{:title "Building my static website with Clojure" :date "2018-08-18" :tags ["clojure"]}

Building my static website with Clojure

Published: 2018-08-18 Sat
last updated: 2018-09-19 Wed

quick note
I made some changes, mostly to align this with what I know now and get rid of some sloppy code. I don't have everything about my website explained here (robots.txt, five links on the home page, and my site map are not included), but I think the hard parts are covered. The most up-to-date and complete version will be the source code https://github.com/nkicg6/website-clj. If I do anything else that I think is interesting or hard, I will add it here in the future. Hopefully it is becoming better-formed and simpler as I go!

Learning through projects

I learn through projects. I have a hard time following the text -> exercises model of learning that is sometimes used in programming and science textbooks, but I found that learning the basics and then jumping into a project is the best (and most fun) way to learn something new.
This particular project is my personal website, which I am rebuilding in Clojure (why Clojure?). I am building a simple, static website to organize my projects and save hints for myself or others for future use.

org-mode

org-mode-unicorn-logo.png

Emacs and org-mode have taken over my life. I will write more about org mode soon, but I plan to write content and publish my site with org-publish, so my org-mode settings will be included throughout this page.

Getting started

The first thing I did was follow along with Christian Johansen's excellent post Building static sites in Clojure with Stasis https://cjohansen.no/building-static-sites-in-clojure-with-stasis/. If you are just getting started, please check this out first!
After following along with that tutorial, I had a much better understanding of what was going on and how things worked. The next step was to think about what I wanted my website and workflow to look like in order to personalize this to my liking. Below are some of my main goals:

  1. Two main categories of posts: Programming and Science.
  2. Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
  3. Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
  4. One-push publishing and integration with org-mode.

Folder layout

To get myself organized, I created the following directory structure within resources/ in my Leiningen project.

resources
├── home
├── org-programming
├── org-science
├── programming
├── public
│   ├── css
│   ├── img
│   └── js
├── science
├── staging-in-progress
└── test
  • home/ contains only the index.html for the homepage.
  • org-programming/ contains the index.html for the programming posts, as well as all of my org-mode write-ups for programming posts.
  • org-science/ is the same as org-programming but for science posts.
  • The corresponding programming/ and science/ pages will contain the html versions of the posts from org-programming/ and org-science. These html pages will be created using org-publish.
  • public/ contains all my static images, css, and js files.
  • staging-in-progress/ contains all the posts I am workin on but don't want to be exported and published yet.
  • test/ just contains test html and org stuff that will not be included in the real site. I just use it for testing.

For reference, here is what the complete directory tree looks like:

├── doc
│   └── img
├── resources
│   ├── home
│   ├── org-programming
│   ├── org-science
│   ├── programming
│   ├── public
│   │   ├── css
│   │   ├── img
│   │   └── js
│   ├── science
│   ├── staging-in-progress
│   └── test
├── src
│   └── website_clj
├── target
└── test
    └── website_clj

Org-publish

Org-mode worg has a nice intro post on org-publishing https://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.html. Starting with that, the emacs-lisp for my export looks like so:

;; emacs-lisp
;; ~/.emacs.d/config.org
(setq org-publish-project-alist
      '(("programming"
         :base-directory "~/personal_projects/website-clj/resources/org-programming"
         :base-extension "org"
         :publishing-directory "~/personal_projects/website-clj/resources/programming"
         :publishing-function org-html-publish-to-html
         :headline-levels 4
         :html-extension "html"
         :body-only t)
        ("science"
         :base-directory "~/personal_projects/website-clj/resources/org-science"
         :base-extension "org"
         :publishing-directory "~/personal_projects/website-clj/resources/science"
         :publishing-function org-html-publish-to-html
         :headline-levels 4
         :html-extension "html"
         :body-only t)
        ("clj-site" :components ("programming" "science"))))

From Emacs, M-x org-publish and I select the project clj-site this will compile (trans-pile?) all the .org files in org-programming/ and org-science/ to html's in the respective programming/ and science/ folders. Note that I set the :body-only argument to t, as all these files will inherit a hiccup based header and footer in my site generation code.

Preaparing pages

Before getting started, I have my src/ directory tree here:

src
└── website_clj
    ├── export_helpers.clj
    ├── process_pages.clj
    └── web.clj

web.clj contains the main site building and export logic. process_pages.clj contains functions for formatting html, parsing edn, and applying the header and footer. export_helpers.clj contains functions for exporting to host on github pages. I'll go over most of these here.

One of the workhorse functions in stasis is stasis/slurp-directory. It takes as arguments the path to a directory and a regex pattern to match and returns a map of {file1-path html1-text ...} for all matching files. I already have my programming posts and science posts in separate directories, so I will use stasis/slurp-directory to read those into two separate maps. This is a very simple and easy to work-with representation of pages, where the path is just root/stasis-map-key. For a page named page1.html, this would be root/page1.html, where root is the url (your page address or localhost:XXXX). Great, so if I want all the programming posts to have /programming/ prepended to them and all the science posts to have /science/ prepended to the address, I can write a really simple function to make this happen.

Note: I am trying to follow the clojure style guide's documentation guidelines.

;; process_pages.clj ns
(ns website-clj.process-pages
  (:require [clojure.string :as str]
            [hiccup.core :refer [html]]
            [hiccup.page :use [html5 include-css include-js]]
            [hiccup.element :refer (link-to image)]
            [net.cgrand.enlive-html :as enlive]
            [clojure.edn :as edn] 
            [stasis.core :as stasis]))  ;; only for testing?

;;--- snip ---

(defn format-images [html] ;; 1
  "formats html image link to appropriately link to static website image directory.
  `html` is a raw html string."
  (str/replace html #"src=\"img" "src=\"/img")) 


(defn format-html ;; 2
  "Composed function to apply multiple html processing steps to raw html.
  `html` is a raw html string."
  [html]
  (-> html
      format-images
      layout-base-header)) ;; other fns for html here 

(defn fmt-page-names  ;; 3
  "removes .html from all non-index.html pages.
  `base-name` is whatever base name you want the string to have prepended to it. 
  `name` is a string."
  [base-name name]
  (str base-name
       (str/replace name #"(?<!index)\.html$" "")))

(defn html-pages ;; 4
  "Composed function that performs html formatting to a map of strings for my blog.
  The argument `base-name` is a new string that will be prepended to all keys in the 
  `page-map` map argument. `page-map` is a map created by the function `stasis/slurp-directory`. 
  The purpose of `html-pages` is to apply formatting to html pages meant for different sections
  of my website. For instance, calling `html-pages` with '/programming' and the a map of pages will prepend 
  '/programming/<page-name>' to every key in the map and strip the html end off all non-index pages."
  [base-name page-map]
  (zipmap (map #(fmt-page-names base-name %) (keys page-map))
          (map #(format-html %) (vals page-map))))

I'll break down these functions briefly, and note that most of them work only on the raw html strings or key name strings returned from the stasis/slurp-directory function.

  1. format-images is simply to fix a silly formatting problem when exporting my image links from org-mode to html (see org-workflow: Handling images). I think it is self explanatory.
  2. format-html will be a function that simply composes other small html formatting functions I may want to use in the future. Right now, I only have format-images and layout-base-header. If I need more in the future, it will be trivial to write and apply them without breaking upstream code (as long as I take and return html strings). Really nice consequence of dealing with simple values rather than objects.
  3. fmt-page-names As the documentation says, this removes html from all html pages that do not contain index in them, and then prepends some base-name to all pages. The pages that are already named index.html are pre-made pages that I have as the landing pages for those subjects. These pages need to retain the .html file endings in order to render as index pages correctly. All others can have the .html endings removed. This allows me to prepend /programming/ to all pages in the programming folder, and the same for science.
  4. html-pages is another composed function of all of the above functions. Instead of taking and returning a string, it takes and returns a map (which comes from stasis/slurp-directory). Just to demonstrate how this is used, I'll show you reading in pages in web.clj:
;;;; web.clj
(ns website-clj.web
  "main namespace for building and exporting the website"
  (:require [optimus.assets :as assets]
            [optimus.export]
            [optimus.link :as link] 
            [optimus.optimizations :as optimizations]      
            [optimus.prime :as optimus]                    
            [optimus.strategies :refer [serve-live-assets]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [stasis.core :as stasis]
            [website-clj.export-helpers :as helpers]
            [website-clj.process-pages :as process]))


;; define page maps and link maps

(def programming-map
  (process/html-pages "/programming"
                      (stasis/slurp-directory "resources/programming" #".*\.(html|css|js)")))

;; --- snip ---

Awesome. All of my html formatting and reading in one place.

A quick note about images and resources

Although this seems simple in hindsight, it caused me a significant amount of headaches and some time to figure out.
Looking back at my folder layout in the resources directory:

resources
├── home
├── org-programming
├── org-science
├── programming
├── public
│   ├── css
│   ├── img
│   └── js
├── science
└── test

How do you refer to images from a post in html?
My first thought was this

<h1>This is the landing page</h1>
<p>
 Welcome to it.

Here is a test image:

 <img src="../public/img/sample-img.png" alt="sample img!" />
</p>

As I figured the working directory was within whatever page you were at, and then I just followed the path to img. But that does not work. Finally I figured out that images can be added by referring to them relative to public as the working directory. For example:

<img src="/img/sample-img.png" alt="sample img!" />

inserts the image stored in public/img/test-img.png. This makes sense (in hindsight), as with stasis I slurped the whole public/ directory.

org-workflow: Handling images

How does this factor into my org-mode workflow?
Let's say I have an example org-mode file, and I'll add an image in org-markup manner.

#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{}
#+HTML_CONTAINER: div



=* This is a test post
Here is a test post and a link to an image. 


[[file:~/personal_projects/website-clj/resources/public/img/test-img.png]]

Exporting this to html gives the following link structure in HTML:

<img src="/img/test-img.png" alt="test-img.png"/>

While this is almost right, it doesn't render properly because all images are referred to /img/. To fix it, I wrote a link formatting function in Preaparing pages called format-images. Now to gather and serve all the resources, I have a function called get-assets which grabs everything from public/ and hands it to optimus for frontend optimizations.

;;;; web.clj
(ns website-clj.web
  "main namespace for building and exporting the website"
  (:require [optimus.assets :as assets]
            [optimus.export]
            [optimus.link :as link] 
            [optimus.optimizations :as optimizations]      
            [optimus.prime :as optimus]                    
            [optimus.strategies :refer [serve-live-assets]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [stasis.core :as stasis]
            [website-clj.export-helpers :as helpers]
            [website-clj.process-pages :as process]))

;; --- snip ---

(defn get-assets
  "get all static assets from the public directory."
  []
  (assets/load-assets "public" [#".*"]))

;;--- snip ---
;; for test rendering
(def app
  "renders the website for experimentation"
  (optimus/wrap
   (stasis/serve-pages get-pages)
   get-assets
   optimizations/none
   serve-live-assets))

get-assets is likely why I refer to images as /img/image.png instead of public/img/image.png.

org-workflow: syntax highlighting

Christian Johanson has an excellent description of formatting markdown fenced code blocks with pygments for nice display on his static site. His approach uses pygments and enliven and is very detailed and nice. However, the amazing org-mode takes care of syntax highlighting for me when I add (setq org-src-fontify-natively t) to my config.org. So here I will just test it real quick and see how it looks.

In my HTML file, I will add a clojure code block like so:

#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{}
#+HTML_CONTAINER: div




=* This is a test post
Here is a test post and a link to an image. 


[[file:~/personal_projects/website-clj/resources/public/img/test-img.png]]

And below is a test code block. 

#+BEGIN_SRC clojure 
(defn format-images [html]
  (str/replace html #"file:///Users/Nick/personal_projects/website-clj/resources/public" ""))

;; main pages function.
(defn html-pages [pages]
  (zipmap (map #(str/replace % #"\.html$" "") (keys pages))
          (map #(fn [req] (layout-base-header req %))
               (map format-images (vals pages)))))
#+END_SRC

How does it look?

This renders upon M-x org-publish-project clj-site to look like this:

syntax-highlighting.png

org-src-fontify-natively uses the currently active theme to highlight your source code. I just exported this using the Leuven theme (great for org-mode) and I like the way it looks. However, if I wanted to change it and use enliven with pygments, I would probably use some emacs-lisp code and packages such as those described here: https://emacs.stackexchange.com/questions/31439/how-to-get-colored-syntax-highlighting-of-code-blocks-in-asynchronous-org-mode-e , but for right now I dont think this is necessary for me so I will go with the raw html formatting from org-export.

You can see the source code for my project here

Summing up

In Preaparing pages we addressed reading in pages with stasis, formatting html, syntax highlighting and adding resources like images. I think we can cross #1 off our list.

  1. Two main categories of posts: Programming and Science.
  2. Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
  3. Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
  4. One-push publishing and integration with org-mode.

The next section will address the metadata-related goals.

Parsing edn metadata

Most static site generators (Jekyll, for instance) contain some way to add metadata in markup format to posts in order to set formatting options, apply themes, add a name, etc. So referring back to my list of goals for my site:

  1. Two main categories of posts: Programming and Science.
  2. Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
  3. Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
  4. One-push publishing and integration with org-mode.

I want to automatically generate a list of posts in reverse chronological order on index pages for the programming and science sections. In order to do this, metadata would be nice, and Clojure offers an excellent solution in the form of extensible data notation or edn. In this section I'll be tackling both 2 and 3.

setting up the metadata in org-mode

First off, I'll put the metadata in a div at the top of my document with the id as edn. Since I write in org-mode, I made a YASnippet (awesome emacs templates, check them out) called blog:

;; yas snippet blog
# -*- mode: snippet -*-
# name: blog
# key: blog
# --
#+HTML: <div id="edn">
#+HTML: {:title "${1:title}" :date "`(format-time-string "%Y-%m-%d")`" :tags ${2:["clojure"]}}
#+HTML: </div>
#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{}
#+PROPERTY: header-args :eval never-export
$0

when I type blog<TAB> it expands to the following

#+HTML: <div id="edn">
#+HTML: {:title "title" :date "2018-08-19" :tags ["clojure"]}
#+HTML: </div>
#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{}
#+PROPERTY: header-args :eval never-export

The important part here is the #+HTML: sections. That tag tells org-mode to export that line as literal HTML. This creates a unique div id containing the metadata with a shorter title for the post, the date (automatically generated with inline emacs-lisp), and a vector of tags. For now I will only deal with the title and date, but I will likely start doing something with the tags vector later.

parsing edn with enlive

So we added metadata under a special tag, but how do we parse it?
The functions I write work with either raw html text, or with the map of {file1-name html1-text ...} returned by the function stasis/slurp-directory, as discussed in Preaparing pages. In order to parse these, I'll use enlive, the amazing selector-based templating and html transformation library.
I'll add some edn metadata to a test html page and start playing. First I need to add enlive to my project.clj

;; project.clj
(defproject website-clj "0.1.0-SNAPSHOT"
  :description "Personal website built with Clojure, Stasis, and Hiccup"
  :url "http://nickgeorge.net"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [optimus "0.14.2"]
                 [enlive "1.1.6"]]
  :ring {:handler website-clj.web/app}
  :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}}
  :aliases {"build-site" ["run" "-m" "website-clj.web/export"]})

and then run lein deps at the command line. I'd recommend going through this enlive tutorial to figure out how to parse with enlive. For REPL based play and testing, my test.org doc looks like this:

#+HTML: <div class="edn">
#+HTML: {:title "renamed" :date "2018-08-05" :tags ["clojure" "testing" "post"]}
#+HTML: </div>
#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{}

Here is my test content

and some code 

#+BEGIN_SRC clojure
(test clj-code)
(def test-me "test string")
#+END_SRC

and I put that in the test-org/ folder and added an export line to my org-mode export file in the project clj-site.
When I run org-publish-project clj-site I get this:

<div class="edn">
{:title "renamed" :date "2018-08-05" :tags ["clojure" "testing" "post"]}
</div>

<div id="outline-container-orgd13af6f" class="outline-2">
<h2 id="orgd13af6f">Here is my test content</h2>
<div class="outline-text-2" id="text-orgd13af6f">
<p>
and some code<br />
</p>

<div class="org-src-container">
<pre class="src src-clojure"><span style="color: #707183;">(</span><span style="color: #006FE0;">test</span> clj-code<span style="color: #707183;">)</span>
<span style="color: #707183;">(</span><span style="color: #0000FF;">def</span> <span style="color: #BA36A5;">test-me</span> <span style="color: #036A07;">"test string"</span><span style="color: #707183;">)</span>
</pre>
</div>
</div>
</div>

With the useful stuff at the top in the tag. I made a new test folder for this, and I moved the test.html there. So now, I'll read that in and start messing around.

I am playing with this code at the bottom of my new process-clj namespace.

;;;; process_pages.clj
;; first step is slurping a directory, applying the path prefix and formatting html. Make sure to add stasis to the ns declaration for testing!

(def slurped-raw
  "holds a map of formatted html pages for my website"
  (html-pages "/test" (stasis/slurp-directory "resources/test" #".*\.(html|css|js)"))) 

(keys slurped-raw)
;; => ("/test/index.html" "/test/test" "/test/test2")
(vals slurped-raw)
;; => html for the pages

;; isolate html for one page

(def test-html (second (vals slurped-raw)))
test-html
;; => html for page /test/test

Now I have raw html to play with.
Figuring out the parsing took some time, but eventually I figured out this code:

;;;; process_pages.clj

(ns website-clj.process-pages
  (:require [clojure.string :as str]
            [hiccup.core :refer [html]]
            [hiccup.page :use [html5 include-css include-js]]
            [hiccup.element :refer (link-to image)]
            [net.cgrand.enlive-html :as enlive]
            [clojure.edn :as edn] 
            [stasis.core :as stasis]))

;; --- snip ---

(defn parse-html
  "Takes raw html and returns keys from edn metadata under the <div id='edn'> html tag
  `html` is raw html"
  [html]
  (as-> html raw-text ;; 1
    (enlive/html-snippet raw-text) ;; 2
    (enlive/select raw-text [:#edn enlive/text-node]) ;; 3
    (apply str raw-text) ;; 4
    (edn/read-string raw-text))) ;; 5


;; --- snip ---

;; isolate html for one page

(def test-html (second (vals slurped-raw)))
;; => html for page /test/test

(def metadata (parse-html test-html))

metadata
;; => {:title "renamed" :date "2018-08-05" :tags ["clojure" "testing" "post"]}

Going through parse-html

  1. start threading the html using the as-> macro. I recently refactored to use this rather than thread first -> because I have one place (#4) where the html needs to be the last argument. Rather than mixing thread first and thread last, I used as-> to overcome this limitation.
  2. turn the html into an enlive/html-snippet. As far as I know, this parses the html for enlive.
  3. use enlive to get the relevant node base on id. You select based on the div id with #id-name. This part is still a little confusing for me…
  4. Now I need to turn that into a string, so I use apply str
  5. uses edn/read-string to parse the resulting string into a clojure map. Note all of my edn metadata will be represented as strings or vectors/lists of strings for now, as I can select some for inserting later.

edn is parsed and in memory, though in order to use it in practice I'll make one more function that takes in the map returned by stasis/slurp-directory, and returns a map of maps with the metadata.

In practice, /programming/index.html will live in the programming/ directory that is parsed by my edn metadata parser. That means if I make links based on the raw output of stasis/slurp-directory I would get a link for the index page, on the index page, which is sloppy. The function remove-index removes the index page.

;;;; process_pages.clj

;; --- snip ---

;; remove index page
(defn remove-index
  "Removes /index.html from map that will be parsed for edn metadata.
  `base-name` is the name prepended to the index.html page. For programming pages it will be '/programming'
  `page-map` is the map returned by `html-pages`. returns `page-map` minus the index pages."
  [base-name page-map]
  (dissoc page-map (str base-name "/index.html")))

This function simply joins the base-name (i.e. "/programming") to the string "/index.html" and removes it from the map.

The function make-edn-page-map works directly with the map from stasis/slurp-directory, and it returns a map of maps, with {page-name1 metadata1 ...}

(defn make-edn-page-map
  "filters the `page-map` to remove index.html and returns a map of page names and edn metadata.
  `page-map` is returned by `stasis/slurp-directory`. 
  `base-name` provides the prepended base for the directory you are filtering by with `remove-index`"
  [base-name page-map]
  (let [filtered-page-map (remove-index base-name page-map)] ;; 1
    (zipmap (keys filtered-page-map) ;; 2
            (map parse-html (vals filtered-page-map))))) ;; 3

;; --- snip ---

;; useage

(def metadata (parse-edn ("/test" slurped-raw)))
metadata
;; => {"/test/test" {:title "renamed" :date "2018-08-05" :tags ["tag1" "tag2"]}, "/test/test2" {:title "renamed2" :date "2018-08-06" ...}}
  1. applies remove-index to the page-map.
  2. Use the keys from the newly filtered page map as the keys in the new map
  3. apply parse-html to the values of the filtered-page-map. This will be the values for the new map.

Hide the metadata

I don't want the metadata showing up at the top of every page. I made a css file called custom.css and had it hide all the id=edn div's.

// css/custom.css
#edn {
    display: none;
}

Easy. Now I will use the include-css hiccup header and add the following to my hiccup-defined header:

;; process-pages ns
(ns website-clj.process-pages
  (:require [clojure.string :as str]
            [hiccup.core :refer [html]]
            [hiccup.page :use [html5 include-css include-js]] ;; include hiccup helpers
            [hiccup.element :refer (link-to image)]
            [net.cgrand.enlive-html :as enlive]
            [clojure.edn :as edn] 
            [stasis.core :as stasis] ;; only for testing?
            ))

;; --- snip ---
(defn layout-base-header [request page]
  (html5
   [:head
    [:meta {:charset "utf-8"}]
    ;;... --- snip ---
    (include-css "/css/custom.css") ;; the new stuff
    ;;... --- snip ---
    ]
   ;;Much more here, I cut it out for simplicity
   ))

Page titles

Now to make good, well-formed html you have to have one last part– a <title>. To add a title, I will use the parsed edn metadata similar to how I did it with the page links.

We have already written parse-html, a function which returns the metadata map. I added a title tag to my layout-base-header that looks like this [:title "Nick's site"]. This is there both to provide a template for enlive to replace, and to ensure that every page has a title even if I am missing a title in the metadata (I read that it is good for search engine ratings to have well-formed html).
Next I wrote insert-page-title:

;; process_pages.clj

;; ---snip---
(defn parse-html
  "Takes raw html and returns keys from edn metadata under the <div id='edn'> html tag
  `html` is raw html"
  [html]
  (as-> html raw-text
    (enlive/html-snippet raw-text)
    (enlive/select raw-text [:#edn enlive/text-node])
    (apply str raw-text)
    (edn/read-string raw-text)))


(defn insert-page-title
  "`insert-page-title` parses edn metadata and return the html with a title inserted
  `page` is the raw HTML of a page including the header."
  [page]
  (let [meta-title (get (parse-html page) :title "Nick's site")] ;; 1
    (-> page ;; 2
        (enlive/sniptest [:title]
                         (enlive/html-content meta-title))))) ;; 3
  1. parse the html and grab the :title tag. Store in a let as meta-title. Note that I provided a default value for get in case I didn't put a title in the metadata.
  2. Start threading with the raw html (note this will have to be after I applied layout-base-header).
  3. swap out the title tag contents with my parsed title.

Great! Now since this is another html processing function, I can add it to format-html to put it right into the normal pipeline without changing anything else!

;; process_pages.clj

;; --- snip --- 

(defn format-images [html]
  "formats html image link to appropriately link to static website image directory.
  `html` is a raw html string."
  (str/replace html #"src=\"img" "src=\"/img"))

;; --- edn parsing for metadata---

;; !!! use as-> instead! see https://learnxinyminutes.com/docs/clojure/
(defn parse-html
  "Takes raw html and returns keys from edn metadata under the <div id='edn'> html tag
  `html` is raw html"
  [html]
  (as-> html raw-text
    (enlive/html-snippet raw-text)
    (enlive/select raw-text [:#edn enlive/text-node])
    (apply str raw-text)
    (edn/read-string raw-text)))

(defn insert-page-title
  "`insert-page-title` parses edn metadata and return the html with a title inserted
  `page` is the raw HTML of a page including the header."
  [page]
  (let [meta-title (get (parse-html page) :title "Nick's site")]
    (-> page
        (enlive/sniptest [:title]
                         (enlive/html-content meta-title)))))

(defn format-html 
  "Composed function to apply multiple html processing steps to raw html.
  `html` is a raw html string."
  [html]
  (-> html
      format-images
      layout-base-header
      insert-page-title)) ;; other fns for html here

Note that insert-page-title is after the base header is applied.

Now we have almost all the parts we need. I'll go over some caveats for publishing with GitHub Pages in Exporting for GitHub Pages, then I demonstrate the workflow in Bringing it all together.

Summing up parsing

In this section, we set up a system for adding edn metadata to files, we parsed the metadata, made a list of links, sorted them, and inserted them into our document. Check a few more options off out list!

  1. Two main categories of posts: Programming and Science.
  2. Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
  3. Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
  4. One-push publishing and integration with org-mode.

Publishing is up next.

Exporting for GitHub Pages

publishing

From lein, Christian gives some nice instructions, so I followed those to see how the export looks and it seems to work nicely. Now, I'd like put my website on-line. I hosted my previous site on GitHub Pages, so I know I need a few config items for hosting. The first is the CNAME file, for mapping your domain name to the github repo.

In Christian's example, he empties the target export directory with (stasis/empy-directory!) before the rest of the export. I definitely want to do this, but looking into the Stasis code, I don't see any options to exclude certain files. That means my CNAME, .gitignore, and .git repo will be wiped out every time I build! No good for GitHub Pages

I decided to use shell commands to get around this for the moment, and I broke these functions out into a namespace called export-helpers.

CNAME and .gitignore will live in the resources/ and target/ directories, respectively. Upon export, they will be copied to the export directory like so

;;;; export_helpers.clj

(ns website-clj.export-helpers
  "helper functions for saving the git directory, cname, and gitignore from `stasis/empty-directory!`
  This exists to help with rendering static sites on github." 
  (:require [clojure.string :as str]
            [clojure.java.shell :as shell])) ;; for shell commands from clojure


(defn cp-cname
  "copy the CNAME file to the export directory.
  `export-dir` is a var that contains the parth to the base of the website. 
  CNAME must be in the directory for github pages domain mapping."
  [export-dir]
  (shell/sh "cp" "resources/CNAME" (str export-dir "/CNAME")))

(defn cp-gitignore
  "copy the gitignore file from a safe location to the base of the github pages repo for rendering."
  [export-dir]
  (shell/sh "cp" "target/.gitignore" (str export-dir "/.gitignore")))

;; --- snip ---

Handling the git repo is a little trickier, as I don't want to maintain the git repo elsewhere. Instead, I made two functions: one to copy .git to a save place, and another to restore it after building.

;;;; export_helpers.clj

;; --- snip ---
(defn save-git
  "copy .git repo to a safe directory to save it from deletion. 
  `safe-dir` is a path to a directory that will not be emptied by `stasis/empty-directory!`
  `export-dir` is the export directory where your site will be made."
  [safe-dir export-dir] 
  (shell/sh "mv" (str export-dir "/.git") (str safe-dir "/.git")))

(defn replace-git
  "Puts the gir directory back into the export directory.
  `safe-dir` is a path to a directory that will not be emptied by `stasis/empty-directory!`
  `export-dir` is the export directory where your site will be made."
  [safe-dir export-dir]
  (shell/sh "mv" (str safe-dir "/.git") (str export-dir "/.git")))

Here is how these will be used in the final product:

(ns website-clj.web
  "main namespace for building and exporting the website"
  (:require [optimus.assets :as assets]
            [optimus.export]
            [optimus.link :as link] 
            [optimus.optimizations :as optimizations]      
            [optimus.prime :as optimus]                    
            [optimus.strategies :refer [serve-live-assets]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [stasis.core :as stasis]
            [website-clj.export-helpers :as helpers]
            [website-clj.process-pages :as process]))

;; --- snip --- 


;; constants for exporting
(def export-dir "target/nickgeorge.net")
(def safe-dir "target")

;; main export function, called by lein build-site
(defn export
  "main export function for static site. See docs for functions included.
  `website-clj.helpers/save-git`
  `website-clj.helpers/cp-cname`
  `website-clj.helpers/cp-gitignore`
  `website-clj.helpers/replace-git`"
  []
  (helpers/save-git safe-dir export-dir)
  (let [assets (optimizations/all (get-assets) {})]
    (stasis/empty-directory! export-dir)
    (optimus.export/save-assets assets export-dir)
    (stasis/export-pages (get-pages) export-dir {:optimus-assets assets}))
  (helpers/cp-cname export-dir)
  (helpers/cp-gitignore export-dir)
  (helpers/replace-git safe-dir export-dir))

This is super hacky and not optimal. It would be better to edit stasis/empty-directory! to include arguments for excluding certain dirs/files– but for now this works.

Bringing it all together

So how does this look in practice? Well just check out my web.clj source. The general format goes like so:

  1. Use stasis/slurp-directory and process-pages/html-pages to read and format the pages for each subject.
  2. pass the resulting map into process-pages/parse-edn to get the metadata map.
  3. pass the metadata map into process-pages/format-html-links to make the html links.
  4. make the get-pages function read all the relevant directories. This is where we also apply process-pages/add-links.
  5. export for serving.
;;;; web.clj

(ns website-clj.web
  "main namespace for building and exporting the website"
  (:require [optimus.assets :as assets]
            [optimus.export]
            [optimus.link :as link] 
            [optimus.optimizations :as optimizations]      
            [optimus.prime :as optimus]                    
            [optimus.strategies :refer [serve-live-assets]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [stasis.core :as stasis]
            [website-clj.export-helpers :as helpers]
            [website-clj.process-pages :as process]))

;; define page maps and link maps
;; define page maps and link maps

(def programming-map
  "constant for all links holding programming pages"
  (process/html-pages "/programming"
                      (stasis/slurp-directory "resources/programming" #".*\.(html|css|js)")))
(def programming-metadata
  "constant for all programming-metadata"
  (process/make-edn-page-map "/programming" programming-map))

(def programming-links
  "constant for all programming links"
  (process/format-html-links programming-metadata))


;; repeat for science...

;; --- snip --
;; load all assets
(defn get-assets
  "get all static assets from the public directory."
  []
  (assets/load-assets "public" [#".*"]))

;; main get pages function for render and export
(defn get-pages ;; 4
  "Gathers all website pages and resources."
  []
  (stasis/merge-page-sources
   {:public (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$") 
    :landing (process/home-page
              (stasis/slurp-directory "resources/home" #".*\.(html|css|js)$"))
    :programming  (zipmap (keys programming-map)
                          (map #(process/add-links % programming-links :#pageListDiv)
                               (vals programming-map)))
    :science (zipmap (keys science-map)
                     (map #(process/add-links % science-links :#pageListDiv)
                          (vals science-map)))}))

;; --- snip ---

(defn export ;; 5
  "main export function for static site. See docs for functions included.
  `website-clj.helpers/save-git`
  `website-clj.helpers/cp-cname`
  `website-clj.helpers/cp-gitignore`
  `website-clj.helpers/replace-git`"
  []
  (helpers/save-git safe-dir export-dir)
  (let [assets (optimizations/all (get-assets) {})]
    (stasis/empty-directory! export-dir)
    (optimus.export/save-assets assets export-dir)
    (stasis/export-pages (get-pages) export-dir {:optimus-assets assets}))
  (helpers/cp-cname export-dir)
  (helpers/cp-gitignore export-dir)
  (helpers/replace-git safe-dir export-dir))

One push publishing with Leiningen :alias

IN PROGRESS!! 2018-09-19

I can already build my website with my current alias, now I will make another to deploy!
The steps I need to do are:

  1. Command line build org-project
    • org-publish-project clj-site from the command line
    • remember to add a header to tell org to not evaluate code like this: #+PROPERTY: header-args :eval never-export
    • This should be a clojure function called with export from build-site
  2. Then run build-site
  3. git add and git push all changes.
    • This could also be a clojure function called with export from build-site

The idea is that I just call build-site and it all happens automatically when I run lein build-site

Right now, to publish, I run:

  1. org-publish-project clj-site from emacs.
  2. lein build-site from the command line in website-clj/ dir.
  3. cd into target/nickgeorge.net/ then git add ., git commit -m "message", and git push.

Further improvements

Will be posted here. On the near horizon:

  1. Tests!