Building static websites with Clojure: an update

Wikipedia defines a static web pages as an "web page delivered to a user exactly as it is stored."

To expand on that, a static website is a collection of web pages (HTML documents, CSS, JavaScript code, movies, images, etc.) that link together and are served on the internet. The defining feature of a static website is that it does not connect to a database or generate dynamic content. So the home page is the same for everyone who requests it, rather than something like your Amazon homepage which will be different depending on if you are logged in and what you bought.

It really depends on the generator and what you need/want. Typically, they transform your documents (typically written in some simple markup language like Markdown), to valid HTML (typically using HTML/CSS templates), and output them in a structured way as your website. They generate all the standard content that a web server then serves to users requesting your site.

Depending on what you want/need, they can do a lot of other things too, such as:

• Adding search features
• Categorizing articles
• Organize posts/pages in a structured way
• syntax highlighting (usually with pygments)
• and more…

So what you need to do is take some directory structure (written in the markdown format of your choice or HTML), apply some conversions, and output it somewhere. All the other stuff is nice but extra. You can write your own code to do it somewhere between the conversion (after reading) step and the output step (writing as HTML).

Writing your own is typically a bad idea, there are tons of good, well-used generators (especially Hugo (Go), Jekyll (Ruby), Distill (R), and Pelican (Python)). They will all impose some structure on you, and you will have to learn a lot of configuration options and rules for the specific generator. If you want the flexibility to do whatever you want and you need a side project, writing your own can be a great way to learn.

I was learning Clojure, so I wanted to write my own. I started using Stasis, using this excellent tutorial by Christian Johansen. Here are the basics: Stasis provides a few simple functions to make reading and writing easier, as well as a server so you can get a preview of what the finished product will look like. The only Stasis functions I will use for this example are:

• slurp-directory (read everything in a directory and return a map of {:path "string-of-file-content"})
• merge-page-sources (ensure there are no path conflicts)
• export-pages (write the pages)
• serve-pages (web server to show you a sample)

We'll use Leiningen to manage this project. Install it and type the following:

lein new example-static-website
cd example-static-website

Next, let's add a few dependencies to our project. We will need a markdown to HTML converter (markdown-clj), an HTML generation library for the header/footer (hiccup), and an http server to view our files (ring). We will also add stasis, which has our helper functions for input-output as a dependency to project.clj:

  (defproject example-static-website "0.1.0-SNAPSHOT"
    :description "FIXME: write description"
    :url "http://example.com/FIXME"
    :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
              :url "https://www.eclipse.org/legal/epl-2.0/"}
    :dependencies [[org.clojure/clojure "1.9.0"]
                   [stasis "2.5.0"]
                   [markdown-clj "1.10.5"]
                   [hiccup "1.0.5"]
                   [ring "1.8.2"]]
    :repl-options {:init-ns example-static-website.core})

And in your shell:

lein deps

I told you a static website was a mapping of markup-content -> HTML website. When learning a new tool, it is always nice to have a minimal working project to play with. Let's use the current directory of our Leiningen project:

.
├── CHANGELOG.md
├── LICENSE
├── README.md
├── doc
│   └── intro.md
├── project.clj
├── resources
│   └── index.md
├── src
│   └── example_static_website
│       └── core.clj
├── target
│   |
│   ... 
│ 
└── test
    |
    ...

I added our homepage, index.md to the resources directory. The markdown homepage looks like this:

# Here is the homepage!

Markdown should be converted *italic* properly and **bold**

Here is a:
- list
- of
- stuff

In this case, we will use the resources directory to build the source for our site, and we will publish in a new directory called website. Let's got to src/example_static_website/core.clj and start working on reading and writing it.

    ;; core.clj
    (ns example-static-website.core
      (:require [clojure.string :as str]
                [stasis.core :as stasis]
                [markdown.core :as md]
                [hiccup.page :as hiccup]))

    (def source-dir "resources")

    (stasis/slurp-directory source-dir #".*\.md$")

We defined the source-dir (relative to project root example_static_website), and then we used stasis to read the contents of that directory.

In this case, stasis will return:

{"/index.md" "# Here is the homepage!\n\nMarkdown should be converted *italic* properly and **bold**\n\nHere is a:\n- list\n- of\n- stuff\n\n\n"}

This is a map of the paths (in this case, index.md is at the root /) to a string of the content. If there were more paths or more nesting, those would be read too. For example, let's make a posts directory in resources and put a sample post in it.

# currently in example_static_website
mkdir resources/posts

This file:

# This is my first post

A blogpost will go *here*

And an index (homepage) for the posts:

# Home page for posts. 

Your posts should be displayed here. 


POSTSHERE

So the directory structure now looks like this:

.
├── CHANGELOG.md
├── LICENSE
├── README.md
├── doc
│   └── intro.md
├── project.clj
├── resources
│   ├── index.md
│   └── posts
│       ├── first_post.md
│       └── index.md
├── src
│   └── example_static_website
│       └── core.clj
...

Now if we re-run the slurp-directory command (but only look at keys, for brevity):

  (keys (stasis/slurp-directory source-dir #".*\.md$"))
  ;; ("/posts/first_post.md" "/posts/index.md" "/index.md")

you can see both posts were read, and we have a list of paths and files.

Browsers don't really understand markdown, so we still need to convert this to HTML, and we need to re-name the files from .md to .html.

First thing's first, let's see how to work with the markdown conversion library (markdown-clj) we decided to use.

From the docs, it looks like md-to-html-string is the function we want to use. Here is how it works:

  (md/md-to-html-string "# This should be h1")
  ;; "<h1>This should be h1</h1>"

Great! Now, we know slurp-directory return a map of paths to a string of content, so let's write a function to read the data, break it into keys and values, then apply our conversion function:

  (defn read-and-convert! [src]
    (let [data  (stasis/slurp-directory src #".*\.md$")
          paths (keys data)
          md-content (vals data)]
      (map md/md-to-html-string md-content)))

Clojure programs typically use a bang (! ) in the function name if we are doing a side-effecty thing (in this case reading files). Then we are using a let to break up the map into keys and values, which we will operate on separately.

You can use this function like so:

  (read-and-convert! source-dir)
  ;; ("<h1>This is my first post</h1><p>A blogpost will go <em>here</em></p>" "<h1>Home page for posts.</h1><p>Your posts should be displayed here. </p><p>POSTSHERE</p>" "<h1>Here is the homepage!</h1><p>Markdown should be converted <em>italic</em> properly and <strong>bold</strong></p><p>Here is a:</p><ul><li>list</li><li>of</li><li>stuff</li></ul>")

That looks great, we applied the md-to-html-string function to each item in the seq returned by (vals data) and returned the HTML content of the string. That's half the stuff we want to do in this funtion. The last part involves replacing the .md file endings with .html and returning a new map of the pages.

Let's use the clojure string library to replace the ".md" with ".html".

  (str/replace "index.md" #".md" ".html")
  ;;index.html

This function takes three arguments and operates on a string. We will be mapping it over the seq of keys, so let's wrap it in a function to make that easier

  (defn key-to-html [s]
    (str/replace s #".md" ".html"))

Now, We can modify our original function and make sure everything works:

  (defn read-and-convert! [src]
    (let [data  (stasis/slurp-directory src #".*\.md$")
          paths (keys data)
          html-content (map md/md-to-html-string (vals data))]
      (map key-to-html paths)))

  (read-and-convert! source-dir)
  ;; ("/posts/first_post.html" "/posts/index.html" "/index.html")

Looks great! Notice how I did the markdown conversion in the let statement that splits the values from data. Since we never use the markdown data again in that function, this is a nice way to make the function more compact. Let's now do the same for keys, and use a zipmap to re-join the new keys and values into a map to return:

  (defn read-and-convert! [src]
    (let [data  (stasis/slurp-directory src #".*\.md$")
          html-paths (map key-to-html (keys data))
          html-content (map md/md-to-html-string (vals data))]
      (zipmap html-paths html-content)))

  (read-and-convert! source-dir)
  ;;{"/posts/first_post.html" "<h1>This is my first post</h1><p>A blogpost will go <em>here</em></p>",
  ;; "/posts/index.html" "<h1>Home page for posts.</h1><p>Your posts should be displayed here. </p><p>POSTSHERE</p>",
  ;; "/index.html" "<h1>Here is the homepage!</h1><p>Markdown should be converted <em>italic</em> properly and <strong>bold</strong></p><p>Here is a:</p><ul><li>list</li><li>of</li><li>stuff</li></ul>}

Great, we are well on our way!

We've done the conversion, but the pages are not yet valid HTML. Let's define a simple header and footer using hiccup:

  (defn apply-header-footer [page]
    (hiccup/html5 {:lang "en"}
                  [:head
                   [:title "Static website!"]
                   [:meta {:charset "utf-8"}]
                   [:meta {:name "viewport"
                           :content "width=device-width, initial-scale=1.0"}]
                   [:body
                    [:div {:class "header"}
                     [:div {:class "name"}
                      [:a {:href "/"} "Home page"]
                      [:div {:class "header-right"}
                       [:a {:href "/posts"} "Posts"]]]]
                    page]
                   [:footer
                    [:p "This is the footer"]]]))

  (apply-header-footer "here is some content!")
  ;; "<!DOCTYPE html>\n<html lang=\"en\"><head><title>Static website!</title><meta charset=\"utf-8\"><meta content=\"width=device-width, initial-scale=1.0\" name=\"viewport\"><body><div class=\"header\"><div class=\"name\"><a href=\"/\">Home page</a><div class=\"header-right\"><a href=\"/posts\">Posts</a></div></div></div>Here is some content</body><footer><p>This is the footer</p></footer></head></html>"

Hiccup is a abbreviated html markup. One cool thing about hiccup is that you can embed Clojure forms/functions within the markup, but here we will just be inserting the text (page). In this case, the header and footer are simple, so I just wrote them right in hiccup.

Now, let's write another function to apply the header to the values:

  (defn format-pages [m]
    (let [html-keys (keys m)
          page-data (map apply-header-footer (vals m))]
      (zipmap html-keys page-data)))

This function is similar to read-and-convert!. It takes in a map, breaks apart the pieces, applies the formatting, then outputs the map.

We could use it like so:

  (format-pages (read-and-convert! source-dir))
  ;; {"/posts/first_post.html" "<!DOCTYPE html>\n<html lang=\"en\"> ..."}
  ;; ...

Returning all of our formatted pages as a map. Now would be a good time to commit your work to a version control system like git.

Typically we will be making changes as we go and will want to see how they will look as a website. We can use Stasis and ring to set up a server and do this live. This code is based on Christian's tutorial:

(def server
  (stasis/serve-pages (format-pages (read-and-convert! source-dir))))

The server variable contains our website so far. Now, let's go to project.clj at the root of our project and define a command telling ring to serve it.

(defproject example-static-website "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.9.0"]
                 [stasis "2.5.0"]
                 [markdown-clj "1.10.5"]
                 [hiccup "1.0.5"]
                 [ring "1.8.2"]]
  :ring {:handler example-static-website.core/server}
  :profiles {:dev {:plugins [[lein-ring "0.12.5"]]}}
  :repl-options {:init-ns example-static-website.core})

Now we will have to close our repl connection and run lein deps in our shell. Now, let's test it out:

lein ring server

You should see something like this:

clicking the link for Posts should take you to the posts homepage too.

So this looks functional. However, there is virtually no style so it looks bad. Let's add some custom CSS.

mkdir resources/css

Let's write some basic CSS:

    /* /css/style.css */
     html{
         font-size: 16px;
         font-family: "sans-serif";
         line-height: 1.35;
     }

     body{
         max-width: 600px;
         margin-left: 25%;
         margin-top: 10px;
         margin-bottom: 10px;
     }
    /* and so on... */
    /* see /css/style.css for full css */

Now we have to add this css to our header function.

  (defn apply-header-footer [page]
    (hiccup/html5 {:lang "en"}
                  [:head
                   [:title "Static website!"]
                   [:meta {:charset "utf-8"}]
                   [:meta {:name "viewport"
                           :content "width=device-width, initial-scale=1.0"}]
                   [:link {:type "text/css" :href "/css/style.css" :rel "stylesheet"}] ;; new!
                   [:body
                    [:div {:class "header"}
                     [:div {:class "name"}
                      [:a {:href "/"} "Home page"]
                      [:div {:class "header-right"}
                       [:a {:href "/posts"} "Posts"]]]]
                    page]
                   [:footer
                    [:p "This is the footer"]]]))

And add a function to read the CSS with stasis:

  (defn get-css [src]
    (stasis/slurp-directory src #".*\.css$"))

and then merge this map with our other pages map (after formatting) in a new function called merge-website-sources:

  (defn merge-website-assets! [root-dir]
    (let [page-map (format-pages (read-and-convert! root-dir))
          css-map (get-css source-dir)]
      (stasis/merge-page-sources {:css css-map
                                  :pages page-map})))

stasis/merge-page-sources works just like Clojure's merge but let's you know if there are any path (key) conflicts. We then combined the page reading and css reading in one function. Later, we could add images or JavaScript reading here too and just add new let's and new keys for our website.

We can then change the server to:

  (def server
    (stasis/serve-pages (merge-website-assets! source-dir)))

And refresh the page to get:

Much better!

Note Obviously we can do better than hard-coding the CSS path in the header (as we did in apply-header-footer). We could add additional parameters to apply-header-footer for the links, or have another function transform the header before applying the page. That is left as an exercise for those interested since this is mostly to serve as an example, so we will take the easy, less flexible way and hard code it for now.

You can't read images with stasis/slurp-directory, since it uses Clojure's slurp under the hood, which only works for text. See the stasis docs or Christian Johansen's tutorial for info about using optimus for that. In the simplest case, you could just copy the image directory to the output directory using a Makefile and shell commands if you don't want to use Optimus.

We are almost done! Now we can easily make a local website and experiment with using CSS to make it look nice. Let's write it out so we can put it on the internet

Let's define an output variable called out-dir as the string docs.

;; core.clj
(def out-dir "docs")

Now, we will use basically the same code as we did for the server var in an export! function.

  (defn export! []
    (stasis/empty-directory! out-dir)
    (stasis/export-pages (merge-website-assets! source-dir) out-dir)
    (println "Website is ready!"))

stasis/empty-directory! will delete everything in the out-dir (consider yourself warned), then it will re-build the updated website for us.

At this point, we would push the website directory up to whatever is hosting our static website, and we would be done!

We can set up an extra lein command so we can build the website from the command line as well:

  ;; project.clj
  (defproject example-static-website "0.1.0-SNAPSHOT"
    :description "FIXME: write description"
    :url "http://example.com/FIXME"
    :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
              :url "https://www.eclipse.org/legal/epl-2.0/"}
    :dependencies [[org.clojure/clojure "1.9.0"]
                   [stasis "2.5.0"]
                   [markdown-clj "1.10.5"]
                   [hiccup "1.0.5"]
                   [ring "1.8.2"]]
    :ring {:handler example-static-website.core/server}
    :profiles {:dev {:plugins [[lein-ring "0.12.5"]]}}
    :repl-options {:init-ns example-static-website.core}
    :aliases {"build-site" ["run" "-m" "example-static-website.core/export!"]})

GitHub pages is a nice option for hosting your personal website for free. Once you've built your website using lein build-site from the command line or (export!) from the REPL, push the whole thing up to GitHub in a repository named your-user-name.github.io.

On GitHub, it will look something like this:

Next, go to the settings tab and scroll to Pages, and choose to serve the website from (from your main or master branch) the docs/ directory:

And that's it! You should now see your website show up at your-user-name.github.io!

Need a CNAME for a custom domain? Just add it to your map:

  (defn write-cname [out]
    (spit (str out "/CNAME") "website-url"))

  (defn export! []
    (stasis/empty-directory! out-dir)
    (stasis/export-pages (merge-website-assets! source-dir) out-dir)
    (write-cname out-dir)
    (println "Website is done!"))

Same goes for any extra text file.

That's a custom static website generator in under 60 lines of code! What else can you do? Well, whatever you want! You could add syntax highlighting, or generate a list of your articles to link to from the posts homepage (a good idea)… Most of it is just simple text manipulation (replace something in the HTML text, or use something fancy like Enlive to do some HTML transformations). Just use the same process we used above:

• figure out what you want to do for one string/page (for example, apply-header-footer)
• map that function across all your pages
• chain the function into your main pipeline