Please see my UPDATED VERSION and not this one! ¶
I wrote an updated version of this https://nickgeorge.net/programming/custom-static-clojure-websites-an-update/ that is much more clear and better organized. Please see that post, I've learned a lot since I wrote this one.
Learning through projects ¶
I learn through projects. I have a hard time following the text -> exercises
model of learning that is sometimes used in programming and science textbooks, but I found that learning the basics and then jumping into a project is the best (and most fun) way to learn something new.
This particular project is my personal website, which I am rebuilding in Clojure (why Clojure?). I am building a simple, static website to organize my projects and save hints for myself or others for future use.
org-mode ¶
Emacs and org-mode have taken over my life. I will write more about org mode soon, but I plan to write content and publish my site with org-publish
, so my org-mode settings will be included throughout this page.
Getting started ¶
The first thing I did was follow along with Christian Johansen's excellent post Building static sites in Clojure with Stasis https://cjohansen.no/building-static-sites-in-clojure-with-stasis/. If you are just getting started, please check this out first! After following along with that tutorial, I had a much better understanding of what was going on and how things worked. The next step was to think about what I wanted my website and workflow to look like in order to personalize this to my liking. Below are some of my main goals:
- Two main categories of posts:
Programming
andScience
. - Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
- Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
- One-push publishing and integration with org-mode.
Folder layout
To get myself organized, I created the following directory structure within resources/
in my Leiningen project.
resources ├── home ├── org-programming ├── org-science ├── programming ├── public │ ├── css │ ├── img │ └── js ├── science ├── staging-in-progress └── test
home/
contains only theindex.html
for the homepage.org-programming/
contains theindex.html
for the programming posts, as well as all of myorg-mode
write-ups for programming posts.org-science/
is the same asorg-programming
but for science posts.- The corresponding
programming/
andscience/
pages will contain thehtml
versions of the posts fromorg-programming/
andorg-science
. Thesehtml
pages will be created usingorg-publish
. public/
contains all my static images, css, and js files.staging-in-progress/
contains all the posts I am workin on but don't want to be exported and published yet.test/
just contains test html and org stuff that will not be included in the real site. I just use it for testing.
For reference, here is what the complete directory tree looks like:
├── doc │ └── img ├── resources │ ├── home │ ├── org-programming │ ├── org-science │ ├── programming │ ├── public │ │ ├── css │ │ ├── img │ │ └── js │ ├── science │ ├── staging-in-progress │ └── test ├── src │ └── website_clj ├── target └── test └── website_clj
Org-publish
Org-mode worg has a nice intro post on org-publishing https://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.html. Starting with that, the emacs-lisp for my export looks like so:
;; emacs-lisp
;; ~/.emacs.d/config.org
(setq org-publish-project-alist
'(("programming"
:base-directory "~/personal_projects/website-clj/resources/org-programming"
:base-extension "org"
:publishing-directory "~/personal_projects/website-clj/resources/programming"
:publishing-function org-html-publish-to-html
:headline-levels 4
:html-extension "html"
:body-only t)
("science"
:base-directory "~/personal_projects/website-clj/resources/org-science"
:base-extension "org"
:publishing-directory "~/personal_projects/website-clj/resources/science"
:publishing-function org-html-publish-to-html
:headline-levels 4
:html-extension "html"
:body-only t)
("clj-site" :components ("programming" "science"))))
From Emacs, M-x org-publish
and I select the project clj-site
this will compile (trans-pile?) all the .org
files in org-programming/
and org-science/
to html
's in the respective programming/
and science/
folders. Note that I set the :body-only
argument to t
, as all these files will inherit a hiccup based header and footer in my site generation code.
Preaparing pages ¶
Before getting started, I have my src/
directory tree here:
src └── website_clj ├── export_helpers.clj ├── process_pages.clj └── web.clj
web.clj
contains the main site building and export logic. process_pages.clj
contains functions for formatting html, parsing edn, and applying the header and footer. export_helpers.clj
contains functions for exporting to host on github pages. I'll go over most of these here.
One of the workhorse functions in stasis is stasis/slurp-directory
. It takes as arguments the path to a directory and a regex pattern to match and returns a map of {file1-path html1-text ...}
for all matching files. I already have my programming posts and science posts in separate directories, so I will use stasis/slurp-directory
to read those into two separate maps. This is a very simple and easy to work-with representation of pages, where the path is just root/stasis-map-key
. For a page named page1.html
, this would be root/page1.html
, where root
is the url (your page address or localhost:XXXX). Great, so if I want all the programming posts to have /programming/
prepended to them and all the science posts to have /science/
prepended to the address, I can write a really simple function to make this happen.
Note: I am trying to follow the clojure style guide's documentation guidelines.
;; process_pages.clj ns
(ns website-clj.process-pages
(:require [clojure.string :as str]
[hiccup.core :refer [html]]
[hiccup.page :use [html5 include-css include-js]]
[hiccup.element :refer (link-to image)]
[net.cgrand.enlive-html :as enlive]
[clojure.edn :as edn]
[stasis.core :as stasis])) ;; only for testing?
;;--- snip ---
(defn format-images [html] ;; 1
"formats html image link to appropriately link to static website image directory.
`html` is a raw html string."
(str/replace html #"src=\"img" "src=\"/img"))
(defn format-html ;; 2
"Composed function to apply multiple html processing steps to raw html.
`html` is a raw html string."
[html]
(-> html
format-images
layout-base-header)) ;; other fns for html here
(defn fmt-page-names ;; 3
"removes .html from all non-index.html pages.
`base-name` is whatever base name you want the string to have prepended to it.
`name` is a string."
[base-name name]
(str base-name
(str/replace name #"(?<!index)\.html$" "")))
(defn html-pages ;; 4
"Composed function that performs html formatting to a map of strings for my blog.
The argument `base-name` is a new string that will be prepended to all keys in the
`page-map` map argument. `page-map` is a map created by the function `stasis/slurp-directory`.
The purpose of `html-pages` is to apply formatting to html pages meant for different sections
of my website. For instance, calling `html-pages` with '/programming' and the a map of pages will prepend
'/programming/<page-name>' to every key in the map and strip the html end off all non-index pages."
[base-name page-map]
(zipmap (map #(fmt-page-names base-name %) (keys page-map))
(map #(format-html %) (vals page-map))))
I'll break down these functions briefly, and note that most of them work only on the raw html strings or key name strings returned from the stasis/slurp-directory
function.
format-images
is simply to fix a silly formatting problem when exporting my image links from org-mode to html (see org-workflow: Handling images). I think it is self explanatory.format-html
will be a function that simply composes other small html formatting functions I may want to use in the future. Right now, I only haveformat-images
andlayout-base-header
. If I need more in the future, it will be trivial to write and apply them without breaking upstream code (as long as I take and return html strings). Really nice consequence of dealing with simple values rather than objects.fmt-page-names
As the documentation says, this removes html from all html pages that do not containindex
in them, and then prepends somebase-name
to all pages. The pages that are already namedindex.html
are pre-made pages that I have as the landing pages for those subjects. These pages need to retain the.html
file endings in order to render as index pages correctly. All others can have the.html
endings removed. This allows me to prepend/programming/
to all pages in the programming folder, and the same for science.html-pages
is another composed function of all of the above functions. Instead of taking and returning a string, it takes and returns a map (which comes fromstasis/slurp-directory
). Just to demonstrate how this is used, I'll show you reading in pages inweb.clj
:
;;;; web.clj
(ns website-clj.web
"main namespace for building and exporting the website"
(:require [optimus.assets :as assets]
[optimus.export]
[optimus.link :as link]
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[clojure.java.io :as io]
[clojure.string :as str]
[stasis.core :as stasis]
[website-clj.export-helpers :as helpers]
[website-clj.process-pages :as process]))
;; define page maps and link maps
(def programming-map
(process/html-pages "/programming"
(stasis/slurp-directory "resources/programming" #".*\.(html|css|js)")))
;; --- snip ---
Awesome. All of my html formatting and reading in one place.
A quick note about images and resources
Although this seems simple in hindsight, it caused me a significant amount of headaches and some time to figure out.
Looking back at my folder layout in the resources
directory:
resources ├── home ├── org-programming ├── org-science ├── programming ├── public │ ├── css │ ├── img │ └── js ├── science └── test
How do you refer to images from a post in html? My first thought was this
<h1>This is the landing page</h1> <p> Welcome to it. Here is a test image: <img src="../public/img/sample-img.png" alt="sample img!" /> </p>
As I figured the working directory was within whatever page you were at, and then I just followed the path to img
. But that does not work. Finally I figured out that images can be added by referring to them relative to public as the working directory. For example:
<img src="/img/sample-img.png" alt="sample img!" />
inserts the image stored in public/img/test-img.png
. This makes sense (in hindsight), as with stasis
I slurped the whole public/
directory.
org-workflow: Handling images
How does this factor into my org-mode workflow? Let's say I have an example org-mode file, and I'll add an image in org-markup manner.
#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{} #+HTML_CONTAINER: div =* This is a test post Here is a test post and a link to an image. [[file:~/personal_projects/website-clj/resources/public/img/test-img.png]]
Exporting this to html gives the following link structure in HTML:
<img src="img/test-img.png" alt="test-img.png"/>
While this is almost right, it doesn't render properly because all images are referred to /img/
. To fix it, I wrote a link formatting function in Preaparing pages called format-images
. Now to gather and serve all the resources, I have a function called get-assets
which grabs everything from public/
and hands it to optimus for frontend optimizations.
;;;; web.clj
(ns website-clj.web
"main namespace for building and exporting the website"
(:require [optimus.assets :as assets]
[optimus.export]
[optimus.link :as link]
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[clojure.java.io :as io]
[clojure.string :as str]
[stasis.core :as stasis]
[website-clj.export-helpers :as helpers]
[website-clj.process-pages :as process]))
;; --- snip ---
(defn get-assets
"get all static assets from the public directory."
[]
(assets/load-assets "public" [#".*"]))
;;--- snip ---
;; for test rendering
(def app
"renders the website for experimentation"
(optimus/wrap
(stasis/serve-pages get-pages)
get-assets
optimizations/none
serve-live-assets))
get-assets
is likely why I refer to images as /img/image.png
instead of public/img/image.png
.
org-workflow: syntax highlighting
Christian Johanson has an excellent description of formatting markdown fenced code blocks with pygments for nice display on his static site. His approach uses pygments and enliven and is very detailed and nice. However, the amazing org-mode
takes care of syntax highlighting for me when I add (setq org-src-fontify-natively t)
to my config.org. So here I will just test it real quick and see how it looks.
In my HTML file, I will add a clojure code block like so:
#+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{} #+HTML_CONTAINER: div =* This is a test post Here is a test post and a link to an image. [[file:~/personal_projects/website-clj/resources/public/img/test-img.png]] And below is a test code block. #+BEGIN_SRC clojure (defn format-images [html] (str/replace html #"file:///Users/Nick/personal_projects/website-clj/resources/public" "")) ;; main pages function. (defn html-pages [pages] (zipmap (map #(str/replace % #"\.html$" "") (keys pages)) (map #(fn [req] (layout-base-header req %)) (map format-images (vals pages))))) #+END_SRC How does it look?
This renders upon M-x org-publish-project clj-site
to look like this:
org-src-fontify-natively
uses the currently active theme to highlight your source code. I just exported this using the Leuven theme (great for org-mode) and I like the way it looks. However, if I wanted to change it and use enliven
with pygments
, I would probably use some emacs-lisp code and packages such as those described here: https://emacs.stackexchange.com/questions/31439/how-to-get-colored-syntax-highlighting-of-code-blocks-in-asynchronous-org-mode-e , but for right now I dont think this is necessary for me so I will go with the raw html formatting from org-export.
You can see the source code for my project here
Summing up
In Preaparing pages we addressed reading in pages with stasis, formatting html, syntax highlighting and adding resources like images. I think we can cross #1 off our list.
Two main categories of posts:Programming
andScience
.- Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
- Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
- One-push publishing and integration with org-mode.
The next section will address the metadata-related goals.
Parsing edn
metadata
¶
Most static site generators (Jekyll, for instance) contain some way to add metadata in markup format to posts in order to set formatting options, apply themes, add a name, etc. So referring back to my list of goals for my site:
Two main categories of posts:Programming
andScience
.- Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.
- Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.
- One-push publishing and integration with org-mode.
I want to automatically generate a list of posts in reverse chronological order on index pages for the programming
and science
sections. In order to do this, metadata would be nice, and Clojure offers an excellent solution in the form of extensible data notation or edn
. In this section I'll be tackling both 2 and 3.
setting up the metadata in org-mode
First off, I'll put the metadata in a div
at the top of my document with the id
as edn
. Since I write in org-mode, I made a YASnippet (awesome emacs templates, check them out) called blog
:
;; yas snippet blog # -*- mode: snippet -*- # name: blog # key: blog # -- #+HTML: <div id="edn"> #+HTML: {:topic "programming" :title "${1:title}" :date "`(format-time-string "%Y-%m-%d")`" :tags ${2:["clojure"]}} #+HTML: </div> #+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{} #+PROPERTY: header-args :eval never-export $0
when I type blog<TAB>
it expands to the following
#+HTML: <div id="edn"> #+HTML: {:topic "programming" :title "title" :date "2018-08-19" :tags ["clojure"]} #+HTML: </div> #+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{} #+PROPERTY: header-args :eval never-export
The important part here is the #+HTML:
sections. That tag tells org-mode to export that line as literal HTML. This creates a unique div id containing the metadata with a shorter title for the post, the date (automatically generated with inline emacs-lisp
), and a vector of tags
. For now I will only deal with the title and date, but I will likely start doing something with the tags vector later.
parsing edn
with enlive
So we added metadata under a special tag, but how do we parse it?
The functions I write work with either raw html text, or with the map of {file1-name html1-text ...}
returned by the function stasis/slurp-directory
, as discussed in Preaparing pages. In order to parse these, I'll use enlive, the amazing selector-based templating and html transformation library.
I'll add some edn
metadata to a test html page and start playing. First I need to add enlive
to my project.clj
;; project.clj
(defproject website-clj "0.1.0-SNAPSHOT"
:description "Personal website built with Clojure, Stasis, and Hiccup"
:url "http://nickgeorge.net"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.8.0"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[optimus "0.14.2"]
[enlive "1.1.6"]]
:ring {:handler website-clj.web/app}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}}
:aliases {"build-site" ["run" "-m" "website-clj.web/export"]})
and then run lein deps
at the command line. I'd recommend going through this enlive tutorial to figure out how to parse with enlive
. For REPL based play and testing, my test.org
doc looks like this:
#+HTML: <div class="edn"> #+HTML: {:topic "programming" :title "renamed" :date "2018-08-05" :tags ["clojure" "testing" "post"]} #+HTML: </div> #+OPTIONS: \n:1 toc:nil num:0 todo:nil ^:{} Here is my test content and some code #+BEGIN_SRC clojure (test clj-code) (def test-me "test string") #+END_SRC
and I put that in the test-org/
folder and added an export line to my org-mode export file in the project clj-site
.
When I run org-publish-project clj-site
I get this:
<div class="edn"> {:topic "programming" :title "renamed" :date "2018-08-05" :tags ["clojure" "testing" "post"]} </div> <div id="outline-container-orgd13af6f" class="outline-2"> <h2 id="orgd13af6f">Here is my test content</h2> <div class="outline-text-2" id="text-orgd13af6f"> <p> and some code<br /> </p> <div class="org-src-container"> <pre class="src src-clojure"><span style="color: #707183;">(</span><span style="color: #006FE0;">test</span> clj-code<span style="color: #707183;">)</span> <span style="color: #707183;">(</span><span style="color: #0000FF;">def</span> <span style="color: #BA36A5;">test-me</span> <span style="color: #036A07;">"test string"</span><span style="color: #707183;">)</span> </pre> </div> </div> </div>
With the useful stuff at the top in the tag. I made a new test folder for this, and I moved the test.html there. So now, I'll read that in and start messing around.
I am playing with this code at the bottom of my new process-clj
namespace.
;;;; process_pages.clj
;; first step is slurping a directory, applying the path prefix and formatting html. Make sure to add stasis to the ns declaration for testing!
(def slurped-raw
"holds a map of formatted html pages for my website"
(html-pages "/test" (stasis/slurp-directory "resources/test" #".*\.(html|css|js)")))
(keys slurped-raw)
;; => ("/test/index.html" "/test/test" "/test/test2")
(vals slurped-raw)
;; => html for the pages
;; isolate html for one page
(def test-html (second (vals slurped-raw)))
test-html
;; => html for page /test/test
Now I have raw html to play with. Figuring out the parsing took some time, but eventually I figured out this code:
;;;; process_pages.clj
(ns website-clj.process-pages
(:require [clojure.string :as str]
[hiccup.core :refer [html]]
[hiccup.page :use [html5 include-css include-js]]
[hiccup.element :refer (link-to image)]
[net.cgrand.enlive-html :as enlive]
[clojure.edn :as edn]
[stasis.core :as stasis]))
;; --- snip ---
(defn parse-html
"Takes raw html and returns keys from edn metadata under the <div id='edn'> html tag
`html` is raw html"
[html]
(as-> html raw-text ;; 1
(enlive/html-snippet raw-text) ;; 2
(enlive/select raw-text [:#edn enlive/text-node]) ;; 3
(apply str raw-text) ;; 4
(edn/read-string raw-text))) ;; 5
;; --- snip ---
;; isolate html for one page
(def test-html (second (vals slurped-raw)))
;; => html for page /test/test
(def metadata (parse-html test-html))
metadata
;; => {:topic "programming" :title "renamed" :date "2018-08-05" :tags ["clojure" "testing" "post"]}
Going through parse-html
- start threading the html using the
as->
macro. I recently refactored to use this rather than thread first->
because I have one place (#4) where the html needs to be the last argument. Rather than mixing thread first and thread last, I usedas->
to overcome this limitation. - turn the html into an
enlive/html-snippet
. As far as I know, this parses the html for enlive. - use enlive to get the relevant node base on id. You select based on the
div id
with#id-name
. This part is still a little confusing for me… - Now I need to turn that into a string, so I use
apply str
- uses
edn/read-string
to parse the resulting string into a clojure map. Note all of myedn
metadata will be represented as strings or vectors/lists of strings for now, as I can select some for inserting later.
edn
is parsed and in memory, though in order to use it in practice I'll make one more function that takes in the map returned by stasis/slurp-directory
, and returns a map of maps with the metadata.
In practice, /programming/index.html
will live in the programming/
directory that is parsed by my edn metadata parser. That means if I make links based on the raw output of stasis/slurp-directory
I would get a link for the index page, on the index page, which is sloppy. The function remove-index
removes the index page.
;;;; process_pages.clj
;; --- snip ---
;; remove index page
(defn remove-index
"Removes /index.html from map that will be parsed for edn metadata.
`base-name` is the name prepended to the index.html page. For programming pages it will be '/programming'
`page-map` is the map returned by `html-pages`. returns `page-map` minus the index pages."
[base-name page-map]
(dissoc page-map (str base-name "/index.html")))
This function simply joins the base-name
(i.e. "/programming") to the string "/index.html" and removes it from the map.
The function make-edn-page-map
works directly with the map from stasis/slurp-directory
, and it returns a map of maps, with {page-name1 metadata1 ...}
(defn make-edn-page-map
"filters the `page-map` to remove index.html and returns a map of page names and edn metadata.
`page-map` is returned by `stasis/slurp-directory`.
`base-name` provides the prepended base for the directory you are filtering by with `remove-index`"
[base-name page-map]
(let [filtered-page-map (remove-index base-name page-map)] ;; 1
(zipmap (keys filtered-page-map) ;; 2
(map parse-html (vals filtered-page-map))))) ;; 3
;; --- snip ---
;; useage
(def metadata (parse-edn ("/test" slurped-raw)))
metadata
;; => {"/test/test" {:topic "programming" :title "renamed" :date "2018-08-05" :tags ["tag1" "tag2"]}, "/test/test2" {:topic "programming" :title "renamed2" :date "2018-08-06" ...}}
- applies
remove-index
to thepage-map
. - Use the keys from the newly filtered page map as the keys in the new map
- apply
parse-html
to the values of thefiltered-page-map
. This will be the values for the new map.
making html links
Now we have a nice map to work with, it is time to make some links. The function below format-html-links
demonstrates the advantage of using hiccup
to generate html from within clojure.
;;;; process_pages.clj
;; --- snip ---
(defn format-html-links
"Makes a list of links in reverse chronological order using hiccup markup.
`metadata-map`comes from the output of `parse-edn`"
[metadata-map]
(html [:ul ;; 1
(for [[k v] ;; 2
(reverse (sort-by #(get-in (val %) [:date]) metadata-map))] ;; 3
[:li ;; 4
(link-to k (get v :title)) (str "<em> Published: " (get v :date) "</em>")])])) ;; 5
;; testing
;; remember what metadata looks like? a map of maps
metadata
;; => {"/test/test" {:topic "programming" :title "renamed", :date "2018-08-05"}, "/test/test2" {:topic "programming" :title "renamed2", :date "2018-08-06"}}
(format-html-links metadata)
;; => "<ul><li><a href=\"/test/test2\">renamed2</a><em> Published: 2018-08-06</em></li><li><a href=\"/test/test\">renamed</a><em> Published: 2018-08-05</em></li></ul>"
I like this function, but it does look a little complicated. Here are what the parts do.
html
is fromhiccup
. Here I am generating anhtml
fragment, and I am initializing an un-ordered html list.- The start of a list comprehension in clojure.
- This is a really cool thing about clojure I found in the docs for sorting maps. You can sort by a value within the map of maps. So I sort by the key
:date
. Then, to get reverse chronological order, I justreverse
the list. This is all part of the main list comprehension. Sok
is the key from the now reverse chronologically orderedmetadata-map
, andv
is the value (inner metadata map) - initializes an element of a list. This will be generated for each item in the
metadata-map
- make a link to the original key
k
with the title of the link being the:title
item provided by the metadata. Then, just for fun, put the published date next to it.
Hide the metadata
I don't want the metadata showing up at the top of every page. I made a css file called custom.css
and had it hide all the id=edn
div
's.
// css/custom.css
#edn {
display: none;
}
Easy. Now I will use the include-css
hiccup header and add the following to my hiccup
-defined header:
;; process-pages ns
(ns website-clj.process-pages
(:require [clojure.string :as str]
[hiccup.core :refer [html]]
[hiccup.page :use [html5 include-css include-js]] ;; include hiccup helpers
[hiccup.element :refer (link-to image)]
[net.cgrand.enlive-html :as enlive]
[clojure.edn :as edn]
[stasis.core :as stasis] ;; only for testing?
))
;; --- snip ---
(defn layout-base-header [request page]
(html5
[:head
[:meta {:charset "utf-8"}]
;;... --- snip ---
(include-css "/css/custom.css") ;; the new stuff
;;... --- snip ---
]
;;Much more here, I cut it out for simplicity
))
inserting the links with enlive
Probably the coolest part of enlive
is how it can transform html based on css selectors. I won't go into detail, but you should definitely look into this.
I wrote this function to add the list to my index pages.
;;;; process_pages.clj
;; -- snip --
(defn add-links
"Adds links of all pages to the index.html page and un-escapes html characters.
The `page` argument is the html for a page.
The `links` argument is an html string, typically generated with the `make-links` function
This returns the modified html"
[page links div] ;; 1
(-> page ;; 2
(enlive/sniptest ;; 3
[div] ;; 4 exists only in index pages.
(enlive/html-content links)))) ;; 5
In the body of my index.html
pages, I added the following div:
<div id="pageListDiv">Page nav list Here</div>
This div
id
will only exist in the index.html pages, so that is the only place that will get the links.
To go through the parts of this add-links
, I have to admit I am still not completely sure what part 3,4,5 are really doing, but I'll do my best.
- This function takes the list of links generated by
format-html-links
and adds them to apage
if if contains the targetdiv id
. - start threading with
page
as an argument. - initialize a
enlive/sniptest
- select the relevant
div id
, In this case it will be:#pageListDiv
, but I made it generic so you just providediv
as an argument. - use
enlive/html-content
to replace the content of thatdiv
with the links.
Page titles
Now to make good, well-formed html you have to have one last part– a <title>
. To add a title, I will use the parsed edn metadata similar to how I did it with the page links.
We have already written parse-html
, a function which returns the metadata map. I added a title
tag to my layout-base-header
that looks like this [:title "Nick's site"]
. This is there both to provide a template for enlive
to replace, and to ensure that every page has a title even if I am missing a title in the metadata (I read that it is good for search engine ratings to have well-formed html).
Next I wrote insert-page-title
:
;; process_pages.clj
;; ---snip---
(defn parse-html
"Takes raw html and returns keys from edn metadata under the <div id='edn'> html tag
`html` is raw html"
[html]
(as-> html raw-text
(enlive/html-snippet raw-text)
(enlive/select raw-text [:#edn enlive/text-node])
(apply str raw-text)
(edn/read-string raw-text)))
(defn insert-page-title
"`insert-page-title` parses edn metadata and return the html with a title inserted
`page` is the raw HTML of a page including the header."
[page]
(let [meta-title (get (parse-html page) :title "Nick's site")] ;; 1
(-> page ;; 2
(enlive/sniptest [:title]
(enlive/html-content meta-title))))) ;; 3
- parse the html and grab the
:title
tag. Store in alet
asmeta-title
. Note that I provided a default value forget
in case I didn't put a title in the metadata. - Start threading with the raw html (note this will have to be after I applied
layout-base-header
). - swap out the title tag contents with my parsed title.
Great! Now since this is another html processing function, I can add it to format-html
to put it right into the normal pipeline without changing anything else!
;; process_pages.clj
;; --- snip ---
(defn format-images [html]
"formats html image link to appropriately link to static website image directory.
`html` is a raw html string."
(str/replace html #"src=\"img" "src=\"/img"))
;; --- edn parsing for metadata---
;; !!! use as-> instead! see https://learnxinyminutes.com/docs/clojure/
(defn parse-html
"Takes raw html and returns keys from edn metadata under the <div id='edn'> html tag
`html` is raw html"
[html]
(as-> html raw-text
(enlive/html-snippet raw-text)
(enlive/select raw-text [:#edn enlive/text-node])
(apply str raw-text)
(edn/read-string raw-text)))
(defn insert-page-title
"`insert-page-title` parses edn metadata and return the html with a title inserted
`page` is the raw HTML of a page including the header."
[page]
(let [meta-title (get (parse-html page) :title "Nick's site")]
(-> page
(enlive/sniptest [:title]
(enlive/html-content meta-title)))))
(defn format-html
"Composed function to apply multiple html processing steps to raw html.
`html` is a raw html string."
[html]
(-> html
format-images
layout-base-header
insert-page-title)) ;; other fns for html here
Note that insert-page-title is after the base header is applied.
Now we have almost all the parts we need. I'll go over some caveats for publishing with GitHub Pages in Exporting for GitHub Pages, then I demonstrate the workflow in Bringing it all together.
Summing up parsing
In this section, we set up a system for adding edn
metadata to files, we parsed the metadata, made a list of links, sorted them, and inserted them into our document. Check a few more options off out list!
Two main categories of posts:Programming
andScience
.Index pages for both Programming and Science. These pages should contain (among other things) all of the links for those sections in reverse chronological order.Metadata similar to YAML front matter in Ruby's Jekyll, containing the date a post was made, the short title, some tags, and other stuff in the future.- One-push publishing and integration with org-mode.
Publishing is up next.
Exporting for GitHub Pages ¶
publishing
From lein
, Christian gives some nice instructions, so I followed those to see how the export looks and it seems to work nicely. Now, I'd like put my website on-line. I hosted my previous site on GitHub Pages, so I know I need a few config items for hosting. The first is the CNAME
file, for mapping your domain name to the github repo.
In Christian's example, he empties the target export directory with (stasis/empy-directory!)
before the rest of the export. I definitely want to do this, but looking into the Stasis code, I don't see any options to exclude certain files. That means my CNAME
, .gitignore
, and .git
repo will be wiped out every time I build! No good for GitHub Pages
I decided to use shell commands to get around this for the moment, and I broke these functions out into a namespace called export-helpers
.
CNAME
and .gitignore
will live in the resources/
and target/
directories, respectively. Upon export, they will be copied to the export directory like so
;;;; export_helpers.clj
(ns website-clj.export-helpers
"helper functions for saving the git directory, cname, and gitignore from `stasis/empty-directory!`
This exists to help with rendering static sites on github."
(:require [clojure.string :as str]
[clojure.java.shell :as shell])) ;; for shell commands from clojure
(defn cp-cname
"copy the CNAME file to the export directory.
`export-dir` is a var that contains the parth to the base of the website.
CNAME must be in the directory for github pages domain mapping."
[export-dir]
(shell/sh "cp" "resources/CNAME" (str export-dir "/CNAME")))
(defn cp-gitignore
"copy the gitignore file from a safe location to the base of the github pages repo for rendering."
[export-dir]
(shell/sh "cp" "target/.gitignore" (str export-dir "/.gitignore")))
;; --- snip ---
Handling the git
repo is a little trickier, as I don't want to maintain the git repo elsewhere. Instead, I made two functions: one to copy .git
to a save place, and another to restore it after building.
;;;; export_helpers.clj
;; --- snip ---
(defn save-git
"copy .git repo to a safe directory to save it from deletion.
`safe-dir` is a path to a directory that will not be emptied by `stasis/empty-directory!`
`export-dir` is the export directory where your site will be made."
[safe-dir export-dir]
(shell/sh "mv" (str export-dir "/.git") (str safe-dir "/.git")))
(defn replace-git
"Puts the gir directory back into the export directory.
`safe-dir` is a path to a directory that will not be emptied by `stasis/empty-directory!`
`export-dir` is the export directory where your site will be made."
[safe-dir export-dir]
(shell/sh "mv" (str safe-dir "/.git") (str export-dir "/.git")))
Here is how these will be used in the final product:
(ns website-clj.web
"main namespace for building and exporting the website"
(:require [optimus.assets :as assets]
[optimus.export]
[optimus.link :as link]
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[clojure.java.io :as io]
[clojure.string :as str]
[stasis.core :as stasis]
[website-clj.export-helpers :as helpers]
[website-clj.process-pages :as process]))
;; --- snip ---
;; constants for exporting
(def export-dir "target/nickgeorge.net")
(def safe-dir "target")
;; main export function, called by lein build-site
(defn export
"main export function for static site. See docs for functions included.
`website-clj.helpers/save-git`
`website-clj.helpers/cp-cname`
`website-clj.helpers/cp-gitignore`
`website-clj.helpers/replace-git`"
[]
(helpers/save-git safe-dir export-dir)
(let [assets (optimizations/all (get-assets) {})]
(stasis/empty-directory! export-dir)
(optimus.export/save-assets assets export-dir)
(stasis/export-pages (get-pages) export-dir {:optimus-assets assets}))
(helpers/cp-cname export-dir)
(helpers/cp-gitignore export-dir)
(helpers/replace-git safe-dir export-dir))
This is super hacky and not optimal. It would be better to edit stasis/empty-directory!
to include arguments for excluding certain dirs/files– but for now this works.
Bringing it all together ¶
So how does this look in practice? Well just check out my web.clj source. The general format goes like so:
- Use
stasis/slurp-directory
andprocess-pages/html-pages
to read and format the pages for each subject. - pass the resulting map into
process-pages/parse-edn
to get the metadata map. - pass the metadata map into
process-pages/format-html-links
to make the html links. - make the get-pages function read all the relevant directories. This is where we also apply
process-pages/add-links
. - export for serving.
;;;; web.clj
(ns website-clj.web
"main namespace for building and exporting the website"
(:require [optimus.assets :as assets]
[optimus.export]
[optimus.link :as link]
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[clojure.java.io :as io]
[clojure.string :as str]
[stasis.core :as stasis]
[website-clj.export-helpers :as helpers]
[website-clj.process-pages :as process]))
;; define page maps and link maps
;; define page maps and link maps
(def programming-map
"constant for all links holding programming pages"
(process/html-pages "/programming"
(stasis/slurp-directory "resources/programming" #".*\.(html|css|js)")))
(def programming-metadata
"constant for all programming-metadata"
(process/make-edn-page-map "/programming" programming-map))
(def programming-links
"constant for all programming links"
(process/format-html-links programming-metadata))
;; repeat for science...
;; --- snip --
;; load all assets
(defn get-assets
"get all static assets from the public directory."
[]
(assets/load-assets "public" [#".*"]))
;; main get pages function for render and export
(defn get-pages ;; 4
"Gathers all website pages and resources."
[]
(stasis/merge-page-sources
{:public (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
:landing (process/home-page
(stasis/slurp-directory "resources/home" #".*\.(html|css|js)$"))
:programming (zipmap (keys programming-map)
(map #(process/add-links % programming-links :#pageListDiv)
(vals programming-map)))
:science (zipmap (keys science-map)
(map #(process/add-links % science-links :#pageListDiv)
(vals science-map)))}))
;; --- snip ---
(defn export ;; 5
"main export function for static site. See docs for functions included.
`website-clj.helpers/save-git`
`website-clj.helpers/cp-cname`
`website-clj.helpers/cp-gitignore`
`website-clj.helpers/replace-git`"
[]
(helpers/save-git safe-dir export-dir)
(let [assets (optimizations/all (get-assets) {})]
(stasis/empty-directory! export-dir)
(optimus.export/save-assets assets export-dir)
(stasis/export-pages (get-pages) export-dir {:optimus-assets assets}))
(helpers/cp-cname export-dir)
(helpers/cp-gitignore export-dir)
(helpers/replace-git safe-dir export-dir))
One push publishing with Leiningen :alias
¶
IN PROGRESS!! 2018-09-19
I can already build my website with my current alias, now I will make another to deploy! The steps I need to do are:
Command line build org-project
org-publish-project clj-site
from the command line- remember to add a header to tell org to not evaluate code like this:
#+PROPERTY: header-args :eval never-export
- This should be a clojure function called with
export
frombuild-site
- Then run build-site
git add
andgit push
all changes.- This could also be a clojure function called with
export
frombuild-site
- This could also be a clojure function called with
The idea is that I just call build-site and it all happens automatically when I run lein build-site
Right now, to publish, I run:
org-publish-project clj-site
from emacs.lein build-site
from the command line inwebsite-clj/
dir.cd
intotarget/nickgeorge.net/
thengit add .
,git commit -m "message"
, andgit push
.
Further improvements ¶
Will be posted here. On the near horizon:
1. Tests!