Total Pageviews

Friday, 27 June 2014

Building static sites in Clojure with Stasis


Stasis is a static site toolkit for Clojure. Unlike pretty much every other static site generator, though, it is not an "opinionated framework", or packed full with flavor-of-the-month templating languages and whatnot. It is just a few functions that helps with creating websites that can be hosted as static files, and developed against a live server (enabling layouts, and various dynamic features to generate the pages). As its Readme states; there are no batteries included. This post will take you through creating your first Stasis site, and serving it with super-optimized frontend assets, courtesy of Optimus. Both Stasis and Optimus are written by my good friend and colleague, Mr. Emacs Wizard, Magnar Sveen.
The source code for this post can be found on GitHub. As I plan to evolve the code to become my new blog (eventually), you should look for the blog-post tag, which represents the code as per this post.

Who is this for?

This post may interest anyone looking to set up a static site of some sort. The range of sites that can be successfully developed as static sites are bigger than you might think. While Stasis is a Clojure tool, only a very basic understanding of Lisp should be necessary to follow along.
I'm hoping this post will not only show you the power of Clojure and Stasis for building static web sites, but also give you a good introduction to some very useful Clojure libraries. Maybe even to Clojure itself. In particular, I will discuss using these libraries: Stasis, Optimus, enlive, hiccup, cegdown, clygments and even write some tests with Midje.

Getting set up

First things first, let's get a project set up to serve our frontpage. If you've never worked with Clojure, install Leiningen. Now create your project: 
lein new cjohansen-no
cd cjohansen-no
That creates an empty project for you. I will use this post to start the new code base for my blog. You might want to give yours a different name. Open project.clj, and add Stasis as a dependency. While you're at it, add a description and tune the license to your desires. When you're done, it should look something like this: 
(defproject cjohansen-no "0.1.0-SNAPSHOT"
  :description "cjohansen.no source code"
  :url "http://cjohansen.no"
  :license {:name "BSD 2 Clause"
            :url "http://opensource.org/licenses/BSD-2-Clause"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]])
Now add a truly static page (i.e. a file). Create and open src/cjohansen_no/web.clj. This is the namespace we will use to define our pages. Put the following content in it: 
(ns cjohansen-no.web
  (:require [stasis.core :as stasis]))

(defn get-pages []
  (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$"))
#"..." is Clojure syntax for regular expressions.
(Note that the namespace is called cjohansen-no, while the directory was called cjohansen_no. If you misspell either of these, Clojure will bark at you with a rather unfriendly message saying it can't find the namespace. Remember to double check file/directory names and namespaces).
The web.clj file pulls in stasis under the local name stasis, and defines one function. Stasis will expect to receive a map of url => content defining pages, so this is what get-pages does. The content can be a function, in which case it will be only be called when Stasis needs to serve this particular page. This lazy loading becomes useful if your site is big enough for it to become slow by loading everything in one go. For now, we'll not worry about it.
slurp-directory is a Stasis function that creates a map of files in a directory (recursively). Every file in the subtree that matches the provided regular expression (i.e. every html, css and js file under resources/public) will be included. The keys in the map (e.g. the URLs), will be the file path relative to resources/public. For example, resources/public/index.html will be served as /index.html.
Put the following in resources/public/index.html: 
<!DOCTYPE html>
<html>
  <head>
    <title>My blog</title>
  </head>
  <body>
    <h1>My blog</h1>
    <p>
      Welcome to it.
    </p>
  </body>
</html>
As Stasis is a "no batteries included" framework, we need to do some work to either export the static site or view it live through a web server. As the file is already static, we'll set up the live server first. Update project.clj to pull in a couple of dependencies and configure a Leiningen plugin: 
(defproject cjohansen-no "0.1.0-SNAPSHOT"
  :description "cjohansen.no source code"
  :url "http://cjohansen.no"
  :license {:name "BSD 2 Clause"
            :url "http://opensource.org/licenses/BSD-2-Clause"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]]
  :ring {:handler cjohansen-no.web/app}
  :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
Ring is the defacto HTTP toolkit for Clojure, and can be compared to Python's WSGI or Ruby's Rack. We have added it as a dependency, and set up the app function in the cjohansen-no.web namespace (e.g. the file we created earlier) to be the entry-point for the web server. The profile configuration loads some convenient Leiningen tasks for the development profile (which is also the default profile).
A web application in Ring is just a function that receives as its only argument a request hash and returns a map like the following: 
{:status 200
 :headers {"Content-Type" "text/html"}
 :body "Hello World"}
Stasis provides the serve-pages function to help us produce this response. It returns a function that will look at the request and find the corresponding page in our map. Add the app function to the src/cjohansen-no/web.clj file: 
(def app (stasis/serve-pages get-pages))
Run the server: 
lein ring server
This will pop up a browser displaying your static HTML file in all its naked glory.

Adding templating

Next we will add a page that is split between the content/body of the page and the wrapping layout. The layout will be shared by many files, so applying it in one central place saves us some work.
Add a partial page to resources/partials/about.html 
<h1>About this site</h1>
<p>
  It is a web page.
</p>
Rather than using HTML manually to create the layout, we will use the popular Clojure templating library called Hiccup. Hiccup allows us to express HTML in a more compact form by using vectors, keywords and maps. It is best illustrated with an example. Add Hiccup to project.clj: 
:dependencies [[org.clojure/clojure "1.5.1"]
               [stasis "1.0.0"]
               [ring "1.2.1"]
               [hiccup "1.0.5"]] ;; Like so
Alter the namespace form in src/cjohansen_no/web.clj to require the html5 function from Hiccup: 
(ns cjohansen-no.web
  (:require [hiccup.page :refer [html5]]
            [stasis.core :as stasis]))
Add the following function to the same file: 
(defn layout-page [page]
  (html5
   [:head
    [:meta {:charset "utf-8"}]
    [:meta {:name "viewport"
            :content "width=device-width, initial-scale=1.0"}]
    [:title "Tech blog"]
    [:link {:rel "stylesheet" :href "/styles/styles.css"}]]
   [:body
    [:div.logo "cjohansen.no"]
    [:div.body page]]))
As you can see, Hiccup markup is mostly a leaner version of HTML. One thing that makes Hiccup very cool is that is accepts elements like the ones in our example, nested lists of elements, and nils. This means that we can map over data structures inline in the Hiccup structure without having to worry about nested lists creating nested markup structures. Think of lists as the DocumentFragments of Hiccup. We can also inline if forms without else forms without worrying about dangling nils causing weird artefacts in the generated markup.
To use the new layout, we will add a page definition for our partial page, and apply the layout on it. To read files from the app's resources directory, require the clojure.java.io package: 
(ns cjohansen-no.web
  (:require [clojure.java.io :as io]
            [hiccup.page :refer [html5]]
            [stasis.core :as stasis]))
Add the about page function: 
(defn about-page [request]
  (layout-page (slurp (io/resource "partials/about.html"))))
Finally, add the new page to the page map created in get-pages: 
(defn get-pages []
  (merge (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
         {"/about/" about-page}))
Because we added a new dependency to project.clj, you now need to quit the server (Ctrl+C) and restart it. When you have, /about/ should present you with your fantastic new page-with-layout.

Serving all partials

Doing all this work for every new partial page seems a tad bit too laborious. We will generalize the code we just added by re-writing it to serve all partial pages under resources/partials in the same way.
Remember how (stasis/slurp-directory "resources/partials" #".*\.html$") will create a map with relative paths as keys and the contents of the corresponding files as values? This is almost what we want, except we now also want to wrap the content in the layout. To achieve this, we will loop through the map and wrap all the values in a function call that adds the layout. Like so: 
(defn partial-pages [pages]
  (zipmap (keys pages)
          (map layout-page (vals pages))))
This function accepts as input the map produced by Stasis' slurp-directory and returns a map where the values have all been wrapped in a layout. The zipmap function takes two collections and returns a map. It builds each map entry by pulling an entry from the first collection to use as the key, and an entry from the second collection to use as the value. Now remove the about-page function, and update get-pages to look like this: 
(defn get-pages []
  (merge (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
         (partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))))
This will almost work. You will find our about page from before at /about.html, instead of /about. The reason is that slurp-directory uses the relative file names as paths. We can fix this in one of two ways:
  1. Mapping the keys as well, to lose the extension. While this will certainly work, it will produce odd results for any file called index.html
  2. Rename resources/partials/about.html to resources/partials/about/index.html
We will opt for option 2 in this case. I will show option 1 when we deal with markdown files.

Path conflicts

Now that we have two page sources, and both of them create root-level URLs, there is a risk that we end up with conflicts. With our current use of merge, any page in resources/partials will silently shadow a page in resources/public with the same path. Update get-pages to the following to avoid the problem: 
(defn get-pages []
  (stasis/merge-page-sources
   {:public (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
    :partials (partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))}))
The merge-page-sources function works pretty much like merge, except it will throw an exception if either source defines duplicate URLs. The map keys are useful to determine the source of the conflict. For instance, if you were to add public/about/index.html, you would get this error: 
URL conflicts between :public and :partials: #{"/about/index.html"}
(That last bit is Clojure set notation by the way).

Writing in markdown

Partial pages are nice, but being able to write in markdown would be even better. Cegdown is a Clojure wrapper for Pegdown, a popular Java library for rendering markdown. Add it to your project.clj: 
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [me.raynes/cegdown "0.1.1"]] ; Like so
Now, add it to the namespace form in src/cjohansen_no/web.clj. While we're at it, we will require the Clojure string library as well (we'll use it shortly): 
(ns cjohansen-no.web
  (:require [clojure.java.io :as io]
            [clojure.string :as str]
            [hiccup.page :refer [html5]]
            [me.raynes.cegdown :as md]
            [stasis.core :as stasis]))
Now we will add a function to render every page in resources/md as markdown. It will be very similar to the partials we did before, but now with markdown rendering as well. Because we don't want ".md" as part of the URL for these pages, we will map the keys as well. 
(defn markdown-pages [pages]
  (zipmap (map #(str/replace % #"\.md$" "") (keys pages))
          (map #(layout-page (md/to-html %)) (vals pages))))
The #( ) form is a function literal. Inside it, % refers to the first argument. In the above example, the following are identical: 
#(str/replace % #"\.md$" "")
;; ...and:
(fn [path] (str/replace path #"\.md$" ""))
I wrote more on the anonymous function literal in a separate post.
The final step is to add the new page source to our map: 
(defn get-pages []
  (stasis/merge-page-sources
   {:public
    (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
    :partials
    (partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))
    :markdown
    (markdown-pages (stasis/slurp-directory "resources/md" #"\.md$"))}))
Add the following to resources/md/my-first-post.md: 
# My first post

It's pretty short for now.
Restart the server again (we added more new dependencies, remember?) When you have, /my-first-post should present you with your brief, but lovely blog post.

Post-processing: Syntax highlighting

Any self-respecting tech blog needs nice syntax highlighting for code blocks. When it comes to syntax highlighting, Pygments is the bee's knees. It supports just about any language you can think of, there's a bunch of color themes around and it is stable and resillient. It is also the library used to highlight code on GitHub. Clygments is a Clojure interface to it (which uses Jython; Pygments is a Python library).
We will add syntax highlighting as a post-processing step for HTML. This way, we can support syntax highlighting for full static pages in the public directory, partial pages and pages rendered from markdown. To do this, we will use another templating library for Clojure, enlive. Actually, enlive is more than just a templating library. As you will see, it can be used to transform documents in various interesting ways.

A code block

Let's add a fenced code block to our markdown file: 
# My first post

It's pretty short for now. Here's our project.clj:

```clj
(defproject cjohansen-no "0.1.0-SNAPSHOT"
  :description "cjohansen.no source code"
  :url "http://cjohansen.no"
  :license {:name "BSD 2 Clause"
            :url "http://opensource.org/licenses/BSD-2-Clause"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [me.raynes/cegdown "0.1.1"]]
  :ring {:handler cjohansen-no.web/app}
  :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
```
In order for this work, we need to inform cegdown that we want to enable the fenced code blocks extension. While we're at it, we'll enable a couple of other useful extensions as well: 
(def pegdown-options ;; https://github.com/sirthias/pegdown
  [:autolinks :fenced-code-blocks :strikethrough])

(defn render-markdown-page [page]
  (layout-page (md/to-html page pegdown-options)))

(defn markdown-pages [pages]
  (zipmap (map #(str/replace % #"\.md$" "") (keys pages))
          (map render-markdown-page (vals pages))))
Reloading the blog post will show you how the fenced code blocks are rendered: 
<pre><code class="clj">...</code></pre>
We will now use enlive to extract this piece of markup and replace it with the version highlighted by clygments. First off, add the new dependencies to project.clj (remember to restart the server!) 
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [me.raynes/cegdown "0.1.1"]
                 [enlive "1.1.5"] ; New
                 [clygments "0.1.1"]] ; New
Add a new namespace (i.e., file) to the project. Copy the following code into src/cjohansen_no/highlight.clj: 
(ns cjohansen-no.highlight
  (:require [clojure.java.io :as io]
            [clygments.core :as pygments]
            [net.cgrand.enlive-html :as enlive]))
Enlive has many tricks up its sleave. Perhaps the most interesting one is the somewhat confusingly named sniptest. It takes some HTML as a string, and selector/function pairs. It will then;
  1. Parse the HTML
  2. Find all nodes matching the selector
  3. Call the corresponding function once for every match
  4. Replace the node with the result of calling the function
  5. Return the transformed HTML as a string
(defn highlight-code-blocks [page]
  (enlive/sniptest page
            [:pre :code] highlight
            [:pre :code] #(assoc-in % [:attrs :class] "codehilite")))
This function will find every code element inside a pre element and pass it through the highlight function (to be shown). Then, it will add a class name to the same elements. In practice, you would probably do both in the highlight function, but this allows me to illustrate how you can perform multiple transformations in one go. The "codehilite" class name just happens to be class name used by the Pygments CSS themes available here (we will include this later). Add the second function: 
(defn- highlight [node]
  (let [code (->> node :content (apply str))
        lang (->> node :attrs :class keyword)]
    (pygments/highlight code lang :html)))
The dash in defn- means that this function is private, and only referrable within the current namespace. The nodes that enlive operate on are maps like this: 
{:tag :code
 :attrs {:class "clj"}
 :content [...]}
The content is a list of new nodes and/or strings.
->> is the thread-last macro. It takes any number of arguments, threads values from left to right; given (->> a (b 1 2) c), it will take the value a, pass it as the last argument to b (i.e. (b 1 2 a)), pass the return value of that expression as the last argument to c, and finally return the result of that. So the above line: 
(->> node :content (apply str))
Is the same as this: 
(apply str (:content node))
I wrote more on threading macros in a separate post.
Using a keyword as a function is one way to look up that key in a map. The apply call means to call str with the following list of arguments, as individual arguments, not one list (e.g. (fn a b c), not (fn [a b c])).
To preview the highlighting in the browser, we need to make some changes to the web namespace. Start by updating the namespace form to pull in our new dependency: 
(ns cjohansen-no.web
  (:require [cjohansen-no.highlight :refer [highlight-code-blocks]] ; This one
            [clojure.java.io :as io]
            [clojure.string :as str]
            [hiccup.page :refer [html5]]
            [me.raynes.cegdown :as md]
            [stasis.core :as stasis]))
Syntax highlighting should apply to all pages. A good place to do it is between our old get-pages function and Stasis' rendering. We will do this by adding a prepare-pages function: 
(defn prepare-pages [pages]
  (zipmap (keys pages)
          (map #(highlight-code-blocks %) (vals pages))))
Again, we use zipmap to produce a new map where the keys are untouched, but the values have been mapped. To make Stasis run through this, rename get-pages to get-raw-pages, and add a new get-pages: 
(defn get-raw-pages []
  (stasis/merge-page-sources
   {:public
    (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
    :partials
    (partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))
    :markdown
    (markdown-pages (stasis/slurp-directory "resources/md" #"\.md$"))}))

(defn prepare-pages [pages]
  (zipmap (keys pages)
          (map #(fn [req] (highlight-code-blocks %)) (vals pages))))

(defn get-pages []
  (prepare-pages (get-raw-pages)))
Reloading the markdown page should show you that what we've done so far both kinda worked and kinda didn't.
We are getting highlighted code. However, Pygments includes some unwanted wrapping markup. As we're already inside a pre element, the result is not quite as desired. To fix this we will use enlive once more to massage the output from Pygments. The output includes a wrapper div and a pre, let's extract just the code: 
(defn- extract-code
  [highlighted]
  (-> highlighted
      java.io.StringReader.
      enlive/html-resource
      (enlive/select [:pre])
      first
      :content))

(defn- highlight [node]
  (let [code (->> node :content (apply str))
        lang (->> node :attrs :class keyword)]
    (assoc node :content (-> code
                             (pygments/highlight lang :html)
                             extract-code))))
Enlive's select function selects elements from a document, but unlike sniptest, it does not accept a string. Instead, we must go through its html-resource function, which only accepts input streams. The end result is that we do a select on what Pygments gives us in order to get just the highlighted code. Refreshing the blog post shows that it works as expected.
The -> is the thread-first macro. It works like thread-last, except it threads values as the first argument to the next function. The above example could be written in either of these ways, but most people find the threading form to be the easiest on the eyes: 
(-> highlighted
    java.io.StringReader.
    enlive/html-resource
    (enlive/select [:pre])
    first
    :content)

;; Same as:
(:content (first (enlive/select (enlive/html-resource
                                  (java.io.StringReader. highlighted))
                                [:pre])))
To add some styling, pick a CSS file from the suggested themes repo, and load it onto the page. Update the layout-page in web.clj to look like this: 
(defn layout-page [request page]
  (html5
   [:head
    [:meta {:charset "utf-8"}]
    [:meta {:name "viewport"
            :content "width=device-width, initial-scale=1.0"}]
    [:title "Tech blog"]
    [:link {:rel "stylesheet" :href "/pygments-css/autumn.css"}]]
   [:body
    [:div.logo "cjohansen.no"]
    [:div.body page]]))

Lazy pages

As we're adding more features to our site, it is becoming apparent that processing all the pages to completion on every request isn't ideal. Fixing this is quite easy with Stasis, because we can give Stasis functions instead of strings, and then Stasis will call the function to build a particular page only when it needs to render that specific page.
To make our pages lazy, update prepare-pages to replace the values with functions instead of strings of highlighted HTML. The function should take one argument, the request map. 
(defn prepare-pages [pages]
  (zipmap (keys pages)
          (map #(fn [req] (highlight-code-blocks %)) (vals pages))))
By having the function literal return a new function that takes one argument, we have significantly improved performance for our development server.

Asset optimization

Now that we have a blog with syntax highlighting, we need to start thinking about delivery. Fast webpages beat slow ones on all sorts of metrics. One way to make our site faster is by employing various frontend asset optimization techniques. For this purpose, there is (among others) Optimus. We will use it to:
  • Concatenate CSS and JavaScript files
  • Minify CSS and JavaScript files
  • Serve CSS and JavaScript from cache-friendly URLs
A "cache friendly" URL is one that is unique every time the contents of the URL changes. This way we can serve assets with aggressive cache headers, and users will only need to download them once. The next time we deploy, if the assets have changed, they will have a new URL. Optimus facilitates this by providing some functions to help us link to assets.
We will start by adding Optimus as a dependency in project.clj. Remember to restart the server after doing this. 
(defproject cjohansen-no "0.1.0-SNAPSHOT"
  :description "cjohansen.no source code"
  :url "http://cjohansen.no"
  :license {:name "BSD 2 Clause"
            :url "http://opensource.org/licenses/BSD-2-Clause"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [me.raynes/cegdown "0.1.1"]
                 [enlive "1.1.5"]
                 [clygments "0.1.1"]
                 [optimus "0.14.2"]] ; New
  :ring {:handler cjohansen-no.web/app}
  :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
Now update the web.clj namespace form to require some functions from Optimus: 
(ns cjohansen-no.web
  (:require [optimus.assets :as assets]                     ; New
            [optimus.optimizations :as optimizations]       ; New
            [optimus.prime :as optimus]                     ; New
            [optimus.strategies :refer [serve-live-assets]] ; New
            [cjohansen-no.highlight :refer [highlight-code-blocks]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [hiccup.page :refer [html5]]
            [me.raynes.cegdown :as md]
            [stasis.core :as stasis]))
Instead of having Stasis serve the files in public, we will hand them to Optimus as assets. We will define a separate function for these assets, as it makes for a natural place to add further assets and/or bundles of assets later: 
(defn get-assets []
  (assets/load-assets "public" [#".*"]))
If your CSS files use @import, Optimus will (by default) take care to inline the import, so there is no need to define bundles at this point. Refer to the Optimus readme for more details.
To make our app use the new assets, we will change the app function: 
(def app
  (optimus/wrap (stasis/serve-pages get-pages)
                get-assets
                optimizations/all
                serve-live-assets))
The call to stasis/serve-pages returns a function (a Ring app, remember?) optimus/wrap returns another function with the same signature that wraps the original one. We pass it the function to get all our assets, optimization rules (a function) and a strategy for serving the assets (also a function). optimizations/all is a grab bag of every trick Optimus knows:
  • Minify JavaScript
  • Minify CSS
  • Inline CSS imports
  • Concatenate bundles
  • Add cache-bust expires headers (replace URL references with generated unique ones)
  • Add last-modified headers
You are free to pick and choose from this list if you want, but for most cases, optimizations/all is what you want.
Lastly, we employed Optimus' serve-live-assets strategy, which means that Optimus will read assets from disk on every request. This is useful in development mode, but in a production setting, you would typically use one that's less resource intensive, like serve-frozen-assets.
Create a CSS file and make sure it gets included from the page layout. 
@import url(../pygments-css/autumn.css);

body {
    font: 16px Helvetica, arial, freesans, clean, sans-serif;
    line-height: 1.5;
    margin: 0 10px;
}
Refreshing the blog in the browser should display the same page as before. However, if you hit the CSS file directly, you will find that Optimus has done what it can to optimize serving it.

Rewriting links

There is one final thing to take care of. In production, we can configure our web server to serve assets with a far future expires header. But in order for that to be safe, we need distinct URLs for every change to the file. Let's add another require to the web namespace form: 
(ns cjohansen-no.web
  (:require [optimus.assets :as assets]
            [optimus.link :as link] ; New
            [optimus.optimizations :as optimizations]
            [optimus.prime :as optimus]
            [optimus.strategies :refer [serve-live-assets]]
            [cjohansen-no.highlight :refer [highlight-code-blocks]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [hiccup.page :refer [html5]]
            [me.raynes.cegdown :as md]
            [stasis.core :as stasis]))
With this in place, we can use Optimus to generate the link to the CSS file. However, to do that, it needs access to the request map, so we need to change a few things. We will start with the layout-page function: 
(defn layout-page [request page]
  (html5
   [:head
    [:meta {:charset "utf-8"}]
    [:meta {:name "viewport"
            :content "width=device-width, initial-scale=1.0"}]
    [:title "Tech blog"]
    [:link {:rel "stylesheet" :href (link/file-path request "/styles/main.css")}]]
   [:body
    [:div.logo "cjohansen.no"]
    [:div.body page]]))
Both the partial-pages and markdown-pages need to pass the request to layout-page. If we change them to return functions, Stasis will call those functions with the request. 
(defn partial-pages [pages]
  (zipmap (keys pages)
          (map #(fn [req] (layout-page req %)) (vals pages))))
Remember that #( ) is a function literal, so the mapping function here is a function that returns another function (which takes a request map as its only argument). The markdown generation is similar, but includes the additional step of running the content through cegdown: 
(defn markdown-pages [pages]
  (zipmap (map #(str/replace % #"\.md$" "") (keys pages))
          (map #(fn [req] (layout-page req (md/to-html % pegdown-options)))
               (vals pages))))
Previously these maps contained strings, so we need to update their use now that they're functions. We start with a new function: 
(defn prepare-page [page req]
  (-> (if (string? page) page (page req))
      highlight-code-blocks))
This function takes a page and a request. Because every page will go through this function, some will be strings, and some will be functions. If the page is a string, we leave it untouched, and if it's a function, we call it with the request map and pipe the result through a series of post-processing steps. There's currently only one step, but the threading macro has set us up for easily adding more steps later. The final piece of the puzzle is to update the prepare-pages function: 
(defn prepare-pages [pages]
  (zipmap (keys pages)
          (map #(partial prepare-page %) (vals pages))))
Again, we use the function literal #( ). We also use partial. This returns a new function that knows the first argument to pass to prepare-page. When you call this new function with one argument (a request), the prepare-page function will be called with a page and a request. Update the page in the browser, view source and note that Optimus has now given our CSS file a nice and unique URL.

Export to disk

So far we've only surfed the server version, but the whole point of this exercise was to create something that can work as a static site. To dump the file to disk, start by adding a custom Leiningen build alias in project.clj: 
(defproject cjohansen-no "0.1.0-SNAPSHOT"
  :description "cjohansen.no source code"
  :url "http://cjohansen.no"
  :license {:name "BSD 2 Clause"
            :url "http://opensource.org/licenses/BSD-2-Clause"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [me.raynes/cegdown "0.1.1"]
                 [enlive "1.1.5"]
                 [clygments "0.1.1"]
                 [optimus "0.14.2"]]
  :ring {:handler cjohansen-no.web/app}
  :aliases {"build-site" ["run" "-m" "cjohansen-no.web/export"]} ; New
  :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
This configures `lein build-site` as a command that will invoke the export function in the cjohansen-no.web namespace. Stasis gives us what we need to build this function: 
(def export-dir "dist")

(defn export []
  (stasis/empty-directory! export-dir)
  (stasis/export-pages (get-pages) export-dir))
While this won't technically fail, it also won't be the whole picture. Had we not been using Optimus, this would be OK. Since we are using Optimus, we want to make sure the export is optimized as well. The fix is simple; tell Optimus to dump assets for us, and add an entry to Stasis' request map extensions so that Optimus finds the assets. First update the namespace form to require the Optimus export library: 
(ns cjohansen-no.web
  (:require [optimus.assets :as assets]
            [optimus.export]            ; New
            [optimus.link :as link]
            [optimus.optimizations :as optimizations]
            [optimus.prime :as optimus]
            [optimus.strategies :refer [serve-live-assets]]
            [cjohansen-no.highlight :refer [highlight-code-blocks]]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [hiccup.page :refer [html5]]
            [me.raynes.cegdown :as md]
            [stasis.core :as stasis]))
Then update the export function: 
(defn export []
  (let [assets (optimizations/all (get-assets) {})]
    (stasis/empty-directory! export-dir)
    (optimus.export/save-assets assets export-dir)
    (stasis/export-pages (get-pages) export-dir {:optimus-assets assets})))
Now, on the command line, run lein build-site. After a short while you will find your entire site ready to ship in the dist directory. This can be directly rsynced to your server.

Testing and verification

Building sites like we've done in this post opens for various interesting ways of programatically performing tests and health checks. I will show you two simple, yet immensely useful tests we can add to a site of this kind. You can of course also add unit tests for individual functions, and doing so in a system composed of mostly pure functions is very straight-forward, yet outside the scope of this post.

Testing for 200 OK

One nice test to put in a site like this is an integration test that checks that every page renders without errors. We will use Midje for our tests, so let's update project.clj: 
(defproject cjohansen-no "0.1.0-SNAPSHOT"
  :description "cjohansen.no source code"
  :url "http://cjohansen.no"
  :license {:name "BSD 2 Clause"
            :url "http://opensource.org/licenses/BSD-2-Clause"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [stasis "1.0.0"]
                 [ring "1.2.1"]
                 [hiccup "1.0.5"]
                 [me.raynes/cegdown "0.1.1"]
                 [enlive "1.1.5"]
                 [clygments "0.1.1"]
                 [optimus "0.14.2"]]
  :ring {:handler cjohansen-no.web/app}
  :aliases {"build-site" ["run" "-m" "cjohansen-no.web/export"]}
  :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}
             :test {:dependencies [[midje "1.6.0"]]    ; New
                    :plugins [[lein-midje "3.1.3"]]}}) ; New
We've added a test profile that includes the midje dependencies. Add the following to test/cjohansen_no/web_test.clj: 
(ns cjohansen-no.web-test
  (:require [cjohansen-no.web :refer :all]
            [midje.sweet :refer :all]))

(fact
 "All pages respond with 200 OK"

 (doseq [url (keys (get-pages))]
   (let [status (:status (app {:uri url}))]
     [url status] => [url 200])))
We simply call our get-pages function, loop the resulting map, and call each page function with a request map consisting only of a URL. The comparison is made with a vector of the URL and the status. The reason for this is that the URL will be included in the error message if this fails. This way we can know which pages fail. To run the tests: 
lein with-profile test midje
Doing this will inform us that the generated core_test.clj fails. Just delete it. Other than that, the test confirms that all is well with our site. To keep the tests running while working on the site, run autotest: 
lein with-profile test midje :autotest

Building a link checker with enlive

Another useful test to have in place is a link-checker. We will make one that at least verifies that the internal links between pages in our app are correct, and that they don't cause any unnecessary redirects (e.g. from /about to /about/).
Enlive is very useful for these things. We will use the select function to find all links, and then make sure that the href attribute points to an existing URL if it is a path (not a full URL, which is treated as an external link). First up is the link-valid? function, which checks if a single link is valid given a map of pages: 
(defn link-valid? [pages link]
  (let [href (get-in link [:attrs :href])]
    (or
     (not (.startsWith href "/"))
     (contains? pages href)
     (contains? pages (str href "index.html")))))
The link is considered valid if the href attribute either points to a URL that isn't a path within our app (relative paths are assumed not used) or if it points to one of the pages in the map. We're lenient enough to allow links to /about/ when we have /about/index.html. Since we will use enlive to select all the links, update the test namespace form to this: 
(ns cjohansen-no.web-test
  (:require [cjohansen-no.web :refer :all]
            [midje.sweet :refer :all]
            [net.cgrand.enlive-html :as enlive]))
Then add the test itself: 
(fact
 "All links are valid"

 (let [pages (get-pages)]
   (doseq [url (keys (get-pages))
           link (-> (:body (app {:uri url}))
                      java.io.StringReader.
                      enlive/html-resource
                      (enlive/select [:a]))]
     (let [href (get-in link [:attrs :href])]
       [url href (link-valid? pages link)] => [url href true]))))
Again, we loop all the pages and get them. For each page, we select all links, and expect all of them to pass the link checker. Again we make a slightly strange comparison in the interest of having more than true/false in the output if one of these fail. If you add an invalid link to the markdown file now, running the tests will produce this: 
FAIL "All links are valid" at (web_test.clj:21)
    Expected: ["/my-first-post" "/about/" true]
      Actual: ["/my-first-post" "/about/" false]
18 lines of code to verify all links on the site. Pretty nifty! Rather than having numerous tests that load all the pages, it would probably be a good way to change the structure of the tests such that we only load each page once, and instead register various test functions we want to run for each page. This is left as an exercise for the reader.

Summary

I hope this post has shown you the power and flexibility of Stasis and all the other tools. Perhaps it has even convinced you further of the value of Clojure. I really do dislike number-of-lines-of-code jerkoffs, but it is worth mentioning that we were able to build a reasonably feature-complete technical blog in roughly 100 lines of code using a simple, yet powerful "no batteries included" library like Stasis (which itself clocks in at just over 100 lines of code). I hope you will consider Clojure and Stasis for your next semi-static web project.
Big thanks to Magnar Sveen for proof-reading and correcting this post.

Discuss

from http://webcache.googleusercontent.com/search?q=cache:xOk4FxU2v2EJ:cjohansen.no/building-static-sites-in-clojure-with-stasis+&cd=1&hl=zh-CN&ct=clnk&gl=uk

http://cjohansen.no/building-static-sites-in-clojure-with-stasis


Posted by at
Labels: ,