dependencies
dev dependencies
cake plugin namespaces
| (this space intentionally left blank) | |||||||||||||||||||||||||||
Core provides all of the functionality around parsing clojure source files into an easily consumable format. | (ns marginalia.core (:require [clojure.java.io :as io] [clojure.string :as str]) (:use [marginalia.html :only (uberdoc-html)] [clojure.contrib.find-namespaces :only (read-file-ns-decl)]) (:gen-class)) | |||||||||||||||||||||||||||
(def *test* "./src/cljojo/core.clj") | ||||||||||||||||||||||||||||
(def *docs* "docs") | ||||||||||||||||||||||||||||
(def *comment* #"^\s*;;\s?") | ||||||||||||||||||||||||||||
(def *divider-text* "\n;;DIVIDER\n") (def *divider-html* #"\n*<span class=\"c[1]?\">;;DIVIDER</span>\n*") | ||||||||||||||||||||||||||||
File System Utilities | ||||||||||||||||||||||||||||
Performs roughly the same task as the UNIX | (defn ls [path] (let [file (java.io.File. path)] (if (.isDirectory file) (seq (.list file)) (when (.exists file) [path])))) | |||||||||||||||||||||||||||
(defn mkdir [path] (.mkdirs (io/file path))) | ||||||||||||||||||||||||||||
Ensure that the directory specified by | (defn ensure-directory! [path] (when-not (ls path) (mkdir path))) | |||||||||||||||||||||||||||
(defn dir? [path] (.isDirectory (java.io.File. path))) | ||||||||||||||||||||||||||||
Returns a seq of clojure file paths (strings) in alphabetical depth-first order (I think?). | (defn find-clojure-file-paths [dir] (->> (java.io.File. dir) (file-seq) (filter #(re-find #"\.clj$" (.getAbsolutePath %))) (map #(.getAbsolutePath %)))) | |||||||||||||||||||||||||||
Project Info ParsingMarginalia will parse info out of your project.clj to display in the generated html file's header. add pom.xml support. | ||||||||||||||||||||||||||||
Parses a project.clj file and returns a map in the following form
by reading the | (defn parse-project-file ([] (parse-project-file "./project.clj")) ([path] (try (let [rdr (clojure.lang.LineNumberingPushbackReader. (java.io.FileReader. (java.io.File. path))) project-form (read rdr)] (merge {:name (str (second project-form)) :version (nth project-form 2)} (apply hash-map (drop 3 project-form)))) (catch Exception e (throw (Exception. (str "There was a problem reading the project definition from " path))))))) | |||||||||||||||||||||||||||
Source File Analysis | ||||||||||||||||||||||||||||
This line should be replaced | ||||||||||||||||||||||||||||
and this one too! | (defn parse [src] (for [line (line-seq src)] (if (re-find *comment* line) {:docs-text (str (str/replace line *comment* ""))} {:code-text (str line)}))) | |||||||||||||||||||||||||||
(defn end-of-block? [cur-group groups lines] (let [line (first lines) next-line (second lines) next-line-code (get next-line :code-text "")] (when (or (and (:code-text line) (:docs-text next-line)) (re-find #"^\(def" (str/trim next-line-code))) true))) | ||||||||||||||||||||||||||||
(defn merge-line [line m] (cond (:docstring-text line) (assoc m :docs (conj (get m :docs []) line)) (:code-text line) (assoc m :codes (conj (get m :codes []) line)) (:docs-text line) (assoc m :docs (conj (get m :docs []) line)))) | ||||||||||||||||||||||||||||
(defn group-lines [doc-lines] (loop [cur-group {} groups [] lines doc-lines] (cond (empty? lines) (conj groups cur-group) (end-of-block? cur-group groups lines) (recur (merge-line (first lines) {}) (conj groups cur-group) (rest lines)) :else (recur (merge-line (first lines) cur-group) groups (rest lines))))) | ||||||||||||||||||||||||||||
Hacktastic, these ad-hoc checks should be replaced with something | ||||||||||||||||||||||||||||
more robust. | (defn docstring-line? [line sections] (let [l (last sections) last-code-text (get l :code-text "")] (try | |||||||||||||||||||||||||||
Last line contain defn && last line not contain what looks like a param vector && current line start with a quote | (or (and (re-find #"\(defn" last-code-text) (not (re-find #"\[.*\]" last-code-text)) | |||||||||||||||||||||||||||
Is the last line's code-text a deftask, and does the current line start with a quote? | (re-find #"^\"" (str/trim (str line)))) (and (re-find #"^\(deftask" (str/trim last-code-text)) | |||||||||||||||||||||||||||
Is the last line's code-text the start of a ns decl, and does the current line start with a quote? | (re-find #"^\"" (str/trim (str line)))) (and (re-find #"^\(ns" last-code-text) | |||||||||||||||||||||||||||
Is the last line's code-text the start of a defprotocol, and does the current line start with a quote? | (re-find #"^\"" (str/trim (str line)))) (and (re-find #"^\(defprotocol" last-code-text) | |||||||||||||||||||||||||||
Is the last line's code-text the start of a defmulti, and does the current line start with a quote? | (re-find #"^\"" (str/trim (str line)))) (and (re-find #"^\(defmulti" last-code-text) | |||||||||||||||||||||||||||
Is the last line's code-text the start of a defmethod, and does the current line start with a quote? | (re-find #"^\"" (str/trim (str line)))) (and (re-find #"^\(defmethod" last-code-text) | |||||||||||||||||||||||||||
Is the last line's code-text the start of a defmacro, and does the current line start with a quote? | (re-find #"^\"" (str/trim (str line)))) (and (re-find #"^\(defmacro" last-code-text) | |||||||||||||||||||||||||||
Is the prev line a docstring, prev line not end with a quote, and the current line empty? | (re-find #"^\"" (str/trim (str line)))) (and (:docstring-text l) | |||||||||||||||||||||||||||
Is the prev line a docstring, the prev line not end with a quote, and the current line not an empty string? | (not (re-find #"\"$" (str/trim (:docstring-text l))))) (and (:docstring-text l) (not (re-find #"[^\\]\"$" (str/trim (:docstring-text l)))) (= "" (str/trim (str line))))) (catch Exception e nil)))) | |||||||||||||||||||||||||||
(defn parse [src] (loop [[line & more] (line-seq src) cnum 1 dnum 0 sections []] (if line (if (re-find *comment* line) (recur more cnum (inc dnum) (conj sections {:docs-text (str (str/replace line *comment* "")) :line (+ cnum dnum)})) (recur more (inc cnum) 0 (if (docstring-line? (str line) sections) (conj sections {:docstring-text (str line) :line cnum}) (conj sections {:code-text (str line) :line cnum})))) sections))) | ||||||||||||||||||||||||||||
How is this handled? I wonder? | ||||||||||||||||||||||||||||
No idea ne | (defn gen-doc! [path] (println "Generating documentation for " path) (with-open [src (io/reader (io/file path))] | |||||||||||||||||||||||||||
and this? | (doseq [section (parse src)] (println section)))) | |||||||||||||||||||||||||||
(defn gen-doc! [path] (with-open [src (io/reader (io/file path))] (parse src))) (re-find *comment* " ;; this is a comment") | ||||||||||||||||||||||||||||
(defn path-to-doc [fn] (let [ns (-> (java.io.File. fn) (read-file-ns-decl) (second) (str)) groups (->> fn (gen-doc!) (group-lines))] {:ns ns :groups groups})) | ||||||||||||||||||||||||||||
Ouput Generation | ||||||||||||||||||||||||||||
Generates an uberdoc html file from 3 pieces of information:
| (defn uberdoc! [output-file-name files-to-analyze] (let [docs (map path-to-doc files-to-analyze) source (uberdoc-html output-file-name (parse-project-file) (map path-to-doc files-to-analyze))] (spit output-file-name source))) | |||||||||||||||||||||||||||
External Interface (command-line, lein, cake, etc) | ||||||||||||||||||||||||||||
(defn format-sources [sources] (if (nil? sources) (find-clojure-file-paths "./src") (->> sources (map #(if (dir? %) (find-clojure-file-paths %) [%])) (flatten)))) | ||||||||||||||||||||||||||||
(defn usage [] (println "marginalia <src1> ... <src-n>")) | ||||||||||||||||||||||||||||
(defn run-marginalia [sources] (let [sources (format-sources sources)] (if-not sources (do (println "Wrong number of arguments passed to marginalia.") (println "Please present paths to source files as follows:") (usage)) (do (println "Generating uberdoc for the following source files:") (doseq [s sources] (println " " s)) (println) (ensure-directory! "./docs") (uberdoc! "./docs/uberdoc.html" sources) (println "Done generating your docs, please see ./docs/uberdoc.html") (println))))) | ||||||||||||||||||||||||||||
The main entry point into Marginalia. | (defn -main [& sources] (run-marginalia sources)) | |||||||||||||||||||||||||||
Example Usage | ||||||||||||||||||||||||||||
Command line example | (comment (-main "./src/marginalia/core.clj" "./src/marginalia/html.clj") | |||||||||||||||||||||||||||
This will find all marginalia source files, and then generate an uberdoc. | (apply -main (find-clojure-file-paths "./src")) | |||||||||||||||||||||||||||
Move these to tests | (merge-line {:docstring-text "hello world" :line 3} {:docs ["stuff"]}) (merge-line {:code-text "(defn asdf" :line 4} {:docs ["stuff"]}) (merge-line {:docs-text "There's only one method in this module", :line 4} {}) ) | |||||||||||||||||||||||||||
Utilities for converting parse results into html.Plus a few other goodies.
Like I said:
hello world | (ns marginalia.html (:use [hiccup.core :only (html escape-html)] [hiccup.page-helpers :only (doctype)]) (:require [clojure.string :as str]) (:import [com.petebevin.markdown MarkdownProcessor])) | |||||||||||||||||||||||||||
(defn css-rule [rule] (let [sels (reverse (rest (reverse rule))) props (last rule)] (str (apply str (interpose " " (map name sels))) "{" (apply str (map #(str (name (key %)) ":" (val %) ";") props)) "}"))) | ||||||||||||||||||||||||||||
Quick and dirty dsl for inline css rules, similar to hiccup. ex. | (defn css [& rules] (html [:style {:type "text/css"} (apply str (map css-rule rules))])) | |||||||||||||||||||||||||||
Stolen from leiningen | (defn slurp-resource [resource-name] (-> (.getContextClassLoader (Thread/currentThread)) (.getResourceAsStream resource-name) (java.io.InputStreamReader.) (slurp))) | |||||||||||||||||||||||||||
(defn inline-js [resource] (let [src (slurp-resource resource)] (html [:script {:type "text/javascript"} src]))) | ||||||||||||||||||||||||||||
(defn inline-css [resource] (let [src (slurp-resource resource)] (html [:style {:type "text/css"} (slurp-resource resource)]))) | ||||||||||||||||||||||||||||
The following functions handle preparation of doc text (both comment and docstring based) for display through html & css. | ||||||||||||||||||||||||||||
Markdown processor. | (def mdp (com.petebevin.markdown.MarkdownProcessor.)) | |||||||||||||||||||||||||||
Markdown string to html converter. Translates strings like "# header! | (defn md -> \"<h1>header!</h1>" [s] (.markdown mdp s)) | |||||||||||||||||||||||||||
Inserts super-fancy characters into the doc section. | (defn replace-special-chars [s] (-> s (str/replace #"->" "→") (str/replace #""" "\""))) | |||||||||||||||||||||||||||
As a result of docifying then grouping, you'll end up with a seq like this one:
| ||||||||||||||||||||||||||||
(defn prep-docs-text [s] s) | ||||||||||||||||||||||||||||
(defn prep-docstring-text [s] (-> s (str/replace #"\\\"" "\"") (str/replace #"^\s\s\"" "") (str/replace #"^\s\s\s" "") | ||||||||||||||||||||||||||||
Don't escape code blocks | (str/replace #"\"$" "") ((fn [t] (if (re-find #"^\s\s\s\s" t) t (escape-html t)))))) | |||||||||||||||||||||||||||
Converts a docs section to html by threading each doc line through the forms outlined above. ex. `(docs-to-html [{:doc-text "#hello world!"} {:docstring-text "I'm a docstring!}]) → "<h1>hello world!</h1><br />"` | (defn docs-to-html [docs] (->> docs (map #(if (:docs-text %) (prep-docs-text (:docs-text %)) (prep-docstring-text (:docstring-text %)))) (map replace-special-chars) (interpose "\n") (apply str) (md))) | |||||||||||||||||||||||||||
(defn codes-to-html [codes] (html [:pre {:class "brush: clojure"} (->> codes (map :code-text) (map escape-html) (interpose "\n") (apply str))])) | ||||||||||||||||||||||||||||
(defn group-to-html [group] (html [:tr [:td {:class "docs"} (docs-to-html (:docs group))] [:td {:class "codes"} (codes-to-html (:codes group))]])) | ||||||||||||||||||||||||||||
(defn dependencies-html [deps & header-name] (let [header-name (or header-name "dependencies")] (html [:div {:class "dependencies"} [:h3 header-name] [:table (map #(html [:tr [:td {:class "dep-name"} (str (first %))] [:td {:class "dotted"} [:hr]] [:td {:class "dep-version"} (second %)]]) deps)]]))) | ||||||||||||||||||||||||||||
(defn cake-plugins-html [tasks] (when tasks (html [:div {:class "plugins"} [:h3 "cake plugin namespaces"] [:ul (map #(vector :li %) tasks)]]))) | ||||||||||||||||||||||||||||
Load Optional ResourcesUse external Javascript and CSS in your documentation. For example: To format Latex math equations, download the MathJax Javascript library to the docs directory and then add
to project.clj. Below is a simple example of both inline and block formatted equations. When \(a \ne 0\), there are two solutions to \(ax^2 + bx + c = 0\) and they are $$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$ | ||||||||||||||||||||||||||||
Generate script and link tags for optional external javascript and css. | (defn opt-resources-html [project-info] (let [options (:marginalia project-info) javascript (:javascript options) css (:css options)] (html (concat (when javascript (map #(vector :script {:type "text/javascript" :src %}) javascript)) (when css (map #(vector :link {:tyle "text/css" :rel "stylesheet" :href %}) css)))))) | |||||||||||||||||||||||||||
Is overloaded? Maybe we should consider redistributing | ||||||||||||||||||||||||||||
header numbers instead of adding classes to all the h1 tags. | (defn header-html [project-info] (html [:tr [:td {:class "docs"} [:div {:class "header"} [:h1 {:class "project-name"} (:name project-info)] [:h2 {:class "project-version"} (:version project-info)] [:br] (md (:description project-info))] (dependencies-html (:dependencies project-info)) (dependencies-html (:dev-dependencies project-info) "dev dependencies") (cake-plugins-html (:tasks project-info))] [:td {:class "codes" :style "text-align: center; vertical-align: middle;color: #666;padding-right:20px"} [:br] [:br] [:br] "(this space intentionally left blank)"]])) | |||||||||||||||||||||||||||
(defn toc-html [docs] (html [:tr [:td {:class "docs"} [:div {:class "toc"} [:a {:name "toc"} [:h3 "namespaces"]] [:ul (map #(vector :li [:a {:href (str "#" (:ns %))} (:ns %)]) docs)]]] [:td {:class "codes"} " "]])) | ||||||||||||||||||||||||||||
(defn floating-toc-html [docs] [:div {:id "floating-toc"} [:ul (map #(vector :li {:class "floating-toc-li" :id (str "floating-toc_" (:ns %))} (:ns %)) docs)]]) | ||||||||||||||||||||||||||||
(defn groups-html [doc] (html [:tr [:td {:class "docs"} [:div {:class "docs-header"} [:a {:class "anchor" :name (:ns doc) :href (str "#" (:ns doc))} [:h1 {:class "project-name"} (:ns doc)] [:a {:href "#toc" :class "toc-link"} "toc"]]]] [:td {:class "codes"}]] (map group-to-html (:groups doc)) [:tr [:td {:class "spacer docs"} " "] [:td {:class "codes"}]])) | ||||||||||||||||||||||||||||
(def reset-css (css [:html {:margin 0 :padding 0}] [:h1 {:margin 0 :padding 0}] [:h2 {:margin 0 :padding 0}] [:h3 {:margin 0 :padding 0}] [:h4 {:margin 0 :padding 0}] [:a {:color "#261A3B"}] [:a:visited {:color "#261A3B"}])) | ||||||||||||||||||||||||||||
(def header-css (css [:.header {:margin-top "30px"}] [:h1.project-name {:font-size "34px" :display "inline"}] [:h2.project-version {:font-size "18px" :margin-top 0 :display "inline" :margin-left "10px"}] [:.toc-link {:font-size "12px" :margin-left "10px" :color "#252519" :text-decoration "none"}] [:.toc-link:hover {:color "#5050A6"}] [:.toc :h1 {:font-size "34px" :margin 0}] [:.docs-header {:border-bottom "dotted #aaa 1px" :padding-bottom "10px" :margin-bottom "25px"}] [:.toc :h1 {:font-size "24px"}] [:.toc {:border-bottom "solid #bbb 1px" :margin-bottom "40px"}] [:.toc :ul {:margin-left "20px" :padding-left "0px" :padding-top 0 :margin-top 0}] [:.toc :li {:list-style-type "none" :padding-left 0}] [:.dependencies {}] [:.dependencies :table {:font-size "16px" :width "99.99%" :border "none" :margin-left "20px"}] [:.dependencies :td {:padding-right "20px;" :white-space "nowrap"}] [:.dependencies :.dotted {:width "99%"}] [:.dependencies :.dotted :hr {:height 0 :noshade "noshade" :color "transparent" :background-color "transparent" :border-bottom "dotted #bbb 1px" :border-top "none" :border-left "none" :border-right "none" :margin-bottom "-6px"}] [:.dependencies :.dep-version {:text-align "right"}] [:.plugins :ul {:margin-left "20px" :padding-left "0px" :padding-top 0 :margin-top 0}] [:.plugins :li {:list-style-type "none" :padding-left 0}] [:.header :p {:margin-left "20px"}])) | ||||||||||||||||||||||||||||
(def floating-toc-css (css [:#floating-toc {:position "fixed" :top "10px" :right "20px" :height "20px" :overflow "hidden" :text-align "right"}] [:#floating-toc :li {:list-style-type "none" :margin 0 :padding 0}])) | ||||||||||||||||||||||||||||
(def general-css (css [:body {:margin 0 :padding 0 :font-family "'Palatino Linotype', 'Book Antiqua', Palatino, FreeSerif, serif;" :font-size "16px" :color "#252519"}] [:h1 {:font-size "20px" :margin-top 0}] [:a.anchor {:text-decoration "none" :color "#252519"}] [:a.anchor:hover {:color "#5050A6"}] [:table {:border-spacing 0 :border-bottom "solid #ddd 1px;" :margin-bottom "10px"}] [:code {:display "inline"}] [:p {:margin-top "8px"}] [:tr {:margin "0px" :padding "0px"}] [:td.docs {:width "410px" :max-width "410px" :vertical-align "top" :margin "0px" :padding-left "55px" :padding-right "20px" :border "none"}] [:td.docs :pre {:font-size "12px" :overflow "hidden"}] [:td.codes {:width "55%" :background-color "#F5F5FF" :vertical-align "top" :margin "0px" :padding-left "20px" :border "none" :overflow "hidden" :font-size "10pt" :border-left "solid #E5E5EE 1px"}] [:td.spacer {:padding-bottom "40px"}] [:pre :code {:display "block" :padding "4px"}] [:code {:background-color "ghostWhite" :border "solid #DEDEDE 1px" :padding-left "3px" :padding-right "3px" :font-size "14px"}] [:.syntaxhighlighter :code {:font-size "13px"}] [:.footer {:text-align "center"}])) | ||||||||||||||||||||||||||||
Notice that we're inlining the css & javascript for SyntaxHighlighter ( | (defn page-template [project-metadata opt-resources header toc floating-toc content] (html (doctype :html5) [:html [:head [:meta {:http-equiv "Content-Type" :content "text/html" :charset "utf-8"}] [:meta {:name "description" :content (:description project-metadata)}] (inline-js "jquery-1.4.4.min.js") (inline-js "xregexp-min.js") (inline-js "shCore.js") (inline-js "shBrushClojure.js") (inline-js "app.js") #_[:script {:type "text/javascript" :src "./../resources/app.js"}] (inline-css "shCore.css") (css [:.syntaxhighlighter {:overflow "hidden !important"}]) (inline-css "shThemeEclipse.css") reset-css header-css floating-toc-css general-css opt-resources [:title (:name project-metadata) " -- Marginalia"]] [:body [:table header toc content] [:div {:class "footer"} "Generated by " [:a {:href "https://github.com/fogus/marginalia"} "Marginalia"] ". " "Syntax highlighting provided by Alex Gorbatchev's " [:a {:href "http://alexgorbatchev.com/SyntaxHighlighter/"} "SyntaxHighlighter"] floating-toc] [:script {:type "text/javascript"} "SyntaxHighlighter.defaults['gutter'] = false; SyntaxHighlighter.all()"]]])) | |||||||||||||||||||||||||||
Syntax highlighting is done a bit differently than docco. Instead of embedding the higlighting metadata on the parse / html gen phase, we use SyntaxHighlighter to do it in javascript. | ||||||||||||||||||||||||||||
This generates a stand alone html file (think | (defn uberdoc-html [output-file-name project-metadata docs] (page-template project-metadata (opt-resources-html project-metadata) (header-html project-metadata) (toc-html docs) (floating-toc-html docs) (map groups-html docs))) | |||||||||||||||||||||||||||
Cake plugin for running marginalia against your project.Usage
| (ns marginalia.tasks (:use marginalia.core [cake.core :only [deftask]])) | |||||||||||||||||||||||||||
Run marginalia against your project code. Optionally, you can pass files or directories to control what documentation is generated and in what order. | (deftask marg {files :marg} (run-marginalia files)) | |||||||||||||||||||||||||||
Leiningen plugin for running marginalia against your project.Usage
| (ns leiningen.marg (:use [marginalia.core])) | |||||||||||||||||||||||||||
(defn marg [project & args] (run-marginalia args)) | ||||||||||||||||||||||||||||
You can pass a file, directory, multiple files and/or directories to marginalia like so:
This allows you to control the order in which sections appear in the generated documentation. For example, in marginalia's docs, the leiningen.marg namespace forced to the bottom of the namespace ordering by using this command:
| ||||||||||||||||||||||||||||
A place to examine poor parser behavior. These should go in tests when they get written. | (ns problem-cases.general ) | |||||||||||||||||||||||||||
Should have only this comment in the left margin. See https://github.com/fogus/marginalia/issues/#issue/4 | ||||||||||||||||||||||||||||
(defn parse-bool [v] (condp = (.trim (text v)) "0" false "1" true "throw exception here")) | ||||||||||||||||||||||||||||
(defn a-function "Here is a docstring. It should be to the left." [x] (* x x)) | ||||||||||||||||||||||||||||
Here is a docstring. It should be to the left. | (defn b-function [x] "Here is just a string. It should be to the right." (* x x)) | |||||||||||||||||||||||||||
This is a macro docstring. It should be on the left. | (defmacro foobar [& body] `~body) | |||||||||||||||||||||||||||
This is a defmulti docstring, it should also be on the left | (defmulti bazfoo class) | |||||||||||||||||||||||||||
This is a defmethod docstring. It should be on the left. | (defmethod bazfoo String [s] (vec (seq s))) (bazfoo "abc") | |||||||||||||||||||||||||||
This is a protocol docstring. It should be on the left. | (defprotocol Foo (lookup [cache e]) (has? [cache e] ) (hit [cache e]) (miss [cache e ret])) | |||||||||||||||||||||||||||
(def ^{:doc "This is also a docstring via metadata. It should be on the left."} a 42) | ||||||||||||||||||||||||||||
(def ^{:doc "This is also a docstring via metadata. It should be on the left."} b 42) | ||||||||||||||||||||||||||||
(def ^{:doc "This is also a docstring via metadata. It should be on the left."} c "This is just a value. It should be on the right.") | ||||||||||||||||||||||||||||
From fnparse | ; Define single-character indicator rules. ; I use `clojure.template/do-template` to reduce repetition. (do-template [rule-name token] (h/defrule rule-name "Padded on the front with optional whitespace." (h/lit token)) <escape-char-start> \\ <str-delimiter> \" <value-separator> \, <name-separator> \: <array-start> \[ <array-end> \] <object-start> \{ <object-end> \}) | |||||||||||||||||||||||||||