janus/text
Convert documents and HTML fragments into text that callers can display, search, or send to a language model.
Require janus.text.api. Do not require the internal namespaces.
The HTML map shape is janus.entities.specs.html.
Vocabulary and internals live next to the code in
components/text/doc/. This page is a how-to.
1. Add the dependency and require the API
;; deps.edn
{janus/text {:local/root "components/text"}}(require '[janus.text.api :as text])2. Parse an HTML string
Use this when Livingdocs or another source gives you a string.
(text/parse-html-fragment "Hello <em>world</em>")
;; => ["Hello " {:tag :em :attrs nil :content ["world"]}]The result is a Janus HTML fragment: a vector of strings and
element maps. An element has :tag, optional :attrs, and optional
:content.
Strip tags when you only need the words:
(text/->text (text/parse-html-fragment "Hey <em>Joe</em>"))
;; => "Hey Joe"Walk every node with the Rama path text/HTML-NODES.
3. Turn Janus HTML into markdown
(text/html->markdown
[{:tag :p :content ["Hello " {:tag :em :content ["world"]}]}
{:tag :h2 :content ["Next"]}])
;; => "Hello *world*\n\n## Next"html→markdown accepts a fragment, an element, a string, or nil.
nil becomes "".
The function trims the string.
It collapses three or more newlines to two.
Unknown tags unwrap. Their children stay. The tag disappears.
(text/html->markdown {:tag :div :content [{:tag :p :content ["Hi"]}]})
;; => "Hi"4. Stream markdown onto a Writer
Use html→markdown-to! when you already have a destination, for
example an HTTP response writer.
(import '[java.io StringWriter])
(let [out (StringWriter.)]
(text/html->markdown-to! out [{:tag :p :content ["Hello"]}])
(str out))The function writes onto any java.lang.Appendable.
It returns that same object.
It flushes a Writer. It does not close it.
It does not trim or collapse whitespace.
Wrap an OutputStream first:
(text/html->markdown-to! (clojure.java.io/writer response-stream) html)5. Convert an entity tree to HTML, then to markdown
An entity is a document tree. Leaves hold Janus HTML under keys in
the janus.html namespace.
(def para
{:janus.node/tag :paragraph
:content {:janus.html/text
(text/parse-html-fragment "Hello <em>world</em>")}})
(text/entity->html para)
;; => ["Hello " {:tag :em :attrs nil :content ["world"]}]
(text/html->markdown (text/entity->html para))
;; => "Hello *world*"entity→html takes keyword args. Pass :variant that way.
html→markdown takes an opts map.
(-> entity
(text/entity->html :variant :sve)
(text/html->markdown))The default entity conversion concatenates :janus.html/* on a leaf.
It wraps each child segment of a non-leaf in a :section.
That is enough for simple paragraphs. It is not enough for images,
headers, or lists. Those need a method. See [extend-entity].
A figure URL that lives on the entity is not in the HTML tree.
Markdown will not show it until a method emits an img or figure.
6. Convert a whole article for a language model
Require the tenant methods before you convert. Minerva does that with
janus.sve.minerva.text.
(require '[janus.sve.minerva.text]) ; registers Livingdocs methods
(require '[janus.text.api :as text])
(defn article->markdown [article]
(text/html->markdown
(text/entity->html article :variant :sve)))Load the method namespace for its side effects.
Call entity→html only after that require.
7. Extend entity conversion
entity→html dispatches on the node tag through
text/node→html-method.
A method receives the node and opts.
It must return a Janus HTML fragment.
(defmethod text/node->html-method :paragraph
[node _opts]
(if-let [content (not-empty (get-in node [:content :janus.html/text]))]
[{:tag :p :content content}]
[]))Two Livingdocs names for the same component share one method. Derive the extra name:
(text/derive-node-tag :p :paragraph)A tenant-specific method dispatches on [variant node-tag].
Pass the variant into entity→html.
(defmethod text/node->html-method [:sve :subtitle]
[node _opts]
[{:tag :h2 :content (get-in node [:content :janus.html/subtitle])}])
(text/entity->html subtitle-node :variant :sve)Without :variant, a [variant tag] method is not used.
A missing pair falls back to the method for the tag.
8. Extend markdown conversion
html→markdown dispatches on the HTML :tag through
text/write-html-method.
A method receives [rf opts out el] and must return out.
rf writes children. out is the Appendable.
(defmethod text/write-html-method :figure
[rf opts out el]
(reduce rf out (or (:content el) [])))Unknown tags already unwrap. Add a method only when the default is wrong.
Built-in tags include headings, p, emphasis, links, lists, img,
code, pre, blockquote, br, and hr.
9. Choose the right function
| Need | Call |
|---|---|
HTML string from Livingdocs |
|
Words only |
|
Markdown string for a prompt |
|
Markdown on an HTTP stream |
|
Entity document to HTML |
|
Tenant-specific entity HTML |
|