nordisk/nornir

The docbook pipeline — design

Class: design · Function axis: f0 (the pipeline as one system) + f1 .nornir/ layout · f2 markers & section renderers · f3 class taxonomy & book routing · f4 export/typst · f5 docs index + search · f6 Codeberg Pages. Strict: this file states only what IS, source-backed. Every why, every rejected alternative, lives in docbook-reasoning.md. How to drive it is in docbook-guide.md.

f0 — the pipeline (the whole)

docbook turns hand-edited .nornir/*.md sources into a repo's published documentation set through one typst engine — no JS, no shellout, no Mermaid. Source code: src/docs/ — mod.rs re-exports the whole surface.

The pipeline is four stages, left to right:

1. Sources (f1) — every .nornir/*.md is the source of truth; the repo-root README.md / CHANGELOG.md are regenerated outputs, not sources.
2. Marker expansion (f2) — sections::rewrite (src/docs/sections.rs:352) replaces each <!-- nornir:gen:start:X -->…<!-- nornir:gen:end:X --> region with a freshly rendered table/graph, reading pre-fetched data off a Ctx.
3. typst (f4) — export::export (src/docs/export.rs:106) compiles the expanded markdown to PDF/HTML/MD via an embedded typst engine.
4. Outputs (f3) — book::build_books (src/docs/book.rs:537) routes the classed sources into the README plus three audience PDFs (manual / idiot_guide / unsorted), and search builds a dedicated docs index over the generated docs; pages publishes the book to Codeberg Pages.

Reuse map (what docbook uses): marker renderers pull from the warehouse function (Ctx::history bench runs, Ctx::coverage) and from introspect::depgraph / DWARF; book.rs is a standalone renderer that takes a routing config plus classed sources, so the same library renders any classed doc set. The release function consumes docbook through release::gate::render_and_stage_docs (freshness gate). The CLI/MCP verbs in f0's surface (see guide) are the only entry points.

f1 — the .nornir/ layout & MANAGED_DOCS

Every repo docbook owns has a .nornir/ directory holding the sources; the top-level artifacts are regenerated (RepoLayout, layout.rs:119).

repo/
├── .nornir/
│   ├── README.md / CHANGELOG.md   ← SOURCE (edited)
│   ├── *-{design,guide,reasoning}.md, *-for-idiots.md  ← classed sources
│   ├── assets/     ← TRACKED media (depgraph.svg, bench charts, figures)
│   ├── warehouse/  ← Iceberg data (gitignored)
│   ├── cache/      ← transient: fetched images, docs index (gitignored)
│   └── docs.toml   ← per-repo render + book routing config
├── README.md / CHANGELOG.md   ← GENERATED, chmod -w
└── docs/           ← GENERATED rendered docs (book/manual/idiot_guide/unsorted PDFs)

• MANAGED_DOCS = ["README.md", "CHANGELOG.md"] (layout.rs:44). These fill from their .nornir/<doc>.md template into the repo-root output. Adding one is a one-line change to that list — it then gets init/render/check for free.
• Any other .nornir/*.md is the published file; render_sources_in_place (layout.rs:239) fills its markers in place.
• Path resolution: RepoLayout::source_of / output_of / docs_dir / export_path (layout.rs:147–161). All nornir-managed media lives under .nornir/assets/ (tracked, one namespace).
• Two output destinations: current artifacts land as files under <repo>/docs/ (tracked, link targets); their full history is historized as rows in the Iceberg doc_exports table.

Generated header. Every managed artifact begins with GENERATED_HEADER_PREFIX / _SUFFIX (layout.rs:38–40) — a loud ⚠ GENERATED … DO NOT EDIT banner. write_atomic sets the read-only bit; init_repo (layout.rs:393) strips the header on re-migration so re-init is idempotent.

Render outputs (docs.toml). DocsRenderCfg (layout.rs:71) with outputs = ["markdown"|"pdf"|"all"] selects what render emits:

RenderReport (layout.rs:168) is a no-op when the output is byte-identical.

f2 — marker syntax & built-in section renderers

Syntax. Inside any .nornir/ source, paired HTML comments mark a region that sections::rewrite overwrites each render:

<!-- nornir:gen:start:symbols binary=target/debug/holger limit=20 -->
<!-- nornir:gen:end:symbols -->

Parsed by parse_markers (sections.rs:190) into Marker (sections.rs:148). Rules: start/end names must match; unclosed start is a hard error with line numbers; markers inside fenced code blocks (```/~~~) are ignored; args are re-emitted in sorted order so output is deterministic.

Renderers (dispatched in render, sections.rs:258; each returns the body only, the dispatcher re-emits the marker comments):

Ctx (sections.rs:62) carries repo_root, workspace_root, history, coverage, doc_dir, no_assets; builders new/with_history/with_coverage/with_doc_dir/ markdown_only (sections.rs:103–141). tests groups by Rust-standard categories parsed with syn in test_inventory::scan (test_inventory.rs). depgraph under docs-export renders a static SVG to .nornir/assets/depgraph.svg (tracked, so it resolves on git hosts); without the feature, a self-contained inline SVG emitted in Rust (svg::render_graph_svg, svg.rs:30) — never Mermaid. Its image link is emitted relative to the doc being rendered via Ctx::with_doc_dir / doc_relative_asset.

f3 — class taxonomy & the three books

Every source has a class — the routing key that decides which PDF a chapter lands in. Class is read by classify (book.rs:409): an explicit <!-- nornir:class: X --> override wins (explicit_class, book.rs:474), else the filename suffix, else the bare stem, else "other" (which no book claims → omitted and logged).

Suffix table: book.rs:420–432. canonical_class (book.rs:455) maps a token to a static class name.

Routing is data. BookRoute (book.rs:384: name, classes, out) is loaded from [book.<name>] in .nornir/docs.toml by load_routing (book.rs:488); absent a config, default_routing (book.rs:509) supplies the three-book default:

[book.manual]       # classes = ["readme","changelog","design","guide","benchmarks"]
out = "docs/manual.pdf"
[book.idiot_guide]  # classes = ["idiot"]
out = "docs/idiot_guide.pdf"
[book.unsorted]     # classes = ["reasoning","history","unsorted"]
out = "docs/unsorted.pdf"

build_books (book.rs:537) classifies every discovered source once, warns on any class no book claims (book.rs:557), then emits one BuiltBook (book.rs:369) per [book.*] entry whose class set matches ≥1 source (no empty PDF). build_book (book.rs:614) is the single-document path (whole-set book.pdf); collect_chapters (book.rs:81) discovers + expands the sources into ordered Chapters (book.rs:34), and assemble_markdown (book.rs:352) joins them (each H1 = a chapter/page break, a missing H1 is synthesised from the filename).

The function axis organises each class file: f0 opens with the whole system, then f1…fn describe each function; a function declares what it uses and f0 draws that graph.

f4 — the export / typst pipeline (feature docs-export)

export::export(markdown, meta, format, image_cache_dir, root) (export.rs:106):

1. Parse markdown with pulldown-cmark.
2. Translate events to typst markup via md_to_typst_with (export.rs:447), parametrised by Target (export.rs:77: Paged for PDF/SVG, Html for HTML). Raw spans / code blocks are fenced with a backtick run longer than anything inside, so docs that show fences round-trip.
3. Compile with typst-as-lib: PDF/MD use report.typ (cover, TOC, numbered headings) → PagedDocument → typst_pdf::pdf; HTML uses web.typ and typst's experimental HTML target. DocFormat (export.rs:37) with parse (:51) / extension (:60).
4. Local images (.nornir/assets/…) resolve because a typst filesystem root is mounted at the repo root (ImageResolver, export.rs:264); a raw-HTML <img src> is recovered into a real #image.
5. Remote images (![](https://…)) are downloaded with ureq+rustls into .nornir/cache/images/ (content-hashed) and embedded; None for image_cache_dir skips fetches (offline).

export_repo (export.rs:795) renders the assembled top-level README. Exports are historized in the Iceberg doc_exports table by record_doc_export (warehouse.rs) — one row per (repo, doc_name, version, format, sha256) with bytes inline; list_doc_exports (warehouse.rs) reads them back. resolve_version (book.rs:883) supplies the version stamped on the cover.

f5 — docs index & BM25 search

The rendered docs get their own Tantivy index, separate from the code index, at <repo>/.nornir/cache/docs-index/ (docs_index_dir, search.rs:27):

• Indexes the generated docs only — repo-root managed docs plus <repo>/docs/*.md (discover_doc_files, search.rs:46). The .nornir/*.md templates are excluded here (their markers are unfilled) — they are indexed as source by the workspace code index instead.
• build_docs_index (search.rs:63) builds it; snapshot_docs_index (search.rs:75) persists it into the docs_index_snapshots / docs_index_blobs Iceberg tables, so it time-travels like the code index; restore_docs_index (search.rs:96) rehydrates on a cold cache (optionally pinned to a --sha); search_docs (search.rs:114) runs BM25.

f6 — Codeberg Pages publishing (feature docs-export)

stage_pages_branch (pages.rs:108) publishes docs/book.pdf to https://nordisk.codeberg.page/<repo>/. It stages an orphan pages branch (PAGES_BRANCH, pages.rs:33) holding exactly three files: .nojekyll (NOJEKYLL, pages.rs:45), book.pdf, and index.html (a dark full-viewport PDF viewer, index_html, pages.rs:51). The tree + orphan commit are built in-process via gix pointing refs/heads/pages without touching HEAD or the working tree; the force-push is left to the operator — the command is printed by force_push_cmd (pages.rs:78). PagesBuild (pages.rs:87) reports the staged file names. The pages module is not feature-gated; only the CLI verb is.

Freshness: hooks & the release gate

• Pre-commit hook — install_hooks (hooks.rs:102) writes a tracked .githooks/pre-commit (pre_commit_script, hooks.rs:53) and runs git config core.hooksPath .githooks (set_hookspath, hooks.rs:166). The hook runs nornir docs render <repo> --markdown-only (README/CHANGELOG only, no SVG/PDF; sets Ctx::no_assets via markdown_only), git adds the result, and exits 0 on any failure (never blocks a commit). InstallReport (hooks.rs:37).
• Release render-and-stage — release::gate::render_and_stage_docs runs render_all + render_sources_in_place, rebuilds SVG/depgraph assets, builds the book PDF, and git adds every changed artifact; then check_all (layout.rs:302) is the final drift assertion. Opt-in per repo under [repo.<name>.gates].docs_fresh.