nordisk/nornir

nornir — complete overview

The real README: the full pitch, quickstart, design, example and measured numbers. The repo-root README.md is the slim face (title · mashup · pointers); this chapter is the heavy detail, and it is also folded into the manual docs/book.pdf by nornir docs book.

nornir is project-management with time-travel — your vegvísir (Old Norse: wayfinder) for a multi-crate Rust workspace: ONE append-only, git-SHA-time-travelled warehouse that unifies code-intelligence, releases, tests and a knowledge graph, fronted by a self-discovering completeness gate that goes green only when every surface is test-wired. The crossover of what Maven + Jira should have been.

Why nornir — it does three tools' jobs (code-intel · dev-portal · build-graph) out of one warehouse — and the rows marked NA are things none of those tools have a concept for at all:

Rows marked NA: the rival has no equivalent concept — not a missing feature, a missing category. Measured numbers below.

What it does

• Accelerate airgapped AI (models/agents) on large codebases. Every index and the Plan-DAG is served over MCP, so a local, offline agent works a huge workspace by query — no full-repo scans, and no code ever leaves the machine.
• Remember & time-travel. Historize every fact on Iceberg, keyed by git hash — the plan DAG, the full-text index, the dependency graph, bench/release lineage — then restore or query the workspace as of any past release.
• Explore. Full-text (Tantivy BM25) and semantic/vector search side by side — and hybrid, fusing the two by Reciprocal-Rank-Fusion (RRF), with a Graph-RAG expansion along the call graph. The exact-flat cosine index is the oracle; a vendored, rayon-free HNSW ANN (znippy_zoomies::vann) takes the live dense path at scale. Plus a syn/DWARF knowledge map (symbols, call graph) and dependency blast-radius across the workspace. Code is embedded with jina-v2-base-code (768-dim) and the vectors are materialized in the warehouse, so semantic search at any past git SHA is a warehouse read — never a re-embed or a git walk. Loading a repo's vector index is single-digit milliseconds (≈7 ms for ~120 vectors, ≈15 ms for ~430); embedding runs on CPU (pure-Rust tract) or GPU (ort + CUDA, ~100× faster) behind one Embedder interface.
• Diagrams from data. Because the whole workspace lives in the warehouse, any graph in it can be drawn as static SVG via typst (no Mermaid, no JS): the dependency graph below is generated, and the nornir-ui visualizer weaves release timelines and cross-repo dependency edges into time-travel threads.
• Plan. Connect real artifacts (source files, built binaries) and the inputs that drive them — a prompt or a written requirement (a krav, in any language) — into a Plan-DAG; topo_ready hands the next ready node to a human or an agent.
• Guard & revert. Lock release-critical files (chmod -w) against chaotic AI agents, models, or teammates, and detect tampering against a recorded manifest. Because every version is keyed by git SHA in the warehouse, time-travel to find the culprit and roll back to a known-good release — the revert itself rides on git.
• Release & document. Build, test, and bench-gate every crate, then cargo publish in dependency order — and (re)generate the README/CHANGELOG and a static website (PDF/HTML via typst) describing exactly what shipped.

Quickstart

# stand in a workspace_<name>/ dir (or add --workspace <name> to any command).
# index a repo into the warehouse — symbols, call-graph, full-text, all keyed by git SHA:
nornir map nornir

# the autonom completeness GATE — exits green ⟺ every surface (CLI/UI/MCP/gRPC/fn) is test-wired:
nornir test coverage nornir

# regenerate THIS README + its dependency diagram, straight from the warehouse:
nornir docs render nornir

Design

• 100% Rust. git via gix, Apache Iceberg via iceberg-rust over skade-katalog.
• One warehouse, append-only. Every fact is an Iceberg row keyed by git SHA: plan-DAG events, dependency edges, bench results, test outcomes, release lineage, full-text index snapshots, and code facts — symbols / call-edges from syn, and DWARF symbols read out of built binaries via gimli.
• One library, four binaries. nornir (CLI) · nornir-mcp (MCP/stdio, feature mcp) · nornir-server (gRPC/tonic, feature server) · nornir-ui (egui time-travel visualizer, feature viz). Build the CLI with cargo build; add --features mcp,server,viz as needed. The CLI runs embedded (opens the warehouse in-process, the default) or as a client of a running nornir-server — mutually exclusive per warehouse, since one process owns the write lock.

Documentation

The full reference — every subsystem chapter — is rendered by nornir docs book into a single manual under docs/ and historized in the warehouse:

• 📖 docs/book.pdf — the complete manual (PDF)
• 📝 docs/book.md — the same, as Markdown

It covers the design overview, docs generation, the Iceberg warehouse, the bench framework, vector/semantic search, and the release & history flow (how a release captures a complete SHA-keyed snapshot of the workspace, and how to read any past release back). The rendered docs get their own full-text index — nornir docs search <repo> "<query>" — kept separate from the code index and time-travelable at any past git SHA.

Example

nornir manages owned repos — the holger artifact server and the znippy compression library — and maps their full build-dependency DAG down through the external crates they don't own, such as Apache Arrow. Inside a repo, cargo owns the workspace (build order and layout); nornir works at the repo level — it finds each repo's root Cargo.toml and only orders across repos. The nornir.toml lists the owned repos and their crates.io publish order; the dependency graph is derived from cargo metadata, internal crates kept separate from external ones:

[repo.holger]                                   # owned — artifact server
remote = "https://codeberg.org/nordisk/holger"

[repo.znippy]                                   # owned — compression library
remote = "https://codeberg.org/nordisk/znippy"
# crates.io upload order: members go up in dependency order — published
# versions are immutable, so it's sequential, not atomic.
publish_order = [
  ["znippy-common"],
  ["znippy-compress", "znippy-decompress"],     # phase 2, uploaded in parallel
]

nornir release run        # test → bench-gate → publish the owned crates, in dep order
nornir docs render holger # regenerate README + dependency diagram (owned + Apache Arrow, …)

Status

Working today: the warehouse, the plan DAG (funnel_* MCP tools), guard, config, bench history + regression gate, dependency-graph oracle, the knowledge map (syn), the Tantivy index + time-machine restore, semantic/vector search (materialized embeddings + git-SHA time-travel, hybrid RRF + HNSW at scale), binary analysis (nornir bin analyze|bloat|checksec|diff|gate|sbom over compiled ELF, persisted into Mimir), the world-coderoom (nornir coderoom ingest|compat|what-breaks|snapshot-diff — Mímir generalized to the crates.io world at symbol granularity, in a dedicated warehouse), docs generation, and the CLI / MCP / gRPC surfaces.

Vector search — measured

Full vectorization of real repos, then time-travel "load a vector blob" (reconstruct the in-memory index for a git SHA straight from the warehouse). Threadripper 3975WX (AVX2) + RTX 4090, f32 model:

Embedding throughput (same f32 model, one Embedder interface, two backends):

Loading a blob is GPU-independent (it's a columnar read); the GPU only accelerates embedding. Unchanged chunks are never re-embedded — a one-line edit at a fresh SHA re-embeds a single chunk and still loads in the same ~7/15 ms.

Enabling GPU embedding (CUDA / AMD ROCm)

Embedding is the one GPGPU-compute step in nornir (populate/nornir vector index). The ort (ONNX Runtime) backend dynamically loads a GPU execution provider at runtime; nothing links against a GPU SDK at build time (ort/load-dynamic), and the backend is chosen at runtime, so ONE binary can carry CPU and GPU paths and fall back to CPU when no GPU is present.

• NVIDIA / CUDA — in the default build already (embed-ort is a default feature). Install the CUDA + cuDNN runtime libs; ort requests the CUDA EP automatically when it finds them, else falls back to CPU. No rebuild needed.

• AMD / ROCm — not default (it turns on ort's migraphx + rocm execution providers, which only make sense on an AMD box). Build with the embed-ort-rocm feature:

cargo install --path . --features embed-ort-rocm    # adds the AMD MIGraphX/ROCm EP
# or, from a checkout:
cargo build --release --features embed-ort-rocm

Runtime prerequisite — ROCm must be discoverable. The ROCm runtime libs (libamdhip64, libMIOpen, librocblas, libmigraphx_c, …) are dlopened lazily; nornir preloads them with RTLD_GLOBAL so the EP's NEEDED sonames resolve. Discovery order (first hit wins):

1. NORNIR_ROCM_LIBS — explicit colon-separated lib dirs (highest priority);
2. $ROCM_PATH / $HIP_PATH → …/lib (+ …/lib64);
3. known system dirs: /opt/rocm*/lib, /usr/lib/x86_64-linux-gnu.

ROCm is commonly installed but not on PATH/loader path (e.g. /opt/rocm-7.2.4). Point nornir at it before running:

export ROCM_PATH=/opt/rocm-7.2.4
export PATH="$ROCM_PATH/bin:$PATH"
export LD_LIBRARY_PATH="$ROCM_PATH/lib:$LD_LIBRARY_PATH"
# or, most targeted:  export NORNIR_ROCM_LIBS=/opt/rocm-7.2.4/lib

The AMD EP is only requested when both the embed-ort-rocm feature is compiled and the runtime probe finds ROCm installed — on any other box the embedder requests CUDA-then-CPU exactly as before, so a non-AMD machine is unaffected. EP precedence: AMD ROCm → NVIDIA CUDA → CPU (MIGraphX is tried ahead of plain ROCm because it graph-compiles the model). Missing libs are not fatal: ort silently falls back to CPU.