nordisk/nornir

Warehouse

Internal reference for Urðr — nornir's single append-only Apache Iceberg store, and the canonical catalogue of every table / projection it holds. This is the "where does each fact live" map. It lives in .nornir/ as a durable design note and is a chapter of nornir docs book.

Source: src/warehouse/ — iceberg_schema.rs (every table schema), iceberg.rs (ensure_layout, the TABLE_* names, append/scan), mod.rs (the Warehouse trait + BenchFilter); the catalogue is the standalone skade-katalog (redb). Decisions: plan.md §4 ("Decided").

What it is

One Iceberg warehouse per workspace, under <storage>/.nornir/warehouse/ (default workspace_holger/.nornir/warehouse/), catalogued by skade-katalog (single-file redb) over object_store::local. Every derived fact about the workspace is an Iceberg row keyed by git SHA (and a snapshot/release id), so the whole history is queryable and time-travellable. The server is the sole holder of the warehouse lock; CLI / MCP / viz are thin clients.

It is live local data — never checked into git (.nornir/warehouse/ is gitignored in every repo). The on-disk .nornir/cache/ materialised views (Tantivy index, vector .nvf) are rehydrated from warehouse blobs on demand.

Laws (locked — see plan.md §4)

• Append-only, git-SHA-keyed. Facts are never mutated in place; a new row supersedes. Identity is the git SHA (+ snapshot/release id), never row order.
• Schema evolution is additive-nullable only. Never rename or retype a required field; add nullable columns (this is how release_lineage gained the *_snapshot_id time-machine pins).
• Idempotent writes. Each table has an identity key (listed below); a re-run with the same key must not double-count.
• One backend. IcebergWarehouse (file:// via the skade-katalog redb catalogue) today; RemoteWarehouse (gRPC, Phase 5) later — same Warehouse trait surface.

The tables (every projection)

Registered in iceberg::ensure_layout; schemas in iceberg_schema.rs. Adding a table is a schema fn + a TABLE_* const + an ensure_layout entry + an append/scan path (see Adding a table). Identity columns are the idempotency key.

Release & publish

Bench & test

Dependency graph (DAG 1)

Plan DAG (DAG 2 — the funnel)

Knowledge map (Mímir — source facts)

Artifact time-machine (snapshot/blob pairs)

A uniform two-table shape (artifact_snapshots + artifact_blobs schemas) reused under several names — byte-for-byte derived artifacts keyed by source git SHA, rehydrated into .nornir/cache/ in ~20–30 ms. See the time-machine note below. Tantivy is live; DWARF/gimli/rustdoc are schema-only (nothing writes them yet).

Rendered documentation

Vector / embeddings (see vector.md)

How a fact gets in

• Release pipeline (release::pipeline::run_pipeline) writes the release/bench/test/dep-graph tables in one pass, tagged release_id + per-repo git_sha + dep-graph snapshot id.
• nornir bench run <repo> spawns the repo's bench example, parses the BenchRun JSON, and appends bench_runs/bench_results/test_outcomes.
• nornir warehouse import-jsonl <repo> loads a legacy bench_history.jsonl into bench_runs (the jsonl stays as the on-disk receipt + the docs renderer's fallback source — see bencher.md).
• knowledge scan / index build / vector index write the knowledge, full-text, and embedding projections.

Surfaces

Warehouse-root precedence & lock-tolerant reads

The server is the sole holder of the warehouse's exclusive catalog.redb lock (skade-katalog is single-writer). But a dev-side local CLI read — the docs_fresh gate, docs render, release trace, a dep-graph load — must not deadlock against a live nornir-server that already holds that lock.

Root resolution (warehouse::warehouse_root, kept in lock-step with config::Loaded::warehouse_root and the server):

1. explicit [storage].local_path → <workspace_root>/<local_path>/warehouse — the repo-/workspace-local home, the recommended default for a CLI;
2. otherwise the shared workspace_holger/.nornir/warehouse fallback — which the live server also defaults to, so a CLI and the server can land on the same file. That collision is exactly why read paths must tolerate the lock.

Lock-tolerant read open (warehouse::open_read_only → IcebergWarehouse::open_read_only). A read first tries a normal exclusive open. If that fails with a redb exclusive-lock error (is_catalog_lock_error), instead of aborting it:

• prints a one-line WARNING: (degraded mode is never silent),
• copies catalog.redb aside into a tempdir and opens that copy (open_snapshot) — a point-in-time read-only catalog snapshot. The temp dir is tied to the warehouse lifetime; only the few-MB catalog index is copied, not the data: table files are referenced by absolute file:// paths, so they still resolve to the live warehouse.

The snapshot is read-only by construction — any write would land in the throwaway temp dir, not the live catalog — so mutating callers must use plain open. This is what lets a CLI read coexist with the server instead of being wedged by its lock; CLI↔server stay isolated without the human having to stop the server first.

Funnel-store root resolution (CLI ↔ server ↔ MCP)

The funnel (funnel_events, DAG 2) has its own root override, separate from the main warehouse root above. This exists so the global server install can point every surface at one shared funnel dir. The single source of truth is funnel::Store::resolve_root(workspace_root, local_path, explicit) — the CLI (nornir funnel …), the gRPC server (nornir-server), and the MCP (nornir-mcp) all call it, so a funnel written by one is read by the others.

Precedence (highest first):

1. explicit --funnel-root <dir> flag (none wired today; the hook exists);
2. NORNIR_FUNNEL_ROOT — the global override, used as-given (no warehouse/ suffix). nornir serve install writes it into the generated nornir.env (default /home/nornir/funnel), so on a real install every surface lands on that exact dir;
3. config [storage].local_path → <workspace_root>/<local_path>/warehouse (kept in lock-step with config::Loaded::warehouse_root);
4. repo-local default → <workspace_root>/.nornir/warehouse (= FunnelStore::default_root).

Why this is its own rule (E4 fix, 2026-06-12): the CLI funnel used to resolve via iceberg_warehouse_root (which ignores NORNIR_FUNNEL_ROOT and whose empty-local_path default is workspace_holger/.nornir/warehouse), while the server/MCP read the global NORNIR_FUNNEL_ROOT. So a nornir funnel submit/plan/node wrote one place and the viz Funnel tab (via the server) read another — the funnel never appeared. Routing all three through resolve_root makes the CLI write exactly where the server reads. Note the level-4 default is .nornir/warehouse (the server's default_root), deliberately not the workspace_holger/.nornir/warehouse the bench/dep-graph tables use — funnel consistency is anchored to the server, which is what viz reads through.

redb single-writer lock: the funnel root is global, so within one server only the first opener holds the exclusive catalog.redb lock. CLI reads (nornir funnel show) use the lock-tolerant Store::open_read_only — they fall back to a copied-aside snapshot instead of deadlocking against a live server. CLI writes (submit/plan/node/link/status) need the real exclusive lock via Store::open; if a server holds it, that surfaces as a clear lock error rather than silently writing to the wrong place. (For a hands-off install where the server is always up, route writes through the funnel RPC instead of the CLI.)

Partitioning (the biggest read win)

A partition spec tells Iceberg how to physically bucket data files by a column's value, so a filtered scan prunes whole files at plan time instead of reading every file ever written. Without it, a scoped read is O(all history); with it, O(the slice you asked for).

Landed — partitioned by repo (identity transform):

How it works in code (src/warehouse/iceberg.rs):

• create_partitioned_table_if_missing(.., &["repo"]) lays the table out with an identity spec (identity_partition_spec binds names→ids against the schema, returns it unbound for TableCreation).
• append_batch calls partition_key_for(table, batch) — it reads the table's default spec and derives a PartitionKey from the batch, then passes it to DataFileWriterBuilder::build(Some(key)) so the file is tagged with its partition. Invariant: identity-on-string only, and each append must be single-partition (all rows share the value) — true for nornir's per-repo appends; it errors loudly rather than mis-tag if a batch ever spans repos.
• Reads are unchanged: the existing repo == X scan predicate now prunes by partition (guaranteed) instead of relying on row-group stats.

No backwards-compat: the partition spec is set at table creation, so an existing warehouse must be recreated to pick it up (accepted, 2026-06-07).

Remaining: optional (repo, machine, month) for bench_runs; then the non-partition wins below (compaction, uuid keys, idempotency index).

Other efficiency gaps (plan.md → "Iceberg warehouse efficiency")

Still open: an idempotency side-index (point lookup vs full scan per write); compaction + expire_snapshots; native uuid/fixed key types instead of String (kills per-row parse); drop always-null test_outcomes columns; and lazy-create the dwarf/gimli/rustdoc stub tables on first write instead of eagerly in ensure_layout.

Adding a table

1. Add a pub fn <name>() -> Result<Schema> in iceberg_schema.rs (additive nullable columns only if evolving an existing one).
2. Add a TABLE_<NAME> const + an ensure_layout entry in iceberg.rs: create_table_if_missing (unpartitioned) or create_partitioned_table_if_missing(.., &["repo"]) when the table is written single-partition and read scoped (see Partitioning above).
3. Add an append path and a query_* method (mirror append_bench_run / query_bench_runs); choose an idempotency key and document it in the table above.
4. If it's a derived artifact, prefer the artifact_snapshots/_blobs two-table shape over a bespoke schema.

Gotchas

• Never commit .nornir/warehouse/ — it's live data; it's gitignored. The catalogue is a single redb file; deleting it orphans the tables.
• Idempotency is per-key, not per-call — re-running a release with the same release_id updates in place; a new id appends. Don't key by timestamp.
• Schema changes are additive-nullable only. Renaming a required field breaks every historical row's read.
• The docs renderer reads bench_history.jsonl first as a fallback, iceberg when present — keep the jsonl receipt in sync via bench run (see bencher.md).