nordisk/nornir

The warehouse — reasoning

Class: reasoning. This file holds the why, the trade-offs, and every design nornir rejected for the warehouse. The descriptive shape that actually runs is in warehouse-design.md / warehouse-guide.md / sql-design.md, which by rule mention none of this. If you want "why not X?" — X is here. Locked decisions with dates live in plan.md §4; this file explains them.

Why one append-only warehouse at all

Every nornir function derives facts about a workspace — a bench number, a call edge, a release outcome, a coverage %, a vuln finding. The alternative was to let each function keep its own store (a jsonl here, a sqlite there, a /tmp log for the live run). We rejected that: facts would live in a dozen incompatible shapes, nothing could join across them, and history would be whatever each format happened to retain. A single substrate that every function reads and writes is what makes cross-cutting reads possible at all — "which release regressed this bench on this machine", "which members depend on the crate that just changed", "the docs index as of that SHA". f0 in the design is the payoff; everything below keeps it true.

Why Iceberg (rejected: raw Parquet, sqlite, DuckDB-as-store, a live SQL server)

• Rejected — a pile of raw Parquet files. Parquet gives columnar reads but no table identity, no snapshots, no schema evolution, no atomic multi-file commit. We'd have hand-rolled a manifest format to get any of that. Iceberg is that manifest format, standardised: every append is an atomic snapshot, and the manifest/row-group statistics are what let a filtered scan prune whole files.
• Rejected — sqlite / a single embedded relational DB. It stores rows fine but is row-oriented (bad for the wide analytical scans bench/knowledge do), has no native time-travel-by-snapshot, and no columnar pushdown to hand DataFusion.
• Rejected — DuckDB as the store of record. DuckDB is a superb engine but making it the durable store would tie the on-disk format to one engine. Iceberg keeps the data engine-neutral: DataFusion reads it today, a DuckDB or Flight backend could read the same files tomorrow. The Warehouse trait (mod.rs:107) exists precisely so a SkadeWarehouse / DuckDB / RemoteWarehouse backend can slot in without touching the ~150 call-sites.
• Rejected — a standing SQL server as the primary store. That adds an always-on process and a network hop to what is mostly a single-machine dev tool. Iceberg-over-local-files means the CLI can read the warehouse with no server running at all; the gRPC nornir-server (Phase 5 RemoteWarehouse) is an optional front for the multi-client case, not a prerequisite.

Iceberg's cost is real — small-file accumulation and per-row string-key parsing — which is why warehouse compact / expire exist and why native uuid/fixed key types are an open efficiency item (plan.md → "Iceberg warehouse efficiency"). We took that cost for the standardised snapshots + pushdown.

Why keyed by git SHA (rejected: keyed by wall-clock time, by row order)

Facts are about a state of the code, and the canonical name of a code state is its git SHA. Keying by SHA makes idempotency free (re-scanning the same SHA produces the same key, so a re-run can't double-count) and makes time-travel exact ("the index as of SHA X" is a lookup, not a guess).

• Rejected — keyed by timestamp. Two scans of the same code at different wall-clock times would produce two rows that should have been one; and "roll back to the state before the release" would be a fuzzy time-window instead of an exact SHA. Timestamps are recorded (ts_micros) for ordering, but they are never the identity.
• Rejected — identity = row order / insertion position. Append-only means rows arrive in write order, but making position the identity would break the moment a compaction rewrote files or a backfill inserted history out of order. Identity is the explicit key columns; order is never load-bearing.

Why append-only (rejected: mutate-in-place / update semantics)

A fact, once true of a SHA, is true forever — a superseding fact is a new row, not an overwrite. Mutate-in-place was rejected because it destroys the whole history: you could no longer ask "what did we believe at release N", and a concurrent reader could observe a half-applied update. Append-only + snapshots means every reader sees a consistent point-in-time, and the past is never lost — which is the precondition for the *_snapshots/*_blobs time-machine (f4).

The discipline it forces: schema evolution is additive-nullable only. Renaming or retyping a required field would break every historical row's read, so new information always arrives as a new nullable column with a fresh field id (this is how release_lineage gained its *_snapshot_id pins without a migration).

Why single-writer redb + a copied-aside read snapshot

skade-katalog is single-writer: exactly one process may hold the exclusive catalog.redb lock. The obvious problem is that a live nornir-server holds that lock continuously, which would wedge every dev-side CLI read (docs render, the docs_fresh gate, release trace, warehouse query).

• Rejected — make readers wait for / evict the writer. Forcing the human to stop the server before any read is hostile, and a shared advisory lock invites torn reads.
• Rejected — a full multi-reader concurrent catalog (e.g. a client/server catalog protocol for every read). Overkill for a single-machine tool and it would drag the network path into the common case.
• Chosen — copy the catalog aside and read the copy (open_read_only → open_snapshot, iceberg.rs:319,:343). The catalog index is a few MB; the bulk table data is referenced by absolute file:// paths, so the copy only duplicates the index and still reads the live data. The read is read-only by construction (a write would land in the throwaway tempdir), and the degraded mode is never silent (one WARNING: line).

The subtlety that made this hard: a naive fs::copy races redb's region growth — the server can extend catalog.redb and advertise the larger layout in the header between the copy's read of the header and its read of the tail, yielding a copy shorter than its own header claims, which redb 2.6 panics on (a length assertion). So the copy is taken only across a size-stable window (copy_catalog_consistent, iceberg.rs:231): copy, re-stat, retry until the source length doesn't change across the window and the copy is not short. This was the fix for the fat-viz "empty Timeline" bug — the reader believed it had "latest" and instead crashed or read empty.

Why the native Iceberg DataFusion provider (rejected: a MemTable copy)

For the SQL surface, the tempting shortcut is to scan_arrow each table into RecordBatches and register a DataFusion MemTable. Rejected: a MemTable materialises the whole table into RAM before the planner can prune it, and throws away Iceberg's manifest/row-group statistics — so SELECT … WHERE repo = 'x' would read every row ever written. The native IcebergStaticTableProvider keeps projection + filter pushdown all the way to the Parquet files, and it lives in the same arrow-58 / iceberg-arrow58 dependency universe already used by iceberg.rs, so the Table/Catalog types unify with zero glue. The full rationale is inlined in src/warehouse/sql.rs's module docs; the descriptive version is sql-design.md.

Why the durable source is the warehouse, not /tmp

The viz Live Run pane originally read a release run only from a /tmp-class release-run-*.events.ndjson file the producer flushed per line. That file is GC'd, lost on reboot, and invisible to any other client — so a run that had already happened, and was durably in the warehouse, showed an empty pane on load. That violates the project LAW persist-to-warehouse-stream-not-tmp. We rejected keeping /tmp as the source: the durable record must be a warehouse table (release_events), and the file/SSE tail is demoted to a fast accelerator layered on top of the warehouse baseline (it's the same run, just lower-latency). The row→event mapping folds one identical render path so the pane never branches on where an event came from. Full write-up: live-run-warehouse.md.

Why the funnel store has its own root rule

The funnel (funnel_events, DAG 2) resolves its root through funnel::Store::resolve_root, separately from the main warehouse root, because the CLI, the gRPC server, and the MCP must all land on one shared funnel dir on a global install. Before this (the E4 fix, 2026-06-12) the CLI funnel resolved via iceberg_warehouse_root — which ignores NORNIR_FUNNEL_ROOT and defaults elsewhere — while the server/MCP read NORNIR_FUNNEL_ROOT, so a funnel submit wrote one place and the viz Funnel tab read another and the funnel never appeared. Routing all three through resolve_root (with NORNIR_FUNNEL_ROOT as the global override, used as-given) makes the CLI write exactly where the server reads.

How we got here (the road, briefly)

The warehouse began as bench history only (a jsonl the docs renderer read), which is why bench_history.jsonl still exists as the renderer's fallback and why warehouse import-jsonl exists to migrate it. Epic A ("Authoritative Schema Reference", warehouse-schema.md) was the deep read that catalogued every table against actual write/read sites in src/. Partitioning by repo was retrofitted table-by-table (accepted with no backwards-compat, 2026-06-07: the partition spec is set at table creation, so an existing warehouse must be recreated to pick it up). The Warehouse trait (mod.rs:107) was carved out so the Iceberg backend became an implementation rather than the store, clearing the path for the Phase 5 RemoteWarehouse (Arrow Flight → server) and a possible SkadeWarehouse/DuckDB backend — see .nornir/warehouse-trait-skade-migration.md for the P2/P3/P4 plan. The SQL function ("SQL over the Iceberg warehouse", DataFusion over Iceberg) was the last layer added on top, reusing the same read-only lock-tolerant open.