The warehouse — design
Class:
design· Function axis:f0(Urðr, the one substrate) +f1layers ·f2table catalog ·f3historization ·f4rehydrate / time-travel. Strict: this file states only the design that is — every claim carries afile:symbol/file:linepointer, no why. The why, the rejected backends and the road here live inwarehouse-reasoning.md. How to drive it (nornir warehouse …, republish) is inwarehouse-guide.md. The SQL read surface is split intosql-design.md/sql-guide.md. Every warehouse table's column shape is the catalogue inwarehouse-schema.md.
f0 — Urðr, one append-only Iceberg substrate keyed by git SHA (the whole)
There is exactly one warehouse per workspace: an append-only Apache Iceberg store under <storage>/.nornir/warehouse/, catalogued by a single-file skade-katalog redb catalog. It is the substrate every other nornir function reads and writes — bench, test, release, knowledge (Mímir), docs, funnel, vector, security. Each derived fact is an Iceberg row keyed by git SHA (plus a snapshot / release id), so the whole workspace history is one queryable, time-travellable body of data.
Four laws hold the substrate (src/warehouse/mod.rs, iceberg.rs; the catalogue of what each holds is warehouse-schema.md):
| law | statement | enforced by / pointer |
| append-only, SHA-keyed | facts are never mutated in place; a new row supersedes; identity is the git SHA (+ snapshot/release id), never row order | iceberg::append_batch (src/warehouse/iceberg.rs) — every append is a fresh Iceberg snapshot |
| additive-nullable schema | never rename or retype a required field; new columns append with a fresh, unused field id | iceberg_schema.rs header rule; Warehouse::ensure_columns (mod.rs:152) |
| idempotent writes | each table has an identity key; a re-run with the same key must not double-count | per-table keys in warehouse-schema.md |
| single-writer | one process holds the exclusive catalog.redb lock; every other surface reads lock-tolerantly |
IcebergWarehouse::open vs open_read_only (iceberg.rs:291, :319) |
Reuse map (who uses the warehouse): it is the shared bottom of the stack. The release pipeline, bench run, knowledge scan, index build, vector index, docs export, and the funnel all write projections into it; the CLI, the MCP server, the gRPC nornir-server, and viz all read from it. The two SQL functions (sql-design.md) read every table generically. The substrate is live local data, never committed to git (.nornir/warehouse/ is gitignored); on-disk .nornir/cache/ materialised views (Tantivy, vector .nvf) are rehydrated from warehouse blobs on demand (f4).
f1 — the layers (backend, catalog, single-writer handle)
The stack, top of src/warehouse/mod.rs down to disk:
| layer | what it is | pointer |
Warehouse trait |
the durable, backend-neutral seam — a generic Arrow core (append_arrow / scan_arrow / scan_filtered / scan_limited), schema plumbing (ensure_table / ensure_columns / table_names), and two named bench wrappers |
mod.rs:107 (trait Warehouse) |
IcebergWarehouse |
the one implementation today; owns a tokio Runtime and block_ons so the ~150 sync call-sites never go async |
iceberg.rs:290 (impl IcebergWarehouse), block_on iceberg.rs:483 |
| skade / skade-katalog | Iceberg-rust append/scan (skade) over a single-file redb catalog (skade-katalog::RedbCatalog); ScanFilter is skade's backend-neutral pushdown predicate, re-exported from mod.rs:71 |
use skade::{Scalar, ScanFilter} (mod.rs:71) |
object_store::local |
Parquet data files addressed by absolute file:// paths under the warehouse dir |
warehouse_uri in IcebergWarehouse::open (iceberg.rs:298) |
Root resolution (warehouse::warehouse_root, mod.rs:218, kept in lock-step with config::Loaded::warehouse_root and the server): an explicit [storage].local_path → <workspace_root>/<local_path>/warehouse; otherwise the home-derived config::warehouse_default_root() — which the live server also defaults to, so a CLI and the server can land on the same file.
Single-writer redb handle, two open paths (iceberg.rs):
open(iceberg.rs:291) takes the real exclusivecatalog.redblock — the only path that may mutate.nornir-serveris the sole long-lived holder.open_read_only(iceberg.rs:319) tries a normalopenfirst; on a redb exclusive-lock error (is_catalog_lock_error,iceberg.rs:178) it prints oneWARNING:line and falls back toopen_snapshot(iceberg.rs:343), which copiescatalog.redbaside into a tempdir withcopy_catalog_consistent(iceberg.rs:231— a size-stable copy that survives the server growing the file under it) and opens the copy. The snapshot is read-only by construction: writes would land in the throwaway temp dir. Table data still resolves to the live warehouse (absolutefile://paths); only the few-MB catalog index is copied.
This is what lets a dev-side CLI read (the docs_fresh gate, docs render, release trace, a dep-graph load, warehouse query) coexist with a live server instead of deadlocking on its lock.
f2 — the table catalog (a table of tables)
Registered in ensure_layout (iceberg.rs:488); every column shape is iceberg_schema.rs; the exhaustive per-column reference is warehouse-schema.md (72 tables). The TABLE_* name constants are iceberg.rs:37–:201. Grouped by the function that writes them:
| group | tables | holds | key (idempotency) |
| bench / test | bench_runs, bench_results, bench_telemetry, test_outcomes, test_results, test_impacted_selections, test_quarantines, test_inventory |
one envelope per bench run + free-form metric values, per-test pass/fail, impact-set selection, flake quarantine state |
run_id (+ result_name/metric_name/test_name) |
| release / publish | release_lineage, release_commits, release_events, release_changes, publish_attempts, crate_metadata_checks, crate_tarball_stats, yank_events, registry_mirrors, path_dep_audits, patch_strip_events, msrv_probes, semver_diffs, version_bump_plans/_targets |
full release provenance + rollback handle, conventional commits, the reversible change ledger, post-publish index-visibility polls, gate outcomes | release_id + repo (+ crate_name/version) |
| dependency graph (DAG 1) | dep_graph_edges, links_declarations, resolved_features |
component dep edges per snapshot, links= declarations, resolved feature sets |
snapshot_id + edge/member cols |
| plan DAG (DAG 2) | funnel_events |
every mutation to the ideas→plan DAG, immutable append (8 event kinds) | event_id, ts_micros, kind |
| knowledge (Mímir) | symbol_facts, call_edges, feature_gate_facts, feature_dep_edges, git_heat_facts, scip_occurrences, knowledge_scans, warehouse_access_edges, architecture_wiring |
source facts from the syn walker + SCIP: public items, call edges, #[cfg] gates, per-file git heat, fn→table access — plus the manifest-derived feature-ACTIVATION graph (feature_dep_edges: feature → dep/feature, incl. the weak pkg?/feat AND-group edge cargo metadata cannot supply) |
snapshot_id + repo + item/edge cols |
| artifact time-machine | tantivy_index_{snapshots,blobs}, docs_index_{snapshots,blobs}, dwarf_{…}, gimli_{…}, rustdoc_json_{…} |
byte-for-byte derived artifacts keyed by source git SHA (two-table snapshot/blob shape) — Tantivy live; DWARF/gimli/rustdoc schema-only | snapshot_id+repo+git_sha / snapshot_id+filename |
| rendered docs | doc_exports |
historized docs export/docs book bytes inline, deduped on content |
repo, doc_name, version, format, sha256 |
| vector / embeddings | embeddings, embedding_snapshots, embedding_manifest |
content-addressed deduped vectors + per-snapshot identity + per-occurrence spans | content_hash+model_profile / snapshot_id |
| coverage / surface / security / agents | coverage, surface_coverage, sbom_components, vuln_findings, mcp_requests, agent_model_runs, viz_actions, clone_events, airgap_events, saga_events, build_provenance |
per-crate coverage, CLI/MCP surface coverage, SBOM + vuln scan, MCP request log, agent bake-off runs, viz action trail, clone/airgap/saga event streams, build provenance | per-table (see warehouse-schema.md) |
Partitioning (the biggest scoped-read win, iceberg.rs create_partitioned_table_if_missing): bench_runs, the knowledge-map tables, sbom_components, knowledge_scans, and every artifact *_snapshots partition by repo; artifact *_blobs partition by snapshot_id. A filtered scan then prunes whole data files at plan time instead of reading all history.
Generic browse: table_names() (iceberg.rs:1055) + scan_preview(table, limit) (iceberg.rs:1073) back the viz Warehouse tab and the server's table_names/scan_preview RPCs, so every table is browsable even without a bespoke reader.
f3 — historization (how a fact gets in)
Every append produces a fresh Iceberg snapshot; the writing function tags the row with its identity key:
| writer | tables written | pointer |
| release pipeline | release/bench/test/dep-graph tables in one pass, tagged release_id + per-repo git_sha + dep-graph snapshot id |
release::pipeline::run_pipeline |
bench run <repo> |
bench_runs/bench_results/test_outcomes from parsed BenchRun JSON |
Warehouse::append_bench_run (mod.rs:160) |
warehouse import-jsonl <repo> |
legacy bench_history.jsonl → bench_runs |
run_warehouse (src/bin/nornir.rs:12395) |
| knowledge / index / vector | knowledge, full-text, embedding projections | append_symbol_scan (sql.rs tests seed via it), index::snapshot, vector index |
| docs export/book | doc_exports (bytes inline, dedup by sha256) |
docs::record_doc_export |
| viz Live Run, release, clone, airgap, saga | the event-stream tables — the durable source the viz reads (never /tmp) |
release_events.rs; see live-run-warehouse.md |
The durable-source law is persist-to-warehouse-stream-not-tmp: viz panes hydrate from the warehouse table and keep any file/SSE tail only as a fast accelerator on top (events_from_release_rows in src/viz/live.rs; folded from live-run-warehouse.md, self-emitters.md).
f4 — rehydrate / time-travel by SHA
Because every derived artifact is stored as *_snapshots + *_blobs rows keyed by source git SHA, any past state is reconstructable:
- Rehydrate — the on-disk
.nornir/cache/materialised views (Tantivy code index, docs index, vector.nvf) are rebuilt from warehouse blobs on a cold cache in ~20–30 ms, via the sharedindex::snapshot::snapshot_dir_to_icebergmachinery parameterised by table name (used by both the code index and the docs index — seedocs-generation.mdDocs index). - Time-travel by SHA —
index restore --sha <sha>/docs search --sha <sha>resolve the latest-snapshot-for-repo (partition-pruned onrepo), fetch that snapshot's blobs (partition-pruned onsnapshot_id), and restore the index as of that SHA.release_lineagecarries additive*_snapshot_idtime-machine pins so a release can be traced or rolled back to exact table state. - Incremental republish — the server's poll loop rebuilds only the changed members ∪ their transitive dependents (the Mímir affected set,
WorkspaceGraph::affected_by_changeinsrc/warehouse/dep_graph.rs), not the whole workspace; seeincremental-republish.mdand the workflow inwarehouse-guide.md.
Adding a table
- Add
pub fn <name>() -> Result<Schema>iniceberg_schema.rs(additive nullable columns only if evolving an existing one). - Add a
TABLE_<NAME>const + anensure_layoutentry (iceberg.rs:488):create_table_if_missing(unpartitioned) orcreate_partitioned_table_if_missing(.., &["repo"])when written single-partition and read scoped. - Add an append path + a
query_*method (mirrorappend_bench_run/query_bench_runs); choose an idempotency key and record it inwarehouse-schema.md. - If it's a derived artifact, prefer the
*_snapshots/*_blobstwo-table shape over a bespoke schema.