nordisk/nornir

The warehouse — guide

Class: guide · audience: a techie driving the warehouse from the CLI. The what is in warehouse-design.md; the why in warehouse-reasoning.md. The SQL query surface has its own guide: sql-guide.md. This file is the CLI + republish workflow.

f0 — the shape of every warehouse command

nornir warehouse <op> (WarehouseOp, src/bin/nornir.rs:2068). Every op resolves the warehouse root the same way the server does ([storage].local_path → <workspace>/<local_path>/warehouse, else the home-default). Reads are lock-tolerant — they open read-only and, if a live nornir-server holds the catalog.redb lock, transparently read a copied-aside snapshot (you get one WARNING: line, never a deadlock). Maintenance (compact/expire) needs the real exclusive lock — run it against a fixture warehouse, never the live server's locked one.

f1 — reading bench history (typed)

# All recorded bench runs for a repo, as one JSON object per run:
nornir warehouse bench-query nornir

# Filter to one machine, keep only the last N runs:
nornir warehouse bench-query nornir --machine ryzen-7950x --last 5

For anything richer than a per-repo bench dump, use SQL (warehouse query) — it reaches every table, joins, and aggregates (sql-guide.md).

f2 — importing legacy bench history

# Ingest <repo>/bench_history.jsonl (or the repo's configured history file)
# into the bench_runs table. Runs with no `machine` field are skipped.
nornir warehouse import-jsonl nornir
# → imported 12/13 runs from /…/nornir/bench_history.jsonl

The jsonl receipt stays on disk as the docs renderer's fallback source; keep it in sync by running bench run (see bencher.md).

f3 — maintenance: compact & expire

Run these on a fixture / offline warehouse — they take the exclusive lock and would fight a live server. Both emit a JobRecord so they show in the Jobs panel.

# Merge small files into a few large SORTED files (tighter manifest bounds,
# fewer Parquet footers per scan). Optionally scope to one table / one repo:
nornir warehouse compact
nornir warehouse compact --table bench_runs --repo nornir
nornir warehouse compact --json          # per-table CompactReport array

# Expire old snapshots (keep the last N and/or last D days), then delete each
# snapshot's now-unreferenced data + manifest files. The head is always kept:
nornir warehouse expire --keep-last 10
nornir warehouse expire --keep-days 30 --json

f4 — the republish / incremental-rebuild workflow

You rarely write the warehouse by hand — the republish path does it. In a monitored multi-repo workspace under nornir-server, the poll loop fetches each member, notes which HEADs moved, and rebuilds only the changed members and the members that depend on them (the affected set), not the whole workspace. Nothing to configure.

# Add a workspace (fat mode) — first publish scans + indexes every member:
nornir workspace add myws

# Force a FULL rebuild of every member on demand:
nornir workspace fetch myws --force

Watch the server log to see which path ran:

nornir-monitor: myws changed [util] → republish
nornir-monitor: myws → incremental rebuild, affected [util, liba, libb, app]
# vs.
nornir-monitor: myws → full rebuild over all members

You get a full rebuild (safe, intentional) when: it's the first workspace add; the warehouse is empty/wiped; you passed fetch --force; cargo metadata couldn't build the dep graph; or the workspace has a single member (no cross-repo edges). The blast radius is computed from the cross-repo crate dependency graph, so a purely-runtime dependency won't be picked up as a dependent (true of every dep-graph-driven feature). Design + the affected-set math: incremental-republish.md, howto-incremental-republish.md.

Verify the affected-set logic:

cargo test --features server --lib monitor
# → monitor::tests::affected_set_is_changed_union_dependents_and_skips_unaffected ... ok

f5 — proving a read is warehouse-sourced (not /tmp, not a checkout)

The durable source of every viz pane is a warehouse table; a file/SSE tail is only an accelerator. To confirm a pane truly reads the warehouse:

# Record a run, delete the /tmp accelerator, relaunch — the run is still there:
nornir release run
rm .nornir/logs/release-run-*.events.ndjson
nornir viz            # 📡 Live Run still shows the run, hydrated from release_events

Thin/remote mode reads the same tables over gRPC (Viz.Architecture carries the metro map's symbol_facts/call_edges fuel; fuel_source == warehouse proves it). Details: live-run-warehouse.md, metro-warehouse-read-and-emitter.md.

Gotchas

• Never commit .nornir/warehouse/ — it's live data, gitignored. The catalog is a single catalog.redb; deleting it orphans the tables.
• Idempotency is per-key, not per-call — re-running a release with the same release_id updates in place; a new id appends. Don't key by timestamp.
• compact/expire are exclusive-lock ops — don't point them at a warehouse a live server holds; use a fixture.
• The docs renderer reads bench_history.jsonl first as a fallback, Iceberg when present — keep the jsonl in sync via bench run (bencher.md).