.nornir/design.md

skade — system design (f0)

Class: design · Function axis: f0 (the whole) — the per-function designs are catalog-design.md, read-path-design.md, commit-design.md, time-travel-design.md, builder-design.md. Strict: this file states only the design that runs. Every why and every rejected alternative lives in reasoning.md; the planning spine is plan.md.

f0 — the whole: one file is your Iceberg catalog

skade-katalog is an embedded, single-file, ACID Apache Iceberg catalog: it implements [iceberg::Catalog] against one redb database file. All catalog metadata — namespaces, namespace properties, table-pointer rows, the immutable commit log — lives inside that file (src/store.rs). The Iceberg table metadata JSON / manifests / Parquet data live in whatever FileIO the builder is given (local fs, S3, GCS). The public surface is five re-exports (src/lib.rs:64): RedbCatalog, RedbCatalogBuilder, RedbCatalogError, and two cache-default constants.

The system is five functions wired into one catalog:

function	what it is	source	design
catalog	the `iceberg::Catalog` surface — namespaces + tables CRUD	`catalog.rs`, `keys.rs`	catalog-design
read-path	the warm read stack — L0 metadata cache, L1 pointer mirror, L1.5 handle cache	`meta_cache.rs`, `pointer_cache.rs`, `table_cache.rs`	read-path-design
commit	the durable write path — single commit, group-commit, atomic multi-table release	`catalog.rs`, `atomic.rs`, `store.rs`	commit-design
time-travel	snapshot-id history — the Ragnar `STree64` index over the commit log	`static_index.rs`, `store.rs`	time-travel-design
builder	construction + configuration (durability, cache budgets)	`builder.rs`	builder-design

The redb tables (the substrate every function shares)

One redb::Database holds five tables (src/store.rs:68), all mutated inside a single WriteTransaction per commit (one fsync):

redb table	key → value	purpose
`tables`	`catalog\x1f table_id` → `metadata_location`	the live table pointer, CAS-advanced on commit
`commits`	`catalog\x1f table_id\x1f snapshot_id:u64_be` → `metadata_location`	append-only commit log — the history the Ragnar index is built from
`namespaces`	`catalog\x1f ns` → marker	namespace existence
`namespace_props`	`catalog\x1f ns\x1f key` → value	namespace properties
`meta`	key → `u64`	monotonic `commit_seq`, `update_counter`, reserved `live_tail_misses`

redb is the source of truth and durability layer; every in-memory layer above it is an accelerator, keyed by an immutable, content-addressed identifier so an entry can only be evicted, never stale.

The reuse graph (what each function `uses`)

                 iceberg::Catalog (trait)
                          │
   ┌──────────────────────┴───────────────────────┐
   │                RedbCatalog                    │
   │   catalog ── uses ──▶ read-path ──▶ store     │
   │   commit  ── uses ──▶ read-path, store        │
   │   time-travel ── uses ──▶ store (commits)     │
   │   builder ── constructs ──▶ all of the above  │
   └───────────────────────────────────────────────┘
                          │ depends on
                          ▼
             redb   ·   znippy-zoomies::STree64   ·   moka / arc-swap / imbl

read-path uses store (the redb read txn is its miss fallback).
commit uses read-path (loads the current table) and store (the group-commit txn + commit-log append).
time-travel uses store (the commits table it indexes and falls back to) and shares nothing mutable with the latest-state read-path.
builder constructs everything and owns the config knobs.

Storage layout

<warehouse-dir>/
  catalog.redb                        # the entire catalog (5 redb tables)
  <namespace>/<table>/
    metadata/<uuid>.metadata.json     # immutable, content-addressed
    data/<uuid>.parquet               # Iceberg data files

One directory is the whole lakehouse: copy it, snapshot it, rsync it.

Concurrency model

All redb access funnels through a tokio::sync::Mutex<Database> (store.rs:137) so the async Catalog trait never blocks a runtime thread on redb's sync API. redb itself allows one writer at a time; the mutex makes that queue explicit and non-blocking. Latest-state reads (table_exists, resolve_metadata, warm load_table) and time-travel index lookups take no lock — they read the ArcSwap-published mirror / static index directly.

Which recipe reader this repo uses, and why (lane AJ, 2026-08-30)

Reader: nornir-build-thing 0.1.11, from crates.io, no path — dev-dependency only (tests/build_variants.rs).

Why: skade is the ONE deliberate registry reader. A hardcoded ../nornir/... path broke any nornir git-worktree whose directory name was not nornir, so this edge stays published-only. Its floor must therefore be a version that EXISTS on crates.io: 0.1.11, not the constellation's local 0.1.12. 0.1.4 had been resolving to 0.1.11 in silence. skade's recipe is schema 1, which is what makes a published reader correct for it.

The rule itself — [nornir-build-thing] schema = N, a refusal that names the version to upgrade to, and a loader that refuses to DROP a key it cannot read — lives once, in edda .nornir/build-design.md, "THE COMPATIBILITY RULE". The cross-version guard that would have caught the 2026-08-30 skew is edda/nornir-build-thing-skew.