nordisk/skade

skade — reasoning (f0)

Class: reasoning · Function axis: f0 (why this overall shape). The descriptive what is in design.md; this file holds the why, the trade-offs, and the roads not taken. Per-function reasoning: read-path-reasoning, commit-reasoning, time-travel-reasoning.

Why embedded, why redb — vs a REST catalog service

The obvious alternative to an embedded catalog is a catalog service — Nessie, Apache Polaris, a Hive Metastore. Those were rejected for the nordisk use case (one process owns the warehouse) because a service means a JVM to run, a network hop per catalog op, and an operational surface to keep alive. skade instead links as a crate and calls functions: no JVM, no REST endpoint, no Postgres.

redb was chosen as the single-file store over the alternatives:

• vs SQLite — SQLite needs libsqlite3 (a C dependency); the nordisk constellation is pure-Rust by policy (airgap-friendly, no FFI). redb is pure Rust.
• vs a log-structured file of our own — redb already gives ACID, a single exclusive file lock, MVCC read snapshots, and crucially global write-txn atomicity across all tables in the file. That last property is what atomic_release is built on; we would have had to re-implement it.
• vs "no catalog, just files" — Iceberg needs an atomic compare-and-swap on the table pointer; a bare filesystem doesn't give that portably. redb's write txn does.

The cost we accept: single process. redb takes an exclusive file lock, so two processes can't open the same catalog. For the "one process owns the warehouse" model this is a feature (it is exactly what makes the lock-free pointer mirror correct — see read-path-reasoning); if you need multi-writer, you need a service, and this crate is the wrong tool.

Why the whole design leans on immutability

Iceberg metadata files are immutable and content-addressed: a commit writes a brand-new <uuid>.metadata.json and flips the pointer. Every architectural shortcut in skade falls out of that one fact — the caches need no invalidation, the time-travel index can be an immutable-swap structure, the commit log is append-only. Rather than fight it, the design makes immutability load-bearing everywhere. The per-function reasoning docs each trace their piece of this.

Why two crates (skade-katalog vs skade)

The published crate is skade-katalog — lean: redb + iceberg + caches. The heavy data-plane deps (arrow, parquet, datafusion) live in a separate detached workspace crate skade so the catalog's manifest and lockfile stay small and its airgap/pure-Rust story stays clean. A consumer that only needs a catalog pays for only a catalog. The data-plane design notes (read-delta / CDC, object-store, S3-ingest, the arrow-58 fork and its perf patch) are preserved alongside this book as reasoning chapters; they describe the skade crate, not the published skade-katalog surface this manual documents.

Rejected / deferred

• A built-in REST endpoint — deliberately out of scope; if you need REST, put an axum shim in front (the bench harness has one). Keeping it out keeps the library dependency-light.
• Multi-process access — see above; a non-goal, not a gap.
• Full schema-evolution TableUpdates — bounded by what iceberg-rust 0.9.1 exposes publicly; unblocks on upstream 0.10. See catalog-design.