nordisk/skade

skade-katalog — GIT-LIKE catalog (branches · tags · merge) within Iceberg

Status: DESIGN NOTE (2026-06-22, Rickard: "what's missing to be more git-like, what's possible and stays within Iceberg?"). skade-katalog today has atomic commit (atomic.rs) + current-snapshot pointer (pointer_cache.rs) + Ragnar static index (static_index.rs) + meta/table caches, but exposes none of Iceberg's git surface (only an internal SnapshotRetention::Branch in skade/src/delete.rs).

Tier 1 — table-level git: NATIVE to Iceberg, just surface it

Every Iceberg table's metadata carries a refs map: branches (movable) + tags (immutable), each → a snapshot, with retention. iceberg-rust (vendored) already has SnapshotReference / SnapshotRetention. Add to skade-katalog (all 100% within Iceberg):

• create / list / drop branch + tag (write refs via the existing atomic commit)
• write-to-branch → WAP (write→audit branch→validate→fast_forward to main)
• cherry-pick snapshot · fast-forward branch · rollback / set-current-snapshot
• time-travel read AS OF snapshot-id / timestamp / branch / tag
• snapshot log (commit history) + snapshot diff (added/deleted data files, from manifests)
• branch/tag retention (max-ref-age, min-snapshots-to-keep) — enum already present

Tier 2 — catalog-level git: BEYOND vanilla Iceberg (the Nessie model)

Iceberg refs are per-table. Git over the whole warehouse — one branch spanning many tables, atomic cross-table commit, global commit log, merge + diff + conflict detection — is NOT in the spec. That's Project Nessie (LakeFS = object-store level).

skade-katalog is well-positioned: atomic commit + pointer-cache + Ragnar snapshot-id tree ≈ the versioned-metadata DAG such a catalog needs.

• catalog branch = named pointer to a set of table-metadata pointers
• catalog commit = atomically advance many tables' pointers together (content-addressed tree node per commit → the Ragnar tree is this)
• merge / diff between catalog branches; a global commit log
• Iceberg REST commit-transaction (atomic multi-table) = stepping stone for the REST shim
• Tables stay pure Iceberg — any reader works at a given ref; git semantics live in the catalog, Rust-native (a Nessie-without-the-JVM).

Ladder (build order)

1. Tier-1 read side: time-travel by snapshot/tag/branch + snapshot log + diff.
2. Tier-1 write side: create/write branch + tag + WAP (fast-forward/cherry-pick/rollback).
3. Tier-2: catalog branch/commit over the Ragnar tree (single-table-set pointer move).
4. Tier-2: merge + diff + conflict detection + REST commit-transaction.

Why korp/nornir want it

• korp ingest (korp/.nornir/collectors.md §Two deployment profiles + assets/collector-profiles.svg): both the Spark-free (korp-lite) and Spark (korp-spark) profiles land in plain Iceberg, so the branches/tags/time-travel/streaming surfaced here apply to either ingest path.
• korp lineage (korp/.nornir/lineage-and-segmentation.md): catalog commits = run-level lineage anchors; WAP audit branches = where the ingest quality-gate runs before publish.
• nornir warehouse: time-travel by git-sha already maps doc/arch exports to snapshots; catalog branches give per-experiment isolation without copying data.

For idiots

Iceberg already lets a single table have branches and tags like git — we just haven't turned that on in skade. The bigger prize is branching the whole warehouse at once (many tables, one commit, merge between branches) — Iceberg can't do that alone, but skade's Ragnar tree is the right shape to add it while keeping every table a normal Iceberg table.