nordisk/skade

Git-like refs (skade_katalog::git) — design & guide

Branches, tags, WAP publish, and rollback for the warehouse — 100% within the Iceberg spec. Every Iceberg table already carries a refs map (movable branches, immutable tags); this surfaces it at the catalog level and adds the git control-plane operations on top, all expressed as standard Iceberg SetSnapshotRef / RemoveSnapshotRef updates driven through the existing atomic commit path. Tables stay pure Iceberg — any reader works at a given ref.

See the capability it lands from: .nornir/git-like-catalog.md (Tier 1) and .nornir/warehouse-capability-gaps.md.

1. Why (the gap it closes)

skade-katalog already had the versioned-metadata spine a git-like catalog needs — atomic multi-table commit (atomic.rs), a current-pointer mirror (pointer_cache.rs), a durable commit log, a Ragnar snapshot index (static_index.rs), and time-travel by snapshot-id / timestamp (load_table_at / load_table_as_of). What it did not expose was Iceberg's git surface: you could not create a branch, tag a release, run write-audit-publish, or roll a table back to a known-good state through the catalog — even though iceberg-rust models all of it natively (SnapshotReference / SnapshotRetention). This closes that gap without a new dependency, without a JVM (a Nessie-without-the-JVM start), and without touching any existing read/write path.

2. The idea (refs are just snapshot pointers with a policy)

An Iceberg branch is a movable named pointer to a snapshot; a tag is an immutable one. Both live in the table metadata's refs map. Every git-like operation is therefore a metadata edit that (a) validates a precondition and (b) sets or removes one ref — exactly what TableUpdate::SetSnapshotRef / RemoveSnapshotRef do. So the whole feature is a thin, correctness-focused layer over the catalog's existing commit_table pointer-advance:

create_branch / create_tag  →  SetSnapshotRef{ ref, SnapshotReference{ snap, retention } }
drop_ref                     →  RemoveSnapshotRef{ ref }
fast_forward / publish       →  SetSnapshotRef{ branch, descendant-snap }   (ancestry-checked)
rollback_to                  →  SetSnapshotRef{ main, ancestor-snap }       (ancestry-checked)

Each goes through one redb write transaction (one fsync), appends the commit log, and updates the L0/L1 caches — the same all-or-nothing discipline the rest of the catalog uses. Reads (list_refs, ref_snapshot_id, resolve_metadata_for_ref) hit the lock-free L1 pointer mirror + L0 metadata cache; no redb transaction on the warm path.

3. The shape that runs

All methods are additive on RedbCatalog (src/git.rs):

The two safety invariants (the correctness core)

Both mutation families are guarded so history can never silently diverge — the same rules git push and git reset enforce:

1. Fast-forward only. fast_forward / publish_branch walk the parent_snapshot_id chain from the target back toward the branch's current head; the move is allowed only if the current head is an ancestor of the target (or equal — a no-op). A non-fast-forward move is rejected with a retryable CatalogCommitConflicts. This is what makes WAP safe: a main that advanced under an audit run refuses to publish rather than rewind.
2. Rewind only. rollback_to allows moving main only to an ancestor of its current head — you can rewind along existing history, never jump to an unrelated line. Nothing is deleted (the newer snapshots stay in the table), so the rollback is reversible with a later fast_forward.

Every move additionally carries a TableRequirement::RefSnapshotIdMatch optimistic precondition on the head it validated, so a concurrent writer that moved the ref out from under us aborts the commit instead of clobbering it.

These invariants are covered by RED-when-broken tests (tests/git_refs.rs): remove the ancestry walk and fast_forward_and_non_ff_guard / rollback_ancestor_guard fail, because a backward fast-forward or a rollback to a sibling line would wrongly succeed.

4. What is NOT here (deferred, honestly)

• Data production on a branch. Landing new snapshots onto an audit branch (the "write" half of WAP) is the writer's job — it needs iceberg-rust's append/manifest transaction actions, which 0.9.1 keeps crate-private. This module operates on snapshots that already exist; it is the ref control plane.
• Cherry-pick / merge / conflict detection (design-doc Tier 1 tail / Tier 2). Cherry-pick needs snapshot re-parenting; catalog-level (cross-table) branches and merge are the Nessie model — a later ladder rung, tracked in .nornir/git-like-catalog.md.
• Ref retention expiry. The retention policy is stored on the ref (BranchRetention, max_ref_age_ms); the expiry sweep that acts on it is not implemented here.

5. Idiot guide (zero-knowledge)

Think of a table's history as a chain of saved states (snapshots), newest last.

• A branch is a sticky note that says "this is the current one" and moves as you save. main is the default branch. create_branch puts a new sticky note on any saved state.
• A tag is a sticky note that never moves — you slap it on a release so you can always find it: create_tag(t, "v1.0", None, None) tags the current state.
• Fast-forward slides a branch's sticky note forward to a newer state, but only if the newer state grew out of where the note is now — you can't skip to an unrelated line, and you can't slide it backward. That is how write-audit-publish works: writers build new states on an audit branch, someone checks them, then publish_branch(t, "audit", "main") slides main forward onto the checked state — atomically, or not at all.
• Rollback slides main backward to an older state it grew from (rollback_to), like an undo. The newer states aren't thrown away, so you can redo by fast-forwarding again.
• Reading a ref: load_table_for_ref(t, "v1.0") gives you the table exactly as it was at that tag — time-travel by name instead of by number.

Nothing here changes how tables are stored: they remain ordinary Iceberg tables, so any Iceberg reader still works. skade just turns on the git buttons.

6. Benchmarks

skade_git_refs (in-proc embedded, bench/examples/nornir-bench.rs) times the git surface over a table with a linear snapshot history and many branches/tags: the lock-free ref-read hot path (ref_snapshot_id, list_refs) and the ref-move write path (create-tag/drop cycles, one redb commit + fsync each). Heavy sweep via BENCH_GIT_REFS / BENCH_GIT_READ_ITERS / BENCH_GIT_WRITE_ITERS.

Measured — HEAVY, host loki (12 cores, 64 GiB)

Table with a 64-snapshot linear history and 256 seeded branches/tags; BENCH_GIT_READ_ITERS=2000000, BENCH_GIT_WRITE_ITERS=5000.

Ref reads ride the same lock-free warm path as resolve_metadata, so they cost a metadata-map read, not a redb hit. Ref moves pay one durable redb transaction (fsync) per commit — the same write cost as any other catalog pointer advance, so the git control plane adds no new hot-path overhead to existing reads/writes.