nordisk/skade

The commit / write path — guide

Class: guide · Function: commit. The what is in commit-design.md; the why in commit-reasoning.md.

Commit one table

The standard trait path, when you already hold a TableCommit:

use iceberg::Catalog;
let updated = catalog.update_table(commit).await?;

When you hold the wire pieces (requirements + updates) as plain vecs — e.g. you received them over an Iceberg-REST commit_table request and can't build a TableCommit (its builder is crate-private upstream) — use the skade primitive:

let updated = catalog.commit_table(ident, requirements, updates).await?;

Both validate every requirement against the current metadata first, then fold the updates, write a new metadata blob, and CAS the pointer. A concurrent writer that moved the pointer first → CatalogCommitConflicts (retryable): reload and retry.

Commit several tables all-or-nothing

When several tables form one logical unit (publish bench_runs + dep_graph + components together), commit them atomically — either all heads advance or none:

let tables = catalog.atomic_release([commit_a, commit_b, commit_c]).await?;

Or, from raw triples (same all-or-nothing semantics, no TableCommit needed):

let tables = catalog.atomic_release_raw([
    (ident_a, reqs_a, updates_a),
    (ident_b, reqs_b, updates_b),
]).await?;

If any one table's pointer moved since staging, the whole batch aborts with CatalogCommitConflicts and nothing moves. (Staged metadata blobs that were already written become orphans — run Iceberg orphan-file cleanup if you care.)

Get throughput on a write burst

Two independent levers:

1. Concurrency does the work for you. Fire many update_table / commit_table calls concurrently; group_commit coalesces whatever is queued into a single redb transaction and a single fsync. You don't call anything special — just don't serialise your writers.

2. Relax durability for bulk import / benchmarks:

RedbCatalogBuilder::default()
    .durability(skade_katalog::WriteDurability::Eventual)   // or ::None for bulk
    // …

Immediate (default) fsyncs every commit. Eventual persists shortly after; None skips fsync entirely (fastest, best-effort durability — the db stays internally consistent, but the last commits can be lost on a hard crash).

Observing commit progress

• catalog.commit_seq().await? — durable monotonic count of all commits, survives restarts.
• catalog.pointer_generation() — in-memory count since open; a cheap change-detector.