nordisk/skade

Fast-commit populate — the commit-side levers — design

Class: design · Function: populate · Uses: write, commit, catalog, atomic, builder. The one shape that runs. This is the umbrella over the three commit-side populate levers and the ergonomic surface that makes the fast path the default; the per-mechanism detail lives in ingest-multi-design.md (cross-table single commit) and committer-overlap-design.md (the un-serialized committer).

f0 — the encode is done; the commit is the frontier

skade's bulk encode is already memory-bandwidth-bound and all-core-saturated (write ≈ 35 M rows/s, 12/12 cores — the core-saturation law is satisfied). The warehouse-populate analysis (workspace_nordisk/bencher_agents_problems.md) confirmed the remaining populate cost is not in the encode but downstream of it: the snapshot-commit fsync and the serial committer that issues it. So the levers here never touch the encode — they cut the number of fsyncs, the cost of each fsync, and the time the committer spends blocking on them:

Each is additive: every per-table path (append, ingest_parallel, ingest_pipelined) is unchanged and stays available. The L2 contract holds — the embedded API and its ~35 M rows/s encode do not regress; these are new lanes.

f1 — lever #1: one commit for many tables (Warehouse::ingest_multi)

Every fast_append is one Iceberg snapshot = one redb write txn = one fsync. A populate that writes several tables as one logical unit (nornir's symbol scan writes ~4 tables/scan) therefore pays one fsync per table. ingest_multi all-core-encodes and writes every table's data files exactly as ingest_parallel would (byte-identical files), then stages each table's fast append (Transaction::stage_fast_append — manifests written, catalog pointer untouched) and flips all of them in one all-or-nothing redb transaction via RedbCatalog::atomic_release_raw. N tables → one fsync; each table still gains exactly one new snapshot; a mid-flush failure commits nothing.

Warehouse::ingest_multi(Vec<(Table, groups)>, &WriteProps) is the ergonomic form — it supplies the warehouse's own catalog, so callers pass their Table handles directly instead of threading &dyn Catalog + IceTable through the free skade::ingest_multi. Full mechanism + failure contract: ingest-multi-design.md.

f2 — lever #1b: shed the fsync for bulk import (open_with_durability)

redb's write durability is a per-database setting (WriteDurability::{Immediate, Eventual, None} in skade_katalog::builder). Immediate (the default, and what Warehouse::open uses) fsyncs on every commit — correct for steady-state serving. A bulk import is a different regime: the whole warehouse is rebuildable from source, so the fsync is pure tax. Warehouse::open_with_durability surfaces the knob at the warehouse level:

• Eventual — persists shortly after commit returns; a hard crash may lose the last few commits but the DB stays internally consistent.
• None — no per-commit fsync; the fastest, durability best-effort.

This composes with lever #1: ingest_multi cuts the count of commits, durability cuts the cost of each. Caveat (redb): pages are only reclaimed at higher durability, so a long-lived process writing exclusively at None grows the catalog file — use None for a bounded import, then reopen at Immediate for durable serving. swap_table overrides durability to Immediate regardless, so a compaction swap is never at risk.

f3 — lever #2: the overlapped committer is the recommended default (Table::populate)

ingest_parallel writes every file with a single serial writer; ingest_pipelined overlaps the FileIO writes (the async gatling::io::run_ordered engine, ≤ channel_depth in flight, re-sequenced to submission order) but keeps a serial commit tail (write-all → commit-all). ingest_pipelined_overlap un-serializes that tail: a concurrent committer consumes the ordered write stream so the fast_append fsyncs of early file-groups overlap the writes of the later ones (mechanism: committer-overlap-design.md).

Table::populate(groups, files_per_commit) is the overlapped path driven at a machine-tuned default_channel_depth() (2× cores, clamped [8, 64] — a wider default than the old hand-picked 8; the bounded channel + submission-order reorder buffer keep memory in check regardless). It is the entry point new callers should reach for — both the I/O round-trips and the commit-fsync tail are hidden behind the encode/write stream, and the result is byte/row-identical with the same snapshot grouping as the lower-level paths. ingest_parallel (strict serial writer) and ingest_pipelined (serial committer) remain for callers that need a specific topology or the proven serial-commit contract.

For a multi-table populate, populate (single-table) composes with lever #1: encode+write each table, then one Warehouse::ingest_multi flush.

f4 — observability + tests

The durable data-commit (fsync) count is a process-global counter, skade_katalog::redb_commit_count, bumped once per redb pointer-advance txn (note_data_commit). The tests read a delta on it to prove the collapse rather than infer it:

• ingest_multi.rs — an ingest_multi of N tables advances the counter by 1 vs N for N separate ingest_parallel calls; all-or-nothing; byte-identical (RED-when-broken: regress to a per-table loop and the delta jumps to N).
• populate.rs::warehouse_ingest_multi_wrapper_is_one_fsync — the same 1-vs-N contrast through the Warehouse::ingest_multi ergonomic wrapper.
• populate.rs::populate_default_matches_parallel — Table::populate is byte/row-identical to ingest_parallel (recommended default changes scheduling, never data) and default_channel_depth() is clamped to [8, 64].
• populate.rs::open_with_durability_bulk_import_roundtrips — a populate under None/Eventual lands every row with the correct snapshot count (the fsync-shedding knob never loses or corrupts data).

Because redb_commit_count is process-global within a test binary, every committing test in populate.rs holds one COMMIT_COUNT_GATE so a measured delta is never polluted by a parallel test's commits.

f5 — benches

Host-stamped, plain-number arms in bench/examples/nornir-bench.rs (authoritative numbers are the Loki bencher's):

• skade.ingest_multi — per-table (N fsyncs) vs cross-table (1 fsync) on a symbol-scan-shaped flush; reports the measured multi_fsyncs/separate_fsyncs and the fsync reduction.
• skade.committer_overlap — serial vs overlapped committer, commit-bound shape.
• skade.ingest_strategies — ingest_parallel vs ingest_pipelined.
• skade.bulk_import_durability — a many-commit populate at Immediate vs None; same data + snapshots, only the per-commit fsync tax differs.