nordisk/skade

Fast-commit populate — guide

Class: guide · Function: populate. The what/why is in populate-fast-commit-design.md; the per-mechanism detail is in ingest-multi-guide.md and committer-overlap-guide.md.

The one-liner

To fill a table fast, call Table::populate. To fill several tables as one atomic unit with a single fsync, call Warehouse::ingest_multi. To shed the per-commit fsync entirely during a bulk load, open with Warehouse::open_with_durability(dir, WriteDurability::None).

Single table — the recommended default

use skade::Warehouse;

let wh = Warehouse::open("/data/lake").await?;
let mut t = wh.create_table("events", &schema).await?;

// groups: Vec<Vec<RecordBatch>> — each inner Vec becomes one Parquet file.
let stats = t.populate(groups, /* files_per_commit */ 16).await?;
println!("{} rows, {} commits", stats.rows, stats.commits);

populate encodes across all cores, overlaps the FileIO writes, and overlaps the commit fsyncs with the ongoing writes, at a machine-tuned channel depth (skade::default_channel_depth()). Prefer it over the lower-level entries unless you need a specific topology:

• Table::ingest_parallel(groups, fpc) — strict serial writer (simplest).
• Table::ingest_pipelined(groups, fpc, depth) — overlapped writes, serial commit tail (the proven serial-commit contract).
• Table::ingest_pipelined_overlap(groups, fpc, depth) — what populate calls, with an explicit depth.

All four produce a byte/row-identical table with the same snapshot grouping; they differ only in scheduling.

Many tables — one fsync for the whole flush

use skade::WriteProps;

let a = wh.create_table("bench_runs", &runs_schema).await?;
let b = wh.create_table("dep_graph",  &dep_schema).await?;
let c = wh.create_table("components", &comp_schema).await?;

let (advanced, stats) = wh.ingest_multi(
    vec![(a, runs_groups), (b, dep_groups), (c, comp_groups)],
    &WriteProps::default(),
).await?;
assert_eq!(stats.commits, 1);           // ONE fsync flipped all three tables
assert_eq!(advanced.len(), 3);          // each gained exactly one snapshot

The batch is all-or-nothing: if any table's groups can't be encoded, nothing commits — no earlier table advances. Each table still gets exactly one new snapshot; only the commit transactions collapse from N to 1. Single-partition per table, same rule as append.

Bulk import — shed the fsync

use skade::{Warehouse, WriteDurability};

// The whole warehouse is rebuildable from source → skip the per-commit fsync.
let wh = Warehouse::open_with_durability("/data/lake", WriteDurability::None).await?;
// … populate / ingest_multi as above (now at zero per-commit fsync cost) …
drop(wh);

// Reopen at the crash-safe default for durable serving.
let wh = Warehouse::open("/data/lake").await?;

• WriteDurability::Eventual — persists shortly after commit; may lose the last few commits on a hard crash, DB stays consistent. A safe middle ground.
• WriteDurability::None — no per-commit fsync, fastest. Use for a bounded import (redb only reclaims pages at higher durability, so exclusive None grows the file), then reopen at Immediate.

Combine the levers: open_with_durability(None) + ingest_multi gives you the fewest commits and the cheapest commits.

How do I know it worked?

IngestStats.commits reports the snapshot-commit count of a call; for the whole process, skade::skade_katalog::redb_commit_count() is the durable fsync counter — read a delta around a populate to confirm a multi-table flush cost exactly one. The test suite (skade/tests/populate.rs, skade/tests/ingest_multi.rs) asserts all of this RED-when-broken.