Un-serialized committer — ingest_pipelined_overlap — guide
Class:
guide· Function:ingest_pipelined_overlap. How to use the overlap-commit fast lane and when to prefer it overingest_pipelined/ingest_multi.
What it does
Table::ingest_pipelined_overlap(groups, files_per_commit, channel_depth) (and the free skade::ingest_pipelined_overlap[_props]) bulk-ingest exactly like ingest_pipelined, but the committer runs beside the writer instead of after it: the fast_append commit fsyncs of already-written file-groups overlap the FileIO writes of the later groups. The resulting table is identical — same rows, same order, same per-files_per_commit snapshot grouping, same commit count — only the commit timing changes.
let mut t = wh.create_table("events", &schema).await?;
// 64 files, commit every 8 → 8 snapshots; up to 4 writes in flight, whose
// FileIO round-trips overlap the committer's fsyncs.
let stats = t.ingest_pipelined_overlap(groups, 8, 4).await?;
println!("{} rows in {} commits", stats.rows, stats.commits);WriteProps control (compression, bloom, row-group size, dictionary) is applied through the Table builder setters (e.g. .compression(ZSTD)) exactly as for the other ingest methods; the free ..._props form takes an explicit &WriteProps.
The knobs
files_per_commit— files folded into eachfast_appendsnapshot. Fewer files per commit = more snapshots = more fsyncs (and more overlap opportunity); more files per commit = fewer, larger snapshots. Identical meaning toingest_pipelined.channel_depth— how far the writer may run ahead of the committer: the bound on both in-flight FileIO writes and the hand-off channel. Larger = more overlap and more peak memory (≈channel_depthencoded files buffered); it backpressures the writer when the committer (fsync) falls behind.
When to use which
| Situation | Use |
| Single table, many commit groups, latency-bound FileIO (S3 / slow fsync) | ingest_pipelined_overlap — hides the serial fsync tail behind writes |
| Single table, one atomic snapshot for the whole load | ingest_parallel (or ingest_multi with one table) |
| Several tables in one flush, want one fsync total | ingest_multi |
| You need all-or-nothing across the whole ingest | ingest_multi (one atomic redb txn) — not the overlap path |
| Conservative / proven serial-commit path | ingest_pipelined |
The one caveat
Like ingest_pipelined's multi-commit behaviour, the overlap path is not atomic across its commits: if a later write or commit fails, an earlier snapshot prefix may already be durable. No partial snapshot is ever produced (a cancelled redb txn rolls back cleanly), but the ingest is not all-or-nothing. If you need that, reach for ingest_multi. This is why the overlap path is a separate opt-in function rather than a silent change to ingest_pipelined.
Verifying
skade/tests/ingest_overlap.rs proves, by direct equality to the serial ingest_pipelined, that the overlap path is byte/row-identical, preserves row order, and produces the same snapshot grouping. The skade.committer_overlap bench arm measures the throughput win (commit-bound shape); the authoritative numbers come from the Loki HEAVY pass.