nordisk/skade

Parallel single-snapshot append — append_parallel — design

Class: design · Function: append_parallel / append_parallel_with / append_parallel_props (Table::append_parallel) · Uses: write.rs, znippy-zoomies/gatling_forkjoin, parquet. The write-path complement to the footer-stat fold (write-stat-parallel-design.md): that one took the serial footer parse off the write; this one takes the serial encode off a lone append.

f0 — the hot cost: a lone append encodes on one core

append (skade::append, Table::append) encodes the whole write into ONE Parquet file on ONE core — a serial zstd/encode that leaves the other ~11 cores idle. By the core-saturation law that idle is a bug: a single large append (the exact shape nornir's Mimir populate hits — one big zstd-compressed batch) strands eleven cores behind one encoder. ingest_parallel already fans the encode across cores, but it requires the caller to pre-split the rows into per-file groups; a caller with one big batch had no all-core single-commit path.

f1 — the fix: split row-aligned, encode all-core, commit one snapshot

write.rs · append_parallel_props(catalog, table, batches, props) -> Result<IceTable>

1. Split — split_batches_row_aligned cuts the input into up to one group per core using zero-copy RecordBatch::slices (no row is copied or re-materialized; slicing only chooses where the file boundaries fall). The concatenation of the groups in order is exactly the input rows in input order.
2. Encode — the groups are encoded across all cores on the ONE gatling fork-join engine (reusing ingest_parallel_props); compression runs inside that stage, so zstd/snappy is parallelised for free.
3. Commit — files_per_commit = groups.len(), so all the files land in a single fast_append snapshot — matching a lone append's one-snapshot result, not N snapshots.

File count is chosen from machine parallelism and row count: nfiles = (total_rows / APPEND_PARALLEL_MIN_ROWS).clamp(1, cores) — one file per core, capped so every file carries at least APPEND_PARALLEL_MIN_ROWS (8 192 ≈ the small end of a Parquet row group).

f2 — functionally equivalent, NOT byte-identical (L2)

Unlike the footer-stat fold (which is byte-identical), append_parallel changes the physical layout: the snapshot holds nfiles data files instead of one, so a full scan reads the rows in a different file order. It is functionally equivalent to append — the same rows with the same per-column values land in the same one snapshot, so any order-independent read (row multiset, count, sum, filtered / point lookups, SQL aggregates) returns an identical result. Callers that require the exact single-file byte layout (or a stable scan order) keep calling append.

Small-write fallback (byte-identical). Below APPEND_PARALLEL_MIN_ROWS total rows, nfiles collapses to 1 and the call takes the plain single-file append_props path — byte-identical to append — because an extra Parquet footer + manifest entry per split file would outweigh the encode win on a tiny write.

f3 — what is asserted (RED-when-broken)

skade/src/write.rs #[cfg(test)]:

• append_parallel_matches_append_functionally — a large append_parallel and a serial single-file append of the same batch are read back and compared as row multisets: identical rows, identical per-column values (order-independent).
• append_parallel_fans_into_multiple_files_and_falls_back_when_small — a large append must fan into >1 data file (if it regressed to one, the encode serialized on one core), while a sub-threshold append must fall back to a single file. Both halves guard the split/fallback decision directly.

f4 — bench arm (LIGHT; needs the Loki HEAVY pass)

bench/examples/nornir-bench.rs · skade.append_parallel_encode A/Bs the two placements over a zstd batch (SKADE_APE_ROWS, default 200 k rows): lane A is the serial single-file append (encode pins ONE core), lane B is append_parallel (encode fans across all cores). It reports serial_rows_per_sec vs parallel_rows_per_sec, the speedup, and the core-saturation of each lane (serial_cores_busy ~1, parallel_cores_busy toward cores_total); the parallel lane is expected faster with far more cores busy.

These are LIGHT (contended-box) measurements — indicative, not authoritative. The parallel lane runs measurably faster than the serial lane, but the exact ratio is not pinned in-repo; the authoritative figure is owed from the serial Loki HEAVY pass (issue #88) on an idle box, as with the rest of the write-path arms. Consistent with the ⛔ root law, the fan-out uses the ONE gatling fork-join engine — no rayon.