nordisk/skade

All-core footer-stat build — guide

Class: guide · Function: ingest_parallel / ingest_multi / ingest_pipelined[_overlap]. What the change means for you as a caller. See write-stat-parallel-design.md for the shape and write-stat-parallel-for-idiots.md for the ground-up version.

What changed

Nothing in the API and nothing in the output. ingest_parallel, ingest_parallel_with, ingest_parallel_props and ingest_multi produce exactly the same Parquet data files and the same Iceberg snapshots as before — byte for byte. The change is internal: the per-file Parquet footer parse that builds each file's prunable DataFile (its per-column min/max bounds, value/null counts, column sizes, split offsets) now runs inside the all-core encode stage instead of in the serial writer loop that follows it.

Why you care

The footer parse is real CPU work. It used to run one file at a time on a single core while the other ~11 cores — which had just finished the encode — sat idle. For a bulk load of many files that serial tail is pure wasted parallelism. Folding it onto the gatling fork-join pool keeps the cores busy through the whole encode phase; the writer that follows is then pure I/O.

You get this automatically — no new argument, no opt-in. Just keep calling ingest_parallel / ingest_multi.

The same fold now also applies to the pipelined writers, ingest_pipelined and ingest_pipelined_overlap. Those run their write stage on the async runtime (the serial tail overlaps I/O via run_ordered; the overlap tail runs the committer beside the writer). Their footer parse used to run inside those async write jobs on the tokio runtime — off the strictly-serial block, but driven one-at-a-time by the reorder task, not on the all-core pool. It now rides the gatling encode burn too, leaving the async write stage as pure I/O. Same API, same byte-identical output, same submission order and snapshot grouping.

When it helps most

• Many files per call. More files ⇒ more footer parses ⇒ more work that was stranded on one core and is now spread across all of them.
• Wide / many-row-group files. The parse cost scales with columns × row groups (more per-column stats to convert), so wide tables and small row groups (finer data-skipping) benefit most.
• CPU-bound loads on a big box. The win is a core-utilisation win; on a fast many-core host the reclaimed cores matter more.

It does not change fsync behaviour: ingest_multi is still one fsync for the whole multi-table flush; ingest_parallel still commits every files_per_commit.

How to observe it

• skade.ingest_parallel_scaling (existing arm) — its cores_busy / core_saturation should rise, since the parse now runs in the parallel stage.
• skade.write_stat_build (arm) — A/Bs the two placements directly for the non-pipelined path and reports serial_stat_cores_busy vs parallel_stat_cores_busy and the speedup.
• skade.pipelined_stat_build (new arm) — the pipelined A/B: runtime_stat (parse on the tokio runtime, the old placement) vs gatling_stat (parse on the gatling worker), reporting runtime_stat_cores_busy vs gatling_stat_cores_busy.

Guarantees

• Byte-identical: same data-file bytes, same DataFile manifest entries, same snapshots (asserted RED-when-broken by write_output_is_byte_identical for the non-pipelined path and pipelined_output_is_byte_identical for the pipelined one).
• Same order: files keep input (submission) order — gatling_map_owned returns in input order, so {prefix}-{seq:08}.parquet is unchanged.
• Same prunability: the DataFile stats are built by the same build_data_file over the same bytes, so scan-planner file/row-group pruning is unaffected.