nordisk/skade

Parallel single-snapshot append — guide

Class: guide · Function: append_parallel. What it means for you as a caller. See append-parallel-design.md for the shape.

What it is

Table::append_parallel(&batches) (and skade::append_parallel / append_parallel_with / append_parallel_props) writes a single fast_append snapshot whose encode is fanned across all cores. Where plain append encodes the whole write into one Parquet file on one core, append_parallel splits the rows into ~one file per core (zero-copy row-aligned slices), encodes them in parallel on the gatling fork-join engine, and commits every file in one snapshot.

Use it for a lone large append — one big RecordBatch (or a few) that you want committed as one snapshot but don't want stranded on a single encoder core. It is the single-write analogue of ingest_parallel (which needs you to pre-split into groups). Compression is applied inside the parallel stage, so a zstd write is parallelised for free.

What you get back — and what changes

• Same rows, one snapshot. The same rows with the same per-column values land in the same single fast_append snapshot.
• Functionally equivalent, NOT byte-identical. The snapshot holds several data files instead of one, so a full scan reads them in a different file order. Every order-independent read — row multiset, count, sum, filtered / point lookups, SQL aggregates — returns an identical result. If you need the exact single-file byte layout or a stable scan order, keep calling append.
• Automatic small-write fallback. Below ~one row group's worth of rows (APPEND_PARALLEL_MIN_ROWS, 8 192) it falls back to the plain single-file append path — byte-identical — because the per-file overhead would outweigh the encode win. You never have to pick; it decides from the row count and core count.

How to observe it

skade.append_parallel_encode (bench arm) A/Bs serial append vs append_parallel directly, reporting serial_cores_busy (~1) vs parallel_cores_busy (toward all cores), the speedup, and rows/sec for each lane. The parallel lane keeps the cores busy and runs faster on a CPU-heavy (e.g. zstd) write.

Numbers are LIGHT. The bench figures are preliminary contended-box measurements — the parallel lane is the faster one, but the authoritative ratio is owed from the idle-box Loki HEAVY sweep (issue #88), not yet recorded here.

Guarantees

• Same rows, same one snapshot as append (functionally equivalent — asserted by append_parallel_matches_append_functionally).
• Fans on a real write, falls back on a small one — asserted RED-when-broken by append_parallel_fans_into_multiple_files_and_falls_back_when_small.
• One gatling engine, no rayon — the encode uses the shared fork-join pool.