nordisk/znippy

arrow-ipc metadata-tail benches — the bench-first gate

LAW (bench-first): the throughput benches for the current arrow-ipc write path must exist and be baselined before any write-path optimisation is written, so a new impl's win is a measurable delta against a real number. This doc records the five baseline benches, the numbers they produced, and the gate the three planned optimisations (A/B/C) must beat.

Spec: workspace_nordisk/znippy-arrowipc-analysis.md §2. Benches live in xtask/examples/nornir-bench.rs (the same nornir Bencher harness as every other znippy bench); baseline rows are appended to bench_history.jsonl at the workspace root. No optimisation is implemented in this pass — only the benches + their baseline.

Why these five exist

The existing end-to-end benches (ArchiveBench, StreamBench) measure compress_mbs / decompress_mbs but fold the metadata seal into compress time and never touch the random-access read path. The whole single-threaded metadata tail — sort + lookup re-materialisation + fst trie + manifest + footer — and the per-call re-read on locate_file are therefore invisible in today's numbers. These five benches make each of those stages a stable, isolated metric.

The blob pipeline (left, the 1→N→1 no-barrier gatling flow) is already near-optimal and well-benched — see .nornir/gatling.md and the assets/gatling_flow.svg skade-style diagram for its full 1→N→1 saturation story. The red box is what this baseline measures: the single-threaded seal tail that runs after the gatling pipeline has drained, plus the cold read path.

The five benches

Row scales (the seal scales with rows, not bytes): S = 10k, M = 100k, L = 1M (heavy, gated behind NORNIR_BENCH_SKIP_HEAVY exactly like StreamBench). B1/B2 drive ArrowIpcSink directly with synthesized BlobMeta rows (one distinct, namespace-scattered path per row — worst case for the sort + fst). B3/B4/B5 build a real sealed v0.7 archive via compress_stream so the true read path (manifest + lookup + trie + blob pread + blake3 verify) is exercised. The random subset uses a deterministic LCG so runs are reproducible.

Each bench run() inject-asserts: e.g. random_lookup fails unless all K targets are located; prefix_meta fails unless the empty prefix returns exactly N rows. A #[cfg(test)] mod arrowipc_bench_tests drives each bench at the S scale and asserts the reported metrics are real and internally consistent (the "tests inject values, not just no-crash" LAW).

BASELINE (2026-06-18 · threadripper-3975wx · 32 cores · znippy 0.9.0)

B1 — subindex_write (push one base-schema sub-index of N rows)

B2 — footer_seal (the headline bottleneck: sort + 2nd materialisation + trie + manifest + footer + fsync)

Seal throughput collapses ~0.91→0.65 Mrows/s from S→L; the sort alone is 357ms of the 1.5s L seal. This is the number idea (A) must beat.

B3 — random_lookup_cold (K random locate_file calls, fresh handle each — today's re-read-everything path)

Per-call latency grows linearly with N (0.84 ms → 281 ms, ~335× for 100× the rows) because every cold call re-decodes the entire lookup sub-index. Idea (B)'s cached reader should make this ~flat (one fst get + array slice) and drop manifest_reads_per_lookup 3 → 0 after open.

B4 — prefix_meta (get_files_meta_with_prefix at ~1% / 10% / 100% selectivity)

Even a 1%-selective prefix pays ~the full-scan cost (the whole lookup is decoded before the range is sliced) — visible as prefix_ms_1pct ≈ prefix_ms_100pct at each scale. Idea (B) decouples it.

B5 — decompress_random_subset (selective restore via get_file, M corpus = 100k files, extraction count capped at 2 000)

*~54 files/s because each get_file is a cold locate_file (full lookup decode)

• pread + decompress + blake3 verify. The lookup re-read dominates; idea (B) should lift extract_files_per_s by ~the same factor it cuts random_lookup. (bytes_out_mbs rounds to 0.0 — the synthetic payloads are tens of bytes each; extract_files_per_s is the comparable metric for this selective-restore path.)*

Run notes: S/M/L = 10k/100k/1M rows. random_lookup and decompress_subset are O(K·N) on the cold path, so K (lookups) and the extraction count are capped as N grows (S=2000, M=500, L=100 lookups; ≤2000 extractions) — the per-call/per-file metric is N-normalised, so fewer samples at L is fine and the wall-time stays bounded.

These are the numbers in bench_history.jsonl (the arrow-ipc baseline run, date 2026-06-18). They are the gate. An optimisation is accepted only if:

• (A) incremental / parallel seal drops footer_seal.seal_ms (raises seal_mrows_s), most visibly at M / L scale, with no regression > 10% on the existing compress_mbs / decompress_mbs (the WORKSPACE.md release gate).
• (B) cached ArchiveReader drops random_lookup.lookup_ns_p50 (the design expects ~1–2 orders of magnitude), collapses random_lookup.manifest_reads_per_lookup from 3 → 0 after open, and proportionally drops prefix_meta.prefix_ms_* and raises decompress_subset.extract_files_per_s.
• (C) append-mode + explicit seal() reuses (A)'s incremental seal; it is measured later by an append_seal variant of B2 (full N-row archive + delta of K rows; append_seal_ms vs full seal_ms). Baselined here as the full seal.

How to run

# Target dir is pinned to fast disk via .cargo/config.toml.
# Full suite (planet skipped); the five families below are part of it:
NORNIR_MACHINE=threadripper-3975wx NORNIR_BENCH_SKIP_PLANET=1 \
  cargo run -p xtask --release --example nornir-bench

# Skip the 1M-row (_l) heavy scale for a fast focused pass:
NORNIR_BENCH_SKIP_PLANET=1 NORNIR_BENCH_SKIP_HEAVY=1 \
  cargo run -p xtask --release --example nornir-bench

# Harness self-test (inject-assert, no timing):
cargo test -p xtask --release --example nornir-bench arrowipc_bench_tests

Sequencing (from the analysis)

1. These benches first (this commit) — implement + baseline. ✅
2. (B) cached ArchiveReader — lowest risk, biggest read win, zero format change. Validate against B3/B4/B5.
3. (D) small seal cleanups (no per-row String clone, streamed sub-index) — validate on B1/B2.
4. (A) incremental / parallel seal — validate on B2.
5. (C) append + seal() — last; add the append_seal bench alongside.

A/B AUDIT (2026-06-18) — ArrowIpcSink (A) vs ArrowIpcSinkAppend (B)

The question: does the new resume/append/seal feature hurt the normal write speed? ("speed inte drabbas") The new capability is a near-identical clone of the original writer, znippy_common::ArrowIpcSinkAppend, that adds append/resume (open_existing → append blobs + rows → re-seal) and the first-class native blob-append (append_files, the compress --append primitive #23 noted was missing). The ORIGINAL ArrowIpcSink hot path is untouched (byte- and perf-identical = the A baseline). The clone's fresh-write path is a line-for-line copy, so it should be a zero-cost superset on the normal path.

Design (clone, not refactor). A new module znippy-common/src/meta_sink_append.rs holds ArrowIpcSinkAppend — field-for-field identical to ArrowIpcSink, same push_subindex + finish (same sub-index serialise, same sorted lookup, same fst trie, same manifest, same ZNPYMIDX footer). The only added surface: open_existing(path) (reads the full manifest incl. reserved sections, finds blob_end = lowest index_offset, recovers existing data rows from the sorted lookup section, positions the cursor at blob_end so new bytes overwrite the rebuildable tail), blob_end()/recovered_rows() accessors, and append_files() (native: codec-compress new files → append blobs past blob_end → re-seal merged old+new rows). finish() re-emits recovered rows as a data sub-index (so the ordinary reader still lists them) and set_len-truncates so no stale footer lingers. One minimal additive extraction in index.rs: read_znippy_full_manifest (public wrapper over the existing private read_full_manifest) — read-only, the original write path unchanged.

B1 — subindex_write (push one base-schema sub-index of N rows): A vs B

B2 — footer_seal (the headline seal: sort + 2nd materialise + trie + manifest + footer + fsync): A vs B

total_bytes is identical A=B at every scale (asserted byte-for-byte by the clone_fresh_path_is_byte_identical_to_original unit test): the clone produces the exact same v0.7 container. The Δ% on seal_ms is single-shot timing noise around 0 (M went the other way at L vs M); none exceeds the gate.

PARITY VERDICT — PASS ✅ (WORKSPACE.md ≤10% no-regression gate)

The append-capable clone is a zero-cost (≈noise) superset of the original on the normal, non-append write path:

• Output is byte-identical — same bytes, same length, A=B, all scales.
• Throughput within the gate — worst seal delta +2.0% (S), M +0.5%, L +1.3%; worst subindex delta +1.5% (L). All well under the ≤10% gate. No regression to report.
• The compress_mbs/decompress_mbs end-to-end path is completely unaffected — the original ArrowIpcSink it uses is untouched (the clone is opt-in, only reached via ArrowIpcSinkAppend::new/open_existing/append_files).

NEW capability MEASURED — resume_append (open existing → append 1 000 blobs → re-seal)

Resume cost is finally measured, not just functionally tested. reopen_ms is the open-existing cost (read manifest + recover the sorted lookup of N rows); append_ms is the full end-to-end (reopen + append 1k blobs + re-sort & re-seal the merged N+1k rows). The append re-seal is ≈ a full fresh seal of N+K rows (L: append_ms 1754ms vs full footer_seal_l 1730ms) because re-sealing rebuilds the sorted lookup + trie + manifest over the whole row set — expected, and the honest cost of an immutable, content-addressed sealed format. The append blob write itself is trivial (57 KB); the cost is the metadata re-seal, which scales with total rows exactly like B2. A future (A) incremental seal would lower both B2 and this in lockstep.

Benches: xtask/examples/nornir-bench.rs — SubindexWriteB/FooterSealB (*_b_{s,m,l}) + ResumeAppend (resume_append_{s,m,l}), heavy _l gated by NORNIR_BENCH_SKIP_HEAVY like A. Inject-assert tests: arrowipc_bench_tests::{ab_clone_write_seal_within_parity_gate, resume_append_recovers_and_reseals} + the structural parity/round-trip unit tests meta_sink_append::tests::{clone_fresh_path_is_byte_identical_to_original, native_append_roundtrips_old_and_new_files}. Row appended to bench_history.jsonl as the arrow-ipc-ab suite, literal date 2026-06-18. Run: ZNIPPY_BENCH_ONLY=subindex_write,footer_seal,resume_append NORNIR_MACHINE=threadripper-3975wx cargo run -p xtask --release --example nornir-bench.