nordisk/znippy-zoomies

znippy-zoomies — benchmarks (the crushing numbers)

Class: benchmarks. The measured head-to-head numbers, HEAVY runs only (no small/smoke mode — see reasoning.md). Host CPU, core count and peak memory are recorded per rival. How to reproduce: guide.md and decompressors-guide.md.

f0 — the decompressor bake-off (the star)

One large deterministic corpus is compressed into every rival's format, then decompressed back by every decompressor on the box — znippy-zoomies' parallel lgz / lbunzip2 against the legacy serial tools and the other parallel C decoders, all at full cores + max memory. The harness warmup-primes each decoder and keeps the min of N≥3 timed passes, so re-runs reproduce rather than chase single-shot noise. Each system is one row; the contested metrics are the columns. Bold = the winning cell for that metric. The table is regenerated from the latest warehouse BenchRun by the marker below.

The honest scope (read this before the numbers). znippy-zoomies is the fastest decoder within the two legacy formats it targets — not the fastest decoder in the field. Concretely, on this corpus:

• gzip: znippy-gz beats serial gzip and parallel pigz per core — a clean win within the format, though gzip decode is Amdahl-/bandwidth-bound and does not scale to all 12 cores (see the core-scaling verdict below).
• bzip2: znippy-bz2 leads the bz2 field measured here — it beats serial bzip2/pbzip2 by ~13× per core and trades the lead with the mature C lbzip2 (808 vs 764 MB/s in the v0.1.15 min-of-3 render; a later same-host run had it a photo-finish the other way — see the Reproducibility caveat below). Against the stock serial/parallel bzip2 tools the win is decisive and repeatable; against C lbzip2 it is a genuine same-format tie, not a decisive lead.
• modern codecs: zstd, lz4, xz/pixz, plzip decode their own formats 1.5–2.6× faster than our lgz. Those formats were designed for fast decode; gzip and bzip2 were not. Our value is multi-core decode of the formats you already have.

Grouping rule (the apples-to-apples fix). Every table below pairs a decoder only with rivals in its own format and its own core-mode: single-core tools sit with single-core tools, all-core tools with all-core tools. A 12-core decompress_mbs is never set against a 1-core rival and called a win — that apples-to-pears framing was a real defect and it is gone. Cross-format and cross-mode reading is led by percore_mbs (throughput per host core), the one metric that stays honest when the core counts differ.

v0.1.15 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB (multi-member gzip = 128 × 8 MB independent members; bzip2 -9) · min-of-3 · 12 cores · 2026-07-10

🍎 DEFLATE family — znippy-gz vs gzip vs pigz

Single-core — serial gzip and the pinned znippy-gz_st (one gatling worker):

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

Multi-core — parallel pigz and full-core znippy-gz:

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

The znippy-gz / znippy-gz_st cells are held for the next bench-machine wave (the local warehouse carries no lgz BenchRun; a render on the bench box fills them from the marker params above). The core-scaling probe measured znippy-gz at ~2.7 / 12 cores decoding this corpus to /dev/null (lgz corpus.gz > /dev/null, the serial-stdout path) and ~7.8 / 12 cores on the same input written with -o file (the positional-pwrite path), at ~75 MB/s/core — ahead of both gzip (60.35/core) and pigz (51.30/core). See the core-scaling verdict below for why gzip decode is bandwidth- rather than core-bound.

🍐 bzip2 family — znippy-bz2 vs bzip2 vs lbzip2 vs pbzip2

Single-core — serial bzip2 and the pinned znippy-bz2_st:

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

Multi-core — parallel lbzip2, pbzip2, and full-core znippy-bz2:

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

Reproducibility caveat — the bz2 lead over C lbzip2 is real but close and occupancy-sensitive (read this before quoting "808 vs 764"). The 808.39 / 764.48 cells above are the v0.1.15 min-of-3 render, normalised to a flat 12 nominal cores (percore_mbs = decompress_mbs / 12 exactly — a nominal core count, not a measured cores_busy). A later heavy bake-off on the same Loki host and the same corpus measured the head-to-head closer, with the sign flipped: znippy-bz2 695 MB/s @ 11.41 cores_busy vs lbzip2 768 MB/s @ 11.81 cores_busy. Nothing in the decode path changed between the two runs — the lbzip2 / gatling decode source is byte-identical since the commit that produced these numbers (33ff958, "beat system lbzip2 on multicore decode"; the only later commits are this docs regroup and an optional, off-by-default nornir-testmatrix feature-dep repoint, neither touching the hot path). The delta is therefore not a code regression but run-to-run variance: bzip2 multi-core decode is L3-latency-bound (diagnosed in 33ff958), so throughput tracks measured core occupancy, and the two runs simply disagree on occupancy (a flat nominal 12 vs a contended 11.41 / 11.81). On a fully idle box the committed min-of-3 has znippy-bz2 ahead; on a loaded box the more core-hungry scaler gives up more throughput than the mature C tool and can trail it. Honest reading: znippy-bz2 and C lbzip2 are within run-to-run variance on this corpus — a genuine same-format photo-finish, not a decisive lead. When quoting the bz2 result, cite the per-core figure and the measured cores_busy, not a bare "808 vs 764".

Columns — decompress_mbs: decode throughput (MB/s) · percore_mbs: decode throughput per host core · decompress_s: decode wall time · peak_mem_mb: peak resident set. znippy-gz / znippy-bz2 are ours; every other row is a stock rival. The tables scope to decode metrics — the only ones that differ between decoders of the same format. Local render note: the warehouse on this box has no BenchRun, so these regions are hand-filled with the numbers the bench-machine committed; nornir docs render znippy-zoomies on the bench box reproduces them from the marker params.

Format properties (identical within a format family — set by the reference compressor, not the decoder): gzip-family compress_mbs ≈ 256 MB/s at ratio 4.64; bzip2-family ≈ 23 MB/s at ratio 7.50. They describe the archive, not the decode race above.

How to read it — the wins. The two formats parallelise very differently, and we say so plainly. znippy-bz2 is the genuine all-cores scaler and trades the bz2 lead with C lbzip2: bzip2's block decode is CPU-heavy, so the parallel path reaches ~8.5× at 12 cores (~71 % efficiency, ~8 cores kept busy) — it decodes bzip2 -9 at 67.37 MB/s/core vs bzip2's 5.02 and pbzip2's 5.04, a ~13.4× lead over the stock serial/parallel bzip2 tools, and at 808 MB/s it edges even the mature C lbzip2 (764 MB/s, 63.71/core) in the v0.1.15 min-of-3 render — the single biggest, same-format gap over the stock tools in the run. Against C lbzip2 specifically the result is a run-to-run photo-finish (a later same-host run had lbzip2 ahead 768 vs 695 at 11.81 vs 11.41 measured cores) — see the Reproducibility caveat above; the decode path did not change between the runs. znippy-gz is a more modest parallel win, and Amdahl-/bandwidth- bound by design: gzip's DEFLATE decode is already so fast that little compute remains to spread across cores (see the verdict below). It still lands the fastest gzip decode measured at ~75 MB/s/core vs gzip's 60.35 and pigz's 51.30, but that is a format-limited speedup, not an all-cores one.

Core-scaling verdict — why lgz uses ~2.7 cores here, and whether it should use all 12. The honest answer is both: real, already-built headroom the bench under-shows and a hard format ceiling.

• Not a single-stream stall on this corpus. The gzip corpus is 128 independent 8 MB members (split -b 8M … | gzip), so it is embarrassingly parallel — the classic "one LZ77 32 KB window serialises everything" argument does not apply here. lgz already exploits every parallel escape hatch in code: parallel multi-member scan (speculative::find_gzip_members_parallel), Z_FULL_FLUSH boundary split (deflate_scan + chunk), and a trailer-verified speculative block-boundary guess (speculative::speculative_decode) for streams with none. It is not being lazy.
• The 2.7 figure is the > /dev/null (stdout) path. The bench runs lgz corpus.gz > /dev/null, which forces the single-writer ordered-sink path (stdout cannot seek). lgz's own -o file path decodes the same input at ~7.8 / 12 cores via positional pwrite (each worker writes its member at a prefix-sum offset, no serial collector). So on this corpus the realistic ceiling is ~7.8 cores, not 2.7 — the gap is a measurement artifact of piping to /dev/null, and it is fixable for free by benching the -o deployment shape.
• ~7.8, not 12, is the inherent gzip ceiling. gzip decode is memory-bandwidth- bound: serial gzip already reconstructs the whole 1 GB corpus in 1.41 s (724 MB/s), so there is only ~1.4 core-seconds of inflate work, most of it DRAM-write-bound LZ77 copies that saturate the memory controller at a handful of cores no matter the thread count. This is exactly why znippy-bz2 does reach 11.7/12 (bzip2 blocks are compute-heavy) while znippy-gz cannot — the difference is per-byte compute intensity, not effort.
• Context — lgz already leads the field. pigz, the reference parallel gzip, reaches only 1.27 / 12 cores on this same corpus (its decoder is essentially serial). lgz at 2.7 (stdout) already ~2.1× that, and at 7.8 (-o) ~6× — no gzip decoder parallelises better, and a plain single-member gzip foo stream is serial-by-format for everyone, pigz included.

Verdict: Rickard is right that the bench leaves cores idle it need not — surfacing the -o path lifts lgz from ~2.7 to ~7.8/12 with zero code change, so "use more cores" is a legitimate, actionable ask for the measurement. But "all 12 cores" is not physically reachable for gzip decode: it is bandwidth-bound, unlike bzip2's compute-bound blocks that genuinely do hit 11.7/12. lgz already implements both parallel paths and already out-scales every gzip tool measured.

How to read it — the losses (stated plainly, not buried). Two things these tables do not show, and we won't pretend otherwise:

1. Modern codecs decode faster than our lgz, full stop — lz4, pixz, xz, zstd and plzip all reconstruct their own formats 1.5–2.6× faster. They were engineered for fast decode; gzip and bzip2 were not.
2. lgz is bandwidth-bound, not all-core — as the verdict above states, gzip decode tops out around ~7.8/12 cores even on the ideal multi-member input; only the bzip2 family (znippy-bz2) is a true all-cores scaler here.

So the defensible claim is narrow and true: multi-core decode for the ubiquitous gzip/bzip2 formats every tool already reads, on data you already hold and can't re-encode — where the stock tools leave the box idle. For bzip2 that is a decisive win over the stock serial/parallel tools and a same-format tie with the mature C lbzip2 (a run-to-run photo-finish — see the Reproducibility caveat); for gzip it is the fastest per-core decode in a format that simply cannot fill 12 cores. It is not "fastest decoder overall", and the numbers above are the reason we don't say that.

Memory is recorded per rival — the peak_mem_mb column is the GNU time -v peak RSS of each decode. Ours buys its throughput with a larger working set (the ring of in-flight slots: znippy-gz 906.8 MB, znippy-bz2 655.2 MB) — a deliberate space-for-cores trade on a 61 GiB host, called out honestly rather than hidden.

f1 — the compressor bake-off (ldeflate, the write side)

Everything above this line is decode. ldeflate is the workspace's one write-side crate — the parallel DEFLATE/zlib engine, plus the gzip, ZIP and JAR deflators built on it — and it is benched on a different question, so it gets its own section rather than a column in f0.

Why there is no rival column. f0 races every decoder on the box to reconstruct one given archive, which is a fair fight because the input is fixed. A compressor's output is not fixed: gzip -6 and this crate produce different bytes, so "MB/s" alone ranks nothing. The contested number for a compressor is what its schedule costs against the same crate's serial path over the same objects at the same level — which is why the baseline row here is compress_objects_serial, not gzip. ldeflate 0.1.0 is published on crates.io (2026-08-03).

One engine, four containers. plan::compress_objects is the only scheduler in the crate. gzip, ZIP and JAR all route through it and differ only in framing — Container::wrap puts a header and a trailer around a body the planner already produced. Total new scheduling code across the three containers: none (LAW 5). The planner picks one of three unit kinds per object, by size:

93.8% of real git objects take the first path — measured over 48 917 objects from eight repositories on this box, not assumed. That is why coalescing is the axis that matters for git and splitting is the axis that matters for LFS, and why one Config serves both: the classification is per object, not per call.

limits.rs is the other half of that story, and it is a scar. Both segment and member are amplifying parameters — the caller passes two small integers and the crate allocates a structure proportional to their quotient. On 2026-08-03 a test crossing len = 3_000_000 with member = 1 (three million one-byte gzip members) reached ~497 GB anon RSS on a 499 GB box; the OOM killer fired and took a shared tmux service and three unrelated sessions down with it. MAX_PIECES = 262 144 now refuses such a request, naming what was exceeded. An Err, never a clamp: quietly raising piece until the count fits would change the segment plan, therefore the output bytes, therefore the determinism guarantee — without the caller ever learning. Every test and bench in the crate now runs under (ulimit -v 25165824).

The honest framing — read this before the scaling table

Above 8 threads the speedup column measures the box, not the code. oden is shared, and ldeflate takes every core it can get: cores actually held tracked 32 − other tenants almost exactly across a seven-run batch. So the same binary over the same corpus produced 32-thread speedups from 10.38× to 23.20× — a 123.5% spread — while the CPU cost of the schedule held to 1.025×–1.054× of serial. The first column is a statement about machine occupancy; the last is a statement about this crate.

An earlier headline — 28.30× at 32 threads, 88% efficiency, 2266 MB/s — is not refuted: it is consistent with, and slightly conservative against, the re-runs, which carried forward predict 29.2–31.2× at that occupancy. But it was presented as a property of the crate when it is a property of that run's occupancy, and it cannot be reproduced on a shared box. It is quoted here only as retired framing. The durable claim is the CPU one.

Scaling — re-measured 2026-08-04

oden · AMD Ryzen Threadripper PRO 3975WX · 32 physical cores · SMT off · 499 GB RAM · 30 770 REAL git objects, 715.4 MB, median 986 B, 93.7% under 64 KiB · level 6 · min-of-3 · load 40–47 during the run · 2026-08-04

Read the last column, not the second-to-last. This run asked for 32 threads and got 12.15 cores — other tenants held the rest — so 11.63× is what 12.15 cores buys, and the crate converted them at 0.96–1.03 speedup per core held at every width from 1 to 32. The 32-thread row is inside the carried 10.38–23.20× band and its efficiency against cores actually held (95.7%) is inside the carried 94.9–97.6% band. Two independent sessions, two different corpora, same conclusion.

Carried from 2026-08-03 (.nornir/ldeflate-design.md Finding 4b — not re-measured here): 49 497 objects / 1684.7 MiB, n = 7 independent runs of min-of-5, each under sudo -n systemd-run --scope -p MemoryMax=32G and run strictly serially.

Speedups at 1–8 threads reproduce to ≤2.5% only because ≤8 cores were always free; the spread grows with thread count exactly as the request starts to exceed what was available. The one column that reproduces at every width is the CPU one.

precompressed.rs — never compress what is already compressed

The largest avoidable cost on the write side is paying an entropy coder to hand back its own input. A git object store is full of packfiles; a JAR is full of PNGs. The gate is a hint → extension → magic-byte probe, in that order, and it is deliberately not an entropy estimate: an estimate is a heuristic with a tunable threshold, costs a scan proportional to the data, and misjudges in both directions. Every probe here is exact — a fixed byte pattern at a fixed offset, at most SNIFF_PREFIX_LEN = 16 byte compares, no threshold, no sample, no allocation. The hint is honoured in both directions, so a caller who knows better than the magic bytes wins either way.

Acting on the verdict is free because of stored_deflate — an RFC 1951 §3.2.4 stored-block emitter: a 5-byte block header per 64 KiB and a memcpy. It is not flate2 at level 0, which still builds and resets a compressor and still walks the input through the encoder state machine. That distinction is where the CPU saving actually comes from.

oden · 24 REAL git packfiles + pack indexes · 113.8 MiB · level 6 · min-of-3 · decided on CPU seconds · load 66 · 2026-08-04 (re-measured)

→ 23.1× less CPU, 95.7% saved, output +0.41% larger.

Decided on CPU seconds, not wall time, because the box was at load 66 and CPU-seconds consumed by our own process is far less sensitive to other tenants than the wall clock is. The wall MB/s column is indicative only — the carried 2026-08-03 run of this identical arm read 267.5 / 2075.5 MB/s on a quieter box while its cpu-s per GiB column read 22.40 / 0.99, i.e. the load-robust metric moved by under 2% and the wall column moved by 3×. That is the whole argument for the choice of metric.

Read the ratio, not the multiplier. Across four runs on 2026-08-03/04 the CPU multiplier wandered 19.4× – 23.1× and the saving held at 94.8% – 95.7%. The multiplier is unstable because its denominator is ~1 cpu-s/GiB — a small absolute number that a shared box perturbs easily. The stable claim is ~95% of the CPU saved for +0.41% of size, and the size penalty came out byte-identical (+0.41%) on every single run. It independently reproduces znippy's own 92–95% figure from a different codebase.

The negative controls, because a 23× that never collapses is not evidence about a probe. Same harness, same two arms, 2026-08-04:

Red 1 collapses because loose objects are commits, trees and source text, so looks_compressed returns false and both arms do identical work — the residual 1.03× and +0.11% is a handful of genuinely-compressed blobs in those repos, which is the honest answer rather than a leak. Red 2 is the tighter control: identical bytes, only the policy differs, and it collapses to 1.04× with the output byte-identical to the compress-everything arm. The 23× is a property of the probe, not of the two arms differing in some other way.

What the compressor costs in ratio

Carried from 2026-08-03 except the last row. Determinism is what makes the first two rows sayable at all: the work plan is a pure function of (object sizes, Config) and never reads the core count, so output is byte-identical at 1 worker and at 32 — parallel_equals_serial_at_every_worker_count asserts exactly that, and segment_size_actually_changes_the_bytes proves the assertion is not vacuous by showing a different config does move the bytes.

The LFS answer is not "parallelise harder"

Carried from 2026-08-03. Level-6 DEFLATE over already-compressed content achieves a ratio of 1.000 — literally nothing — while running 42× slower than a hash over the same bytes. Splitting makes it 11× faster at achieving nothing. Re-measured on 2026-08-04 the deflate.lfs_incompressible arm read 357.5 MB/s at ratio 1.000, which is the same result: the number to act on is the ratio, and the action is to route that content to precompressed's store path before a compressor sees it. Making waste cheaper is not making progress, and someone will otherwise bench this crate on LFS-shaped data, find it does nothing, and conclude the design failed when it is working exactly as intended.

Reproduce

# the publishable arms (nornir bencher — never criterion). These rows warehouse in
# the SAME BenchRun as the decompressor bake-off above.
NORNIR_BENCH_DEFLATE_ONLY=1 cargo run --release --example nornir-bench

# the tuning loop, including the precompressed arm (which does NOT warehouse)
CARGO_TARGET_DIR=/home/rickard/scratch/cargo/zoomies-deflate \
  cargo run --offline --release -p ldeflate --example measure -- precompressed

NORNIR_BENCH_DEFLATE_ONLY=1 skips the heavy bake-off and emits no BenchRun contract — the numbers go to stderr, so a compressor fix can be measured in a minute without a preview run reaching a published table. The scaling arm carries its own gate: check_deflate_scaling_real fails the run below 4× speedup or 4 cores held, and deflate_scaling_gate_trips_on_serial feeds it a serial implementation's numbers (speedup 1.0, 1 core) and asserts it trips — verified red on 2026-08-04 by dropping both floors to 0.0, which made the test fail with its own message, then reverted and re-confirmed green. Whole suite: 86 tests, green serially under (ulimit -v 25165824), 2026-08-04.

osm-katana — read the regime before the numbers

osm-katana (OSM-XML/PBF → GeoParquet, all-core on the gatling pipeline) has two very different benchmark regimes, and only one of them is a throughput measurement.

Smoke run (small synthetic corpus) — NOT a scaling result. The default bake-off runs a tiny synthetic OSM-XML corpus that exists to exercise the emit path on any box. It finishes in ~0.2 s, so the numbers are dominated by process start-up, thread-pool spawn and file open — fixed overhead that min-of-3 cannot remove. Every tool shows ~1 core busy because the parallel path never engages on ~24 MB; a lean serial C tool naturally "wins" a micro-input where thread-spawn cost exceeds the work. Do not read 12-core scaling — or any throughput ranking — from this table.

Representative run (real all-core face) — the region-clip planet cut. osm-katana's actual scaling shows only at planet scale, where there is enough work to fill the box. The committed warehouse BenchRun (Sweden cut out of the full planet with our own tool, resolved geometry: 19.35 M ways + 14.5 M relations) is:

That is the honest ~11/12-core figure (perf(osm-katana): parallelize … ~3→12 cores, f6dc01b; run committed in 6518559). There is no head-to-head at this scale: osmium / osmconvert have no planet→clipped-GeoParquet equivalent, so the region-clip row is osm-katana-only and we do not manufacture rival numbers for it. The multi-hour planet run is not reproduced on this box (it is a heavy run); these are the committed warehouse numbers. A lighter but still representative all-core figure would need a region-sized corpus that runs several seconds all-core — a bench-machine job, not a laptop one.

osm-katana — per-phase core occupancy

Throughput alone cannot say why the box sits idle, so the bench now reports the converter's own per-phase occupancy (--log JSONL → one bench row per phase). The numbers below are PREVIEW — a 319 MB PBF, not a publishable heavy run — and the harness enforces that: a light run never emits a BenchRun, so these can never reach the warehouse or a README table. They are here because the shape is the finding.

oden · AMD Ryzen Threadripper PRO 3975WX · 32 physical cores · SMT off · 499 GB RAM · input 319 MB PBF · min-of-3 · 2026-07-22

How to read it:

• convert sits below pass1 and pass2 ⇒ core-time is lost between the bursts, not inside them. That is the pass1 → sort_and_stree → pass2 barrier.
• sort_and_stree at 4.74 / 32 is the barrier itself: pass 1 must be fully drained before the tail coord chunk can be sorted, and pass 2 cannot start until the STree exists.
• pass1's collector_pct 55.3 with busy_pct 51.1 ⇒ collector-bound. The single in-order sink copies every packed coord buffer into the sorter and writes the whole nodes.parquet — 565 MB + 533 MB on one thread for this input — while 31 workers wait behind it.

What that measurement already bought

The first thing the per-phase view exposed was not in the pipeline at all: every phase was reporting a wall time that was an exact multiple of 100 ms. Phase::done() joins the phase's sampler thread before it reads the clock, and the sampler only noticed the stop flag between 100 ms sleeps — so each of the four phase boundaries charged up to a full poll interval of dead time with the entire worker pool already idle. Replacing the polled flag with a condvar:

Contention caveat. oden was carrying other agents' workspace builds for most of this session (load average swinging 18 → 159), so the 319 MB figures are quoted two ways: the bench harness's min-of-3 from a quiet window, and a stricter old-binary/new-binary interleaved A/B (11 pairs, round-robin so both binaries see the same turbo/contention windows) whose best pair is the −6.9 % above. The new binary won 9 of 11 pairs. The 3.3 MB figure needs no caveat — it repeated to the millisecond across 5/5 runs, because on a small input the four 100 ms boundary stalls were the runtime.

Output unchanged across the change — osm-katana verify --digest gave the same multiset digest for all three tables (nodes 19b60a8f97bfc25e / 35316700 rows, ways 04bb35929ff74b2d / 4170200, relations 00eff866fcbfa1f7 / 93800).

A negative result worth keeping

CoordFileWriter::write_raw — the pass-1 collector's coord path — walks chunks_exact(16) rebuilding each [u8; 16] element by element, which reads like an obvious 565 MB-per-run win over a bulk memcpy. It is not. Isolated at the real shape (35316700 records over 370 segments, 3 passes):

0.5 % — the TrustedLen specialisation already compiles the extend to the same thing, and both sit at memory bandwidth. The unsafe version was written, measured and reverted: not faster, so it does not ship. The pass-1 collector's real cost is the 533 MB nodes.parquet stitch, not the coord copy.

The disguised-pool sweep (2026-07-22) — what each conversion actually cost

ROOT LAW #0 names std::thread::scope + s.spawn as the disguise: a hand-rolled scoped work pool is a rayon replacement and is forbidden just as hard, because it dodges a thread::spawn grep while being the identical violation. Twelve such pools were left in the tree; this pass converted them all except the root crate's psort/xml/vtd. Every conversion is quoted honestly — a law conversion lands even at parity, but parity is what it is called.

All numbers on oden (AMD Ryzen Threadripper PRO 3975WX, 32 physical cores, SMT off, 499 GB), interleaved old/new A/B so both binaries see the same contention windows. oden was shared with other agents all session; single before/after numbers are not quoted anywhere below.

geo2arrow was also a memory bug. The old code claimed its bounded channel kept memory in check. It did not: row groups were carved into contiguous per-worker partitions while the writer emits in strict global order, so nothing worker 1..N−1 produced could be written until worker 0 finished its whole partition — and the writer kept draining the channel into an unbounded pending HashMap to unblock the decoders. run_ordered_sink's permit pool cannot do that: a permit is released only after an in-order emit. 104 M rows / 214 row groups → 6.03 GB Arrow IPC, md5 identical from both binaries.

The search index was silently non-reproducible. gatling_features concatenated each worker thread's private Vec in handle order, so the dense feature ids — and every posting list, and the whole .idx — depended on which thread won which row group. Three runs of the old binary on liechtenstein gave three different files; three runs of the new one are byte-identical. Same 61 082 features, same 18 558 tokens, every posting list equal under the id relabelling. (A second hole in the same file: postings is a HashMap and was serialised in HashMap iteration order — the same class of bug as the serialize_tags HashMap. It is key-sorted now, via gatling_sort_unstable_by.)

A second negative result worth keeping — the stree band. The obvious gatling unit for the per-layer block fill is ONE BLOCK, and that is what was written first. It is 2× slower: a block is 64 bytes of fill and the widest layer of a 100 M-key tree is 12.5 M blocks, so the shared fetch_add and its cache line become the entire cost.

The lesson generalises: gatling's self-dispatch is not free, and the right unit is the one where the work per fetch_add dwarfs the fetch_add. Finer is better only until it is not — measure it.

What did NOT move. convert itself is untouched by this pass (its phases were already on gatling), and the interleaved liechtenstein A/B says so: pass1 0.080 → 0.078 s, pass2 0.107 → 0.104 s, convert 0.197 → 0.197 s. The one visible change is sort_and_stree occupancy, 4.43 → 5.48 of 32 cores at the same 0.009 s wall — the stree band conversion showing up. Output gate: verify --digest multiset digests unchanged (nodes 16a8386cf4b7ad76 / ways 02ed64602fadfe31 / relations 119bffec8cc479f0) and node_coords.arrow compares equal under cmp.

Left standing, deliberately. xml_to_pbf.rs's blob_writer_loop is ONE named, long-lived thread doing blocking file I/O off the gatling collector, fed by a bounded mpsc channel. That is IO-offload, not CPU fan-out — the law's own carve-out — and it is commented as such at the call site.

Reproduce (PREVIEW lens)

NORNIR_ZOOMIES_CORPUS_MB=0 NORNIR_ZOOMIES_WORK=/tmp/zb \
NORNIR_OSM_KATANA_INPUT=~/work/liechtenstein-latest.osm.pbf \
  cargo run --release --example nornir-bench

NORNIR_ZOOMIES_CORPUS_MB=0 means "no decompressor corpus" — the decoder roster is empty and only the osm-katana arm runs, so a fix can be measured in seconds instead of minutes. Same code path, same rows, same metric keys as the heavy run. See osm-katana-guide.md.

static search tree — stree32 vs binary_search

Measured with -C target-cpu=native, the in-test micro-bench, 1M hit queries (src/stree32.rs, stree32_bench / stree32_bench_ram). The win tracks the cache→RAM cliff and rises toward Ragnar's headline ~40× at 1 GB:

The u32 density win flips on once RAM-bound: in cache both widths are ~equal, but at 512 MB the i64 tree's extra level costs a real cache miss, so STree32 is 1.43× faster than STree64 over the same keys — the whole reason to restore the 32-bit width. Reproduce:

cargo test -p znippy-zoomies --release stree32_bench -- --nocapture           # in-cache sweep
cargo test -p znippy-zoomies --release stree32_bench_ram -- --ignored --nocapture   # 512 MB RAM-bound

engine + scanners — the criterion benches

Three criterion benches ship in benches/ (harness = false, Cargo.toml:58): gatling_roundtrip (the no-barrier streaming engine, split → N decode → in-order collect → sink), xml_scan (VTD-style OSM-XML scan), and json_scan (zero-copy NDJSON scan). All three run all-cores by design; the operational contract is flat 100 % core utilisation (the "all cores busy" tell in guide.md).

cargo bench -p znippy-zoomies --bench gatling_roundtrip
cargo bench -p znippy-zoomies --bench xml_scan
cargo bench -p znippy-zoomies --bench json_scan

The json_scan bench also pins the no-rayon port: after moving the NDJSON line-scan off rayon::into_par_iter onto gatling_forkjoin::gatling_for_each, the ported scan is if anything faster than the rayon version it replaced (no global-pool contention).