nordisk/znippy-zoomies

znippy-zoomies

Single-purpose parallel Rust primitives built on one all-core engine, not rayon's fork-join model.

• gatling — multicore streaming engine; zero copy, allocation, I/O-wait, and CPU-pause.
• lgz / lbunzip2 — pure-Rust parallel gzip / bzip2 decoders on gatling.
• ldeflate — pure-Rust parallel DEFLATE/zlib compressor; gzip / ZIP / JAR on one planner.
• stree / stree32 — AVX2 static search trees, ~40 ns/query.
• vtd — parallel OSM-XML scanner → compact byte-offset index; bounded RAM.
• psort — parallel sample sort over 16-byte i64-keyed records.

Decompressor benchmarks

Each of our parallel decoders against the standard tools for its own format, grouped by format — every one of ours vs every one of theirs, in two modes. One deterministic 1 GB corpus, min-of-3. Columns are hand-picked (MB/s · cores · wall s · peak RSS). A 🏆 marks a decoder that beat a long-reigning record — see records.toml.

Ordinary gzip, one pass — real OCI layers, CLI (round 2)

The lgz CLI streams (stdin the same as files): one gatling ordered pipeline, read windows from gatling::window_plan. Produced by benches/r2_matrix.py (rendered by benches/r2_table.py). Output to /dev/null, to a file, and the time to the first output byte.

2026-11-02 · oden (Threadripper PRO 3975WX, 32 cores; shared: load 10–14 during the run) · 15 real Docker Hub layers (2398 MB out) · best of 3 · wall summed over the layers

Window plan (lgz CLI, all threads, to /dev/null)

Ordinary gzip — real OCI image layers

.tar.gz / OCI tar+gzip layers as Docker Hub serves them: ONE DEFLATE stream, no flush points. lgz splits it at real block starts found at the bit level (lgz::bitsplit). Produced by benches/oci_layers.sh (plan: .nornir/BENCH-PLAN.md); the layers are fetched by digest from benches/corpora/oci-layers.json, never committed. lgz decodes into memory; the C tools write to /dev/null.

2026-11-02 · dwarf (AMD, 12 threads, shared host: load 3–7 during the run) · 15 real Docker Hub layers (nginx, postgres, python, clickhouse; 2398 MB out) · best of 3 per layer · wall summed over the layers

The multi-core gzip rows below decode multi-member gzip (concatenated members), not one ordinary stream.

Multi core variants — all cores

gzip

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

bzip2

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

zip

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

jar

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

Single core variants — one core each

gzip

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

bzip2

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

zip

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

jar

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

cores_busy = avg CPU cores busy during decode (12 = full saturation). znippy-* run all cores; _st rows are pinned to one worker. 🏆 = beat a long-reigning record (records.toml).

Latest run — 32 cores, 2026-07-23: the harness refused to publish it

Not benched. This repo contributed no numbers to the latest run.

An entrypoint exists but produced no run — it was not part of the latest pass, or it exited without emitting a BenchRun.

The published tables above are untouched by this run, and that is deliberate: a preview may never overwrite a measured, published table. nornir docs render hides a preview run from every marker that has not opted in with preview=yes, so the blocks below are the only ones it can reach.

Nothing here failed. The decoders did exactly what they were asked and the roundtrip tests passed — but the OSM-XML input was 24 MB against the 1024 MB heavy floor, so the harness declined to emit a publishable BenchRun for the whole run rather than let an under-floor pass become a number somebody quotes. Read every row below as a preview: honest measurements of a workload too small to publish.

gzip — preview

v0.1.20 · PREVIEW · AMD Ryzen Threadripper PRO 3975WX 32-Cores · 499 GiB · corpus 1024 MB · osm-katana input 24 MB · min-of-3 · 32 cores · 2026-07-23

bzip2 — preview

v0.1.20 · PREVIEW · AMD Ryzen Threadripper PRO 3975WX 32-Cores · 499 GiB · corpus 1024 MB · osm-katana input 24 MB · min-of-3 · 32 cores · 2026-07-23

one core each — preview

v0.1.20 · PREVIEW · AMD Ryzen Threadripper PRO 3975WX 32-Cores · 499 GiB · corpus 1024 MB · osm-katana input 24 MB · min-of-3 · 32 cores · 2026-07-23

OSM convert — preview, and the arm that put the whole run under the floor

v0.1.20 · PREVIEW · AMD Ryzen Threadripper PRO 3975WX 32-Cores · 499 GiB · corpus 1024 MB · osm-katana input 24 MB · min-of-3 · 32 cores · 2026-07-23

No zip / jar rows: those arms did not run in this pass. The host string above is the harness's own — it composes machine from the CPU/RAM/corpus it measured and ignores NORNIR_MACHINE, so this run reads as its Threadripper rather than as oden. To promote any of this to a published table: NORNIR_ZOOMIES_CORPUS_MB=1024 and an osm-katana input ≥ 1024 MB (or NORNIR_OSM_KATANA_CLIP=<region> against a planet file).

Compressor benchmarks

Everything above is decode. ldeflate (published 2026-08-03) is the workspace's one write-side crate — the parallel DEFLATE/zlib engine plus the gzip, ZIP and JAR deflators built on it. It gets its own section rather than a column above, because a compressor cannot be raced the way a decoder can: the bake-off reconstructs one fixed archive, while gzip -6 and this crate emit different bytes, so MB/s alone ranks nothing. The contested number is what the schedule costs against the same crate's serial path over the same objects at the same level — so the baseline row is compress_objects_serial, not gzip.

plan::compress_objects is the only scheduler in the crate. gzip, ZIP and JAR route through it and differ only in framing — Container::wrap puts a header and trailer around a body the planner already produced, and the three containers added no scheduling code at all. The planner picks a unit kind per object by size: Unit::Batch below coalesce_below = 64 KiB (one task drains many, reusing one encoder), Unit::Whole, or Unit::Segment above 4 MiB (cut at Z_FULL_FLUSH, so one big blob cannot serialise a worker). Unit::Batch covers 93.8% of real git objects — measured over 48 917 of them — which is why one Config serves both git and LFS. limits.rs refuses an amplifying request past MAX_PIECES = 262 144 rather than attempting it.

Scaling — the same crate against its own serial path

oden · AMD Ryzen Threadripper PRO 3975WX 32-Cores · 499 GiB · 30 770 real git objects, 715.4 MB, median 986 B · level 6 · min-of-3 · load 40–47 · 2026-08-04

Read the last column. Above 8 threads the speedup_vs_serial column measures the box, not the code — oden is shared and this crate takes every core it can get, so a run that asks for 32 threads and is given 12.15 cores returns 11.63×. Across a 7-run batch on 2026-08-03 the 32-thread speedup spanned 10.38×–23.20× (123.5% spread) while the CPU cost of the schedule held to 1.025×–1.054× of serial, and efficiency against cores actually held stayed in a 94.9–97.6% band at every occupancy. An earlier headline of 28.30× / 88% / 2266 MB/s is not refuted — carried forward it predicts 29.2–31.2× at that occupancy — but it was a statement about machine occupancy dressed as one about the crate, it cannot be reproduced on a shared box, and it is retired as a crate claim. Full tables, the negative controls and both falsified hypotheses: benchmarks.md.

Never compress what is already compressed

precompressed.rs gates the entropy coder on hint → extension → magic-byte probe — no entropy estimate, no tunable threshold, at most 16 byte compares. Acting on a store verdict is free because stored_deflate emits RFC 1951 §3.2.4 stored blocks: a length-prefixed memcpy, not flate2 at level 0, which would still build, reset and walk the encoder.

oden · 24 real git packfiles + pack indexes · 113.8 MiB · level 6 · min-of-3 · decided on CPU seconds · load 66 · 2026-08-04

→ 23.1× less CPU, 95.7% saved, output +0.41% larger.

Decided on CPU seconds, not wall time: at load 66 the wall column is noise (the same arm read 267.5 / 2075.5 MB/s on a quieter box) while cpu-s per GiB moved under 2%. Quote the saving, not the multiplier — across four runs on 2026-08-03/04 the multiplier wandered 19.4×–23.1× because its denominator is ~1 cpu-s/GiB, while the saving held at 94.8–95.7% and the size penalty came out byte-identical (+0.41%) every time. It independently reproduces znippy's own 92–95% figure. Negative controls, same harness: with the probe given nothing to find (git loose objects) it collapses to 1.03×, and with the probe vetoed by ContentHint::Compressible over the same packfiles to 1.04× at byte-identical output — so the 23× is a property of the probe.

OSM converter

osm-katana (in-repo sub-crate) converts OSM-XML/PBF → GeoParquet all-core on a no-barrier gatling pipeline, head-to-head with the legacy reference tools osmium / osmconvert for the same input. One convert pass, min-of-3. Columns: MB/s, cores busy, wall seconds, peak RSS (MB).

v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13

convert_mbs = input MB / wall; cores_busy = average CPU cores busy (12 = full saturation on this box). osm-katana runs all cores; osmium / osmconvert are the single-stream C++ references, benched only when installed. By default a small synthetic OSM-XML corpus exercises the emit path on any box — the HEADLINE number comes from the off-box planet run (NORNIR_OSM_KATANA_INPUT=<planet.osm> nornir bench run znippy-zoomies). Region clip headline (cut Sweden out of the planet with our tool): add NORNIR_OSM_KATANA_CLIP=10.9,55.3,24.2,69.1.

Staged pipeline

The classic planet chain — each stage eats the previous stage's output file: xml2-pbf (OSM XML/bz2 → PBF) → pbf2-geo (PBF → GeoParquet) → geo2-arrow (GeoParquet → Arrow IPC). NORNIR_OSM_KATANA_CHAIN=1 nornir bench run znippy-zoomies warehouses one row per stage (stage_mbs = that stage's input MB / wall); point NORNIR_OSM_KATANA_INPUT at a planet for the heavy chain. Off by default — it adds a full three-stage pass — so the table lands after that run.

Docs

• Manual · Idiot's guide

License

MIT. The stree module retains Ragnar Groot Koerkamp's copyright notice (see LICENSE).