nordisk/znippy-zoomies

znippy-zoomies — guide (f0, the whole)

Class: guide · audience: a techie building on the crate or running the benches. The what is in design.md; the why in reasoning.md; the crushing numbers in benchmarks.md.

f0 — a full session

# 1. build the crate + the codec CLIs at full optimisation
cargo build --release                       # the library + workspace members
cargo build --release -p lgz -p lbzip2      # the two bake-off decoders

# 2. prove the primitives in-repo (fast, deterministic)
cargo test --release stree32_bench -- --nocapture   # AVX2 tree vs binary_search
cargo test --release --test rayon_free_law          # the no-rayon law holds

# 3. run the HEAVY decompressor bake-off (writes one warehouse BenchRun row)
nornir bench run znippy-zoomies                      # all cores, max memory

# 4. render the numbers into the docs
nornir docs book znippy-zoomies                      # → docs/manual.pdf (+ the other two)

Rule zero for anything you add: no rayon, and the ends never cook — reader reads, collector stitches in order, all heavy work in the workers.

Running the HEAVY bake-off

The bake-off lives in examples/nornir-bench.rs. nornir bench run znippy-zoomies subprocess-spawns it (cargo run --release --example nornir-bench), reads the last stdout line as one nornir::bench::BenchRun JSON object, and appends it to the warehouse bench_runs table. It is HEAVY by law — a full corpus, all cores, max memory; there is no small or smoke mode.

Env knobs (examples/nornir-bench.rs):

What it does, once: build a deterministic corpus, compress it into every rival's format at full cores (bzip2 -9, zstd -12 -T{n}, lz4 -9, xz -6 -T{n}, brotli -q5, plzip -6 -n{n}), then race every decompressor on the box to reconstruct it — ours (znippy-gz, znippy-bz2) against the legacy serial tools and the other parallel C decoders (pigz, lbzip2, pbzip2, pixz). Rivals not installed are skipped and named. Each system is one row; the contested metrics are the columns (decompress_mbs, percore_mbs, compress_mbs, ratio, decompress_s, peak_mem_mb). Host CPU, core count and memory are stamped on the header line.

Reading the result

nornir docs render znippy-zoomies (or the docs book above) fills the <!-- nornir:gen:start:mashup … --> markers in README.md and benchmarks.md from that BenchRun: one row per system, the winning cell per metric bolded, and a one-line biggest-gap headline. Direction is inferred from the unit (_mbs higher-better, _s/_mem_mb lower-better).

Using each function

Look & feel

Every parallel surface reads the same way in htop/btop: all cores busy = you did it right; one core pegged, the rest asleep = you made an end (reader/collector) cook, or reached for a serial loop. That single tell is the whole operational contract.