nordisk/znippy

Benchmarking: why nornir bench, not cargo bench

znippy has exactly one benchmark suite, and it does not run through cargo's bench harness. This chapter records why, and how the single source of truth is wired, so nobody re-adds a parallel criterion or #[bench] path.

Has nornir forked its own way? Yes — deliberately.

nornir does not use cargo's #[bench] / cargo bench / criterion harness at all. From nornir/src/bench/api.rs, the contract is tiny:

• Each bench is a small struct implementing nornir's Bencher trait, registered with register_bench! / register_bench_ordered!. The inventory crate collects them at link time — there is no hand-maintained registry.
• examples/nornir-bench.rs::main() is literally nornir::bench::api::run_main_json(). It runs every registered bencher and prints one BenchRun JSON line to stdout.
• The release pipeline subprocesses cargo run -p xtask --release --example nornir-bench --quiet, parses that stdout line, and writes it into the Iceberg bench_runs table. The README tables are then rendered from the warehouse, not by hand.

⚠ Correction, 2026-08-03 — there is no git_sha column. This chapter used to say the row is "tagged with release_id + per-repo git SHA + dep-graph snapshot", and so does nornir/.nornir/bencher.md. READ: nornir/src/warehouse/iceberg_schema.rs:127-140 gives bench_runs exactly seven fields — run_id, repo, ts_micros, date, version, machine, cores — and build_bench_runs_batch (nornir/src/warehouse/iceberg.rs:2440-2456) writes exactly those seven. No git_sha, no release_id. A bench number is keyed by (repo, machine, version, timestamp); SHA attribution is indirect, through release_lineage / regression_trace. Do not design a suite around per-SHA keying it will not get.

Why a fork, not cargo bench/criterion

1. dev-deps. An example may use [dev-dependencies] (osm-katana, znippy-compress, the plugins); a bin may not — and nothing about nornir is allowed to leak into a published crate. That is the whole reason the entry point is an example, not a bin or a [[bench]].
2. Different goal than criterion. criterion does statistical micro-sampling with its own human-readable output. nornir wants single-shot, real-workload numbers (multi-GB archives, the planet OSM pipeline), historized by git SHA, gated for >10% regression, and rendered into the README. criterion's model and cargo's bench output are not machine-historizable the way nornir needs.

The xtask pattern (this repo)

znippy is a multi-crate workspace, so it uses nornir's recommended shape:

xtask/                       publish = false  → heavy deps never reach crates.io
├── Cargo.toml               nornir + osm-katana + znippy-* under [dev-dependencies]
├── src/lib.rs               doc-only; exists so the package has a target the example builds against
└── examples/nornir-bench.rs the ONE bench source: registers every Bencher, calls run_main_json()

Run a release-gate-equivalent locally:

cargo run -p xtask --release --example nornir-bench            # full suite (JSON to stdout)
NORNIR_BENCH_SKIP_PLANET=1 NORNIR_BENCH_SKIP_HEAVY=1 \
  cargo run -p xtask --release --example nornir-bench          # skip the multi-GB workloads

The old before_cratering.sh and xtask/src/main.rs runners (which duplicated each other) and the throughput tests in tests/tests/perf_bench.rs / compress_dir_bench.rs were retired — their unique workloads were ported into nornir-bench.rs so nothing was lost:

• archive_text_500mb / archive_binary_500mb / archive_random_500mb / archive_mixed_repo / archive_huge_2gb — synthetic codec throughput via compress_stream (heavy ones gated behind NORNIR_BENCH_SKIP_HEAVY).
• archive_real_wheels — the Python-wheel corpus alongside jars/crates.
• maven_extract — native (ljar) vs fallback metadata extraction, µs/call.
• read_only_uring_10k — the raw io_uring open+read+close NVMe ceiling that every compress_dir number is measured against (no codec).
• stree_8mb / stree_128mb / stree_1gb — static-search-tree id lookup (single find_exact + batched batch_stream) vs a binary_search baseline, swept across the cache→RAM cliff. Reproduces Ragnar's "40× faster than binary search". Benched as Bencher impls, NOT criterion — even though upstream uses a criterion-style harness, nornir bencher stays the single source of truth (no [[bench]]/criterion connection needed). stree_1gb is gated behind NORNIR_BENCH_SKIP_HEAVY. Full write-up, numbers, and the two perf fixes: stree.md.

Maven correctness tests stayed in tests/tests/maven_bench.rs; only the timing functions moved.

The git package format (znippy.git_*) — where its workloads live

Six arms measure the git package format: git_oid_lookup_{100k,1m,8m}, git_graph_{linear,merges}_200k, git_reach_10k, git_sections_300k and git_archive_60k. The registrations are in xtask/examples/nornir-bench.rs like every other arm, but the workloads are not — they live in plugins/native/znippy-plugin-git/src/bench_kernels.rs behind the crate's bench-kernels feature, and the Bencher impls are thin callers.

That is deliberate, and it is the direct consequence of the worktree trap above: no git worktree of znippy can build xtask at all, so a workload written in nornir-bench.rs could only ever be executed from /home/rickard/git/znippy. Written in the crate it compiles anywhere, cargo test -p znippy-plugin-git --features bench-kernels guards it, and there is still exactly one implementation (LAW 5). The feature is off by default — it is what drags in the fst rival and znippy-compress, and none of that belongs in a shipped archiver.

git_oid_lookup is the arm that matters: __gunnar_oid__ chose an stree keyspace over the stock fst trie and lookup_batch_pipeline::<8> was the whole justification, so the arm times binary search, stree serial, stree batch and an fst::Map built over the same oids — after cross-checking that all four return the same row for every query. The verdict is reported as a neutral *_x ratio rather than asserted, so a loss is visible instead of failing the run. The archive arm writes its fixture to /home/rickard/scratch/… (md1), never /tmp, which on oden is RAM and would have measured the page cache.

The cargo bench muscle-memory — wired

cargo bench -p xtask runs the exact same suite, via a thin [[bench]] shim. A [[bench]] is a separate binary, so it must link the same register_bench! registrations — xtask/benches/nornir.rs gets that by #[path]-including the example source (benches may use dev-deps, like examples), so inventory collects every bencher into the bench binary too:

// xtask/benches/nornir.rs
#[allow(dead_code)]
#[path = "../examples/nornir-bench.rs"]
mod suite;

fn main() -> anyhow::Result<()> {
    nornir::bench::api::run_main_json()
}

# xtask/Cargo.toml
[[bench]]
name    = "nornir"
harness = false

Caveats: it still emits nornir's JSON, not pretty criterion output, and nornir itself keeps calling cargo run --example (it never calls cargo bench). So the shim is only the familiar command with nornir semantics — not a second bench system. See nornir's .nornir/bencher.md → Surfaces for the canonical writeup.