nordisk/nornir

Scalable bench design

How nornir's benchmark pipeline scales — from one developer running nornir bench run on a laptop to many machines streaming results into one warehouse served over the network. This is the scaling companion to bencher.md (the bench framework / contract) and warehouse.md (the Iceberg store). Both design rationale and a reference for the work still open.

The shape of the problem

A bench result is one BenchRun (version, machine, cores, date, a flat set of <corpus>[_<variant>] metrics + optional per-test pass/fail). At scale there are three independent axes:

• Many repos — every workspace member can carry a bench example.
• Many machines — the same bench runs on different hardware; results are partitioned by machine so they're never compared across boxes by accident.
• Much history — every release adds runs; reads (bench_history, gates) must stay fast as the table grows.

The design keeps the producer (running benches) and the store (Iceberg) loosely coupled so each axis scales on its own.

Today's path (the baseline)

child repo  examples/nornir-bench.rs  ──prints one BenchRun JSON line──►  nornir
nornir  ──parse──►  append_bench_run  ──►  bench_runs + bench_results (Iceberg)
                                       └─►  bench_history.jsonl (chmod -w receipt)

A child repo opts in via [dev-dependencies] nornir + impl Bencher + register_bench! + examples/nornir-bench.rs; nornir is invisible to production builds. The JSONL file is the on-disk receipt + migration source; Iceberg is the source of truth (last_run reads iceberg-first).

This is correct but has two scaling limits: the JSON-file hop on the live path, and a single-threaded warehouse write.

Write path & parallelism — the catalog pipeline.rs pattern

warehouse::iceberg::append_batch is the simple path: one task builds one ParquetWriter, writes one RecordBatch, close(), fast_append().commit(). iceberg-rust does not parallelize this — the parquet ArrowWriter encodes serially; it just hands you the writer + transaction primitives.

The scalable topology is the one skade-katalog prototyped in its bench harness (bench/src/pipeline.rs, the "znippy pattern"):

N encode workers              bounded channel            1 writer task
RecordBatch -> Parquet bytes  ──►  mpsc(depth)  ──►  sequential put + fast_append
     (CPU, all cores)                                  (I/O + commit, 1 core)

Many CPU threads encode Parquet in parallel; one thread does the I/O + commit (the single-writer invariant the redb catalog requires anyway).

When to adopt it: only for bulk writes — the embeddings table (768-dim float lists across many chunks) and large bench-result imports (many machines × many runs ingested at once). For nornir's typical append (one BenchRun, a few rows → one snapshot) the parallel encode just adds overhead; the catalog's own header warns the single-writer split can lose to per-task encode+write on fast NVMe at small scale. Measure before adopting; leave append_batch serial until a bulk path proves the win.

Bencher under the service — RPC ingest, no JSON hop

To scale across machines, the producer should record straight into the warehouse over the network instead of via a local JSON file:

bench host (any machine)  ──RecordBench RPC──►  nornir-server  ──►  append_bench_run
                                                (sole warehouse-lock owner)

• A RecordBench RPC on the bench service carries (workspace, repo, BenchRun); the server validates against the repo's required_results and appends.
• Drops the JSONL hop on the live path (JSONL stays as the chmod -w receipt / offline-import fallback).
• Pairs with multi-workspace routing: the RPC names the workspace, the server routes to that workspace's warehouse handle (serialized writes per workspace).
• Lets a remote/airgapped bench host stream results to a central server — the same gRPC path the CLI uses in client mode (embedded or client, never both on one warehouse — see design.md).

Storage layout for scale

• bench_runs partitioned by (repo, machine, year-month) so reads prune to one machine's recent history instead of scanning all rows. (Partition specs are the open warehouse-efficiency win.)
• Single-core variants land as _st-suffixed metrics from a pinned run (NORNIR_BENCH_PIN=0 / --single-core), so one nornir bench run yields both multi- and single-core rows reproducibly.
• Reads are warehouse queries: query_bench_runs(BenchFilter::for_repo(name)) with repo pushdown. The shipped bench_history doc renderer is exactly this — N rows newest-first — so READMEs render their own history with no hand-maintenance.

Open work (tracked in plan.md)

• Bench-over-service RPC — landed. nornir bench run with NORNIR_SERVER set ships the locally-run BenchRun to the server over the Bench.Submit gRPC RPC (protobuf), which persists it (server owns the lock); default stays in-process append. First tonic client in the tree (submit_bench_remote). Lossless: proto BenchRun carries results + tests, and the server Submit persists into iceberg bench_runs (not just the JSONL receipt), so a remote submit is fully queryable.
• Parallel-encode bulk appends (catalog pipeline.rs pattern) — embeddings + bulk bench imports; measure first.
• bench_runs partition spec by repo — landed (prunes scoped reads to one repo's files). Optional (repo, machine, month) refinement still open.
• NORNIR_BENCH_PIN / --single-core so one run yields both _st and multi-core rows.
• bench_history MCP/server parity — done: CLI, MCP (docs_render/check/ book/export + docs_fresh/gate-all), and server (render/check + docs_fresh/gate-all) all feed history (from the JSONL receipt, matching how each surface already reads last_run), so the section renders everywhere and the docs_fresh gate doesn't false-drift on a bench_history marker.