nordisk/nornir

Mimir 100-bench baseline — 2026-07-18

Machine Loki · 12 cores · nornir v0.5.4 · HEAVY (serial, multi-core pass) · target on /mnt/work4t · sccache.

Suite: src/bench_mimir.rs — 94 Mimir benchers (35 populate + 59 search) + 11 existing nornir benchers = 105 rows. All 105 tests green, 0 red (incl. the hard RRF recall guard). Run: cargo run --release --example nornir-bench. See mimir-bench-optimize-loop.md for the cycle procedure.

Throughput key per row is the natural one (rows/keys/edges/chunks/commits/queries/lookups/fuse per sec). cores = telemetry cores_busy_max (of 12).

Optimize cycle 1 — RRF hybrid fusion (SHIPPED, PROVEN)

src/knowledge/fusion.rs fuses the persisted Tantivy BM25 index and the dense cosine kNN by Reciprocal-Rank-Fusion (no new dep). On the planted corpus (half the relevant docs semantic-only, half lexical-only) the dense-only MCP path that ships today gets recall@k = 50%; the RRF hybrid gets 100% — +50 points at ~9 µs fuse cost:

• hybrid_rrf_1k: dense 50% → hybrid 100% (bm25 50%), fuse p50 8.8 µs, 111612 fuse/s
• hybrid_rrf_200: dense 50% → hybrid 100% (bm25 50%), fuse p50 8.3 µs, 118403 fuse/s

The bench carries a red-when-broken guard (anyhow::ensure! — runs in release, recorded as a bench TestOutcome): if the fusion ever drops below the dense-only path the bench turns red. Plus unit tests in fusion.rs + tests/mimir_fusion.rs.

Optimize cycle 2 — core saturation (SHIPPED, PROVEN)

Rickard's complaint: the populate + search paths run on ONE CPU. Cycle-2 audit (100-bench cores_busy telemetry + a sat_probe wall-clock probe under the same /proc sampler — cargo run --release --example sat_probe -- <probe> <size> <secs> [k]) found the whole suite pinned at 1–2 / 12 cores. Root cause, path by path:

• Graph-RAG context-slicing (knowledge::context::context_slices) — the symbol_facts JOIN (symbols_in_span) did a FULL linear scan of the symbol table per hit, single-core: a k-hit slice over an N-symbol repo is k·N comparisons on one CPU. FIXED → new symbols_in_spans fans the single scan over ALL symbols across the znippy-zoomies gatling pool (gatling_map_balanced), each worker emitting the (hit, symbol) pairs its batch lands in; results collected in symbol order so every hit's list is byte-identical to the serial per-hit filter (red-when-broken unit test parallel_symbol_join_matches_serial crosses the 32 768 fork threshold). Proof (sat_probe context 20000 6 500, join-dominant — the real warm-view Graph-RAG shape with a large candidate set):

→ 2.9× throughput, 4→12/12 cores, output byte-identical. (At the suite's k=10 the per-query WAREHOUSE RE-READ dominates, not the join, so the microbench row barely moves — the win lands on the real retrieval where the candidate set is large and the view is warm.)

• symbol_facts / call_edges columnar write (warehouse::iceberg:: append_symbol_scan_async) — the wide-table encode built its 9–12 Arrow columns with serial .iter().map(clone).collect() passes on one core. FIXED → new build_columns_par fans the independent per-column builds across the gatling pool; each column sees the same rows in the same order ⇒ the assembled batch is byte-identical (iceberg roundtrip + warehouse_writes_are_sorted… tests green). Gain is modest (sat_probe append 25000 280k→304k rows/s, cores 1→2) because the skade single-file Parquet zstd encode + commit dominates the append — see the honest ceiling below.

Ceilings hit (documented, not forced)

Most remaining suite rows are NOT bounded by an our-code serial loop — forcing a gatling map there would be theatre. The real bottlenecks:

• Parquet encode/commit (skade) dominates every iceberg_commit_* / symbol_facts_* / embeddings_* / real_src_embed row. skade already ships an all-core encode (ingest_parallel_with), but it splits one snapshot into N data files, which changes the DataFusion scan order load_latest reads back → would break the byte-identical symbols/calls row order the call-graph posting lists depend on. Deferred until load_latest imposes a deterministic read order (then the parallel encode is a drop-in reuse, no new code).
• RagnarLog::build / linear_build were assumed sort-bound and filed as "waiting for a gatling parallel sort". Cycle 3 tested that assumption with the now-existing gatling_sort and it does NOT hold — see the cycle-3 section below. The sort is neither the build's dominant cost nor a saturable win on this workload; NOT repointed.
• vann_build_* / per-query vec_knn / ragnar_* point lookups are external (znippy vann SIMD) or genuine single-ITEM latency microbenches — 1-core by design, exempt per the cycle rule.
• context_slice_* at k=10 is per-query warehouse-read-bound (the join fix applies but isn't the microbench's dominant cost; see the proof table above).

Net: the two our-code bulk paths that were serial AND CPU-bound are now gatling — the Graph-RAG join saturates 12/12 when it is the bulk cost. Parallelism is gatling-only (grep -E 'rayon|thread::spawn|thread::scope' on the diff is clean); every conversion is byte-identical with a red-when-broken test.

Optimize cycle 3 — RagnarLog::build sort: tested, NO net win (NOT shipped)

Cycle 2 filed "RagnarLog::build is sort-bound; repoint onto a gatling parallel sort once znippy-zoomies has one." That primitive now exists (znippy_zoomies::gatling_sort::gatling_sort_by, znippy-zoomies 027e4ce — a byte-identical drop-in for slice::sort_by; path-dep already resolves it, no bump needed). Cycle 3 A/B-tested the repoint before shipping — via a new ragnar probe added to examples/sat_probe.rs (serial slice::sort_by vs gatling_sort_by on the exact (key, orig, payload) tuples the build sorts, under the same /proc Sampler; asserts the two orders are byte-identical — they are). 10M keys, Loki, contended:

Two reasons the repoint is not a win here, both load-bearing:

1. The input is (near-)sorted. The warehouse commit-log keys are monotone snapshot-ids / ts_micros arriving in append order; the bench fixture ((k*2, k)) is likewise pre-sorted. slice::sort_by is adaptive (timsort) — O(n) single-pass on sorted runs — so a parallel merge-sort does strictly more work and loses (1.6×). gatling only wins on genuinely shuffled input, which this path does not see.

2. The sort is not the bottleneck. Sort-only throughput is ~578M keys/s (1M, sorted) but the full RagnarLog::build is ~95M keys/s — the STree64 construction

   • Eytzinger layout + allocations dominate, not the sort_by. Even a free sort would barely move the build.

Also: even the shuffled "win" tops out at 2/12 cores — the sort is memory-bandwidth-bound on 24-byte tuples (10M ≈ 240 MB), so it does not approach the 11/12 the core-saturation law targets. Decision: do NOT repoint RagnarLog::build (the lib change was written, A/B-measured, and reverted). The sort primitive is sound and re-usable elsewhere — but the right consumer is a large unsorted collection on a CPU-bound (not bandwidth-bound) hot path, and no such sort exists in the populate/index-build paths (the other candidates — StreeIndex hash sort, event-log sort_by ts — are either small or likewise append-ordered). The ragnar sat_probe arm is kept as the reusable evidence tool. Next-cycle target: move off sorts entirely — the remaining honest saturation levers are the write-cadence ceiling (iceberg_commit_*, ~35 commits/s, Parquet+manifest per commit) and vec_knn_100k brute-force flat-scan → HNSW ANN (done in cycle 4 below).

Optimize cycle 4 — HNSW ANN behind the VectorIndex seam (SHIPPED)

Mimir's dense kNN was a brute-force flat scan (znippy_zoomies::vann SIMD cosine over every row) — the vec_knn_100k 2.85 ms single-query wall. Cycle 4 adds a real HNSW (Hierarchical Navigable Small World) approximate index so the live vector_search MCP path gets O(log n) dense retrieval, keeping the flat scan as the exact 100%-recall oracle + small-corpus fallback.

ANN engine — VENDORED, rayon-free. The two embeddable Rust HNSW crates both break the gatling-only ROOT LAW: hnsw_rs (hnswlib-rs) and instant-distance each pull rayon into their parallel build. So the HNSW is written directly onto std in znippy_zoomies::vann::hnsw — zero new external deps. Proof the crate that gained the code stays clean:

$ cd znippy-zoomies && cargo tree -e normal | grep -i rayon   # → (empty)

(nornir itself shows rayon only transitively via the pre-existing candle embedder — not introduced here and not on the vector path.) Parallelism, when added, goes through this crate's own gatling_forkjoin pool, never rayon. The build is single-threaded (graph mutation is serial); the win is the query descent. Follow-up lever if the 100k build proves too slow single-threaded: parallelise construction onto the gatling pool (see #88 HEAVY sweep).

Wiring.

• VectorIndex::build_hnsw (znippy-zoomies src/vann/mod.rs) reuses the stored normalized matrix verbatim → an HnswIndex (src/vann/hnsw.rs).
• store::Reconstructed::ann (nornir src/vector/store.rs) lazily builds + memoizes the graph when the corpus ≥ HNSW_MIN_ROWS (4096) and NORNIR_VECTOR_HNSW isn't off; dense_search_ann routes hybrid_search's dense arm through it (falls back to the exact flat scan otherwise). search() stays exact — it is the oracle every test/bench measures against. RRF fusion on top is unchanged, so the shipped vector_search MCP tool keeps its recall gain AND gains O(log n) dense retrieval.

Recall (PROVEN, red-when-broken). HnswParams default M=16, efConstruction=128, efSearch=64, cosine metric (dist = 1 − cosine). Two gates, both green:

• vector::store::tests::live_dense_arm_hnsw_recall_floor_vs_flat_oracle (cargo test --lib, PASS) — a 5k-row live warehouse snapshot; confirms the ANN path engages and its recall@10 vs the exact flat oracle ≥ 0.90.

• vann::hnsw::tests::hnsw_recall_floor_vs_flat_oracle (znippy-zoomies, PASS)

  • a planted-exact-match test.

• Bench guard: SearchHnswKnn (vec_hnsw_1k/10k/100k) turns red if recall@10 drops below its floor.

Latency. New benchers nornir.mimir.search.vec_hnsw_{1k,10k,100k} report HNSW p50 + recall@10 + the flat baseline on the same corpus (speedup). Measured numbers land via the bench runner on a quiet box:

Latency/recall table pending the HEAVY sweep on a quiet box (#88); the recall@10 floor (≥0.90 vs the flat oracle) is already proven green by the unit tests above.

POPULATE (35 benchers)

SEARCH (59 benchers)

Optimize cycle 5 — append_symbol_scan fsync batching: tested, NO measurable win (NOT shipped)

Cycle 4 left "iceberg_commit_64 35 commits/s … batch commits" as the top POPULATE lever (see Slowest paths below). Cycle 5 built it and A/B-measured it before shipping — the win did not materialise, so it was NOT shipped.

What was built. IcebergWarehouse::append_symbol_scan_async issued one append_batch → one fast_append → one redb transaction per non-empty table — up to 4 fsyncs per scan. The patch extracts the columnize into symbol_scan_batches(), stages every table through stage_for_multi() / split_for_append(), and commits all of them in one skade::ingest_multi ⇒ 1 fsync — the same cross-table primitive the deep-scan flush (ingest_symbol_scans_async) already uses. split_for_append mirrors skade's own append_parallel_props rule (APPEND_PARALLEL_MIN_ROWS = 8192, one file per core, row-aligned zero-copy slices) so the data-file layout is unchanged, not merely equivalent.

Byte-identity: PROVEN (this part is keepable and reusable). The test warehouse::iceberg::tests::multi_commit_matches_per_table_append_byte_for_byte drives BOTH strategies from the SAME symbol_scan_batches output — one arm replaying the old per-table append_batch loop, one the new single-transaction path — and compares, per table, the Parquet data-file count, per-file byte length, and full payload, plus decoded row equality. Fixture crosses the 8192-row split threshold so both the multi-file and single-file paths are covered. Green, and it genuinely runs (lib test count 926 → 927; not one of the #![cfg(feature)]-silenced files). Notably the payload assert did not fail, which settles an open question: skade's ArrowWriter encode and iceberg's ParquetWriter single-file path emit identical bytes here.

Performance: NO measurable win. Measured on the t14s working box (16 cores, heavily contended — ~60 concurrent fleet cargo processes), light arms only (heavy sweep still owed to an idle box, #88). A first naive BEFORE/AFTER pass suggested iceberg_commit_64 +34 % — that was drift, not signal. Re-measured with two saved binaries run interleaved round-by-round so machine drift hits both arms equally (scratchpad/interleave.sh, median-of-3 per round):

In the two clean rounds the arms are statistically indistinguishable (+0.6 % on the target arm); round 3 is a load spike that degraded both. Same binary re-run gave iceberg_commit_64 11.9–16.6 /s across the session — the noise band is wider than any effect being claimed.

Two load-bearing reasons the batching does not pay here:

1. The benchers structurally cannot show it. mk_scan (bench_mimir.rs) sets features: vec![] and tests: vec![], so every arm writes only 2 of the 4 tables — the benched case is 2 fsyncs → 1, never the full 4 → 1. The 4-table shape is only reachable from the real single-scan callers (bin/nornir.rs, monitor.rs); the deep-scan bulk path already batches.
2. The fsync is not the bottleneck at these sizes. At 200–5 000 rows the Parquet encode + manifest/metadata rewrite dominates a commit (exactly what the cycle-4 note said: "Parquet+manifest per commit"), so removing one fsync moves ~nothing.

Decision: do NOT ship the write-path change (loop law: keep only if faster AND byte-identical — byte-identical yes, faster no). Patch preserved at /home/rickard/nornir-append-symbol-scan-batching.patch. It remains a genuine crash-consistency improvement (the four tables would flip all-or-nothing instead of a reader being able to observe symbol_facts advanced while call_edges is not) — that is a correctness argument, not a perf one, and is Rickard's call to make on its own merits, not something to smuggle in under a perf banner.

Before this lever is revisited, fix the measurement: give mk_scan non-empty features/tests (or add a 4-table arm) so the benched path matches the shipped one, and re-run on an idle box. A lever whose benefit the suite cannot express is not a proven lever.

Slowest paths (next optimize targets)

• SEARCH: context_slice_10k_radius ~40 ms p50 and context_slice_10k ~11.7 ms (each query re-reconstructs the VectorIndex + reloads symbol_facts/call_edges — cache the per-repo reconstruct across queries). vec_knn_100k single-query 2.85 ms is the brute-force flat-scan wall → cycle 4 = HNSW ANN (SHIPPED, see above) routes the live hybrid_search dense arm through an O(log n) graph, recall@10 ≥ 0.90 vs the flat oracle (batch already amortises to 0.45 ms/query at 11/12 cores).
• POPULATE: iceberg_commit_64 35 commits/s (29 ms/commit) is the write-cadence bottleneck (Parquet+manifest per commit — batch commits / larger row-groups). vann_build_100k_d256 uses only 3/12 cores — a gatling fork-join over the quantize/add is a core-saturation win.

Ragnar STree64/Eytzinger point+range lookups beat the LinearScan baseline by 5,500×–340,000× at 1M–10M keys (O(log n) vs O(n)); pure latency microbenches (1 core, exempt from core-saturation).