nordisk/nornir

Mimir performance — 100-bench suite + never-ending optimize loop

The Mimir knowledge warehouse has two north-star numbers: POPULATE fast (ingest / index build / commit) and SEARCH fast (the MCP retrieval tools). This is the standing epic that measures both and drives them down, cycle after cycle.

The suite

• Benchers: src/bench_mimir.rs — ~100 benchers registered via register_bench!, linked into the nornir lib (so the release completeness-gate's bencher-discovery seam sees them), run by examples/nornir-bench.rs.
• Naming / grouping: nornir.mimir.<populate|search>.<op>_<size>. The mashup and the dwarves testmatrix-benches view group by operation × size.
• Per bench: deterministic fixture → warmup → N timed iters. Records throughput (*_per_sec), latency percentiles (p50_us / p99_us), plus the harness's automatic core-utilisation telemetry (cpu_* / cores_busy_*). A/B arms (Ragnar vs LinearScan; RRF-hybrid vs dense-only) carry speedup_x / recall_*_pct so a cycle can prove keep-or-drop.

Groups

• POPULATE: symbol_facts write, call_edges index build, embeddings + embedding_manifest, Ragnar/Eytzinger/STree64 build (+LinearScan baseline), VectorIndex (vann) build, Iceberg commit throughput. Sizes 1k…100k rows; Ragnar to 10M (cheap in-memory). Plus real_src_embed — ingest nornir's own src/ (the real-codebase north-star).
• SEARCH: Graph-RAG context-slicing (knowledge_context, radius off/on), dense cosine kNN single + batch across corpus sizes and k, call-graph queries, change-radius traversal, STree64/Eytzinger point + range lookups (+A/B baseline), DataFusion SQL templates (def-use, change-radius, call-chain, table-access lineage = the "why" provenance query) + raw point/agg SQL, the RRF hybrid arm, and real_src_vec_knn.

Run it

cd <nornir-worktree>
export CARGO_TARGET_DIR=/mnt/work4t/cargo-target/nornir RUSTC_WRAPPER=sccache NORNIR_MACHINE=Loki
# HEAVY baseline (authoritative), multi-core pass, serial:
cargo run --release --example nornir-bench > /tmp/mimir-bench.json
# ALSO the single-core (_st) denominator in the same invocation:
NORNIR_BENCH_ST=1 cargo run --release --example nornir-bench > /tmp/mimir-bench-st.json
# Persist into the warehouse bench_runs table (+ git receipt):
nornir bench run nornir
# Read history back:
nornir bench history-show nornir

Each row is one BenchResult; results[].metrics carries the numbers. HEAVY is the authoritative figure (memory: heavy-bench law); the run envelope stamps machine+cores+version.

The optimize loop (re-fire this every cycle)

1. Baseline — run the suite HEAVY, stamp a report under .nornir/ (mimir-bench-baseline-<date>.md), diff the regenerated README (docs-render regression guard — report, never silently drop rows).
2. Pick the target — the slowest populate and slowest search path by throughput/p99 (or lowest recall_*_pct for a quality target).
3. Fix — reuse, don't fork. Parallelism is gatling-only (znippy_zoomies::{gatling, gatling_forkjoin, chunk_revolver}); NO rayon / thread::spawn / thread::scope / CPU tokio::spawn. Hot paths must saturate ~11/12 cores (core-saturation law); pure point-lookup latency microbenches are exempt (release::gate::is_latency_microbench).
4. Prove — re-bench; show before/after (throughput/p99 and, for retrieval, recall). Add/keep a red-when-broken test on any new index/populate code.
5. Commit + push the worktree branch. Repeat.

Ranked SEARCH-quality targets (from workspace_nordisk/.nornir/warehouse-graph-vector-market.md)

1. RRF hybrid fusion — DONE (cycle 1). src/knowledge/fusion.rs fuses the persisted Tantivy BM25 index and the dense cosine kNN by Reciprocal-Rank-Fusion (no new dep, ~150 LOC incl. tests). Proven by nornir.mimir.search.hybrid_rrf_* (recall_hybrid_pct vs recall_dense_pct → recall_gain_pct). Next: wire fuse_bm25_dense into the knowledge_context / MCP search tool retrieval path so the shipped tool returns the fused ranking.
2. HNSW ANN (cycle 2, NEXT). Dense kNN is today a BRUTE-FORCE flat SIMD scan (znippy_zoomies::vann), no ANN — nornir.mimir.search.vec_knn_100k shows the flat-scan latency wall. Add a real ANN index behind the existing VectorIndex seam: pure-Rust hnsw_rs or instant-distance (MIT/Apache, embeddable). ⛔ NOT rust-diskann — it pulls rayon internally (ROOT LAW #0). If DiskANN is wanted later, vendor the algorithm onto gatling. Bench kNN latency AND recall (recall@10 vs the exact flat scan) before/after across vec_knn_* sizes; the flat scan is the ground-truth oracle for recall.
3. Embedded PGQ graph-query path (later). DuckPGQ preferred (queries the columnar rows we already hold). Avoid FalkorDB (SSPLv1+server) and upstream Kùzu (OSS frozen after the 2026 Apple acquisition).