nordisk/nornir

SCIP bench — ours (in-process ra-ingest) vs rust-analyzer scip

A small, controlled head-to-head: nornir's in-process resolved-index builder (--features ra-ingest) against the external rust-analyzer scip CLI, with the external binary pinned to the same rust-analyzer cut so the comparison isolates approach from version. Companion to scip-resolved-knowledge.md (the design) and callgraph-perf.md.

TL;DR

• Same engine. Both paths resolve with rust-analyzer (the ra_ap_* libs / the rust-analyzer binary) on the salsa incremental DB — same find_all_refs / goto_definition. What differs is delivery: OURS links it in-process and fans the resolve phase across all cores with gatling; THEIRS shells out to a batch CLI that emits an index.scip file.
• Version drift = 0. Pinning the external binary to the 2026-06-15 cut (the same cut as ra_ap_* = 0.0.337, which OURS links) produced identical occurrence counts to the older stable-1.95.0 (2026-04-14) binary. So the count difference below is 100 % by-design, not version skew.
• OURS ≈ 11× faster, ≈ 9× leaner on this crate, and lands straight in the warehouse with no temp file — while keeping fewer occurrences on purpose (it feeds a call graph, not a rename engine).

Setup

• Target: a synthetic 128-LOC, zero-dependency crate (3 modules: a Shape trait + Circle/Rect impls, a Stats aggregator, a pipeline that wires them) — small and dep-free so rust-analyzer scip runs fast and the comparison is about the indexer, not the dep closure. Git-committed (the ingest is SHA-keyed).
• OURS: nornir scip ingest <crate> --in-process --no-save (binary built --features ra-ingest; ra_ap_* = 0.0.337).
• THEIRS: rust-analyzer scip <crate> --output index.scip, then parsed + counted through nornir scip ingest <crate> --index index.scip --no-save so both paths report defs/refs through the same counting code.
• Pin: the external binary is the GitHub release tagged 2026-06-15 (rust-analyzer 0.3.2937-standalone) — the same source cut rust-analyzer published ra_ap_* 0.0.337 from. A second run used the rustup stable component (1.95.0, 2026-04-14) to measure drift.
• Machine: local box; CARGO_TARGET_DIR off the T9 SSD (on /home). Numbers are warm, best-of-3; this is an order-of-magnitude bench, not a microbenchmark.

Results

Drift check: the stable-1.95.0 (2026-04-14) binary produced the same 268 (64 def, 204 ref) as the pinned 0.0.337 cut → version drift contributed nothing to the gap.

RE-MEASURED 2026-09-07 (t14s, 16 cores, 46 GB) — the flip's evidence

Re-run because the numbers above were about to be used to justify turning ra-ingest on for the dwarves-server build (dwarves 1b6e731), and a doc figure is not evidence. Different external binary: the rustup rust-analyzer component for stable 1.97.1 (2026-07-14), not the 2026-06-15 cut ra_ap_* 0.0.337 was published from — so this run does NOT have the original's zero-drift property, and its own drift check is the one above. Fixture: an equivalent 120-LOC, zero-dependency 3-module crate (git-committed). Both roads driven through the SAME release nornir binary (--in-process vs --subprocess), best of 3, warm, box otherwise idle.

≈14× faster, ≈6× leaner. The order-of-magnitude claim REPRODUCES. The RSS ratio is lower than the original's 9× only because OURS is measured here through the fat default+ra-ingest release binary (423 MB, viz linked in), which starts heavier than the original's build.

Full nornir — and the one claim that did NOT reproduce

OURS beat its own documented 63.5 s (43.7 s here). But THEIRS did not fail — the "19 minutes AND IT FAILED" line elsewhere in these docs did not reproduce on this box: the external run exited 0 in 10.1 minutes, having printed 2 019 ERROR Encountered enclosing definition with no name lines on the way. Treat that failure as an oden-specific observation, not a property of the subprocess road. That matters for the flip: the shell-out is a fallback that WORKS, which is part of why it was kept rather than deleted (the other part being that it is the only producer of real SCIP monikers — see scip-resolved-knowledge.md).

Note the def/ref inversion at scale: OURS finds ~2× the definitions (it walks file_structure over every workspace file) and ~5× fewer references (workspace- only goto_definition, locals dropped). Subset-by-intent, in both directions.

Why the counts differ — and why that's correct

The 135-vs-268 gap is intentional, not a miss:

• THEIRS is exhaustive. A SCIP index records every occurrence token. The raw index.scip carries ~99 local N symbols — every local-variable binding and use (let mut stats, params area/w/h, …) — plus all package-scoped refs.
• OURS is call-graph-scoped. ra_ingest runs goto_definition per NameRef, workspace-only, and deliberately drops local-variable noise — because it feeds the metro / knowledge map (who-calls-whom), not a rename engine. So it emits the leaner, resolved 135.

Both resolve the same named defs/refs via the same engine; OURS's output is a subset by intent, not a lower-fidelity approximation.

"Can we output the same?"

• Substance: yes. Both populate one scip_occurrences table via the scip crate's own types; counts/positions are directly comparable and load_preferred consumes either identically.
• Wire format: not yet. We do not emit a byte-identical index.scip — scip.rs only parses theirs. The in-memory scip::Index emit and real global monikers are stubbed (OURS keys defs by file#Lline:Ccol def-site identity). Wiring ra_ap_ide::moniker → a real scip::Index is what would enable a true byte/symbol diff (and unblock the cross-repo moniker join).

What OURS buys beyond the numbers

• In-process — no subprocess, no 50 MB binary startup, no temp file.
• Parallel by gatling — the resolve phase fans across all cores rayon-free (gatling_map_owned), one owned Analysis snapshot per chunk (salsa is Send + !Sync + !Clone, so you mint-and-move rather than share). The LOAD phase still uses rust-analyzer's own threads (num_worker_threads) and is salsa-bound.
• Historized + served — rows are SHA-keyed in the warehouse (rust-analyzer never historizes), queryable at any past SHA, and served to CLI / MCP / server / metro through query::load_preferred.

Caveats

• Trivial crate. Absolute times are tiny. OURS's advantage grows at scale (THEIRS amortizes a large fixed startup; the full-nornir figures in scip-resolved-knowledge.md are 63.5 s for OURS vs a 19-min-and-failed external run). The shape — faster, leaner, warehouse-native, subset-by-design — is what generalizes, not the exact ratios.
• Both paths off by default (scip / ra-ingest features).

Reproduce

# (2026-09-07 form: one binary, two flags — `--subprocess` forces the shell-out
#  road even in a build that links the in-process engine, as does
#  NORNIR_SCIP_SUBPROCESS=1.)
#   nornir --workspace <ws> scip ingest scipcrate --in-process --no-save
#   nornir --workspace <ws> scip ingest scipcrate --subprocess  --no-save

# 1. build nornir with the in-process engine (target off the T9 SSD)
export CARGO_TARGET_DIR=/home/rickard/nornir-target
cargo build --release --features ra-ingest --bin nornir

# 2. a tiny git-committed crate in a workspace with a one-line nornir.toml:
#    <ws>/nornir.toml   ->   [repo.scipcrate]
#    <ws>/scipcrate/    ->   Cargo.toml + src/*.rs   (git init && commit)

# 3. pin the external binary to the matching cut
curl -sL https://github.com/rust-lang/rust-analyzer/releases/download/2026-06-15/\
rust-analyzer-x86_64-unknown-linux-gnu.gz | gunzip > ra-pinned && chmod +x ra-pinned

# 4. OURS (in-process)
nornir --workspace . scip ingest scipcrate --in-process --no-save     # 3 runs

# 5. THEIRS (pinned) → parse for comparable counts
./ra-pinned scip <ws>/scipcrate --output index.scip                    # 3 runs
nornir --workspace . scip ingest scipcrate --index index.scip --no-save