nordisk/nornir

RESOLVED knowledge map via rust-analyzer (task #24 spike)

Status: feasibility spike, landed behind two features (both off by default): --features scip (parse a rust-analyzer scip index FILE) and the relaunch's --features ra-ingest (link rust-analyzer in-process, no subprocess). Ties to workspace_nordisk/PLAN.md §Q (ingest rust-analyzer's resolved semantics). This note records what was proven, the precision win, what is stubbed, and the path to full integration.

Where this sits in the picture: the resolved-index pipeline (source scan → single-pass resolution → the ~356k-row persisted index → load_preferred query layer → consumers, plus the planned static-search-tree query layer) is drawn in assets/resolved-index.svg, embedded in design.md. The beginner-level "who calls this?" framing is in nornir-for-dummies.md.

Beyond the spike (landed 2026-06-25): the cold ingest is now a single-pass resolver (one cached goto_definition per NameRef, file fan-out over the rayon-free gatling_map_owned) with a workspace-only scope fix — 35 min → 63.5 s — and the call-graph consumers now PREFER the resolved map (query::load_preferred: CLI / MCP / server / metro feed, tagged resolved/scip vs syn). The full perf roadmap (incremental next, then the static search tree for the query layer) is in the Future perf roadmap section below.

Relaunch (task #24, hard constraints): rust-analyzer IN-PROCESS — PROVEN

The relaunch forbade shelling. ra_ap_* (rust-analyzer's own crates, published to crates.io, synced weekly) was linked AS A LIBRARY and builds + runs against this toolchain:

• Deps resolve + build (rustc 1.95.0): ra_ap_ide, ra_ap_load-cargo, ra_ap_vfs, ra_ap_project_model, ra_ap_ide_db, ra_ap_paths, all pinned =0.0.337 (the 2026-06-15 cut). Pulls the whole r-a graph (salsa 0.27, ra_ap_hir*, ra-ap-rustc_* 0.165). cargo check --features ra-ingest GREEN. Default cargo check stays clean + fast (≈48s lib) — zero ra_ap_* cost.
• In-process pipeline (src/knowledge/ra_ingest.rs): load_workspace_at(root, &CargoConfig, &LoadCargoConfig, &progress) → (RootDatabase, Vfs, _); AnalysisHost::with_database(db).analysis() → Analysis. Definitions via Analysis::file_structure(&FileStructureConfig, file_id); resolved references via Analysis::find_all_refs(FilePosition, &FindAllRefsConfig). Line/col via Analysis::file_line_index().line_col(). No index.scip, no child process. The ScipRow/ScipScan data model + the scip_occurrences warehouse table are REUSED (so ra-ingest implies scip).
• It really runs: the smoke test tests/ra_ingest_smoke.rs writes a tiny cargo crate, loads it into the salsa DB in-process, pulls resolved defs+refs, persists scip_occurrences SHA-keyed, reads load_latest_scip back, and asserts the precision (below). Green: cargo test --features ra-ingest --test ra_ingest_smoke.

Key API findings (0.0.337, the unstable surface)

• LoadCargoConfig gained num_worker_threads + proc_macro_processes fields.
• file_structure / find_all_refs now take a config arg (FileStructureConfig { exclude_locals }, FindAllRefsConfig { search_scope, ra_fixture, exclude_imports, exclude_tests } — neither is Default; RaFixtureConfig IS Default). These shift across patch releases — hence the = pins.

The PARALLELISM blocker (genuine finding) + how it's honored

rust-analyzer's salsa RootDatabase — and thus AnalysisHost and Analysis — is Send + !Sync (its ZalsaLocal query stack is a RefCell), and Analysis is not Clone. So you CANNOT share one Analysis (or &host) across rayon workers: map_init/map_with need Sync/Clone, both absent — the compiler rejects it (E0277, RefCell<QueryStack> not Sync). The salsa-correct parallel pattern, which ra_ingest uses: build the def-site list on the main thread, then create ONE owned Analysis snapshot per chunk (each is cheap + Send, sharing the immutable DB underneath) and move it into its rayon task — defs.chunks(n).map(|c| (c, host.analysis())).collect() → into_par_iter().flat_map_iter(...). Real parallel find_all_refs over the resolved engine, no N reloads. Zero-copy: file paths borrow the Vfs string; the row strings are the only owned data (they must outlive the dropped DB).

Stubbed (relaunch §5 — documented, not built)

• Real SCIP monikers in-process. Today ra_ingest keys each def by a file#Lline:Ccol def-site identity (exact, but local — file is workspace-RELATIVE). ra_ap_ide::moniker can emit the same global moniker strings the SCIP-file path carries (enables the cross-repo moniker join). Still not wired. What CHANGED: the gap is no longer just "the join is missing", it was actively wrong. global_symbol_table is built across ALL members and scip_call_edges_with consults it whenever the local lookup misses — which it does routinely, since sym_kind holds only CALLABLE defs while refs_in_file emits a row for every resolved NameRef (fields, locals, types). Two members' src/lib.rs#L7:C4 are the same string, so a field read in member A joined onto a function in member B and came out as a resolved call edge that does not exist (red-proven: scip.rs::def_site_keys_must_never_join_across_scans). global_symbol_table now refuses a def-site-keyed scan and says so on stderr, naming the member; same-scan edges are untouched. The cross-repo join needs either the moniker wiring or a SCIP-file scan (nornir scip ingest <repo> --subprocess).
• SCIP emit in-memory. ra_ap can build a scip::Index directly (the in-memory data model the relaunch allows) for other tools to consume. Not done.
• Full syn replacement, cross-repo, incremental — see the SCIP-path list below.

The problem

nornir's knowledge_* map (symbol_facts + call_edges, producer src/knowledge/symbols.rs) is syntactic — it parses every .rs with syn and emits:

• one SymbolRow per item (fn/struct/trait/impl-fn/…),
• one CallEdgeRow per call expression, where callee_ident is the raw, unresolved identifier.

That means find-usages / callers-of is a name match: callers_of("name") returns every call whose callee's last path segment is name, regardless of which name it actually is. No type resolution, no trait-method dispatch, no macro expansion, no re-export following. The doc comment on symbols.rs says so outright ("No type resolution — callee_ident is the raw name").

The fix: ingest SCIP

rust-analyzer scip <dir> resolves all of that (it IS the IDE engine) and emits a portable SCIP index (index.scip, a protobuf). Each Document carries:

• Occurrences — every defs/ref token, each tagged with a globally-unique RESOLVED symbol moniker and a symbol_roles bitset (Definition / Import / Read / Write / …),
• SymbolInformation — the definitions, with kind (Function/Method/Struct/ TraitMethod/Macro/…) and display_name.

Because a reference occurrence carries the same resolved moniker as its definition, find-usages is an exact symbol match — no name collision, no last-segment heuristic. We then persist that SHA-keyed (git_sha column), which rust-analyzer never does — so the resolved map becomes historized and time-travelable, the thing nornir's warehouse adds on top of an IDE index.

What landed (this spike)

• Cargo.toml: feature scip = ["dep:scip"] (OFF by default). Pulls the scip crate (0.8) → protobuf 3.x, isolated behind the feature.

• src/knowledge/scip.rs — the producer + in-memory queries:

  • ingest_index_file() / ingest_index() — parse the protobuf Index into flat ScipRows (resolved symbol, role, is_definition, kind, display_name, file, 1-based line/col). BUILD-FREE: it only reads the protobuf that the separately-invoked rust-analyzer produced, so the parse path is unit-tested against a hand-built index.
  • ScipScan::resolve_symbols() / usages_of() / definitions_matching() — the precise find-usages (exact moniker match).

• src/warehouse/iceberg_schema.rs::scip_occurrences() — the table schema.

• src/warehouse/iceberg.rs — TABLE_SCIP_OCCURRENCES registered as a repo-partitioned table; append_scip_scan[_async]() (SHA-keyed write) + load_latest_scip() (max-ts snapshot read-back).

• src/bin/nornir.rs — opt-in nornir scip ingest <repo> (runs r-a or takes --index PATH) and nornir scip query <repo> usages|defs|delta <arg>.

• Unit tests in scip.rs: protobuf round-trip + role/kind mapping + the resolved-vs-syn precision case.

The precision delta (the "is it worth it" evidence)

Canonical case — a trait method with two impls, or a name shared by an unrelated item (e.g. a trait method name() and a struct field name):

• syn stores both call sites as the bare ident name. callers_of("name") folds them — plus any other name in the repo (the Config#name field access) — into one undifferentiated bucket. It cannot tell you which name is meant, and it over-counts.
• SCIP gives each its own resolved moniker (… Greet#name(). vs … Config#name.). usages_of("… Greet#name().") returns exactly the true call sites of the trait method and excludes the field access — IDE-grade.

This is exactly the ingest_maps_roles_and_resolves_exact_symbol test: syn's last-segment match on name yields 3 hits (2 real + 1 false), SCIP yields the 2 real ones keyed to an exact symbol. nornir scip query <repo> delta <name> prints the live count on real data. (Real-index numbers from this repo are in the spike report / commit message.)

Headline delta — the in-process run on the name fixture (real resolved data)

tests/ra_ingest_smoke.rs runs rust-analyzer in-process over a crate with a trait method Greet::name() (called twice) AND a struct field Config.name (read once). The RESOLVED map produced THREE distinct def-site symbols:

The syn side (KnowledgeView::callers_of("name"), query.rs:157 — matches callee_ident == "name" OR ends-with ::name) folds the two g.name() calls into ONE undifferentiated name bucket and cannot say they belong to Greet::name rather than any other .name(); it also cannot see the field read at all (not a call). The resolved map separates all three and attributes every usage to an exact def-site — the precision win, measured on real resolved data, not a hand-built index.

The other resolution syn misses entirely and SCIP captures: macro-generated references (role bit Generated), method calls through generic/dyn bounds, and re-exported paths.

Stubbed / not done (path to full integration)

1. Replace syn as the default knowledge producer. SCIP requires a full cargo build (r-a compiles the workspace), so it stays heavier than the build-free syn scan that powers initial load + the test inventory — syn remains the instant/build-free map, SCIP is the on-demand enrichment pass. The query-side seam is now WIRED, though: query::load_preferred reads the resolved scip_occurrences rows when present (edges by containment via scip::scip_call_edges) and falls back to syn otherwise, returning a resolved/scip vs syn source tag; the CLI knowledge query, the MCP tools, the server RPCs, and the metro feed all route through it. So consumers already prefer the resolved map; what's left is making the producer run automatically (on republish/release) rather than via an explicit nornir scip ingest.
2. Cross-repo. SCIP monikers already encode package + version (rust-analyzer cargo <crate> <ver> …), so a usage in repo A of a symbol defined in repo B is resolvable by moniker join across scip_occurrences rows from both repos. Not wired — needs a cross-repo query that groups by the global moniker rather than by (repo, symbol).
3. Incremental. Today one ingest = one full index = one snapshot. r-a has no incremental SCIP; the historization is per-SHA full snapshots (cheap to store, the warehouse dedups data files). A diff-since-last-SHA view is a query, not a producer change.
4. SCIP emit. So other tools consume nornir's map: the scip crate's write_message_to_file + our rows → an Index is the inverse of ingest_index. Not done; the row shape already carries everything an Index needs (symbol, role, range, kind, display_name), so emit is a mechanical mapping back.
5. MCP/gRPC surface. The knowledge_* MCP tools have syn counterparts; a scip_usages MCP tool would mirror knowledge_callers over load_latest_scip. Not added in the spike (kept to the CLI to bound surface area).

Operational notes

• rust-analyzer is the rustup proxy: rustup component add rust-analyzer, then rust-analyzer scip <dir> --output index.scip. It COMPILES the workspace (loads every proc-macro + build script), so first run is minutes and needs the same toolchain the repo builds with.
• The benign ERROR Encountered enclosing definition with no name def=Module(…) lines r-a logs during indexing are not failures — it streams per-module while resolving and still writes a complete index.

Future perf roadmap — PARKED 2026-06-25 (chase INCREMENTAL next, then back-burner)

Cold-ingest perf is good enough now: 35 min → 63.5 s via the scoping fix + prefill. Ordered by payoff:

1. INCREMENTAL (the big practical win — do this first when resumed). scip_occurrences is SHA-keyed; re-index ONLY changed files, not all 270. Cold run 63.5 s; re-runs → seconds. Diff the workspace file set vs the last snapshot's git SHA, re-resolve changed files + their reverse-dependents, splice into the prior snapshot. Nobody should pay the cold 63.5 s twice.
2. More cores. refs phase scales ~linearly on the custom gatling; 12-core box ≈ 25 s refs. World scale: ~3 concurrent ra per 64 GB box (RAM-bound, ~17 GB/crate), distribute crates across a fleet.
3. refs saturation ceiling. prefill_caches=true got refs 62→35 s (CPU 381→551 %, ~5.5/8 cores). The last ~2.5 cores are salsa first-inference serialization (ra-internal) — diminishing returns.
4. Static search tree (Ragnar Groot Koerkamp) for the QUERY layer — billions-of-rows lookups (callers_of/refs_to), NOT the build. Batchable; SIMD+prefetch. Also the term-dict of a future full-text index.
5. Stream-into-Arrow merge — parked for world/deep scale only; merge is 3 ms at workspace scale (irrelevant now).
6. Load (28 s) = cargo metadata (external) + serial salsa DB build — ~irreducible on a single box.

LANDED this session (2026-06-25): scoping fix (vfs_rel_path strip_prefix? → workspace-only, 2 M→293 k defs, 35 min→80 s) · rayon→custom gatling_map_owned (owned-item work-stealing, constellation-no-rayon) · prefill_caches=true (80→63.5 s) · phase timing · single-pass goto_definition validated vs find_all_refs (defs identical, refs a superset).