nordisk/nornir

Mímir v2: what it is, for reviewers

This page describes Mímir as it is on nornir main on 2026-11-04. It is written for a reviewer who has not read the code. Every claim was checked against the code; file:line references are extras for those who want to look. Paths are in nornir unless they start with dw: (dwarves, crates/dwarves/). The graph lenses and the folds are nornir's crate crates/mimir-lens/ (moved from dwarves on 2026-11-05); dwarves is their CLI door. The detailed manuals are README.md, tables.md, compiler-to-graph.md and lenses.md.

0. Design stance (owner, 2026-11-04)

Mímir is meant to hold the whole Rust code universe: every workspace crate, every dependency at its locked version, std, and the system calls below them, across history. Large memory use is accepted as the price of that. The machines it runs on have hundreds of gigabytes of RAM. Reviewers should therefore weigh proposals by coverage, exactness and answer time on the full universe, not by peak memory. A change that saves memory but drops facts, approximates an exact answer or narrows the universe is not wanted. A change that keeps every fact and makes the full universe faster or cheaper to rebuild is.

1. Purpose

Mímir turns the whole code universe of a Rust workspace into one typed graph. "Whole" means the workspace's own crates, every dependency crate at the exact version the lockfile resolves, the toolchain's std, and, below them, the kernel's system calls. The facts come from the compiler itself, not from a parser of ours.

Questions it answers, with evidence and named gaps:

• Who calls this, what does it reach, and what is the shortest path?
• Which tests reach this code, and which tests can never fail?
• What did the last commit break, and for which other repos?
• Which code reaches which warehouse table, and which system calls a program can possibly make?
• Which functions are copies, near copies or independent twins?
• Which features are dead, which gates can be bypassed, where are the cycles and the chokepoints?

Who calls it:

2. The pipeline end to end

 repo at commit C
   │  cargo mimir-dump  (cargo check --workspace --all-targets --keep-going,
   │                     RUSTC_WRAPPER=mimir-driver, nightly-2026-10-05)
   ▼
 mimir-driver: rustc in process, after_analysis hook walks TyCtxt + MIR
   │  per compiled unit (member, dependency, std):
   │    <crate>@<ver>-<hash>.{nodes,edges,spans,symbols}.arrow + .stats.json
   │  (Arrow IPC file format, one RecordBatch, temp name + rename)
   │
   ├──────────────► nornir-mimir-lens (offline; door: dwarves lens):
   │                Graph::from_dump → adjacency lists
   │                → GraphBLAS matrices (crates/mimir-lens/src/lenses/grb.rs) → 12 graph lenses,
   │                  verdict, zombie/risk/bloat/coverage folds → Report
   ▼
 nornir consumer (src/code_room/consumer_loop.rs, crates/mimir-ingest)
   │  read_out_dir → delta vs the repo's open set → +/− events
   │  ONE skade atomic release per (repo, sha): events + offset + ledger
   ▼
 lake: skade (Iceberg catalog in redb, Parquet zstd-3, 256k-row groups)
   │  code_{node,edge,span}_events, universe_*_events, code_symbol_events,
   │  bin_*; consumer_offsets (repo → seq, sha); READ-FAST interval tables
   │  code_{nodes,edges,spans}_v  [valid_from_seq, valid_to_seq)
   ▼
 mimir-room: fold at a Scope {repo → seq, universe on/off} → Snapshot
   │  sorted monikers = dense u32 ordinals; one boolean n×n GrB_Matrix per
   │  edge kind; unions, transposes, the Jikes matrix built on first use, kept
   ▼
 readers: CodeRoom.Answer surfaces (Viz.CrateGraph, Knowledge.Source, …)
          mimir_lens::answer (26 table-fed lenses + 2 source lenses)
   ▼
 answer envelope: worst[…], feeds[], evidence_note (MISSING / LOWER BOUND),
                  lens_meta.state (complete | partial | empty_fact | empty_gap | …)

The producer is the only source of graph facts. It is a clippy-shaped rustc driver (crates/mimir-driver/src/main.rs). It does not write the lake. It writes files, and the nornir consumer is the only writer of the lake (README §5). dwarves reads the same files and never the lake.

Other feeds ride the consumer's offsets and write the warehouse beside the lake: git_heat, sbom, dep_graph, code_index, binary, clone_shapes and embeddings (src/code_room/feeds.rs:13-21).

3. The data model

Nodes (code_nodes, tables.md §1). Kinds: crate mod fn method struct enum union trait trait_alias impl const static type macro ffi foreign_type field variant param ret local generic assoc_type assoc_const mono_instance ffi_import feature, plus the MIR parts mir_body and mir_block, and the boundary's syscall nodes in the universe. Each node has a level (workspace · repo · crate · module · item · part), its span, cfg, visibility, effective_visibility, has_doc, is_test, defined_here (false = a reference stub of another crate's item) and origin:

• local = a workspace member;
• library = a dependency crate;
• lang = a toolchain crate.

Edges (code_edges). Kinds: CONTAINS DEPENDS_ON CALLS IMPLEMENTS FOR_TYPE USES_TYPE READS_FIELD EXPANDS GATED_BY TESTS TOUCHES_TABLE COMPATIBLE INSTANCE_OF ENABLES ENABLES_WEAK BUILT_FROM SYSCALLS CFG_SUCC WRITES_FIELD (the last since schema 6: a body → each field it writes; READS_FIELD is every mention, so a read-only use is READS_FIELD without WRITES_FIELD). An edge has a site count; positions live in code_spans under the key from ␞ to ␞ KIND, so moving code changes spans and never an edge.

Ids. Every id is a SCIP moniker string, rust-analyzer cargo <package> <version> <descriptors>, made by one formatter (crates/mimir-schema/src/moniker.rs). Only the grammar is SCIP's; there is no index.scip and no SCIP tooling. A "stable" form drops the version, for identity across bumps (moniker::stable).

MIR payloads (compiler-to-graph.md §6). The driver reads mir_promoted. Members get full facts: one mir_body per fn, one mir_block per basic block, CFG_SUCC edges. Dependencies get the body fingerprint only (MIMIR_MIR changes this). A block's payload holds its statements, terminator, the locals it touches with their types, and the data-flow pairs a>b (local a read where b is written). Constants are recorded by type, never by value. The block fingerprint is FNV-1a 64 over the block text with locals renamed by first use, then mixed with the stable callee moniker. So two blocks that do the same thing to the same types share one fingerprint.

Caps. No block cap per body since schema 6 (every block is a row; MIMIR_MIR_BLOCKS / cargo mimir-dump --mir-blocks N sets one), and STMTS_PER_BLOCK = 128 statements per block (crates/mimir-schema/src/mir.rs:62). Ordinals are 32-bit. The room drops mir_block rows and CFG_SUCC unless MIMIR_ROOM_MIR_BLOCKS=in; lenses read block payloads from the lake per repo. Pane lists are capped at 200 entries.

4. Time and history

The time axis is the consumer sequence number seq, per repo: the ordinal of the commits the consumer has walked, first-parent order. Each event row carries repo, seq, sha, op (+/−), payload_hash. consumer_offsets maps a repo to its current (seq, sha).

As of a commit. A Scope names repo → seq (and universe on or off). A sha is not a scope key: Coderoom.SnapshotDiff takes repo@seq and refuses a sha (src/code_room/reader/wired/tests.rs:433). The room at a scope is built in one of two ways:

1. The fold. For each key, the rows with a + at or before seq and no later − (the open set). This scans the repo's event history.
2. The READ-FAST zone (crates/mimir-ingest/src/readfast.rs). Per source kind there is a derived interval table, code_nodes_v, code_edges_v, code_spans_v, partitioned by repo. Each row is one version with [valid_from_seq, valid_to_seq); valid_to_seq = NULL means open. A table property mimir.readfast.anchor.<repo> says how far the zone is complete. A read takes the interval rows live at min(seq, anchor) plus the event tail after the anchor. The event tables stay the truth; a repo without an anchor falls back to the fold. The server refreshes the zone as a lake_readfast job after each consume (src/server_grpc.rs:6439).

Diffs.

• Room::diff(older, newer) (crates/mimir-room/src/diff.rs): nodes by a merge of the two sorted moniker lists (equal moniker, other payload hash = changed); edges per kind in the union node space as added = A_new⟨¬A_old⟩, removed = A_old⟨¬A_new⟩. This needs two rooms.
• EventLog::interval_diff(kind, repo, from, to) and the room's commit window (crates/mimir-room/src/window.rs) answer seq−1 → seq from the zone with range predicates on the interval columns, without folding either room. It answers only when both seqs are inside the anchor and the repo has no binary stream at either seq; otherwise the caller folds two rooms. world_impact uses the window first and the two-room diff as the fallback, and keeps two windows per repo (lenses.md, world_impact).

History. A member's first consume starts at its tip. Older commits come from a backfill job that consumes one commit per turn (src/code_room/backfill.rs). Each turn is a full cargo mimir-dump of that commit, with cargo's incremental fingerprints deciding what recompiles. A past commit is built against today's sibling repos; when it does not load, it is recorded as error and skipped.

Cost, as measured (sources and dates in §6): a tip consume of one member with 608 crates took 299 s (dump 156 s, delta 121 s, commit 22 s); the next commit with two bodies changed took 58 s (58 crates re-dumped). A room for the whole workspace took 28 s on dwarf. There is no measured number for the READ-FAST read against the fold.

5. Vectors and similarity

All of these exist in code today. "Real" means a producer writes the input in the normal flow.

Model and backends. The default model is jina-v2-base-code (768 dimensions). The registry also lists a q8 variant, minilm-l6-v2, bge-base-en-v1.5 and the static potion-base-8m (src/vector/embed_registry.rs:113-255). Backends: tract on CPU, onnxruntime on CPU, CUDA or ROCm, picked at run time (src/vector/select.rs), or a remote GPU worker ($NORNIR_EMBED_REMOTE, src/vector/remote.rs). Vectors from two models never mix (model_profile key).

The pair index (src/vector/pair_index.rs). similar_code (cosine 0.92) and semantic_clones (0.80) need every pair above a bar. The index is an exact cosine self-join over the distinct vectors, kept in the process: FLOOR = 0.80, used from MIN_ROWS = 2,048 rows. The cold build is n²/2 dots in AVX2 register tiles (src/vector/tiled.rs; an int8 prefilter gives the same pairs and floats); a commit adds k × n dots for k new vectors. There is no ANN, and the index runs on the CPU only. Pairs are held as Vec<(u32, u32, f32)>. Clone lenses then apply mutual top-k (MUTUAL_K = 8) and union-find classes. semantic_twins uses TWIN_COSINE = 0.882, the operating point calibrated for jina-v2-base-code on 3,222 nornir functions (ROC-AUC 0.920, src/mimir_lens/room_feed/clones.rs:2044).

6. Compute

GraphBLAS (SuiteSparse 7, through crates/graphblas-sys and the safe layer crates/mimir-graph). The operations, with A = boolean adjacency:

(crates/mimir-graph/src/ops.rs; table with lines in lenses.md §3.4.) Matrices are packed from sorted from·2³² + to pairs, then set to GxB_AUTO_SPARSITY so GraphBLAS can store a mostly empty matrix hypersparse (crates/mimir-graph/src/matrix.rs:207). The only global option set is the thread count (GxB_GLOBAL_NTHREADS, crates/mimir-graph/src/runtime.rs). The graph lenses (crates/mimir-lens) add components, Leiden (misplaced), closure by repeated squaring (dsm), dominators and betweenness (chokepoints), and a CFPQ walk (gate-bypass). The possible-system-calls walk is a plain breadth-first walk with parents over edge reads, not a matrix product (crates/mimir-room/src/possible.rs:461).

Caches.

• Rooms are cached per process, keyed by the log identity and the scope; at most one ring view (universe on) is kept (crates/mimir-room/src/lib.rs:480-500).
• Lens answers are kept per (room version, lens, request, build sha) (src/mimir_lens/queued.rs).
• clone_inc.rs and reach_inc.rs bring clone folds and reach walks forward incrementally; both must equal the full recompute.
• The driver side has the cargo target dir, a mirror of the latest dump of every unit (keyed by cargo's unit hash), and a semantic ring ledger for dependency units (§7 of the table below).

jera lanes (src/server_grpc.rs:205-262): backfill, feeds (deferred embeddings), rooms (room warm-up, the ring's embed), five lenses lanes (LENS_LANES = 5), panes (pane answers, high priority) and lake (compaction, READ-FAST refresh). A Mimir.* door answers a kept result at once, or queues the lens and answers queued / running with a job id.

Numbers (only what a file records; one run each unless said):

7. What is NOT there

• No SCIP import from other indexers; only the moniker grammar is SCIP's.
• No "world ring": crates.io callers and published semver diffs have no producer. Coderoom.Compat and Coderoom.WhatBreaks answer not_wired.
• No binary call edges (only linked symbols, sizes and sections).
• CALLS is a lower bound: calls through operators, closures and fn pointers are not all resolved. The possible-syscalls walk widens such sites; the lenses say LOWER BOUND.
• No literal values in MIR facts. (Up to schema 5 also no MIR past the 1,024th block of a body; since 6 every block is written.)
• No per-commit file sets; git heat holds counts only.
• No query language; the API is typed calls and named surfaces.
• No sha-keyed scope; time is repo@seq.
• No commit snapshots as bitmaps (Roaring or otherwise); the READ-FAST interval tables play that role.
• No ANN, no GPU and no top-k heaps in the pair index; no centroid pruning.
• No GxB_GLOBAL_POOL setting and no explicit GxB_HYPERSPARSE; AUTO sparsity only.
• No content-addressed cache of member dumps across commits: a driver change that alters member rows needs a re-dump of every commit to rebuild history.
• No embeddings for dwarves dupes.
• No measured comparison of the READ-FAST read with the fold.
• File text is not in the lake; it is read on demand with gix.

8. The old review vs today

The old review was written against an older description. Each row was checked against the code on 2026-11-04.