nordisk/nornir

Mímir v2 — the Rust code graph

Mímir v2 is nornir's code graph. One producer (a rustc driver) writes Arrow IPC files per compiled crate. One consumer folds them into an Iceberg/Parquet lake. One in-memory room (GraphBLAS) answers questions through named reader surfaces and lenses. Every node id is a SCIP moniker string.

This document describes the code on main as it is today. File references are repo-relative paths in nornir unless another repo is named. A short list of planned items sits at the end, marked as planned.

New to Mímir? Start with overview.md (what it is, for reviewers).

Chapters:

1. Glossary

2. Flow

  repo clone at commit C
        │
        ▼
  cargo-mimir-dump ──► cargo check --workspace --all-targets
                       RUSTC_WRAPPER=mimir-driver  RUSTUP_TOOLCHAIN=nightly-2026-10-05
                       target dir: target/mimir     env: MIMIR_OUT MIMIR_REPO MIMIR_SHA …
        │
        │  per unit:  <crate>@<version>-<hash>.nodes.arrow
        │             <crate>@<version>-<hash>.edges.arrow
        │             <crate>@<version>-<hash>.spans.arrow
        │             <crate>@<version>-<hash>.symbols.arrow
        │             <crate>@<version>-<hash>.stats.json
        ▼
  ┌────────────────────────────────────────────┐   ┌──────────────────────────────────┐
  │ nornir code room (server)                  │   │ dwarves `cargo lens graph-package` │
  │ consumer: read_out_dir → delta vs open set │   │ reads *.edges.arrow, kind CALLS    │
  │ ONE skade atomic release per (repo, sha)   │   │ joins with cargo metadata + lock   │
  │ lake: <builds>/mimir_v2/lake (Parquet)     │   │ writes target/dwarves/             │
  │ mimir-room: fold → Snapshot (GraphBLAS)    │   │   graph-package.znippy             │
  │ reader::door surfaces + mimir_lens::answer │   │ (see dwarves docs/v2/README.md)    │
  │ gRPC CodeRoom.Answer :7878                 │   └──────────────────────────────────┘
  │   proxied by the dwarves forge on :7900    │
  └────────────────────────────────────────────┘
        │
        ▼
  dwarves-gui (Coderoom tab, graph source pane) → facett CoderoomPane

3. The producer

3.1 cargo-mimir-dump

crates/mimir-driver/src/bin/cargo-mimir-dump.rs. It is a cargo subcommand (cargo mimir-dump) that does two things:

1. One cargo metadata, written to <out>/crate-versions.json (MIMIR_CRATE_VERSIONS).
2. cargo check --workspace --all-targets (or cargo build with --build) with the driver as the rustc wrapper.

The nornir consumer runs it with --keep-going, so every crate that compiles is dumped even when another does not (crates/mimir-ingest/src/driver.rs).

3.2 mimir-driver

crates/mimir-driver/src/main.rs. A clippy-shaped rustc driver: it drops cargo's rustc from argv, compiles in process with its linked nightly, and registers a rustc_driver::Callbacks whose after_analysis hook walks the TyCtxt and returns Compilation::Continue, so rustc still emits the metadata cargo expects. Proc macros, build scripts and --print calls are plain rustc. The walk is crates/mimir-driver/src/walk.rs (and walk/).

3.3 Helpers

3.4 The per-unit files

For every compiled unit the driver writes, under MIMIR_OUT:

<crate>@<version>-<hash>.nodes.arrow      table code_nodes
<crate>@<version>-<hash>.edges.arrow      table code_edges
<crate>@<version>-<hash>.spans.arrow      table code_spans
<crate>@<version>-<hash>.symbols.arrow    table code_symbols
<crate>@<version>-<hash>.stats.json       counts and timings
<crate>@<version>-<hash>.mir.arrow        table code_mir        (schema 8, one row per body)
<crate>@<version>-<hash>.inst.arrow       table code_instances  (schema 8, one row per instance)
<crate>@<version>-<hash>.data.arrow       table code_data       (schema 8, one row per static)

<hash> is the short hash of the rustc invocation. A toolchain crate (core, std, …) has a URL as its version; its files say @lang (walk.rs, the stem computation).

Format: Arrow IPC file format (arrow_ipc::writer::FileWriter), one RecordBatch per file, written to a temporary name and renamed (walk.rs write_ipc). Every batch carries two schema metadata keys: mimir.schema = 8 and mimir.table = the table name (crates/mimir-schema/src/lib.rs SCHEMA_VERSION, META_SCHEMA, META_TABLE).

The three lossless files have zstd buffers and up to 2,048 rows per batch; a unit may lack any of them. A program's --exec unit has only these three and <stem>.exec.json.

The columns of the tables, the node kinds and the edge kinds are in tables.md. In short:

MIR facts have ONE lossless form since schema 8: the code_mir rows of <stem>.mir.arrow, one row per body, for every crate. The mir_body node of a fn (level part, in code_nodes) stays: its payload, the block fingerprints, is derived from those rows. The driver writes no mir_block node and no CFG_SUCC edge. A reader that wants the blocks derives them from the same rows with mimir_schema::code_mir::legacy and gets the bytes schema 7 wrote.

Positions live apart from identity: an edge row has no line numbers; its sites are code_spans rows with target = "edge" and key = from ␞ to ␞ KIND (mimir_schema::edge_key, separator U+001E). Moving code changes spans, never an edge.

For the whole path from the compiler to the lens matrices, with the rustc queries, the MIR payloads, the loaders and the semirings, see compiler-to-graph.md.

4. Ids: SCIP monikers, no SCIP tooling

Every node id is a SCIP symbol string in the shape rust-analyzer scip emits:

rust-analyzer cargo <package> <version> <descriptors>

The one formatter is crates/mimir-schema/src/moniker.rs. Nothing else assembles that string, so two producers that name the same definition write the same bytes.

A non-lib target (bin, test, bench, example) gets one leading Meta descriptor naming the target, because rust-analyzer gives those the lib's ids. Impl scopes are rust-analyzer's: impl#[Self]new()., impl#[Dog][Greet]greet()..

What this is not: there is no index.scip, no scip-* indexer and no SCIP protobuf anywhere in the path. Only the id grammar is SCIP's. Readers that need the package of a callee parse it out of the moniker (the dwarves graph-package join does this in crates/mimir-reach/src/lens_graph.rs moniker_package).

5. Ingest into the lake

The nornir server's code room (src/code_room/) is the only writer of the lake.

5.1 The consumer loop

src/code_room/consumer_loop.rs, crates/mimir-ingest/src/consumer.rs. For a member:

1. Its own clone under <builds>/mimir_v2/clones/<member> (git::sync_clone); the server's registry clone is not touched (src/code_room/mod.rs).
2. Checkout of the next commit after the member's offset (checkout::checkout_changed).
3. cargo-mimir-dump on that tree (section 3.1), on one of three roads:

1. read_out_dir reads the unit files back (crates/mimir-ingest/src/driver.rs): every <stem>.stats.json unit with its lossless files where it has them, and every program's <stem>.exec.json unit, under the owner package@version:bin:name+exec.
2. Delta against the open set: rows the new dump has and the open set lacks become + events, rows the open set has and the dump lacks become - events.
3. One skade atomic release: the event tables, the consumer offset and the ledger rows (and the universe manifest when it moved) advance together or not at all (crates/mimir-ingest/src/log.rs commit, under one commit lock).

5.2 Where and how it is stored

The tables (details in tables.md):

Provenance (source_only / binary_only / both) is derived at read time by joining the source and binary streams on the moniker; it is not stored as a fact of the room.

The lossless tables are event tables of the source stream like the graph's: the same delta, the same atomic release, the same ring. A row is keyed by its identity (def, inst, id, made stable) and re-emitted when its content changes; moved code changes no content. They hold no mir_block row and no CFG_SUCC edge. They are read on demand only (crates/mimir-ingest/src/lossless.rs): EventLog::code_mir, code_instances and code_data give the open rows of one repo, or of one crate of the ring, bounded by a byte cap. A lake stamped before schema 8 is wiped at open and read again from git (tables.md, section 2).

6. mimir-room: the in-memory snapshot

crates/mimir-room/src/lib.rs, snapshot.rs, scope.rs.

A Scope is a map repo → seq plus universe: bool. Folding the log at a scope gives an immutable Snapshot:

A room holds a fn's mir_body node and nothing of the lossless tables: no fold reads them, so they cost a room no memory and no load time. The one lens that needs the MIR blocks (the clone lens) reads them beside the room, for one repo and under a byte cap: MirSide::load calls EventLog::mir_block_payloads, which derives the schema-7 block payloads from the repo's code_mir rows (crates/mimir-room/src/mir.rs).

Room = Arc<Snapshot>. A built snapshot is cached per process, keyed by the log identity (the event tables' UUIDs) and the resolved scope; a new commit to any event table moves the log's stamp and drops that log's entries.

The graph engine is crates/mimir-graph: a small safe layer over SuiteSparse:GraphBLAS through crates/graphblas-sys. It imports boolean adjacency from CSR or from Arrow (from_ord, to_ord) batches and provides the semiring operations the lenses use: reachability forward and reverse with many seeds, fan-in/fan-out, fewest hops (MIN_PLUS), worst value on a path (MAX_SECOND), scope masks, A·Aᵀ similarity, connected components, diff. Every handle is RAII (GrB_free on drop); the library is initialised once per process.

The typed query API (no query language):

Every answer carries Evidence: which tables fed the room, which edge kinds the query walked, and what is missing. With the universe off, walks stop at the reference stubs of dependency and std items and the evidence says so (query.rs). A SYSCALLS walk says MISSING until the universe is folded in: the boundary producer writes its edges into the universe (section 10).

7. Readers: surfaces, the code room door, and the panes

src/code_room/reader/wired.rs dispatches a surface name to a function over a Reach (the room plus the workspace it was read from). The surfaces on main:

Knowledge.SymbolLookup  Knowledge.DefinedIn   Knowledge.Callers    Knowledge.Callees
Knowledge.CallPath      Knowledge.FeatureGates Knowledge.Context   Knowledge.Map
Knowledge.Symbols       Knowledge.Source      Knowledge.TestReach
Viz.Knowledge  Viz.Architecture  Viz.CrateGraph  Viz.CallGraph  Viz.FunctionTier
Viz.CodeMap    Viz.Binary        Viz.TestMatrix.inventory
Test.List      Test.Affected     Arch.History    Mimir.SqlTemplate   Mimir.SystemBoundary

Two of them are what the code room panes draw:

Wire: gRPC CodeRoom.Answer on the nornir server (loopback :7878 by default, --addr / NORNIR_SERVER_ADDR; src/server_grpc.rs). The dwarves forge proxies the same service on its own port (:7900, dwarves crates/dwarves/src/warehouse_proxy.rs CodeRoomProxy), so one URL gives a client the forge and the room.

Who asks: dwarves-gui's Coderoom tab (dwarves-gui/src/coderoom.rs) mounts facett's shared CoderoomPane (facett-coderoom-view) behind a CoderoomDataSeam that asks Viz.CrateGraph and Coderoom.SnapshotDiff; the graph source pane (dwarves-gui/src/graph_source.rs) asks Knowledge.Source for the node a person clicked. Neither reads a table or a file; the seam is dwarves-gui/src/room.rs (ForgeRoom, CodeRoom.Answer through the forge).

8. Lenses

Start with the user guide, lens-guide.md: install, dump, run, the target architecture, the reports, every lens in one table. The reference manual is lenses.md (https://www.gunnar.rs/nordisk/nornir/tree/docs/v2/mimir/lenses.md). It covers the nornir lenses behind src/mimir_lens/mod.rs answer(lens, reach, params) and the graph lenses of crates/mimir-lens/ over a Mímir v2 dump (doors dwarves lens … and cargo lens …), with inputs, algorithms, outputs, tests and worked examples.

9. What is NOT there

10. The system boundary and the possible system calls

The kernel tooling as a whole (Build Thing, Heimdall tooling, Mímir, and the files between them) is described in heimdall docs/v2/kernel-tooling.md.

10.1 The boundary in the universe

crates/mimir-ingest/src/boundary/ writes the system boundary into the dependency universe, once per toolchain and target, after a consume (src/code_room/consumer_loop.rs ensure_boundary → boundary::ensure):

The syscall tables:

The libc map names x86_64's calls. aarch64 has no legacy calls (open, stat, fork, pipe …): boundary::calls_on(arch, name) turns each one into the call(s) the C library makes there instead (boundary::GENERIC_STAND_INS: open → openat, rename → renameat, renameat2, …), so no mapped call is dropped on aarch64. A legacy call with no stand-in (iopl, modify_ldt, the x86 thread-area calls) has no edge.

10.2 The possible system calls (the max)

crates/mimir-room/src/possible.rs, Room::possible_syscalls(entries, opts), and the CLI verb:

mimir-room <lake> possible [ENTRY…] [--bin NAME] [--target TRIPLE] [--binary NAME] [--out FILE] [--runtime off] [--repo NAME]…

ENTRY is a moniker or a readable path of a fn; --bin NAME takes the main of that bin target (moniker ending bin=NAME:main(). , - read as _: the crate name). The room is opened with the universe folded in. The verb refuses to write a file when an entry is unknown or the room holds no syscall node of the target's architecture.

With --bin the program is a linked executable (PossibleOpts::runtime, off with --runtime off): std's rt::lang_start and rt::lang_start_internal are entries too when the room holds them, and the calls the C runtime's start-up makes before main are possible, with the path <C runtime start-up, before main> (mimir_ingest::boundary::libc_map::startup: musl on x86_64 arch_prctl, set_tid_address; glibc adds set_robust_list and, uncertain, rseq, prlimit64, brk, mmap, mprotect, readlinkat, getrandom). A source graph never sees that code.

The walk is breadth-first over CALLS and SYSCALLS from the entries; a syscall node is an end (the walk does not go on into kernel functions). It runs twice: first over resolved edges only (what it reaches is direct), then from everything reached, widened at every unresolved site:

Closure bodies are part of the fn that defines them (walk.rs BodyVisitor), so their calls are reached with that fn.

The output is the Graph JSON that dwarves-tune-graph's read_graph reads, with extra fields its reader ignores:

The evidence (stderr of the verb) says MISSING when the universe is off, when no syscall table exists for the architecture, and when a reached C function is unmapped.

The binary walk and the source-graph stubs inside dwarves-tune-graph (walk.rs, srcgraph/) stay the owner's and are not used here.

10.3 Dependency sources from Holger

The walk covers third-party crates because the room holds them: the driver compiles and dumps every crate a program links (RUSTC_WRAPPER, §3.1). Their exact sources come from a registry. With MIMIR_CRATE_MIRROR set to a cargo sparse index that mirrors crates.io (Holger's pull-through sparse+<holger>/production/crates-io/, or the dev pair's sparse+http://127.0.0.1:18444/cache/index/), they come from that mirror and never from the network (crates/mimir-ingest/src/crate_mirror.rs):

Before the dump, cargo fetch --locked gets every Cargo.lock entry from the mirror, so a crate the mirror cannot serve fails there, by name. Holger is read through the registry door it already serves (the sparse index and the download). Cargo checks each .crate against the lockfile's checksum. A repo's own .cargo/config.toml (a vendored source) still wins. The code room's dumpers (src/code_room/consumer_loop.rs) and the consume example take the mirror from the environment.

Proof (2026-11-04, Holger's scan worker as in §10.2, Holger's dev pair with the rust-remote download fix): 758 registry crates came from Holger's /cache and none from crates.io; the max is 118 calls (77 direct), the scan probe's 23 and the job probe's 30 measured calls are all inside it, 0 gaps; 25 of the 118 are made in a registry crate Holger served (libc 0.2.191, rustix: io_uring_setup, landlock_create_ruleset, epoll_*, futex, mmap, …).

11. Consumers outside nornir

Next (planned, not built)

• A second producer: a universal SCIP import of any scip-* indexer's index.scip into the same code_nodes / code_edges tables, same ids.
• A viewer trait in facett (crates, symbols, calls, span, text) with a znippy reader and a CodeRoom.Answer reader behind it.