nordisk/nornir

Mímir v2 — land (integration branch mimir-v2-land)

2026-10-13. Spec: workspace_nordisk/PLAN-MIMIR-V2-2026-10-12.md. Inventory of the removal: .nornir/mimir-v2-inventory.md (§8–§10 = what answers not_wired today).

This branch is the first tree where the old roads are gone AND the new engine is present. Nothing is wired yet: every surface in §6 still answers not_wired. No merge to main (plan §7: deletion + rebuild land as one).

Merge base c18e6d1 (nornir main).

1. Merge: conflicts and resolutions

git merge origin/mimir-v2-room on mimir-v2: 2 conflicted files, 3 hunks. Outside the new crate directories the room branch changed only Cargo.toml and src/coderoom.rs.

Auto-merged in src/coderoom.rs, and KEPT (the part of the room's change that touches surviving code): scip_descriptor → mimir_schema::moniker::descriptor_for_kind, scip_moniker → mimir_schema::moniker::moniker, the ItemExtent::scope doc (impl#[B]), and root Cargo.toml's mimir-schema dependency. One writer of moniker bytes (LAW 5).

Follow-ups of the resolution, in the merge commit:

• escape_scip_name (the room made it pub(crate) for walk_extents) has NO caller left after hunk 1 → removed, not kept unused. scip_descriptor is private again.
• one stale comment line (Type#method(). → impl#[Type]method().).

Check that nothing deleted came back: the 62 paths git diff --diff-filter=D c18e6d1 mimir-v2 lists — none exists in the merged tree.

2. Guard (tests/no_second_parser.rs)

The guard iterates [workspace].members, so the six new member crates were covered the moment they became members. Made explicit and widened:

Not forbidden, noted: graphblas-sys has an OPTIONAL build-dependency bindgen (feature regenerate, off by default; bindings are checked in). bindgen parses C headers, not Rust; it pulls syn transitively the way any proc-macro crate does (plan §7.1: not ours).

Result and red-proof: §4.

3. What is NOT in this branch

• No wiring (§6 is the map, no code).
• mimir-driver is not built by anything in the workspace; it is built by hand on its nightly (cd crates/mimir-driver && cargo build), see lane A in §6.
• The root crate depends on mimir-schema only. It does not link GraphBLAS yet.

4. Compile and test matrix

Box: dwarf (12 threads, load 25–95 during the runs — shared), stable rustc 1.98.1, siblings from ~/d-scip, Cargo.lock seeded from the room worktree, --offline. Logs: ~/d-scip/tmp/land-logs/. Removal baseline: the mimir-v2-L2 row in nornir-workspace.toml and the removal lane's logs (~/d-scip/tmp/mimirv2/).

New crates (cargo test -p … --no-fail-fast, one run, 466 s):

Guard red-proof (the compiled guard binary re-run against the edited tree, each edit reverted, then green again — log guard-red.log):

Not run here: --all-features (red on main and on the removal lane: gpu.rs wgpu API), and the root crate's integration tests (the removal lane runs those against its own head; this merge changes none of them except the guard).

5. libgraphblas-dev

graphblas-sys links the system SuiteSparse:GraphBLAS dynamically (cargo:rustc-link-lib=dylib=graphblas). Its build.rs only WARNS when the header is missing, so nothing fails until a LINK happens: cargo check stays green on a box without the library; cargo build / cargo test of anything that links mimir-graph / mimir-room fails at link time (-lgraphblas).

Today only --workspace (or -p) commands reach those crates, because the root crate does not depend on them yet.

After lane A/B make the root crate depend on mimir-room, every gate that LINKS the root crate trips too (cargo test -p nornir … lines in CI, xtask feature-matrix builds, tests/atom_linkage.rs, the solo tests), and nornir-server / dwarves-full gain a RUNTIME dependency on libgraphblas.so.7 (+ libgomp).

Proposed fix (not implemented): a build requirement, not a feature flag — plus one narrow feature for linkage.

1. GraphBLAS is THE engine (plan §4), not an optional extra. A flag that lets the workspace build without it creates a second "engine absent" state on every surface and a new matrix dimension — the shape §7 forbids. And a root-crate feature would not help the gates above anyway: they say --workspace, which builds every member regardless of root features.
2. So: add libgraphblas-dev where builds link — .woodpecker.yml apt list (check the image's package is 7.x: graphblas-sys refuses another major), the nordisk-cargo-toolchain image / cargo-test.Containerfile (edda nornir-build-thing), README build prerequisites — and libgraphblas7 in every runtime image / appliance compose that ships nornir-server or dwarves-full (ops kits: both gunnar and holger).
3. One preflight line instead of a linker wall: the gates that link (run_cargo_test, the build gate) probe for the library first and fail with SuiteSparse:GraphBLAS 7.x not found — apt-get install libgraphblas-dev.
4. The ONE feature gate that buys something: the root crate's dependency on mimir-room is optional (code-room), ON in server / full-server / solo, OFF in the thin atoms (nornir-cli thin, nornir-ui, wasm). Thin clients ask the server over gRPC and must not link the engine (tests/atom_linkage.rs already audits that nornir-ui carries no server symbols). That gates LINKAGE of thin atoms; it is not a way to run a server without the engine.
5. Take regenerate out of cargo features (an env var read by build.rs, or an xtask verb) so --all-features never runs bindgen.

Owner decision inside this: dynamic system library (above) vs a vendored static build of GraphBLAS for self-contained sold media (cmake, minutes of build, no package on the target).

6. Wiring map (for the next lanes — no code here)

6.1 The handle every lane uses

One log per workspace, one consumer writing it, rooms opened from it:

<registry_root>/<ws>/git/<member>/              server's clone (monitor::fetch_workspace) — KEPT
<registry_root>/<ws>/builds/mimir_v2/lake/      skade lake, namespace mimir_v2 (EventLog) — NEW
<registry_root>/<ws>/builds/mimir_v2/clones/<member>/   the consumer's own clone (RepoSpec.worktree)
<registry_root>/<ws>/builds/mimir_v2/target/<member>/   nightly target dir (DriverDumper.target_dir)
<registry_root>/<ws>/builds/mimir_v2/out/<member>/      driver IPC dumps (DriverDumper.out_root)
<registry_root>/<ws>/builds/mimir_v2/open-set.bin       open-set cache (Consumer::boot)

The lake is its OWN skade catalog, not builds/catalog.redb: redb is single-writer and IcebergWarehouse holds that file; plan §7.5 wants a new namespace nobody else reads. (Decision D1.)

6.2 Where the deleted poll loop + populate are replaced (lane A)

Registry → repos → offsets:

Registry::list() → Workspace{name, mode, poll, members: Vec<MemberState{name, remote, git_ref, last_seen_sha}>}
  → RepoSpec { name:  member.name,
               source: <root>/<ws>/git/<name>      (local path; the fetch keeps auth + HTTPS in ONE place)
               branch: member.git_ref or "main",
               worktree: <root>/<ws>/builds/mimir_v2/clones/<name>,
               initial_offset: D2 }
  → offsets live in mimir_v2.consumer_offsets ONLY (never copied into the registry)

Non-graph feeds that rode the deleted populate and now have NO trigger (P7): they need a side-step on consume, or their lenses stay blind although nothing about them changed — git_heat_facts (knowledge::git_heat::scan_repo, today only release::pipeline::run_pipeline writes it), the sbom / vuln scan, the dep-graph snapshot, the tantivy index, embeddings (monitor::embed_workspace is kept, untriggered).

6.3 What the room API lacks

Read side (crates/mimir-room):

Producer / schema gaps (a surface stays not_wired until its row here closes; none is a room API question):

Write side (crates/mimir-ingest):

6.4 Lane B — the lenses (src/mimir_lens)

Mimir.* RPCs, the MCP mimir_* tools and nornir mimir analyze all reach the same 18 lenses through mimir_lens::answer (+ subclone, MCP only). Wired on mimir-v2-wire-b (the two vector lenses on mimir-v2-wire-j; only data_reach still answers not_wired): the scoring / report halves are kept, their inputs are typed facts read from the code room by src/mimir_lens/room_feed (and the warehouse for the feeds that are not a graph). Every answer is the lens envelope with lens_meta.state = complete | partial | empty_fact | empty_gap, or a refusal state (not_wired, not_consumed, no_room, lake_unavailable, not_linked).

| data_reach | not_wired | sensitivity_of, ranking | — | P3 TOUCHES_TABLE has no producer | | similar_code, semantic_clones | live (lane J, §10) | analyze_duplicates / analyze_semantic_clones whole: the one predicate shares_one_chunk_vector, the trivial-family suppression, the precision filters | one vector per function node (vector::fnvec: node id → content hash → vector); signature from the PARAM nodes, callee set from CALLS, impl header from the block + IMPLEMENTS | no vector → vectors_missing / no_embedder (no count at all); N of M without one, or vectors of an older seq than the room → partial with the numbers; P1 (no return type in the signature); P6; the dependency ring's vectors are written but not compared |

The 6 mimir_calibrate tests deleted with the syn corpus are back (mimir_calibrate::tests::on_a_room_corpus): the corpus is lifted out of a code room (room_feed::corpus::from_room — item spans, parameter nodes, CALLS edges) on a generated tree; real_lake_calibration_corpus (ignored) runs the same oracle on a real lake. room_tests::real_lake_every_lens (ignored) asks every lens against a real lake.

6.5 Lane C — the other readers

6.6 Three lanes, disjoint files

Proposed first step, done by lane A ALONE before B and C branch — the seam commit. It adds no behaviour:

1. root Cargo.toml: optional deps mimir-ingest, nornir-mimir-room behind code-room (§5 point 4); src/lib.rs: pub mod code_room;.
2. src/code_room/mod.rs: the per-workspace handle (lake path, shared log handle, room(scope)), built in open_workspace_ctx, held as a new WorkspaceCtx field (WorkspaceCtx.mimir is already the dep graph — pick another name).
3. Two dispatch points that today return exactly what the stubs return: mimir_lens::answer(lens, handle, params) -> Value (file owned by B) and code_room::reader::* (file owned by C). Every not_wired site in server_grpc.rs, mcp_stdio.rs and bin/nornir.rs is re-pointed at them ONCE.

After the seam, B and C edit only their own modules; the three shared files are touched again only by A.

Unavoidable overlaps and the split:

Each lane's hunks in the three big files sit in different impl blocks / functions, hundreds of lines apart, so they merge textually even without the seam; the seam makes them not overlap at all. The real contention is crates/mimir-room: B needs R1 / R2 / R4 / R5 / R8 that C owns. Order: A's seam + R13 first; then C lands R1–R8 as its first commits (API only, with tests) and B rebases on them; B's room work (R9–R12) is in files C does not open (ops.rs in mimir-graph, a new symbols.rs in mimir-room).

What can go live without any missing API, as the proof each lane starts with: A — consume one member + Repopulate report; B — anomaly (needs only P7's git-heat trigger); C — Mimir.RepoOverview knowledge block (Room::stats()).

7. Decisions needed

8. Lane A — what landed (mimir-v2-wire-a, 2026-10-13)

Decisions as implemented: D1 the lake is <ws>/builds/mimir_v2/lake, its own catalog. D2 TIP, backfill by nornir workspace populate <ws> --from <sha|date> (inline; the wire has no field). D6 the driver is located by settings and probed; NORNIR_MIMIR_DRIVER_HOST is the wrapper line of the toolchain image. A lake of an OLDER mimir-schema that only lacks nullable columns is evolved in place on open (lane E, §9); any other difference is refused (LakeSchemaMismatch), and the message names the directory to remove.

Follow-ups outside this repo (edda)

Known limits

• The feeds are keyed on the consumer's offsets: a box without the driver runs no feed either (git heat included), and says driver_not_available.
• A consume and its feeds hold the workspace gate; embeddings on a CPU embedder is minutes per member (measured in a DEBUG build: 30 chunks, 415 s). A fetch waits for it, readers do not.
• ingest_runs.ms does not include the commit phase (the row is built before the release); the job row's phase_ms.commit does.
• The toolchain's std dump (StdDumper) and the binary layer are not run by the server (source stream only).
• The root crate cannot be published from this branch: it depends on the unpublished mimir crates (D7), by design until landing.

9. Lane E — closing items (mimir-v2-wire-e, 2026-10-13)

What an existing lake sees

• A schema-2 lake under this build: opens; code_node_events and bin_node_events gain ignored, doc_hidden; every event table is stamped mimir.schema = 3. Nothing is written to the event tables until the member's next commit. That consume dumps every unit (mirror discarded) and writes + events for the nodes the producer now has a value for — every defined item (doc_hidden is false or true wherever has_doc is set) — and nothing for edges or spans. Measured with the real driver on the driver fixture, a lake written by the schema-2 build (308 node, 408 edge, 317 span events of the repo): opened by this build → code_node_events 24 → 26 columns, the open-set cache still valid, 0 events; the next commit (one new fn in ONE crate) → all 7 units dumped, 112 node + events (110 identities that existed and now carry a value, 2 for the new fn under its two owners), 2 edge events, 4 / 2 span events. 198 of the 308 open node identities — stubs, params, generic params, features — were not written again.
• The target ids: at the member's next consume every node of a non-lib target (bin, integration test, bench, example) closes under its old id and opens under the new one, with its edges and spans. Measured on the same lake: a comment-only commit in the crate that has the bin bctl → its 2 nodes under their 2 owners (the bin and its harness) move: 4 - / 4 + node events, 21 / 21 edge events, 16 / 16 span events. No migration.
• The dependency ring (@universe) is dumped once per crate version: its rows keep null in the new columns.

Known limits (lane E)

• skade's all-core raw decode declines a table that has more than one schema (skade/src/read.rs: metadata.schemas_iter().len() > 1): an evolved lake is read through the Iceberg engine's scan. Same rows, slower. The fix is in skade (a field-id remap in the raw path).
• Under a version bump the doctor's blast is exact for REMOVED symbols; a change of a surviving item's type alone is not told apart (every id of the crate moves with its version, and the dry-run delta carries keys, not columns).
• doc_hidden is the item's own attribute; undoc_api walks up to find a hidden module, other readers must do the same.
• The semantic blast is a lower bound like every reachability answer (P6), and it reads the sites of CONSUMED repos only — which is why a dependent that is not consumed refuses instead of answering.

10. Lane H — the wire door and the panes' readers (mimir-v2-wire-h, 2026-10-13)

A thin client (dwarves) had no way to ask the reader: only five Knowledge.* verbs and three Viz.* verbs were on the wire, typed or as JSON, and a refusal was a gRPC status (the state lost in a code). Lane H adds ONE door and the surfaces the dwarves panes draw.

Unchanged on purpose: Coderoom.Ingest / Compat / WhatBreaks (P5), table lineage (P3), the vector lenses, binary call edges (P4) — not_wired through the door as through every other front door.

10. Lane K — which code touches which table (mimir-v2-wire-k, 2026-10-13)

P3 closes: TOUCHES_TABLE has ONE producer, the rustc driver (crates/mimir-driver/src/walk/tables.rs — its header is the definition of "touching a table").

Measured on real code (the nornir lib, default features)

The real driver (release build) over -p nornir --lib, tree = 46f10dd + this lane's work as it stood before the last lens / reader edits (the warehouse sources are those of the head):

• 191 TOUCHES_TABLE edges, 213 sites, 67 tables by name; 105 write, 79 read, 7 both; 5 sites in 5 fns name their table at runtime (the gRPC Warehouse.Scan / ScanVectors handlers, the warehouse browser pane, compact_all, drop_v1_tables — each checked in the source: runtime names all five).
• The pass looked at 154 bodies (227 looks) of the lib's 7 049 fns and methods, kept 44 summaries, read 1 185 dependency sidecars: 0.16 s of a 1.72 s walk.
• Fed through the consumer into a scratch lake (the driver's Arrow files, one commit): 165 567 node + edge rows and 126 341 span rows in 3.8 s. The room opens in 4.9 s cold (36 898 nodes); data_reach answers in 0.2 s (186 (table, fn) rows, 67 tables, the 5 runtime-named sites listed); the lineage of bench_runs = 6 sites, 255 fns reach them; the board has 67 Table nodes and 153 reads / writes edges; 87 gRPC handlers get a nearest table (114 pairs, 1.3 s for the whole board).
• NOT run: the consumer with the real DriverDumper over the whole workspace and the dependency ring (started, stopped after a minute when the owner stopped driver consumes on dwarf, 2026-10-14 01:25 CEST).

Against the v1 scan (nornir main's access_scan.rs, unchanged, run over the tree of 46f10dd: 441 site rows, 22 tables by name + <dynamic>): v1 marks the CALLER of a named accessor, this marks the fn where the name meets the read / write, so the same fact sits one CALLS hop lower and the reachers hold v1's fn. v1 never saw 48 of the 67 tables (its closed list of accessor names did not have them, and it read string literals only — scan_preview(TABLE, …) was <dynamic>); it counted table_names() as an access (most of its 20 <dynamic> rows); about two thirds of its rows (290, counted by path and module name) are #[cfg(test)] modules, tests/ and examples/, which here belong to the harness / test / example units — not in this lib-only run. v1 is right where this is blind: its 5 release_events rows in the lib are calls into edda's release ledger (below).

Not covered (the readers say so; none of it is silently empty)

• Calls through &dyn Warehouse made inside edda (nornir-release-ledger: release_events, release_changes; nornir-knowledge-view): those crates are compiled before any impl of the trait exists, so there is nothing to derive. The fix is ten lines in edda/crates/nornir-warehouse-trait (#[cfg_attr(mimir, mimir::table(write, table))] on append_arrow / append_arrow_async / ensure_table / ensure_columns, read on the scan_* / describe_columns methods, and cfg(mimir) declared in its [lints.rust]). korp-collectors' WarehouseSink is the same case. Inside nornir a &dyn Warehouse call resolves through nornir's own impl.
• SQL: a query string is not read for table names, and a table DataFusion scans through a registered provider is no edge.
• A table reached through a closure parameter or a fn pointer (P6's shape).
• The namespace / catalog: the id is the table's name. The warehouse, the Mímir v2 lake and modgunn's own store share one name space of ids.
• A loop over a list of name PAIRS attributes each site to every name of the list (the right tables, each site counted for all of them).
• Creating a table (ensure_layout, create-if-missing) counts as a WRITE of it: on nornir that one fn writes 66 tables and, being reached by everything that opens a warehouse, heads data_reach's ranking. 24 of the 67 tables are touched by it alone.
• A handler's nearest table is nearest in the CALL GRAPH: a handler that goes through a dispatcher fn ends at whatever the dispatcher can reach.
• After a change to what a summary says (TABLE_PASS), dependencies cargo keeps are not summarised until they are compiled again: every dump names them (table_sidecars_stale). For the schema-3 → 4 step itself the dumper discards its mirror and cleans + compiles again every unit that has no dump (read from mimir-ingest driver.rs, not run on an existing lake here).

10. Lane I — the server runs all of it (mimir-v2-wire-i, 2026-10-13)

Measured (one box, load 25–35 of 12 cores, release build of 82827fc)

Through the server door, workspace of one member (znippy-zoomies, 608 crates):

• std: 25 crates, 937,960 rows, 51 s (90 s on a second server at the same time).
• tip: +3,242,285 events in 299 s (dump 156 s, delta 121 s, commit 22 s).
• binary feed with no artifact: no_artifact and the directory, 8 ms. A hand build (cargo build --release -p lgz --target-dir …/<sha>) moved into place: picked up by the next sweep, 36 s, +3,353 rows.
• call path thp_enable_for_process → open64: 9 hops through std to libc's extern item; 110 s the first time (the room with the ring is built), 3.8 s after. → prctl (libc, one hop): 0.2 s.
• backfill from 20 commits back: the request returned in 0.6 s; 21 turns (20 + one commit pushed during the walk, consumed last) in 173 s. 18 of the past commits do not load against TODAY's sibling (nornir-build-thing = "^0.1.12", two with merge markers in Cargo.lock): each is recorded error with that reason in 0.5–4.7 s and the walk goes on.
• embeddings (tract CPU, 100 chunks, budget off): a commit pushed 20–28 s into the embed began its consume 848.0 s after the push before, 1.3 s after; Workspaces.Fetch returned after 864.0 s before, 17.5 s after.
• memory of the server: 57 MB idle; 4.4 GB at the peak of the tip consume (the delta step), 0.33 GB 40 s later; 4.6 GB while the room with the ring was built and the embedder ran; 0.7 GB during the backfill walk.

Known limits (lane I)

• A past commit is checked against the siblings as they are today. When it does not load it is recorded error; its rows are not rebuilt.
• The artifact directory is filled by whoever builds; a descriptor key for another place needs a field on edda's RepoSpec.
• A member removed by a re-register is said; its rows stay in the lake.
• A sibling that is not a member of the workspace must be beside the clones by hand (git/ and builds/mimir_v2/clones/).
• The sweep can queue one more backfill unit just as the walk ends: it finds nothing open and leaves a done job row that says so.
• Not run: the image against the installed driver (same ids, cold / warm), a restart in the middle of a walk on a real server, edda itself, the real-driver tests added here (real::a_member_and_the_published_copy_…, real::a_member_that_is_a_path_dependency_…).

10. Lane J — the two vector lenses (mimir-v2-wire-j, 2026-10-13)

similar_code and semantic_clones answer from the code room again, on one vector per function. Design note: .nornir/vector.md § "One vector per function node".

Measured (znippy-zoomies, dwarf, 2026-10-13)

Release build --no-default-features --features server,vector,embed-tract,mcp, scratch workspace, tract CPU, the box shared with three other lanes.

Not measured there: the ring's price and its embed, the lenses through the server door after the second commit (through the CLI door they answered complete at seq 2 in 1.2–1.8 s), and the line-window half of the feed (never priced: stopped by hand after the function half). The server was stopped on the owner's order before those. With the default 10-minute embed budget the function half itself is refused on this CPU (priced at 30–100 min): the feed row is error and the lenses say vectors_missing with that reason.

The threshold (DupThresholds::cosine = 0.92, kept)

All pairs of the 3 546 vectors (6 285 258; mimir analyze duplicates --calibration): p50 0.204 · p90 0.381 · p99 0.561 · p99.9 0.750 · p99.99 0.931. At 0.92: 736 pairs, 211 of identical text; of the 525 others, 400 pass the structural gate and 97 are also in different crates (what the lens reports: 100 clusters with the identical ones).

Hand-checked against the source, cross-crate pairs that pass the structural gate:

0.92 is where the precision breaks, so it stays. What it does NOT find: real copies with systematic renames score far lower — ljar::main ≈ lzip::main (89 lines, 99 % the same text) 0.856, batch::fill_buffer (62 lines, 96 %) 0.813, batch::decompress_jars ≈ decompress_zips 0.866 — in a band (0.80 – 0.92) where 77 of the 91 pairs of ten lines or more share less than 85 % of their text. No cosine separates those; they are the sub-clone (text) lens's findings. Of the reference pair ljar.rs ≈ lzip.rs: decode_entry and physical_cores (identical) and local_data_offset (0.928) are reported, main is not.

Tests and red-proofs

cargo test --lib --no-default-features --features server,vector,viz (vector_tests, fnvec, room_feed, mimir_analyze, mimir_clones, mimir_calibrate, code_room, vector::, iceberg_schema, consumer_door_tests): 232 passed, 0 failed, 4 ignored. Guards no_second_parser 7/7, code_room_seam 5/5. cargo check green for server,vector,embed-tract,mcp, full-server, viz (no code-room).

NOT run: the default-feature cargo test --lib and cargo check (the box was needed elsewhere), and nothing on a GPU. --features vector-gpu did not compile on this tree (gpu.rs: wgpu 30 made get_mapped_range a Result); fixed in one line, compiled, its GPU tests skip on dwarf (no adapter).

What is not in this lane

• The ring's vectors are written and linked; no lens compares against them yet.
• The lenses' candidate scan is the exact all-pairs cosine on the CPU (VectorIndex::pairs_above). The WebGPU kNN (vector/gpu.rs, top-8, int8) is untouched and not on the lens path: a threshold scan needs every pair above the bar.
• Free-text search still reads line windows: Rust code is embedded twice (once per function, once per window).
• A driver that runs inside a toolchain image keeps its cargo registry there: the ring job then finds no source and writes skipped rows naming the crates.