Mímir v2 lenses
This manual describes every lens that exists on main today, in two repos.
It is written from the code. Each claim names its file and line. To run the
lenses, start with the user guide lens-guide.md; this file
is the reference behind it.
- nornir lenses run over the code room and the warehouse. Their entry is
answer(lens, reach, params)in nornirsrc/mimir_lens/mod.rs:198. - graph lenses run offline over a Mímir v2 dump directory. The code is the
nornir crate
nornir-mimir-lens(crates/mimir-lens/, moved out of dwarves on 2026-11-05, see section 4 "Layout"); the doors aredwarves lens <name>/cargo lens <name>. The foldsdwarves mimir zombie|risk|bloatanddwarves coverageread the same dump through the same crate;dwarves dupesstays in dwarves.
Path conventions. A bare path such as src/mimir_lens/trust.rs:141 is in nornir;
the graph lenses and folds are in nornir too, under crates/mimir-lens/ (for
example crates/mimir-lens/src/lenses/twins.rs:64). A path that starts with dw:
is in dwarves, under crates/dwarves/: the CLI door only. Line numbers are for
nornir main; the files of crates/mimir-lens/ moved verbatim from dwarves main
3b3b384 (2026-11-04), so their numbers did not change (the tests lost their CLI
cases to dwarves tests/lens_cli.rs).
Related documents: README.md (the code graph, the producer, the
room), tables.md (every table and column) and
compiler-to-graph.md (how rustc's facts become the
tables and the matrices every lens reads).
How each lens section reads. Every lens has the same parts:
- Math: the exact formulas the code computes, with every constant. Where the code and its comment or intent differ, the section says so.
- GraphBLAS steps: each product with its semiring as the code names it
(
GxB_ANY_PAIR_BOOLis the structural OR-AND reach, the same as LOR.LAND on booleans), or a plain statement that the step is not GraphBLAS. - Dataflow: an ASCII picture from the tables to the output.
- Worked example: real Holger numbers where the plan or the goldens record them, otherwise a test fixture computed by hand.
- Run, output, limits and tests, with
file:linereferences as extras.
The dataflow of every lens starts at the compiler:
rustc (mimir-driver, after_analysis)
→ code_nodes / code_edges / code_symbols (Arrow IPC per crate unit)
→ dwarves: Graph::from_dump → adjacency lists → grb::pattern matrices → lens products → Report
→ nornir: consumer → lake → mimir-room per-edge-kind matrices → room_feed folds → envelope
compiler-to-graph.md describes each arrow, with the
semirings and one fn traced from source to matrix entries.
Contents:
- Quickstart: run the lenses on your workspace, with Rusternetes as the worked example
- The two families and what they read
- nornir lenses (
answer,LENSES) - Graph lenses (
nornir-mimir-lens, doordwarves lens …), including the study lenses taint, chokepoints, misplaced and dsm (4.9 to 4.12), the package-graph views (4.26, a renderer), the macro lenses (4.27 to 4.34) and fail-open (4.35) - The super lens
verdict - The dwarves folds on the dump (zombie, risk, bloat, coverage, dupes)
- Worked examples on Holger
- Cost
- What is not wired
1. Quickstart: run the lenses on your workspace
This path was run end to end on Rusternetes, on two hosts (section 1.6). It needs Linux x86_64, about 5 GB for the sources, and room for the dumps (5.5 GB and 3.8 GB on Rusternetes).
1.1 System packages
On Ubuntu:
sudo apt-get install -y build-essential clang cmake pkg-config protobuf-compiler \
curl git git-lfs time libgraphblas-dev
| package | why |
|---|---|
libgraphblas-dev | SuiteSparse:GraphBLAS 7.x, linked by the lenses. There is no bundled build. At run time it is libgraphblas7 and libgomp1 |
protobuf-compiler (protoc) | needed by a workspace that runs prost-build, as Rusternetes does. Your own workspace may not need it |
clang, cmake | named by the build files of openzl-sys-rs. A build without them was not tried |
time, git-lfs | time only for measuring. Cloning facett without LFS prints a harmless warning about 13 files |
1.2 Install the dump tool (nightly, nornir only)
mimir-driver is a rustc driver, so it is built against one pinned nightly. It
is its own workspace and needs only the nornir checkout
(crates/mimir-driver/Cargo.toml:12, :31).
rustup toolchain install nightly-2026-10-05 --profile minimal \
--component rustc-dev,llvm-tools-preview,rust-src
git clone ssh://git@git.gunnar.rs:2222/nordisk/nornir
cd nornir/crates/mimir-driver # this directory's rust-toolchain.toml picks the nightly
cargo install --locked --path . # cargo-mimir-dump, mimir-driver, and two helpers
The cd matters: the toolchain comes from that directory. cargo-mimir-dump
looks for mimir-driver beside itself, so keep the two binaries in one
directory (crates/mimir-driver/src/bin/cargo-mimir-dump.rs:196-201). It runs the check under the
pinned nightly by itself, so your workspace keeps its own toolchain.
1.3 Install dwarves (the 15 sibling repos)
From crates.io (2026-11-06): cargo +nightly-2026-10-05 install --locked cargo-mimir-dump
and cargo install --locked dwarves-lens give the dump tool and the lens door with no
sibling repos; see lens-guide.md "Quick start". The source build below
is for the full forge.
A build cannot avoid the sibling repos today. crates/dwarves/Cargo.toml
names them by path, and Cargo reads every path dependency, optional ones too. A
lone dwarves clone stops with failed to read ../edda/crates/autonom/Cargo.toml.
crates.io is no way out either: graphblas-sys and nornir-mimir-graph are
publish = false, and the published dwarves has no lenses feature. So there
are two ways:
-
Copy the binaries. Build once on a host that has the siblings, then copy
dwarves,cargo-lens,cargo-mimir-dumpandmimir-driverto the same directory on yourPATH. On a built hostdwarveslinks onlylibgraphblas7,libgomp1, libc, libm and libgcc_s, so the target needs those two packages, plus the nightly of section 1.2 for the driver. Copying was checked withlddonly; no run was made on a third host. -
Build with the siblings. Put these beside dwarves in one directory, one
git cloneat a time. The code host limits parallel transfers.from repos ssh://git@git.gunnar.rs:2222/nordisk/dwarves, nornir, edda, Skidbladnir, facett, holger, skade, znippy, znippy-zoomies, gunnar, monetize ssh://git@git.gunnar.rs:2222/mikronova/draupnir, funnel-core, tunnr, modgunn, openzl-sys-rs That is dwarves and 15 siblings, 4.8 GB in all. Then, from the dwarves checkout:
cargo install --locked --force --root ~/lens-tools --path crates/dwarves \ --no-default-features --features cli,lenses,mimir-fat export PATH=~/lens-tools/bin:$PATH--no-default-featuresmatters.defaultis the full tool with five binaries (11:40 against 6:21 on oden), andlensesandmimir-fatare not in it (dw:Cargo.toml:141,:329-333). The slim line ran every lens and fold of section 1.6.--rootkeeps an existing~/.cargo/bin/dwarvesas it is. Without it,cargo installoverwrites that binary.--lockedneeds the siblings at commits that fit dwarves' lock. Section 1.6 gives a set that was proven together.
1.4 Dump and run
cd <your workspace>
# Dump 1: every crate with its dependencies. The graph lenses read this one.
cargo lens graph-package --local # writes target/dwarves/mimir
# Dump 2: your own crates only. The folds and cfg-sprawl read this one (see 1.5).
cargo mimir-dump --workspace-only --out target/dwarves/mimir-ws \
--target-dir target/dwarves/mimir-target-ws --repo <name>
# Graph lenses. Each finds target/dwarves/mimir by itself.
cargo lens size --local --limit 0
cargo lens twins --local --limit 0
cargo lens cycles --local --limit 0
cargo lens dsm --local --limit 0
cargo lens misplaced --local --limit 0
cargo lens dispatch-erosion --local --src . --limit 0
cargo lens cfg-sprawl --local --mimir-dump target/dwarves/mimir-ws --limit 0
# The gate, then the super lens: one band per crate and module.
cargo lens gate-bypass --local --limit 0 \
--sink 'Store#put()' --sink '[Store]put()' \
--choke 'Gate#check()' --entry 'my-server handlers/'
cargo lens verdict --local --limit 0 \
--sink 'Store#put()' --sink '[Store]put()' \
--choke 'Gate#check()' --entry 'my-server handlers/'
# The folds, on the own-crates dump. <dir> holds only your checkout.
mkdir -p /tmp/empty-warehouse
dwarves mimir --local --wh target/dwarves/mimir-ws --checkouts <dir> --limit 100 zombie
dwarves mimir --local --wh target/dwarves/mimir-ws --checkouts <dir> --limit 100 risk
dwarves coverage --local --mimir-dump target/dwarves/mimir-ws \
--warehouse /tmp/empty-warehouse --checkouts <dir>
What to watch:
-
--localruns here and ignores$DWARVES_SERVERand$NORNIR_SERVER, which would otherwise reroute the verb (dw:src/cli.rs:92-95). -
--limit 0prints every row. Text output stops at 50 rows while the headline counts all rows, so a top list from the default output can mislead. -
Sink patterns: name the trait AND the impls. A pattern is a substring of
<crate> <short moniker>.Store#put()matches the trait's method only. A handler that callsimpl Store for Arc<S>calls a separate node, spelled[Store]put(), and thedynexpansion runs only from trait to impl. On Rusternetes the trait pattern alone gave 14 write paths, one impl pattern 15, and all impls 201. An impl pattern also matches test impls such asCountingStorage. -
gate-bypass reads check-then-write as BYPASS. A path that calls the gate, returns and then writes is not mediated in the lens's model: the write must sit inside the gate's own frame. Servers that authorize inline therefore read BYPASS, and any non-zero gate term makes the unit RED in
verdict. Read a door before you call it a missing check (section 1.6). -
dwarves coveragereads the host's default warehouse under~/.nornirand takes the parent directory as its repo roster. The empty--warehouseand a--checkoutsdirectory holding only your repo keep the answer from depending on the machine. -
An empty dump.
cargo mimir-dumpwith a new--outon a warm--target-direxits 0 and writes nothing, since cargo has nothing to recompile. The next lens then says no dump exists. Give each dump its own target dir, or removetarget/dwarvesand start over. Rerunninggraph-packageon the same out dir is safe. -
-pand--featureson a--workspace-onlydump (Gunnar re-compare row 12). With--workspace-only,-p Xdumps only the named packages, not the whole workspace. With several-p, a package-prefixed feature (--features gunnar-server/metered) silently enables nothing: give the features unprefixed (-p gunnar-server -p gunnar-grpc --features metered). The Gunnar delivery dump was made that way:cargo mimir-dump --workspace-only -p gunnar-server-metered --features metered \ --out target/dwarves/mimir-metered --target-dir target/dwarves/mimir-target-meteredCheck the result before you trust a count: the metered dump must hold
account_room_at()(crates/gunnar-grpc/src/service/accounts.rs:323, behind#[cfg(feature = "metered")]), whichcargo lens fail-openlists. -
One dump is one build. An impl behind a cfg the dump did not compile is invisible, so
single-implanddyn-to-enumjudge that build only. Give the other build with--also-dump DIR(repeatable): the impl sets become the union by label (§4.5, §4.15).
1.5 Why two dumps
The graph lenses keep to the workspace's own code (origin = local) on a full
dump. The folds do not yet, and neither does cfg-sprawl or the crate level of
cycles. On a full dump risk ranks syn and num-bigint-dig first, zombie
names 194 dependency crates, and cfg-sprawl reports 2,250 features in 292
crates instead of 46 in 9. So until that scope is fixed in dwarves, the folds
and cfg-sprawl read a second dump made with --workspace-only.
Neither dump suits everything. A full dump peaks at about 8 GB per lens on
Rusternetes. An own-crates dump costs less (5.4 to 5.8 GB) but loses facts about
dependency types: dispatch-erosion falls from 34 to 23 foreign ladders, and
verdict moves from 2 AMBER / 6 GREEN to 3 AMBER / 5 GREEN crates. The bench in
section 8 used --workspace-only dumps.
1.6 Worked example: Rusternetes
Rusternetes 1f73ec89 (https://github.com/Ignalina/rusternetes), 9 crates. All
16 lenses and folds exit 0 on both hosts, with the same headlines. oden has 32
cores and ran at load 35 to 40 beside other work, with the default dwarves build
unless marked slim. loki has 12 cores and was quiet, at nice 10, with the slim
build. Its build rows had a warm sccache, so they are not cold numbers.
| step | oden | loki |
|---|---|---|
| nightly toolchain | 50 s / 0.56 GB | not recorded |
| dwarves, default features | 11:40 / 1.74 GB | not run |
| dwarves, slim line of section 1.3 | 6:21 / 0.67 GB | 2:12 / 0.63 GB |
| cargo-mimir-dump + mimir-driver | 1:57 / 0.88 GB | 9 s / 0.85 GB |
cargo lens graph-package (full dump, 5.5 GB) | 2:05 / 3.44 GB | 75 s / 3.28 GB |
cargo mimir-dump --workspace-only (3.8 GB) | 1:51 / 3.44 GB | 69 s / 3.27 GB |
| size, cycles, misplaced, single-impl, error-shape, dispatch-erosion | 21–22 s / 8.07 GB | 13 s / 7.68 GB |
| twins, taint, chokepoints | 22–25 s / 8.07 GB | 14 s / 7.68 GB |
| gate-bypass | 36 s / 8.07 GB | 20 s / 7.68 GB |
| verdict | 41 s / 8.07 GB | 22 s / 7.68 GB |
| dsm | 1:41 / 8.07 GB | 63 s / 7.68 GB |
| cfg-sprawl (own crates on loki, full on oden) | 21 s / 8.07 GB | 9 s / 5.35 GB |
| zombie (own crates) | 15 s / 5.64 GB | 9 s / 5.37 GB |
| risk (own crates) | 26 s / 5.80 GB | 15 s / 5.52 GB |
| coverage (own crates) | 14 s / 5.64 GB | 9 s / 5.37 GB |
The clones plus everything above took roughly 15 minutes on loki; that figure is an estimate from log times. Every lens above is mostly the dump load (section 8). A rerun on warm tools and warm dumps takes about 4 minutes.
The findings:
| lens | headline | top rows |
|---|---|---|
| size | 105 fns over 150 lines, 35 over 300, of 2,568 | Kubelet::sync_pod 2,364 lines; ProtoRegistry::new 1,854; build_router 1,771 |
| twins | 210 groups, 977 of 2,545 fns have a twin, 225 DIVERGENT pairs | STRONG groups ×60, ×54 and ×49 (mostly kubectl execute) |
| cycles | 1 module cycle | rusternetes_storage with etcd, memory, rhino, 29 calls |
| dsm | propagation cost 7.2 %, hierarchical | core = those 4 storage modules; per crate: storage 75 %, kubelet 40 %, scheduler 40 % |
| misplaced | Q 0.627 for Leiden against 0.225 for modules; 107 fns, 8 modules to split | 13 fns scheduler plugins → framework |
| dispatch-erosion | 34 foreign ladders, 1,476 string compares, 189 parallel enum pairs | bollard LogOutput 10 ladders; types::Phase 10 |
| single-impl | 8 traits with no impl, 1 with one | all 8 are scheduler framework plugins, for example BindPlugin |
| error-shape | 315 anyhow sites against 9 typed errors | kubectl 190; controller-manager 51; kubelet 40 |
| cfg-sprawl (own crates) | 46 features, 30 never gate anything, 18 gated-out stubs | storage 12 item + 26 body gates |
| taint | 90 tainted sink calls in 22 fns | resync_volumes 14 fs; IptablesManager::flush_rules 10 exec. The default sources look too wide: axum State::deref alone names 28 rows |
| chokepoints | no fn dominates every path; 324 of 1,612 entries reach 18 sinks | best betweenness check_resource_quota 0.001: authorization is per handler |
| gate-bypass | 201 write paths, 161 doors, complete mediation NO | see below |
| verdict | crates 1 RED, 2 AMBER, 6 GREEN; 59 RED modules | api-server RED on the gate term |
| zombie (own crates) | 4 unreached features, 0 severe | aws 8 items; integration-tests 4 |
| risk (own crates) | 398 files; no churn feed, so the key is callers × uncovered | common/resources/pod.rs 8,108; common/types.rs 3,794 |
| coverage (own crates) | 2,611 of 13,318 fns (20 %), 3,610 tests | no bench feed, no line coverage |
Two figures need a caution. risk's real first row is the generated
OUT_DIR/openapi.v2.rs at 43,268, because the fold does not yet treat OUT_DIR/
as generated. verdict says 1,950 of 2,545 fns are reached while coverage says
2,611 of 13,318: the two count different populations, and the difference is not
settled.
The gate, read properly. The run used:
SINKS=(); for m in create update update_raw delete; do
SINKS+=(--sink "Storage#$m()" --sink "[Storage]$m()"); done
cargo lens gate-bypass "${SINKS[@]}" --choke 'Authorizer#authorize()' \
--entry 'rusternetes-api-server handlers/' --local --limit 0
The answer, 161 doors and NO, is not 161 missing checks. 155 of the 161 door fns
call authorize themselves (computed from the CALLS edges; that shows the call
exists, not its order). The other 6 were read by hand or through their callers,
and all sit behind an authorize: priorityclass/patch and service/patch
delegate to generic_patch, and four are helpers. No handler with a write and
no authorize was found, which is not a proof. Rusternetes authorizes inline, then
writes, and the lens counts that as BYPASS (section 1.4). That is also why
verdict shows 59 RED modules.
The versions. This run predates schema 6, so its dump had the 1,024-block cap (compiler-to-graph.md). These commits were proven together:
| repo | commit | repo | commit |
|---|---|---|---|
| dwarves | abbe6fd6 | znippy | d93fc343 |
| nornir | deebfdde | gunnar | dbaaee26 |
| edda | be8ec302 | monetize | 4e230c13 |
| Skidbladnir | 1c6913a5 | draupnir | ee46d026 |
| facett | f4c3dd57 | funnel-core | d3a0f062 |
| holger | aa5f0adc | tunnr | 17cd493b |
| skade | cbb8c9b1 | modgunn | f43f0ec8 |
| znippy-zoomies | 03ff44e8 | openzl-sys-rs | 7e96eb11 |
1.7 What each step does
-
cargo lens graph-packagerunscargo metadata, thencargo-mimir-dump --out <out>/mimir --target-dir <out>/mimir-target --repo <dir>, then writestarget/dwarves/graph-package.znippy(crates/mimir-lens/src/lens_graph.rs:622-791).--mimir-dump DIRreuses an existing dump instead.--mimir-binor$DWARVES_MIMIR_DUMPnames the dump binary. -
cargo lens Xis the same command asdwarves lens X(dw:src/bin/cargo-lens.rs:1-5). -
Every graph lens looks for the dump in the given path, then
<path>/dwarves/mimir, then<path>/target/dwarves/mimir(crates/mimir-lens/src/lenses/mod.rs:60-78). -
Shared flags of every graph lens (
DumpLensArgs, dw:src/cli.rs:553-584):flag default meaning --mimir-dump DIR./target/dwarves/mimirthe dump, or a workspace root holding one --crate NAMEevery crate the dump defines repeatable --include-testsoff count test code too; generated/paths are always left out--include-depsoff report on the dependency crates of a full dump too; by default only the workspace's own code ( origin = local) counts--src DIRnone a checkout, for the text fallbacks --jsonoff one JSON object with every row --limit N50 rows printed; 0 = all. Headlines count every row, whatever the limit --fail-on-findingsoff exit 1 when the lens found anything A missing dump or a lens error exits 2 (dw:
src/cli.rs:586-636). -
The nornir lenses are asked through a front door: the gRPC
Mimir.*service (src/server_grpc.rs:786, one handler per lens from:3322), the MCPmimir_*tools (src/mcp_stdio.rs:1700-2314), or the dwarves CLI throughdwarves::lens(dw:src/lens.rs). See section 3.1.
2. The two families and what they read
| nornir lenses | dwarves graph lenses | |
|---|---|---|
| entry | answer(lens, reach, params) (src/mimir_lens/mod.rs:198) | dwarves lens <name> (enum RepoLens, dw:src/cli.rs:263) |
| reads | the code room (mimir-room, GraphBLAS) built from the consumed dump, plus warehouse feeds (git heat, test and bench runs, SBOM, advisories, Build Thing filings) | one dump directory: *.nodes.arrow, *.edges.arrow, *.symbols.arrow (crates/mimir-lens/src/coverage/v2.rs:44-46) |
| runs | in the nornir server, queued as jera jobs (src/mimir_lens/queued.rs:1-5) | in process, offline, byte-stable output (crates/mimir-lens/src/lenses/mod.rs:26-28) |
| answer | the lens envelope (section 3.2) | a Report with headline, summary, findings, needs, notes (crates/mimir-lens/src/lenses/mod.rs:608-656) |
The only producer of the code graph is mimir-driver, run by cargo mimir-dump
(see README.md §3). Both families read the same node and edge
kinds: CONTAINS, CALLS, IMPLEMENTS, FOR_TYPE, GATED_BY, ENABLES, USES_TYPE,
READS_FIELD, WRITES_FIELD (schema 6), TESTS, EXPANDS, TOUCHES_TABLE, CFG_SUCC,
DEPENDS_ON, SYSCALLS, and the name-free MIR nodes mir_body / mir_block.
READS_FIELD is every mention of a field (read or written); WRITES_FIELD is the
writes (=, op=, &mut, a &mut auto-ref, a struct literal), so a
read-only use is READS_FIELD ∧ ¬WRITES_FIELD
(compiler-to-graph.md §3.1). The dwarves Graph
indexes both (reads_field, writes_field and their reverses).
The honesty rule is the same in both. A fact the input does not carry is named with its producer. It is never printed as zero.
- nornir:
evidence_note,lens_meta.gapsandlens_meta.state(src/mimir_lens/mod.rs:23-28). - dwarves:
needs <fact>: <producer>lines in every output mode (crates/mimir-lens/src/lenses/mod.rs:608-636).
3. nornir lenses
3.1 The entry and the lists
answer(lens, reach, params) (src/mimir_lens/mod.rs:198-219):
subcloneandgroup_twinsgo to their source folds (:201-206).- A name not in
LENSESis refused withinvalid_argumentand the known list (:207-210). - A name in
WAITINGanswersnot_wired.WAITINGis empty today (:185-186). anomalyreads the warehouse only (:215-217).- Every other lens goes to
room_feed::from_room(src/mimir_lens/room_feed.rs:486), which dispatches inroom_feed/lenses.rs:19-63.
The lists in src/mimir_lens/mod.rs:
| list | members | line |
|---|---|---|
LENSES | the 26 table-fed lenses below | :95-128 |
SOURCE_LENSES | subclone, group_twins: not in LENSES; gRPC Mimir.Subclone / Mimir.GroupTwins (src/server_grpc.rs:3451, :3462) and the MCP tool mimir_subclone (src/mcp_stdio.rs:2136) are their doors | :178 |
REPO_SCOPED | villa, churn, undoc_api, trust, system_boundary: asked with a repo, they open that repo's room alone | :146 |
RING_ON | system_boundary: its room has the dependency universe folded in | :133 |
ROOMLESS | anomaly, generic_cost, binary_drift: read no room, never queued | :154 |
BUILD_FED | generic_cost, binary_drift, zombie_features: read Build Thing filings | :162 |
Common request fields: repo (default "" = every repo) and limit
(0 = DEFAULT_LIMIT 50, :180). They are read by Query::new
(src/mimir_lens/room_feed.rs:103-122). A missing number reads as 0, a flag
as false.
Feature gates. Without code-room, every room-fed lens refuses with
not_linked (room_feed.rs:496-499). Without vector, the clone and vector
lenses and both source lenses refuse with not_linked
(room_feed/lenses.rs:58-70, room_feed.rs:425, :457). vector is a
default feature; code-room comes with server.
Front doors:
- gRPC.
lens_answer(src/server_grpc.rs:786) returns a kept answer when it is ready, else queues the lens on the jeralenseslane and answersqueued/runningwith a job id (src/mimir_lens/queued.rs:182). Handlers sit atserver_grpc.rs:3322-3524(generic_cost to data_reach). - MCP.
mimir_<lens>tools insrc/mcp_stdio.rs(for examplemimir_generic_cost:1700,mimir_data_reach:2314). WithNORNIR_SERVERset a tool calls gRPC; otherwise it callsanswerin process (lens_embedded,:5041-5048). - dwarves.
dwarves mimir bloat|zombie|riskasks nornir's lens throughdwarves::lenswhen no local dump is given (dw:src/cli.rs:14995-15011). The CLI waits for a queued answer up to$DWARVES_LENS_WAITseconds, default 120 (dwarvesdocs/v2/README.md§8). - nornir CLI.
nornir mimir analyze <op>(src/bin/nornir.rs:783) covers the clone lenses, undoc-api, name-collision, trust, reachable-vuln and symbol-identity (:789-987). It refuses to open the room's lake locally and asks for--server <url>(:6422,:6527).
3.2 The envelope and its states
Every answer is built by finish (room_feed.rs:332):
| key | content |
|---|---|
kind | the lens name |
report_kind | the fold's own kind (for example bloat, risk, churn_filter) |
worst | the ranked rows (aliased from findings, anomalies, worst_artifacts or per_repo when the fold has no worst) |
feeds | [{table, rows, root, missing}] per input |
evidence_note | text; gaps follow MISSING:, bounds follow LOWER BOUND: |
lens_meta | {state, lens, repo_scope, room_scope, gaps, lower_bounds, ms} |
lens_meta.state:
| state | meaning | set at |
|---|---|---|
complete | an answer, no gaps | room_feed.rs:190-197 |
partial | an answer with named gaps | same |
empty_fact | no rows, and the producer is known to have run | same (Basis::fact, :177) |
empty_gap | no rows, and an input is missing | same |
not_wired | a lens waits for a producer (none today) | mod.rs:79, room_feed.rs:322 |
queued / running | the answer is being computed as a jera job | queued.rs:182 |
| a refusal name | not_linked, no_room, lake_unavailable, not_consumed, driver_not_available, no_artifact, invalid_argument, internal, no_embedder, vectors_missing, not_wired | room_feed.rs:307, src/code_room/mod.rs:168-235 |
The source lenses compute their own state (room_feed.rs:465-483).
Every caller and reach figure is a lower bound. The answers say so with the
note CALLS_LOWER_BOUND (room_feed.rs:89), which names "operators, closures
and fn pointers". Read against the driver, only the operators are missing:
- A call through a value (a closure, a
Box<dyn Fn…>, a genericF: Fn…) is a CALLS edge to theFn*trait method rustc records for the overloaded call, resolved to the impl when the type is concrete (crates/mimir-driver/src/walk.rs:1843-1852). - A call through a fn pointer is a CALLS edge to
Fn::call(:1853-1856). The fn that was turned into a value gets a CALLS edge from the body that names it (:1860-1869). - A closure body is part of the fn that defines it, so its calls are edges of
that fn (
README.md§10.2,compiler-to-graph.md). - An overloaded operator (
a + bon anAddimpl,Index,Deref) has no CALLS edge: the HIR walk handlesCall,MethodCalland paths only (walk.rs:1828-1906; schema 6 added arms for field writes, not for operators). MIR still has the call terminator, so the MIR-based lenses (gate-bypass phase 2, taint, twins' shapes) see it.
So the figures are a lower bound because of the operators, and because a
Fn* or Fn::call edge stops at the trait method unless a reader widens it
(the system-boundary walk does; most lenses do not). The note's wording is
older than the driver's value-call edges.
The GraphBLAS walks of the room live in nornir crates/mimir-graph/src/ops.rs:
- single reach is a BFS
q⟨¬v⟩ = q ANY.PAIR A(vxm,:6-7,:55-95); reach_each/reach_setsbatch every seed into one frontier matrix,F⟨¬V⟩ = F ANY.PAIR A(mxm,:668-699);- reverse walks use the cached transpose (
crates/mimir-room/src/query.rs:430-440).
Kept (incremental) answers:
clone_inc.rskeeps the clone folds per workspace and plans Carry, Advance or Full; it must equal the full recompute (src/mimir_lens/clone_inc.rs:1,:114).reach_inc.rskeeps the walks of system_boundary, data_reach and reachable_vuln.NORNIR_REACH_INC=0forces the reference walk (src/mimir_lens/reach_inc.rs:1,:73).
3.3 The lens table
| lens | finds | reads | module |
|---|---|---|---|
generic_cost | monomorphisation tax per artifact | Build Thing filings: symbol_sizes, binary_facts | bloat.rs |
zombie_features | features never built, or gating code nobody calls | feature nodes, ENABLES, GATED_BY, CALLS, USES_TYPE, Cargo.toml files | zombie.rs |
change_risk | files ranked by callers × churn × uncovered | CALLS, test reach, coverage_fn, git_heat_facts | risk.rs |
weak_tests | tests that cannot go red | test_outcomes, bench_runs, git_heat_facts, is_test nodes, CALLS | weak.rs |
bench_gaps | hot modules with no bench arm | CALLS fan-in, git_heat_facts, bench_runs, bench_results | benchcov.rs |
villa | impure call sites (process, shell, exit) | CALLS into std/libc/nix items | villa.rs |
churn | band filter on change frequency | git_heat_facts series, fn counts per file | churn_filter.rs, change_series.rs |
anomaly | outliers in the change series | git_heat_facts (all snapshots) | anomaly.rs |
world_impact | blast radius of the last commit | the room's commit window, dep_graph_edges | world_impact.rs |
binary_drift | the same artifact line filed twice with different bytes | Build Thing filings | binary_drift.rs |
data_reach | which code reaches which warehouse table | TOUCHES_TABLE, CALLS (reverse), effective_visibility | data_reach.rs |
trust | act / hold / refuse per fn by its weakest hop | CALLS, impure items, test reach | trust.rs |
name_collision | calls that land on a merged node id | CALLS, the room's duplicate keys | name_collision.rs |
symbol_identity | merged ids of our own definitions, by cause | definitions, duplicate keys | symbol_identity.rs |
reachable_vuln | advisory crates our code calls | CALLS, USES_TYPE, vuln_findings, sbom_components | reachable_vuln.rs |
undoc_api | undocumented public fns in churny files | effective_visibility, has_doc, git_heat_facts | undoc_api.rs |
system_boundary | which syscall classes a repo reaches, with a path | syscall nodes, SYSCALLS, CALLS through the universe | room_feed/boundary.rs |
similar_code | copied-not-moved clusters | graph structure, optional vectors | room_feed/vectors.rs |
semantic_clones | clones in the name-free model | fn_shape_events, embeddings_norm | room_feed/clones.rs |
exact_clones | the same normalised body | fn_shape_events | room_feed/clones.rs |
near_clones | MinHash near bodies | fn_shape_events | room_feed/clones.rs |
structural_clones | fns that touch the same things | params, return, types, fields, callees | room_feed/clones.rs |
role_neighbours | fns in the same role | engine=wl (default): WL role fingerprints on the item graph, fn_role_labels; engine=fastrp: FastRP role vectors over CALLS, USES_TYPE, READS_FIELD | room_feed/clones.rs, clone_signals/role_wl.rs |
same_intent | fns that read alike, names included | raw-model vectors, name words | room_feed/clones.rs |
semantic_twins | Type-3 copies and Type-4 twins | vectors plus the graph signals | room_feed/clones.rs |
reimplements_shared | a crate holds a copy of a function of a library it depends on | kept twins and similar_code answers, DEPENDS_ON | room_feed/reuse.rs |
subclone (source) | shared token runs inside items | the checkout, the room's item spans | subclone.rs |
group_twins (source) | the same group of statements in two fns | the room's per-line facts, optional vectors | group_twins.rs |
Producers of the warehouse feeds (room_feed/lenses.rs:378, :458, :613,
src/code_room/feeds.rs:13-17):
| table | producer |
|---|---|
| code room | the consumer, from cargo-mimir-dump output (src/code_room/consumer_loop.rs:161) |
git_heat_facts | the consumer's git_heat feed |
dep_graph_edges | the consumer's dep_graph feed |
test_outcomes | a Runa Nextest / CargoTest step |
bench_runs, bench_results | a Runa Bench step |
coverage_fn | a Runa Coverage step |
| Build Thing filings | Build Thing runs (builder 2), read from bin_ingest_runs of @build-thing (room_feed/built.rs:1-21, :62-98) |
vuln_findings | modgunn warehouse scan --persist (reachable_vuln.rs:25) |
syscall nodes, SYSCALLS | mimir_ingest::boundary::ensure, once per toolchain and target (room_feed/boundary.rs:5-14) |
fn_shape_events, embeddings_norm | the clone_shapes feed (src/clone_signals/feed.rs:1, :106, :153) |
embeddings (per fn) | the embeddings feed, vector::fnvec (src/vector/fnvec.rs:1-24, :126) |
3.4 Lens by lens
The room's GraphBLAS operations
Every walk a nornir lens runs on the code room is one of the operations below.
A is the boolean adjacency of one edge kind, or the union of several
(A[i][j] = edge i → j, "i calls / uses / depends on j"). The node space is the
room's ordinals 0..n. The lens sections below name the operation and point
here for the recurrence.
One note on the main semiring. GxB_ANY_PAIR_BOOL multiplies with PAIR (1
wherever both operands hold an entry) and adds with ANY (any present entry).
On a boolean pattern that is the structural OR-AND semiring (LOR.LAND): y[j]
is true when some frontier node has an edge to j. No value is read.
| op | recurrence | semiring / operator as the code names it | file:line |
|---|---|---|---|
reach (one walk) | f₀ = v₀ = seeds; f_{k+1}⟨¬v_k, struct, replace⟩ = f_k ANY.PAIR A (vxm); v_{k+1}⟨f_{k+1}, struct⟩ = true; stop when f is empty or k = max_hops | GxB_ANY_PAIR_BOOL, descriptor GrB_DESC_RSC; assign with GrB_DESC_S | crates/mimir-graph/src/ops.rs:49-99 (vxm :66-74, assign :87-95, stop :62) |
reverse_reach | the same on Aᵀ without building it: f⟨¬v⟩ = A ANY.PAIR f (mxv) | GxB_ANY_PAIR_BOOL | ops.rs:76-84, :115-121 |
reach_each / reach_sets (batched) | F is a B × n boolean matrix, row i = the frontier of walk i (one seed, or one seed set). F⟨¬V, struct, replace⟩ = F ANY.PAIR A (mxm); V⟨F, struct⟩ = true. All rows start at hop 0, so max_hops bounds each row exactly. Seeds are cut into chunks of B | GxB_ANY_PAIR_BOOL with GrB_DESC_RSC; GrB_Matrix_assign_BOOL with GrB_DESC_S | ops.rs:672-729 (mxm :695-703, assign :705-715, chunks :682); B = NORNIR_BFS_BATCH, default 64 (crates/mimir-room/src/query.rs:203-206) |
| kind union | C = A ∨ B per pair of kinds, cached per kind set | GrB_Matrix_eWiseAdd_BinaryOp with GrB_LOR | ops.rs:624-640; cache crates/mimir-room/src/snapshot.rs:426-444 |
dependents_by (transpose) | the backward walk is reach on Aᵀ; Aᵀ is built once per kind set by GrB_transpose and cached | GrB_transpose, then reach / reach_each with GxB_ANY_PAIR_BOOL | crates/mimir-graph/src/matrix.rs:178-188; snapshot.rs:446-451; query.rs:438-461, :526-535 |
dependents (Jikes walk) | W = Jᵀ ∨ diag(impls)·CONTAINS, with J = CALLS ∨ USES_TYPE ∨ IMPLEMENTS ∨ FOR_TYPE. Walking W forward is walking J backward, plus one step down from every impl block to the items it contains. Then reach (one set) or reach_sets (many sets) on W | diag(impls)·CONTAINS = GrB_mxm with GxB_ANY_SECOND_BOOL (mask_rows); ∨ = GrB_LOR; walk GxB_ANY_PAIR_BOOL | query.rs:18, :386-425, :500-520; snapshot.rs:471-488; ops.rs:644-661 |
within scope | A_s = diag(s) ANY.SECOND A ANY.FIRST diag(s): only edges with both ends in the node set s | GxB_ANY_SECOND_BOOL, then GxB_ANY_FIRST_BOOL | ops.rs:409-436; used at query.rs:348, :370, :407 |
bottleneck | over node weights w: best⟨seeds⟩ = w; loop: c = f MAX.FIRST A (vxm); c = min(c, w) (intersection: an unweighted node drops out); gt = c > best; f⟨gt, replace⟩ = c; f⟨¬best, struct⟩ = c; best = max(best, c); stop when f is empty | GxB_MAX_FIRST_UINT64; GrB_MIN_UINT64, GrB_GT_UINT64 (eWiseMult); GrB_MAX_UINT64 (eWiseAdd) | ops.rs:329-405 (vxm :344-352) |
propagate | y = A·x over a weighted A (FP64): y[i] = Σ_j A[i][j]·x[j] | GrB_PLUS_TIMES_SEMIRING_FP64 (mxv) | ops.rs:492-506 |
diff | added = A_y⟨¬A_x⟩, removed = A_x⟨¬A_y⟩ | GrB_Matrix_apply GrB_IDENTITY_BOOL, GrB_DESC_RSC | ops.rs:596-620 |
The room's reach, walk_each and dependents_by return the walk WITHOUT
its seeds (query.rs:352-353, :376-378, :413-415). ops::reach itself
includes them.
Not every room read is a product. Room::edges extracts the (from, to)
tuples of the per-kind matrix and filters them by two node bitmaps; it uses no
semiring (crates/mimir-room/src/export.rs:187-214). Room::mask,
Room::lookup (name search) and Room::sites are table reads.
Of these operations, bottleneck serves trust and propagate serves the role
vectors. The lenses of this part use reach, reach_each, reach_sets, the
cached transpose, the Jikes matrix, the within scope and edge reads.
generic_cost
Call. gRPC Mimir.GenericCost (src/server_grpc.rs:3316), MCP
mimir_generic_cost (src/mcp_stdio.rs:1700), or dwarves mimir bloat
without a local dump. Params repo, min_instantiations, limit. Roomless:
it reads Build Thing filings, never the code room.
Math. Per filed artifact a (only the latest filing of each line, see
below), per group g = (artifact, crate, generic):
instances of a = { s ∈ symbol_sizes(a) : size(s) > 0, section(s) starts ".text",
and ( is_monomorphization(s) -- v0 name carries generic args
or |{ mangled names with base(s) }| ≥ 2 ) } -- legacy: one hash per instance
base(s) = demangled name without generics, trailing "::" trimmed
N_g = number of instances in g
total_g = Σ size over the instances of g
tax_g = total_g − ⌊total_g / N_g⌋ (integer division, saturating)
kept iff N_g ≥ max(min_instantiations, 2)
class_g = glue if base contains "drop_in_place" or "drop_glue"
trait-method if base starts "::", contains "_::::", or its last segment is empty,
or base contains " as " (a <T as Trait>::m method)
named otherwise -- the only actionable class
total_bytes(a) = Σ size over ALL symbols of a (not the file size)
tax_bytes(a) = Σ_g tax_g
tax_pct(a) = 100 · tax_bytes(a) / total_bytes(a) (0 when total is 0)
actionable(a) = Σ tax_g over class named
Groups sort by tax_bytes descending, then symbol. Artifacts sort by
tax_bytes descending. worst keeps the top limit groups; top_named in the
JSON shows five named groups. (bloat.rs:49-58, :161-189, :193-249;
room_feed/built.rs:170-203.)
A line is (repo, variant, file, target, sorted features)
(binary_drift.rs:749-753). Two landings of one build with the same sha256 are
one filing (built.rs:62-101). The latest filing of a line is the last by
(rev, built_at, owner) (built.rs:99, :183).
The module doc says the feed is the code room's mono_instance nodes and
INSTANCE_OF edges (bloat.rs:3-5). The code reads the symbol_sizes of the
latest Build Thing filing instead (built.rs:184-185); no room is opened.
GraphBLAS steps. None. Every step is a scan or a BTreeMap group-by.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | read bin_ingest_runs of @build-thing, dedupe by (build_id, sha256) | not GraphBLAS | the filings | built.rs:62-101 |
| 2 | group by line, take the last | not GraphBLAS | latest filing per line | built.rs:104-110, :182-189 |
| 3 | load symbol_sizes by sha256 | not GraphBLAS (warehouse read) | the analysed symbols | built.rs:131-161 |
| 4 | filter instances, group, tax | not GraphBLAS | the formulas above | bloat.rs:161-249 |
Dataflow.
bin_ingest_runs(@build-thing).ledger ──► Filing{repo,variant,file,target,features,sha256,rev}
│ group by line, last by (rev, built_at, owner)
▼
symbol_sizes[sha256] ──► instances_of(): .text, size>0, mono or name shared ≥2
│ └─► total_bytes(a) = Σ all sizes
▼
group (artifact, crate, generic) ──► N, total, tax = total − ⌊total/N⌋, class
▼
artifacts[ repo, artifact, total_bytes, tax_bytes, tax_pct, actionable_tax_bytes, top_named ]
Worked example. Fixture bloat.rs:268: four instances of
znippy::parse_stream (u8, u16, u32, u64), 100 B each, in an artifact
whose symbols total 1,000 B.
N = 4, total = 4 · 100 = 400
tax = 400 − ⌊400 / 4⌋ = 400 − 100 = 300
tax_pct = 100 · 300 / 1000 = 30.0
class = named → actionable = 300
min_instantiations = 5 → floor max(5, 2) = 5 > 4 → no group, no artifact (bloat.rs:283)
one instance only → N = 1 < 2 → no group (bloat.rs:282)
Fixture bloat.rs:287: two instances of znippy::Codec::encode, 50 B each,
marked as trait-impl methods. tax = 100 − ⌊100/2⌋ = 50, class trait-method,
actionable 0. A real build (room_feed/built_tests.rs:149) instantiates
widen six times (u8, u16, u32, u64, bool, char) and the lens
reports instantiations = 6 for it (built_tests.rs:183).
On Holger, only the dwarves counterpart has a recorded number: the golden
bloat row reads artifacts 0, total_tax_bytes 0 because that dump has no
binary layer (crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json). That is dwarves'
fold over a dump, not this one.
Output. artifacts, total_tax_bytes, actionable_tax_bytes,
source: "build", worst_artifacts[repo, artifact, total_bytes, tax_bytes, tax_pct, actionable_tax_bytes, top_named[symbol, crate, instantiations, tax_bytes]] (bloat.rs:116-134, built.rs:201).
Limits. Nothing filed gives the gap no_build, which says to run
dwarves buildthing <repo> --variant <variant> (built.rs:43-48). A filing
whose bytes were not analysed on this server reads ABSENT, not zero
(built.rs:143-150). Only the latest filing per line is read.
Tests. bloat.rs:268, :287, :300; room_feed/built_tests.rs:149;
room_feed/room_tests.rs:380.
zombie_features
Call. gRPC Mimir.ZombieFeatures (server_grpc.rs:3325), MCP
mimir_zombie_features (mcp_stdio.rs:1731), or dwarves mimir zombie
without a local dump. Params repo, limit. Room: every consumed repo.
Math. Keys are crate/feature (a feature belongs to one crate). For each
repo r of the request:
declared(r) = { label(f) : f feature node of r, origin Local, defined }
built(r) = { label(f) : CRATE node ─ENABLES→ f } -- the consumed build
∪ { d ∈ declared(r) : a file of r turns d on } -- see "enablers"
enablers = every Cargo.toml under the clone (depth ≤ 6; target, .git, node_modules,
vendor skipped): [features] items (dep/feat and own names, not dep:),
features = [..] on [dependencies], [dev-dependencies], [build-dependencies],
[workspace.dependencies], [target.*.*]; .nornir/build-variants.toml variants
exempt(f) = f is a [[bin]] required-feature (opt_in)
or the feature NAME is in build-variants `offers` / `allow_unbuilt`
A gate is one GATED_BY edge item → feature, skipped when exempt. Per
(repo, feature):
partial = items with GATED_BY.partial = true (counted, never judged)
negated = own gates whose every mention is under not(…) (counted, never judged)
own = own, non-negated gates (by site)
compiled_out = own gates whose item kind is cfg_gated_out
judged = own gates that were compiled
inbound = Σ over judged items of (CALLS + USES_TYPE site counts arriving at the item)
never_built = declared ∋ f and built ∌ f (build space known)
unreached = judged ≠ ∅ and inbound = 0
direction = both | never-built | unreached | (not a zombie)
A second pass adds every declared − built feature that has no gate at all as
never-built, with the site text "declared in Cargo.toml; no crate was compiled
with it, and no code is gated by it". Severe = never-built or both. Rows sort
severe first, then gated_items descending, then repo, then feature. Totals
are counted before the cut to limit. (zombie.rs:282-413;
room_feed/lenses.rs:152-277.)
Three code facts a reviewer should know:
inboundcounts every caller, including another item gated by the same feature. A gated item reached only from its own feature's code reads as reached (lenses.rs:221-223).- The room path sets
build_space_known = truefor every repo of the request (lenses.rs:168-170).build_space_unknownis therefore always empty there; only the unit test reaches it (zombie.rs:539). - The exemption drops the gates only (
lenses.rs:232-235). The declared-but-not-built pass does not consult it (zombie.rs:351-375). An opt-in feature that nothing enables therefore still reads never-built, withgated_items 0. The basis note atlenses.rs:267says such features are "no zombie". The code and that note differ in this case.
GraphBLAS steps. None. All steps are masks and edge reads.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | mask feature nodes, origin Local, request repos | not GraphBLAS | the features | lenses.rs:163 |
| 2 | edge read ENABLES * → features | not GraphBLAS (Room::edges) | built by the consumed build | lenses.rs:181-189 |
| 3 | edge read GATED_BY * → features | not GraphBLAS | the gates, partial, negated | lenses.rs:216 |
| 4 | edge read CALLS ∪ USES_TYPE * → gated items | not GraphBLAS | inbound per item | lenses.rs:217-224 |
| 5 | Cargo.toml / build-variants parse | not GraphBLAS (TOML) | enablers, opt-in, excused | zombie.rs:203-264 |
| 6 | fold | not GraphBLAS | directions, totals | zombie.rs:282-413 |
Dataflow.
code_nodes(feature) ─┬─ ENABLES from crate ──► built
├─ defined ─────────────► declared
└─ GATED_BY ◄── items ──► own / partial / negated / compiled_out
items ◄── CALLS + USES_TYPE (site counts) ────► inbound
clone/**/Cargo.toml, .nornir/build-variants.toml ──► enablers ∪→ built ; opt_in, excused → exempt
Build Thing latest artifacts ──► LinkedBytes (path_key join) ──► bytes_in_binary
▼
never_built = declared ∖ built ; unreached = judged ≠ ∅ ∧ inbound = 0
▼
worst[ repo, feature, direction, gated_items, compiled_out_items, partial_items,
negated_items, inbound_calls, bytes_in_binary, sites ]
bytes_in_binary = per item path (generics and <> erased), the largest
total over the latest artifact of every line, for repos some analysed artifact
was built from; else null (built.rs:276-340).
Worked example. Room fixture room_feed/room_tests.rs:190, repo shop, 4
features seen:
| feature | facts | rule | direction |
|---|---|---|---|
shop/legacy | declared, no ENABLES; its one item is a cfg_gated_out stub | declared ∋ f ∧ built ∌ f | never-built (gated 1, compiled_out 1) |
shop/unused | built; dead() exists only under it; 0 callers | judged = {dead} ≠ ∅ ∧ inbound = 0 | unreached (inbound 0, bytes null) |
shop/stripe | built; called at 4 sites | inbound = 4 | not a zombie |
shop/trace | only a cfg!(…) inside audit() and bye() | partial only, judged = ∅ | not a zombie |
With the ENABLES edges removed (room_tests.rs:216), built = ∅, so all four
are never-built and unused becomes never-built+unreached.
Unit fixture zombie.rs:478: korp/spark41 gates one compiled item with 0
inbound, korp/live one with 2. Both built. Result: one zombie, spark41,
unreached.
The offline fold over a dump (dwarves mimir --wh <dump> zombie,
crates/mimir-lens/src/mimir/zombie.rs) judges the same way. The adapter
(coverage/v2.rs) marks each feature_gate_facts row with compiled_out
(the item is a cfg_gated_out stub) and item_kind. Per (repo, feature)
the item keys split into four sets. A key compiled in one unit is judged even
when another unit compiled it out.
negated = keys whose cfg names f only under not(…) (feature_mentions)
compiled_out = keys present only as cfg_gated_out stubs
roots = crate roots gated by #![cfg(…)] (a target is run, never called)
judged = every other key
unreached = judged ≠ ∅ ∧ Σ inbound CALLS over judged = 0
not_judged = features with gates and judged = ∅ (and not never-built), listed with
the three counts and named in the evidence note: UNKNOWN, not clean
Worked example: on the Holger h32 dump, front gates 27 keys. 25 are
cfg_gated_out stubs, because the dump built holger-server-lib without
front: the 6 helpers and 2 tests in exposed/http.rs, the same 8 again in
each of the two test crates that compile that module, and one test in
what_is_open_is_the_rons.rs. The other 2 sit under not(feature = "front"):
the crate tests/a_build_without_front_compiles.rs and its test. No key is
judged, so front is NOT JUDGED, with 25 compiled out and 2 under not(…). It
used to read "unreached 27": the stubs have no callers because rustc never
compiled them. In the source, handle calls all six helpers when front is
on (exposed/http.rs:1013-1091). Test fixture: tests/mimir_v2_dump.rs
a_feature_the_dump_compiled_out_names_only_under_not_or_gates_a_crate_root_is_not_unreached.
The golden reads zombies 7, severe 7, features_seen 80 (all 7 never-built),
with 25 features not judged (crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json).
Before the FRONT-ZOMBIE fix it read 25 zombies, of which 18 were unreached
only because the dump compiled their code out or the fold read not(…) and
crate-root gates as the feature's code.
Output. features_seen, repos_with_gates, build_space_unknown, zombies, severe, zombies_shown, resolution; rows repo, feature, direction, gated_items, compiled_out_items, partial_items, negated_items, inbound_calls, bytes_in_binary, sites (zombie.rs:95-115).
Limits. A cfg naming a feature with no GATED_BY edge is a gap
(lenses.rs:251-262). cfg_attr predicates, macro $metavariable feature
names and dependency gates of compiled-out items are lower bounds
(lenses.rs:270-272). The feature excused set matches by feature name in
every crate of the repo (zombie.rs:268-272).
Tests. zombie.rs:433, :445, :478, :488, :501, :510, :524,
:531, :539; room_tests.rs:190, :216; built_tests.rs:243.
change_risk
Call. gRPC Mimir.ChangeRisk (server_grpc.rs:3334), MCP
mimir_change_risk (mcp_stdio.rs:1761), or dwarves mimir risk without a
local dump. Params repo, limit.
Math. Per file F of the request's repos, over the callables (fn,
method) in F that are not tests (is_test, or inside a cfg that requires
test):
callers(F) = Σ_{f ∈ F} Σ_{e : CALLS → f} count(e) -- sites from anywhere in the room, tests included
churn(F) = git_heat_facts.commits_90d of (repo, file), latest snapshot; 0 if absent
cov(F) = min(covered, functions) / functions (0 when functions = 0)
covered = line basis: |{ f ∈ F : some coverage_fn row landing on f has
regions_covered > 0 or lines_covered > 0 }|
reach basis: |{ f ∈ F : f ∈ T }|
T = reach(is_test nodes of the request, CALLS) ∪ { targets of TESTS from them }
risk(F) = callers(F) · churn(F) · (1 − cov(F))
The basis is per repo: line when that repo's newest coverage_fn run exists
(newest by (ts_micros, run_id)), else reach (lenses.rs:365-370,
rows.rs:316-325). A coverage row lands on the innermost room function whose
span holds its first line: among candidates with line ≤ row.line ≤ max(end_line, line), the largest start line, then the smallest end line
(risk.rs:135-160). Files with risk = 0 are dropped after files_scored is
counted. Rows sort by risk descending, then file (risk.rs:164-226). Reach is
more generous than line coverage, so the reach basis under-reports risk; the
answer says so (lenses.rs:383-388).
GraphBLAS steps.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | mask fn + method, origin Local, request repos | not GraphBLAS | callables | room_feed/graph.rs:169-177 |
| 2 | edge read CALLS * → callables, sum count | not GraphBLAS (Room::edges) | call sites per fn | graph.rs:204-211 |
| 3 | reach(tests, CALLS): f⟨¬v⟩ = f ANY.PAIR A_CALLS until empty | GxB_ANY_PAIR_BOOL (vxm) | the test-reachable set T | graph.rs:223; query.rs:482-491; ops.rs:66-74 |
| 4 | edge read TESTS tests → * | not GraphBLAS | direct test targets | graph.rs:226 |
| 5 | span join of coverage_fn rows | not GraphBLAS | line covered | risk.rs:135-160 |
| 6 | per-file sum and product | not GraphBLAS | risk | lenses.rs:358-372; risk.rs:187-204 |
Dataflow.
code_nodes(fn,method) ──► callables ──┬─ CALLS in-edges (Σ count) ──► callers(F)
└─ ∈ T ? ──────────────────────► covered (reach)
code_nodes(is_test) ──reach CALLS (ANY.PAIR)──► T ∪ TESTS targets
coverage_fn (newest run per repo) ──span join──► covered (line)
git_heat_facts (latest) ──(repo,file)──► churn(F)
▼
risk(F) = callers · churn · (1 − covered/functions) → drop 0 → sort desc
▼
worst[ repo, file, risk, downstream_callers, churn_90d, coverage_frac, coverage_basis, explain ]
Worked example. Room fixture room_feed/room_tests.rs:237, repo shop,
no coverage_fn (reach basis), files_scored = 4:
| file | callers | churn_90d | cov | risk |
|---|---|---|---|---|
src/pay.rs | 4 (charge) + 2 (audit) + 1 (shell) + 1 (refund, from web) = 8 | 12 | 3 of 6 fns reached = 0.5 | 8 · 12 · (1 − 0.5) = 48 |
src/lib.rs | 1 (checkout, from web) | 5 | 0 of 1 = 0.0 | 1 · 5 · 1 = 5 |
src/cart.rs | 2 (add) + 2 (total) + 1 (open) = 5 | 2 | 3 of 4 = 0.75 | 5 · 2 · 0.25 = 2.5 |
The fourth file has risk 0 and is scored but not listed. The test pins the
three rows and risk = 48.0 for the first (room_tests.rs:249-250).
Unit fixture risk.rs:266, heat commits_90d = 20: one fn, uncovered, 10
callers gives 10 · 20 · 1 = 200. Two fns, one covered, gives 10 · 20 · 0.5 = 100. Fully covered gives 0, so the file is scored (files_scored 1) and not
listed.
On Holger, only the dwarves counterpart has recorded numbers: golden
files_scored 616, units 324, without churn (crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json;
plan §D, "risk ranks without git heat").
Output. files_scored, units, coverage_basis, semver_available (false), model_attribution, changed_uncovered[repo, file, fn, line, commit]; rows
repo, file, risk, downstream_callers, churn_90d, coverage_frac, coverage_basis (line | reach), explain (risk.rs:67-87, lenses.rs:405-407).
Limits. Without a coverage run the basis is reach and risk is
under-reported (lenses.rs:383-388). semver_diffs has no producer
(lenses.rs:390). A fn without a span belongs to no file and is not scored
(lenses.rs:373-375). Every caller count is a lower bound
(CALLS_LOWER_BOUND, room_feed.rs:89).
Tests. risk.rs:255, :266, :288; room_tests.rs:237;
room_feed/feed_tests.rs:297, :341 (ignored: needs cargo-llvm-cov).
weak_tests
Call. gRPC Mimir.WeakTests (server_grpc.rs:3424), MCP
mimir_weak_tests (mcp_stdio.rs:2050). Params repo, limit.
Math. Constants: MIN_RUNS = 3, MIN_VERSIONS = 2, PLACEBO_CHURN = 5
(weak.rs:18-26). A case "ran" unless its status is ignored or skipped
(rows.rs:152-154). Its repo and code version come from its bench_runs row
(version = commit, else version string), else from the case row itself
(weak.rs:144-157, rows.rs:112-114). Per (repo, test_name):
runs = |distinct run_id| versions = |distinct non-empty code versions|
failures = cases with passed = false
churn = Σ commits_90d over the DISTINCT files of
{ o ∈ reach_each(test node, CALLS) ∖ {test} : origin Local, defined }
symbols = | that set |
verdict = OK (proven red) if failures > 0
= NO-DATA else if runs < 3
= NO-DATA else if versions < 2
= PLACEBO else if churn ≥ 5
= CANNOT-GO-RED otherwise
Findings are PLACEBO and CANNOT-GO-RED. They sort PLACEBO first, then churn
descending, runs descending, test name (weak.rs:184-243). PLACEBO_CHURN
is a floor: the code compares with ≥, so the fifth commit counts, and the
constant's doc says why (five commits in a quarter is the least churn that
gave the test a real chance to go red; weak.rs, nornir 624abb34). The dwarves
copy in crates/mimir-lens/src/coverage/weak.rs says the same (dwarves 847d752).
The test node is found by name: same repo, same leaf name, and the node's path
equals the name or ends with ::name. Exactly one hit joins; none or several
leave the test on run history alone (lenses.rs:626-638). With a repo
filter only the worst list is cut to that repo (lenses.rs:669-674); the
counters tests_seen, judgeable, placebos and the rest are folded over
every repo in test_outcomes (lenses.rs:667).
GraphBLAS steps.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | mask is_test nodes, request repos | not GraphBLAS | the test nodes | lenses.rs:590-596 |
| 2 | name join outcomes → one test node | not GraphBLAS | seeds | lenses.rs:626-638 |
| 3 | reach_each(seeds, CALLS): F⟨¬V, struct, replace⟩ = F ANY.PAIR A_CALLS, 64 tests per frontier matrix | GxB_ANY_PAIR_BOOL (mxm) | each test's forward reach | lenses.rs:640-641; query.rs:430-461, :360-380; ops.rs:695-703 |
| 4 | file of each reached node, hash join to git_heat_facts | not GraphBLAS | churn per test | lenses.rs:642-658 |
| 5 | outcome fold | not GraphBLAS | verdicts | weak.rs:137-289 |
Dataflow.
test_outcomes ──ran()──► (repo, test_name) ◄── bench_runs (run_id → repo, version)
│ runs, versions, failures
code_nodes(is_test) ──name join──► one test node ──reach_each CALLS (ANY.PAIR, mxm)──► reached
reached ∩ Local ∩ defined ──file──► git_heat_facts(latest).commits_90d ──Σ distinct files──► churn
▼
verdict rules (MIN_RUNS 3, MIN_VERSIONS 2, PLACEBO_CHURN 5)
▼
worst[ repo, test_name, verdict, runs, versions, failures, churn_commits_90d, reachable_symbols ]
Worked example. Unit fixture weak.rs:366: three runs of korp at versions
0.1.0, 0.1.1, 0.2.0; korp.hot and korp.cold pass in all three.
korp.hot : failures 0; runs 3 ≥ 3; versions 3 ≥ 2; churn 40 ≥ 5 → PLACEBO
korp.cold: failures 0; runs 3 ≥ 3; versions 3 ≥ 2; churn 1 < 5 → CANNOT-GO-RED
sorted: PLACEBO first → korp.hot, korp.cold; placebos = 1
the same churn keyed to repo skade → korp.hot has churn 0 → placebos = 0 (weak.rs:405-406)
Room fixture room_tests.rs:326, repo shop: tests_seen 3, judgeable 3, placebos 1, churn_joined 2. pay::charges_once reaches charge, audit, shell
and libc_write (4 symbols), all in src/pay.rs (12 commits in 90 days):
PLACEBO, churn 12. cart::tests::adds_up reaches add, total and open in
src/cart.rs (2 commits): CANNOT-GO-RED, churn 2, symbols 3.
ghost::never_compiled joins no test node: CANNOT-GO-RED on run history alone,
churn 0, symbols 0.
Fixture weak.rs:331: adds_up passes in runs s1, s2 (commit c1) and s3
(c2), so runs 3 and versions 2: CANNOT-GO-RED. pay::charges_once fails once:
OK. pay::slow is ignored and is no test seen.
Output. outcome_rows, tests_seen, judgeable, proven_red, insufficient, placebos, churn_joined; rows repo, test_name, verdict, runs, versions, failures, churn_commits_90d, reachable_symbols (weak.rs:113-130).
Limits. A repo without a filed test run is a gap (lenses.rs:611-619).
Names that join no single #[test] node are counted and named
(lenses.rs:659-666). The reach is a lower bound (lenses.rs:622-624).
Tests. weak.rs:331, :351, :366, :410; room_tests.rs:326;
feed_tests.rs:216.
bench_gaps
Call. gRPC Mimir.BenchGaps (server_grpc.rs:3433), MCP
mimir_bench_gaps (mcp_stdio.rs:2080). Params repo, limit.
Math. A module is the nearest Module or Crate ancestor of a callable
(crates/mimir-room/src/attrs.rs:174-187). Per module m of repo r, over
its non-test callables:
fan(m) = Σ CALLS sites into its fns churn(m) = Σ commits_90d over its files
cov(m) = min(fns_covered, fns_total) / fns_total (reach basis: fn ∈ T, as in change_risk)
hint(m) = 1 if the module path (lowercase) contains one of the 23 PERF_MODULE_HINTS, else 0
max_fan = max(1, max_m fan(m)) max_churn = max(1, max_m churn(m))
score(m) = 0.40 · fan(m)/max_fan + 0.25 · churn(m)/max_churn + 0.25 · (1 − cov(m)) + 0.10 · hint(m)
Coverage enters as 1 − cov: the less a module's fns are test-reached, the
hotter the gap, and a module every test reaches is the last to need a bench
(W_COVERAGE * (1.0 - cov), benchcov.rs; nornir 922c3714). Before that
commit the code added + 0.25 · cov, so a better-tested module scored higher.
The module doc says hot paths are ranked by "fan-in × churn × perf-name hint ×
test coverage" (benchcov.rs:4-6); the code computes the weighted sum above.
Test modules are no bench surface: a segment tests, test, *_tests,
test_*, or ::benches in the path (benchcov.rs:113-118). The rest sort by
score descending, then module, and are cut to limit BEFORE the gap is
counted (benchcov.rs:207-214). So hot_paths = min(modules, limit).
arms(r) = { result_name of bench_results whose run_id is a bench_runs row of r }
covers(a, m) = tokens(scenario(a)) ∩ tokens(m) ≠ ∅
scenario(a) = text after the first '.', lowercased
tokens(s) = split on non-alphanumerics, length ≥ 3,
not a STOP_TOKEN (23 words), not a part of the repo name
benched(m) = ∃ a ∈ arms(r) : covers(a, m)
gap = HotPaths − Benched − Allowlist (edda_autonom::compute_gap; allowlist = ∅)
never_benched(r) = no bench_runs row of r
(benchcov.rs:44-49, :122-164, :167-264; lenses.rs:438-511.) The
top-level worst concatenates each repo's unbenched modules, repos with the
most unbenched first, cut to limit (lenses.rs:491-502).
GraphBLAS steps.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | callables, call sites, test reach (as change_risk 1–4) | GxB_ANY_PAIR_BOOL for the test reach; the rest not GraphBLAS | fan, cov per fn | graph.rs:264-287 |
| 2 | parent chain to module | not GraphBLAS | module per fn | lenses.rs:447-456 |
| 3 | weighted score, sort, cut | not GraphBLAS | hot paths | benchcov.rs:167-215 |
| 4 | token match arms × modules | not GraphBLAS | benched | benchcov.rs:130-155 |
| 5 | gap | not GraphBLAS (set difference) | unbenched | benchcov.rs:240-263 |
Dataflow.
callables ─ CALLS Σ count ─► fan ; ∈ T ─► fns_covered ; span file ─► files
│ group by module_of
git_heat_facts(latest, repo) ──Σ over files──► churn
▼
score = .40 fan/max + .25 churn/max + .25 (1−cov) + .10 hint ──► sort, cut to limit
bench_runs(repo) ⋈ bench_results ──► arms ──token match──► benched
▼
per_repo[ repo, hot_paths, benched, unbenched, bench_runs, bench_arms, never_benched,
worst[ module, surface_id, fan_in, commits_90d, test_coverage, score, hint ] ]
Worked example. Unit fixture benchcov.rs:333, repo korp, heat 30 commits
on the file of korp::hot:
modules (tests excluded): called fan 9 · hot churn 30 · covered cov 1 · quiet
(each module has one fn; only covered's fn is test-reached, so the others have cov 0)
max_fan = 9, max_churn = 30, no module name carries a hint
called = 0.40 · 9/9 + 0.25 · (1 − 0) = 0.65
hot = 0.25 · 30/30 + 0.25 · (1 − 0) = 0.50
quiet = 0.25 · (1 − 0) = 0.25
covered = 0.25 · (1 − 1) = 0.00
order: called, hot, quiet, covered (as asserted at benchcov.rs:342)
Unit fixture benchcov.rs:324: arm znippy.decode_mt → scenario decode_mt
→ tokens {decode} (mt is shorter than 3). Module znippy::decode → tokens
{decode} (znippy is the repo). They intersect, so decode is benched;
znippy::encode and znippy::caller are not.
Room fixture room_tests.rs:281, repo shop: shop::pay has fan 8 (the
maximum), churn 12 (the maximum) and coverage 0.5, so its score is `0.40 + 0.25
- 0.25 · (1 − 0.5) = 0.775
. With1 − cov, the less-reached module is the hotter gap; the room test asserts the ordershop::pay,shop,shop::cart,shop::util(room_tests.rs:290`). The fixture has 3 runs and 0 arms, so all 4 hot paths are unbenched.
Output. repos, per_repo[...] as above, worst (rows plus repo)
(lenses.rs:474-509).
Limits. No bench run filed is a gap (lenses.rs:463-468). The allowlist
is empty (lenses.rs:473). The arm match is by name tokens, not by what the
bench calls.
Tests. benchcov.rs:295, :314, :324, :333; room_tests.rs:281;
feed_tests.rs:259.
villa
Call. gRPC Mimir.Villa (server_grpc.rs:3442), MCP mimir_villa
(mcp_stdio.rs:2155). Params repo, limit. Repo-scoped: with a repo it
opens that repo's room alone.
Math. The impure items are nodes whose name is one of 13 IMPURE_NAMES
and whose resolved path classify accepts (villa.rs:170, graph.rs:293-307):
classify(path, kind) =
process if path ends "process::Command::new"
shell if last segment ∈ {system, popen, execvp, execv, execve, execvpe,
execl, execlp, execle, exec}
and (kind = ffi or root ∈ {libc, nix, rustix}
or (last = exec and path contains "process::"))
process-control if root = std and path ends "process::exit" or "process::abort"
none otherwise
process + a string literal in Command::new("…") on the site line or the next two
→ external-tool, known_tool = the literal (no '\' and no '{' in it)
One finding per positioned CALLS site from a Local caller of the request's
repos. When an edge counts more sites than it has positions, one extra finding
is placed at the caller's own line (lenses.rs:124-133). Sort key: weight
external-tool 3 > shell 2 > process 1 > process-control 0, then repo, file,
line, symbol (villa.rs:62-69, :301-309). Severity: external-tool and shell
high, process medium, process-control low (villa.rs:71-77).
The doc of Heresy::known_tool still speaks of a KNOWN_OFFENDERS list
(villa.rs:92-95). No such list exists; the tool is the literal read at the
call site (villa.rs:157-166, :257-263).
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | name lookup of IMPURE_NAMES, classify | not GraphBLAS | impure items | graph.rs:293-307 |
| 2 | edge read CALLS * → impure | not GraphBLAS (Room::edges) | candidate sites | lenses.rs:80-82 |
| 3 | filter Local callers of the request | not GraphBLAS | sites | lenses.rs:101-106 |
| 4 | read the source line from the clone | not GraphBLAS | known_tool | lenses.rs:90-99; villa.rs:157-166 |
Dataflow.
name lookup (new, exit, abort, system, exec*, popen) ──classify(path, kind)──► impure items
CALLS edges ─► to impure, from Local caller in repos ──► sites (file, line) per edge
clone/<repo>/<file> lines[line..line+2] ──"Command::new(\"…\")"──► binary
▼
kind ← process | shell | control ; process + literal → external-tool
▼
findings[ repo, symbol, loc, kind, severity, callee, known_tool, evidence_note ] ; by_kind
Worked example. Room fixture room_tests.rs:98, repo shop, 3 heresies:
| site | callee | classify | weight | rank |
|---|---|---|---|---|
shop::pay::shell src/pay.rs:34 | libc::system | shell (root libc) | 2 | 1 |
shop::pay::charge src/pay.rs:12 | std::process::Command::new | process (no literal) | 1 | 2 |
shop::pay::bye src/pay.rs:39 | std::process::exit | process-control | 0 | 3 |
Unit fixture villa.rs:386: a Command::new site in znippy whose line spells
"osmium" becomes external-tool, known_tool = osmium, severity high.
Output. edges_seen (all CALLS edges of the room, from its stats,
lenses.rs:83), heresies, repos_with_heresy, by_kind; rows repo, symbol, loc, file, line, kind, severity, callee, known_tool, evidence_note
(villa.rs:120-149).
Limits. A binary held in a variable or built by format! is not named
(villa.rs:157-166). Builder methods (spawn, output, arg) are never
anchors, so one shell-out is one finding (villa.rs:198-200). A room without
CALLS reads as a missing feed, not as "no heresy" (villa.rs:443).
Tests. villa.rs:344, :360, :375, :386, :401, :411, :431,
:443, :451; room_tests.rs:98, :134, :157, :174;
tests/mimir_villa_robot.rs:17.
churn
Call. gRPC Mimir.Churn (server_grpc.rs:3475), MCP mimir_churn
(mcp_stdio.rs:2186). Params band (high default, low, band),
cutoff_low, cutoff_high (both default 0), repo, limit. Repo-scoped.
Math. Every git_heat_facts generation forms one series per (repo, file), sorted by (ts_micros, snapshot_id) (change_series.rs:88-127). The
filter reads the newest point p:
frequency = p.commits_90d
complexity(F) = | fn + method nodes of the room whose span lies in F | (tests included)
hotspot = frequency · complexity (saturating)
high-pass : keep frequency ≥ cutoff_high sort hotspot ↓, frequency ↓, repo, file
low-pass : keep frequency ≤ cutoff_low stable_but_changed = (p.commits_30d > 0)
sort stable_but_changed first, commits_30d ↓,
complexity ↓, repo, file
band-pass : keep cutoff_low ≤ frequency ≤ cutoff_high sort as high-pass
(churn_filter.rs:150-228; lenses.rs:412-433.) Band::parse maps low,
low-pass, lowpass, stable to low-pass, band, band-pass, bandpass to
band-pass, and everything else to high-pass (churn_filter.rs:49-56). With the
default cutoff 0 the high pass keeps every file.
The module doc says complexity is the count of fn items from symbol_facts
(churn_filter.rs:20-21, :147). The code counts the room's fn and method
nodes per span file (lenses.rs:415-422). That count includes test fns: the
fixture's pay.rs holds 7 = 6 + its test (room_tests.rs:268), while
change_risk excludes tests.
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | mask fn + method, count per span file | not GraphBLAS | complexity | lenses.rs:415-422 |
| 2 | parse every heat generation into series | not GraphBLAS | NodeSeries | change_series.rs:88-127 |
| 3 | band filter and sort | not GraphBLAS | rows | churn_filter.rs:159-213 |
Dataflow.
git_heat_facts (ALL snapshots) ──group (repo,file), sort ts──► series ──latest()──► p
code_nodes(fn,method) ──span file──► complexity(F)
▼
band filter on p.commits_90d ; hotspot = commits_90d · complexity ; low: commits_30d > 0
▼
worst[ repo, file, frequency, commits_total, commits_30d, complexity, hotspot,
stable_but_changed, explain ]
Worked example. Room fixture room_tests.rs:263, repo shop, high pass,
cutoff 0:
| file | commits_90d | complexity | hotspot |
|---|---|---|---|
src/pay.rs | 12 | 7 | 12 · 7 = 84 |
src/cart.rs | 2 | 5 | 2 · 5 = 10 |
src/lib.rs | 5 | 1 | 5 · 1 = 5 |
src/util.rs | 0 | 1 | 0 · 1 = 0 |
The test pins complexity and hotspot. The frequency column follows from them
(84 / 7 = 12, and so on) and matches the churn that change_risk reads from the
same fixture. With band low, cutoff_low 2, the files with commits_90d ≤ 2
pass: cart.rs (2) and util.rs (0).
Unit fixture churn_filter.rs:265, cutoff_high 10: hot (30 · 12 = 360) and
warm (12 · 2 = 24) pass; cold (1) is filtered.
Output. band, cutoff_low, cutoff_high, nodes_seen, passed, stable_changes_flagged, model_attribution; rows as in the dataflow
(churn_filter.rs:117-142).
Limits. An empty series names the git_heat feed; a repo filter with no
row is a gap (room_feed.rs:381-397). A churned file with no function the
room knows has complexity 0 (lenses.rs:426-429).
Tests. churn_filter.rs:265, :296, :327, :339;
change_series.rs:179, :198, :210, :224; room_tests.rs:263.
anomaly
Call. gRPC Mimir.Anomaly (server_grpc.rs:3484), MCP mimir_anomaly
(mcp_stdio.rs:2221). Params z_threshold (default 3.0; a value ≤ 0 keeps
the default, room_feed.rs:366-369), repo, limit. Roomless: it reads
git_heat_facts only.
Math. Constants: z_threshold 3.0, min_history 6, stable_baseline 0.5,
ewma_alpha 0.4 (anomaly.rs:135, :154-163). Per file, the increments are
the change counts per interval (change_series.rs:66-79):
x₀ = commits_total(p₀) -- the backfilled history of the first scan
x_k = max(0, commits_total(p_k) − commits_total(p_{k−1})) for k ≥ 1
μ, σ = population mean and standard deviation (σ = √(Σ(x−μ)²/n))
EWMA: s₁ = x₁; s_k = α·x_k + (1 − α)·s_{k−1}, α = 0.4
z(o; μ, σ) = (o − μ)/σ if σ > 1e-9
= max(z_threshold, 4.0) if σ ≤ 1e-9 and o > μ -- saturated
= 0 otherwise
z_ewma = (o − s)/σ with s = the EWMA of the baseline, if σ > 1e-9; else z
For kinds (a) and (b) a row needs z ≥ z_threshold, o > μ AND
z_ewma ≥ z_threshold (anomaly.rs:181-205). A slow ramp lifts the EWMA with
it, so its latest interval is far above the plain mean but not above the EWMA,
and it does not fire. (Before nornir ca92f7b7 the EWMA was computed and printed
but gated nothing.) The unit test a_slow_ramp_is_not_a_spike
(anomaly.rs:473) uses increments 1, 2, 3, 4, 5, 6, 7 with z_threshold 1.5:
the code comment records the latest interval as 2.1 σ above the plain mean of
its history but only 1.4 σ above the EWMA, so it is not a spike.
The four kinds:
| kind | needs | baseline | observed | fires when |
|---|---|---|---|---|
(a) stable-node-churn | ≥ 6 increments | x₁ … x_{n−2} (first and last excluded) | x_{n−1} | z ≥ z_threshold, z_ewma ≥ z_threshold, o > μ, μ ≤ 0.5 |
(b) spike | ≥ 6 increments | same | same | z ≥ z_threshold, z_ewma ≥ z_threshold, o > μ, μ > 0.5 |
(d) breadth-outlier | ≥ 6 snapshots in the repo | breadth of every snapshot but the last | breadth of the last | z ≥ z_threshold, o > μ |
(c) new-coupling | ≥ 6 increments per file | x₁ … x_{n−2} | x_{n−1} | ≥ 2 files of one repo with μ ≤ 0.5 and o > 0; pairs of the first 6 files (sorted), z := z_threshold, baseline 0, observed 1 |
Breadth of a snapshot = the number of files of the repo whose increment at
that snapshot is > 0. Its baseline includes the first snapshot, whose
increments are backfill totals. Rows sort by z descending, then kind
(stable-node-churn < spike < breadth-outlier < new-coupling), then subject.
(anomaly.rs:166-271, :276-342, :347-400.)
Code-vs-doc differences:
Params::min_history's doc says "≥ 3 ⇒ ≥ 2 baseline increments" (anomaly.rs:142-143). The default is 6 (MIN_HISTORY,:135).- The breadth fold orders snapshots by the STRING
"{ts}\u{1}{sid}"(anomaly.rs:296-301). That is numeric order only while everyts_microshas the same number of digits. - The test comment says b, c and d "all fired at s3" (
anomaly.rs:595). The fixture fires them at s6 (:571).
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | parse all heat generations into series | not GraphBLAS | series | change_series.rs:88-127 |
| 2 | increments, mean, std, EWMA | not GraphBLAS | per-file stats | change_series.rs:66-153 |
| 3 | four detectors | not GraphBLAS | anomalies | anomaly.rs:171-240 |
Dataflow.
git_heat_facts (ALL snapshots) ──(repo,file) series──► increments x₀..x_{n−1}
├─ n ≥ 6: baseline x₁..x_{n−2}, observed x_{n−1} ──z──► stable-node-churn | spike
├─ per repo per snapshot: #files with x>0 ──z over snapshots──► breadth-outlier
└─ quiet (μ ≤ 0.5) and fired (o > 0), ≥ 2 per repo ──pairs──► new-coupling (proxy)
▼
worst[ repo, subject, kind, baseline, observed, z, evidence_note ] ; insufficient_history
Worked example. Fixture anomaly.rs:430: src/core.rs has
commits_total 5, 5, 5, 5, 5, 5, 25 over seven scans.
x = [5, 0, 0, 0, 0, 0, 20] n = 7 ≥ 6
baseline = x₁..x₅ = [0, 0, 0, 0, 0] μ = 0, σ = 0
observed = 20 > μ, σ = 0 → z = max(3.0, 4.0) = 4.0 ≥ 3.0
μ = 0 ≤ 0.5 → stable-node-churn
Fixture anomaly.rs:495: totals 5, 10, 15, 20, 25, 30, 70 give x = [5, 5, 5, 5, 5, 5, 40], baseline [5, 5, 5, 5, 5], μ = 5, σ = 0, observed 40 > 5,
so z = 4.0. μ = 5 > 0.5, so the kind is spike.
Fixture anomaly.rs:552: file a moves at each of six snapshots; b, c,
d hold 0 and jump to 5 at s6.
breadth per snapshot: s1..s5 = 1, s6 = 4
baseline [1,1,1,1,1] μ = 1, σ = 0; observed 4 > 1 → z = 4.0 → breadth-outlier
b, c, d: x = [0,0,0,0,0,5], baseline [0,0,0,0] μ = 0, observed 5 → z = 4.0 → stable-node-churn each
a: x = [1,1,1,1,1,1], observed 1 = μ → nothing
quiet and fired: b, c, d → new-coupling b~c, b~d, c~d at z = 3.0
Fixture anomaly.rs:533: four scans give 4 increments < 6, so nothing is
judged and insufficient_history = 1.
The minimum of 6 is measured. On loki on 2026-10-16 (about 2.25 generations
per file), min_history = 3 gave 50 alarms, all "baseline 0.00, churned 1, z
4.0" (anomaly.rs:130-134, :528-531).
Output. z_threshold, nodes_seen, insufficient_history, min_history, anomalies_found, model_attribution; rows repo, subject, kind, baseline, observed, z, evidence_note (anomaly.rs:105-128).
Limits. When no file has 6 scans, nothing is judged and the answer says
"not enough history" as a gap (room_feed.rs:371-376). New coupling is a
churn-correlation proxy: git_heat_facts holds no per-commit file set
(change_series.rs:14-21).
Tests. anomaly.rs:430, :495, :518, :533, :552;
room_tests.rs:62, :83.
world_impact
Call. gRPC Mimir.WorldImpact (server_grpc.rs:3493), MCP
mimir_world_impact (mcp_stdio.rs:2253). Params repo, limit.
Math. Per repo with consumer seq s ≥ 2, the commit window s−1 → s. It
is read from the READ-FAST zone when it can answer, else folded from two as-of
rooms and Room::diff (lenses.rs:1220-1249, :1288-1371). The window is
kept per (repo, seq), two per repo (world_impact.rs:141-154).
item(n) = level Item, origin Local, defined
public(n) = effective_visibility ∈ {direct, reexported}
per crate c of the window:
added = |added item nodes| added_public = |… that are public|
removed = |removed item nodes| removed_public = |… that are public|
changed = |item nodes whose payload hash differs| (doc-only edits count)
seeds(c) = monikers of the removed and changed items (added items are no seed)
breaking = removed_public > 0
bump = major if breaking, else minor if added_public > 0, else patch
world_callers(c) = { repo(o) : o ∈ dependents(seeds(c)) } ∖ {own repo, repos starting '@'}
constellation(c) = BFS over dep_graph_edges (latest snapshot), crate names normalised (−→_, lowercase):
seeds = consumers_of_crate(c) ∪ dependents_of_repo(p) for p ∈ producers_of_crate(c);
step = dependents_of_repo(repo)
world_impact = |world_callers| + |constellation|
sort: breaking first, world_impact ↓, |world_callers| ↓, world_repo, crate_name
(world_impact.rs:187-244, :260-332; lenses.rs:1250-1280.) The diff
matches nodes by moniker in a merge of the two sorted moniker lists; equal
monikers with different payload hashes are changed
(crates/mimir-room/src/diff.rs:76-128).
GraphBLAS steps.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | READ-FAST window, or two as-of rooms and the moniker merge | not GraphBLAS | added / removed / changed nodes | lenses.rs:1288-1371; diff.rs:76-128 |
| 1b | Room::diff edge part: added = A_y⟨¬A_x⟩ per kind | GrB_Matrix_apply GrB_IDENTITY_BOOL, GrB_DESC_RSC | edge diff (computed, not read by this lens) | diff.rs:130-155; ops.rs:596-620 |
| 2 | Jikes matrix W = Jᵀ ∨ diag(impls)·CONTAINS, J = CALLS ∨ USES_TYPE ∨ IMPLEMENTS ∨ FOR_TYPE | GrB_LOR; GrB_transpose; GxB_ANY_SECOND_BOOL | the dependents graph, cached | snapshot.rs:471-488 |
| 3 | reach_sets(seed sets, W): F⟨¬V, struct, replace⟩ = F ANY.PAIR W, 64 sets per matrix | GxB_ANY_PAIR_BOOL (mxm) | dependents per touched crate | lenses.rs:1257-1258; query.rs:386-425; ops.rs:695-703 |
| 4 | dep_graph_edges BFS | not GraphBLAS (VecDeque) | constellation dependents | world_impact.rs:260-295 |
Dataflow.
code room seq s−1 → s ──(zone | as-of diff)──► added / removed / changed items per crate
│ removed + changed monikers = seed set per crate
▼
W = (CALLS ∨ USES_TYPE ∨ IMPLEMENTS ∨ FOR_TYPE)ᵀ ∨ diag(impls)·CONTAINS
reach_sets(seed sets, W) ──repo of reached ∖ own ∖ '@*'──► world_callers
dep_graph_edges (latest) ──consumers ∪ producers' dependents, BFS──► constellation_dependents
▼
worst[ world_repo, crate_name, added_items, removed_items, changed_items, suggested_bump,
breaking, world_callers, world_caller_count, constellation_dependents,
constellation_count, world_impact, explain ]
Worked example. Room fixture room_tests.rs:606. Shop's seq 2 adds
hello (pub), removes bye (pub(crate)) and removes refund (pub, called
by web).
added = 1, removed = 2, changed = 0
removed_public = 1 (refund) > 0 → breaking, bump major
seeds = {bye, refund} → dependents reach web's caller → world_callers = [web]
dep_graph_edges missing → constellation = [] (named as a missing feed)
world_impact = 1 + 0 = 1 ; world_repo "shop@1→2"
Unit fixture world_impact.rs:412: edges app → lib via lib_c, lib → util via util_c; a change to crate util-c with no callers.
norm("util-c") = util_c
consumers_of_crate[util_c] = {lib} → seen {lib}
producers_of_crate[util_c] = {util}, dependents_of_repo[util] = {lib} → already seen
BFS: dependents_of_repo[lib] = {app} → seen {lib, app}; dependents_of_repo[app] = ∅
constellation = [app, lib], world_impact = 0 + 2 = 2
Fixture world_impact.rs:369: serde removes 3 public items and has 3 caller
repos (breaking, major, impact 3). leaf adds 5 (minor, impact 0). serde
ranks first.
Output. changes_seen, breaking_changes, model_attribution; rows as in
the dataflow (world_impact.rs:88-113).
Limits. A repo needs two consumed commits (lenses.rs:1222-1225, :1273).
Callers outside the workspace and published semver diffs have no producer; the
answer always says so (lenses.rs:1276). A two-seq window folded on the fold
path builds two uncached rooms; kept windows avoid that on later asks
(room_tests.rs:633).
Tests. world_impact.rs:369, :394, :404, :412, :425, :433;
room_tests.rs:606, :633, :650.
binary_drift
Call. gRPC Mimir.BinaryDrift (server_grpc.rs:3506), MCP
mimir_binary_drift (mcp_stdio.rs:2284). Params repo, limit, history
(default 1; gRPC and MCP do not pass it, so it is always 1 from those doors,
built.rs:217). Roomless.
Math. Per line (repo, variant, file, target, sorted features), the
filings oldest first. Runs of equal sha256 collapse to the last of the run
(built.rs:224-233). Fewer than 2 distinct filings: the line is single.
Otherwise the last history successive pairs (from, to) are diffed with
nornir_introspect::diff::diff_reports of the two analyses (built.rs:242-251).
For one pair:
explained_by = source if from.commits ≠ to.commits (any repo's commit)
= toolchain else if from.toolchain ≠ to.toolchain
= nothing otherwise
per changed symbol s, kind ∈ {symbol-added, symbol-removed, symbol-grew, symbol-shrunk}:
first_party = crate(s) (− → _) equals the package, the repo, or the file name
reach = suspicious_tag(name(s)) if kind = symbol-added (14 fragments:
process-spawn, process-exit, exec, shell, network)
class = glue | runtime | derive | code (by GLUE_MARKERS, RUNTIME_PREFIXES,
DERIVE_TRAITS on the erased name)
severity = high if explained_by = nothing or reach present
= medium else if first_party
= low otherwise
rank = (severity 3/2/1, first_party, |delta_bytes|) descending, then symbol
unexplained = |{severity = high}| explained = |changes| − unexplained
attested = unexplained = 0 and at least one analysed pair
(binary_drift.rs:749-807, :811-821, :869-981.) Features are part of the
line, so they never differ inside a pair. provenance_changed lists commits
and toolchain; its doc also names builder … (binary_drift.rs:786-787), which
the code does not compare. The JSON field repos_with_drift carries the number
of LINES with drift (binary_drift.rs:1033, :948-950).
The module's source-versus-binary fold (report_from, fold_repo,
binary_drift.rs:524-624) has no feed today: no room holds linked symbols, so
the wire lens answers successive only (binary_drift.rs:57-64).
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | read and dedupe filings, group by line | not GraphBLAS | lines | built.rs:62-110 |
| 2 | collapse equal sha256, pair successive | not GraphBLAS | steps | built.rs:223-252 |
| 3 | diff_reports(a, b) | not GraphBLAS (nornir_introspect) | symbol / section / crate deltas | built.rs:247 |
| 4 | classify, severity, rank | not GraphBLAS | changes | binary_drift.rs:880-981 |
Dataflow.
bin_ingest_runs(@build-thing) ──► filings per line (oldest first) ──collapse equal sha256──► distinct
│ windows(2), last `history`
▼
symbol_sizes[from.sha256], symbol_sizes[to.sha256] ──diff_reports──► added / removed / grew / shrunk
provenance(from, to) ──► explained_by ∈ {source, toolchain, nothing}
▼
severity, class, reach, rank
▼
attested, attestation, steps[…], single, cant_catch,
findings/worst[ repo, variant, artifact, binary_symbol, demangled, kind, severity, crate, class,
before, after, delta_bytes, first_party, explained_by, from/to_commit, from/to_sha ]
Worked example. Real build room_feed/built_tests.rs:200: the probe crate
is built and filed twice; the second commit replaces the body of checksum.
sha256(first) ≠ sha256(second) → 2 distinct filings → 1 pair
commits differ → provenance_changed[0] = "commit binlens-probe …" → explained_by = source
checksum: crate binlens-probe = repo → first_party; not an added network/process reach
severity = medium (not "nothing", no reach, first party)
no high change → unexplained = 0, one analysed pair → attested = true
rank: the first-party change sorts above dependency churn (findings[0].first_party = true)
A single filing gives empty_gap with "filed once … ⚒ Build it again after a
change" and attestation UNKNOWN (built_tests.rs:190-192). The test records
no byte counts.
Output. As in the dataflow; also binary_symbols_seen,
first_party_symbols, explained, unexplained, by_kind,
edges_evaluated: false (binary_drift.rs:1003-1043).
Limits. One filing per line has nothing to compare (built.rs:254-258).
Sizes, not bytes: a body change that keeps every size shows only as a
different sha256 (built.rs:265). Binary call edges have no producer, so an
injected call inside an unchanged symbol is not evaluated (built.rs:266).
The full list is SUCCESSIVE_CANT_CATCH (binary_drift.rs:707-713).
Tests. room_feed/built_tests.rs:149, :200 (the successive fold);
binary_drift.rs:1070, :1082, :1105, :1130, :1138, :1148, :1162,
:1172, :1201 (the source-versus-binary fold).
data_reach
Call. gRPC Mimir.DataReach (server_grpc.rs:3518), MCP
mimir_data_reach (mcp_stdio.rs:2314). Params repo, limit, class
(syscall classes; gRPC and MCP do not pass it).
Math. code = every Local, defined node of the request's repos
(lenses.rs:1074). One row per TOUCHES_TABLE edge f → table with f ∈ code:
reachers(f) = sorted, deduplicated display paths of
dependents_by(f, CALLS, within code) ∖ {f} -- backward CALLS walk inside code
reach_breadth = |reachers(f)| -- the full count
public_entry = f or some reacher has an effective visibility that is public API
sensitivity = label of the FIRST of 14 SENSITIVE_PATTERNS whose substring the lowercased
table name contains: pii, personnummer, person → pii; secret → secret;
token, cred, password, passwd, apikey, api_key, vault, key_material
→ credential; saga, audit → audit-pii; else ""
writes = access ∈ {write, both}
sort: sensitive first, writes first, reach_breadth ↓, public_entry first, (table, accessor_fn)
per table: readers = Σ [access ≠ write], writers = Σ [access ≠ read], sites = Σ site_count,
reach = |{accessor_fn} ∪ reachers| over its rows
leak path = a row with sensitivity ≠ ""
A table name not known at compile time is no row: it goes to unresolved,
sorted by site count (data_reach.rs:222-303). reachers lists at most 64
names; reach_breadth stays exact (data_reach.rs:57, :254-255). Edges
without access read unknown (lenses.rs:1135-1152).
The module doc lists "key" among the sensitive name fragments
(data_reach.rs:18-19). There is no bare key pattern; only apikey,
api_key and key_material (data_reach.rs:39-54).
The walk has two paths. The reference path (NORNIR_REACH_INC=0) walks every
toucher (lenses.rs:1099-1115). The default path is the kept walk
reach_inc::table_reachers (lenses.rs:1116-1122). It fingerprints every
code node by its CALLS successors inside code, its path and its API flag.
It re-walks only the touchers that changed, that a changed node reaches now, or
whose kept reachers hold a changed node. The others keep their reachers
(reach_inc.rs:779-888). The two paths must give equal answers
(reach_tests.rs:196).
GraphBLAS steps.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | edge read TOUCHES_TABLE code → * | not GraphBLAS (Room::edges) | touches with access, sites | lenses.rs:1079 |
| 2 | A_CALLSᵀ, cached | GrB_transpose | the backward graph | snapshot.rs:446-451 |
| 3 | within code: diag(code) ANY.SECOND Aᵀ ANY.FIRST diag(code) | GxB_ANY_SECOND_BOOL, GxB_ANY_FIRST_BOOL | walks stay inside the request's code | query.rs:367-374; ops.rs:409-436 |
| 4 | reach_each(touchers, Aᵀ_code): F⟨¬V, struct, replace⟩ = F ANY.PAIR Aᵀ_code, 64 per matrix | GxB_ANY_PAIR_BOOL (mxm) | reachers per toucher | query.rs:438-461, :375; ops.rs:695-703 |
| 4' | kept path: reach(changed, CALLS, within code) forward, then step 4 for the affected touchers only | GxB_ANY_PAIR_BOOL (vxm, then mxm) | which touchers to re-walk | reach_inc.rs:828, :845 |
| 5 | sensitivity, sort, per-table fold | not GraphBLAS | rows, tables | data_reach.rs:222-303 |
| 6 | syscall classes from the ring room (when the lake holds syscall nodes) | see system_boundary | syscalls[…] | lenses.rs:1165-1199 |
Dataflow.
code_edges(TOUCHES_TABLE) from code ──► (f, table | unresolved, access, sites, count)
A_CALLSᵀ ─mask_by(code)─► Aᵀ_code ──reach_each(touchers) ANY.PAIR──► reachers(f)
effective_visibility(f, reachers) ──► public_entry
table name ──SENSITIVE_PATTERNS──► sensitivity
▼
sort (sensitive, writes, breadth, public) ; per-table readers / writers / sites / reach
▼
worst[ table, sensitivity, accessor_fn, crate_name, repo, access, sites, site_count,
reachers (≤ 64), reach_breadth, public_entry, explain ] ; tables ; unresolved ; syscalls
Worked example. Room fixture room_tests.rs:721, repo shop: 4 edges
with a known table, 3 tables, 2 leak paths.
| table | accessor | access | sensitivity (pattern) | reachers | breadth | public |
|---|---|---|---|---|---|---|
audit_log | shop::pay::audit | write | audit-pii (audit) | checkout, pay::charge, pay::charges_once | 3 | true |
payment_tokens | shop::pay::charge | write | credential (token) | (2) | 2 | |
orders | shop::pay::dead | both | "" | none | 0 | false |
orders | shop::cart::open | read | "" | Cart::total, tests::adds_up, checkout | 3 |
rank: audit (sensitive, write, 3) > charge (sensitive, write, 2)
> dead (not sensitive, both = write) > open (not sensitive, read)
orders: readers = [both ≠ write] + [read ≠ write] = 2 ; writers = [both ≠ read] = 1
sites = 2 ; reach = |{dead} ∪ {open, Cart::total, adds_up, checkout}| = 5
unresolved: shop::pay::shell, 2 sites (src/pay.rs:33, :35)
Unit fixture data_reach.rs:367: one table, three fns. append (write,
reachers cli::run, srv::submit), query (read, cli::run, viz::pane),
compact (both, none). readers = 0 + 1 + 1 = 2, writers = 1 + 0 + 1 = 2,
sites 3, reach = |{append, cli::run, srv::submit, query, viz::pane, compact}| = 6.
Output. access_edges_seen, tables_seen, tables[table, sensitivity, readers, writers, sites, reach], sensitive_tables, leak_paths_seen, unresolved_edges, unresolved_sites, unresolved[…], model_attribution, syscalls, syscall_functions_reaching, syscall_functions_seen; rows as in the
dataflow (data_reach.rs:187-218; lenses.rs:1175-1195).
Limits. Closures, fn pointers, trait impls the producer did not see, and
table names inside SQL text are a lower bound (lenses.rs:1035-1036). A lake
made before mimir-schema 4 cannot tell "touches no table" from "not yet
written" (lenses.rs:1078-1088). Without effective_visibility,
public_entry is false everywhere and named as a gap (lenses.rs:1123-1125).
Folding the dependency universe for the syscall part cost 8.2 M nodes and 951 s
on the lake copy on 2026-10-19; it is skipped when the lake holds no syscall
node (lenses.rs:1168-1179).
Tests. data_reach.rs:330, :351, :367, :380, :397, :407;
room_tests.rs:721, :765, :787; room_feed/reach_tests.rs:120, :196.
trust
- Call. gRPC
Mimir.Trust, MCPmimir_trust, CLInornir mimir analyze trust --server <url> --repo <repo> [--act-at 0.66] [--hold-at 0.33] [--limit N] [--svg]. Repo-scoped (REPO_SCOPED,src/mimir_lens/mod.rs:146). A parameter that is 0 or missing keeps the default (room_feed/lenses.rs:986-991). - Finds. For every program function: act, hold or refuse. The verdict is the
best call chain from the entry surface to the function, scored by its weakest
hop. The weakest hop is named (
trust.rs:1-46).
Dataflow.
code room (Mímir v2 tables)
code_nodes fn + method, minus is_test and #[cfg(test)] code ──► scope S (mask) lenses.rs:865-866
code_edges CALLS S → impure items (villa::IMPURE_NAMES) ──► impure kinds per fn lenses.rs:874-901
code_edges CALLS from is_test nodes, plus TESTS ──► covered set C graph.rs:216-228
code_edges CALLS S → S ──► A (boolean n×n), entry set E
│
local trust ℓ(v) per fn ──► weight w(v) = round(1000·ℓ)+1 ──────┤
▼
ops::bottleneck(E, A⟨S⟩, w) (MAX.FIRST, MIN) ──► b(v)
│
t(v) = (b(v) − 1)/1000 ──► Bounds::decide ──► act / hold / refuse
witness: Room::path (MIN.PLUS layers) + hops_from ──► weakest_hop, depth
▼
output: act, hold, refuse, unreachable, worst[func, crate_name, decision,
inherited_trust, local_trust, weakest_hop, reason, depth, explain]
Math.
Local trust of one function v (trust.rs:323-358). Let K(v) be the impurity
kinds of the impure items v calls, T(v) its sensitive table touches, and
fed = the scope holds at least one test node.
penalty(kind): Shell = 0.8, ExternalTool = 0.8, Process = 0.5, ProcessControl = 0.3 trust.rs:163-169
sens(access): write = 0.5, any other access = 0.25 trust.rs:172-174
untested: 0.2 trust.rs:177
ℓ₀ = 1 − max_{k ∈ K(v)} penalty(k) (1.0 when K(v) is empty)
ℓ₁ = clamp₀¹( ℓ₀ − Σ_{(a,t) ∈ T(v), t sensitive} sens(a) ) each touch stacks
ℓ = clamp₀¹( ℓ₁ − 0.2 ) if fed and v ∉ C, else ℓ₁
So impurity takes the WORST penalty, while sensitive touches and the untested
penalty SUBTRACT on top of it. The production lens never fills T(v)
(access: Vec::new(), room_feed/lenses.rs:915), so ℓ₁ = ℓ₀ there, and the
answer always carries the gap "the sensitive-data term of local trust is NOT
applied" (lenses.rs:907).
The integer weight scale (trust.rs:182-196): w(v) = round(1000·ℓ(v)) + 1, so
w ∈ [1, 1001]. The +1 keeps "reached with trust 0" (weight 1) apart from
"not reached" (no entry). The way back is t = (w − 1)/1000.
The entry surface (lenses.rs:922-932): E = { v ∈ S : no u ∈ S, u ≠ v, has CALLS u → v }. When E is empty (a fully cyclic scope), E = S.
Inherited trust is the widest path (bottleneck) value, computed as a frontier
fixpoint on GraphBLAS (crates/mimir-graph/src/ops.rs:329-403):
A′ = diag(m_S) · A · diag(m_S) (mask_by, ops.rs:409-436)
b⁰ = w⟨E⟩ f⁰ = b⁰ (ops.rs:337)
repeat while nvals(f) > 0:
c(j) = max_{i : A′(i,j)} f(i) c = f ⊕.⊗ A′, ⊕ = MAX, ⊗ = FIRST
c = min(c, w) eWiseMult (intersection)
g = (c > b) eWiseMult GT (where both exist)
f = c⟨g, replace⟩ ; f⟨¬struct(b)⟩ = c (risen values, then first-time values)
b = max(b, c) eWiseAdd (union)
fixpoint: b(v) = max_{paths P: e ⇝ v, e ∈ E} min_{u ∈ P} w(u)
t(v) = (b(v) − 1)/1000, or 0 with "unreachable" when v has no b(v)
The values only rise and are drawn from the finite set of weights, so the loop ends. The seed's own weight caps every path from it (the path includes both ends).
Decision (trust.rs:131-158): t ≥ act_at (0.66) ⇒ act; else t ≥ hold_at
(0.33) ⇒ hold; else refuse. An unreached node counts in unreachable and in
refuse (trust.rs:382-406).
Ranking (trust.rs:382-406): only non-act verdicts are listed, sorted by
decision (refuse before hold), then t ascending, then function name, then
index; cut at limit (default 50).
Weakest hop (lenses.rs:994-1015). For a listed node v with value b(v):
caps(v) = { u : w(u) = b(v) and b(u) = b(v) }
at_least(v) = { u : b(u) ≥ b(v) }
weakest(v) = v if v ∈ caps(v)
= first node of Room::path(caps(v) → v, CALLS, within at_least(v)) otherwise
depth(v) = Room::hops_from(E, CALLS, within S)(v)
The path is the shortest one; ties go to the smallest target ordinal, then the
smallest predecessor ordinal (crates/mimir-room/src/path.rs:37-44).
The kept walk (trust.rs:479-780). The lens keeps the last walk per workspace
and repo, keyed by moniker hash, and plans Carry (nothing the walk reads
changed), Advance (locals rose, nodes or edges appeared: a raise pass; a
local fell: re-walk its forward cone, or the whole graph when the cone holds
over half the nodes, trust.rs:689-697), or Full (entry surface moved, a
node or edge is gone, a key collides). Both cheap paths are pop-the-max
Dijkstra over the same weights and must equal ops::bottleneck.
GraphBLAS steps. GxB_ANY_PAIR_BOOL on a boolean matrix is the structural
OR-AND (LOR.LAND) reach.
| step | operation | semiring / op | computes | file:line |
|---|---|---|---|---|
| 1 | v⟨¬visited⟩ = q · A_CALLS per hop, seeds = test nodes | GxB_ANY_PAIR_BOOL (vxm) | the covered set C | graph.rs:224, ops.rs:66-74 |
| 2 | edge extraction CALLS S → impure, S → S | none (row scan of the adjacency) | K(v), entry surface | lenses.rs:876, :922; crates/mimir-room/src/export.rs:186 |
| 3 | A′ = diag(m) · A · diag(m) | GxB_ANY_SECOND_BOOL, then GxB_ANY_FIRST_BOOL (mxm) | CALLS inside the scope | ops.rs:409-436 |
| 4 | c = f · A′ | GxB_MAX_FIRST_UINT64 (vxm) | best value offered to each successor | ops.rs:344-352 |
| 5 | c = c ∘ w | GrB_MIN_UINT64 (eWiseMult) | the hop is capped by the node entered | ops.rs:355-363 |
| 6 | g = c > b | GrB_GT_UINT64 (eWiseMult) | where the value rose | ops.rs:365-373 |
| 7 | f⟨g, replace⟩ = c; f⟨¬s(b)⟩ = c | GrB_IDENTITY_UINT64 (apply, masked) | next frontier | ops.rs:375-393 |
| 8 | b = b ∪max c | GrB_MAX_UINT64 (eWiseAdd) | best value so far | ops.rs:394-401 |
| 9 | hops_from(E) | GrB_MIN_PLUS_SEMIRING_UINT64 (vxm) | depth | crates/mimir-room/src/query.rs:540-558, ops.rs:170-200 |
| 10 | Room::path(caps → v) | GrB_MIN_PLUS_SEMIRING_UINT64 layers (min_hops_to), back-step on the cached transpose | weakest hop | path.rs:85-118, ops.rs:212-262 |
| 11 | kept walk (Carry / Advance) | not GraphBLAS: binary-heap widest path | the same b | trust.rs:663-760 |
Worked example (fixture a_pure_leaf_behind_a_shellout_hop_is_refused_not_acted,
trust.rs:973-1001). The chain is app::main → app::spawn_shell → app::format_arg,
and spawn_shell calls libc::system (an ffi node, so villa::classify gives
Shell, villa.rs:212-217). No coverage input.
ℓ(main) = 1.0 w = 1001
ℓ(spawn_shell) = 1 − 0.8 = 0.2 w = 201
ℓ(format_arg) = 1.0 w = 1001
E = {main} (nobody calls it)
b⁰ = {main: 1001} f = {main: 1001}
hop 1: c(spawn_shell) = 1001, min(1001, 201) = 201; new ⇒ f = {spawn_shell: 201}, b += 201
hop 2: c(format_arg) = 201, min(201, 1001) = 201; new ⇒ f = {format_arg: 201}, b += 201
hop 3: format_arg has no callee ⇒ c empty ⇒ f empty ⇒ stop
t(format_arg) = (201 − 1)/1000 = 0.20 < 0.33 ⇒ refuse
caps = {u : w(u) = 201, b(u) = 201} = {spawn_shell} ⇒ weakest_hop = app::spawn_shell
depth = 2
The test asserts exactly this: local 1.0, inherited 0.2, refuse, weakest hop
app::spawn_shell, depth 2. The sibling fixture
max_over_paths_takes_the_cleaner_route (trust.rs:1021-1036) gives target
two entries: clean_entry → target offers 1001 and
dirty_entry → shell → target offers min(1001, 501, 1001) = 501
(Command::new is Process, 1 − 0.5 = 0.5). The max is 1001, so target is
act and absent from the list. With coverage, an untested helper is 1 − 0.2 = 0.8 (trust.rs:1073-1083).
- Output.
nodes_seen, entry_points, act, hold, refuse, unreachable, bounds, coverage_fed, evidence_note, feeds, and rowsfunc, crate_name, decision, inherited_trust, local_trust, weakest_hop, reason, depth, explain(trust.rs:261-289). Withsvg=true, a card grid (verdict_svg.rs:61). - Limits. The sensitive-data term is not applied (no TOUCHES_TABLE read).
Without a test node, the untested penalty is dropped and the note says trust
is an upper bound (
trust.rs:452-457). CALLS is a lower bound (closures, fn pointers). - Code vs comment. The existing manual says local trust "takes the worst
penalty". The code takes the worst only among impurity kinds; sensitive
touches and the untested penalty subtract on top (
trust.rs:323-358). - The external-tool hop.
villa::classifyreturnsProcessfor aCommand::new. The trust feed then reads the call site from the consumed checkout (villa::binary_at,villa.rs:157); when the site spells its binary as a string literal,villa::named_toolupgrades the kind toExternalTool(penalty 0.8, as villa ranks it;villa.rs:230,room_feed/lenses.rs:876-904). The answer carries a note with the count of such hops. Without a checkout, or with the binary in a variable, the hop keeps the process penalty 0.5. (Before nornir 600e22f0 the external-tool penalty could never fire.) - Tests.
trust.rs:933-1101(semiring laws, weight scale, the decision fixtures),:1146-1375(plan, incremental equals full over random mutations, scale, upper bound);room_feed/room_tests.rs:443,:475(the room walk equals the reference walk),:536,:560,:585;tests/support/mcp_harness.rs:1340.
name_collision
- Call. gRPC
Mimir.NameCollision, MCPmimir_name_collision, CLInornir mimir analyze name-collision --server <url> [--repo R] [--limit N]. - Finds. How many CALLS edges land on a node id the fold had to merge,
beside the leaf-name baseline a name matcher would have
(
name_collision.rs:1-23).
Dataflow.
code_nodes fn + method of the request ──► defs (repo, leaf name) lenses.rs:772-774
code_edges CALLS from own (origin = Local) nodes ──► E (one row per from→to pair)
per edge: callee endpoint_only? callee leaf name, callee id lenses.rs:777-790
Room::duplicate_keys(within callables) ──► merged ids M (rows, owners) symbols.rs:82
│
hash maps: leaf counts, inbound(m) ─┴─► conflation_pct, leaf_conflation_pct, worst[…]
Math (name_collision.rs:180-246, production inputs lenses.rs:768-811):
E = CALLS (from, to) pairs whose source is own code (distinct pairs, not sites)
edges_seen = |E|
unresolved = |{ e ∈ E : to(e) is endpoint_only }|
resolved = edges_seen − unresolved
M = ids with distinct_payloads > 1 or key_dup_rows > 0 (callables only)
inbound(m) = |{ e ∈ E : to(e) = m }|
conflated = Σ_{m ∈ M} inbound(m)
pinned = resolved − conflated
conflation_pct = 100 · conflated / resolved (0 when resolved = 0)
defs = callable definitions (repo filter applied)
colliders = { name : |{ d ∈ defs : leaf(d) = name }| ≥ 2 }
leaf_conflated = Σ_{name ∈ colliders} |{ e ∈ E : leaf(to(e)) = name }|
leaf_conflation_pct = 100 · leaf_conflated / edges_seen
Each merged id becomes a row with distinct_defs = max(rows, 2)
(lenses.rs:801). Rows sort by cross_repo first, then conflated_edges
descending, then distinct_defs descending, then name (name_collision.rs:224-232).
GraphBLAS steps.
| step | operation | semiring | computes | file:line |
|---|---|---|---|---|
| 1 | CALLS out of the own mask | none (row scan of the adjacency) | E | lenses.rs:779, export.rs:186 |
| 2 | merged keys | none (the snapshot's dup list, filtered by mask) | M | symbols.rs:82-89 |
| 3 | counts | none (hash maps) | every figure | name_collision.rs:180-246 |
Worked example (same_named_fns_are_pinned_and_only_a_merged_id_is_conflated,
name_collision.rs:266-278). Definitions new (a), new (a), new (b),
parse, helper; 10 CALLS edges: 6 to a new, 3 to parse, 1 to helper;
one merged id a::cart::helper with 1 inbound edge.
callable_names = |{new, parse, helper}| = 3; colliders = {new} (3 defs); cross_repo_colliders = 1
resolved = 10 − 0 = 10; conflated = 1; pinned = 9
conflation_pct = 100 · 1/10 = 10.0
leaf_conflation_pct = 100 · 6/10 = 60.0
The 50-point gap is what rustc-resolved ids bought.
- Output.
callable_names, colliding_names, cross_repo_colliders, edges_seen, resolved_edges, pinned_edges, conflated_edges, unresolved_edges, conflation_pct, leaf_conflation_pct; rowsname, distinct_defs, distinct_repos, conflated_edges, cross_repo, sites (≤ 6), explain(name_collision.rs:107-129). - Limits. A room without CALLS reads as a missing feed, never "sound".
- Code vs comment. The module promises a CROSS-REPO class ("two repos
disagree about one id"). The production lens fills
reposwith the defining repo only (lenses.rs:797-798), sodistinct_reposis at most 1 andcross_repois always false on a live room; only the unit tests reach the cross-repo ordering (name_collision.rs:280-286). - Tests.
name_collision.rs:266-311(5 tests);room_tests.rs:399.
symbol_identity
- Call. gRPC
Mimir.SymbolIdentity, MCPmimir_symbol_identity, CLInornir mimir analyze symbol-identity --server <url> [--repo R] [--limit N]. - Finds. The merged ids of the request's own definitions, split by cause,
and a one-word verdict on the one-definition-per-id invariant
(
symbol_identity.rs:1-27).
Dataflow.
code_nodes own definitions (g.local) ──► ids = |own| lenses.rs:820-821
Room::duplicate_keys(all) ──► per merged id: rows, distinct_payloads, key_dup_rows, owners
keep: defining repo in the request and node ∈ own; else count "elsewhere" lenses.rs:823-844
spans ──► files (distinct span files), sites lenses.rs:757-763
│
classify ──► cause ──► by_cause, verdict, doubt_ratio, worst[…]
Math (symbol_identity.rs:178-257):
cause(k) = repeated_key_in_unit if key_dup_rows(k) > 0
= one_id_several_places else if files(k) > 1
= units_disagree otherwise
ours(cause) ⇔ cause ∈ {repeated_key_in_unit, one_id_several_places}
rows_under_ambiguous = Σ_k rows(k)
definition_rows = ids + Σ_k (rows(k) − 1)
doubt_ratio = rows_under_ambiguous / definition_rows (0 when definition_rows = 0)
verdict = unknown if ids = 0
= sound if no merged id
= units_disagree_only if no merged id is ours
= unsound otherwise
Rows sort by ours first, then definitions (= rows) descending, then repo, then
symbol (symbol_identity.rs:227-235).
GraphBLAS steps. None. The lens reads the snapshot's dup list and the span
table (symbols.rs:82-89, lenses.rs:757-763) and folds them with maps.
Worked example (a_key_repeated_in_one_unit_is_unsound_and_units_disagreeing_is_not,
symbol_identity.rs:282-298). ids = 10; keys a helper(). (rows 2,
key_dup_rows 0, one file) and a impl#[T]new(). (rows 3, key_dup_rows 2).
cause(helper) = units_disagree; cause(new) = repeated_key_in_unit (ours)
rows_under_ambiguous = 2 + 3 = 5
definition_rows = 10 + (2−1) + (3−1) = 13
doubt_ratio = 5/13 = 0.3846
verdict = unsound; worst[0] = new (ours ranks first)
- Output.
verdict, verdict_label, symbols_seen, ambiguous_symbols, definition_rows, rows_under_ambiguous, doubt_ratio, by_cause, repos_with_repeated_keys; rowsrepo, symbol, kind, cause, cause_label, definitions, files, sites(symbol_identity.rs:126-158). - Limits. Merged ids of dependency stubs and binary-only nodes are counted
in a note, not judged (
lenses.rs:845-847). - Tests.
symbol_identity.rs:282-330(5 tests);room_tests.rs:415.
reachable_vuln
- Call. gRPC
Mimir.ReachableVuln, MCPmimir_reachable_vuln, CLInornir mimir analyze reachable-vuln --server <url> [--repo R] [--limit N] [--svg]. - Finds. An advisory
crate@versionthat a repo resolves: refuse when the repo's own code calls or names an item of that crate at that version, hold when it is only present.actis never emitted (reachable_vuln.rs:1-50).
Dataflow.
warehouse vuln_findings (crate, version, ids, summary) ──► advisories map A reachable_vuln.rs:203-217
warehouse sbom_components, latest (repo, name, version) ──► candidates C = sbom ⋈ A :220-236
code room CALLS ∪ USES_TYPE from own (origin = Local) nodes ──► U(repo, crate, version) lenses.rs:691-702
│
hash join on (norm crate, version) ──► verdict refuse/hold, call_sites, rank
│
for each row: ring room (universe folded in) ── reach over CALLS ∪ SYSCALLS ──► boundary{classes, syscalls, entries_called, path}
Math.
norm(c) = trim(c) with '-' → '_' reachable_vuln.rs:36-38
A = { (norm(crate), trim(version)) ↦ (ids, summary) } (a later row with the same key replaces an earlier one)
C = { (repo, norm(name), version) : row of latest sbom_components, (norm(name), version) ∈ A, repo filter }
U(r, c, v) = Σ count(e) over e ∈ CALLS ∪ USES_TYPE, from a Local node of repo r,
to a node of crate c at version v, skipping a Local target in the same repo r
call_sites = U(r, c, v) (0 when absent)
verdict = refuse if call_sites > 0, else hold
Rank (reachable_vuln.rs:268-283): refuse before hold; then "exploit" first,
where exploit ⇔ the lowercased summary contains [public-exploit] or kev;
then call_sites descending; then repo; then crate. Cut at limit.
The boundary of each listed finding (room_feed/boundary.rs:453-511):
entries = callees of crate@version that the repo's own fns call; reach over
CALLS ∪ SYSCALLS from entries; classes = the classes of the syscall nodes
reached; one shortest path to the most severe class in the order network,
process, file, system, ipc, signal, memory, event, time, random, then indirect
(boundary.rs:446-447), prefixed by our calling function.
GraphBLAS steps.
| step | operation | semiring | computes | file:line |
|---|---|---|---|---|
| 1 | CALLS ∪ USES_TYPE out of the own mask | none (row scan) | U | lenses.rs:694 |
| 2 | advisory ⋈ sbom ⋈ U | none (hash join, BTreeMap) | candidates, verdicts | reachable_vuln.rs:200-262 |
| 3 | forward reach from the crate's entries over CALLS ∪ SYSCALLS | GxB_ANY_PAIR_BOOL (vxm BFS) | the syscalls reachable | boundary.rs:475, ops.rs:66-74 |
| 4 | shortest path to the shown class | GrB_MIN_PLUS_SEMIRING_UINT64 layers | boundary.path | boundary.rs:482, path.rs:85-118 |
| 5 | kept walks per finding | carry when entries and the universe half held | the same boundary | boundary.rs:536-608 |
Worked example (reachable_is_refused_present_but_unreachable_is_held,
reachable_vuln.rs:314-335). vuln_findings: time 0.1.45 RUSTSEC-2020-0071.
sbom_components: appA and appB both resolve time 0.1.45. Uses:
U(appA, time, 0.1.45) = 1, U(appB, serde, 1.0.0) = 9.
findings_seen = 1; C = {(appA, time, 0.1.45), (appB, time, 0.1.45)}
appA: call_sites = 1 > 0 ⇒ refuse
appB: call_sites = U(appB, time, 0.1.45) = 0 ⇒ hold (serde's 9 uses do not count)
order: appA (refuse) before appB (hold)
A second fixture shows that a use of time 0.3.0 does not make time 0.1.45
reachable (reachable_vuln.rs:359-365). On the net fixture the hyperish
finding's boundary is classes = ["network"] with the path fetch → get → … → syscall connect #42 (network) (room_feed/boundary_tests.rs:84-95).
- Output.
findings_seen, components_seen, reachable_refuse, present_hold; rowsrepo, crate, version, advisory_ids, summary, reachable, call_sites, verdict, colour, explain, boundary(reachable_vuln.rs:146-166,lenses.rs:712-741). - Limits. Reachable means a DIRECT edge from own code: a crate used only
through another dependency reads
hold(lenses.rs:704-707). An unfedvuln_findings(producermodgunn warehouse scan --persist) is a missing feed, never "clean". - Code vs comment. The module calls the marker "a
[public-exploit]/ KEV marker". The code tests the substringkevanywhere in the lowercased summary (reachable_vuln.rs:268-271), so any word that contains those three letters also sorts first. - Tests.
reachable_vuln.rs:314-372(5 tests);room_tests.rs:348;boundary_tests.rs:84;reach_tests.rs:138.
undoc_api
- Call. gRPC
Mimir.UndocApi, MCPmimir_undoc_api, CLInornir mimir analyze undoc-api --server <url> --repo R [--min-commits-90d N] [--limit N]. Repo-scoped. - Finds. Public functions without a doc comment in files that many commits
and authors change (
undoc_api.rs:1-20).
Dataflow.
code_nodes fn + method (attrs effective_visibility, has_doc, doc_hidden; mimir-schema ≥ 2)
minus tests, #[cfg(test)] code, methods of trait impls (IMPLEMENTS), #[doc(hidden)], spanless
keep effective_visibility ∈ {direct, reexported} ──► api[] lenses.rs:516-570
warehouse git_heat_facts (latest) (repo, file, commits_90d, commits_total, authors_total) ──► churn map
│
join on (repo, file); floor on commits_90d ──► rank ──► worst[…] undoc_api.rs:122-196
Math (undoc_api.rs:122-196):
floor = 2 if min_commits_90d ≤ 0, else min_commits_90d
churn(repo, file) = (max commits_90d, max commits_total, max authors_total) over its rows
pub_fns = |api| after the repo filter
undocumented_pub_fns = |{ f : ¬has_doc(f) }|
finding f ⇔ ¬has_doc(f) ∧ churn(repo_f, file_f) exists ∧ commits_90d ≥ floor
rank: commits_90d ↓, authors_total ↓, commits_total ↓, repo ↑, func ↑
Churn is a file property: every undocumented public function in a hot file inherits the file's heat.
GraphBLAS steps. None. One edge extraction (IMPLEMENTS sources, row scan,
lenses.rs:529) and a hash join.
Worked example (only_undocumented_pub_fns_in_churny_files_rank_by_churn,
undoc_api.rs:221-247). Four public fns; heat: hot.rs 40/120/5,
warm.rs 6/20/2, cold.rs 1/3/1; floor 2.
pub_fns = 4; undocumented = {hot_api, warm_api, cold_api} = 3 (documented_api has a doc)
hot_api: 40 ≥ 2 ✓ warm_api: 6 ≥ 2 ✓ cold_api: 1 < 2 ✗
churny_undocumented = 2; order: hot_api (40) before warm_api (6)
- Output.
pub_fns, undocumented_pub_fns, churny_undocumented, min_commits_90d; rowsrepo, func, file, line, commits_90d, commits_total, authors_total, explain(undoc_api.rs:79-97). - Limits. A room without
effective_visibilityorhas_docnames the gap (lenses.rs:520-524). A file with no heat row is not ranked. - Tests.
undoc_api.rs:221,:250;room_tests.rs:302.
system_boundary
- Call. gRPC
Mimir.SystemBoundary(also the code-room doorCodeRoom.Answer), MCPmimir_system_boundary. Paramsrepo,class(comma list; an unknown class isinvalid_argument,boundary.rs:400-408),limit. Repo-scoped and ring-on: the room has the dependency universe folded in (RING_ON,src/mimir_lens/mod.rs:133). - Finds. Per repo, which classes of system call (network, process, file,
system, ipc, signal, memory, event, time, random) its functions can reach,
each function with one shortest path you can show (
boundary.rs:1-19).
Dataflow.
ring room (own repo + std + dependencies, mimir_ingest::boundary written once per toolchain/target)
code_nodes kind syscall (ty = class) ──► targets per class boundary.rs:163-186
code_nodes own ffi (link_name) ── libc_map::lookup ──► extra targets, call name boundary.rs:189-206
code_nodes own fn + method (origin Local, defined) ──► O (seeds) boundary.rs:168-175
A_W = A_CALLS ∨ A_SYSCALLS
│
fwd = reach(O, A_W) (ANY.PAIR BFS)
per class κ: hit_κ = targets_κ ∩ (fwd ∪ O)
back_κ = dependents_by(hit_κ, A_W) (BFS on the transpose)
fns_κ = back_κ ∩ O
border_κ = { f ∈ fns_κ : CALLS f → (back_κ ∪ hit_κ) \ O }
rows: first per_class of fns_κ ordered (¬border, repo, path)
path = Room::path(f → hit_κ, A_W) (MIN.PLUS layers)
▼
output: classes[class, functions, syscalls, from[rows]], functions_seen, functions_reaching,
unmapped_own_ffi, worst
Math.
WALK = {CALLS, SYSCALLS} boundary.rs:34
per_class = clamp(limit, 1, ROWS_PER_CLASS = 12) boundary.rs:37, :421
functions_seen = |O|
functions_reaching = |⋃_{κ asked} fns_κ|
classes sorted by |fns_κ| ↓, then class name boundary.rs:219
row order inside a class: border first, then repo, then display path boundary.rs:264
uncertain(row) = the last hop is a SYSCALLS edge flagged partial, or the C map is not sure
hops(row) = edges on the path (the C-map suffix of an own ffi adds a label, not a hop)
kernel_entry = the first CALLS edge out of the syscall node (from the kernel BTF)
worst = the classes' rows concatenated, cut at limit boundary.rs:423
Path choice (crates/mimir-room/src/path.rs:37-44): fewest hops to any target
of the class; among those the target with the smallest ordinal; at every hop
back the predecessor with the smallest ordinal. The default path is the kept
walk (reach_inc::boundary_sets, per-function class bit masks advanced over
the room's delta, boundary.rs:211-216, reach_inc.rs:428).
NORNIR_REACH_INC=0 forces the reference walk above; the tests hold the two
equal.
GraphBLAS steps.
| step | operation | semiring | computes | file:line |
|---|---|---|---|---|
| 1 | fwd = reach(O, A_W): q⟨¬v⟩ = q · A_W per hop | GxB_ANY_PAIR_BOOL (vxm), the structural OR-AND reach | everything our code reaches | boundary.rs:242, ops.rs:66-74 |
| 2 | back_κ = dependents_by(hit_κ) on the cached A_Wᵀ | GxB_ANY_PAIR_BOOL | who reaches the class | boundary.rs:253, query.rs:333-355 |
| 3 | border: CALLS edges fns_κ → onward | none (row scan) | border | boundary.rs:260-262 |
| 4 | row path | GrB_MIN_PLUS_SEMIRING_UINT64 (min_hops_to), predecessors by transpose rows | path, hops | boundary.rs:363, ops.rs:212-262 |
| 5 | kept walk: per-function walks batched | GxB_ANY_PAIR_BOOL (mxm frontier matrix) | class bit masks | reach_inc.rs:406-413, ops.rs:672-699 |
Worked example (fixture boundary_ws, room_feed/fixture.rs:926-1078, test
room_feed/boundary_tests.rs:22-51). Own fns start, fetch, save, sum. Edges:
start → fetch, fetch → hyperish::get, save → std::fs::write;
get → TcpStream::connect → Socket::connect → libc connect (ffi);
write → File::open_c → libc open64 (ffi). The boundary maps connect to the
syscall connect (#42, network) and open64 to openat (#257, file).
O = {start, fetch, save, sum}; fwd ∋ syscall connect, syscall openat
network: hit = {connect}; back = {Socket::connect, TcpStream::connect, get, fetch, start}
fns = {fetch, start} ⇒ functions = 2
border: fetch → get (get ∈ onward) ✓; start → fetch (own) ✗ ⇒ fetch first
path(fetch) = fetch → get → TcpStream::connect → Socket::connect → libc connect → syscall connect
hops = 5; kernel_entry = __x64_sys_connect; uncertain = false
file: fns = {save} ⇒ functions = 1; syscalls = ["openat"]
sum reaches nothing ⇒ functions_reaching = |{fetch, start, save}| = 3 of functions_seen = 4
class order: network (2) before file (1)
All of these figures are asserted by the test.
- Output.
classes[{class, functions, syscalls, from[{class, function, repo, crate, at, syscall, path, hops, uncertain, kernel_entry, border}]}], functions_seen, functions_reaching, unmapped_own_ffi, worst(boundary.rs:84-143). - Limits. No syscall node gives the gap
NO_BOUNDARY(boundary.rs:48). A dependency dumped by a driver older than the fn-as-value CALLS edge misses those hops (boundary.rs:177). An ownextern "C"symbol the C map does not know is listed inunmapped_own_ffi. - Cost. Folding the universe for a lake the boundary never ran on was
8.2 M ring nodes in 951 s (lake copy, 2026-10-19);
boundary_in_lakenow answers "no boundary" from a node-stream scan (boundary.rs:50-82). - Tests.
room_feed/boundary_tests.rs:22-153(8 tests);room_feed/reach_tests.rs:50-196(9 tests, kept walk equals reference).
The clone lenses
Graph first (room_feed/graph_first.rs:1-21). The compiled graph is the
primary index and answers alone with state complete. The embedding index
(when vectors exist) adds its pairs beside the graph's with their cosine.
Every finding carries source = graph, embedding or both
(graph_first.rs:33-40), and every answer counts them in indexes
(graph_first.rs:50-67). No vector is a note ("the embedding index added
nothing"), never a refusal.
Shared rules of every clone lens:
- Population (
room_feed/clones.rs:201-268). The fn and method nodes of the scope's repos that have text of their own (fnvec::room::function_nodes,src/vector/fnvec/room.rs:74), minus files undertarget/, absolute or OUT_DIR paths,vendor/,vendored/,third_party/,generated/,*.generated.rs,*.pb.rs,examples/,benches/(clones.rs:1257-1277), minus trait-impl methods whose span is at most 3 lines (TRIVIAL_LINES,clones.rs:1292). A function in#[test], amod tests,tests/or a*_tests.rsis a test (clones.rs:1376); tests leave the pairs unlessinclude_tests. - Shapes come from
fn_shape_eventsat the room's sequence number per repo (theclone_shapesfeed,src/clone_signals/feed.rs). A missing shape is a gap for the shape lenses and a counted note for the graph-first ones (clones.rs:252-263). - Mutual kNN (
graph_first.rs:89-104): for scored pairs,top_k(a)= thekpartners ofawith the highest score (ties: the lower index); a pair(a,b)survives whenb ∈ top_k(a)anda ∈ top_k(b). A hub that resembles everything pairs with nothing. - Deliberate families leave the list and are counted: every member a
SIMD / width variant (a body with
target_featureor intrinsics, or a name part such asavx2,neon,scalar,clones.rs:89-97), and insemantic_twinstwo methods of one name on two types that take onlyself(clones.rs:1399-1408). A member whose body is ONE literal (json!{…}or a struct literal) drops the finding (clones.rs:1304-1374). - Clone classes.
structural_clonesandsemantic_twinscollapse pairs into connected components by union-find (one finding per class, its best pair,class_size, ≤ 12class_members,clones.rs:1431-1493).
Shared dataflow:
code room nodes fn + method ── filters ──► population P clones.rs:201
warehouse fn_shape_events (exact key, tokens, MinHash, simd) ──► shape(f) clone_signals/store.rs
code room edges CONTAINS(param, ret) · USES_TYPE · READS_FIELD · CALLS out/in ──► features(f) structure.rs:143
warehouse embeddings (raw, one vector per fn) / embeddings_norm (per exact key) ──► vectors (optional)
│
exact / near / structural / role / name / cosine indexes ──► pairs ──► gates ──► findings{source}
The table of constants:
| constant | value | file:line |
|---|---|---|
MIN_TOKENS (exact, near, semantic_clones) | 40 normalised tokens | clones.rs:35 |
NORM_FLOOR (semantic_clones) | 0.90 | clones.rs:37 |
ROLE_FLOOR (role_neighbours, engine=fastrp) | 0.90 | clones.rs:42 |
ROLE_FLOOR, ROUNDS, MAX_ROUNDS, D_HUB, RARE_DF, MIN_NEIGHBOURS (engine=wl) | 0.60, 2, 4, 256, 64, 3 | clone_signals/role_wl.rs:37-54 |
MAX_BUCKET (LSH) | 200 | clones.rs:44 |
PERMS, BANDS, ROWS, SHINGLE, NEAR_FLOOR | 64, 16, 4, 5, 0.80 | clone_signals/minhash.rs:17-24 |
RARE_DF, MIN_SHARED, MIN_FEATURES, STRUCT_FLOOR | 64, 2, 3, 0.5 | clone_signals/structure.rs:46-54 |
ROLE_DIM, PROJ_NNZ, W1, W2 | 128, 4, 1.0, 0.5 | structure.rs:56-61 |
STRUCT_BODY_FLOOR, WRAPPER_LINES, TRIVIAL_LINES | 0.3, 6, 3 | clones.rs:1285-1292 |
COPY_COSINE, COPY_STRUCT_FLOOR, TWIN_COSINE, TWIN_STRUCT_CEILING, MUTUAL_K | 0.92, 0.50, 0.882, 0.35, 8 | clones.rs:1232-1243 |
TWIN_MIN_LINES, TWIN_MIN_TOKENS, TWIN_BODY_CEILING, COPY_BODY | 5, 20, 0.80, 0.5 | clones.rs:1245-1280 |
NAME_FLOOR (same_intent graph) | 0.5 | graph_first.rs:143 |
pair index FLOOR, MIN_ROWS | 0.80, 2,048 rows | src/vector/pair_index.rs:37, :40 |
DupThresholds (similar_code) | cosine 0.92, structural 0.5, cross-crate only | src/mimir_analyze.rs:408-417 |
CloneThresholds (same_intent) | cosine 0.80 (0.882 for the calibrated jina-v2-base-code space), structural ≤ 0.35, mutual k 8, cluster ≤ 50 | src/mimir_clones.rs:348-360, src/mimir_calibrate.rs:1416-1430 |
reuse TWIN_BODY_FLOOR, RANK_LINES_CAP | 0.3, 30 | room_feed/reuse.rs:30-32 |
How to run (CLI ops under nornir mimir analyze, src/bin/nornir.rs:789-987):
exact-clones, near-clones, structural-clones, role-neighbours,
semantic-clones, same-intent, semantic-twins, duplicates
(= similar_code), reimplements-shared, each with --server <url> [--repo R] [--limit N]; MCP mimir_<lens>; gRPC Mimir.<Lens>.
Tests for the family: room_feed/clone_tests.rs, room_feed/vector_tests.rs,
room_feed/graph_first_tests.rs, room_feed/reuse_tests.rs,
clone_signals/{norm,minhash,structure}.rs unit tests,
tests/support/mcp_harness.rs:1331, :1394-1403.
exact_clones
- Finds. Groups of functions whose normalised bodies are identical: the Type-1 and Type-2 clones (consistent renames, other literals of the same kind, other layout and comments).
Dataflow.
function text (driver span at the commit) ──► proc-macro2 tokens (one lexer) ──► normalize ──► canonical text
SHA-256(canonical text) = exact key (stored in fn_shape_events) norm.rs:162-252
population P with shape, tokens ≥ 40, not test, one node per (repo, file, start, end)
──► group by exact key ──► groups with ≥ 2 members ──► rank ──► worst[…] clones.rs:270-342
Math. The normalisation (src/clone_signals/norm.rs:131-252):
| token | becomes | rule |
|---|---|---|
keyword, primitive, prelude or common std name (KEEP) | itself | norm.rs:42-66 |
common std method (METHODS) used as one: after ., before (, beside :: | itself | norm.rs:72-105 |
pub, pub(…) | dropped | norm.rs:169-191 |
identifier after ' | L1, L2, … | norm.rs:136-138 |
identifier before ! + a group | M1, … | :139-141 |
identifier after fn, before (, or before ::< | F1, … | :142-146 |
identifier after . | D1, … | :147-149 |
| identifier starting upper case | T1, … | :150-152 |
| any other identifier | V1, … | :153 |
| literal | #s string, #c char, #n number (by kind) | :205 |
Numbers count first occurrences per role, so a consistent rename gives the same
text. The text is laid out canonically: a new line after ;, { and }, four
spaces per brace level, joint punctuation glued (norm.rs:211-250). Then:
exact(f) = SHA-256(canonical text of f) (vector/chunk.rs:111)
tokens(f) = number of normalised tokens (attributes and pub left out)
G_key = { f ∈ P : shape(f), tokens(f) ≥ min_tokens (40), include_tests ∨ ¬test(f),
first node per (repo, file, start_line, end_line) }
finding ⇔ |G_key| ≥ 2
rank = tokens(G) · (|G| − 1) ↓, then key ↑ clones.rs:304
pairs = Σ_G |G|(|G| − 1)/2
GraphBLAS steps. None. Hashing and a BTreeMap group-by.
Worked example (a_consistent_rename_reformatted_and_recommented_is_the_same_normalised_text,
norm.rs:274-283). KILL is pub fn kill(pid: u32, table: &mut HashMap<u32, Proc>) -> Option<Proc> { let found = table.remove(&pid)?; found.stop_all(); Some(found) }
with a comment. ERASE is the same code renamed (erase, id, registry,
hit), reformatted, with a doc comment, #[inline] and pub(crate). By the
rules above, kill/erase → F1 (after fn), pid/id → V1,
table/registry → V2, Proc → T1, found/hit → V3,
stop_all → F2 (before (), while u32, HashMap, Option, Some and
.remove( stay. Both texts start fn F1 ( V1 : u32 and their exact keys are
equal, which the test asserts. On the clone fixture, tally and summarize
form one exact group and tally_more (one statement added) does not
(room_feed/clone_tests.rs:57-68).
- Output.
groups, pairs, groups_matching, deliberate_families, deliberate, worst[{signal: "exact", found_by, why, exact_key, tokens, size, members}](clones.rs:308-342). Parammin_tokens,include_tests. - Limits. A text that does not tokenise has no shape. Names play no part:
killanddeletewith different bodies are no pair (clone_tests.rs:127-140).
near_clones
- Finds. Pairs of DISTINCT normalised bodies that share most of their token runs: Type-3 copies (a statement added, changed or dropped).
Dataflow.
exact groups (as exact_clones, tokens ≥ 40) ──► one MinHash signature per exact key (first member's)
──► 16 bands × 4 rows ──► buckets (≤ 200 members) ──► candidate pairs ──► estimate ≥ 0.80 ──► worst[…]
clones.rs:344-395, minhash.rs:36-101
Math (src/clone_signals/minhash.rs):
classes(f) = per normalised token its role class hashed (V, F, T, D, L, M, a kept name, a punctuation, #s/#c/#n)
FNV-1a 64 of the class string; numbers dropped (V1 and V2 are one class) norm.rs:126-128, :252
shingle s = fold over 5 consecutive classes: h ← mix(h ⊕ c), h₀ = 0x517cc1b727220a
mix = splitmix64 finaliser minhash.rs:27-32
sig_p(f) = min_{s ∈ shingles(f)} ( mix(s ⊕ p·0xa0761d6478bd642f) >> 32 ), p = 0 … 63
estimate(a,b) = |{ p : sig_p(a) = sig_p(b) }| / 64 (an unbiased estimate of the shingle-set Jaccard J)
candidate ⇔ ∃ band β ∈ 0…15 : sig[4β … 4β+3](a) = sig[4β … 4β+3](b), and the bucket has ≤ 200 members
near pair ⇔ candidate ∧ estimate ≥ 0.80 (so at least 52 of 64 minima equal: 52/64 = 0.8125, 51/64 = 0.797)
P(candidate | J = s) = 1 − (1 − s⁴)¹⁶
Pairs sort by estimate descending, then index (minhash.rs:100). A body
shorter than 5 tokens has no signature.
GraphBLAS steps. None. Hash buckets per band.
Worked example (computed from the constants). The banding curve
1 − (1 − s⁴)¹⁶:
s = 0.8: s⁴ = 0.4096; 1 − s⁴ = 0.5904; 0.5904¹⁶ = e^(16·ln 0.5904) = e^(16·(−0.52696)) = e^(−8.4313) = 0.000218
P = 1 − 0.000218 = 0.99978
s = 0.5: s⁴ = 0.0625; 0.9375¹⁶ = e^(16·(−0.064539)) = e^(−1.03262) = 0.35607; P = 0.644
s = 0.3: s⁴ = 0.0081; 0.9919¹⁶ = e^(16·(−0.0081330)) = e^(−0.130128) = 0.87798; P = 0.122
A true near copy at J = 0.8 is a candidate with near certainty; a J = 0.3 pair
is a candidate one time in eight and is then refused by its estimate. On the
clone fixture, tally and tally_more are a near pair found by
"near: MinHash" (room_feed/clone_tests.rs:70-73); the unit test holds a
one-token change in a 400-token text at an estimate ≥ 0.85 and an unrelated
text < 0.2 (minhash.rs:113-129).
- Output.
pairs, pairs_matching, deliberate_families, deliberate, worst[{signal: "near", found_by, why, jaccard_estimate, a[], b[]}];aandblist every member of each exact group (clones.rs:352-395). - Limits. A bucket over 200 bodies (a shape every small function has) is
skipped and counted, so a pair that only meets in such buckets is lost
(
minhash.rs:79-82). - Code vs comment. The module doc says J = 0.8 is a candidate "with
probability 0.9996". The formula it states,
1 − (1 − 0.8⁴)¹⁶, gives 0.99978 (minhash.rs:9-12). The other two figures (0.64, 0.12) match.
structural_clones
- Finds. Pairs of functions that touch the same rare things (parameter and return types, types used, fields read, callees), names ignored.
Dataflow.
code room: CONTAINS → param (not self) / ret parts (ty) ──► p:<type>, r:<type>
USES_TYPE (minus the impl's own self type via FOR_TYPE) ──► t:<stable id>
READS_FIELD ──► f:<stable id>; CALLS out ──► c:<stable id> structure.rs:143-243
──► Features: sets[f], df, idf ──► postings of rare features ──► candidates ──► weighted Jaccard
──► gates (≤ 3 lines, MinHash < 0.3, wrapper) ──► literal drop ──► clone classes ──► worst[…]
clones.rs:405-500
Math (src/clone_signals/structure.rs):
N = functions in the feature population (tests out unless include_tests)
df(x) = functions carrying feature x (structural sets and the k: caller sets both count)
idf(x) = ln(1 + N / df(x)) structure.rs:121-130
J_w(A,B) = Σ_{x ∈ A∩B} idf(x) / Σ_{x ∈ A∪B} idf(x) structure.rs:251-286
cap = min(RARE_DF = 64, max(⌊N/2⌋, 2))
rare x ⇔ 2 ≤ df(x) ≤ cap
candidates(a) = { b > a : sets(b) shares a rare x with sets(a) }, only rows with |sets| ≥ 3
pair ⇔ J_w ≥ floor (STRUCT_FLOOR 0.5, param `structural`) ∧ |A∩B| ≥ 2 structure.rs:327-385
lens gates (clones.rs:430-450):
drop if either span ≤ 3 lines
drop if both have shapes, exact keys differ, and MinHash estimate < 0.3 (STRUCT_BODY_FLOOR)
drop if one side is ≤ 6 lines with ≤ 1 callee that the other side also calls (a wrapper)
order: J_w ↓, then (a, b) ↑; then one finding per clone class
Candidate generation is exact over the posting lists: every pair that shares a rare feature is scored, and a pair that shares none is never compared.
GraphBLAS steps.
| step | operation | semiring | computes | file:line |
|---|---|---|---|---|
| 1 | CONTAINS, FOR_TYPE, USES_TYPE, READS_FIELD, CALLS out and in | none (edge extraction, row scans) | the feature sets | structure.rs:143-243 |
| 2 | posting lists, candidate scoring | none (gatling fork-join over rows, jaccard_score) | pairs | structure.rs:327-385 |
| 3 | CALLS out of the pair members | none (out_pairs, CSR row read) | the wrapper gate | clones.rs:417-420, crates/mimir-room/src/lib.rs:785 |
Worked example (unrelated_names_sharing_a_rare_return_type_and_fields_are_structural_clones,
structure.rs:747-770). 200 filler functions each have r:Result<()>,
p:&str and their own c:own{i}. KILL has {r:MapLayout, t:MapLayout#, f:MapLayout#repos., f:MapLayout#plates., c:a, p:&str}; PERVERT the same with
c:b. Fns 3 and 4 have {r:Result<()>, p:&str, c:x} and {…, c:y}.
N = 204
df: r:MapLayout = t:MapLayout# = f:repos = f:plates = 2; c:a = c:b = c:x = c:y = 1
p:&str = 204; r:Result<()> = 202
idf(df=2) = ln(1 + 204/2) = ln 103 = 4.63473
idf(df=1) = ln(1 + 204) = ln 205 = 5.32301
idf(&str) = ln(1 + 204/204) = ln 2 = 0.69315
idf(Result) = ln(1 + 204/202) = 0.69809
KILL × PERVERT: ∩ = 4 rare + p:&str = 4·4.63473 + 0.69315 = 19.23207
∪ = ∩ + c:a + c:b = 19.23207 + 2·5.32301 = 29.87809
J_w = 19.23207 / 29.87809 = 0.6437 ≥ 0.5, shared = 5 ≥ 2 ⇒ pair (the test asserts ≥ 0.6)
fns 3 × 4: ∩ = 0.69809 + 0.69315 = 1.39124; ∪ = 1.39124 + 10.64602 = 12.03726
J_w = 0.1156 (the test asserts < 0.2)
cap = min(64, 102) = 64: Result (202), &str (204) are not rare and c:x, c:y have df 1,
so 3 and 4 are never even compared.
The shared words list the rare features first: "both return MapLayout",
"both read .MapLayout.repos", with &str last.
- Output.
pairs, pairs_matching, pass{functions, features, rare_features, candidates, pairs, ms}, worst[{signal: "structural", found_by, why, score, shared, a, b, class_size, class_members}](clones.rs:455-500). - Limits. Two functions on one shared API with unrelated text are
refused by the 0.3 body floor (
clone_tests.rs:80-82). - Tests.
structure.rs:725-770;clone_tests.rs:57-85,:248-262.
role_neighbours
- Finds. Pairs of functions that play the same role in the code graph: the same kind of rare types, fields, callees and callers, whatever their names and bodies. No embedding is read.
- Engines.
engine=wlis the default (theengine=wlsection below);engine=fastrpselects the FastRP engine described first here.
Dataflow.
Features (as structural_clones, plus k:<caller> sets) structure.rs:143
role graph G: rows = functions, then every other endpoint of
CALLS / USES_TYPE / READS_FIELD out of a function and CALLS into one; symmetric structure.rs:434-462
 = D⁻¹·A (row-normalised, weight 1/deg) on GraphBLAS FP64 CSR structure.rs:503-517
X0 (functions × 128) ── propagate ──► X1 = ·X0 ── propagate ──► X2 = ·X1 (per column, GrB_mxv)
role(f) = n( n(X0) + 1·n(X1) + 0.5·n(X2) ) ──► fn_role_vectors (kept, 2-hop ball recomputed)
──► has_role filter ──► cosine self-join (pair index ≥ 2,048 rows, else exact scan) ≥ 0.90 ──► worst[…]
clones.rs:554-733
Math (FastRP, Chen et al. CIKM 2019, as coded in structure.rs:400-596):
proj(x) : h = FNV-1a64(key(x)); four times: h ← mix(h); dim = h mod 128; sign = +1 if bit 40 of h is 0, else −1
(4 non-zeros; two draws on one dim add) structure.rs:406-415
X0[f] = n( Σ_{x ∈ sets(f) ∪ callers(f)} idf(x) · proj(x) ) n(v) = v/‖v‖₂ (0 stays 0) :552-560
X0[u] = 0 for every non-function row u (type, field, external callee)
Â(i,j) = 1/deg(i) for each neighbour j of i :503-517
X1 = Â · X0, X2 = Â · X1 (one GrB_mxv per dimension and step) :565-578
role(f) = n( n(X0[f]) + W1·n(X1[f]) + W2·n(X2[f]) ), W1 = 1.0, W2 = 0.5 :582-592
has_role(f) ⇔ |sets(f)| + |callers(f)| ≥ 3 ∧ ∃ x ∈ sets(f) ∪ callers(f) : df(x) > 1 clones.rs:574
pair (f,g) ⇔ has_role(f) ∧ has_role(g) ∧ role(f)·role(g) ≥ floor (0.90, param `cosine`)
order: cosine ↓, then (a, b) ↑
Because a type or field node starts at 0, X1 of a function carries only its
neighbouring functions; two functions that use one struct meet at X2, through
the struct's row (whose X1 is the mean of its users' X0). The vectors are
L2-normalised, so the dot product is the cosine. When a kept role space exists,
only the 2-hop ball of the changed functions is recomputed
(structure.rs:672-681); inside the ball the result equals a full run.
GraphBLAS steps.
| step | operation | semiring | computes | file:line |
|---|---|---|---|---|
| 1 | edges for the role graph | none (edge extraction) | G | structure.rs:451-457 |
| 2 | Â from CSR with weights 1/deg | Matrix<f64>::from_csr | row-normalised adjacency | structure.rs:503-517 |
| 3 | y = Â · x per column, twice | GrB_PLUS_TIMES_SEMIRING_FP64 (mxv) | X1, X2 | structure.rs:520-530, ops.rs:492-509 |
| 4 | cosine self-join | none (SIMD dot, pair index or exact scan) | pairs ≥ 0.90 | clones.rs:657-675, pair_index.rs:106 |
Worked example (functions_in_the_same_role_are_nearest_neighbours_and_a_lone_utility_is_not,
structure.rs:805-826). KILL and PERVERT (unrelated names, different callees,
the same rare return type and fields, the same callers) are each other's
nearest neighbour with cosine > 0.8, and a lone utility has cosine < 0.3 to
both; after one function changes, only its ball is recomputed and every row
inside it equals a full run. The test records no exact floats. Two
consequences follow from the formulas without any data: a function with an
empty feature set and no function neighbour has role = 0 and pairs with
nothing; and two functions with the same feature set and the same function
neighbours have equal X0, X1 and X2, so their cosine is exactly 1.
- Output.
pairs, pairs_matching, mode, recomputed, deliberate_families, deliberate, worst[{signal: "role", found_by, why, cosine, shared, a, b}](clones.rs:686-733). - Limits. Two-feature accessors have no role (
clone_tests.rs:248-262). - Code vs comment. The module doc calls
X0"the IDF-weighted sum of its features' projections". The code L2-normalises that sum before the propagation (structure.rs:559), soX1is the mean of normalised neighbour vectors. - Tests.
structure.rs:805-830;clone_tests.rs:248,:832;graph_first_tests.rs:281.
engine=wl: role fingerprints (space role-wl-v1)
FastRP above is built from neighbour IDENTITIES, undirected and untyped: two
functions are close when they touch the same nodes. That is proximity. The
wl engine compares what the neighbours ARE, per edge kind and direction,
wherever they are. It is the default engine (since 2026-11-05, after the
comparison in ~/plan/handback/wl-role.md); engine=fastrp selects FastRP
above, which stays the 3D code map's vector producer.
- Finds. Pairs of functions in the same role, in one component or in two with no shared node, and it keeps apart near-neighbours of different roles.
- Params.
engine(wl, the default |fastrp),rounds(1..=4, default 2),score(the floor, default 0.60),kept: false(no stored label is read or written),test_helpers(defaultfalse: test helpers are out of the population;include_testskeeps them too).
Dataflow.
room: nodes (kind, ty of param/ret parts, visibility, is_test, defined, repo)
edges CALLS, USES_TYPE, READS_FIELD, WRITES_FIELD, TOUCHES_TABLE(access),
CONTAINS, IMPLEMENTS, FOR_TYPE
│ from_room: rows = functions, then what they touch; derived IMPL_OF, SELF_OF, HAS_FIELD
▼
ℓ⁰ (n × u64, FNV-1a + splitmix) A_κ boolean, one per kind; Aᵀ_κ cut by the hub/boundary rule
│ │
▼ ▼
for t = 0..T−1: O_t = onehot(ℓ^t) ──► N_{κ,d,t} = A_{κ,d} ⊕.pair O_t ──► sort, hash rows ──► ℓ^{t+1}
│
├─► fn_role_labels (ℓ⁰..ℓ^T + fp per stable id; only the T-hop ball of a change is recomputed)
▼
F_T(v) tokens ──► df, idf, cut df > N/2 ──┬─► postings of rare tokens (df ≤ 64), probed by every token
└─► prefix P_τ(v) in (df ↑, id ↑) order: postings of its
common tokens, probed by the prefix ──► candidates (exact)
──► J = S / (‖w_v‖₁ + ‖w_w‖₁ − S) ≥ 0.60 on gatling fork-join ──► worst[{score, class, shared}]
Math (clone_signals/role_wl.rs, fixed hash mimir_graph::ops::stable_hash):
population P = own fns \ { f : is_test(f) ∨ test_like(f) } unless test_helpers (or include_tests)
test_like(f): a path segment tests / test / test_* / *_tests, a file under tests/,
*tests.rs, */test.rs, benches/, test_* (clones.rs test_like, Func::is_test)
item graph V = P, then every endpoint of their channels; own ADTs, their fields; stubs
channels κ ∈ {CALLS, USES_TYPE, READS_FIELD, WRITES_FIELD, TABLE_R, TABLE_W, HAS_FIELD, IMPL_OF, SELF_OF}
d ∈ {out, in}; tag(κ, d) = 2κ + d
CALLS into a noise callee (`?`, fmt, clone, drop …) and from a test fn are left out
IMPL_OF(m,tm) ⇔ ∃ im, tr: im —IMPLEMENTS→ tr ∧ im —CONTAINS→ m ∧ tr —CONTAINS→ tm ∧ name(m) = name(tm)
SELF_OF(m,T) ⇔ ∃ im: im —CONTAINS→ m ∧ im —FOR_TYPE→ T
ℓ⁰(fn) = H("fn" | item_kind | receiver | arity | sort⦃tclass(params)⦄ | tclass(ret) | bucket(#generics) | vis | is_test)
ℓ⁰(adt) = H("adt" | kind | bucket(#fields) | bucket(#variants) | bucket(#generics) | vis)
ℓ⁰(field) = H("field" | tclass(ty) | vis) ℓ⁰(table) = H("table") ℓ⁰(trait) = H("trait" | vis)
ℓ⁰(stub) = H(kind | "stub") a node outside the workspace
tclass(t) = ref-shape ⊕ head: bool, int, float, char, str, path, bytes, option<·>, result<·,err-class>,
seq<·>, map, set, box<·>, dyn, impl, future<·>, fn-like, generic, assoc, tupleN, own, std, foreign,
unit, never; nested classes cut at depth 2; err-class: own-err, dyn-err, io-err, str-err, unit-err, other-err
bucket(c) = 0 if c = 0, else ⌊log₂ c⌋ + 1
N_{κ,out}(v) = { u : A_κ(v,u) }
N_{κ,in}(v) = { u : A_κ(u,v) } if v is in the workspace and in_deg_κ(v) ≤ D_hub (256), else ∅
c_{κ,d}(v,l) = | { u ∈ N_{κ,d}(v) : ℓ^t(u) = l } |
ℓ^{t+1}(v) = H( ℓ^t(v), sort⦃ (tag(κ,d), l, bucket(c_{κ,d}(v,l))) ⦄ )
F_T(v) = ⦃(t, self, ℓ^t(v)) : t = 0..T⦄ ⊎ ⦃(t, κ, d, ℓ^t(u)) with count c : t = 0..T−1, u ∈ N_{κ,d}(v)⦄
df(k) = own functions whose F_T holds k; a token with df(k) > max(N/2, 2) is cut
idf(k) = ln(1 + N / df(k)); w_v(k) = idf(k) · c_v(k)
S(v,w) = Σ_k min(w_v(k), w_w(k)); J(v,w) = S / (‖w_v‖₁ + ‖w_w‖₁ − S)
has_role(v) ⇔ Σ_{κ,d} |N_{κ,d}(v)| ≥ 3 (round-0 neighbours)
cand(v,w) ⇔ ∃ k ∈ F_T(v) ∩ F_T(w): 2 ≤ df(k) ≤ min(64, N/2) (a rare token)
∨ ∃ k ∈ P_τ(v) ∩ P_τ(w) (a prefix token)
P_τ(v) = the shortest head of v's tokens in the global order (df ↑, then id ↑)
whose remaining weight is below τ·‖w_v‖₁
pair(v,w) ⇔ has_role(v) ∧ has_role(w) ∧ cand(v,w) ∧ J(v,w) ≥ τ (0.60)
class = exact if ℓ^T(v) = ℓ^T(w), else graded
order : J ↓, then (a, b) ↑
fp(v) = H(ℓ⁰(v), sort⦃(tag, H(stable id of u))⦄ over the followed edges)
The prefix rule is what makes the join exact (prefix filtering, Chaudhuri et
al. 2006; Bayardo et al. 2007, here for weighted multisets). From J ≥ τ:
S ≥ τ·(‖w_v‖₁ + ‖w_w‖₁ − S) ≥ τ·‖w_v‖₁ (as ‖w_w‖₁ ≥ S); the same for w
Say v's prefix ends first in the global order. A token of P_τ(v) that w holds
lies before the end of P_τ(w), so it is in P_τ(w). If the prefixes share nothing,
every token v shares with w lies in v's suffix, so S ≤ w(suffix(v)) < τ·‖w_v‖₁,
a contradiction. Every pair above the bar therefore shares a prefix token. The rare
postings stay (a rare token in a prefix is found by them); only the COMMON tokens of a
prefix get a second posting list, probed by the prefix alone. The bar is taken a hair
low (τ·(1 − 10⁻⁶)) because J is f64 and compared after the cast to f32.
Worked example (a_pair_that_shares_only_common_tokens_is_still_a_candidate): 400
rows over 24 tokens, every df > 64, so no token is rare and the old rule compared no
pair (0 of the scan's 2,126 at τ = 0.60). A row with weights (df ↑) 2.0, 1.5, 1.0, 0.5
has ‖w‖₁ = 5.0 and the bar 0.6·5.0 = 3.0: after the first token 3.0 remains (not
below the bar), after the second 1.5 does, so the prefix is its first two tokens.
With the prefix rule the join equals the scan at τ = 0.3, 0.6 and 0.9.
Measured (build-1, release, role_compare on the replayed lakes; lost = pairs of the
exact all-pairs scan at τ = 0.60 the join does not return):
| lake | T | rare rule only: lost | rare + prefix: lost | candidates (was → now) | join ms / scan ms |
|---|---|---|---|---|---|
| Holger 9dbe5cbf | 1 / 2 / 3 | 4,015 / 0 / 0 | 0 / 0 / 0 | T = 2: 276,514 (prefix tokens 15) | 14 / 492 |
| Holger fix2 (schema 6) | 2 | 0 | 0 | 297,989 (26) | 17 / 564 |
| Gunnar 5a2d1da1 | 1 / 2 / 3 | 17,712 / 13 / 0 | 0 / 0 / 0 | T = 2: 486,410 → 528,550 (101) | 28 / 1,435 |
The 13 Gunnar pairs at T = 2 were lost neither by depth nor by a label collapse nor by
the τ cut: each pair's ONLY shared token is a common one, "both are called by a
fn(&str, &str, option, option, own, own) -> own" ×49 with df 120 > 64 (the
test-server boot copies that all call init_tracing, ServeConfig::ephemeral,
Grant::server, AuthTable::bootstrap_admin and the Server::*_addr getters), so no
rare token opened them. All 13 survive the filters: the Gunnar answer moves 2,143 →
2,156 pairs (exact 1,506 unchanged), golden re-pinned. Read by hand they are WEAK (a
shared caller, not a shared role: init_tracing against an address getter); they sit
at J = 0.65-0.72, inside the band the τ calibration already scored 17/19. Holger's
answers do not move (answer hashes unchanged), so the 50/50 hand-checked pairs hold.
T = 3 is no fix: the rare rule loses none at T = 3, but T = 3 moves every answer
(Holger 1,091 → 1,055 pairs, Gunnar 2,143 → 1,545) and adds a hop to the incremental
ball, for a gap that is in the candidate rule, not in the labels.
The counts are bucketed in the hash (seven callers and eight are one bucket
apart at most) and raw in J, so the grade is kept where it is read. In-edges
stop at the workspace boundary and at hubs: Vec::push keeps one label for the
whole room, and a caller's change never reaches through it to its other
callers. A dyn call is an edge to Trait#m (compiler-to-graph.md); the impl
methods reach it by IMPL_OF, a 2-hop route (f → tm ← m), so a trait with 40
impls does not multiply its callers' edges by 40.
GraphBLAS steps.
| step | operation | semiring | computes | file |
|---|---|---|---|---|
| 1 | A_κ from the item graph's edges | Matrix<bool>::from_edges | one boolean matrix per kind | role_wl.rs followed |
| 2 | in-degree per kind | ops::fan_in, PLUS.PAIR | in_deg_κ | ops.rs |
| 3 | Aᵀ_κ on the in-scope rows under D_hub | ops::mask_rows, ANY.SECOND | the in-channels | ops.rs |
| 4 | one-hot of round t | ops::label_onehot, pattern matrix | O_t (n × L_t) | ops.rs |
| 5 | N = A_{κ,d} · O_t | PLUS.PAIR UINT64 (GxB_PLUS_PAIR_UINT64), ops::label_counts | neighbour-label counts | ops.rs |
| 6 | sort per row, hash | CPU, ops::wl_relabel | ℓ^{t+1} | ops.rs |
| 7 | candidates and J | CPU on gatling fork-join, posting lists | pairs | role_wl.rs pairs_rare |
| 7′ | S⟨M⟩ = W ⊕.min Wᵀ (same value, the shared form) | PLUS.MIN FP64 masked, ops::multiset_min_masked | numerator of J | ops.rs |
Step 7 runs on the CPU: the sum over a row's tokens is then in token order, so a score is the same bits on every run. Step 7′ is the GraphBLAS form of the same intersection, kept for the shared crate and tested against it.
Worked example (two_components_of_one_shape_get_one_label_per_position,
the_jaccard_is_weighted_by_rarity_and_counts, role_wl.rs; fixture
fixture/roles.rs). Two components with no shared node:
main_a → load_a → read_a, parse_a; load_a, parse_a USES_TYPE CfgA
main_b → load_b → read_b, parse_b; load_b, parse_b USES_TYPE CfgB
round 0 ℓ⁰(load_a) = ℓ⁰(load_b): the signature fn(&path) -> result<own,own-err>, type names gone.
FastRP has r:Result<CfgA, ErrA> against r:Result<CfgB, ErrB>, and c:/t: ids that differ.
round 1 tokens of load_a: (out CALLS, ℓ⁰(read), 1), (out CALLS, ℓ⁰(parse), 1),
(in CALLS, ℓ⁰(main), 1), (out USES_TYPE, ℓ⁰(adt), 1); load_b has the same four.
So ℓ¹(load_a) = ℓ¹(load_b).
round 2 the same at ℓ¹, so ℓ²(load_a) = ℓ²(load_b): class exact, J = 1.
read_a has the same ℓ⁰ shape class as nothing else in the room and only one
caller: J(load_a, read_a) shares no self token and no neighbour token above
the first round, so it stays under 0.5. Fixture route and parse_head share
their caller, the type and the field; their ℓ⁰ and most round-0 tokens are
equal, but route's eight callees make ℓ¹ and ℓ² differ and add tokens the
leaf does not have, so J falls under the floor. FastRP pairs them.
Test helpers out by default (lane LENS-POLISH, 2026-11-05). A helper in tests/
or a mod tests is code nobody ships, and its copies across test files are the same
role by construction: with helpers in, 932 of Holger's 1,091 pairs and 1,952 of Gunnar's
2,143 had a test helper on both sides. Measured on the replayed lakes (release,
role_compare asks wl-helpers-in / wl-helpers-out; 25 pairs drawn uniformly from
each answer and read blind from source; "weak" = a constructor against its builder, or
a shared caller only):
| lake | helpers in: pairs (exact) | out: pairs (exact) | helpers left out | precision in / out (strict) | prod pairs of "in" kept by "out" |
|---|---|---|---|---|---|
| Holger 9dbe5cbf | 1,091 (977) | 151 (76) | 2,509 | 25/25 (23 of them test copies) / 23/25 (2 weak) | 144 of 154 (+7 new) |
| Gunnar 5a2d1da1 | 2,156 (1,506) | 156 (82) | 8,194 | 21/25 (2 no, 2 weak) / 24/25 (1 weak) | 134 of 198 (+22 new) |
The production pairs that leave with the helpers are mostly pairs whose evidence was a
shared TEST caller (8 of Holger's 10, 61 of Gunnar's 64, among them all 13 pairs the
exact join recovered, init_tracing against Server::grpc_addr): weak by the same
reading. With the helpers out, the answer is the production roles, at least as precise,
and an order of magnitude shorter; test_helpers=true gives the old answer.
Kept state. fn_role_labels holds ℓ⁰..ℓ^T and fp per stable id. Rows
whose fp moved and their T-hop ball over the followed channels are
recomputed; every row when that is a quarter of them or nothing is kept
(refine_ball; equal to a full run, a test). The pairs are carried only while
every function's token multiset holds. idf is population-wide, so J is not
pairwise: any change is the full join, which costs less than the labels.
- Output.
engine, space ("role-wl-v1" | "fastrp-v1"), rounds, pairs, pairs_exact, pairs_matching, mode, recomputed, changed, stored, distinct_labels, pass, deliberate_families, deliberate, worst[{signal: "role", found_by, why, score, class, shared, a, b}]. FastRP keepscosineand addsscore(= the cosine) andclass(graded).Jis no cosine, so awlfinding carries none. - Limits. CALLS is a lower bound (operators, closures, fn pointers). The
tystrings are short names, soErroris one class whicheverErrorit is. Stubs have one label per kind (the universe's labels are phase B). The join is exact (the prefix rule above);role_comparechecks it against the scan (lost_by_posting_lists= 0). - Tests.
role_wl.rs(type classes, labels, hub rule, ball);role_wl_tests.rs(the planted twins, the dyn impls, the near-neighbours, posting lists, incremental);crates/mimir-graph/tests/wl.rs(the CPU oracle, the pinned hash). - Goldens.
src/mimir_lens/room_feed/golden/role-*.json(Holger h32/next 9dbe5cbf, Gunnar 5a2d1da1, Holger fix2 schema 6): counts, labels per round and a fixed answer hash per engine. Check: replay the dump (clone_tests::lake_from_dumps), thenLENS_DUMP_BUILDS=<lake>/builds ROLE_OUT=<dir> ROLE_GOLDEN=<file> cargo test --release --lib role_compare -- --ignored.
semantic_clones
- Finds. Clones by behaviour, held by the graph: signal 1 (exact or near body) OR signal 2 (structural). The model over the NORMALISED body (names removed) confirms a graph pair or adds a pair alone through gates.
Dataflow.
population (graph first) ──► exact groups (tokens ≥ 40) ──► near pairs (MinHash, as near_clones)
──► features ──► structural pairs ≥ 0.5 (kept candidate set)
map every pair onto (exact key a, exact key b) ──► agree{near?, structural?}
warehouse embeddings_norm (one vector per exact key, the newest green run's profile)
──► cosine self-join ≥ 0.90 (pair index, or exact scan ≤ 20,000 rows) ──► emb_pairs
agree pairs: + cosine emb-only pairs: family gate, cross-crate, mutual kNN k = 8
──► exact groups first, then pairs by weight ↓ ──► worst[{source}]
clones.rs:766-1140
Math (clones.rs:766-1140):
graph findings:
every exact group with ≥ 2 members source = both if its key has a vector (cosine 1), else graph
agree(a,b) for keys a ≠ b: near(a,b) = MinHash estimate ≥ 0.80 and/or
struct(a,b) = best J_w ≥ 0.5 over member pairs
c(a,b) = v_a · v_b when both keys have vectors of one length
confirmed ⇔ c ≥ floor (NORM_FLOOR 0.90, param `cosine`) source = both if confirmed, else graph
weight = c if present, else max(near, struct)
embedding-only findings: (a,b) ∈ emb_pairs \ agree with
usable(key) = members with no twin exclusion (test, trait-impl method, constructor returning Self,
span < 5 lines, tokens < 20), unless include_all
usable(a) ≠ ∅ ∧ usable(b) ≠ ∅ ∧ ∃ members in different crates ∧ (a,b) ∈ mutual kNN (k = 8 by cosine)
weight = c, source = embedding
order: exact groups (key order) first, then the rest by weight ↓
GraphBLAS steps. None for the pair logic (hash groups, LSH buckets,
posting lists, SIMD dot products). The features are edge extractions
(structure.rs:143-243); the family sets read IMPLEMENTS and CONTAINS edges
(clones.rs:1547-1587).
Worked example (the_normalised_model_finds_the_renamed_copy_and_semantic_clones_states_its_signals,
room_feed/clone_tests.rs:92-125). tally and summarize are one normalised
body, so they share ONE vector in embeddings_norm. The raw model, which reads
the names, puts their cosine below 0.95 (asserted). semantic_clones lists the
pair from the exact group with found_by = "exact: the same normalised body"
and "normalised model: cosine 1.000 (one vector)", source both. Without any
vector it answers complete from the graph alone (graph_first_tests.rs:183),
and an embedding pair of a test, a one-liner or two functions of one crate is
never added (graph_first_tests.rs:330).
- Output.
findings, rule, indexes, embedding_refused, functions_without_shape, deliberate_families, deliberate, worst(clones.rs:1131-1139). - Limits. Over 20,000 normalised vectors at a floor below 0.80 the
embedding index is not joined and the note says so (
clones.rs:940-945).
same_intent
- Finds. Functions whose names and texts read alike (two
delete-like functions), names included on purpose. Every answer saysname_driven: true. It is not a clone finding (clones.rs:1142-1229).
Dataflow.
code room fn nodes + param nodes ──► name words (own name + parameter names) ──► name Features
──► name_pairs: rare words (2 ≤ df ≤ cap), ≥ 2 words per fn, J_w ≥ 0.5, ≥ 2 shared graph_first.rs:106-216
warehouse embeddings (raw model, one vector per fn) ──► kept cosine join ≥ floor vectors.rs:871-934
union ──► detect_clusters: precision filters, same-chunk refusal, same kind,
structural_agreement ≤ 0.35, partition guard ──► mutual kNN k = 8 + Louvain ──► clusters
mimir_clones.rs:820-1028; relabel (name_driven, SIMD family) clones.rs:1142-1229
Math.
words(ident) = split on non-alphanumerics and lower→Upper boundaries, lower-cased, digits kept,
words of length 1 and "self" dropped, deduplicated graph_first.rs:108-128
name features(f) = { n:<w> : w ∈ words(name(f)) ∪ ⋃_p words(param_name(p)) }
name pair ⇔ |feat| ≥ 2 on both sides ∧ share a word with 2 ≤ df ≤ min(64, max(⌊N/2⌋, 2))
∧ J_w ≥ 0.5 (NAME_FLOOR) ∧ shared ≥ 2 graph_first.rs:179-216
emb pair ⇔ cos(raw_f, raw_g) ≥ floor; floor = 0.882 when the space is jina-v2-base-code (recorded),
else 0.80, or param `cosine` vectors.rs:827-841
kept edge ⇔ both rows pass the filters ∧ not one chunk ∧ same kind class
∧ structural_agreement ≤ 0.35
∧ ¬(cos ≥ 0.92 ∧ structural_agreement ≥ 0.5) (similar_code's territory) mimir_clones.rs:862-897
structural_agreement = 0.6·J(sig tokens) + 0.4·J(callees) (sig only when both call nothing;
callees only when a signature is missing) mimir_analyze.rs:622-637
clusters = Louvain communities of the mutual-kNN (k = 8, by edge weight = cos, else graph score) edges
cluster score = mean(edge weight) · size; clusters over 50 members leave the ranked list
GraphBLAS steps. None. Word features and postings (hash maps), a SIMD
cosine join, mutual kNN and Louvain in plain Rust (mimir_clones.rs:1029-1084).
Worked example (name_pairs_need_two_shared_rare_words,
graph_first.rs:238-251). Rows: bump_alpha(xs, by), bump_beta(xs, by),
workspace_root(), repo_root(), and eight other{i}_thing{i}().
N = 12; cap = min(64, max(6, 2)) = 6
words: 0 {alpha, bump, by, xs} 1 {beta, bump, by, xs} 2 {root, workspace} 3 {repo, root}
df: bump = by = xs = root = 2; alpha = beta = workspace = repo = 1; other_i, thing_i = 1
idf(df=2) = ln(1 + 12/2) = ln 7 = 1.94591; idf(df=1) = ln 13 = 2.56495
(0,1): ∩ = {bump, by, xs} = 3·1.94591 = 5.83773; ∪ = 5.83773 + 2·2.56495 = 10.96763
J_w = 0.5323 ≥ 0.5, shared 3 ≥ 2 ⇒ pair
(2,3): ∩ = {root} = 1.94591; ∪ = 1.94591 + 2·2.56495 = 7.07581; J_w = 0.275, shared 1 ⇒ no pair
The test asserts that (0, 1) is the only pair.
- Output. the
SemanticCloneReportfields (cluster_count_total, cluster_count_matching, clusters_shown, pair_count, thresholds, filters, clusters), pluskind: "same_intent",name_driven: true,indexes,vectors,knn,deliberate(vectors.rs:940-1001,clones.rs:1142-1229). - Code vs comment. The existing manual's table says the graph index is
"name-word Jaccard ≥ 0.5". The code weights each word by its rarity
(
idf = ln(1 + N/df)) and needs two shared words (graph_first.rs:200-212). The signature tokens ofstructural_agreementare whitespace tokens offn name(p: T, …), so the function and parameter names are part of them (vectors.rs:97-101,mimir_analyze.rs:611-615). - Tests.
graph_first.rs:222-251;graph_first_tests.rs:139,:158;vector_tests.rs:275;clone_tests.rs:841.
semantic_twins
- Finds. Type-3 copies and Type-4 twins (one idea written twice), v1's
gates on v2's per-function vectors, graph first (
clones.rs:1695-1708).
Dataflow.
population (graph first) ── same lines once ── twin exclusions (unless include_all) ──► kept K
features(K) ──► graph pairs J_w ≥ 0.5 (kept per population fingerprint) ──► graph mutual kNN (k = 8, by J_w)
warehouse embeddings (raw, one vector per fn) ──► cosine join ≥ min(0.92, 0.882) (space `twins`)
──► embedding mutual kNN (k = 8, by cosine)
union of both ──► per pair: c, s = J_w, body = MinHash / 1.0 ──► copy / twin gates
──► families, literal drop ──► relabel by body ──► drop same-file twins ──► classes ──► worst[…]
clones.rs:1709-2045
Math (clones.rs:1740-1988):
exclusion(f) = test or fixture | method of an `impl Trait for Type` | returns Self
| span < 5 lines | tokens < 20 clones.rs:1524-1545
floor_e = min(copy_floor 0.92, twin_floor 0.882) (params `copy_cosine`, `cosine`)
pairs_e = { (a,b) : v_a·v_b ≥ floor_e }
pairs_g = structural pairs of K at J_w ≥ 0.5
per pair in pairs_e ∪ pairs_g:
c = cosine (from the join, or computed when both have vectors)
s = J_w(a, b)
body = 1 if exact keys equal; MinHash estimate if both have shapes; none otherwise
gate 1 body ≥ 0.80 ⇒ copy, rank = (c or s) · body
gate 2 c_e ≥ 0.92 ∧ s ≥ 0.50 ⇒ copy, rank = c · s
gate 3 c_e ≥ 0.882 ∧ s ≤ 0.35, body known, mutual_e(a,b) ⇒ twin, rank = c · (1 − s)
gate 4 (a,b) ∈ pairs_g, body known, mutual_g(a,b) ⇒ twin, rank = s · (1 − body)
else "between" (c_e = a cosine from the embedding join only)
list = twins by rank ↓, then copies by rank ↓
relabel (when body is known): copy ⇔ body ≥ COPY_BODY (0.5) ∨ the gates said copy; else twin kind_by_body, clones.rs:1687, :1961
drop a twin whose two members are in one repo and one file clones.rs:1965-1969
then one finding per clone class (unless `pairs`), cut at limit
The 0.882 floor is v1's calibration: the 80th percentile of hard negatives
(same signature, another body) over 3,222 nornir functions, ROC-AUC 0.920, for
jina-embeddings-v2-base-code only (clones.rs:1236-1238,
src/mimir_calibrate.rs:1416-1430). In another space the note says to read it
as a heuristic.
GraphBLAS steps. None for the gates. Features and family sets are edge
extractions (structure.rs:143, clones.rs:1547-1587); the joins are SIMD
dot products or posting lists; classes are union-find.
Worked example. On the clone fixture bump_alpha and bump_beta are the
same normalised body on two types: body = 1.0 ≥ 0.80, so gate 1 makes them a
copy, clone_type 3, whatever their structural score; every reported twin has
body_similarity < 0.8 (room_feed/clone_tests.rs:226-240). With the recorded
real jina vectors (vector_tests.rs:396-411), local_data_offset × 2 has
cosine 0.9276 ≥ 0.92 (gate 2 when s ≥ 0.50) and bench_vs_miniz /
bench_inflate 0.9158, which is under 0.92 and over 0.882: it can only be a
twin, and only if s ≤ 0.35, the pair is mutual and its bodies are known.
- Output.
pairs, groups, twins, copies, deliberate_by_family, thresholds{copy_cosine, copy_structural_floor, twin_cosine, twin_structural_ceiling, twin_body_ceiling, mutual_k, min_lines, graph_structural_floor}, excluded, candidates, graph_candidates, functions_without_shape, indexes, worst[{kind, clone_type, found_by, why, structural, body_similarity, members, source, cosine, class_size, class_members}], evidence_notewith the theory text (clones.rs:2021-2044). - Code vs comment. The doc comment, the evidence note and
THEORYsay a copy is "cosine ≥ 0.92 AND structural ≥ 0.50, or MinHash ≥ 0.80" and a twin is "MinHash < 0.80". After the gates, the body audit only UPGRADES: a pair whose bodies share≥ COPY_BODY = 0.5of their normalised token runs is a copy whatever the gates said, and a copy from the gates stays a copy (kind_by_body,clones.rs:1687; nornir 9943fd4c). So a gate-3 or gate-4 twin with0.5 ≤ body < 0.8is reported as a copy, which differs from the doc's "MinHash < 0.80" twin rule. (Before 9943fd4c the audit could also demote a gate-2 copy to a twin.) Twins inside one file are dropped, which the doc comment does not mention. - Tests.
clone_tests.rs:226,:849,:860,:909,:922, andthe_body_share_only_upgrades_a_pair_to_a_copy;graph_first_tests.rs:227,:255.
similar_code
- Finds. Copied-not-moved code: function pairs in different crates whose
vectors AND literal structure agree, clustered, with a keep / kill /
repoint recommendation (
room_feed/vectors.rs:677-822).
Dataflow.
code room fn nodes + param nodes + CALLS ──► SymbolVec rows (signature "fn name(self, p: T…)", callees)
warehouse embeddings (raw, one per fn) ──► kept cosine join ≥ 0.92 ──► gate_pairs ──► embedding edges
features ──► graph pairs J_w ≥ 0.5 (kept) ──► gate_pairs ──► graph edges (cosine added when both have vectors)
drop pairs where one side CALLS the other
──► union-find clusters ──► canonical pick ──► rank ──► trivial-family suppression ──► worst[…]
vectors.rs:681-822, mimir_analyze.rs:801-1050
Math (src/mimir_analyze.rs:575-1050):
gate_pairs (a,b):
refuse when both rows were cut from the same lines (same chunk hash)
refuse when cross_scope_only (default) and crate(a) = crate(b) (param same_crate lifts it)
structural_agreement(a,b) = 0.6·J(sig_a, sig_b) + 0.4·J(callees_a, callees_b)
where J = plain set Jaccard (two empty sets ⇒ 0), sig tokens = whitespace tokens;
= J(sig) when both call nothing; = J(callees) when a signature is missing
keep ⇔ structural_agreement ≥ 0.5 (param `structural`)
embedding edge ⇔ cos ≥ 0.92 (param `cosine`) ∧ gate; graph edge ⇔ J_w ≥ 0.5 ∧ gate
edge weight = cos if finite, else the graph score
cluster = union-find component of the edges
centrality(m) = mean over other members of cos (or structural_agreement when a vector is missing)
canonical = a member in {edda, skidbladnir, znippy-zoomies, facett}, else the most central; ties by id
disposition = keep (canonical), repoint (others, when the canonical is in a shared home), kill (else)
score = mean(edge weight) · size; order: score ↓, size ↓, canonical ↑
GraphBLAS steps. None for the pairs (SIMD dot products through the pair
index or the full scan, posting lists, union-find). The wrapper drop reads CALLS
rows (out_pairs, vectors.rs:725-733).
Worked example (real recorded vectors, jina-v2-base-code,
room_feed/vector_tests.rs:396-436; ljar and lzip-parallel are one program
copied into two crates):
| pair | cosine | at 0.92 | judged by a person |
|---|---|---|---|
decode_entry × 2, physical_cores × 2 | > 0.9999 | kept | the same text |
local_data_offset × 2 | 0.9276 | kept (0.9276 ≥ 0.92) | a true copy |
bench_vs_miniz ~ bench_inflate | 0.9158 | refused (0.9158 < 0.92) | two different benchmarks |
main × 2 | 0.8559 | refused | a true copy the vectors miss |
The test asserts cluster_count = 3. The usable band is 0.012 wide here.
- Output.
kind: "duplicates", cluster_count, cluster_count_matching, clusters_shown, pair_count, same_chunk_pairs, suppressed_trivial_clusters, suppressed_trivial, thresholds, vectors, indexes, knn, worst[clusters with members{disposition, centrality}, pairs{a, b, cosine, graph_score, same_crate}, source]; withcalibration=truethe threshold instrument (vectors.rs:792-821,:398-490). - Code vs comment. The structural agreement is described as "signature +
callee set". The signature string contains the function name and the
parameter names (
fn name(p: T, …),vectors.rs:97-101), and its tokens are split on whitespace, so a name is glued to its neighbour token (name(p:); names therefore count in this gate. - Tests.
vector_tests.rs:211,:319,:415;graph_first_tests.rs:92,:111.
reimplements_shared
- Finds. A crate A that depends on a library B and holds a copy of a B
function: replace A's function with a call to B's (
room_feed/reuse.rs:1-15). It folds no pairs of its own.
Dataflow.
kept answers: semantic_twins (pairs = true, limit 1e6) + similar_code (limit 1e6) ──► pairs + evidence
code room: crate of each fn (parent chain) ── DEPENDS_ON reach per crate ──► imports(A → B)
CALLS rows (wrapper), fan_in(CALLS) (callers), CONTAINS param/ret (fit),
effective_visibility + file of B (public library side)
checkout: A's own Cargo.toml [dependencies] (+ target deps), non-optional ──► tier
──► score ──► one finding per replaced fn ──► direct tier first ──► worst[…]
reuse.rs:230-476
Math (reuse.rs:148-476):
similarity(pair) = the first present of cosine, avg_cosine, body_similarity, graph_score, structural,
clamped to [0,1] reuse.rs:156
left out: either side test code; one crate or one package; no DEPENDS_ON path either way;
A already CALLS B; only similar_code names the pair (no semantic_twins evidence);
B not a public fn of a library target (not bin/, main.rs, tests/, excluded paths);
no copy evidence and max body_similarity < 0.3 (TWIN_BODY_FLOOR)
direction: A → B when DEPENDS_ON reaches B's crate (or its package) from A's crate
tier = Direct if B's package is a normal, non-optional dependency of A's own manifest, else AddDependency
fit = drop_in if param and return types are equal, else adapter; partial when both bodies read
the same ALL_CAPS own static
score = similarity · (1 + ln(1 + callers(A_fn))) · min(lines(A_fn), 30) · (1 if Direct else 0.5) reuse.rs:392
order = Direct tier first, then score ↓; one finding per A function, other libraries under `also`
callers is the in-degree of A's function over CALLS (Room::fan_in,
reuse.rs:296).
GraphBLAS steps.
| step | operation | semiring | computes | file:line |
|---|---|---|---|---|
| 1 | DEPENDS_ON reach, one row per crate, batched | GxB_ANY_PAIR_BOOL (mxm frontier matrix), the structural OR-AND reach | imports | reuse.rs:281-288, ops.rs:672-699 |
| 2 | in-degree over CALLS: ones · A | GxB_PLUS_PAIR_UINT64 (vxm) | callers | reuse.rs:296, query.rs:672-691, ops.rs:140-162 |
| 3 | CALLS rows of the pair members; CONTAINS params | none (CSR row reads) | wrapper, fit | reuse.rs:295, :207-219 |
Worked example. The fixture writes parse_header three times (alpha,
beta, gamma) and makes gamma depend on alpha
(room_feed/reuse_tests.rs:1-50). The answer has exactly one finding:
replace gamma::parse_header, use alpha::parse_header, tier
already_imports (gamma's own manifest names alpha), fit drop_in (same
parameters). beta depends on neither, so its copy is a twin, not a finding.
Without shapes, semantic_twins has no copy and the pair is left out as
"only similar_code's cluster" (reuse_tests.rs:57-66). The rank factors,
from the formula:
callers c: 0 1 10 100
1 + ln(1 + c): 1.000 1.693 3.398 5.615
lines L: min(L, 30) — a 200-line copy weighs as 30
tier: × 1 (already imports) × 0.5 (add a dependency first)
- Output.
findings, functions, already_imports, add_dependency, pairs_kept, drop_in, pairs_read, left_out, worst[{tier, replace, use_instead, library, fit, partial, own_statics, signature, use_instead_public, action, why, evidence, gain{lines, callers}, similarity, score, also}](reuse.rs:396-476). - Code vs comment. The evidence note says "Ranked by similarity × lines ×
(1 + callers)". The code ranks by
similarity · (1 + ln(1 + callers)) · min(lines, 30), halves the add-dependency tier, and sorts the direct tier first (reuse.rs:392,:440). - Tests.
room_feed/reuse_tests.rs(6 tests).
subclone
What it finds. Duplicated regions that are smaller than the item they sit
in: a run of tokens shared by two different functions whose other code
differs. A chunk-keyed detector (one hash or one vector per item) cannot state
such a run at any threshold, because the enclosing items differ
(src/mimir_subclone.rs:1-21). The lens folds source text, not a warehouse
table and not the graph. The code room supplies only the item spans
(src/mimir_lens/subclone.rs:8-24).
Run it.
- MCP:
mimir_subclone {repo, limit}(in process,src/mcp_stdio.rs:2124-2147;limit0 → 50,:364-369). - gRPC:
Mimir.Subclone(LensQuery{repo, limit})goes throughlens_answerand the jeralensesqueue, on the source laneLENS_LANES − 1(src/server_grpc.rs:3451-3460,:6955-6961). - In process:
mimir_lens::answer("subclone", reach, {"repo": …})(src/mimir_lens/mod.rs:201-203). repois required.repo = ""refuses withscope_required: true,lens_meta.state = invalid_argument, and lists the repos that have a checkout (subclone.rs:203-206,:233-256;room_feed.rs:472).- Environment knobs (
src/mimir_subclone.rs:289-304, parsed at:352-397):NORNIR_SUBCLONE_NORMALIZE(0/false/off/no or 1/true/on/yes),NORNIR_SUBCLONE_MIN_LINES(R2),NORNIR_SUBCLONE_MIN_TOKENS(R1),NORNIR_SUBCLONE_OVERLAP(fraction 0..1). A value that does not parse is named in the note as⚠ IGNOREDand the default stays (:325-341). - Volume instrument (reads the live tree, ignored by default):
cargo test -p nornir --lib subclone_volume -- --ignored --nocapture(subclone.rs:470-512).
Corpus. The checkout at <ws>/git/<repo>, the sibling of the warehouse
root (subclone.rs:121-126). Every .rs file with a src path component at
any depth. Directories target, .git, vendor, node_modules are skipped.
tests/, examples/, benches/ and a top-level build.rs are out
(subclone.rs:85, :150-191; test :436-468). Unreadable files are skipped,
not read as empty (:146-149).
Math
1. Tokens (the alphabet). Each file is lexed with proc-macro2
(src/mimir_subclone.rs:456-510). With normalisation ON (the default,
normalize_identifiers: true, :266):
| token | becomes | line |
|---|---|---|
pub | dropped (visibility is not shape) | :472-474 |
a keyword from the 40-word KEYWORDS list (fn, let, for, if, match, self, Self, …) | itself | :402-407, :475 |
| any other identifier | i | :475 |
a literal starting with ", r or b | #s | :482 |
a literal starting with ' | #c | :483 |
| any other literal (numbers) | #n | :484 |
| a punctuation character | itself, one token per character (:: is two tokens) | :490-493 |
a group ( … ), { … }, [ … ] | open token, the inner tokens, close token; an invisible group is ∅ … ∅ | :494-505 |
With NORNIR_SUBCLONE_NORMALIZE=0 identifiers and literals stay verbatim
(:475, :487). Comments and whitespace are gone by construction. Each token
keeps its 1-based source line. A file that does not tokenise is counted in
files_unparsed, never partially guessed (:451-461, :822).
The symbol id is the 32-bit FNV-1a of the symbol text,
sym_id(s) with offset 0x811c9dc5 and prime 0x01000193
(:1111-1118). This makes ids equal across files without a shared interner
(:796-813).
The module doc says "every literal by KIND". The code classifies a byte
character literal b'x' as #s, because the rule looks only at the first
byte (:481-485); the FlatTok lexer of the clone signals classes it as a
character (:576-580).
2. k-grams and winnowing. Constants (src/mimir_subclone.rs:141-174):
k = KGRAM_TOKENS = 25 (k-gram length, tokens)
t = GUARANTEE_TOKENS = 60 (guarantee threshold, tokens)
w = WINNOW_WINDOW = t − k + 1 = 36 (window, in k-grams)
For a file of n tokens x₀ … x₍ₙ₋₁₎:
gᵢ = FNV1a64( le_bytes(id(xᵢ)) ‖ … ‖ le_bytes(id(x₍ᵢ₊ₖ₋₁₎)) ), i = 0 … n−k
offset 0xcbf29ce484222325, prime 0x100000001b3 (:597-606)
if n − k + 1 ≤ w: select the single (gᵢ, i) with the least gᵢ,
ties → the largest i (:619-625)
else for each window s = 0 … (n−k+1) − w:
best(s) = argmin_{j∈[s, s+w)} gⱼ, ties → the rightmost j (:628-634)
emit (g_best, best) unless best equals the previous emit (:636-639)
The guarantee (Schleimer, Wilkerson and Aiken 2003): any shared run of at
least w + k − 1 = 36 + 25 − 1 = 60 tokens contains one whole window, so it
selects at least one common fingerprint and is found. A run shorter than
k = 25 tokens has no k-gram and can never match
(:28-33, guarantee_tokens() :274-276). The expected density is
2/(w+1) = 2/37 ≈ 5.4 % of the k-grams (:154-157).
3. Seeds (phase B). An inverted index fingerprint → [(file, k-gram index)] over all selected fingerprints (:827-833). A bucket is dropped when:
- its occurrences lie in fewer than 2 distinct files and
cross_file_only(default true) (:839-841); - it has fewer than 2 occurrences (
:842-844); - it has more than
max_bucket = 50occurrences: skipped and counted inbuckets_skipped(:845-848,:264).
The single-file test runs before the size test, so a big bucket inside one file is not counted as skipped.
4. Extension (phase C). For every pair of occurrences in a kept bucket
from two different files, extend first checks the k tokens are really
equal (a hash collision returns None), then grows the match one token at a
time to the left and to the right while the ids agree (:1137-1157). The run
has len = e − s + 1 tokens, the same on both sides. Gates
(:869-882, defaults :249-268):
R1: len ≥ min_run_tokens = 60 (= GUARANTEE_TOKENS)
R2: lines(site) = end_line − start_line + 1 ≥ min_run_lines = 20, at BOTH sites
A run that fails is counted in runs_rejected. A kept run is keyed by
run_hash = FNV-1a 64 over its token ids (:1121-1131), so the same run
reached from many seeds lands in one group. Each key collects the set of sites
(file, start_line, end_line, len).
5. Groups (phase D). Per run key (:889-965):
- A site contained in another site of the same group (same file, start ≤,
end ≥, more tokens) is dropped (
:899-906). A group needs ≥ 2 sites left. enclosing= the innermost item span that contains the site's start line (largest depth, then fewest lines;src/vector/spans.rs:54-59).sub_item = truewhen that item starts before the run or ends after it (:920-923).- R6, callable body (
require_callable: true): every site must overlap (i.start ≤ s.end ∧ s.start ≤ i.end) a span of kindfnormethod, else the group is dropped and counted ingroups_non_callable. If any site's file has no spans, R6 cannot judge: the group is kept and counted ingroups_r6_unjudged(:931-951). lines= the lower median of the site heights,hs[(|hs|−1)/2];tokens= the lower median of the site token counts (:952-962).idiom = sites > max_sites = 12(kept, counted inidiom_groups).damage = lines × (sites − 1)(:960).
6. Subsumption, R5 (phase E). Sort groups by tokens desc, then damage
desc, then key. Walk in that order and drop a group when every site of it
lies inside some site (same file, t.start ≤ s.start ∧ t.end ≥ s.end) of an
already kept group. Re-sort the survivors by damage desc, then key
(:967-997). groups_before_subsumption keeps the count before.
7. Findings, the headline (phase F). A finding is a connected component of
groups linked by an overlapping site (cluster_findings, :1029-1108):
intervals = { (file, start, end, group) for every site }, sorted by (file, start, end)
per file: walk with a running high-water end h;
if start_j ≤ h → union(group of previous interval, group_j); h = max(h, end_j)
else → h = end_j
This is union-find with path halving; it yields the transitive closure of
"two sites in one file share at least one line". Per component:
lead = its highest-damage group (groups are damage-sorted, so the first
member), sites = Σ sites of the members, files = the distinct files,
lines and damage = the lead's (not a sum, :722-729). Findings are
sorted by damage desc, then key (:1106). The lens shows the first
limit findings in worst; findings_total always gives the full count.
8. The overlap threshold (item granularity, not used by the lens).
pair_items is a comparison instrument (:1532-1626). Candidates are leaf
fn/method spans. An item's signature A is the set of winnowed
fingerprints whose whole k-gram (first and last token line) lies inside the
item. Two items in different files pair when
overlap(A, B) = |A ∩ B| / max(|A|, |B|) ≥ floor = OVERLAP_FLOOR = 0.70 (:1511-1517, :174)
Only candidates sharing a fingerprint are compared, through an inverted index
that skips buckets over max_bucket (:1590-1602). max in the denominator
(SourcererCC) never scores lower than Jaccard's |A ∪ B|.
detect never reads overlap_floor, and no lens calls pair_items. So
NORNIR_SUBCLONE_OVERLAP changes only the echoed options and the environment
line, never the answer. The lens module doc says the overlap threshold is
"NOT exposed" (subclone.rs:52-58); the code reads it from the environment
anyway (src/mimir_subclone.rs:300, :386-394).
GraphBLAS steps
None. Lexing, winnowing, the inverted index, the extension and the union-find
are plain Rust over vectors and hash maps. The fan-out over files is
gatling_map_balanced, largest source first (src/mimir_subclone.rs:797-818).
The code room is read only for the item spans of each file
(room.nodes_in_file, room_feed/lenses.rs:1395-1432), which is a node
lookup, not a matrix product.
Dataflow
<ws>/git/<repo>/**/src/**/*.rs ──rust_sources──► (path, text) subclone.rs:144-191
code room nodes_in_file(repo, path) ──item_spans──► ItemSpan[] | None room_feed/lenses.rs:1396-1432
│
▼ phase A (gatling, per file)
proc-macro2 lex → shape tokens (i, #s/#c/#n, keywords, punct) → sym_id → winnow(k=25, w=36)
│
▼ phase B
index: fingerprint → [(file, k-gram idx)] (drop: 1 file, <2 occ, >50 occ = counted)
│
▼ phase C
extend each cross-file seed pair → run (len, lines) R1 len ≥ 60, R2 ≥ 20 lines → runs{run_hash → sites}
│
▼ phase D phase E phase F
groups (contained sites dropped, R6, median lines, damage) → R5 subsumption → overlap components = findings
│
▼
envelope: findings_total, groups, groups_before_subsumption, groups_non_callable,
groups_r6_unjudged, files_without_spans, idiom_groups, buckets_skipped,
runs_rejected, guarantee_tokens, options, environment, worst[≤ limit]
Worked example 1: winnowing by hand at k = 25, w = 36
Input: the body of the test winnow_selects_the_same_fingerprints_wherever_a_run_appears
(src/mimir_subclone.rs:1371):
fn f() { let a = 1; let b = 2; let c = a + b; let d = c * 3; let e = d - 1; let g = e / 2; }
Normalised tokens (rule of step 1):
fn i ( ) { 5
let i = #n ; let i = #n ; 5 + 5
let i = i + i ; 7
let i = i * #n ; 7
let i = i - #n ; 7
let i = i / #n ; 7
} 1
n = 44 tokens
Arithmetic with the code's constants:
- k-grams:
n − k + 1 = 44 − 25 + 1 = 20. 20 ≤ w = 36, so the "fewer k-grams than one window" branch applies (:619-625): exactly one fingerprint is kept, the least of the 20 k-gram hashes, ties to the rightmost.- Replaying
sym_idandkgram_hashexactly as written (:597-606,:1111-1118) on these 44 symbols, the least of the 20 hashes isg₁₇ = 0x0106862aad6f4550. The selection is[(0x0106862aad6f4550, 17)]. - The body is 44 tokens, under the 60-token guarantee and under R1 (60). Even
if another file shared this whole body, phase C would reject the run
(
len 44 < 60) and count it inruns_rejected.
The window rule itself is visible with the test's small values k = 6, w = 4 (:1376). The same 44 tokens give 44 − 6 + 1 = 39 k-grams and
39 − 4 + 1 = 36 windows. The first windows, with the hashes from the same
replay (first 4 hex digits shown, comparison on the full 64 bits):
k-gram i : 0 1 2 3 4 5 6 7 8
g_i : d9b2 f8fc 5637 71c5 000a 2947 2513 5f7e b402
window 0 = g0..g3 → min 5637 at i=2 → emit (5637…, 2)
window 1 = g1..g4 → min 000a at i=4 → emit (000a…, 4)
window 2 = g2..g5 → min 000a at i=4 → same index, not emitted again
window 3 = g3..g6 → min 000a at i=4 → not emitted
window 4 = g4..g7 → min 000a at i=4 → not emitted
window 5 = g5..g8 → min 2513 at i=6 → emit (2513…, 6)
Across all 36 windows this keeps 18 fingerprints of 39 k-grams. With a
prefix fn pad() { let z = 9; } (11 tokens) every k-gram inside the body is
the same hash at index i + 11, so the same fingerprints are selected. That
is the property the test asserts (f1 ⊆ f2, :1377-1379).
Worked example 2: the recorded volumes
All numbers are the code's own records (src/mimir_subclone.rs:56-82,
subclone.rs:27-31, :89-112, src/mimir_subclone.rs:226-228, :1636-1641).
| measurement | value |
|---|---|
nornir src/, defaults (Type-2 ON) | 219 files, 1.32 M tokens, 2.9 s: 65 findings, 195 groups, 191 true sub-item |
nornir src/, Type-2 OFF | 23 findings, 26 groups, 25 true sub-item |
| nornir + dwarves + edda, ON | 1,139 files, 4.56 M tokens, 7.2 s: 347 findings, 810 groups, 733 true sub-item |
| same three repos, OFF | 132 findings, 171 groups, 143 true sub-item |
lens corpus (every src tree) / top-level src/ / whole checkout | 278 / 220 / 407 files → 70 / 65 / 120 findings |
R6 on nornir src/ | 394 groups without R6, 191 with it |
largest single region on nornir src/ | restated by 38 groups |
item granularity (pair_items) at overlap ≥ 0.70 / 0.80 / 0.90 | 1,057 / 1,001 / 979 item pairs |
| Type-2 ON vs OFF, three repos (975 files, earlier run) | 680 vs 150 groups |
Arithmetic on these records:
- Groups to findings on nornir:
195 / 65 = 3.0groups per region. R5 and phase F remove no group; they only merge restatements. - The overlap curve:
(1057 − 1001) / 1057 = 5.3 %from 0.70 to 0.80, and(1057 − 979) / 1057 = 7.4 %from 0.70 to 0.90. The code calls this "8 %"; the records give 7.4 %. This flatness is why the threshold is not the knob. - Normalisation:
680 / 150 = 4.53, the "4.5×" of the docs. On findings,65 / 23 = 2.8×. - R6 removes
394 − 191 = 203groups, theuse-block class. - The
default_limitpage: a run withlimit = 25shows 25 of the 70 lens findings. The lens doc says 25 is thedefault_limitinLENSES(subclone.rs:110-112); the code has no such entry.subcloneis not inLENSES(it is inSOURCE_LENSES,src/mimir_lens/mod.rs:178), and a missinglimitreads asDEFAULT_LIMIT = 50(room_feed.rs:103-105,mod.rs:181) or 50 through MCP (mcp_stdio.rs:364-369). - Damage of the verified top find,
mimir.rs:332-434↔viz/diagram.rs:222-325(src/mimir_subclone.rs:91-104), taken as a two-site group with exactly these spans: heights 103 and 104 lines,lines = hs[(2−1)/2] = hs[0] = 103,damage = 103 × (2 − 1) = 103. With Type-2 OFF the detector states only a 36-line fragment (mimir.rs:344-379) of this pair; the testnormalisation_off_loses_the_verified_top_findholds that line (:1737).
Output fields
Scanned answer (subclone.rs:383-411): kind, scanned, scope_required,
repo, source_root, files_scanned, files_unparsed, tokens_total,
findings_total, groups, groups_before_subsumption,
groups_non_callable, groups_r6_unjudged, files_without_spans,
idiom_groups, buckets_skipped, runs_rejected, guarantee_tokens,
options, environment, span_source, worst, feeds, evidence_note.
Each worst row (:294-330): key, lead_file, files, file_count,
sites, lines, damage, restated_by (member groups), location (the
lead group's first site, file:start-end), sub_item, explain.
States (room_feed.rs:465-480): refusal = invalid_argument; no checkout or
no .rs file = scanned: false, feed <source tree> missing, note
NO SOURCE TREE (subclone.rs:259-281); scanned and clean = empty_fact
with "clean READING"; any of files_without_spans, files_unparsed,
buckets_skipped > 0 = partial / empty_gap, and the note says the counts
are a lower bound (subclone.rs:347-381).
Limits and needs
- One repo per ask; no workspace scope (by design,
subclone.rs:33-38). - A contiguous token run only. A copy interleaved with unrelated lines breaks
into pieces shorter than the gates and is not found
(
src/mimir_groupclone.rs:6-15, test:1235).group_twinscovers that case. - Spans come from the room. A file the room has no span for is still scanned,
but its sites get no
enclosing, and R6 leaves its groups unjudged (counted). - Needs the
vectorfeature; withoutcode-roomit runs with no spans (room_feed.rs:404-427).
Tests
src/mimir_lens/subclone.rs:415-578 (corpus, sibling root, volume instrument,
refusal, absent vs empty tree); src/mimir_subclone.rs:1275-1425 (planted
block found, rename, retuned constant, unrelated files, short run rejected,
same-file repeat, winnow position independence :1366, winnow discards
:1382, hash collision :1395, guarantee :1405, unparseable file :1417);
R6 :1441, :1481; normalisation :1645, :1737; findings
one_region_restated_by_two_groups_is_one_finding :1768,
two_unrelated_duplications_stay_two_findings :1801; environment :1845,
:1864; room spans src/mimir_lens/room_feed/room_tests.rs:682; one-lexer
rule tests/no_second_parser.rs:151-164.
group_twins
What it finds. The same group of statements inside two different
functions of one repo, also when the copy is interleaved with unrelated
statements and its locals are renamed, and the same group twice inside one
function (src/mimir_lens/group_twins.rs:1-30). Two indexes answer side by
side; each row says which found it (source: graph, embedding, both,
text):
- the graph (
src/mimir_groupclone_graph.rs), primary and complete on its own: compiled facts from the code room, no model, no vector; - the embedding index (
src/mimir_groupclone.rs), optional: the checkout cut into def-use groups, paired on one normalised body or on group vectors inembeddings_groups.
Run it.
- gRPC:
Mimir.GroupTwins(LensQuery{repo, limit}), queued on the source lane (src/server_grpc.rs:3462-3471,:6955). - In process:
mimir_lens::answer("group_twins", reach, {"repo": …, "limit": …, "model_profile": …})(src/mimir_lens/mod.rs:204-206).model_profilepicks the vector space;""means the newest green function-embedding run's space (group_twins.rs:89-99). repo = ""refuses and lists the repos with a checkout (group_twins.rs:179-181,:203-220).- Environment knobs (
src/mimir_groupclone_graph.rs:218-240):MIMIR_GT_MIN_BLOCKS,MIMIR_GT_MIN_WEIGHT,MIMIR_GT_BUDGET_MS,MIMIR_GT_MIN_SRC_LINES,MIMIR_GT_MIN_CALLEES,MIMIR_GT_THREADS; andMIMIR_MIR_SIDE_CAP_MB(default 2048) for the MIR side (room_feed/lenses.rs:1448-1459).
Math: the graph index
1. Population and facts. gather (src/mimir_groupclone_graph.rs:983-1109)
reads two node masks from the room: all = every defined, non-test fn or
method of the room; mine = those of repo (:986-988). For every node of
all it reads these edges and turns each site into a name-free key on a
source line (facts_of, :470-491):
| fact | room source | key | line |
|---|---|---|---|
| a call (callee resolved by rustc, trait methods to the chosen impl) | CALLS edge, one code_spans site per call | c:<stable moniker> | :1016-1024, :475-477 |
| a field read or written | READS_FIELD edge and sites | f:<stable moniker> | :1025-1033, :478-480 |
| a table a storage call names | TOUCHES_TABLE edge, sites, access | t:<table>:<access> | :1034-1038, :481-483 |
a let binding with its inferred type | CONTAINS → local part (name@line, ty) | l:<type> (the name is dropped) | :1045-1052, :484-486 |
The stable moniker leaves the version out, so a dependency bump moves no key
(:476, test :1364). Only facts inside the function's own span count; one
key per line once (:487-489). For a function outside repo the facts carry
line 0 and count only for rarity (:1008-1011).
A fact line is (line, sorted distinct keys) (lines_of, :598-611). This is
the finest unit: sites carry a line, no column and no order within a line
(:20-24).
MIR granularity. A function of repo whose MIR blocks the MIR side holds
is cut by basic block instead (:1053-1099, mir_facts_of :424-467).
Units are the blocks in source order (first line, then body order). Keys per
block:
b:<block fingerprint> the whole block (statements, terminator, callee, local types)
s:<fnv64(canonical statement)> each statement, locals renumbered
c:<callee stable moniker> the call terminator's callee
e:<fp_x>><fp_y> each CFG_SUCC edge x→y, on the source block x
The module doc lists the first three; the code also adds the e: edge key
(:449-452). Functions of the two granularities are never paired with each
other (:681). Functions outside repo with fingerprints only contribute
their b: keys to the rarity (:1061-1063, :1085-1087).
2. Rarity. Over the population P (every function of all,
N = |P|; :1100-1107, Rarity::of :569-580):
df(k) = |{ f ∈ P : k ∈ keys(f) }| (a key never seen counts as df 1)
idf(k) = ln(1 + N / df(k)) (:585-587)
rare(k) ⇔ df(k) ≤ RARE_DF = 64 ∧ df(k) ≤ max(⌊N/2⌋, 2) (:591-594)
W(S) = Σ_{k∈S} idf(k)
The module doc gives the rare rule as df ≤ RARE_DF; the code adds the second
condition (at most half the population).
3. Eligibility and candidates. A function is eligible when its ident does
not contain ::_:: (derive expansions), it has at least min_units units and
at least MIN_GROUP_KEYS = 4 distinct keys (:661-663).
min_units = MIN_GROUP_LINES = 3 for fact lines and
max(MIN_GROUP_BLOCKS = 8, 3) = 8 for MIR (:294-300). Candidates come only
from posting lists of rare keys (:670-675):
cand(a, b) = |{ k rare : k ∈ keys(a) ∩ keys(b) }|, a < b, different monikers, same granularity
Candidates are judged in order of cand desc, then (a, b) (:689-690). At
most MAX_CANDIDATES = 60,000 are judged, and the judging stops at the
BUDGET_MS = 45,000 ms wall clock; the rest is counted in
candidates_unjudged with bounded_by (:691-698, :778-783). The fan-out
is gatling with GRAPH_WORKERS = 6 (:151, :777).
4. The group of a pair (a, b) (judge, :704-775):
shared = keys(a) ∩ keys(b)
for each run_a in runs( lines of a with a key in shared, MAX_GAP = 3 ):
ka = keys(run_a) ∩ shared
run_b = argmax over runs( lines of b with a key in ka, 3 ) of W(keys(run) ∩ ka)
(tie → the run that starts first)
ga = lines of run_a with a key in keys(run_b) trim A
gb = lines of run_b with a key in keys(ga) trim B
MIR: ga, gb ← the largest CFG_SUCC-connected part (tie → the earlier one) (:840-876)
runs(marked, g) splits a sorted index list where two consecutive marked
indices are more than g + 1 apart, so at most MAX_GAP = 3 unmarked fact
lines (the interleaved noise) may sit between two members (:614-623; test
:1561: runs([0,1,5,6,20], 3) = [[0,1,5,6],[20]]).
5. Score, order and gates. With Sa = keys(ga), Sb = keys(gb),
qa, qb = the ordered key sequences (lines in order, keys sorted inside a
line):
score = W(Sa ∩ Sb) / W(Sa ∪ Sb) IDF-weighted Jaccard (:739-741)
exact ⇔ shape(ga) = shape(gb) (the per-line key lists are equal, line for line) (:743)
kgrams(q) = { FNV1a64 of each window of KGRAM = 2 consecutive keys } (:637-651)
shared_k = |kgrams(qa) ∩ kgrams(qb)|
rare = { k ∈ Sa ∩ Sb : rare(k) }
The pair is a twin when all of these hold (:735-751):
|ga| ≥ min_units ∧ |gb| ≥ min_units ∧ |Sa| ≥ 4 ∧ |Sb| ≥ 4
rare ≠ ∅
shared_k ≥ MIN_SHARED_KGRAMS = 2
exact ∨ score ≥ SCORE_FLOOR = 0.6
W(Sa ∩ Sb) ≥ min_weight (0 for fact lines, MIN_GROUP_WEIGHT = 40.0 for MIR)
substantive(…) (always true for fact lines; the MIR boilerplate gates below)
The reported score is 1.0 when exact, else score (:753).
MIR boilerplate gates (substantive, :246-286), each per side:
distinct source lines ≥ MIN_GROUP_SRC_LINES = 4; lines more than 2 away
from the item's first and last line ≥ MIN_INTERIOR_LINES = 3; not the same
file and span under two monikers; cover = distinct group lines in the span ÷
span lines, with min(ca, cb) ≥ MIN_COVER_BOTH = 0.3 and
max(ca, cb) ≥ MIN_COVER_ONE = 0.5; shared c: keys whose crate is not in
BOILERPLATE_CRATES (core, alloc, std, serde, serde_json, clap,
clap_builder) ≥ MIN_GROUP_CALLEES = 4 (:122-153).
6. The same group twice in one function (self_twins, :886-962). For a
body with 2·min_units ≤ n ≤ SELF_SCAN_MAX_UNITS = 800 units, for each offset
d = min_units … n−1, every maximal stretch of positions p with
unit(p) = unit(p + d) gives a pair of non-overlapping runs of length
k = min(stretch, d), kept when k ≥ min_units, both runs are CFG-connected
(MIR), and the gates hold (≥ 4 keys, a rare key, ≥ 2 k-grams in the run, the
weight, substantive). These are exact only, score 1.0, same_function. They
run before the cross pairs, so a spent budget cannot starve them (:664-667,
test :1224).
7. Ranking. Twins sort by score desc, rare_shared desc,
shared_kgrams desc, then the first group line of side a and of side b
(:792-800). shared_words = the labels of the 4 shared keys with the
highest idf (:754-758). A group's key is SHA-256 of its key sequence joined
by newlines (:833).
Math: the embedding index (optional)
Input: the checkout (subclone::rust_sources), each leaf fn/method span
cut out (fnvec::cut) and segmented (group_twins.rs:45-72, :110-128).
- Segmentation (
src/mimir_groupclone.rs:439-531). Statements join by def-use, through union-find in statement order: a statement joins the group that owns a name it reads or writes; it then owns the names it writes. The signature owns nothing. A closing brace joins its header; a header with no def-use link of its own joins the dominant group of its block. Noise that shares no name stays in its own group. - Normalisation. The group's tokens go through
clone_signals::norm::normalize_tokens(names → numbered roles per group, literals → kinds, visibility gone). The exact key is SHA-256 of the normalised text, which is also the vector key inembeddings_groups(:41-52). k-grams: FNV-1a 64 over windows ofKGRAM = 8role classes (:534-550). - Gates (
pair,:617-730):
eligible ⇔ norm.tokens ≥ MIN_GROUP_TOKENS = 30 (:628)
candidate ⇔ a shared 8-gram bucket of ≤ MAX_BUCKET = 100 groups, different functions (:653-664)
df(g8) = number of DISTINCT normalised bodies holding the 8-gram
pair ⇔ |shared 8-grams| ≥ 2 ∧ |{ g8 shared : df(g8) ≤ RARE_DF = 2 }| ≥ 1 (:681-686)
cosine = 1.0 if the exact keys are equal
= Σᵢ vaᵢ · vbᵢ if both vectors exist with equal length (:692)
→ counted in candidates_unjudged otherwise
twin ⇔ cosine ≥ cosine_floor
The code takes the plain dot product. It equals the cosine only when the stored vectors are unit length; the module does not normalise them.
- The floor.
COSINE_FLOOR = 0.90(fixture value) unless the model profile has a recorded row. One row exists:jina-v2-base-code, floor 0.967 = μ_pos − 1·σ_pos, ROC-AUC 0.997, recall 93.6 %, hard-negative survival 0.6 % (:795-858; chosen ingroup_twins.rs:135-147). - Ranking.
cosinedesc,shared_kgramsdesc, then positions (:715-721).
Merge (group_twins.rs:312-410). Graph rows come first. An index twin
whose two group lines fall inside the item spans of a graph row's two
functions is agreement: its line is added to that row's found_by, and
when it is not exact the row's source becomes both and its cosine the
embedding cosine. Any other index twin is added after the graph rows
(source text or embedding). worst = the first limit rows of
graph rows then added rows.
On a graph row the field cosine carries the graph score, an IDF-weighted
Jaccard, and cosine_from: "graph_score" says so (group_twins.rs:342-348).
Likewise GraphOptions.cosine_floor carries the score floor; no cosine is
computed in the graph index (src/mimir_groupclone_graph.rs:155-158).
GraphBLAS steps
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | C = A_kind(rows, :) with GrB_Matrix_extract (rows = every fn ordinal of all) for kind ∈ CALLS, READS_FIELD, TOUCHES_TABLE, CONTAINS | none (an extract, no semiring) | the out-edges of every population function, as (from, to) pairs | crates/mimir-room/src/lib.rs:785-791, crates/mimir-graph/src/matrix.rs:153-174; called at src/mimir_groupclone_graph.rs:1017, :1045 |
Everything else is not GraphBLAS. The node masks are a column scan of the
room's node table (crates/mimir-room/src/query.rs:704-710). Site lines,
spans and MIR blocks are lookups. Rarity is a hash count. The candidate
generation is posting lists in a hash map. The run cut, the trim and the score
are set operations in Rust. The MIR connectivity is a small union-find over the
group's CFG edges (:840-876). The embedding index uses no graph at all.
Dataflow
mimir-driver dump ─► code room (lake) ───────────────────────────────────────────┐
mask: fn|method, defined, non-test (all repos → P, repo → mine) │
GrB_Matrix_extract rows of CALLS / READS_FIELD / TOUCHES_TABLE / CONTAINS │
sites (line per call / field / table), local parts (ty) │
MirSide: mir_block payloads beside the room, ≤ MIMIR_MIR_SIDE_CAP_MB = 2048 MiB│
│ │
▼ gather │
CompiledFn[mine]: fact lines {c:, f:, t:, l:} or MIR blocks {b:, s:, c:, e:} │
Rarity over P: df, idf = ln(1 + N/df), rare │
│ │
▼ pair (gatling, ≤ 60,000 candidates, ≤ 45 s) │
rare-key postings → candidates → runs (gap ≤ 3) → trim → score / exact / 2-grams │
+ self_twins (repeats in one body) → GraphTwin[] sorted by score, rare, k-grams │
│ │
│ <ws>/git/<repo> src trees + room item spans (optional index) │
│ → fnvec::cut → segment (def-use) → normalize → 8-grams │
│ → embeddings_groups vectors (model space) → pair → GroupTwin[] │
▼ │
envelope: graph rows, then index rows added / agreeing → worst[≤ limit] ◄─────────┘
functions, granularity, functions_mir, functions_without_span, graph_unread,
candidates, graph_candidates_unjudged, twins_total, twins_graph,
twins_embedding_added, twins_embedding_agreed, vectors_read, options,
embedding{…}, floor_basis, feeds, evidence_note
Worked example: the near-copy fixture computed by hand
Fixture: a_near_copy_pairs_on_the_score_and_common_keys_never_open_a_pair
(src/mimir_groupclone_graph.rs:1347-1361), with the gates small()
(:1117-1119; for fact lines these equal the defaults: 3 lines, 4 keys, gap 3,
floor 0.6, 2-grams, 2 shared, weight 0).
Functions (raw :1234-1262, planted :1265-1275, background
:1288-1296):
near_a (lines 11-17) near_b (lines 41-47)
11 l:Vec<(String,u64)> same
12 l:u64 same
13 c:ledger_open f:entries same
14 c:weigh_entry l:u64 same
15 c:u64::wrapping_mul f:seed same
16 c:Vec::push same
17 c:sort_by_weight f:entries c:sort_unstable_by_weight f:entries ← the one swapped callee
control: {c:Vec::push l:u64} {c:u64::wrapping_mul} {c:Vec::push} {c:String::len l:u64}
bg0..bg29: {c:Vec::push l:u64} {c:step_i} {l:u64 c:Vec::push} {c:String::len}
The local names (a_acc, b_acc, …) are dropped, so both planted copies
give the same l: keys. Population N = 33. Document frequencies and idf
(ln(1 + 33/df)):
| key | df | idf | rare (df ≤ 64 and ≤ max(⌊33/2⌋, 2) = 16) |
|---|---|---|---|
c:sort_by_weight, c:sort_unstable_by_weight | 1 | ln 34 = 3.5264 | yes |
l:Vec<(String,u64)>, c:ledger_open, f:entries, c:weigh_entry, f:seed | 2 | ln 17.5 = 2.8622 | yes |
c:u64::wrapping_mul | 3 | ln 12 = 2.4849 | yes |
c:String::len | 31 | ln 2.0645 = 0.7249 | no |
l:u64, c:Vec::push | 33 | ln 2 = 0.6931 | no |
Candidates: (near_a, near_b) share 6 rare keys; (near_a, control) and
(near_b, control) share one (c:u64::wrapping_mul). The step_i keys are
unique, so no background pair is a candidate. candidates = 3.
Pair (near_a, near_b). shared = 8 keys (all but the two sort callees).
Every line of A holds a shared key, so run_a = lines 11-17; the same on B.
The trim keeps all 7 lines on both sides (line 17 holds f:entries).
Sa ∩ Sb = 8 keys, Sa ∪ Sb = 10 keys.
W(Sa ∩ Sb) = 5·2.8622 + 2.4849 + 2·0.6931 = 18.1822
W(Sa ∪ Sb) = 18.1822 + 2·3.5264 = 25.2349
score = 18.1822 / 25.2349 = 0.7205 ≥ 0.6
exact = false (line 17 differs)
sequence A = Lvec Lu64 c_lo f_e c_we Lu64 c_wm f_s c_push c_sort f_e (11 keys, 10 2-grams)
shared_k = 10 − 2 = 8 (only the two 2-grams around the swapped callee differ)
rare = 6 shared rare keys
All gates pass: one twin, score 0.7205, exact: false, shared_kgrams: 8,
rare_shared: 6. The code comment records "0.72" for this case
(:104-107).
Pair (near_a, control). shared = {l:u64, c:u64::wrapping_mul, c:Vec::push}. Marked lines of A: 12, 14, 15, 16 (one run). All four lines of
control are marked. After the trim, Sa = {l:u64, c:weigh_entry, c:u64::wrapping_mul, f:seed, c:Vec::push} (5 keys) and
Sb = {c:Vec::push, l:u64, c:u64::wrapping_mul, c:String::len} (4 keys).
W(Sa ∩ Sb) = 0.6931 + 2.4849 + 0.6931 = 3.8712
W(Sa ∪ Sb) = 3.8712 + 2·2.8622 + 0.7249 = 10.3205
score = 3.8712 / 10.3205 = 0.3751 < 0.6
2-grams A = (Lu64,c_we) (c_we,Lu64) (Lu64,c_wm) (c_wm,f_s) (f_s,c_push)
2-grams C = (c_push,Lu64) (Lu64,c_wm) (c_wm,c_push) (c_push,c_len) (c_len,Lu64)
shared_k = 1 < 2
It has one rare shared key, so it fails on the k-grams and on the score. The
code records "0.38" for this control (:104-107). The test asserts exactly
one twin and that control is in none (:1356-1360).
The interleaved case (:1307-1327). B's copy gets a log line and an
unrelated call inserted after its 2nd and 4th planted lines, and its locals
renamed. B's run is lines 41-49, the noise lines 43 and 46 carry no shared key
and are trimmed. Both groups then have the same 7 per-line key lists, so the
pair is exact, score 1.0, group_lines A = 11-17 and B = 41, 42, 44, 45,
47, 48, 49, and both sides have one SHA-256 key.
Worked example: recorded numbers
From the code's records (src/mimir_groupclone_graph.rs:111-147,
src/mimir_groupclone.rs:795-858):
- MIR blocks made one repo's candidates 197,545 (nornir on dwarf, 186 s); hence the 60,000 cap.
- nornir answered 27,837 MIR groups at 3 blocks, against 3,486 by fact line;
hence
MIN_GROUP_BLOCKS = 8. - Of 23 hand-checked nornir MIR findings, 16 were derive code and 3 were
two-line idioms; hence
MIN_GROUP_SRC_LINES = 4. A third check (15 of 390) found shared prologues with cover 0.04-0.28, while real copies covered ≥ 0.59 of one side and ≥ 0.38 of the other; hence the cover gates 0.5 / 0.3. - Embedding space calibration on nornir/src: 6,701 functions, 6,282 groups of
≥ 30 normalised tokens; positives μ 0.9900, σ 0.0234; floor
0.9900 − 0.0234 = 0.9666 ≈ 0.967. Survivors on the corpus: floor 0.90 → 657 twins (286 by vector), 0.95 → 417 (46), 0.967 → 383 (12); 371 exact-key twins do not depend on the floor.
Output fields
Top level: see the dataflow box (group_twins.rs:487-528). Each worst row:
granularity, cosine, cosine_from (graph_score, embedding,
text_exact), score, source, exact, shared_kgrams, rare_shared,
shared (top words), found_by, why (says "interleaved" when fact lines
not in the group sit inside its span), a and b (repo, file, line,
end_line, ident, item_span, group_lines as ranges like 41-42,44-45,
tokens, key, moniker, and for MIR group_blocks), location
(:340-367, :246-290).
States (room_feed.rs:465-480): refusal = invalid_argument; no graph and no
tree = scanned: false, NO COMPILED GRAPH … and NO SOURCE TREE
(group_twins.rs:223-244); graph unread but tree present = partial, note
⚠ MISSING: the compiled graph was not read … a LOWER bound; candidates
unjudged = ⚠ BOUNDED (:438-443); functions without a span =
⚠ MISSING … (a LOWER bound) (:450-452).
Limits and needs
- One repo per ask; rarity is over the whole room.
- The graph needs a code room with
CALLS,READS_FIELD,TOUCHES_TABLEsites andlocalparts. MIR granularity needs the room's MIR bodies and the MIR side payloads beside it; a function without them keeps fact lines. - A line carries no column and no order inside it; a site inside a macro
expansion has no position (
src/mimir_groupclone_graph.rs:20-24). - Facts are what the compiler resolved: control flow and literals are not keys at fact-line granularity. Two groups that call the same functions in the same line order with different arithmetic look alike there.
- The embedding index needs the checkout, the room's item spans, and vectors in
embeddings_groupsfor one model space. With none of these it adds nothing, and the lens is still complete on the graph. - Needs the
vectorfeature (room_feed.rs:434-462).
Tests
Lens: src/mimir_lens/group_twins.rs:531-794 (refusal, not scanned, graph
without tree :623, no graph = lower bound :632, text-exact without graph
:721, ranges :782, source root :790). Graph:
src/mimir_groupclone_graph.rs:1126-1565, including interleaved :1307,
local names :1332, near copy :1347, dependency bump :1364, MIR block
pair :1407, granularities never mix :1443, repeats in one function
:1465, :1492, :1224, connected MIR fragment :1515, bounded pairing
:1542, runs :1561, MIR boilerplate gates :1126, :1185, worker parity
:1199. Embedding index: src/mimir_groupclone.rs:1002-1260, including the
oracles :1066-1119, constants :1156, recorded space :1203, and
winnowing_is_blind_to_the_interleaved_copy_and_this_module_is_not :1235.
subclone, group_twins and dwarves twins: which to use
subclone | group_twins | dwarves twins | |
|---|---|---|---|
| input | the repo's checkout (<ws>/git/<repo>), the room for item spans only | the code room (compiled facts, MIR side); optional checkout + embeddings_groups | one mimir-driver dump (Graph::load) |
| grain | a contiguous token run inside items (≥ 60 tokens, ≥ 20 lines) | a statement group: fact lines (≥ 3) or MIR blocks (≥ 8), gaps of ≤ 3 lines allowed | a whole function (api reach, signature, WL shape of its MIR graph) |
| name-free | yes for locals and literals with Type-2 ON (identifiers → i); keywords and punctuation kept | yes for locals (l:<type>); callees, fields and tables are resolved identities | shape class: yes (labels = terminator kind + callee family); api class: resolved API families; name+sig and DIVERGENT: use the name stem on purpose |
| needs MIR | no | no (MIR raises the grain to blocks when present) | yes for shape, NEAR, inlined, DIVERGENT; api and name+sig work without |
| needs vectors | no | no (optional index) | no |
| scope | one repo, cross-file only | one repo, cross-function plus repeats in one body; rarity over the whole room | the dump's crates (workspace, deps with --include-deps), across crates |
| GraphBLAS | none | one row extract (GrB_Matrix_extract), no semiring | yes: reach on ANY.PAIR, masked IDF cosine on PLUS.TIMES FP64, WL counts on PLUS.PAIR, multiset Jaccard on PLUS.MIN, components (twins.rs:1-28, section 4.2) |
| similarity | exact token-run equality after normalisation | IDF-weighted Jaccard Σidf(∩)/Σidf(∪), idf = ln(1 + N/df), plus key order (2-grams) | IDF cosine with idf = max(ln(n/df), 0.05) (crates/mimir-lens/src/lenses/twins.rs:276-336); WL label multisets; multiset Jaccard ≥ 0.8 |
| unit of the answer | a finding = a region (overlap component of groups), ranked by damage = lines × (sites − 1) | a pair of groups with line lists, ranked by score, rare keys, k-grams | groups of functions (connected components), STRONG when two classes agree |
What each finds that the others miss.
subclonealone sees a copied block of text inside two otherwise different functions, including code the compiler resolves to nothing specific: arithmetic, literals, control flow, string building,matcharms. It also sees code outside compiled configurations, since it reads text. It misses an interleaved copy (the run breaks into pieces underk), a copy whose statements were reordered, and twins with different text.group_twinsalone sees an interleaved or reordered-by-call-syntax copy inside a function:a.f(x)andT::f(&a, x)are one resolved callee, renamed locals move no key, and up to 3 noise lines may sit between members. It also reports the same group twice inside one body. It misses copies that share no rare compiled fact (pure arithmetic), and functions that are twins only as a whole.- dwarves
twinsalone sees whole-function twins that share no text and no statement group: the same outside API reached through different helpers (sha2 ≈ ring digest), the same MIR shape under different names, NEAR shapes that differ by one branch, inlined twins, and DIVERGENT pairs (same name and signature, a different body). It works across crates and on dependencies. It does not locate a shared fragment inside two larger functions.
When to use which.
- "Which block of code was pasted into two places?" →
subcloneon the repo. - "Is the same sequence of calls and field writes done in two functions, maybe
with logging or renames in between?" →
group_twinson the repo. - "Which functions across crates do the same job, or claim to do it and do not
(DIVERGENT)?" → dwarves
twinson the dump. On Holger at the defaults it finds 327 groups, 41 STRONG, 71 DIVERGENT pairs (section 7.3).
Shared and different.
- Shared:
group_twinsand dwarvestwinsboth read compiled facts written bymimir-driver(resolvedCALLS, MIR blocks andCFG_SUCC): the room is folded from the same dump that dwarves loads. All three are deliberately name-free for locals and literals.subcloneand the embedding index ofgroup_twinsuse the one allowed lexer,proc-macro2(tests/no_second_parser.rs:151-164), and share the checkout resolution and corpus rule (subclone::source_root,rust_sources). Both nornir lenses refuserepo = "". - Different:
subcloneis text-based and exact after normalisation;group_twinsis fact-based and weighted by rarity over the room; dwarvestwinsis graph-based on GraphBLAS over a whole dump and compares whole functions. The idf formulas differ (ln(1 + N/df)in nornir,max(ln(n/df), 0.05)in dwarves), and so do the rarity bounds (df ≤ 64 and ≤ N/2 ingroup_twins; ≤rare_max12 fns in dwarves).
4. Graph lenses (nornir-mimir-lens; door: dwarves lens)
Layout. Since 2026-11-05 (lane LENS-MOVE, owner order) every lens, graph,
loader and cache line lives in nornir, in the crate nornir-mimir-lens
(library mimir_lens, crates/mimir-lens/, beside mimir-graph):
| module | what | moved from dwarves |
|---|---|---|
lenses/ | the graph lenses, Graph::load / from_dump, the exact cache (cache.rs, GraphAr cache_graphar.rs), all.rs (lens all / lens batch: the held graph and the plan) | src/lenses/, src/cli.rs (LOADED_GRAPH, LENS_ALL_DEFAULT) |
coverage/ | the coverage index, the Mímir v2 dump adapter (v2.rs), report.rs (roster, --detail text) | src/coverage/, src/cli.rs (coverage_roster, the detail print) |
mimir/ | the zombie / risk / bloat / collide folds, fold.rs (manifest scan, risk coverage input, renderers) | src/mimir/, src/cli.rs (manifest_features, render_*) |
lens_graph.rs | graph-package | src/lens_graph.rs |
lens_defaults.rs, retired.rs | the knobs, the retired v1 tables | the same files |
testkit.rs | fixture writers for these tests and the dwarves CLI tests | the test files |
Features: lenses, mimir-fat, coverage-fat, lens-graph,
graph-cache-graphar (the same names dwarves had). dwarves re-exports the
modules (dwarves::lenses, dwarves::coverage, dwarves::mimir, ...), so its
callers did not change; it keeps argument parsing, data-source selection
(local dump, --wh, server) and printing. The history of the moved files came
along (git log crates/mimir-lens in nornir).
All of them read one dump through Graph::load (crates/mimir-lens/src/lenses/mod.rs:124).
The index holds CONTAINS parents and children, CALLS with site counts, the
dyn Trait expansion (a trait method → the same-named method of every impl,
via IMPLEMENTS and CONTAINS), FOR_TYPE, EXPANDS, CFG_SUCC and the MIR blocks of
each fn.
Scope (crates/mimir-lens/src/lenses/mod.rs:713-737): a node counts when it is defined in
the dump, in the scoped crates, the workspace's own code, not under
generated/, not written by a derive macro (EXPANDS), and not test code unless
--include-tests. Own code means the node's origin is local, or absent in
an old dump (is_local, :420-424). A full dump (cargo mimir-dump without
--workspace-only) carries every dependency crate too. Those crates count only
with --include-deps, or when --crate names them (:719). Before this rule,
twins on a full Holger dump counted 181,922 fns and 10,907 groups of tokio,
serde and other dependencies (plan §D round 9). Test code = an
is_test node, a tests/*.rs crate, a file under tests/, or a cfg(test)
ancestor (:369-393).
LENSES (crates/mimir-lens/src/lenses/mod.rs:695-709): gate-bypass, size, twins,
dispatch-erosion, cycles, single-impl, error-shape, cfg-sprawl,
verdict, and the four study lenses taint, chokepoints, misplaced and
dsm (sections 4.9 to 4.12). The verdict folds the first eight (it recomputes the single-impl, error-shape
and cfg-sprawl terms inline, section 5). It does not fold the study lenses.
The GraphBLAS wrapper is crates/mimir-lens/src/lenses/grb.rs. It calls GrB_mxm with these
semirings:
| function | semiring | product | line |
|---|---|---|---|
mxm_bool | GxB_ANY_PAIR_BOOL | A·B or A·Bᵀ | grb.rs:32-54 |
mxm_f64_masked | GrB_PLUS_TIMES_SEMIRING_FP64 | C⟨M⟩ = A·Bᵀ, structural mask | grb.rs:56-81 |
shared_count / shared_count_rect | GxB_PLUS_PAIR_UINT64 | A·Bᵀ / A·B | grb.rs:83-117 |
multiset_shared | GxB_PLUS_MIN_UINT64 | A·Bᵀ | grb.rs:119-136 |
reach_k | repeated mxm_bool | (I + A + … + Aᵏ)·L | grb.rs:143-155 |
components | mimir_graph::ops::components | weakly connected components | grb.rs:165-178 |
4.0 The load: the parallel reader, the exact graph cache, one load for many lenses
(Lane graph-fast, 2026-11-04, dwarves branch graph-fast/loader-cache; the dw: lines of this section are that branch's.) Every lens
starts with Graph::load(dump) (crates/mimir-lens/src/lenses/mod.rs:223). The load has two paths
that end in the same function, so a cached graph IS the fresh graph:
dump dir (*.nodes.arrow, *.edges.arrow; schema in each file's metadata)
│
key = blake3( LOADER_VERSION ‖ Σ_files sorted ( |name| ‖ name ‖ blake3(bytes) ) )
│ (cache.rs:124, files hashed in parallel)
┌──────── snapshot <cache>/<key>.gar exists? ────────┐
│ no │ yes
▼ ▼
FRESH (v2.rs:291 read_with) CACHED (cache_graphar.rs:329)
1 IPC read, per file, biggest first ─ gatling vertex chunks ∥ (kind, partition) chunks ─ gatling
2 bins = ⋃ bin targets of the raw id column nodes = vertices 0..n, refs = n..n+m
3 rows decoded per ≤ 2^15 rows of a batch, edges[seq] = (id(src), id(dst), kind, props)
columns downcast once, ids built normalised ends[seq] = (src < n ? src : NONE, …)
4 dedupe sharded by hash(key) % 64 │
5 IdIndex + ends resolved ∥ │
│ write the snapshot (temp dir + rename) │ │
└──────────────────────────┬─────────────────────────────┘
▼
Graph::from_resolved / from_parts (mod.rs:267, :275)
parent, children, calls/callers, call_count, dyn expansion, field maps,
mir_blocks_of, cfg_succ, edges_by_kind, edge_kinds, node_kinds
Exactness. Each step keeps the serial reader's answer, not an approximation of it:
| step | rule | why it is the serial answer |
|---|---|---|
| decode | unit u = (file f, batch b, rows r); output = concat over u in (f, b, r) order | the serial reader pushed rows in that order |
| normalise | id' = id[..s] ‖ id[e..] when (s, e) = strip_range(id, bins), else id | normalize_in_place replaced the same range |
| dedupe nodes | shard(i) = fnv(id_i) mod 64; in a shard, walk i ascending: keep the first defined_here row, else the first row | rows of one id are all in one shard, in order |
| dedupe edges | key = (from, to, kind); the first row is the representative; count = max, negated, partial = first non-null in row order | same fold, same order (v2.rs:794) |
| id index | open addressing over node indices, probe fnv(id) & mask, compare the string in dump.nodes | a repeated id resolves to the LAST row, as HashMap::insert did |
| ends | ends[k] = (idx(from_k), idx(to_k)), NONE when no node | the two idx.get of the old edge loop |
edges_of_kinds(K) | the positions of the kinds in K, merged ascending | = edges.filter(kind ∈ K) with both index_of (mod.rs:455) |
| cache | key = content of every graph file + LOADER_VERSION; seq restores edge order | a changed byte is a new key; a snapshot of another key is never read |
Tested in crates/mimir-lens/tests/graph_cache.rs: the fast reader = the reference reader
(read_with_reference, the old row-at-a-time code) on a fixture with stub rows, a
bin= segment, a test crate, a dangling edge end, repeated edges, LargeUtf8 and
UInt64 columns, two batches and an empty file; the cached graph = the fresh graph
field by field (both snapshot formats); a changed file = a new key and the old
snapshot removed; a cut snapshot is rebuilt; validate_graph_dir finds no error in
the GraphAr snapshot; and the same checks on the pinned Holger dump (ignored test).
The snapshot (crates/mimir-lens/src/lenses/cache_graphar.rs) is an Apache GraphAr gar/v1
graph written with nordisk/graph-ar, Arrow IPC chunks:
| part | layout |
|---|---|
vertex type vertex | the n dump nodes, then the m edge ends that are no node; chunks of 2^17 vertices |
| property groups | id · kind name parent crate version file start_line end_line is_test defined_here is_node · repo cfg origin · ty |
| edge types | one per edge kind (vertex_CALLS_vertex, …), ordered_by_source, offsets per vertex chunk, chunks of 2^20 edges |
| edge properties | seq (the edge's position in the deduped dump.edges), count, negated, partial |
dwarves.json | key, file count, n, m, edge count, edge type → kind |
Default place: <dump>/.graph-cache/<key>.gar; DWARVES_GRAPH_CACHE=<dir> moves it,
=off reads the Arrow files every time; DWARVES_GRAPH_CACHE_FORMAT=dwgc uses the
custom one-file snapshot instead (cache.rs: varint rows in 2^14-row blocks plus
the end table). The two measured close on Holger with graph-ar bade573: median cached
load 0.96 s / 2.0 GB against 1.04 s / 2.6 GB, first run with the write 3.6 s against
2.7 s. The rule for a close result is GraphAr: it is a standard that FalkorDB (via
knut) and Spark read, and it holds 0.5 GB less.
One load, many lenses. dwarves lens all (plan and held graph: crates/mimir-lens/src/lenses/all.rs; door: dw:src/cli.rs lens_all_cmd) builds each lens's
argv and parses it with the same RepoLens parser as the verb, on one graph held in
a process static (:624, never dropped, so the process skips freeing it at exit);
dwarves lens batch --plan <jsonl> (:882) runs any list of lens lines, one load per
dump, each line's answer byte for byte the verb's. DWARVES_LOAD_TRACE=1 prints each
load step and each lens's compute time.
The default set (all.rs DEFAULT, plan_src; lane LENS-POLISH, 2026-11-05): every
graph lens with no required input, 20 on one graph: size, twins, dispatch-erosion,
cycles, single-impl, error-shape, cfg-sprawl, dsm, misplaced, taint, verdict, then the
plan-H lenses taxonomy-drift (H9), dyn-to-enum (H1), foreign-helper-sprawl (H3),
late-validation (H4: --entry = the gate entries when given, else the roots),
asymmetric-seam (H5), deferred-init-sprawl (H10), typestate-candidate (H2),
pipeline-subclass-leak (H7), monolithic-router (H8). The ones that need an input ride
on it: history-comments (B4) on --src, gate-bypass on --sink + --choke,
chokepoints on --entry + --sink, runtime-guard-erosion (H6) on --sink.
reach-set (B10: its --entry is the target set, not a filter) and
export-refactor-envelope (H11: a finding of another report) run only by name.
plan_src(L, sinks, chokes, entries, sources, src) =
L if L ≠ ∅ (a repeat runs once)
[gate-bypass | sinks ≠ ∅ ∧ chokes ≠ ∅] ++ DEFAULT ++ [history-comments | src]
++ [chokepoints | entries ≠ ∅ ∧ sinks ≠ ∅] ++ [runtime-guard-erosion | sinks ≠ ∅] otherwise
plan(…) = plan_src(…, src = false)
Every lens of the plan reads the one held graph (loaded), so a lens added to the set
costs its compute only. Bench (build-1, release door, graph cache warm, the box under a
load of 40-50 from other lanes; timeit wall / peak RSS, median of three runs
alternated with the old plan): Holger lens all --mimir-dump holger-s6 6.7 s /
2.73 GB (11 lenses) → 7.5 s / 2.75 GB (20 lenses). The nine new lenses cost 1.0 s
together (DWARVES_LOAD_TRACE: dyn-to-enum 265 ms, monolithic-router 183,
pipeline-subclass-leak 150, taxonomy-drift 149, typestate-candidate 140,
late-validation 67, foreign-helper-sprawl 46, deferred-init-sprawl 28,
asymmetric-seam 13); the RSS is the held graph. With the gate inputs and --src
(24 lenses, history-comments 70 ms of them): 9.4 s / 2.4 GB.
The CLI uses mimalloc (dw:src/main.rs:18): the decode allocates millions of small
Strings on every core, and glibc's arenas made the cache decode 2.0 s against 0.4 s.
4.1 gate-bypass
The question. Does the gate (the choke) sit on every path from a request entry to a write into a store (a sink)? This is complete mediation. The lens answers it twice: first on the fn call graph (plain BFS), then, when the dump carries MIR, context-sensitively on the block-level interprocedural CFG (ICFG), where a path must return to the call site it came from.
Run. cargo lens gate-bypass --sink PAT… (--choke PAT… | --spec FILE) [--entry PAT…]
(dw:src/cli.rs:277-292). --sink is required, and --choke unless --spec names
the gate (below). A pattern is a
substring of the label <crate> <short moniker> (crates/mimir-lens/src/lenses/mod.rs:299,
:447). A pattern that matches no fn is an error, not an empty answer
(crates/mimir-lens/src/lenses/gate_bypass.rs:122-135, :156-162).
Name the trait AND its impls as sinks (Store#put() and [Store]put()). An
impl method is its own node, and the dyn expansion runs from trait to impl
only, so a trait pattern alone under-reports: 14 write paths against 201 on
Rusternetes (section 1.4). An impl pattern also matches test impls.
cargo lens gate-bypass --crate holger-server-lib \
--sink 'RepositoryBackendTrait#put()' --choke 'impl#[IntakeChain]' \
--entry 'exposed/http/handle()'
From the compiler to the graph
- fn nodes and CALLS.
cargo mimir-dump(the nornirmimir-driver) writescode_nodes/code_edges(*.nodes.arrow/*.edges.arrow). The lens reads CALLS, IMPLEMENTS, CONTAINS, EXPANDS and CFG_SUCC from them (Graph::load,crates/mimir-lens/src/lenses/mod.rs:124, edge switch:143-170). - MIR blocks. For every fn, method and closure of a workspace crate the
driver takes rustc's
mir_promotedbody (built MIR, before drop elaboration and optimisation;crates/mimir-driver/src/walk/mir.rs:63-100). Closures andasyncbodies are separate MIR bodies under the same typeck root; the driver writes their blocks under the root fn with the prefixcK.(walk/mir.rs:642-674). Each non-cleanup block becomes onemir_blocknode namedbbNorbbcK.N(walk/mir.rs:680,:737). Itstypayload holds the terminator text, for a callcall <callee moniker>⟨args⟩->_n@line:col(grammar:crates/mimir-lens/src/lenses/mir.rs:1-11). - CFG_SUCC. One edge per normal-flow successor of the terminator; cleanup
(unwind) blocks are no successors (
walk/mir.rs:546-548). For aCallthe return target is a successor (walk/mir.rs:484), so CFG_SUCC already holds the edgecall block → its continuation. This matters below. - The cap. Since schema 6 there is none: the driver writes every
non-cleanup block of each body (Holger's
exposed/http/handle(), 12,016 blocks, 9,926 of them in its coroutine, is whole). A cap exists only on request (MIMIR_MIR_BLOCKS,cargo mimir-dump --mir-blocks N), and then a capped body's CFG_SUCC stays inside its written blocks. Up to schema 5 the driver wrote the first 1,024 blocks and dropped the rest, so a long body's later call blocks were missing from the ICFG. The lens'sneedstext (crates/mimir-lens/src/lenses/gate_bypass.rs:397, "writes no mir_block for a body over its cap") describes neither; only a fn with nobb0at all lands indoors_without_mir.
Phase 1: the sets on the fn graph (plain BFS, not GraphBLAS)
Let F be the fn and method nodes of the dump. scope(f) is the common
dwarves scope: defined in the dump, in the --crate set, own code
(origin = local) unless --include-deps, not under generated/, not
written by a derive (EXPANDS), not test code unless --include-tests
(crates/mimir-lens/src/lenses/mod.rs:729-736).
The dyn expansion (crates/mimir-lens/src/lenses/mod.rs:188-218). For every trait t and
every impl i with i IMPLEMENTS t, a fn child tm of t and a fn child m
of i with the same name give one edge:
DYN = { (tm, m) : tm ∈ CONTAINS(t), m ∈ CONTAINS(i), (i, t) ∈ IMPLEMENTS, name(tm) = name(m) }
H = CALLS ∪ DYN (the forward graph of the walk)
pred(v) = callers(v) ∪ dyn_sources(v) (gate_bypass.rs:103-106)
A call to a trait method thus reaches every impl's method of that name. This over-approximates on purpose: for a gate, a missed edge would hide a bypass.
The sets (crates/mimir-lens/src/lenses/gate_bypass.rs:125-191):
S = { f ∈ F : label(f) contains a --sink pattern } sinks
C = { v : label(v) contains a --choke pattern } chokes (any node kind; only fns are walked)
E = { f ∈ F : label(f) contains an --entry pattern } with --entry
E = { f ∈ F : scope(f), f not a trait method, callers(f) = ∅, dyn_sources(f) = ∅ } without --entry (roots)
E′ = E ∖ S (is_entry, :175)
D = { (f, s) : s ∈ S, (f, s) ∈ CALLS, f ∈ F, f ∉ S, scope(f) } doors (:177-190)
T = { v : ¬include_tests ∧ (test_code(v) ∨ generated(v)) } (:191)
A door is a pair, so one fn calling two sinks is two doors. Only direct CALLS
make a door: a fn that reaches a sink only through DYN is not a door.
For a blocked set B, let Anc_B(f) be the nodes u with a path
u = v₀ → v₁ → … → v_k = f in H where every vᵢ with i < k is not in B.
The reverse BFS (reverse_bfs, :82-117) computes the shortest such path to
the first entry it dequeues. The door itself is never tested against B. The
two blocked sets are:
B_strict = C ∪ S ∪ T (:204)
B_loose = S ∪ T (:216)
The classes of a door (f, s) (:193-232):
gate-door f ∈ C
BYPASS f ∉ C ∧ ( E′ ∩ Anc_B_strict(f) ∖ {f} ≠ ∅ ∨ f ∈ E′ )
through-gate f ∉ C ∧ ¬BYPASS ∧ E′ ∩ Anc_B_loose(f) ∖ {f} ≠ ∅
latent otherwise (a pub API, a test-only caller, or dead code)
A BYPASS row carries the path entry → … → f → s; a door that is itself an
entry carries [f, s]. Rows are sorted by class (BYPASS, through-gate,
gate-door, latent), then door label, then sink label (:233-239). The
headline counts rows (write paths) and distinct BYPASS door fns (:248-260).
This phase is breadth-first search over adjacency lists (VecDeque), not a
GraphBLAS product.
Phase 2: the ICFG (crates/mimir-lens/src/lenses/icfg.rs)
The fn set of the ICFG is Φ = {f ∈ F : scope(f)} ∪ S ∪ C ∪ E ∪ ⋃_{s∈S} dyn_targets(s)
(crates/mimir-lens/src/lenses/gate_bypass.rs:342-351). Its blocks are the mir_block rows
of the fns in Φ (icfg.rs:61-91). succ(c) is the CFG_SUCC successors of
block c inside the ICFG. ret is the set of blocks whose terminator is
return (:136). entry(f) is f's bb0; a fn without blocks has no entry
and is opaque (it returns at once, :54-55).
V = { blocks of the fns in Φ } n = |V|
Intra = { (b, b′) : (b, b′) ∈ CFG_SUCC, b, b′ ∈ V, b is not an in-set call } (:97-145)
∪ { (bb0(f), bbcK.0(f)) : every nested body K of f } (:80-87)
in-set call: term(b) = call|tailcall g′, label(g′) ∈ Φ, and some g ∈ {g′} ∪ dyn_targets(g′) has blocks
succ(c) = the CFG_SUCC successors of c in V (kept for every call block c, :146)
Calls = { (c, g) : term(c) = call|tailcall g′, label(g′) ∈ Φ, g ∈ {g′} ∪ dyn_targets(g′) } (:136-142)
Call = { (c, entry(g)) : (c, g) ∈ Calls, g not opaque } (descend_edges, :182-187)
Return = { (r, s) : (c, g) ∈ Calls, r ∈ ret ∩ blocks(g), s ∈ succ(c) } (return_edges, :190-207)
The edge bb0 → bbcK.0 is how an async fn or a closure body is entered:
bb0 of an async fn only builds the coroutine, and its body lives in the
bbc1.* blocks. A callee whose label is not in Φ gives no Calls entry at
all; the walk still steps over that block through Intra. The fall-through
after a call INTO the set is not an Intra edge: it exists only as a matched
edge, once the callee is proven to return (icfg.rs:136-146).
The Dyck grammar. Label each Call edge out of call block c with (_c
and each Return edge into succ(c) with )_c; Intra edges carry no
label. A path is realizable when its label word is in the language of
partially matched parentheses:
M → M M | (_c M )_c | ε | intra (matched: every return goes back to its own call site)
P → M | P (_c M (pending calls allowed, as when a walk descends and never returns)
Entries are roots (nobody calls them in the walk), so unmatched ) never
occur on a realizable path from an entry. The insensitive walk accepts any
word over the three edge kinds, so it also accepts (_c₁ … )_c₂: enter a
helper from one call site and "return" into another.
The summary formulation the code uses (matched_edges, icfg.rs:212-261).
Each matched (_c M )_c is replaced by one summary edge c → succ(c):
Summary₀ = ∅
Matched_k = { (c, s) : (c, g) ∈ Calls, s ∈ succ(c), g opaque ∨ g ∈ Summary_k }
A_k = pattern(Intra ∪ Matched_k)
Summary_{k+1} = Summary_k ∪ { g : (entry(g) · A_k*) ∩ ret ≠ ∅ }
stop when Summary_{k+1} = Summary_k, or after round 64 (`rounds >= 64`, :256)
The summaries are computed on the unmasked graph.
What the matching does. Because the fall-through c → succ(c) after an
in-set call exists only as a matched edge, a path continues after a call only
when the callee can reach one of its return blocks. A callee that never returns
(a panic-only body, unreachable) blocks the path after it, in both walks. The
matched walk then differs from the insensitive walk only by Return: the
insensitive walk can enter a helper from one call site and "return" into the
successor of another. (Before dwarves f36754d, succ_of was also written into
Intra, so Matched ⊆ Intra and the summary rounds added nothing; the walk was
realizable only because Return was left out, and it stepped over callees that
never return.)
Phase 2: the masked reach and the doors
Let k ∈ {0,1}ⁿ be the keep vector: k[b] = 0 for every block of a fn in
C ∩ F, else 1 (rows_of, icfg.rs:172-179). x = Σ_{e∈E′} e_{entry(e)}
is the seed vector (entries without blocks give no seed, :271-275).
A_ins = pattern(Intra ∪ Call ∪ Return) (doors_reached, icfg.rs:408-410)
A_mat = pattern(Intra ∪ Matched ∪ Call) (:412-414)
A′ = diag(k) · A · diag(k) (mask_by: drop every edge into or out of a choke block)
v₀ = q₀ = x
q_{t+1}⟨¬v_t, struct, replace⟩ = q_t ANY.PAIR A′ (one BFS level)
v_{t+1}⟨q_{t+1}, struct⟩ = true
v = v_t when q_t is empty (no hop limit) (ops::reach, crates/mimir-graph/src/ops.rs:103-110, bfs :49-98)
ANY.PAIR on bool is the structural OR-AND (LOR.LAND) product: an entry is
true when any predecessor in the frontier has an edge. A seed in a choke block
stays in v, but it has no out-edges after the mask.
The ICFG doors (icfg.rs:418-433):
D_icfg = { (fn(c), g) : (c, g) ∈ Calls, g ∈ S, fn(c) ∉ C }
reached_ins(f, g) = ∃ c ∈ blocks(f) : (c, g) ∈ Calls ∧ v_ins[c]
reached_mat(f, g) = ∃ c ∈ blocks(f) : (c, g) ∈ Calls ∧ v_mat[c]
doors_insensitive_walk = |{ (f, g) ∈ D_icfg : reached_ins }|
doors_matched_walk = |{ (f, g) ∈ D_icfg : reached_mat }|
impossible_paths_dropped = doors_insensitive_walk − doors_matched_walk (saturating, gate_bypass.rs:356)
D_icfg is not D. It does not drop a sink fn that calls a sink, and it does
not test scope (both are filtered in Phase 1, gate_bypass.rs:185, but not in
icfg.rs:419-427). The block walk also blocks neither sinks nor test code,
while the fn-level BFS does (B_strict). For each matched door the evidence
path is a plain BFS over A_mat without the choke blocks, folded to fns
(path_to, icfg.rs:298-352).
Complete mediation, as coded (gate_bypass.rs:357, :377-388, :439-440):
doors_without_mir = { f : (f, s) is a Phase-1 BYPASS row, entry(f) undefined }
complete_mediation ⇔ ¬∃ e ∈ E′, (f, g) ∈ D_icfg : x_e ⇝ c over A′_mat (no realizable path e ⇝ sink call avoiding C)
∧ doors_without_mir = ∅
complete_mediation_blockwalk ⇔ the first conjunct alone
Read formally: no word of the language P labels a path from an entry block to
a sink call block that avoids every choke block. A path counts as mediated only
when the sink call sits inside a choke fn's own blocks (its frame): a path that
calls the choke, returns, and then writes is a bypass. Under that reading,
doors_insensitive − doors_matched are the doors reached only by an impossible
call/return word.
GraphBLAS steps
| step | operation | semiring / op as the code names it | computes | file:line |
|---|---|---|---|---|
| 1 | fn-level classes | none: reverse BFS over adjacency lists | BYPASS / through-gate / gate-door / latent | crates/mimir-lens/src/lenses/gate_bypass.rs:82-117, :193-232 |
| 2 | A = pattern(edges) | GxB_Matrix_build_Scalar, iso true, duplicates collapse | the union of edge lists as one boolean matrix | crates/mimir-lens/src/lenses/grb.rs:21-30, crates/mimir-graph/src/csr.rs:208-221 |
| 3 | summaries, per round | ops::reach_each(seeds, A, None, 256): F⟨¬V, struct, replace⟩ = F · A with GxB_ANY_PAIR_BOOL, then V⟨F, struct⟩ = true (GrB_Matrix_assign_BOOL, GrB_DESC_S); 256 fn entries per frontier matrix F (B × n) | which fn entries reach a return block over Intra ∪ Matched | crates/mimir-lens/src/lenses/icfg.rs:243-255; crates/mimir-graph/src/ops.rs:672-729 |
| 4 | choke mask | ops::mask_by: GrB_Matrix_diag(k), then diag(k) · A with GxB_ANY_SECOND_BOOL, then (·) · diag(k) with GxB_ANY_FIRST_BOOL | A′ = diag(k)·A·diag(k): edges with both ends outside the choke | crates/mimir-lens/src/lenses/icfg.rs:281-287; crates/mimir-graph/src/ops.rs:409-436 |
| 5 | reach from the entries (twice: insensitive, matched) | ops::reach: q⟨¬v, struct, replace⟩ = q · A′ by GrB_vxm with GxB_ANY_PAIR_BOOL and GrB_DESC_RSC; v⟨q, struct⟩ = true by GrB_Vector_assign_BOOL with GrB_DESC_S | the blocks an entry reaches | crates/mimir-lens/src/lenses/icfg.rs:288-292; crates/mimir-graph/src/ops.rs:49-98, :103-110 |
| 6 | doors, counts, evidence path | none: a scan of Calls and a plain BFS | doors_*, the CFPQ-BYPASS path | crates/mimir-lens/src/lenses/icfg.rs:418-460, :298-352 |
Dataflow
code_nodes (fn, method, trait, impl, mir_block + ty) code_edges (CALLS, IMPLEMENTS, CONTAINS, CFG_SUCC)
│ │
└──────────────► Graph::load (mod.rs:115) ◄────────────────┘
callers / calls · dyn_targets / dyn_sources (IMPLEMENTS + CONTAINS, same name)
cfg_succ · mir_blocks_of
│
--sink / --choke / --entry patterns ──► S, C, E′, D (gate_bypass.rs:125-191)
│
┌───────────────────────┴───────────────────────────┐
▼ Phase 1 (BFS) ▼ Phase 2 (only when mir_block nodes exist)
reverse BFS over pred = callers ∪ dyn_sources Icfg::build over Φ: V, Intra (+bb0→bbcK.0), Calls (dyn-expanded)
blocked C∪S∪T, then S∪T │ │ │
│ Call (→bb0) Return (ret→succ(c)) Matched (summaries,
▼ │ │ reach_each, ANY.PAIR)
rows {class, door, sink, entry, path} │ │ │
bypass, through_gate, gate_doors, latent ├── A_ins = I ∨ C ∨ R A_mat = I ∨ M ∨ C ──┤
▼ mask_by(¬choke blocks) ▼
ops::reach from entry bb0s (ANY.PAIR, vxm)
│
sink call blocks reached ──► doors_insensitive_walk, doors_matched_walk,
impossible_paths_dropped, CFPQ-BYPASS / dropped rows
Phase-1 BYPASS doors without bb0 ──► doors_without_mir
──► complete_mediation
Worked example 1: the fixture, by hand
Fixture graph() in crates/mimir-lens/tests/graph_lenses.rs:340-396, test
gate_bypass_prints_the_door_an_entry_reaches_without_the_gate
(crates/mimir-lens/tests/graph_lenses.rs:646-683). Sinks Store#put(), choke
gate_ingest, entries http_handle() and mirror_run().
CALLS: http_handle → gate_ingest → Store#put, mirror_run → Store#put,
mirror_run → log, restore → Store#put, restore → log,
t_restore (test) → restore, Disk::put → fs_write. Store#put has one impl
method, so DYN = {(Store#put, Disk::put)}.
Phase 1. S = {Store#put} (the label of Disk::put is
app impl#[Disk][Store]put()., which does not contain Store#put()).
D = {(gate_ingest, put), (mirror_run, put), (restore, put)}.
| door | test | class |
|---|---|---|
gate_ingest | gate_ingest ∈ C | gate-door |
mirror_run | no callers, but mirror_run ∈ E′ | BYPASS, path [mirror_run, Store#put] |
restore | its only caller t_restore is test code (∈ T), blocked in both walks | latent |
So bypass 1, gate_doors 1, latent 1, through_gate 0, as the test asserts.
Phase 2. The blocks of the five fns that matter (other fixture fns add blocks that no entry reaches):
http_handle h0 = bb0: return hc0 = bbc1.0: call gate_ingest hc1 = bbc1.1: return
gate_ingest g0 = bb0: call Store#put g1 = bb1: return (choke)
mirror_run m0 = bb0: call Store#put m1 = bb1: call log m2 = bb2: return
restore r0 = bb0: call log r1 = bb1: call Store#put r2 = bb2: return
log l0 = bb0: return
Store#put, Disk::put: no blocks (opaque)
Calls = (hc0, gate_ingest), (g0, put), (g0, Disk::put), (m0, put), (m0, Disk::put),
(m1, log), (r0, log), (r1, put), (r1, Disk::put)
in-set calls (the callee has blocks): hc0 (gate_ingest), m1 (log), r0 (log)
Intra = h0→hc0 (closure entry), g0→g1, m0→m1, r1→r2 (no fall-through after hc0, m1, r0)
Call = hc0→g0, m1→l0, r0→l0 (put and Disk::put are opaque)
Return = g1→hc1, l0→m2, l0→r1 (log returns to BOTH call sites' successors)
Round 1 (Summary = ∅): Matched = calls to opaque callees: g0→g1, m0→m1, r1→r2
reach from each entry: h0 is a return → http_handle; g0→g1 → gate_ingest; l0 → log
mirror_run (m0→m1, stuck) and restore (r0, stuck) do not reach a return yet
Round 2: Matched adds hc0→hc1, m1→m2, r0→r1 → mirror_run and restore now reach a return
Round 3: nothing new → stop, rounds = 3 (computed by hand from the rule above)
Mask: k = 0 on g0, g1 → drops hc0→g0, g0→g1, g1→hc1 (the matched hc0→hc1 stays: the gate returns)
Seeds: h0, m0
The two walks:
insensitive A′ = I ∨ C ∨ R : h0 → hc0 ; m0 → m1 → l0 → { m2, r1 } ; r1 → r2
sink call blocks reached: m0 (mirror_run, put), r1 (restore, put) → 2
matched A′ = I ∨ M ∨ C : h0 → hc0 → hc1 ; m0 → m1 → { m2, l0 } (l0 has no out-edge)
sink call blocks reached: m0 → 1
The word of the dropped path is m0 · m1 (_m1 l0 )_r0 r1: it enters log from
the call site m1 and returns to the successor of the call site r0. The
parentheses do not match, so it is not in P. impossible_paths_dropped =
2 − 1 = 1. g0 is not a door of the walk because gate_ingest ∈ C. The
matched walk reaches the door mirror_run, so complete_mediation is false
and the text says CFPQ-BYPASS app mirror_run(). and
dropped app restore()., as asserted at crates/mimir-lens/tests/graph_lenses.rs:672-682.
Worked example 2: Holger before and after R1
Before: h32/next 9dbe5cbf (plan §D rounds 5 and 6, §E.2; golden
gate-bypass in crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json). Sinks:
RepositoryBackendTrait::{put, put_from_reader, handle_http2_upload, handle_http2_request_bytes} and dyn ….write_from_reader. Choke:
impl#[IntakeChain]. Entries: the HTTP handle, the gRPC services,
JobRunner::run, mirror run_cycle / run_pair, replication, the import
door, migration and offsite restore.
- Phase 1: 21 doors = 13 BYPASS rows (through 11 distinct door fns) + 4
gate-doors (
ingest_pulled,ingest_pushed,npm_publish,cargo_publish, all insideIntakeChain) + 0 through-gate + 4 latent (Holger::put,LocalHolger::put,ProxyBackend::{put, put_from_reader}). Check: 13 + 4 + 0 + 4 = 21. - Phase 2 on
holger-server-lib:cfpq_blocks194,731,cfpq_intra_edges199,569,cfpq_call_edges11,386,cfpq_return_edges30,354,cfpq_rounds2,cfpq_matched_edges10,230. Insensitive 19 doors, matched 13, dropped 19 − 13 = 6:Holger::put,LocalHolger::put,ProxyBackend::{fetch, put, put_from_reader}andSignedIndexes::handle_http2_request_bytesare reached only through a return into another call site.doors_without_mir0, so complete mediation: NO (the matched walk reaches 13). The intra-edge, matched-edge and round counts were recorded before dwarves f36754d (which took the fall-through after in-set calls out ofIntra). That commit reports the Holger doors unchanged: 13 matched. The golden row pins the door counts, not the edge counts. - One door, as the plan records it.
offsite::restore, reached from the named entryoffsite::restore_latest. The fn-level row isBYPASS offsite::restore, pathoffsite::restore_latest → … → offsite::restore → <sink>; the plan records the entry and the door, not the intermediate fns or which of the five sinks the row names. The door is also among the 13 doors of the matched walk, so the path is realizable with calls and returns matched. The review listed the offsite restore as a raw write; the lens shows it as BYPASS becauserestore_latestwas passed as an entry (plan §E.2). A second recorded door of the same shape ismirror_cycle::run_pair_skipping, reached from the entryrun_pair. - The async doors were found only after the coroutine bodies (
bbcN.0) were entered frombb0(dwarves 6f881c0). Before that fix the matched walk reached 3 doors, not 13 (plan handback lens-build).
After: Holger main with the R1 Admitted gate (plan §D round 9; holger
main 91768f29 / bb80377b → 8fd5d586, the holger-refactor lane's dump). Chokes:
impl#[IntakeChain], admitted/impl#[Admitted], admitted/impl#[Gate].
Sinks: put, put_from_reader, handle_http2_upload, dyn ….write_from_reader.
- Phase 1: 0 BYPASS, 3 gate-doors, 2 latent (
ProxyBackend::{put, put_from_reader}). - Phase 2: matched 0, insensitive 2, dropped 2 − 0 = 2. With 0 BYPASS rows,
doors_without_mir = ∅holds by definition. Complete mediation: YES. - With
handle_http2_request_bytesas a sink too, 4 fn-level doors stay (exposed/http/handle,ChainViewandSignedIndexeshandle_http2_request_bytes,IndexSigner::answer), and the matched walk reaches 3 of them. That verb serves the whole collected request and writes only on PUT, DELETE or PATCH. The method is a runtime value, so the lens cannot call that door mediated. Splitting the verb is a Holger change.
Output
Summary (crates/mimir-lens/src/lenses/gate_bypass.rs:261-289, :358-389, :440):
sinks, choke_fns, entries, doors, bypass, through_gate, gate_doors, latent, dyn_expansions (= Σ |dyn_targets|), then complete_mediation_blockwalk, cfpq_blocks, cfpq_intra_edges, cfpq_call_edges (= |Call|),
cfpq_return_edges, cfpq_matched_edges, cfpq_rounds, doors_insensitive_walk, doors_matched_walk, impossible_paths_dropped, entries_with_blocks, doors_without_mir, entries_without_mir, complete_mediation. Findings:
{class, door, door_loc, sink, entry, path, path_locs} with class BYPASS,
through-gate, gate-door or latent (:318-329); then
{class: CFPQ-BYPASS | dropped, door, door_loc, sink, path}, matched rows
first, then by door and sink label (:400-438).
Limits and needs
- A runtime
if(a config toggle such as Holger's*_refusal()) is not a graph edge, so a door guarded only by a toggle reads open. - The dyn expansion and the
bb0 → bbcK.0edges over-approximate (the safe side,:331-333,icfg.rs:82-84). - Roots mode (no
--entry) is wrong for a server: its roots sit in hyper and tonic (on Holger it gave 6 bypass rows; plan handback lens-build). needslines: IMPLEMENTS edges (:158-163), CALLS edges (:164-169),mir_blocknodes (the whole Phase 2 is skipped without them,:353-359), and MIR facts for each BYPASS door without abb0(:408-417). Each namescargo mimir-dump.
Tests
crates/mimir-lens/tests/graph_lenses.rs:
the_graph_indexes_calls_contains_and_the_dyn_expansion (:628),
gate_bypass_prints_the_door_an_entry_reaches_without_the_gate (:646;
insensitive 2, matched 1, dropped 1, mediation false),
gate_bypass_says_yes_to_complete_mediation_when_every_door_is_the_gate
(:686), gate_bypass_expands_a_dyn_call_to_every_impl (:701; path
mirror_run → Store#put → Disk::put → fs_write),
gate_bypass_refuses_a_pattern_that_matches_nothing (:726),
the_matched_walk_does_not_fall_through_a_callee_that_never_returns (:986:
boot calls never (bb0 unreachable) and then the sink; fn-level bypass 1,
matched walk 0, insensitive walk 0). MIR parsing:
crates/mimir-lens/src/lenses/mir.rs:216, :228. The Holger numbers are pinned in
crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json (gate-bypass).
4.2 twins
twins finds groups of fns that do the same thing. It needs no embeddings. It
builds six kinds of pair edge (api, name+sig, shape, NEAR, inlined, DIVERGENT)
from four matrices, joins them into connected components, and reports each
component as a group. A group whose edges come from two or more classes is
STRONG. The code is crates/mimir-lens/src/lenses/twins.rs, the products are in
crates/mimir-lens/src/lenses/grb.rs, and the knobs are in crates/mimir-lens/src/lens_defaults.rs.
Run
cargo lens twins [--hops N] [--sim X] [--rare-max N] [--wl-rounds N] \
[--min-blocks N] [--near X] [--min-apis N] [--name SUBSTR] \
[--mimir-dump DIR] [--crate NAME…] [--include-tests] [--include-deps] [--json] [--limit N]
# the golden runs (crates/mimir-lens/tests/golden_holger.rs:124-156):
dwarves lens twins --mimir-dump DIR --json --limit 0
dwarves lens twins --mimir-dump DIR --json --limit 0 --name normalize_name --min-blocks 4
The flags are in dw:src/cli.rs:306-338. The CLI's default_value_t and
TwinsArgs::default() both read one table, so they cannot drift apart
(crates/mimir-lens/src/lens_defaults.rs:1-18, crates/mimir-lens/src/lenses/twins.rs:64-79). verdict
runs twins with TwinsArgs::default() and its own scope
(crates/mimir-lens/src/lenses/verdict.rs:258-264).
| knob | default | meaning | lens_defaults.rs |
|---|---|---|---|
hops (k) | 2 | helper depth of the API reach, (I + A + … + Aᵏ)·L | :6 |
sim | 0.8 | cosine floor of an api or name+sig edge | :8 |
rare_max | 12 | an API is rare when at most this many fns reach it | :10 |
wl_rounds (h) | 2 | Weisfeiler–Lehman rounds | :12 |
min_blocks | 8 | MIR blocks a fn needs for shape / NEAR / inlined / DIVERGENT | :14 |
near | 0.8 | multiset-Jaccard floor of a NEAR edge | :16 |
min_apis | 2 | reached APIs each fn of an api pair needs | :18 |
Round 9 of the plan made this one table. Before it, the CLI said
rare_max 50, min_blocks 4, which gave 562 groups and 118 STRONG on Holger
(plan §D rounds 6 and 9). At the defaults the golden reads 327 groups and 41
STRONG under the STRONG rule of dwarves 27eaa1d (78 under the old rule).
Reads (from the compiler to the graph)
The driver (cargo mimir-dump) writes Mímir v2 *.nodes.arrow and
*.edges.arrow. Graph::from_dump indexes them (crates/mimir-lens/src/lenses/mod.rs:130-252).
twins uses these facts:
| fact | from the dump | index field | used for |
|---|---|---|---|
| fns | nodes of kind fn / method (FN_KINDS, mod.rs:72) | dump.nodes | the row set |
| calls | CALLS edges, both ends resolved, deduped per caller (mod.rs:155-159, :173-180) | calls[f] | A and L |
| types | param / ret children (CONTAINS) with a ty column | children[f] | T |
| MIR blocks | mir_block nodes, owned by the CONTAINS parent or by the id <fn>bbN: / <fn>`bbcK.N`: (mod.rs:210-224, :547-559) | mir_blocks_of[f], read sorted by block name (mod.rs:481-496) | the PDG nodes and ℓ⁰ |
| control flow | CFG_SUCC edges, block → block (mod.rs:168) | cfg_succ | PDG control edges |
A mir_block payload is name-free text (crates/mimir-lens/src/lenses/mir.rs:1-11):
<fp 16 hex> ␞ <terminator>@l:c ␞ <stmt>@l:c ␟ … ␞ <local>:<ty> ␟ … ␞ <a>><b>,…
terminator of a call: call <callee moniker>⟨<args>⟩-><dest local>@l:c
parse_block needs a 16-hex fingerprint, else the block is unparsable
(mir.rs:69-100). term_kind is the leading word of the terminator (call,
switch, return, goto …; mir.rs:118-124). callee is the moniker
between call / tailcall and ⟨ (mir.rs:127-134).
has_const_str_arg is true when an argument is k:&str or k:&&str
(mir.rs:145-148). strip_pos drops a trailing @line:col (mir.rs:103-114).
The fn set. A row is an fn or method that Scope::holds (defined in the
dump, in the scoped crates, origin = local unless --include-deps or
--crate, not generated/, not derive-written, not test code unless
--include-tests; mod.rs:716-724), that is not a trait method declaration
(is_trait_method, mod.rs:363-365), and whose name contains --name when
given (twins.rs:550-554). n is the size of this set. --name changes the
math, not only the output: n, A, L and every IDF weight are computed
over the filtered set.
Math
1. API reach (the api class). For rows f, g (twins.rs:564-582):
A[f][g] = 1 iff CALLS f → g and g is in the fn set (n × n)
L[f][c] = 1 iff CALLS f → x, x not in the fn set, x not noise,
and api_family(label(x)) = column c (n × apis)
R = (I + A + A² + … + Aᵏ)·L on the boolean semiring, k = hops
label(x) is <crate> <short moniker> (mod.rs:290-292). A callee outside the
fn set includes a trait method declaration, a dependency fn, and an own fn
excluded by scope or by --name. A callee is noise when its short moniker
starts with one of 22 prefixes (is_noise_callee, twins.rs:82-108):
ops/try_trait/, fmt/, hint/must_use, convert/, ops/deref/,
clone/Clone#clone, ops/drop/, borrow/, ops/function/,
iter/traits/collect/IntoIterator#into_iter, option/impl#[Option<T>],
result/impl#[Result<T, E>], string/impl#[String][Deref],
str/impl#[str]as_bytes, slice/impl#[[T]]len, string/impl#[String]as_str,
ops/arith/, cmp/, panicking/, mem/drop, alloc/, ptr/.
API-family folding (API_FAMILIES, 48 rows, twins.rs:112-161;
api_family, :164-176). The first row whose prefix starts the label wins.
The column is then <family>::<tail>, where tail is the text after the last
/, ], # or space, with trailing (). removed. A label with no matching
row is its own column. So sha2 digest/impl#[D][Digest]update(). and
ring digest/impl#[Context]update(). are both digest::update, and
hex encode(). is hex::encode (unit test twins.rs:1186-1201). The
families are:
| family | prefixes |
|---|---|
| digest | sha2 , sha1 , sha3 , blake3 , blake2 , digest , md-5 , ring digest/ |
| hex | hex , base16ct , faster-hex |
| base-encoding | data-encoding , base64 |
| http-client | reqwest , ureq , hyper |
| fs / io | tokio fs/, std fs/ / tokio io/, std io/ |
| json, serde, toml, ron | serde_json / serde_core , serde / toml / ron |
| tls | rustls , rustls-pki-types , webpki-roots , native-tls , rustls-native-certs |
| clock | chrono , std time/, time |
| version, log, regex | semver / tracing , log / regex |
| compress | zstd , flate2 , lz4_flex , xz2 |
| archive | tar , zip |
| sleep | tokio time/, std thread/sleep |
| random | rand , getrandom , uuid |
The method name is kept, so digest::update ≠ digest::finalize. Two
unrelated methods of one family with the same last segment share a column.
reach_k (crates/mimir-lens/src/lenses/grb.rs:144-155) computes R without the
identity matrix: R ← L; C ← L; repeat k times { C ← A·C; if nnz(C) = 0 stop; R ← R ∨ C }. A·C is GxB_ANY_PAIR_BOOL; on bool that is the structural
OR-AND (LOR.LAND) product, so R[f][c] = 1 iff f reaches API c through at
most k own helpers. The loop stops early when a power is empty.
Document frequency and IDF (twins.rs:597-608, idf_cosine :276-329):
df(c) = |{ f : R[f][c] = 1 }| (column count of R)
|R_f| = |{ c : R[f][c] = 1 }| ("per_fn": every API f reaches)
w(c) = max( ln(n / df(c)), 0.05 ) if df(c) > 0, else 0
rare(c) ⇔ 2 ≤ df(c) ≤ rare_max (df = 1 is never rare)
ln is the natural log and n the size of the fn set. The floor is
IDF_FLOOR = 0.05 (twins.rs:272, :304). It applies when
df(c) > n·e^(−0.05) ≈ 0.951·n. Without it a column every fn has would weigh
ln(1) = 0 and drop out of every norm; with it, that column stays a weak
column. Before idf_cosine counts df, it dedupes the (row, column)
entries, so each column counts once per fn (twins.rs:285-290).
Mask and cosine. R_rare is R restricted to the rare columns.
M = R_rare · R_rareᵀ (pairs that share ≥ 1 rare API)
W[f][c] = w(c) where R[f][c] = 1 (the IDF-weighted pattern)
S⟨M⟩ = W · Wᵀ S[i][j] = Σ_c w(c)² · R[i][c] · R[j][c]
‖w_i‖² = Σ_c w(c)² · R[i][c]
cos(i,j) = S[i][j] / (‖w_i‖ · ‖w_j‖) for i < j with ‖w_i‖·‖w_j‖ > 0
The mask chooses which pairs to compute. The cosine itself runs over every column, common APIs included. The IDF weight enters squared, as in a TF-IDF cosine with binary term frequencies.
api edge (twins.rs:617-639): for i < j,
cos(i,j) ≥ sim ∧ |R_i| ≥ min_apis ∧ |R_j| ≥ min_apis
∧ |rare(i) ∩ rare(j)| ≥ 2
score = cos(i,j); evidence = "cos 0.xx, rare API shared: <first 3>"
The code comment says why the last test is there: "one rare API in common is a
coincidence, two are a recipe" (twins.rs:628).
2. Type matrix (the name+sig class) (twins.rs:650-730). For each fn with
at least one param or ret child:
T[f]["p:" + norm_ty(param ty)] = 1 for each param child
T[f]["r:" + norm_ty(ret ty)] = 1 ("r:()" when there is no ret child)
sig_of(f) = "(p₁, p₂, …) -> r" (display text, params in child order)
norm_ty deletes every space and every lifetime (' plus the identifier
chars after it): &'a mut Vec<u8> → &mutVec<u8> (twins.rs:193-213,
test :1200).
stem(name) (twins.rs:179-190): lowercase; strip each suffix of
_impl, _inner, _once, _v2, _v3, _2, _3 once, in that order; then pop
trailing ASCII digits while the length is above 1; then trim trailing _.
sanitize_seg2 → sanitize_seg, tls_roots_impl → tls_roots,
sha256_hex_v2 → sha256_hex (test :1178-1183). Digit popping also turns
sha256 into sha and base64 into base, which the length rule below then
drops.
The stem mask (twins.rs:683-707) pairs every i < j with the same stem,
after leaving out: trait-impl methods (is_trait_impl_method, mod.rs:415-418),
fns without a type row, stems shorter than 4 chars, the 17 CONSTRUCTORS
stems (new, default, from, into, fmt, with, build, builder, get, set, of, run, name, parse, read, open, load), and stem groups of one or more than 2,000.
The same idf_cosine then runs on T (n = the whole fn set, df over the
T entries).
name+sig edge: same stem ∧ cos_T(i,j) ≥ sim; score = cos_T; evidence = "<stem> <sig>"
The module doc says that a name+sig edge together with an api edge
"strengthens the pair" (twins.rs:13-14). The code has no such step; the two
edges only count as two classes of the group.
A fn with two params of the same printed type sends one column twice. The
entries are deduped first (twins.rs:285-290), so the column is one entry of
W and one term of the norm, and the type cosine stays in [0, 1]. The unit
test the_type_cosine_never_passes_one_when_a_param_type_repeats pins this
(twins.rs, tests module). Before dwarves 27eaa1d the duplicate made W hold
2w against a norm of 2w², and the cosine could exceed 1.
3. The PDG (build_pdg, twins.rs:345-438). One node per MIR block of
every fn in the set, blocks in name order (bb0 first, so bs[0] is the entry
block).
- Control edges
B_c[u][v] = 1iffCFG_SUCC u → vand both blocks belong to the same fn (:399-407). - Data edges
B_d[u][v] = 1iffu ≠ v, same fn, and some local_nis indefs(u)and inuses(v)(writer → reader,:408-424). This is flow-insensitive: any writer links to any reader of the same local, in either block order. There is no reaching-definitions step.
defs_uses (twins.rs:442-492) reads every statement and the terminator,
each with strip_pos applied:
split s at its first '='; if the text after it does not start with '=':
lhs = before, rhs = after else lhs = none, rhs = s (the whole text)
defs ∋ n if lhs.trim() is "_n"
defs ∋ n if s contains "->" and the text after the LAST "->" starts with "_n"
uses ∋ n for every "_n" in rhs whose '_' is at the start or after a non-alphanumeric byte
A call terminator has no =, so its whole text is the right side, and its
destination ->_n is both a def and a use of the same block. An unparsable
payload gets the label hash("?") and no defs, uses or callee (:365-372).
Initial labels (twins.rs:373-385):
callee(b) = api_family( <3rd space-separated token of the moniker> + " " + short_id(moniker) )
("" when the terminator is not a call)
ℓ⁰(b) = hstr( term_kind(b) + "|" + callee(b) + "|" + ("k" if has_const_str_arg else "") )
hstr and h64 are Rust's DefaultHasher over a str or a [u64]
(:215-225). Only label equality matters, never the value. The block
fingerprint fp, statements, locals and switch targets are not hashed into
ℓ⁰. callee_of(b) is the same family string, but None for a noise callee
(:386-393); a fn has a real call when one of its blocks has
callee_of ≠ None (:752-758).
The module doc says the labels are name-free. That holds for locals and
literals, but a call to one of the workspace's own fns hashes its full label
(app gate_ingest().), so the callee's name is in ℓ⁰. Noise callees are
hashed into ℓ⁰ too; noise only decides the real-call test.
WL rounds (wl, twins.rs:497-546). For t = 0 … h−1:
onehot⁽ᵗ⁾[b][λ] = 1 iff ℓ⁽ᵗ⁾(b) = λ (blocks × distinct labels)
N_κ = B_κ · onehot⁽ᵗ⁾ κ ∈ {c = 1, d = 2}, GxB_PLUS_PAIR_UINT64
N_κ[v][λ] = |{ x : B_κ[v][x] = 1 ∧ ℓ⁽ᵗ⁾(x) = λ }| (out-neighbours only)
H(v) = sort{ h64([κ, λ, N_κ[v][λ]]) : κ ∈ {1,2}, N_κ[v][λ] > 0 }
ℓ⁽ᵗ⁺¹⁾(v) = h64([ℓ⁽ᵗ⁾(v)] ++ H(v))
The histogram counts out-neighbours only (successors and readers). A block's predecessors do not change its label. The CCGraph reference in the module doc uses the same product.
Shape signature (twins.rs:759-767). Only fns with
|blocks| ≥ min_blocks and a real call get one (these are fns_with_mir):
sig(f) = h64( sort[ ℓ⁽ʰ⁾(b) : b ∈ blocks(f) ] )
This is the multiset of the final round's labels only. Earlier rounds are
not kept. shape edge: sig(i) = sig(j), every pair inside one hash bucket;
score 1.0 (:770-789).
NEAR, multiset Jaccard (twins.rs:790-846). Over the fns with a sig:
Wc[f][λ] = |{ b ∈ blocks(f) : ℓ⁽ʰ⁾(b) = λ }| (u64 counts), |f| = Σ_λ Wc[f][λ]
C = Wc · Wcᵀ on GxB_PLUS_MIN_UINT64: C[i][j] = Σ_λ min(Wc[i][λ], Wc[j][λ])
J(i,j) = C[i][j] / (|i| + |j| − C[i][j])
NEAR edge: i < j ∧ J(i,j) ≥ near ∧ sig(i) ≠ sig(j); score = J
Σmax is not a second product. It is
Σ_λ max(a_λ, b_λ) = |i| + |j| − Σ_λ min(a_λ, b_λ). The product runs without
a mask; GraphBLAS stores an entry only for pairs that share a label. The
module doc speaks of "Jaccard of the label sets" (twins.rs:21); the code uses
label multisets.
Inlined (twins.rs:848-912). Not GraphBLAS; hash maps over the final
labels.
small fns s: min_blocks ≤ |s| ≤ 4·min_blocks ∧ s has a real call
entry(s) = ℓ⁽ʰ⁾(bs[0]) (the first block by name)
hosts(λ) = { fns with any block labelled λ } (s itself included)
inlined edge (s, H) when a block of H has label entry(s), and
|hosts(entry(s))| ≤ 4 (more than 4 hosts = an idiom, skipped)
H ≠ s ∧ |H| > |s|
∀λ: Wc[s][λ] ≤ Wc[H][λ] (the whole multiset of s sits in H)
score = 0.9; each (s, H) once
The host needs MIR blocks but neither min_blocks nor a real call of its own.
DIVERGENT, the B-7 twin-rules class (twins.rs:915-942). For each
name+sig edge (a, b):
DIVERGENT edge iff sig(a) and sig(b) exist ∧ sig(a) ≠ sig(b) ∧ (a,b) is not a NEAR edge
score = 1.0
So both fns need min_blocks blocks and a real call. A name+sig pair where
either side is too small or has no MIR is neither proven same nor proven
different.
5. Components and groups (twins.rs:944-1004). The twin graph has every
edge except inlined. grb::components builds its pattern and calls
ops::components (nornir crates/mimir-graph/src/ops.rs:511-592):
S = A ∨ Aᵀ (GrB_Matrix_eWiseAdd, GrB_LOR), then label propagation
t = S ·(MIN.SECOND) labels until no label drops, with one pointer jump
labels(i) = min(labels(i), labels(labels(i))) per round. The label is the
smallest row index in the component.
- A non-inlined edge
(a, b)putsaandbinto grouplabel(a). - An inlined edge
(s, H)putssandHinto grouplabel(s). Ifshas no other edge, that is a new two-member group. IfHis in another group,His listed in both. - A group's edges ("mine") are the edges with
label(a) = group, plus inlined edges withlabel(H) = group. So an inlined edge also adds the classinlinedto the host's group. classes= the set of edge classes in "mine".near= the number of NEAR edges,best= the maximum score, evidence = "<class> <a> ↔ <b>: <why>", sorted and deduped.- STRONG ⇔
|classes ∩ {api, name+sig, shape, NEAR}| ≥ 2(is_strong,twins.rs:254-260). Two INDEPENDENT classes must agree. DIVERGENT is derived from name+sig (the same pair, a different body), and inlined is containment, so neither is a second vote. (Before dwarves 27eaa1d the rule was|classes| ≥ 2over all six classes, so every DIVERGENT group was STRONG.)
Sort order (twins.rs:1005-1020), descending unless noted:
- has a NEAR edge;
- has a DIVERGENT edge;
- number of classes;
- number of members;
- best score;
- label of the first (smallest) member, ascending.
BLOB (MAX_GROUP = 60, twins.rs:1081-1101). A group with more than 60
members prints as BLOB ×N in C crates [classes] with the hint "raise --sim /
--min-blocks or use --name", and its JSON is
{size, crates, classes, blob: true}. It still counts in groups,
groups_strong and fns_with_twin.
GraphBLAS steps
| step | operation | semiring / op as named in code | computes | file:line |
|---|---|---|---|---|
| 1 | A, L from edge lists | Matrix::<bool>::from_edges (duplicates collapse) | call pattern, API pattern | twins.rs:593-594, grb.rs:21-30 |
| 2 | C ← A·C, k times | GxB_ANY_PAIR_BOOL (structural OR-AND) | one more helper hop | grb.rs:33-54, :148 |
| 3 | R ← R ∨ C | GrB_Matrix_eWiseAdd_BinaryOp with GrB_LOR | R = (I + A + … + Aᵏ)·L | grb.rs:152, nornir ops.rs:624-641 |
| 4 | df, rare filter | not GraphBLAS (counts over edges_of(R)) | df(c), R_rare | twins.rs:596-608 |
| 5 | M = R_rare·R_rareᵀ | GxB_ANY_PAIR_BOOL, GrB_DESC_T1 | pairs sharing a rare API | twins.rs:611, grb.rs:33-54 |
| 6 | S⟨M⟩ = W·Wᵀ | GrB_PLUS_TIMES_SEMIRING_FP64, GrB_DESC_ST1 (structural mask, transpose B) | Σ_c w(c)² over shared APIs | twins.rs:316, grb.rs:58-81 |
| 7 | norms and division | not GraphBLAS | cos(i,j) | twins.rs:312-325 |
| 8 | stem mask | not GraphBLAS (BTreeMap by stem), then pattern | same-stem pairs | twins.rs:683-707 |
| 9 | S_T⟨stem⟩ = W_T·W_Tᵀ | GrB_PLUS_TIMES_SEMIRING_FP64, GrB_DESC_ST1 | type cosine | twins.rs:708, grb.rs:58-81 |
| 10 | B_c, B_d, onehot | from_edges | PDG and label one-hot | twins.rs:503-517 |
| 11 | N_κ = B_κ·onehot, per round and kind | GxB_PLUS_PAIR_UINT64, no descriptor | out-neighbour label histogram | twins.rs:527, grb.rs:102-117 |
| 12 | relabel, sig(f), shape buckets | not GraphBLAS (DefaultHasher, BTreeMap) | ℓ⁽ᵗ⁺¹⁾, sig, shape edges | twins.rs:533-543, :759-789 |
| 13 | C = Wc·Wcᵀ | GxB_PLUS_MIN_UINT64, GrB_DESC_T1 | multiset intersection Σ min | twins.rs:822, grb.rs:121-136 |
| 14 | inlined, DIVERGENT | not GraphBLAS (hash maps, set lookups) | containment, divergence | twins.rs:848-942 |
| 15 | components | GrB_LOR for A ∨ Aᵀ; GrB_mxv with GxB_MIN_SECOND_UINT64; GrB_MIN_UINT64; GrB_Vector_extract (pointer jump) | weakly connected components | twins.rs:952, grb.rs:166-178, nornir ops.rs:511-592 |
Dataflow
Mímir v2 dump matrices / indexes products edges output
───────────── ────────────────── ──────── ───── ──────
CALLS (fn→fn in set) ───────────► A n×n ─┐
CALLS (fn→other, −noise) ──┐ ├─► R=(I+A+A²)·L ─┬─► df, w=max(ln(n/df),.05)
API_FAMILIES fold ───────┴────► L n×apis┘ ANY.PAIR ├─► M=R_rare·R_rareᵀ (ANY.PAIR)
└─► S⟨M⟩=W·Wᵀ (PLUS.TIMES) ─► cos ──► api ─────┐
param/ret ty (norm_ty) ─────────► T n×types ─┐ │
fn name (stem, −CONSTRUCTORS) ──► stem mask ───┴─► S_T⟨stem⟩=W_T·W_Tᵀ ──────────────────► cos ──► name+sig ──┤
│ │
mir_block payload ─► defs_uses ─► B_d blocks×blocks ─┐ ▼ │
CFG_SUCC ───────────────────────► B_c blocks×blocks ─┼─► N_κ=B_κ·onehot(ℓᵗ) (PLUS.PAIR) ×h DIVERGENT ──┤
mir_block term/callee/k ────────► ℓ⁰ per block ──────┘ ℓᵗ⁺¹=h(ℓᵗ,H) (sig≠, ¬NEAR)│
sig(f)=h(sorted ℓʰ) ─────────────► shape ──────┤
Wc fn×label counts │
C=Wc·Wcᵀ (PLUS.MIN) ─► J ──────────► NEAR ───────┤
entry label ⊆ host ───────────────► inlined ┐ │
│ ▼
components(A∨Aᵀ, MIN.SECOND) ◄── all but inlined ───┼── edges
│
groups (inlined filed under label(s)) ◄──────────────┘
→ classes, STRONG (≥2), sort, BLOB > 60
→ summary {fns … fns_with_twin, name_families_top}
→ findings {size, crates, classes, strong, near_pairs,
divergent, members, locs, evidence}
Output
Summary keys (twins.rs:1041-1080): fns (n), fns_with_mir (fns with a
sig), apis (columns of L), rare_apis, pairs_api, pairs_name_sig,
pairs_shape, pairs_near, pairs_divergent, pairs_inlined, groups,
groups_strong, fns_with_twin (distinct members over all groups, BLOBs
included), name_families_top. The last one is the top 12 stems (non-trait-impl
fns) whose members sit in at least 2 distinct files, as stem ×N, sorted by
size descending, then stem.
Each group finding (:1102-1148) prints
[NEAR ][DIVERGENT ][STRONG ]×N in C crates [classes], one line per member
with file:line, and up to 6 evidence lines (then "… K more pairs"). Its JSON is
{size, crates, classes, strong, near_pairs, divergent, members, locs, evidence}.
Class words, in this order: NEAR, DIVERGENT, api, name+sig, shape, inlined.
The headline repeats the counts (:1026-1040), and one note line names the
formulas with the run's knob values (:1150-1153).
Worked example 1: normalize_name ×3 on Holger (golden)
The pinned dump is Holger h32/next 9dbe5cbf (crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json:2-7).
The full run at the defaults gives (:21-35):
| fns | fns_with_mir | apis | rare_apis | api | name+sig | shape | NEAR | DIVERGENT | inlined | groups | STRONG | with twin |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 6,603 | 4,243 | 2,496 | 1,496 | 473 | 312 | 179 | 149 | 71 | 10 | 327 | 41 | 889 |
The plan's §E row prints "1,496 rare APIs of 2,504"; the golden file, which a
test pins, says apis = 2,496.
The B-7 golden row runs --name normalize_name --min-blocks 4
(crates/mimir-lens/tests/golden_holger.rs:136-155) and pins (golden json :36-46):
fns 3, pairs_api 1, pairs_name_sig 3, pairs_shape 1, pairs_near 0, pairs_divergent 2, groups 1, groups_strong 1, fns_with_twin 3. The plan
(§D round 7) names the three fns: pip.rs and znippy-python, 5 blocks each,
and wheel.rs, 30 blocks. Here is why each edge forms.
- Fn set.
--namekeeps only the three fns, son = 3. Every other callee is outside the set, soAis empty unless the three call each other, andR = L. IDF is over 3 rows: an API reached by 2 of them weighsln(3/2) = 0.405, one reached by all 3 weighsmax(ln 1, 0.05) = 0.05. Withn = 3, every API reached by at least 2 of the 3 is rare (2 ≤ df ≤ 12). - name+sig = 3.
stem("normalize_name") = "normalize_name"(14 chars, not a constructor), so the stem mask holds all 3 pairs. The golden says all 3 type cosines are ≥ 0.8. If the three signatures are equal, everyTcolumn hasdf = 3, weight 0.05, and every cosine is exactly 1. - shape = 1.
pip.rsandznippy-pythonhave 5 blocks (≥ the run'smin_blocks4) and a real call, and the same final WL label multiset. At the defaultmin_blocks = 8they would get nosig, which is why the golden row passes--min-blocks 4. - api = 1. The same two fns pass the api test: cosine ≥ 0.8, at least 2
APIs each, at least 2 shared rare APIs. The pairs with
wheel.rsfail one of these tests; the golden does not say which. - NEAR = 0. For a 5-block fn against the 30-block fn,
Σmin ≤ 5, soJ ≤ 5 / (5 + 30 − 5) = 5/30 = 0.167 < 0.8. - DIVERGENT = 2. The name+sig pairs
(pip.rs, wheel.rs)and(znippy-python, wheel.rs)have twosigs (30 ≥ 4 blocks, a real call), thesigs differ, and they are not NEAR (step 5). The pair(pip.rs, znippy-python)has equalsigs, so it is not DIVERGENT. - One group, STRONG. The edges are pip–znippy {api, name+sig, shape},
pip–wheel {name+sig, DIVERGENT} and znippy–wheel {name+sig, DIVERGENT}. That
is one component of 3 with classes {DIVERGENT, api, name+sig, shape}. Three
of them are independent (api, name+sig, shape), so it is STRONG, and it
prints as
DIVERGENT STRONG ×3 … [DIVERGENT+api+name+sig+shape].
Reading: two of the PEP 503 normalisers are the same rule, and wheel.rs
claims the same name and signature with a different body. The lens cannot tell
which body is wrong; that needs a test. The review's "PEP 503 ×5 with 2 wrong"
is these 3 plus two normalisers under other names (normalize,
normalise_*), found with --name normal (plan §D round 7).
Worked example 2: the fixture by hand
Fixture fixture() in crates/mimir-lens/tests/graph_lenses.rs:293-608 (twins part
:397-533), run by twins_finds_api_name_sig_shape_near_and_inlined_twins_on_graphblas
(:767-845) with min_blocks = 4 and the other defaults. The numbers below
are computed by hand from the fixture. The test asserts lower bounds
(pairs_api ≥ 3, pairs_near ≥ 1, …) and the exact values
pairs_name_sig = 2 and pairs_divergent = 1.
The fn set, n = 27. The fixture defines 29 local fn/method nodes. Two
are dropped: t_restore is test code and Store#put is a trait declaration.
The sha2, hex, ring and anyhow fns are not defined in the dump, and
dep_x huge has origin = registry, so they are not rows either. The 27 rows are: impl Disk put, fs_write,
http_handle, gate_ingest, mirror_run, restore, big, log, h_new,
h_enc, sha_a, sha_b, fmt sha_c, sha_near, small_fn, host_fn,
p1/normalize, fmt normalize, a/tls_roots, b/tls_roots, ladder,
f1, f2, boom, stringly, gated, partly.
API reach. L has 4 columns: digest::new (h_new → sha2, sha_c → ring:
the family table folds both), hex::encode (h_enc, sha_c),
app Store#put(). (gate_ingest, mirror_run, restore: a trait declaration is
outside the set) and anyhow __private/format_err(). (boom). A holds
sha_a/sha_b → h_new, h_enc; http_handle → gate_ingest; mirror_run, restore → log;
Disk put → fs_write; f1 ↔ f2. A·L adds sha_a, sha_b → {digest::new,
hex::encode} and http_handle → Store#put. A²·L is empty, so reach_k stops.
| column | reached by | df | rare | w = max(ln(27/df), 0.05) |
|---|---|---|---|---|
| digest::new | h_new, sha_c, sha_a, sha_b | 4 | yes | ln 6.75 = 1.9095 |
| hex::encode | h_enc, sha_c, sha_a, sha_b | 4 | yes | 1.9095 |
| app Store#put(). | gate_ingest, mirror_run, restore, http_handle | 4 | yes | 1.9095 |
| anyhow format_err | boom | 1 | no | ln 27 = 3.2958 |
So apis = 4, rare_apis = 3. With w = 1.9095, w² = 3.6464:
cos(sha_a, sha_c) = (w² + w²) / (√(2w²) · √(2w²)) = 7.2928 / 7.2928 = 1.000
|R| = 2 each ≥ min_apis, shared rare = {digest::new, hex::encode} = 2 → api edge
(sha_a, sha_b) and (sha_b, sha_c): the same → 3 api edges
cos(sha_a, h_new) = w² / (√(2w²) · √(w²)) = 1/√2 = 0.707 < 0.8 → no edge
cos(gate_ingest, mirror_run) = 1.000, but |R| = 1 < min_apis = 2 → no edge
pairs_api = 3. Here all rare weights are equal, so IDF cancels; it matters
when the shared APIs have different df.
name+sig. Stems normalize (2 fns) and tls_roots (2 fns) pass the mask;
stringly has a type row but no stem partner. Each pair has identical type
rows (p:&str, r:String and p:&Path, r:Vec<u8>), so cosine = 1 and
pairs_name_sig = 2.
PDG and WL for sha_a (:430-451). The fixture's call terminators are
call rust-analyzer cargo <crate> * <path>⟨_1⟩->_2@1:1 and its switch is
switch(^_2;0,1)@1:5 (:130-139).
| block | statements, terminator | defs | uses | ℓ⁰ text | control out | data out |
|---|---|---|---|---|---|---|
| bb0 | _3=use(_1), call sha2 … new() | 2, 3 | 1, 2 | call|digest::new| | bb1 | bb1, bb2 |
| bb1 | _4=use(_3), call hex encode() | 2, 4 | 1, 2, 3 | call|hex::encode| | bb2 | bb0, bb2 |
| bb2 | _2=discr(_4), switch | 2 | 2, 4 | switch|| | bb3, bb4 | bb0, bb1 |
| bb3, bb4 | return | – | – | return|| | – | – |
The data edges come from _2: bb0, bb1 and bb2 all write it, and all three
read it, so they form a 6-edge clique (plus _3: bb0 → bb1, and _4:
bb1 → bb2, already in it). bb3 and bb4 have no out-neighbours, so
ℓ¹ = h64([ℓ⁰(R)]) and ℓ² = h64([ℓ¹]) for both: one label ρ, twice.
sha_b is a copy, so sig(sha_a) = sig(sha_b): 1 shape edge.
sha_near is the same five blocks plus bb5: goto → bb0 (:452-453). bb5 is
only a predecessor of bb0, and WL counts out-neighbours only, so bb0 … bb4 get
the same ℓ² as in sha_a. The multiset Jaccard is then:
Wc[sha_a] = {ℓ(bb0):1, ℓ(bb1):1, ℓ(bb2):1, ρ:2} |sha_a| = 5
Wc[sha_near] = {ℓ(bb0):1, ℓ(bb1):1, ℓ(bb2):1, ρ:2, ℓ(bb5):1} |sha_near| = 6
Σmin = 1 + 1 + 1 + 2 = 5
J = 5 / (5 + 6 − 5) = 5/6 = 0.833 ≥ 0.8, and sig differs → NEAR (sha_a, sha_near)
The same holds for (sha_b, sha_near). small_fn (4 blocks) and host_fn
(the same 4 blocks after a leading goto, :454-466) give
J = 4 / (4 + 5 − 4) = 4/5 = 0.8, which meets the floor: a third NEAR edge.
p1/normalize (4 blocks) against fmt normalize (7 blocks) share only one
return label (p1's switch has 1 return successor, fmt's has 2), so
J = 1 / (4 + 7 − 1) = 0.1. That pair has a name+sig edge, two different
sigs and no NEAR edge, so it is the 1 DIVERGENT pair the test asserts,
with the evidence "same name and signature ((&str) -> String), a different body".
Inlined. small_fn's entry label is host_fn's bb1 label, 2 hosts, its
multiset sits in host_fn's, and 5 > 4 blocks: an inlined edge. sha_a and
sha_b are small fns too (5 ≤ 16 blocks), and their multisets sit in
sha_near (6 blocks): two more inlined edges.
Groups. The components over the non-inlined edges are
{sha_a, sha_b, sha_c, sha_near} with classes {NEAR, api, shape, inlined},
{small_fn, host_fn} with {NEAR, inlined}, {p1/normalize, fmt normalize} with
{DIVERGENT, name+sig}, and {a/tls_roots, b/tls_roots} with {name+sig}. Only
the sha group is STRONG: it has three independent classes (NEAR, api, shape).
small/host has one (NEAR; inlined does not vote), and normalize has one
(name+sig; DIVERGENT does not vote). The sort puts the sha group first (NEAR, 4 classes),
then small/host (NEAR, 2 classes), then normalize (DIVERGENT), then tls_roots.
The test checks that the sha group holds fmt sha_c() and app sha_near(),
and that name_families_top contains tls_roots ×2 (two files). With
--name tls, n = 2 and there is 1 group (:831-844).
Limits
- MIR is name-free for locals and literals: two bodies that differ only in a literal of the same type have the same shape. The two wrong PEP 503 normalisers differ in shape, so they come out as non-twins, not as NEAR.
- Up to schema 5 a body over the driver's 1,024-block cap was cut to its first 1,024 blocks. Since 6 every block is written, so a long body is compared whole.
- Data edges are flow-insensitive (any writer of
_nto any reader of_n). WL sees out-neighbours only. - DIVERGENT says that two bodies differ. It cannot say which one is wrong.
--namerecomputesn,A,Land IDF over the filtered set, so its counts are not a slice of the full run.
needs
Each refusal names cargo mimir-dump as the producer:
"CALLS edges" when there are none in scope (twins.rs:642-647), "param /
return types" when T is empty (:725-730), and "mir_block nodes" when no fn
has MIR (:740-746). Without MIR the shape, NEAR, inlined and DIVERGENT
classes are empty.
Tests
crates/mimir-lens/tests/graph_lenses.rs:767twins_finds_api_name_sig_shape_near_and_inlined_twins_on_graphblas(fixture:293, twins part:397-533).- Unit tests
stems_fold_numbered_and_impl_suffixes(crates/mimir-lens/src/lenses/twins.rs:1178) andthe_api_family_table_folds_libraries_that_are_the_same_thing(:1186, includesnorm_ty). - MIR payload parsing:
crates/mimir-lens/src/lenses/mir.rs:216,:228. - Holger goldens
golden_twins_on_holger(crates/mimir-lens/tests/golden_holger.rs:124), the full row and thetwins.normalize_namerow (section 7.6). On the lossless graph (schema 8, lens upgrade batch 1). A shape pair is two fns of one WL shape; the shape is name-free and value-free, sobase * 2andbase * 3were one shape with nothing to tell them apart. The row holds the constants:
const(f) = the integer and string constants of f's bodies, in site order (k_lo, k_bytes)
PARAM(f, g) ⇔ shape(f) = shape(g) ∧ const(f) ≠ const(g) the pair's `why` names the differing positions (a→b)
same ⇔ shape(f) = shape(g) ∧ const(f) = const(g) the pair's `why` says `same constants`
The twin graph does not change (a PARAM pair is still a shape pair): the row
gains the reading. Summary key pairs_param. Fixture
twins_name_the_constants_that_differ_in_a_shape_pair.
4.3 dispatch-erosion
- Run.
cargo lens dispatch-erosion [--min-variants 4] [--min-arms 3] [--src <checkout>] [--crate NAME…] [--include-tests] [--include-deps](dw:src/cli.rs:341-352, scope flags dw:src/cli.rs:553-573). - Finds. Three shapes of behaviour chosen by a value instead of by a type:
match ladders over an enum that lives in another crate or another module,
string compares against a constant, and parallel enums (two enums with almost
the same variant names) (
crates/mimir-lens/src/lenses/dispatch.rs:1-13). - Reads.
enumnodes and theirvariantchildren (CONTAINS), and themir_blockpayloads of every fn in scope (thetycolumn, grammar incrates/mimir-lens/src/lenses/mir.rs:5-11). Nothing else.
Math. Let V(e) be the set of variant names of enum e (its CONTAINS
children of kind variant).
E = { e : kind = enum, defined_here, ¬derived(e), ¬test_code(e), |V(e)| ≥ min_variants } min_variants = 4
F = { f : kind ∈ {fn, method}, Scope.holds(f) }
B_f = the mir_block nodes of f
Ladder: a block b ∈ B_f whose terminator is switch(_n; t₁,…,t_k) and that holds a
statement _n = discr(_m) (a `*` deref suffix is allowed); τ = the type of _m in
b's local list.
arms(b) = k + 1 (every listed target plus the otherwise arm)
keep b only when arms(b) ≥ min_arms min_arms = 3
name = bare_type(τ) (strip &, mut, lifetimes, Option<…>, Result<…,_>,
Box<…>, Arc<…>, Rc<…>, Cow<…>; cut at the first
non [A-Za-z0-9_] char)
C = { c ∈ E : name(c) = name }
e = the c ∈ C with crate(c) = crate(f); else the only c when |C| = 1;
else no pick (the name goes to the "ambiguous" note)
class(b) = ForeignCrate if crate(e) ≠ crate(f)
OtherModule if module_path(e) ≠ module_path(f)
Home otherwise
String site: a block b ∈ B_f whose terminator is call/tailcall to callee c, where
short(c) starts "str/traits/impl#[str][`PartialEq" and ends "eq()." → "=="
short(c) starts "str/impl#[str]" and ends "]contains()." or
"]eq_ignore_ascii_case()." → "==~"
short(c) starts "str/impl#[str]" and ends "]starts_with()." / "]ends_with()." /
"]strip_prefix()." / "]strip_suffix()." → "sniff"
and the argument list holds the token k:&str or k:&&str (a constant string).
eq(f) += 1 for "==" and "==~"; sniff(f) += 1 for "sniff".
ladders_foreign = |{b : class = ForeignCrate}| arms_foreign = Σ arms(b) over them
ladders_other_module = |{b : class = OtherModule}| arms_other_module = Σ arms(b) over them
ladders_home = |{b : class = Home}|
string_eq_sites = Σ_f eq(f) string_sniff_sites = Σ_f sniff(f)
string_sites_files = |{ file(f) : f has a string site }|
enums_considered = |E|
Parallel enums: for every unordered pair e₁ < e₂ of E (by node index):
J(e₁,e₂) = |V(e₁) ∩ V(e₂)| / |V(e₁) ∪ V(e₂)| keep when J ≥ 0.5
sort: J descending, then the label of e₁ ascending.
The code is in crates/mimir-lens/src/lenses/dispatch.rs:84-100 (E), :126-192 (ladders and
string sites), :146-167 (candidate pick and class), :39-63 (the callee
classes), :249-268 (Jaccard), and crates/mimir-lens/src/lenses/mir.rs:144-148
(k:&str), :152-178 (switch_on_discr, arms = targets + 1 at :176),
:183-209 (bare_type).
Ranking of the findings:
enumfindings: one per enum with at least one non-home ladder, sorted byladders_foreign + ladders_other_moduledescending, then label; inside, the sites sorted by arms descending, then fn label; the text shows 12 sites, the JSON all (crates/mimir-lens/src/lenses/dispatch.rs:306-371).parallelfindings: in the Jaccard order above (:372-385).stringsfindings: files sorted byeq + sniffdescending, then path. Only the first 40 files get a finding (:396-448).string_sites_filesstill counts every file.- With
--src, the literals are read from the source lines of each string-site fn in those 40 files: lines[start_line, end_line]of the fn, not starting with//, that contain== ",!= ",_with(",contains(", or start with"and contain=>. Every quoted literal of 1–40 bytes on such a line counts once (:66-79,:402-436).vocabulary_top= the 20 most frequent literals, count descending, then literal (:454-465).
Where the module doc and the code differ:
- The module doc names two classes: "foreign when the enum's crate is not the
fn's crate, home otherwise" (
crates/mimir-lens/src/lenses/dispatch.rs:4-7). The code has three: another crate, another module of the same crate, and home (:109-167). - The module doc lists
str::eq/starts_with/ends_with/contains(:8-9). The code also countseq_ignore_ascii_case(with==) andstrip_prefix/strip_suffix(as sniffs), andcontainscounts with==, not as a sniff (:39-63). - E is filtered by
defined_here, derive and test only (:88), not by--crateororigin. The scope applies to the fns that hold the switch, not to the enum it switches over.
GraphBLAS steps. None. This lens is a scan of MIR block payloads and set
arithmetic in BTreeMap / BTreeSet.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | collect E and V(e) | none (CONTAINS children scan) | enums with ≥ 4 variants | crates/mimir-lens/src/lenses/dispatch.rs:84-100 |
| 2 | parse each mir_block, match switch(_n;…) + _n = discr(_m) | none (string parse) | ladders and their arms | crates/mimir-lens/src/lenses/mir.rs:152-178 |
| 3 | candidate pick, class | none (hash lookup by bare name) | ForeignCrate / OtherModule / Home | crates/mimir-lens/src/lenses/dispatch.rs:142-167 |
| 4 | callee classes on call blocks | none (prefix/suffix test) | == and sniff sites | crates/mimir-lens/src/lenses/dispatch.rs:175-188 |
| 5 | pairwise Jaccard over E | none (O( | E | ²) set loop) |
| 6 | literal text fallback | none (file read, --src) | literals, vocabulary_top | crates/mimir-lens/src/lenses/dispatch.rs:402-465 |
Dataflow.
code_nodes kind=enum ──CONTAINS──▶ kind=variant ──▶ E, V(e) ─────────────┬──▶ Jaccard pairs ──▶ parallel_enum_pairs, kind=parallel
│
code_nodes kind=mir_block (ty payload) ──parse──▶ switch(_n;…) + discr ──▶ arms, bare_type(τ)
(owner fn ∈ Scope) │
pick e by name/crate ◀─┘
│
class Foreign / OtherModule / Home
│
ladders_*, arms_*, kind=enum findings
code_nodes kind=mir_block ──call terminator──▶ str callee + k:&str ──▶ eq / sniff per fn
│
per file (top 40) ── --src lines ──▶ literals, vocabulary_top
└──▶ string_eq_sites, string_sniff_sites, string_sites_files
Worked example (fixture). The fixture in crates/mimir-lens/tests/graph_lenses.rs:528-560
defines crate fmt with enum Kind {A, B, C, D} and enum Kind2 {A, B, C, E},
and in crate app a fn ladder with two MIR blocks:
bb0:switch(^_2;0,1,2)with_2=discr(_1)and local1:Kind. The switch lists 3 targets, soarms = 3 + 1 = 4 ≥ 3.bare_type("Kind") = Kind;C = {fmt Kind}; noKindin crateapp, and|C| = 1, soe = fmt Kind.crate(e) = fmt ≠ app, so the class is ForeignCrate.bb1: a call tostr/traits/impl#[str][PartialEq]eq().with arguments_65,k:&str: one==site.
Both enums have 4 variants, so enums_considered = 2. Jaccard:
|{A,B,C}| / |{A,B,C,D,E}| = 3 / 5 = 0.60 ≥ 0.5, one parallel pair. The test
asserts ladders_foreign = 1, arms_foreign = 4, ladders_other_module = 0,
string_eq_sites = 1, parallel_enum_pairs = 1, the text
PARALLEL 0.60 fmt Kind# ≈ fmt Kind2#, and, without --src, "needs the
compared literals' values" (crates/mimir-lens/tests/graph_lenses.rs:844-864). A second worked
check is the unit test in crates/mimir-lens/src/lenses/mir.rs:216-225: a switch with the 15
targets 0,…,14 gives 16 arms, which is why an exhaustive match over Holger's
15-variant ArtifactFormat counts 16 arms (the otherwise arm is counted even
when it is unreachable).
Worked example (Holger). Golden row on h32/next 9dbe5cbf
(crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json, key dispatch-erosion):
| field | value |
|---|---|
enums_considered | 119 |
ladders_foreign / arms_foreign | 31 / 136 |
ladders_other_module / arms_other_module | 73 / 389 |
ladders_home | 187 |
string_eq_sites / string_sniff_sites | 1,066 / 495 |
string_sites_files | 187 |
parallel_enum_pairs | 14 |
Mean arms per non-home ladder: (136 + 389) / (31 + 73) = 525 / 104 ≈ 5.05.
The plan (§E.2) names the rows behind these: ArtifactFormat (15 variants)
with identity::Format::of 16 arms and security::ecosystem_of 16 arms;
pull_intake::RemoteKind from door::remote 8 arms; parallel pairs
RemoteKind ≈ SourceKind 0.67, the two ConnectorAuth enums 1.00, and
Credential ≈ UpstreamAuth 0.5 (exactly at the threshold, kept because the
test is ≥).
-
Output. Summary
enums_considered, ladders_foreign, arms_foreign, ladders_other_module, arms_other_module, ladders_home, string_eq_sites, string_sniff_sites, string_sites_files, parallel_enum_pairs, plusvocabulary_topwhen--srcfound literals (crates/mimir-lens/src/lenses/dispatch.rs:280-296,:457). Findings{kind: enum, enum, loc, variants, ladders, arms, ladders_other_module, arms_other_module, files, crates, home_ladders, sites[{fn, loc, arms, at}]},{kind: parallel, jaccard, a, b, shared}and{kind: strings, file, eq, sniff, literals}(:356-369,:383,:446). -
Limits. Literal values need
--src: constants are name-free in MIR (:449-453). Amatchon&strcompiles to<str as PartialEq>::eqcalls and counts as==. Two enums with the same bare name in two crates, neither in the fn's crate, are skipped and named in a note (:157-159,:297-303). With nomir_blocknodes the lens says "needs mir_block nodes" (:101-106). Since schema 6 a body over 1,024 blocks (Holger'shandle) has all its blocks. -
The enum by the full type (schema 8, lens upgrade batch 1). The lossy view knew the switched local's short type (
1:Role), so a ladder went to an enum by its bare name. The lossless row holds the full type, and the lens reads it:ty(sw) = the full type of the place whose discriminant the switch reads: local_ty of the local (references peeled), or the field type when the place ends in a field path(ty) = `<crate>::<modules>::<Name>` (rustc prints a type of the body's own crate without the crate) enum(sw) = the table enum whose module path + name = path(ty) on_the_enum ⇔ the last segment of ty(sw) is the enum's name ladder(sw) ⇔ enum(sw) exists; ¬on_the_enum ⇒ no ladder (the switch is over a holder: Option<E>, Result<E, _>) fallback the bare name, when the dump is schema ≤ 7 or the path finds no enum (counted: ladders − ladders_by_type)Two wrong results go. (1) Two enums of one name: the fixture
dispatch_erosion_finds_the_enum_by_the_full_type_of_the_switched_valuehasaaa::Roleandmirror::Rolein one crate, and the lossy rule took the first (Gunnar'smirror::Roleladders went toxtask setup_svg::Rolethe same way). (2) A switch over a holder was counted as a ladder over the enum inside it. Checked by hand on Holger 9dbe5cbf:plugin-abi v2::check_grants(v2.rs:951) isfor want in &manifest.requires, a loop whosenext()is anOption<&Capability>, counted three times as a 3-arm ladder overCapability;sandbox::remote::handle(remote.rs:484) matches anOption<Ask>;holger run_promote(cli lib.rs:2651) matches aResult<nornir_gate::Decision, PromoteError>, and the bare name gave it to Holger's ownsecurity::Decision. A nested match (Some(Format::Npm) => …) still counts: its inner switch reads the field's discriminant, and the field's type names the enum.dump ladders foreign other module home not on the enum by type Holger h32 9dbe5cbf 28 → 10 73 → 53 187 → 173 54 227 of 236 Gunnar c17a6ef4 default 70 → 61 44 → 20 102 → 101 51 151 of 182 The lenses that read these rows follow. Holger
pipeline-subclass-leak: 5 → 4 structs.Metricsleaves, becausemetrics::Verb,WireVerbandAuditActionno longer have 3 real ladders (a counter per verb is no leaf state).dyn-to-enumenums_seen40 → 35. Summary keysladders_by_typeandswitches_not_on_the_enum; the first 8 dropped switches are named in a note with their type.NORNIR_LENS_LOSSLESS=offgives the lossy reading on the same dump. The lens takes 1.1 s on Holger (0.7 s lossy). -
The compared literals from the graph (schema 8, lens upgrade batch 1). A
&strconstant is bytes in the row (k_kind = BYTES,k_bytes), so the compared literals and the literal arguments need no--src:lits(f) = ⋃ { str_args(c) : c a compare or table call site of f } str_args = the &str constants among the call's arguments LA(c, i) = { (f, s) : argument i of call site s in f is k:&str } its value = the i-th &str constant of the site vocabulary = the multiset of lits over the filesThe STR rows and the LITERAL-ARG rows carry the values on every schema-8 dump;
--srcis read only forconst NAME: &stritems andconst TABLE: &[&str]lookups, whose values live in static memory (code_data, not written yet): the needs row says so. Fixturedispatch_erosion_reads_the_compared_literals_from_the_constants. -
Self ladders (round 4, Gunnar re-compare row 8). An enum with 2 or 3 variants is under the variant floor, but a
match selfin many of its own methods is behaviour picked per variant at every call:SELF(E) = { m ∈ methods(impl E) : ∃ b, term(b) = switch(discr(_1 | _1*)) over E }, a row when|SELF(E)| ≥ min_self_ladders = 10. On Gunnar's default dump:SELF-LADDERS gunnar-server storage/Engine# (2 variants, crates/gunnar-server/src/storage.rs:172) — 18match selfladders in 18 methods of its own impl: one trait with 2 impls would carry them(goldengolden_gunnar_dispatch_engine_is_matched_on_self_in_18_methods; the source has 20match self, two of them in fns whose arms the build folds). The metered dump has 0: thereEnginehas one variant (the other is#[cfg(feature = "engine-gix")]). Holger: 0. Summaryself_ladder_enums; finding{kind: self_ladders, enum, loc, variants, ladders, methods}. -
Const strings and literal arguments (round 4, Gunnar re-compare row 9). Three string shapes the first passes missed:
x == Ok(CONST)/x == Some("v"): rustc promotes the wrapped constant, so the compare isResult::eq/Option::eqagainst ak:&Result<&str, _>/k:&Option<&str>operand. It counts as==.TABLE.contains(&x)over ak:&[&str]operand (aconst TABLE: &[&str]) is a table lookup: summarystring_table_sites,/ N tablein the STR row.- With
--src, a compared line that names aconst NAME: &strorconst NAME: &[&str]item of the same file gets the item's literals. - The literal-argument table (§5.2):
LA(c, i) = { call sites of c whose argument i is k:&str (or a local set by use(k:&str)) }, aLITERAL-ARGrow when|LA(c, i)| ≥ min_literal_args = 10; with--srcthe literals are read from the call's line and up to 3 more. The STR rows now list every file, not the top 40.
On Gunnar's default dump (golden
golden_gunnar_dispatch_method_names_and_verbs_are_literals):==sites 659 → 674, 8 table sites, 12 literal-argument rows. The heaviest:LITERAL-ARG gunnar-grpc service/impl#[ServiceContext]require(). arg 5 ← 94 literal sites in 91 fns of 14 files [Access.AddKey, Access.ListKeys, Admin.Capacity, Admin.ReadLogs, Admin.StorageInfo, Admin.Topology, …] LITERAL-ARG gunnar-grpc service/impl#[ServiceContext]authorize(). arg 5 ← 29 literal sites in 29 fns of 7 files [Accounts.Gate, Admin.Health, Admin.Stats, Usage.ReadAll] STR 11 == / 1 sniff / 2 table crates/gunnar-ssh/src/server.rs [GIT_PROTOCOL, LC_, git-lfs-authenticate, gunnar-fire-drill, gunnar-recipient-keys, gunnar-repo, gunnar-tunnel-address, gunnar-whoami] STR 2 == / 0 sniff / 2 table crates/gunnar-grpc/src/service/mod.rs [Access.ListGrants, …, Repos.Create, Repos.Delete, …]Hand check: the server.rs verbs are the
*_VERBconsts (server.rs:907–972, B3), compared as== Ok(FIRE_DRILL_VERB)(:2699). mod.rs isrefuse_on_browse(§7c #8): thematch method { "Repos.Delete" | "Repos.Create" => .. }at :377 and theBROWSE_STILL_ANSWERS/TWIN_STILL_ANSWERSlookups.require(.., method: &'static str)is the §5.2 table candidate. Holger:==1,067 → 1,094, 5 table sites, 16 rows (flow::Row::new136 sites,grpc::coded64,ErrCode::tag54). -
Tests.
crates/mimir-lens/tests/graph_lenses.rs:844dispatch_erosion_sees_the_mir_switch_over_a_foreign_enum_and_the_string_compare;dispatch_erosion_counts_const_str_compares_tables_and_literal_arguments;dispatch_erosion_counts_match_self_ladders_of_a_two_variant_enum;crates/mimir-lens/src/lenses/mir.rs:216and:228(payload parse); goldengolden_dispatch_erosion_on_holger(crates/mimir-lens/tests/golden_holger.rs:160).
4.4 cycles
- Run.
cargo lens cycles [--crate NAME…] [--include-tests] [--include-deps]. - Finds. Strongly connected components with more than one member in the
module graph and in the crate graph (
crates/mimir-lens/src/lenses/cycles.rs:1-5). - Reads. CALLS edges (deduplicated per (caller, callee) pair), the CONTAINS
parent chain (for
module_path) and DEPENDS_ON edges betweencratenodes.
Math.
F = { f : kind ∈ {fn, method}, Scope.holds(f) }
mod(x) = module_path(x) = the names of the `mod` / `crate` ancestors of x on the
CONTAINS parent chain, crate name first, joined by "::"
(the crate name alone when x has none)
calls(f) = the DISTINCT callees of f over CALLS (sorted, deduplicated: one entry per
(f, t) pair, whatever the site count); `dyn Trait` targets NOT added
Module graph M (weighted, directed, no self loops):
M[a, b] = |{ (f, t) : f ∈ F, t ∈ calls(f), Scope.holds(t), mod(f) = a, mod(t) = b, a ≠ b }|
Crate graph K:
K[p, q] = |{ (f, t) : same filter, crate(f) = p, crate(t) = q, p ≠ q }|
+ 0 for every DEPENDS_ON edge crate p → crate q with p, q in --crate scope, p ≠ q
(the edge is added with weight 0: it can close a cycle, it adds no calls)
SCC = Tarjan's strongly connected components of the unweighted graph (`order::sccs`, `crates/mimir-lens/src/lenses/order.rs`)
keep = { S ∈ SCC : |S| ≥ 2 }
calls_inside(S) = Σ_{a,b ∈ S} M[a,b] (resp. K[a,b])
heaviest(S) = the edges (a → b) with a, b ∈ S, sorted by M[a,b] descending, then (a, b)
order of cycles = |S| descending, then the sorted member list
module_cycles = |keep(M)|; modules_in_cycles = Σ_{S ∈ keep(M)} |S|; module_edges = |{(a,b) : M[a,b] > 0}|
crate_cycles = |keep(K)|; crate_edges = number of (p, q) keys of K (incl. weight-0 DEPENDS_ON keys)
Code: module and crate edges crates/mimir-lens/src/lenses/cycles.rs:66-104, the dyn rule
:74-76, weight 0 at :101, Tarjan and calls_inside :21-55, summary
:121-125, heaviest :128-132. calls(f) is deduplicated in
crates/mimir-lens/src/lenses/mod.rs:181-188.
The word "calls" in the output (calls_inside, heaviest[].calls, "1,860
calls inside") counts distinct fn → fn CALLS pairs, not call sites: the
lens reads g.calls, which is deduplicated per pair, and never the
call_count site map (crates/mimir-lens/src/lenses/cycles.rs:76-82). The module note says
"an edge = one or more direct CALLS between fns of two modules" (:168), which
is right for the edge; the weights are pair counts.
GraphBLAS steps. None. The SCCs come from Tarjan's walk in crates/mimir-lens/src/lenses/order.rs (sccs; nornir links no petgraph)
(crates/mimir-lens/src/lenses/cycles.rs:10, :36), not from GraphBLAS; the roll-up is a
BTreeMap count.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | roll CALLS up to modules | none (map count over g.calls) | M | crates/mimir-lens/src/lenses/cycles.rs:68-89 |
| 2 | roll CALLS up to crates, add DEPENDS_ON at weight 0 | none (map count) | K | crates/mimir-lens/src/lenses/cycles.rs:84-104 |
| 3 | Tarjan SCC | none (order::sccs) | SCCs of M and K | crates/mimir-lens/src/lenses/cycles.rs:36 |
| 4 | inner weight | none (edge filter + sum) | calls_inside | crates/mimir-lens/src/lenses/cycles.rs:41-48 |
Dataflow.
code_edges CALLS (dedup per pair) ──▶ (f,t), f,t ∈ Scope
code_nodes parent / CONTAINS ──▶ module_path(f), module_path(t), crate(f), crate(t)
│
├──▶ M[mod(f), mod(t)] += 1 ──▶ adjacency lists ──order::sccs──▶ |S| ≥ 2 ──▶ module_cycles, modules_in_cycles,
│ findings {kind: module, size, calls_inside,
│ members, heaviest[]}
code_edges DEPENDS_ON (crate→crate) ──+0──┐
└──▶ K[crate(f), crate(t)] += 1 ──▶ DiGraph ──tarjan_scc──▶ crate_cycles, findings {kind: crate, …}
Worked example (fixture). crates/mimir-lens/tests/graph_lenses.rs:561-567: modules m1
and m2 in crate app, m1::f1 calls m2::f2 and m2::f2 calls m1::f1.
mod(f1) = app::m1, mod(f2) = app::m2, so M[app::m1, app::m2] = 1 and
M[app::m2, app::m1] = 1. Tarjan finds S = {app::m1, app::m2}, |S| = 2,
calls_inside = 1 + 1 = 2. Every other call of the fixture stays inside the
module app, so no other module edge exists. No call crosses a crate inside
the scope (the fmt fns call only reference stubs, which are not
defined_here), so crate_cycles = 0. The test asserts module_cycles = 1,
members ["app::m1", "app::m2"], crate_cycles = 0
(crates/mimir-lens/tests/graph_lenses.rs:867-879).
Worked example (Holger). Golden: module_cycles = 9,
modules_in_cycles = 115, module_edges = 1,540, crate_cycles = 1. The plan
(§E.2) splits it: the largest SCC holds 90 modules of holger-server-lib with
1,860 fn pairs inside; the other 8 cycles hold 115 − 90 = 25 modules, for
example holger_index_rows with its 6 format modules (7) and audit with
audit::{chain, arrow_ipc, trail} (4). Its heaviest edges are
pull_intake::ecosystems → pull_intake ×56 and pull_intake → promote::intake ×43 (fn pairs). The crate cycle is holger-plugin-host ↔ holger-server-lib (13 fn pairs).
- Output. Summary
module_cycles, modules_in_cycles, module_edges, crate_cycles, crate_edges(crates/mimir-lens/src/lenses/cycles.rs:121-125). Findings{kind: module, size, calls_inside, members, heaviest[{from, to, calls}]}(12 heaviest in JSON, 6 in text) and{kind: crate, size, calls_inside, members}(:150-165). - Limits. It cannot say which edge is wrong; that needs the intended
layering. Calls through
dyn Traitare left out on purpose: a trait object is the seam that breaks a cycle (:74-76). Overloaded operators have no CALLS edge, and a call through a closure or fn pointer ends at theFn*trait method (section 3.2), so neither links two modules. With no CALLS edges the lens says so (:59-64). - Tests.
crates/mimir-lens/tests/graph_lenses.rs:867cycles_finds_the_two_modules_that_call_each_other; goldengolden_size_cycles_single_impl_error_shape_cfg_sprawl_on_holger(crates/mimir-lens/tests/golden_holger.rs:183).
4.5 single-impl
- Run.
cargo lens single-impl [--crate NAME…] [--include-tests] [--include-deps]. - Finds. Traits with zero or one production impl (
crates/mimir-lens/src/lenses/single_impl.rs:1-5). - Reads.
traitnodes, IMPLEMENTS (impl → trait), FOR_TYPE (impl → type), and the trait's CONTAINS children (methods).
Math.
T = { t : kind = trait, Scope.holds(t) }
I(t) = { i : IMPLEMENTS i → t } (every impl in the dump, any crate, derive-written included)
prod(t) = { i ∈ I(t) : ¬test_code(i) } test(t) = { i ∈ I(t) : test_code(i) }
blanket(t) = |{ i ∈ I(t) : FOR_TYPE(i) is a node of kind `generic` }|
methods(t) = |{ c ∈ children(t) : kind(c) ∈ {fn, method} }|
traits = |T|
zero_impl = |{ t ∈ T : |prod(t)| = 0 }|
single_impl = |{ t ∈ T : |prod(t)| = 1 }|
impls = Σ_{t ∈ T} ( |prod(t)| + |test(t)| )
findings = { t : |prod(t)| ≤ 1 }, sorted by |prod(t)| ascending (ZERO first), then label
Code: crates/mimir-lens/src/lenses/single_impl.rs:34-70 (rows), :43-55 (test split and
blanket), :71-81 (counts), :82-88 (order). test_code is
crates/mimir-lens/src/lenses/mod.rs:378-402 (an is_test node, a `test= `` id, a file
under tests/, or a cfg(test) ancestor).
The union of builds (round 4, Gunnar re-compare row 11). --also-dump DIR
(repeatable; SingleImplArgs::also) adds the graphs of other builds:
I∪(t) = prod_g(t) ∪ ⋃_k prod_k(label(t)) (impls matched by label)
single_impl = |{ t : |I∪(t)| = 1 }|, zero_impl = |{ t : |I∪(t)| = 0 }|
BUILD-SPLIT = { t : |prod_g(t)| ≤ 1 ∧ |I∪(t)| ≥ 2 }
On Gunnar, default dump with --also-dump the metered dump (golden
golden_gunnar_single_impl_and_h1_on_the_union_with_the_delivery_build):
single-impl: 27 traits with exactly one production impl and 0 with none, of 58 traits (289 impls seen); across 2 builds, 5 traits have ≤ 1 impl here and ≥ 2 in the union (BUILD-SPLIT)
BUILD-SPLIT gunnar-repair exec/Estate# (crates/gunnar-repair/src/exec.rs:29, 20 methods) — 1 impl in this build, 2 across 2 builds [NoEstate, ServerEstate]
BUILD-SPLIT gunnar-repair exec/GrowSet# (crates/gunnar-repair/src/exec.rs:229, 1 methods) — 1 impl in this build, 2 across 2 builds [AskOn, NoAsk]
BUILD-SPLIT gunnar-server grow/go_ahead/GoAheadPolicy# (crates/gunnar-server/src/grow/go_ahead.rs:113, 2 methods) — 1 impl in this build, 2 across 2 builds [MonetizePolicy, OperatorPolicy]
BUILD-SPLIT gunnar-server grow/world/World# (crates/gunnar-server/src/grow/world.rs:111, 23 methods) — 1 impl in this build, 2 across 2 builds [BoxWorld, NoEngine]
BUILD-SPLIT gunnar-ssh plan/PlanGate# (crates/gunnar-ssh/src/plan.rs:111, 6 methods) — 1 impl in this build, 2 across 2 builds [DiskHeadroomGate, EntitlementSlot]
Alone, the default dump said 32 SINGLE; 5 of them are single only because the
metered impls (MonetizePolicy, BoxWorld, EntitlementSlot, …) were compiled
out. Bench: 5.9 s / 4.4 GB for both graphs (the second one loads uncached).
Note: the scope rule (no derive-written code) applies to the trait, not to the
impls. A derive-written impl of a workspace trait counts as a production impl,
as the lens note says ("a derive-generated impl counts too", :128). The
module doc says "blanket impls included and marked" (:2); blanket is a
count of such impls per trait, not a flag.
GraphBLAS steps. None. One pass over the impls_of_trait hash map built at
load time (crates/mimir-lens/src/lenses/mod.rs:169-172).
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | IMPLEMENTS reversed | none (hash map at load) | I(t) | crates/mimir-lens/src/lenses/mod.rs:169-172 |
| 2 | split by test_code, FOR_TYPE → generic | none (lookup) | prod, test, blanket | crates/mimir-lens/src/lenses/single_impl.rs:42-56 |
| 3 | count and sort | none | summary, findings | crates/mimir-lens/src/lenses/single_impl.rs:71-88 |
Dataflow.
code_nodes kind=trait (Scope) ──────────────────────────────┐
code_edges IMPLEMENTS impl→trait ──▶ impls_of_trait[t] ─────┼──▶ prod(t) / test(t) ──▶ zero_impl, single_impl, impls
code_edges FOR_TYPE impl→type (kind=generic?) ──▶ blanket ──┘ └──▶ findings {trait, loc, methods,
code_edges CONTAINS trait→method ──▶ methods ───────────────────────────────────────▶ prod_impls, test_impls, blanket}
Worked example (fixture). crates/mimir-lens/tests/graph_lenses.rs:293-302: trait
Store with one impl impl Store for Disk, and trait Lonely with none. The
trait core::error::Error of the fixture is a reference stub
(defined_here = false), so it is not in T. T = {Lonely, Store},
|prod(Store)| = 1, |prod(Lonely)| = 0: single_impl = 1, zero_impl = 1.
The text lists ZERO app Lonely# before SINGLE app Store#
(crates/mimir-lens/tests/graph_lenses.rs:882-891).
Worked example (Holger). Golden: traits = 43, single_impl = 19,
zero_impl = 2, impls = 176. So 43 − 19 − 2 = 22 traits have two or more
production impls, and 19 + 2 = 21 of 43 (49 %) are findings. The plan
(§E.2) names the two ZERO traits: apps::hub::Notify and plugin-abi
v2::MeterHost (its 2 impls are test code), and SINGLE ones such as
PackageHandler, JobRunner, UpstreamConnector and ImportAudit (+6 test
impls).
- Output. Summary
traits, single_impl, zero_impl, impls(crates/mimir-lens/src/lenses/single_impl.rs:78-81). Findings{trait, loc, methods, prod_impls, test_impls, blanket}(:118-125), tagged ZERO or SINGLE. - Limits. With no IMPLEMENTS edges the lens answers "not answerable"
instead of calling every trait unimplemented (
:18-25). It does not countdyn Traitcall sites per trait. - Tests.
crates/mimir-lens/tests/graph_lenses.rs:882single_impl_counts_implements_edges_per_trait; golden atcrates/mimir-lens/tests/golden_holger.rs:183.
4.6 size
- Run.
cargo lens size [--fn-lines 150] [--fn-lines-big 300] [--file-lines 5000] [--src <checkout>] [--crate …] [--include-tests] [--include-deps](dw:src/cli.rs:293-309). - Finds. Long fns and big files.
- Reads.
start_line,end_lineandfileof every node in scope. With--src, the files themselves.
Math.
N = { n : Scope.holds(n) } (every node kind, not only fns)
len(f) = end_line(f) − start_line(f) + 1 for f ∈ N, kind ∈ {fn, method},
when both lines are set and end ≥ start; else f counts in fns_without_lines
(a trait method with end_line = start_line, a declaration without a body, is skipped)
fns = |{ f with len(f) }|
fns_long = |{ f : len(f) > fn_lines }| fn_lines = 150 (strictly greater)
fns_very_long = |{ f : len(f) > fn_lines_big }| fn_lines_big = 300
bound(file) = max { end_line(n) : n ∈ N, file(n) = file } (lower bound)
lines(file) = number of lines of src/file when --src has the file (exact)
= bound(file) otherwise (marked "+")
files = |{ file : some n ∈ N lies in it }|
files_big = |{ file : lines(file) > file_lines }| file_lines = 5000
lines = Σ_file lines(file)
lines_exact = (every file was counted exactly) ∧ files > 0
order: fns by len descending, then label; files by lines descending, then path
Code: crates/mimir-lens/src/lenses/size.rs:48-66 (fns and file bounds; the skip rule at
:59, len at :63), :74-75 (bands), :80-91 (files),
:101-109 (summary).
GraphBLAS steps. None. One pass over the nodes.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | per-node scan | none | len(f), bound(file) | crates/mimir-lens/src/lenses/size.rs:48-66 |
| 2 | file line count | none (file read, --src) | exact lines(file) | crates/mimir-lens/src/lenses/size.rs:37-40, :80-86 |
Dataflow.
code_nodes (Scope) start_line, end_line ──▶ len(f) = end − start + 1 ──▶ fns, fns_long (>150), fns_very_long (>300),
findings {kind: fn, lines, fn, loc, module, crate}
code_nodes (Scope) file, end_line ──▶ bound(file) = max end_line ──┐
--src <checkout>/file ──▶ lines().count() ─────────────────────────┴▶ lines(file) ──▶ files, files_big (>5000), lines,
lines_exact, findings {kind: file, lines, exact, file}
Worked example (fixture). crates/mimir-lens/tests/graph_lenses.rs:344-352: big() in
src/big.rs spans lines 100–400, so len = 400 − 100 + 1 = 301. 301 > 150
and 301 > 300: fns_long = 1, fns_very_long = 1. The struct Tail in the
same file ends at line 6000, so bound(src/big.rs) = 6000 > 5000:
files_big = 1, printed FILE 6000+ src/big.rs (a bound, no --src). The
dependency crate dep_x (origin registry) has huge() with
900 − 1 + 1 = 900 lines; it counts only with --include-deps, where
fns_long = 2 (crates/mimir-lens/tests/graph_lenses.rs:739-765).
Worked example (Holger). Golden with --src: fns = 6,668,
fns_long = 66, fns_very_long = 19, files = 370, files_big = 5,
lines = 273,777, lines_exact = true. So 66 / 6,668 ≈ 1.0 % of the fns
are over 150 lines, and the mean file holds 273,777 / 370 ≈ 740 lines. The
plan (§E.2) names the five big files (lib.rs 9,847; exposed/http.rs 8,781;
promote/intake.rs 7,765; grpc/services.rs 7,514; flow.rs 5,359) and the
longest fn, handle at http.rs:883, with 4,071 lines.
- Output. Summary
fns, fns_without_lines, fns_long, fns_very_long, files, files_big, lines, lines_exact, files_counted_exactly(crates/mimir-lens/src/lenses/size.rs:101-109). Findings{kind: file, lines, exact, file}and{kind: fn, lines, fn, loc, module, crate}(:120-138). - Limits. The dump has no file line counts; without
--srcevery file count is a lower bound and the lens says "needs file line counts" (:110-119).filescounts only files that hold an in-scope node. Attributes are not in the span. - Tests.
crates/mimir-lens/tests/graph_lenses.rs:739size_reads_fn_lengths_from_the_graph_and_names_the_file_count_as_a_bound; golden atcrates/mimir-lens/tests/golden_holger.rs:183.
4.7 error-shape
- Run.
cargo lens error-shape [--src <checkout>] [--crate …] [--include-tests] [--include-deps]. - Finds. Per crate: untyped error creation (anyhow), typed error types,
stringly results,
.context()sites, and, with--src, lines that branch on error text (crates/mimir-lens/src/lenses/error_shape.rs:1-13). - Reads. CALLS with site counts, IMPLEMENTS + FOR_TYPE of the trait
core::error::Error/std::error::Error, and thetyof each fn'sretchild.
Math. For a crate k:
F_k = { f : kind ∈ {fn, method}, Scope.holds(f), crate(f) = k }
sites(f, c) = the CALLS site count of f → c (the edge's `count`, at least 1)
anyhow_kind(c) = "create" if crate(c) = anyhow and short(c) contains "format_err"
or ends with "]msg()." or ends with "]new()."
= "context" if crate(c) = anyhow and short(c) contains "Context#" or "context"
anyhow_sites_k = Σ_{f ∈ F_k} Σ_{c ∈ calls(f), kind(c)=create} sites(f, c)
anyhow_fns_k = |{ f ∈ F_k : some create callee }|
context_sites_k = Σ_{f ∈ F_k} Σ_{c : kind(c)=context} sites(f, c)
typed_k = |{ ty : IMPLEMENTS i → t, t = trait named Error in crate core or std,
FOR_TYPE i → ty, Scope.holds(ty), crate(ty) = k }| (distinct types)
r(f) = ty of f's `ret` child with spaces removed
string_results_k = |{ f ∈ F_k : r(f) starts "Result<" and ends ",String>" or ",&str>" }|
boxed_results_k = |{ f ∈ F_k : r(f) starts "Result<", contains ",Box<dyn" and "Error",
not already a string result }|
text_branches_k = lines (not starting "//") of the crate's files that contain
".to_string().contains(", "{e}\").contains(", "{e:#}\").contains(" or
"err.to_string().contains(" (--src only; a file goes to the crate
of the first fn found in it)
untyped_vs_typed_types_heuristic_k = anyhow_sites_k / (anyhow_sites_k + typed_k) (0 when both are 0)
summary: anyhow_sites = Σ_k anyhow_sites_k, typed_error_types = Σ_k typed_k,
string_results = Σ_k (string_results_k + boxed_results_k), context_sites = Σ_k context_sites_k
rows: crates with F_k ≠ ∅ or typed_k > 0, sorted by anyhow_sites_k descending, then crate name
Code: crates/mimir-lens/src/lenses/error_shape.rs:44-58 (anyhow_kind), :64-86 (typed),
:88-118 (sites and return types), :121-168 (text fallback), :178-199
(summary), :200-219 (rows and share).
The ratio divides a count of call sites by a count of sites plus a count
of types. The units differ on purpose: the graph has no "typed error site"
(an Err(MyErr::X) is a variant construction, not a call). So it is a
heuristic, not the fraction of errors that are untyped, and the field is named
untyped_vs_typed_types_heuristic to say so (error_shape.rs:15-19, :236;
renamed from untyped_share in dwarves c099cc1). It exists per crate only;
there is no workspace-level value in the summary. The verdict's err term is
the same ratio.
The module doc lists stringly results as Result<_, String> or
Result<_, Box<dyn Error…>> (:9-10); the code also counts Result<_, &str>
(:112).
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | IMPLEMENTS of Error → FOR_TYPE | none (hash lookup) | typed types per crate | crates/mimir-lens/src/lenses/error_shape.rs:64-86 |
| 2 | CALLS with site counts into anyhow | none (label match, sum) | create and context sites | crates/mimir-lens/src/lenses/error_shape.rs:94-104 |
| 3 | ret type test | none (string test) | stringly results | crates/mimir-lens/src/lenses/error_shape.rs:105-117 |
| 4 | text grep | none (file read, --src) | text branches | crates/mimir-lens/src/lenses/error_shape.rs:121-168 |
Dataflow.
code_edges CALLS (count) f→anyhow::* ──anyhow_kind──▶ create / context ──▶ anyhow_sites_k, anyhow_fns_k, context_sites_k ──┐
code_edges IMPLEMENTS impl→core/std Error + FOR_TYPE impl→ty ──▶ typed_k ─────────────────────────────────────────────────┼▶ heuristic_k
code_nodes kind=ret (ty) under f ──▶ Result<_,String|&str> / Result<_,Box<dyn …Error>> ──▶ string_results_k, boxed_results_k
--src files ──grep──▶ text_branches_k └▶ one finding per crate
Worked example (fixture). crates/mimir-lens/tests/graph_lenses.rs:568-596: boom() has
one CALLS edge (count 1) to the stub anyhow __private/format_err()., which
is "create": anyhow_sites = 1. impl Error for MyErr points at the stub
trait core error/Error#, and MyErr is in scope: typed_error_types = 1.
stringly() returns Result<(), String>: string_results = 1. For crate
app, untyped_vs_typed_types_heuristic = 1 / (1 + 1) = 0.5. Without --src the lens says
"needs callers that branch on error text" (crates/mimir-lens/tests/graph_lenses.rs:894-907).
Worked example (Holger). Golden: anyhow_sites = 936,
typed_error_types = 37, string_results = 378, context_sites = 279.
Computed from these, over the whole workspace:
936 / (936 + 37) = 936 / 973 ≈ 0.962. The plan (§E.2) gives the per-crate
rows behind it: holger-server-lib 509 sites and 21 typed, so
untyped_vs_typed_types_heuristic = 509 / (509 + 21) = 509 / 530 ≈ 0.960; holger
(server/cli) 132 sites and 0 typed, so 132 / 132 = 1.0. The 936 sites sit in
401 fns, and 76 lines branch on error text (with --src).
- Output. Summary
anyhow_sites, typed_error_types, string_results, context_sites(crates/mimir-lens/src/lenses/error_shape.rs:193-199). One finding per crate{crate, fns, anyhow_sites, anyhow_fns, typed, string_results, boxed_results, context_sites, text_branches, untyped_vs_typed_types_heuristic}(:220-225). - Limits.
format!strings insideErr(…)without anyhow have no constructor call to count. A dump without theanyhowcrate (as reference stubs) counts no untyped site and says so (:175-177). The "context" match is any anyhow callee whose short id containscontext(:53). - Tests.
crates/mimir-lens/tests/graph_lenses.rs:894error_shape_counts_anyhow_sites_typed_errors_and_stringly_results; golden atcrates/mimir-lens/tests/golden_holger.rs:183.
4.8 cfg-sprawl
- Run.
cargo lens cfg-sprawl [--crate …] [--include-tests] [--include-deps]. - Finds. Per crate: declared features, item gates, gates inside fn bodies,
features that gate nothing, and
cfg_gated_outstubs (crates/mimir-lens/src/lenses/cfg_sprawl.rs:1-6). - Reads.
featurenodes,cfg_gated_outnodes, GATED_BY edges (item → feature node) with theircountandpartialcolumns.
Math. For a crate k:
declared_k = { name(n) : kind(n) = feature, crate(n) = k, crate in --crate scope }
gated_out_k = |{ n : kind(n) = cfg_gated_out, crate(n) = k, crate in --crate scope }|
G = { e : GATED_BY e = (x → φ), Scope.holds(x) } (x = the gated item)
sites(e) = max(count(e), 1)
for e ∈ G with crate(x) = k:
referenced_k ∪= { name(φ) }
partial(e) = true → body_gates_k += sites(e); body_gate_fns_k ∪= { enclosing fn of x }
otherwise → item_gates_k += sites(e)
per file(x): (item, body) site counts
never_referenced_k = declared_k \ referenced_k (set difference by feature name)
features_declared = Σ_k |declared_k|
item_gates, body_gates = Σ_k item_gates_k, Σ_k body_gates_k
features_never_referenced = Σ_k |never_referenced_k|
gated_out_stubs = Σ_k gated_out_k
rows: crates with declared_k ≠ ∅ or a gate, sorted by item + body gates descending, then name;
per crate the files sorted by item + body descending (15 in JSON, 5 in text)
undeclared note: a gate whose feature node is not defined_here but in the item's own crate
Code: crates/mimir-lens/src/lenses/cfg_sprawl.rs:39-58 (feature and stub nodes),
:60-84 (gates; item vs body at :73-80), :89-92 (never referenced),
:98-105 (summary), :106-156 (rows), :157-167 (undeclared note).
"Item gate" = a #[cfg(feature = …)] on the item itself. "Body gate" = a
GATED_BY edge with partial = true: a #[cfg] or cfg! inside the item's
body (the "ifdef inside a function" class).
Where the module doc and the code differ: the doc says gates on a feature no
crate declares "are listed by name" (:5-6). The code lists them only in a
note, and only when the feature node is a stub of the gated item's own crate
(:81-83, :157-167). Feature nodes and stubs are filtered by --crate
only (:41), not by origin or the test rule; the gates are filtered by the
full scope (:64). On a full dump without --crate, dependency crates'
feature nodes therefore count in features_declared.
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | node scan | none | declared features, gated-out stubs | crates/mimir-lens/src/lenses/cfg_sprawl.rs:39-58 |
| 2 | GATED_BY scan, split by partial | none (sum of count) | item and body gates | crates/mimir-lens/src/lenses/cfg_sprawl.rs:60-84 |
| 3 | set difference | none (BTreeSet::difference) | never-referenced features | crates/mimir-lens/src/lenses/cfg_sprawl.rs:89-92 |
Dataflow.
code_nodes kind=feature ──▶ declared_k ─────────────────────────────┐
code_edges GATED_BY item→feature (count, partial), item ∈ Scope ──▶ referenced_k ──▶ never_referenced_k = declared_k \ referenced_k
├── partial=true ──▶ body_gates_k, body_gate_fns_k
└── otherwise ──▶ item_gates_k ──▶ per-file (item, body)
code_nodes kind=cfg_gated_out ──▶ gated_out_k ───────────────────────▶ summary + one finding per crate
Worked example (fixture). crates/mimir-lens/tests/graph_lenses.rs:597-609: crate app
declares features x, y, z. gated() has a GATED_BY edge to x with
partial = false (an item gate); partly() has one to y with
partial = true (a body gate). So declared = {x, y, z},
referenced = {x, y}, never_referenced = {x, y, z} \ {x, y} = {z}:
features_declared = 3, item_gates = 1, body_gates = 1,
features_never_referenced = 1, text "never gating: z"
(crates/mimir-lens/tests/graph_lenses.rs:910-920).
Worked example (Holger). Golden: features_declared = 155,
item_gates = 830, body_gates = 1,099, features_never_referenced = 29,
gated_out_stubs = 435. So 830 + 1,099 = 1,929 gate sites, and
1,099 / 1,929 ≈ 57 % of them sit inside bodies. The plan (§E.2): 155
features in 32 crates, the body gates in 259 fns; holger-server-lib alone has
61 features, 640 + 756 = 1,396 gate sites in 201 fns, and 345 gated-out
stubs. Its never-gating features include default and umbrellas such as
fmt-all and supply-chain: the dump writes a feature node for default
too, so default is counted as declared here (the zombie fold, section 6.2,
leaves default out of its declared set).
- Output. Summary
features_declared, item_gates, body_gates, features_never_referenced, gated_out_stubs(crates/mimir-lens/src/lenses/cfg_sprawl.rs:98-105). Findings{crate, declared, referenced, never_referenced, item_gates, body_gates, body_gate_fns, gated_out, files[{file, item, body}]}(:149-154). - Limits. Whether a feature is ever BUILT is
dwarves mimir zombie's question (it reads the manifests;:168). With neither GATED_BY edges nor feature nodes the lens says "needs" (:32-37). - Tests.
crates/mimir-lens/tests/graph_lenses.rs:910cfg_sprawl_tells_item_gates_from_gates_inside_bodies_and_names_unused_features; golden atcrates/mimir-lens/tests/golden_holger.rs:183.
4.9 taint
The question. Does a value that came from outside (a request parameter, a
request body, stdin, the environment, a socket) reach the argument of a
dangerous call (spawn a process, touch the file system, join a path, open a
connection) without passing a sanitizer on the way? This is the taint-style
traversal of a code property graph (Yamaguchi, Golde, Arp and Rieck, Modeling
and Discovering Vulnerabilities with Code Property Graphs, IEEE S&P 2014:
source ⇝ sink along data-dependence edges with no sanitizer between), set up
in the shape of CodeQL's global taint tracking (isSource / isSink /
isSanitizer). It is one boolean reach on GraphBLAS over a value-flow graph
whose nodes are (fn, MIR local) pairs. It is context-insensitive: it does not
use the realizable-path (IFDS) restriction of Reps, Horwitz and Sagiv (POPL
1995). The code is crates/mimir-lens/src/lenses/taint.rs; the plan row is F-1 (plan §F).
Run
cargo lens taint [--source PAT…] [--source-api PAT…] [--sink PAT[:FAMILY]…] \
[--sanitizer PAT…] [--no-default-sinks] \
[--mimir-dump DIR] [--crate NAME…] [--include-tests] [--include-deps] [--json] [--limit N]
# Holger, plan §F.2 (5 source patterns; the exact strings are not recorded):
cargo lens taint --crate holger-server-lib \
--source 'exposed/http/handle()' --source '<gRPC service pattern>' --source 'JobRunner#run()' …
The flags are in dw:src/cli.rs:392-415, the shared dump flags in
dw:src/cli.rs:553-583, the dispatch in dw:src/cli.rs:753-772. A pattern is
a substring of the label <crate> <short moniker>, as in gate-bypass.
| flag | effect | code |
|---|---|---|
--source PAT | fns whose parameters are seeds; a pattern that matches no fn is an error | taint.rs:244-252 |
--source-api PAT | callees whose result is a seed. Replaces the default table SOURCE_APIS (it does not add to it) | :253-257, table :71-83 |
--sink PAT[:FAMILY] | adds a sink to the default table; the family is the text after the last :, default custom | :258-269 |
--no-default-sinks | drops the default table SINKS; then --sink must give one, or the run is an error | :258-272, table :86-112 |
--sanitizer PAT | adds a sanitizer to the default list SANITIZERS | :273-274, list :115-124 |
With no --source and the default API table the lens still runs, seeded by
API results only. The :FAMILY split uses the last : of the argument, so a
pattern that itself contains : must carry an explicit family.
From the compiler to the graph
-
Fns and params.
cargo mimir-dump(the nornirmimir-driver) writes the fn and method nodes (FN_KINDS = fn, method,crates/mimir-lens/src/lenses/mod.rs:81) and theirparamchildren (CONTAINS) in declaration order. The lens reads the children'snameto skipself(taint.rs:448-463). -
MIR blocks. For every fn of a workspace crate the driver writes one
mir_blocknode per non-cleanup block of rustc's built MIR (see 4.1, From the compiler to the graph). Closure andasyncbodies are separate MIR bodies whose blocks are written under the root fn (bbcK.N). The payload (crates/mimir-schema/src/mir.rs:25-40, readercrates/mimir-lens/src/lenses/mir.rs:1-11,:69-100) is:<fp 16 hex> ␞ <terminator>@l:c ␞ <stmt>@l:c ␟ … ␞ <local>:<ty> ␟ … ␞ <a>><b>,… call terminator: call <callee moniker>⟨<args>⟩-><dest local>@l:c operand: _3 (copy) | ^_3 (move) | k:<type> (constant); places _1.0, _3*, &_2 -
The flow section
a>b. The driver records, for every statement and for the terminator, the locals it reads and the local it writes, and emits one def-use pair(read, written)per read (Ctx::flow,crates/mimir-driver/src/walk/mir.rs:563-576). For aCallterminator the reads are the argument operands and the write is the destination (walk/mir.rs:476-486, flow call:535). So the flow section of a call block already holdsa>rfor every argument localaand the resultr. The set is per block, with no kill analysis (mir.rs:38-39in the schema). A write through a pointer ((*_3).0 = …) is a read of_3, not a def (walk/mir.rs:216-228). -
MIR local numbering.
_0is the return place,_1 … _kare the parameters in order (rustc's convention), the rest are temporaries.
Math
The fn set and the nodes (taint.rs:279-321).
Src = fns_matching(--source) (any crate, any scope)
Φ = { f : is_fn(f) ∧ Scope::holds(f) } ∪ Src (:280-286)
Φ_M = { f ∈ Φ : f has at least one mir_block with a payload } (:293-297)
maxl(f) = max { ℓ : ℓ in a block's locals table, or "_ℓ" occurs in a statement or terminator of f } (:298-310)
nl(f) = maxl(f) + 1
base(f) = Σ { nl(f′) : f′ ∈ Φ_M, f′ before f in node-id order } (:313-315)
V = { (f, ℓ) : f ∈ Φ_M, 0 ≤ ℓ < nl(f) } id(f, ℓ) = base(f) + ℓ N = |V| = Σ nl(f)
node(f, ℓ) = id(f, ℓ) if f ∈ Φ_M and ℓ < nl(f), else undefined (an edge with an undefined end is dropped)
locals_in(s) is every _n in s whose _ is not preceded by a letter or
digit (:164-185). arg_local(x) strips leading ^ & * and spaces and reads
the digits after _, so ^_3 → 3, _1.0 → 1, &_2 → 2, k:&str →
none (:189-197).
All blocks of f, including the nested bodies bbcK.*, share the one local
range of f. Local _1 of a closure or async body (in rustc the captured
environment or the coroutine) is therefore the same node as the outer fn's
first parameter. This merges values across bodies of one fn: it can add flows
that do not exist, and it is how a seeded parameter reaches an async body at
all.
The edges of the boolean pattern matrix A (N × N), all within the call
loop :338-437. For a block of f, let its call terminator (kind call or
tailcall) have callee label λ (= <crate> <short> of the callee moniker,
:142-148), argument locals a₁ … a_k (the parsed locals of the argument
list in order, constants dropped, :382-387) and result r (->_r,
:211-220). san(x) ⇔ ∃ p ∈ SANITIZERS ∪ --sanitizer : p ⊂ x (:322).
R1 block flows (f,a) → (f,b) for every a>b in the 5th section of the payload (:352-359)
R2 statements (f,m) → (f,n) for every statement "_n… = rhs" (lhs via arg_local,
so "_n.0 = …" and "_n* = …" write _n), every m ∈ locals_in(rhs), m ≠ n
(a statement whose "=" is followed by "=" is skipped) (:199-208, :363-375)
R3 pass-through (f,a_j) → (f,r) ∀ j, when ¬san(λ) and r is defined (:405-415)
R4 enter callee (f,a_i) → (t, i) i = 1 … k, and (t,0) → (f,r)
for every t ∈ {g} ∪ dyn_targets(g) where label(g) = λ, g ∈ Φ,
t ∈ Φ_M, ¬san(λ) and ¬san(label(t)) (:416-435)
A = pattern(R1 ∪ R2 ∪ R3 ∪ R4) (duplicates collapse, :466-467)
dyn_targets(g) is the dyn expansion of 4.1: the same-named method of every
impl of g's trait (crates/mimir-lens/src/lenses/mod.rs:179-209). A callee outside Φ
(every std or other dependency fn on a workspace-only scope) is never
entered; only R1 and R3 carry a value across it.
Two code facts change the reading of R3 and R4:
- R4 maps the i-th argument local, not the i-th argument.
icounts the argument list after constants are dropped (enumerateoverarg_nodes,:413-417). Forg(k:&str, _3)the local_3goes to(g, 1), not to(g, 2). The module doc says "(f, a_i) → (g, i+1)(MIR's_1..are the params)" (:39-40), which holds only when no argument beforea_iis a constant. - R1 already holds R3. The driver writes the def-use pairs of the call
terminator into the flow section (see above), so every call block carries
a_j > r. R1 adds those edges, and R3 duplicates them for an ordinary call. For a sanitizer call, R1 drops every flow into the call's result local before the block flows become edges (sanitized_dest,taint.rs:340-356).
Seeds (a boolean vector s of size N).
S_param = { (f, i+1) : f ∈ Src ∩ Φ_M, p_i the i-th param child of f (0-based), name(p_i) ≠ "self", i+1 < nl(f) } (:441-465)
S_api = { (f, r) : a call in a block of f whose label λ contains a source-API pattern, r defined } (:400-404)
s = 1 on S_param ∪ S_api
self keeps its index: a method m(&self, req) seeds _2. A source fn in
Src ∖ Φ_M goes to sources_without_mir and a needs line; a source fn with
no param children is counted in source_fns_without_params (:453-456).
The source-API test runs before the sanitizer test, so a sanitizer call whose
label also matches a source API still seeds its result.
The step (:491-503).
T = reach(s, A) = the least fixpoint T = s ∨ Aᵀ T (no hop limit; s ⊆ T)
Sinks (:258-269, :323-328, :397-399). The table is the ordered list
SINKS ++ --sink (or --sink alone). The family of a call is that of the
first row whose pattern is a substring of λ. Every such call is a sink
call, whatever its sanitizer status (the sink test runs before the sanitizer
test).
TAINTED(call in f with argument locals a₁ … a_k) ⇔ ∃ j : node(f, a_j) defined ∧ T[(f, a_j)] (:536-539)
Every argument counts, including a receiver (Path::join(base, seg) is
tainted when base is).
The default sink table (:86-112):
| family | callee patterns |
|---|---|
| exec | std process/impl#[Command]new(, …arg, tokio process/impl#[Command]new(, …arg |
| fs | std fs/write(, create_dir, remove_, rename(, copy(, hard_link(, read(, read_to_string(, read_dir(, impl#[File]create(, impl#[File]open(, impl#[OpenOptions]open(, std os/unix/fs/symlink(, tokio fs/ (every tokio fs fn) |
| path | std path/impl#[Path]join(, std path/impl#[PathBuf]push( |
| net | std net/impl#[TcpStream]connect(, tokio net/impl#[TcpStream]connect(, reqwest , ureq , hyper client/ |
The module doc lists the fs family as writes, removes, renames, symlinks and
File::create/open (:17-21). The table also holds the reads fs::read,
read_to_string, read_dir and every tokio fs/ fn, so fs counts reads of
a tainted path too.
The default source APIs (:71-83): tonic request/impl#[Request,
hyper body/, http_body_util , http request/impl#[Request,
axum extract/, std env/args, std env/var, std io/stdin,
tokio io/stdin, std net/impl#[TcpStream]read,
tokio net/impl#[TcpStream]read.
The default sanitizers (:115-124): sanitize, normalize, canonical,
validat, verif, percent_decod, impl#[Path]file_name(,
impl#[Path]components(, all substrings of the callee label.
How a sanitizer cuts. A call with san(λ) gets no R3 edge (argument →
result) and no R4 edges (arguments → the callee's params and the callee's _0
→ the result), and is counted in sanitizer_calls (:405-408). A dyn target
whose own label is a sanitizer is not entered (:423). The cut is on the
call, in both directions across the callee; the sanitizer fn's own body edges
stay in A.
So the result of sanitize(x) is clean even though the driver writes the
call's arg > dest pair into the sanitizer's call block. The fixture now
carries that pair (1>2 on the clean() call block,
crates/mimir-lens/tests/graph_lenses_study.rs), and the sanitized value does not reach
fs::write. (Before dwarves ba122c9 R1 took that pair before the sanitizer
test, so on a real dump the result of sanitize(x) stayed tainted whenever
x was.)
The path (:504-563). A plain BFS (not GraphBLAS) over the edges of A
with both ends in T, started from every seed at once in ascending node id,
gives each tainted node one parent. For a tainted sink call the walk starts at
its first tainted argument and follows parents to a seed (at most 10,000
hops). The reported source is that seed, so a sink reached from several sources
names only the one nearest in hops (ties go to the lower node id and the
sorted edge order). The path is printed as fn steps: consecutive nodes of the
same fn collapse to the first one (:602-613).
Ranking. Rows are sorted by (family, fn label, callee label, loc)
(:564-571).
Context-insensitivity. R4 adds (t,0) → (f,r) at every call site of t.
A value that enters t from one call site and reaches _0 therefore taints
the result at every other call site too. The IFDS formulation (Reps, Horwitz
and Sagiv) would accept only paths whose call and return edges match; the
module doc names the Dyck-matched walk of gate-bypass (4.1) as the follow-up
(:26-28). It is not built.
GraphBLAS steps
| step | operation | semiring / op as the code names it | computes | file:line |
|---|---|---|---|---|
| 1 | parse payloads, build R1–R4 and the seeds | none: string parsing and hash lookups | the edge list and s | crates/mimir-lens/src/lenses/taint.rs:293-465 |
| 2 | A = pattern(edges) | GxB_Matrix_build_Scalar, iso true | the N × N value-flow matrix | crates/mimir-lens/src/lenses/taint.rs:493, crates/mimir-lens/src/lenses/grb.rs:21-30, crates/mimir-graph/src/csr.rs:208-221 |
| 3 | T = reach(s, A) | ops::reach: q⟨¬v, struct, replace⟩ = q · A by GrB_vxm with GxB_ANY_PAIR_BOOL and GrB_DESC_RSC; v⟨q, struct⟩ = true by GrB_Vector_assign_BOOL with GrB_DESC_S; until q is empty | every value node a seed reaches | crates/mimir-lens/src/lenses/taint.rs:495-499; crates/mimir-graph/src/ops.rs:49-98, :103-110 |
| 4 | sink test, parents, paths | none: a scan of the sink calls and a plain BFS (VecDeque) inside T | TAINT rows | crates/mimir-lens/src/lenses/taint.rs:504-563 |
ANY.PAIR on bool is the structural OR-AND (LOR.LAND) product: a node
enters the frontier when any frontier node has an edge to it.
Dataflow
code_nodes: fn/method (+ param children), mir_block (ty payload) code_edges: CONTAINS, IMPLEMENTS, CALLS
│ │
└────────────────────► Graph::load (mod.rs:115) ◄─────────────────────┘
│ mir_blocks(f), dyn_targets, children
--source ──► Src │
▼
Φ_M, nl(f), base(f) ──► V = (fn, local) nodes, N = Σ nl(f)
│
per block: flows a>b ──► R1 statements ──► R2 call ⟨args⟩->_r:
├─ sink table? ──► sink calls (family, callee, loc, arg nodes)
├─ source API? ──► seed (f, r)
├─ sanitizer? ──► skip R3, R4
├─ R3 args → r
└─ R4 args → callee params, callee _0 → r (callee ∈ Φ_M, dyn expanded)
param children of Src (not self) ──► seeds (f, i+1)
│
A = pattern(R1 ∪ R2 ∪ R3 ∪ R4) s = seeds
│
T = ops::reach(s, A) (vxm, ANY.PAIR)
│
sink calls with an argument in T ──► BFS parents ──► rows {family, sink, fn, loc, source, path, hops}
──► tainted, tainted_<family>, tainted_fns, …
Worked example 1: the fixture, by hand
Fixture taint_fixture() (crates/mimir-lens/tests/graph_lenses_study.rs:263-379), test
taint_tracks_params_and_api_results_to_sinks_through_callees_and_stops_at_sanitizers
(:600-677). Sources handle() safe() via() deep() nomir() me(), extra
sanitizer clean(), default sinks and source APIs. Every fn has one param
req, except me, whose only param is self. No block of the fixture has a
statement except wrap and clean (_0=_1).
The fns with MIR and their locals (nl(f) from the locals tables and the
_n in each terminator):
| fn | blocks (terminator, flows) | nl |
|---|---|---|
me | bb0 call std fs/write⟨^_1⟩->_2, flows 1>2 | 3 |
handle | bb0 call std fs/write⟨^_1⟩->_2, flows 1>2 | 3 |
safe | bb0 call app clean⟨^_1⟩->_2, no flows; bb1 call std fs/write⟨^_2⟩->_3 | 4 |
via | bb0 call app wrap⟨^_1⟩->_2; bb1 call std process/impl#[Command]new⟨^_2⟩->_3 | 4 |
wrap | bb0 return, stmt _0=_1 | 2 |
deep | bb0 call app helper⟨^_1⟩->_2 | 3 |
helper | bb0 call std fs/remove_file⟨^_1⟩->_2 | 3 |
other | bb0 call tonic request/impl#[Request<T>]into_inner⟨^_1⟩->_2; bb1 call std fs/write⟨^_2⟩->_3 | 4 |
clean | bb0 return, stmt _0=_1 | 2 |
nomir has no blocks. So fns_with_mir = 9 and
value_nodes = N = 3+3+4+4+2+3+3+4+2 = 28. Below, x.ℓ is the node
(x, _ℓ); absolute ids depend on the node order of the load.
Edges (after dedup):
me R1 me.1→me.2 (R3 the same)
handle R1 handle.1→handle.2 (R3 the same)
safe bb0: clean() is a sanitizer → no R3, no R4; its flow 1>2 goes into the result safe.2 → dropped. bb1: R3 safe.2→safe.3
via R3 via.1→via.2; R4 via.1→wrap.1, wrap.0→via.2; bb1: R3 via.2→via.3
wrap R2 wrap.1→wrap.0
deep R3 deep.1→deep.2; R4 deep.1→helper.1, helper.0→deep.2
helper R3 helper.1→helper.2
other R3 other.1→other.2, other.2→other.3
clean R2 clean.1→clean.0
value_edges = 1+1+1+4+1+3+1+2+1 = 15
calls_entered = 2 (wrap, helper); calls_passed_through = 9 (every
call with a result except the one sanitizer call); sanitizer_calls = 1;
sink_calls = 6 (the four fs/write, Command::new, fs/remove_file).
Seeds: handle.1, safe.1, via.1, deep.1 (params req); me has only
self, so no seed, and it is not counted as without params; nomir has no
MIR, so it lands in needs; other.2 is the result of into_inner
(source API tonic request/impl#[Request). seeds = 5.
The reach:
level 0 {handle.1, safe.1, via.1, deep.1, other.2}
level 1 + handle.2, via.2, wrap.1, deep.2, helper.1, other.3
level 2 + via.3, wrap.0, helper.2
level 3 nothing new (wrap.0 → via.2 is already in T)
T = 14 nodes = tainted_nodes
The sink calls:
| sink call | argument | in T | row |
|---|---|---|---|
me fs/write | me.1 | no (self not seeded) | — |
handle fs/write | handle.1 | yes (seed) | TAINT fs, path app handle(). _1, hops 1 |
safe fs/write | safe.2 | no (the sanitizer call has no edge into safe.2) | — |
via Command::new | via.2 | yes | TAINT exec, path app via(). _1, hops 2 |
helper fs/remove_file | helper.1 | yes | TAINT fs, path app deep(). _1 → app helper(). _1, hops 2 |
other fs/write | other.2 | yes (seed) | TAINT fs, source tonic request/impl#[Request<T>]into_inner(). result |
Sorted by family, then fn label: via exec; handle, helper, other fs.
This is the order the test asserts (:622-631). The headline is
4 tainted sink calls in 4 fns (exec 1, fs 3) of 6 sink calls seen; 5 seeds (6 source fns, 1 API results), 28 value nodes, 1 sanitizer calls honoured.
"6 source fns" is |Src| (the fns the patterns matched, me and nomir
included), not the number of seeded fns (:565-580).
Two readings of the fixture:
- The
viarow's path does not passwrap. BFS reachesvia.2at level 1 over the R3 pass-through edgevia.1 → via.2, before the route through the callee (via.1 → wrap.1 → wrap.0 → via.2, level 3). The test file's header says the flow goes "through the callee's body" (:10-11); the printed path shows the shorter pass-through. safeis clean only because its call block has no flow section. With the driver's1>2insafebb0 (whatcargo mimir-dumpwrites for that call), R1 would addsafe.1 → safe.2, and thesafewrite would be TAINTED.
Worked example 2: Holger h32/next 9dbe5cbf
Plan §F.2 (server-lib; sources = the params of handle, the gRPC services,
JobRunner::run, import_door, migration, plus the default API table).
- Round 0 was RED: 0 of 783 sink calls tainted, because the lens read
arguments as
_3while the driver writes^_3. Round 1 (args fixed,selfstill seeded): 674 of 783 tainted, 230k nodes, 281k edges, 624 seeds,T= 151k nodes (plan §F.3). - Round 2 (
selfis not a source; args^_n,_n.0,&_n): 672 tainted sink calls in 303 fns, fs 338, path 294, net 40, exec 0 (check: 338 + 294 + 40 + 0 = 672), from 447 seeds (209 source fns, 160 API results; so 447 − 160 = 287 parameter seeds), over 230k value nodes and 281k edges, 150k nodes reached, 357 sanitizer calls,fns_with_mir4,115 (handlehas MIR). The bench note for the same run says 262k edges (plan §F.1); the plan records both numbers. - The current code (dwarves ba122c9: the sanitizer cut comes before R1):
671 tainted sink calls in 302 fns; path 294 → 293, fs 338 and net 40 as
before (check: 338 + 293 + 40 + 0 = 671); 357 sanitizer calls honoured. One
Path::jointhat sat behind a sanitizer call is no longer tainted. The per-class rows below are from round 2. - By source class (the lane's grouping of the
sourcefield): request derived 348 (path 137, fs 192, net 19; check 137 + 192 + 19 = 348), config derived 156, env 168 (std env/var(_os)130 + 38). Check: 348 + 156 + 168 = 672. The plan names the largest request sources:handle(req)55,(client_cn)29,(peer)22,JobRunner::run(job)32,import_bundle_unaudited(bundle/source)22,intel_in_job(source)37,recharge/watched_app/twin_status/become_twin(request)48,http Request::uri/headers27, tonicinto_inner/metadata/peer_certs/remote_addr35. These sum to 307, so 41 of the 348 come from sources the plan does not list. - Request →
Path::joinwithout a sanitizer call: 137 rows, among themAttestStore::accept(attest.rs:906),FileAuditLog::read_records(audit/mod.rs:538),import_bundle_unaudited×3 (import_door.rs),Catalog::catch_up,apk::Signer::load_or_create, andPullIntake::{apk_signer, keep_materials ×12, materials_dir, read}. The fns with most request-tainted sinks:PullIntake::keep_materials12,ScanWorker::run_job10,Fetcher::one_job9.
The lens is context-insensitive (a return reaches every call site), so these counts are still upper bounds on the realizable flows.
Output
Summary (taint.rs:468-486, :594-600): fns_with_mir, value_nodes (N),
value_edges (|R1 ∪ R2 ∪ R3 ∪ R4|), seeds, sink_calls,
sanitizer_calls, calls_entered (R4 targets entered, one per target),
calls_passed_through (calls with an R3 result), source_fns_without_params,
then tainted, tainted_fns, tainted_sink_apis (distinct callee labels),
tainted_<family> per family, tainted_nodes (|T|). Findings
(:614-633): {family, sink, fn, loc, source, path, hops}, where loc is
<fn file:line> +<line:col relative to the fn> and hops is the length of
the (fn, local) path before compression.
Cost
Plan §F.1 (the lens-study bench table, build-1, debug build, one container per
run; peak = cgroup memory.peak, page cache included): taint on Holger
server-lib with 5 source patterns, 79 s wall / 3.7 GB peak. The graph read
floor in the same table (lens cycles) is 77 s / 5.2 GB, so the lens's own
work (parsing, one BFS, the paths) is a few seconds of the 79 s. No dwarves or
gunnar row was recorded.
Limits and needs
- Sanitizers do not clean a result on a real dump (R1, see Math). Until
the sanitizer test also drops the call block's
a > rflow pairs, a sanitizer only stops the walk into its body. - Context-insensitive. A callee's
_0returns to every call site. - Flow-insensitive and field-insensitive. Any writer of
_nreaches any reader of_nin the fn;_n.0and_n.1are both_n. A write through a pointer is an R2 def of the pointer local. - Control dependence is not an edge. A segment checked by an
ifbeforejoinreads as tainted (the CPG's CONTROLS edges are the follow-up, plan §F.2). - Positional R4. Constant arguments shift the parameter index (Math, fact 1).
- Nested bodies share locals. Closures and
asyncbodies of one fn share its local range. - Callees outside the scope are not entered. Only R1/R3 carry a value across a dependency call. Any derived value counts (a constructor that keeps only a name still passes the taint, plan §F.2).
- needs.
mir_block nodes (the value-flow graph)when the dump has none (:236-243);MIR for N source fn(s)forSrc ∖ Φ_M(:462-471). That text says "the driver writes no mir_block for a body over its cap (1,024 blocks)". Since schema 6 the driver has no cap (4.1, The cap); this line names only fns with no blocks at all. - Errors, not empty answers: a
--sourcethat matches no fn, no sink patterns, no sources (:249,:271,:276).
Tests
crates/mimir-lens/tests/graph_lenses_study.rs:600taint_tracks_params_and_api_results_to_sinks_through_callees_and_stops_at_sanitizers: the four rows in order, thehelpersourceapp deep(). param _1 (req),source_fns_without_params = 0, theothersource namesinto_inner,nomirinneeds,sanitizer_calls = 1(fixture:263-379).crates/mimir-lens/tests/graph_lenses_study.rs:680the_study_lenses_are_listed.- Unit test
statement_and_call_parsing(crates/mimir-lens/src/lenses/taint.rs:643-661):stmt_flow,arg_local(^_99,_1.1,k:&str),call_ret,block_flows,locals_in,callee_label. - No Holger golden:
crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.jsonhas notaintentry.
How it relates to the other lenses
gate-bypass (4.1) asks a control question: is there a call path from an
entry to a write that avoids the gate fn? taint asks a value question:
does a value derived from the request reach the argument of the sink call? A
door can be a BYPASS without any request byte reaching the write (a migration
writing its own data), and a tainted Path::join is not a store write at
all, so neither lens subsumes the other. They also differ in precision.
gate-bypass Phase 2 is context-sensitive: it drops call/return words that do
not match (the CFPQ walk). taint is context-insensitive, so it
over-approximates exactly where gate-bypass stopped doing so. verdict
does not fold taint: crates/mimir-lens/src/lenses/verdict.rs has no reference to it (its
only choke is the gate-bypass --choke pattern, verdict.rs:60, :328).
4.10 chokepoints
The question. Where could one gate sit so that every path from a request
entry to a sink passes it? On the fn call graph restricted to the fns that lie
on some entry → sink path, the answer is the set of fns that dominate every
reached sink from a virtual root over the entries (dominators, Cooper, Harvey
and Kennedy, A Simple, Fast Dominance Algorithm, 2001, through order::immediate_dominators in
crates/mimir-lens/src/lenses/order.rs). When that set is empty, no single fn mediates every path. The
lens then ranks the fns by the share of entry → sink shortest paths they lie
on (betweenness, Brandes, A Faster Algorithm for Betweenness Centrality,
J. Math. Sociol. 2001, accumulated towards the sinks only): the place a gate
would catch most paths. The code is crates/mimir-lens/src/lenses/chokepoints.rs; the plan
row is F-2 (plan §F).
Run
cargo lens chokepoints --entry PAT… --sink PAT… [--top N] \
[--mimir-dump DIR] [--crate NAME…] [--include-tests] [--include-deps] [--json] [--limit N]
# Holger, plan §F.1/§F.2 (9 entry and 5 sink patterns; the exact strings are not recorded):
cargo lens chokepoints --crate holger-server-lib \
--entry 'exposed/http/handle()' … --sink 'RepositoryBackendTrait#put()' …
The flags are in dw:src/cli.rs:416-431: --entry and --sink are
required and repeatable; --top (betweenness rows to list) defaults to 15.
Dispatch: dw:src/cli.rs:773-788. Patterns are label substrings as in
gate-bypass. A pattern set that matches no fn is an error
(chokepoints.rs:117-130). --include-deps has no effect here (see the fn set).
From the compiler to the graph
The lens reads only fn nodes and three edge kinds from cargo mimir-dump:
CALLS (caller → callee), IMPLEMENTS (impl → trait) and CONTAINS (trait →
method, impl → method), through Graph::load (crates/mimir-lens/src/lenses/mod.rs:115).
The last two give the dyn expansion dyn_targets(t) of 4.1: a call to a trait
method also reaches the same-named method of every impl. No MIR is used; the
lens is fn-level by design.
Math
The sets (chokepoints.rs:120-150).
E = fns_matching(--entry) S = fns_matching(--sink) (any crate; the scope does not filter them)
K = { v : is_fn(v) ∧ ¬(¬include_tests ∧ (test_code(v) ∨ generated(v))) ∧ (v ∈ S ∨ v ∈ E ∨ crate(v) ∈ --crate or no --crate) } (:145-150)
K uses in_scope (the crate list only), not Scope::holds. It does not test
defined_here, origin = local or derive-written code. On a full dump without
--crate the call graph therefore includes the dependency crates (whatever
--include-deps says). Entries and sinks that are test code drop out of K.
The call matrix (:151-171), boolean n × n over all dump nodes:
C = pattern( { (f,t) : f, t ∈ K, (f,t) ∈ CALLS } ∪ { (f,d) : f, d ∈ K, (f,t) ∈ CALLS, d ∈ dyn_targets(t) } )
The dyn edge f → d is added for every callee t of f, whether or not
t ∈ K.
The live subgraph (:172-197).
F = reach(1_E, C) every node an entry reaches (entries included)
B = reverse_reach(1_S, C) every node that reaches a sink = reach of S over Cᵀ (sinks included)
L = F ∩ B E_L = E ∩ L, S_L = S ∩ L
S_L = ∅ ⇒ headline "no entry reaches a sink: nothing to mediate", stop (:203-206)
Every fn on an entry → sink path of C is in L, and every fn of L lies on
one, so L is exactly the union of the entry → sink paths. A path may pass
through one sink on its way to another; sinks are not blocked.
The dominator tree (:207-251). Index L in ascending node id as
1 … |L|, add a virtual root ρ = 0, and take
G_L = (ρ ∪ L, C|_{L×L} ∪ { ρ → e : e ∈ E_L })
Dom(ρ) = {ρ}, Dom(v) = {v} ∪ ⋂_{p ∈ pred(v)} Dom(p) (greatest fixpoint; order::immediate_dominators = Cooper–Harvey–Kennedy,
iterating idom over reverse postorder until stable)
idom(v) = the dominator of v nearest to v, other than v
u dominates v when every path ρ ⇝ v in G_L passes u. A path in
G_L is a call chain, so "passes u" means u's frame is on the stack when
v runs (the same reading as the fn-level phase of gate-bypass).
Gate positions (:239-269).
chain(s) = ( idom(s), idom(idom(s)), … ) stopping before ρ = Dom(s) ∖ {ρ, s} as a sequence
D(s) = chain(s) ∖ (S ∪ E)
G = ⋂_{s ∈ S_L} D(s)
depth(v) = number of idom steps from v to ρ (a child of ρ has depth 1) (:270-282)
order: depth descending, then label ascending (:283-289)
Code vs doc. The module doc's math (:33) and plan §F.3 write
G = ⋂ (Dom(s) ∖ {ρ, s} ∖ E); the module's list item 3 (:10-12) and plan
§F says "root and the sinks themselves excluded". The code removes every
sink and every entry from every chain (:257-260): a sink that dominates
another sink (a sink method that calls another sink) is never a gate position.
The deepest position is the one nearest the sinks. "Nearest" is by tree depth: two positions with the same depth are ordered by label, not by distance.
Betweenness towards the sinks (sink_betweenness, :71-113; ranking
:290-307). For each e ∈ E_L, a BFS over G_L from e (ρ has no in-edge,
so it is never reached):
d_e(e) = 0, σ_e(e) = 1
P_e(w) = { v : (v,w) ∈ G_L, d_e(v) + 1 = d_e(w) } shortest-path predecessors
σ_e(w) = Σ_{v ∈ P_e(w)} σ_e(v) number of shortest e → w paths
δ_e(v) = Σ_{w : v ∈ P_e(w)} (σ_e(v) / σ_e(w)) · ( [w ∈ S_L] + δ_e(w) ) (backward, over the BFS order reversed)
cb(v) = Σ_{e ∈ E_L, e ≠ v} δ_e(v)
bc(v) = cb(v) / max(1, |E_L| · |S_L|) (:294)
By Brandes's lemma, δ_e(v) = Σ_{t ∈ S_L} σ_et(v) / σ_et: the fraction of
shortest e → t paths through v, summed over the sinks t that e
reaches. So bc(v) is the mean over the |E_L|·|S_L| entry–sink pairs of
that fraction, with unreachable pairs counted as 0. It lies in [0, 1], and it
is 1 only when every entry reaches every sink and v is on all their shortest
paths.
Code vs doc. The module's list item 5 (:16-17) and the plan §F table
say "normalised by |E|·|S|". The code divides by |E ∩ L| · |S ∩ L| (the live
entries and sinks), as the module's math section (:36) and plan §F.3 say. On
Holger the difference is large: 217 matched entry fns, 13 live.
ranked = { v ∈ L : v ∉ S_L, v ∉ E, bc(v) > 0 }, sorted by bc descending, then label ascending; the first --top rows (:295-307, :342)
A row carries [dominates] when v ∈ G. Betweenness counts shortest paths
only: a fn on every path but not on every shortest path ranks below 1, and the
dominator set, not the rank, says "every path".
GraphBLAS steps
| step | operation | semiring / op as the code names it | computes | file:line |
|---|---|---|---|---|
| 1 | C = pattern(edges) | GxB_Matrix_build_Scalar, iso true | the dyn-expanded call matrix over K | crates/mimir-lens/src/lenses/chokepoints.rs:171, crates/mimir-lens/src/lenses/grb.rs:21-30 |
| 2 | F = reach(1_E, C) | ops::reach: GrB_vxm with GxB_ANY_PAIR_BOOL, GrB_DESC_RSC; GrB_Vector_assign_BOOL with GrB_DESC_S | fns an entry reaches | :174-178; crates/mimir-graph/src/ops.rs:49-98, :103-110 |
| 3 | B = reverse_reach(1_S, C) | ops::reverse_reach: GrB_mxv on C (no transpose built) with GxB_ANY_PAIR_BOOL, GrB_DESC_RSC; same assign | fns that reach a sink | :179-183; ops.rs:76-85, :115-121 |
| 4 | L = F ∩ B | none: BTreeSet intersection | the live fns | :184-187 |
| 5 | dominators | none: order::immediate_dominators on the adjacency lists | idom, chains, G, depth | :231-289 |
| 6 | betweenness | none: Brandes BFS + backward pass in Rust (VecDeque) | bc | :71-113, :290-307 |
ANY.PAIR on bool is the structural OR-AND (LOR.LAND) reach. The plan
cites LAGraph's batched LAGr_Betweenness; the code does not use it.
Dataflow
code_nodes (fn, method, trait, impl) code_edges (CALLS, IMPLEMENTS, CONTAINS)
│ │
└──────────► Graph::load (mod.rs:115) ◄───────┘
calls · dyn_targets (same-named impl methods)
│
--entry ──► E --sink ──► S --crate / --include-tests ──► K
│
C = pattern(CALLS|K ∪ dyn edges|K) (n × n, bool)
│ │
ops::reach(E) (vxm, ANY.PAIR) ops::reverse_reach(S) (mxv, ANY.PAIR)
└───────────── F ∩ B ──────────┘
│
L, E_L, S_L ──► entries_reaching_a_sink, sinks_reached, live_fns, live_edges
│
G_L = C|L + ρ → E_L
│ │
order::immediate_dominators from ρ Brandes from each e ∈ E_L, targets S_L
│ │
chain(s) ∖ (S ∪ E) ──► ⋂ = G ──► GATE-POSITION rows bc / (|E_L|·|S_L|) ──► CHOKE rows (top N)
chain(s) per sink ──► SINK rows
Worked example 1: the fixture, by hand
Fixture gate_fixture() (crates/mimir-lens/tests/graph_lenses_study.rs:229-261), test
chokepoints_finds_the_gate_position_for_one_entry_and_none_for_two
(:421-486). CALLS: http_handle → router → gate_ingest → Store#put,
mirror_run → Store#put, Disk::put → fs_write, f1 ↔ f2. Disk implements
Store, so dyn_targets(Store#put) = {Disk::put}. Sink pattern
Store#put() matches only the trait method app Store#put(). (the impl
method's label is app impl#[Disk][Store]put().), so S = {Store#put}.
C = http_handle→router, router→gate_ingest, gate_ingest→Store#put,
gate_ingest→Disk::put (dyn), mirror_run→Store#put,
mirror_run→Disk::put (dyn), Disk::put→fs_write, f1→f2, f2→f1.
One entry, E = {http_handle}.
F = {http_handle, router, gate_ingest, Store#put, Disk::put, fs_write}
B = {Store#put, gate_ingest, mirror_run, router, http_handle}
L = {http_handle, router, gate_ingest, Store#put} live_fns 4, live_edges 3
G_L: ρ → http_handle → router → gate_ingest → Store#put
idom: http_handle ← ρ, router ← http_handle, gate_ingest ← router, Store#put ← gate_ingest
chain(Store#put) = (gate_ingest, router, http_handle); D = chain ∖ (S ∪ E) = {gate_ingest, router}
G = {gate_ingest, router}; depth(gate_ingest) = 3, depth(router) = 2 → order gate_ingest, router
This is the order the test asserts (:441-446). Betweenness: from
http_handle there is one shortest path to the sink, so σ = 1 everywhere,
δ(gate_ingest) = 1·(1 + 0) = 1, δ(router) = 1·(0 + 1) = 1; the norm is
1·1 = 1; so bc = 1.000 for both, each tagged [dominates], tie broken by
label (app gate_ingest(). before app router().). The headline is
2 fn(s) dominate every entry → sink path (1 entries, 1 sinks, 4 live fns); nearest to the sinks: app gate_ingest().
Two entries, E = {http_handle, mirror_run}.
L = {http_handle, router, gate_ingest, Store#put, mirror_run} live_fns 5, live_edges 4
pred(Store#put) = {gate_ingest, mirror_run}: Dom = {ρ, Store#put} → chain(Store#put) = ()
G = ∅ → headline "NO fn dominates every entry → sink path …"; SINK row "dominated by: nothing but the entries"
from http_handle: δ(gate_ingest) = 1, δ(router) = 1
from mirror_run: the one path mirror_run → Store#put has no inner fn
cb(gate_ingest) = cb(router) = 1; norm = |E_L|·|S_L| = 2·1 = 2; bc = 1/2 = 0.5 each
Two CHOKE rows at 0.500 (:462-475), and the headline ends
Top choke by betweenness: app gate_ingest(). (0.50). With entry
nothing_like_this the run is an error (:476-485).
Worked example 2: Holger h32/next 9dbe5cbf
Plan §F.2 (server-lib, 9 entry and 5 sink patterns, the gate-bypass set of
4.1):
- 217 fns match the entry patterns; 13 reach a sink (
|E_L| = 13); 5 sinks are reached (|S_L| = 5); 80 live fns, 148 live edges. - Gate positions: 0. Every sink's chain is "nothing but the entries".
- Top betweenness:
ChainView::handle_http2_request/…_bytes0.031, thenoci_push::Staged::get,PullIntake::runandtwin_follow::Follower::{cycle, recharge, spawn, start}at 0.015, thenrun_ecosystem,pip::project_files,run_go,run_file,run_project,maven::metadata_text. - The arithmetic (computed here from the recorded counts): the norm is
13 · 5 = 65.2/65 = 0.0308prints as 0.031 and1/65 = 0.0154as 0.015, soChainView's two fns each carry the equivalent of two whole entry–sink pairs (cb = 2) and the 0.015 row fns one (cb = 1), out of 65. No fn is on the shortest paths of more than 2 of the 65 pairs.
The plan reads this as the measured form of the review's §7(b): no fn exists
where a runtime gate could mediate every entry → sink path, so a type (the
Admitted typestate of R1, 4.1) is the only gate shape.
Output
Summary (:198-202, :226-229, :308): entries, entries_reaching_a_sink
(|E_L|), sinks, sinks_reached (|S_L|), live_fns (|L|), live_edges
(edges of G_L without the root edges), gate_positions (|G|). Findings, in
this order: {class: gate-position, fn, loc, depth} (:330-341);
{class: choke, fn, loc, betweenness, dominates_all}, the first --top
(:342-354); {class: sink, sink, dominators} per live sink, in node-id
order (:355-369).
Cost
Plan §F.1 (lens-study bench table): chokepoints on Holger server-lib, 9
entry and 5 sink patterns, 86 s wall / 4.3 GB peak. The note says the live
subgraph is 80 fns, dominators and betweenness are instant, and the cost is
the read (the graph read floor in that table is 77 s / 5.2 GB). Betweenness is
O(|E_L| · |G_L| edges); the dominator solver is near-linear on 80 nodes.
Limits and needs
- Fn-level. No MIR, no call/return matching, no runtime
if: a toggle in front of a sink call is not a dominator. - Shortest paths only in the betweenness; longer paths count 0.
- The dyn expansion over-approximates (every impl is a target), which can only remove dominators, the safe side for "no gate exists" but not for "this fn dominates".
- The scope is the crate list only (see the fn set), so on a full dump
dependency fns join the live subgraph unless
--crateis given. - needs (
:131-142):CALLS edgesandIMPLEMENTS edges (dyn expansion)when the dump has none.
Tests
crates/mimir-lens/tests/graph_lenses_study.rs:421chokepoints_finds_the_gate_position_for_one_entry_and_none_for_two: positionsgate_ingest, routerfor one entry;NO fn dominates, two CHOKE rows, the first at 0.5 for two entries; an error for an entry that matches nothing (fixture:229-261).crates/mimir-lens/tests/graph_lenses_study.rs:680the_study_lenses_are_listed.- No unit test of
sink_betweennessand no Holger golden (crates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.jsonhas nochokepointsentry).
How it relates to the other lenses
gate-bypass (4.1) takes the gate as given (--choke) and lists the doors
that escape it. chokepoints takes no gate and asks where one would
have to sit: the common dominators if one fn can mediate everything, the
betweenness rank if none can. The fns a top-ranked choke would miss are the
paths gate-bypass reports once that fn is passed as --choke. Both work on
the same dyn-expanded call graph, but gate-bypass Phase 1 blocks sinks and
test code during its walk, while chokepoints blocks nothing inside L.
chokepoints has no MIR phase. Against risk (change risk = downstream
callers × churn × (1 − coverage), 3.4 change_risk): risk ranks files by
how likely a change breaks something, over the whole code. chokepoints ranks
fns only by their position on entry → sink paths, with no churn or coverage
input. A high-betweenness fn is where a security gate would pay off, not
necessarily a risky file. verdict does not fold chokepoints:
crates/mimir-lens/src/lenses/verdict.rs has no reference to it (its chokes field is
gate-bypass's --choke list, verdict.rs:60, :328-334).
4.11 misplaced
misplaced finds code that lives in the wrong module. It draws an undirected
weighted graph over the fns in scope and the types they use, finds its
communities with the Leiden algorithm, and compares them with the partitions
that the modules and the crates draw. A fn whose community belongs mostly to
another module, and which has more edge weight into that module than into its
own, is MISPLACED. The lens prints MOVE rows (home module → target module,
with a fn count), SPLIT rows (a module pulled apart by three or more
communities), the modularity Q of all three partitions and the MoJoFM
agreement of the module and crate partitions with the communities. The code is
crates/mimir-lens/src/lenses/misplaced.rs; the aggregation product is in
crates/mimir-lens/src/lenses/grb2.rs.
Sources: Traag, Waltman and van Eck, From Louvain to Leiden: guaranteeing
well-connected communities, Scientific Reports 9:5233, 2019 (Leiden);
Wen and Tzerpos, An Effectiveness Measure for Software Clustering
Algorithms, IWPC 2004 (MoJoFM); Tzerpos and Holt, MoJo: a distance metric
for software clusterings, WCRE 1999 (the move + join heuristic)
(crates/mimir-lens/src/lenses/misplaced.rs:1-23, plan §F row F-3).
Run
cargo lens misplaced [--gamma 1.0] [--max-levels 20] [--split-share 0.2] \
[--weight-cap 20] [--top 50] \
[--mimir-dump DIR] [--crate NAME…] [--include-tests] [--include-deps] [--json] [--limit N]
# the Holger run of plan §F.2:
dwarves lens misplaced --mimir-dump DIR --json --limit 0
The flags are in dw:src/cli.rs:434-452, the shared dump and scope flags in
dw:src/cli.rs:553-583, and the call into the lens in dw:src/cli.rs:789-808.
The CLI defaults equal MisplacedArgs::default()
(crates/mimir-lens/src/lenses/misplaced.rs:94-105), but they are written twice: the lens
has no lens_defaults.rs row, unlike twins.
| knob | default | meaning | file:line |
|---|---|---|---|
gamma (γ) | 1.0 | resolution of modularity | crates/mimir-lens/src/lenses/misplaced.rs:98 |
max_levels | 20 | maximum number of local-move rounds (levels) | :99 |
split_share | 0.2 | share of a module's fns a community must hold to count for SPLIT | :100 |
weight_cap | 20 | cap on the call-site weight of one directed CALLS edge | :101 |
top | 50 | rows listed per class (MOVE, SPLIT, MISPLACED) | :102 |
| (fixed) | 5 | a module needs at least 5 fns to be a SPLIT candidate | :583 |
| (fixed) | 3 | communities needed for SPLIT | :591 |
--top caps each class, and the global --limit (default 50) caps all
findings together. The findings come in the order MOVE, SPLIT, MISPLACED
(:638-678), so at the default --limit 50 a run with 50 MOVE rows prints no
MISPLACED rows. Use --limit 0 to see all classes.
Reads (from the compiler to the graph)
The driver (cargo mimir-dump) writes Mímir v2 *.nodes.arrow and
*.edges.arrow; Graph::from_dump indexes them (crates/mimir-lens/src/lenses/mod.rs:130-243).
| fact | from the dump | index field | used for |
|---|---|---|---|
| fns | nodes of kind fn / method that Scope::holds (prod code of the scoped crates by default; crates/mimir-lens/src/lenses/mod.rs:729-736) | g.ids(), is_fn | fn nodes |
| calls | CALLS edges, deduplicated and sorted per caller (mod.rs:164-168, :181-188); plain calls only, no dyn expansion | calls[f] | fn–fn edges |
| call sites | the count column of CALLS, summed over duplicate edges as Σ max(count, 1) (mod.rs:167) | call_count, call_sites(f,t) (mod.rs:276-278) | the CALLS weight |
| type coupling | USES_TYPE and READS_FIELD edges; the source node is mapped to its enclosing fn by the parent chain (enclosing_fn, mod.rs:355-370) | dump.edges | fn–type edges |
| modules | the mod / crate ancestors on the CONTAINS parent chain, crate first, joined by :: (module_path, mod.rs:314-334) | module_of[row] | module partition |
| crates | the node's crate column | node.krate | crate partition |
Math
1. The node set and the weights (crates/mimir-lens/src/lenses/misplaced.rs:427-471).
F = { f : kind ∈ {fn, method}, Scope.holds(f) }
T = { t : some USES_TYPE / READS_FIELD edge x → t with enclosing_fn(x) ∈ F,
Scope.holds(t), t is not a fn }
nodes = F ∪ T, n = |F| + |T| (rows numbered in first-seen order: g.ids() order,
callees as met, then types as met)
directed CALLS weight, for f, g ∈ F, f ≠ g, g ∈ calls(f):
c(f→g) = clamp(call_sites(f, g), 1, weight_cap) weight_cap = 20
directed type weight, for each USES_TYPE / READS_FIELD edge x → t kept above:
one tuple (enclosing_fn(x), t) of weight 1
symmetric adjacency (WGraph::from_edges, :119-147): every tuple (i, j, w) is folded to the
unordered pair {i, j} and the weights are summed:
A_ij = A_ji = c(i→j) + c(j→i) (fn–fn)
A_ft = A_tf = number of USES_TYPE / READS_FIELD edges from f or its children to t
A_ii = 0 at the first level (no self calls are kept, `t == f` is skipped at :445)
k_i = Σ_j A_ij, 2m = Σ_i k_i
The module doc says "w(f,t) = 1 per USES_TYPE / READS_FIELD edge"
(:32-33). That is right per edge, but the edges of one fn and its params,
locals and blocks to the same type are summed into one weight by from_edges
(:120-124). The count column of a USES_TYPE edge is not read. A pair of
fns that call each other gets up to 2 · weight_cap = 40, because the cap is
applied to each direction before the sum.
2. Modularity (modularity, :150-171).
Q(P) = Σ_{C ∈ P} [ Σ_in(C)/2m − γ · (Σ_tot(C)/2m)² ]
= (1/2m) · Σ_ij ( A_ij − γ · k_i·k_j / 2m ) · δ(c_i, c_j)
Σ_in(C) = Σ_{i ∈ C} Σ_{j ∈ C} A_ij (an internal undirected edge counts twice, once from
each end; a self loop (i, i) counts once)
Σ_tot(C) = Σ_{i ∈ C} k_i
Q = 0 when 2m = 0. γ = gamma = 1.0
The code matches the module doc (:36-37). It sums Q over a HashMap of
communities, so the order of the floating-point additions is the hash order.
The partition does not depend on that order (see Determinism), but the last
bits of Q may.
3. Leiden, as coded (leiden, :319-379). One level is: local move on the
current graph, then refinement, then aggregation of the refined partition.
Phase 1, fast local move (local_move, :174-228).
start: level 1 from singletons, part(i) = i; later levels from the carried partition
tot(C) = Σ_{i: part(i)=C} k_i
queue = all nodes 0 … n−1 in index order (FIFO)
pop v: own = part(v); tot(own) −= k_v (v is taken out of its community)
k_{v,C} = Σ_{j ≠ v, part(j) = C} A_vj (only communities of neighbours)
Δ(C) = k_{v,C} − γ · k_v · tot(C) / 2m
best = own; then for C over the neighbour communities in ascending id:
best = C if Δ(C) > Δ(best) + 1e-12
tot(best) += k_v
if best ≠ own: part(v) = best, and every neighbour j with part(j) ≠ best that is not
already queued is pushed to the back of the queue
stop: when the queue is empty; returns "something moved"
Δ(C) is 2m/2 times the modularity gain of moving the isolated v into
C, so the argmax is the same as in the paper. A tie goes to own, then to
the lowest community id. The code has a second tie clause (c < best with
equal gain, :209), but the candidates are visited in ascending order, so a
later c is never smaller than best: the clause never fires.
Phase 2, refinement (refine, :232-305). For each community C of the
local-move partition (in ascending community id, nodes in ascending index):
start: every node is its own sub-community, r(i) = i
K_C = Σ_{i ∈ C} k_i, K_S = Σ_{i ∈ S} k_i, E(X, Y) = Σ_{i ∈ X, j ∈ Y} A_ij
for v in C (ascending), only while v is still a singleton:
skip v unless E(v, C∖v) ≥ γ · k_v · (K_C − k_v) / 2m (v well connected to C)
for each neighbour sub-community S ⊂ C, S ≠ {v}, in ascending id:
skip S unless E(S, C∖S) ≥ γ · K_S · (K_C − K_S) / 2m (S well connected to C)
Δ_S = k_{v,S} − γ · k_v · K_S / 2m
v joins the S with the largest Δ_S, if that Δ_S > 1e-12
(ties: the lowest sub-community id; no S passes: v stays alone)
Traag et al. draw S at random with probability ∝ exp(Δ_S / θ) among the
S with Δ_S ≥ 0, and visit the nodes in random order. The code takes the
θ → 0 limit (the largest Δ_S), visits the nodes in index order, and needs
Δ_S > 1e-12 instead of ≥ 0, so a zero-gain merge is not made. The two
well-connectedness conditions are the paper's, written for modularity
(:262, :286). E(S, C∖S) is recomputed by a scan of all of C for every
candidate (:279-285), so this phase is O(|C|² · degree) per community in
the worst case.
Phase 3, aggregation (:336-374). With the refined partition relabelled to
dense ids 0 … k−1 in order of first appearance (dense, :308-316):
P[i][c] = 1 iff r(i) = c (n × k, UINT64)
A' = Pᵀ · (A · P) (k × k, PLUS.TIMES on UINT64)
A'_IJ = Σ_{i ∈ I, j ∈ J} A_ij
next graph: keep A'_IJ for I < J (an undirected edge once) and A'_II as a self loop
carried once; from_edges then gives
k'_I = A'_II + Σ_{J ≠ I} A'_IJ = Σ_{i ∈ I} k_i (degrees and 2m are kept)
next start partition: part'(I) = part(i) for the members i of I (the NON-refined partition)
The module doc says "the diagonal = the internal weight" (:49). As coded,
A'_II = Σ_in(I), twice the internal undirected weight (plus old self loops).
That is what keeps k' and Q equal on both levels, so the code is right and
the wording is loose.
Iteration and stop rule (:325-375).
loop:
moved = local_move(cur, part); levels += 1
if not moved or levels ≥ max_levels: stop max_levels = 20
refined = refine(cur, part); k = number of refined parts
if k = |nodes of cur|: stop (nothing to aggregate)
cur, part = aggregate(cur, refined), carried part
result: part mapped back to the original nodes, relabelled dense in first-seen order
levels in the output counts local-move rounds, including the last one that
moved nothing. When the run stops at max_levels, the last local move is not
refined. The paper stops when the local move leaves every community a single
aggregate node; the code stops one step earlier or at the same point, when
nothing moves.
Determinism. The partition is byte-stable: the queue and the refinement visit
nodes in index order, candidates are sorted, all weights are integers held
exactly in f64 (below 2⁵³), and dense relabels in order. The graph is
built in g.ids() order, so the same dump gives the same rows.
4. The three partitions and Q (:479-496). Every node of F ∪ T gets a
module (module_path) and a crate (krate). Q_leiden, Q_modules and
Q_crates are Q of the Leiden, module and crate partitions on the same
graph and the same γ. Type nodes count in all three.
5. MoJoFM (mojofm, :385-411), computed as mojofm(modules, communities)
and mojofm(crates, communities), so A = the drawn partition and B = the
communities, over all n nodes (fns and types):
o(a, b) = |{ i : A(i) = a, B(i) = b }|
tag(a) = argmax_b o(a, b) (tie: the smaller b id)
moves = n − Σ_a o(a, tag(a))
joins = |{A clusters}| − |{ distinct tags }|
mno = moves + joins (Tzerpos–Holt heuristic)
max_mno = max( n − |B|, n − max_j |B_j|, 1 )
MoJoFM = clamp( 1 − mno / max_mno, 0, 1 ) (JSON: a fraction; text: × 100 %)
n = 0 → 1.0
The tag heuristic gives a valid sequence of moves and joins, so mno is an
upper bound on the true minimum, and the printed MoJoFM is a lower bound, as
the module doc says (:22-23). The code's max_mno is the worse of two
extreme partitions (all singletons: n − |B| joins; one cluster:
n − max_j |B_j| moves). This text does not check that expression against the
maximum max(mno(∀A, B)) that Wen and Tzerpos derive; if their maximum is
larger, the code's figure is lower still, so it stays a lower bound. MoJo is
not symmetric: these figures say how many moves and joins turn the module (or
crate) partition into the communities.
6. MISPLACED, MOVE, SPLIT (:499-608). Only fns count here; type nodes are
in the graph but are never listed.
maj(C) = the module with the most fns in community C (tie: the smaller module name)
share(C) = |{f ∈ C : mod(f) = maj(C)}| / |{f ∈ C}|
w(f, M) = Σ_{j : mod(j) = M} A_fj (neighbours j are fns AND types)
MISPLACED f ⇔ maj(C(f)) ≠ mod(f) ∧ w(f, maj(C(f))) > w(f, mod(f))
order = w_target − w_home descending, then the fn label ascending
MOVE (home, target) = |{ MISPLACED f : mod(f) = home, maj(C(f)) = target }|
order = count descending, then (home, target) ascending
SPLIT M ⇔ |{f : mod(f) = M}| ≥ 5 ∧
|{ C : |{f ∈ C : mod(f) = M}| ≥ split_share · |{f : mod(f) = M}| }| ≥ 3
order = number of such communities descending, then fns descending, then name
The plan (F-3) and the module doc (:17-18) give SPLIT as "≥ 3 communities
holding ≥ 20 % each". The code has one more rule that neither states: a module
with fewer than 5 fns is never a SPLIT (:583-585). At split_share = 0.2
at most five communities can pass.
GraphBLAS steps
| step | operation | semiring / op as named in code | computes | file:line |
|---|---|---|---|---|
| 1 | A (n×n, UINT64) from the adjacency lists, both directions, self loops once | GrB_Matrix_build_UINT64, dup GrB_PLUS_UINT64 | the current level's adjacency | crates/mimir-lens/src/lenses/misplaced.rs:337-343, crates/mimir-lens/src/lenses/grb2.rs:21-32, nornir crates/mimir-graph/src/matrix.rs:46-74, types.rs:188-195 |
| 2 | P (n×k, UINT64), one 1 per row | same build | membership of the refined partition | crates/mimir-lens/src/lenses/misplaced.rs:344-349 |
| 3 | AP = A·P | GrB_mxm with GrB_PLUS_TIMES_SEMIRING_UINT64, no descriptor | weight of each node into each refined part | crates/mimir-lens/src/lenses/misplaced.rs:350, crates/mimir-lens/src/lenses/grb2.rs:35-55 |
| 4 | A' = Pᵀ·AP | GrB_mxm with GrB_PLUS_TIMES_SEMIRING_UINT64, GrB_DESC_T0 (transpose the first input) | the aggregated graph | crates/mimir-lens/src/lenses/misplaced.rs:351, crates/mimir-lens/src/lenses/grb2.rs:46-50 |
| 5 | tuples, keep I < J and I = I | GrB_Matrix_extractTuples_UINT64, then a Rust filter (not GraphBLAS) | the next level's edge list | crates/mimir-lens/src/lenses/misplaced.rs:352-364, crates/mimir-lens/src/lenses/grb2.rs:58-68 |
| 6 | local move, refinement, Q, MoJoFM, majority, MOVE, SPLIT | not GraphBLAS (Rust, hash maps and sorted vectors) | the partition and the rows | crates/mimir-lens/src/lenses/misplaced.rs:150-411, :499-608 |
GraphBLAS is used only for the aggregation, once per level.
Dataflow
Mímir v2 dump graph Leiden (per level) output
───────────── ───── ────────────────── ──────
CALLS (dedup, plain) + count ─► c(f→g)=clamp(sites,1,20) ─┐
USES_TYPE / READS_FIELD ──────► (enclosing_fn, type), 1 ──┼─► WGraph A (u64, symmetric, sums)
+ parent chain │ k_i, 2m
fn / method nodes, Scope ─────► F ; types ─► T ───────────┘ │
▼
┌──────── local move (queue, Δ = k_vC − γ k_v tot/2m)
│ │ part
│ ▼
│ refine (well-connected, θ→0) ─► refined r
│ │
│ ▼
│ P (n×k) ─► A' = Pᵀ·(A·P) GrB PLUS.TIMES UINT64
│ │ keep I<J and diagonal
└── next level ◄┘ part carried (non-refined)
│ stop: nothing moved / k = n / 20 levels
▼
CONTAINS chain ─► module_path ─► module partition ─┐ communities ─┬─► Q_leiden, Q_modules, Q_crates
crate column ───► crate partition ─────────────────┼───────────────┼─► MoJoFM modules→comm, crates→comm
│ └─► maj(C), w(f,M)
└──────────────────► MISPLACED, MOVE, SPLIT rows
summary {nodes, fns, edges, communities,
levels, gamma, q_*, mojofm_*, misplaced,
split_modules}
Output
Summary keys (:625-637): nodes (n), fns (|F|), edges (undirected
edges of the first-level graph, Σ |adj| / 2), communities, levels,
gamma, q_leiden, q_modules, q_crates, mojofm_modules,
mojofm_crates (fractions in [0, 1]), misplaced, split_modules.
The headline (:610-624) reads
K communities over F fns + T types (γ G, L levels): Q leiden … vs modules … vs crates …; MoJoFM modules→communities X %, crates Y %; M misplaced fns, S modules to split.
Findings (:638-678):
| class | text | JSON |
|---|---|---|
| move | MOVE c fn(s) home → target | {class: "move", from, to, fns} |
| split | SPLIT M (N fns) spreads over K communities: cX ×n (mostly M'), … | {class: "split", module, fns, communities: [{community, fns}]} |
| misplaced | MISPLACED label (file:line) home H → T (community P % that module; edges a home vs b there) | {class: "misplaced", fn, loc, home, target, w_home, w_target, community_share} |
One note line names the method (:679).
Worked example 1: the fixture, by hand
Fixture misplaced_fixture (crates/mimir-lens/tests/graph_lenses_study.rs:381-407), crate
app: module a holds a1 a2 a3, module b holds b1 b2 b3; inside each
module every fn calls the other two; a::stray calls b1 b2 b3 and each of
them calls stray. Every CALLS edge has count = 1 (:201). There are no
USES_TYPE edges, so T = ∅ and the lens adds a needs line for them
(:422-427).
Graph. n = 7. Rows in first-seen order: a1 = 0, a2 = 1, a3 = 2, b1 = 3, b2 = 4, b3 = 5, stray = 6 (b1's sorted callees are b2, b3, stray). Each
pair that calls both ways gets A = 1 + 1 = 2. Nine undirected edges of weight
2: three in a, three in b, three stray–b_i.
k(a_i) = 2 + 2 = 4, k(b_i) = 2 + 2 + 2 = 6, k(stray) = 3 · 2 = 6
2m = 3·4 + 3·6 + 6 = 36
Level 1, local move (γ = 1, singletons, tot(i) = k_i):
| v | own | candidates Δ(C) = k_{v,C} − k_v·tot(C)/36 | goes to |
|---|---|---|---|
| a1 (0) | 0 | own 0; {1}: 2 − 4·4/36 = 1.556; {2}: 1.556 | 1 (first strictly best) |
| a2 (1) | 1 | own (tot 4 after removal): 2 − 16/36 = 1.556; {2}: 1.556 | stays (tie goes to own) |
| a3 (2) | 2 | own 0; {1} (a1, a2): 4 − 4·8/36 = 3.111 | 1 |
| b1 (3) | 3 | own 0; {4}, {5}, {6}: 2 − 6·6/36 = 1 each | 4 |
| b2 (4) | 4 | own (tot 6): 2 − 1 = 1; {5}: 1; {6}: 1 | stays |
| b3 (5) | 5 | own 0; {4} (b1, b2): 4 − 6·12/36 = 2; {6}: 1 | 4 |
| stray (6) | 6 | own 0; {4} (b1, b2, b3): 6 − 6·18/36 = 3 | 4 |
The queue is then empty (every re-queued neighbour was still queued). Result:
{a1, a2, a3} and {b1, b2, b3, stray}; something moved.
Refinement. In C = {a1, a2, a3}, K_C = 12: a1 is well connected
(4 ≥ 4·8/36 = 0.889), joins {a2} (E({a2}, C∖{a2}) = 4 ≥ 0.889,
Δ = 2 − 16/36 = 1.556); a2 is no longer a singleton; a3 joins {a1, a2}
(E = 4 ≥ 8·4/36 = 0.889, Δ = 4 − 32/36 = 3.111). In
C = {b1, b2, b3, stray}, K_C = 24: b1 joins {b2} (Δ = 1), b3 picks
{b1, b2} (Δ = 4 − 72/36 = 2) over {stray} (Δ = 1), stray joins
{b1, b2, b3} (E = 6 ≥ 18·6/36 = 3, Δ = 6 − 108/36 = 3). The refined
partition equals the local-move partition, k = 2 < 7, so the lens aggregates.
Aggregation. A' = Pᵀ·A·P = [[12, 0], [0, 24]]: A'_00 = Σ_in(a) = 3 edges · 2 directions · 2 = 12, A'_11 = 6 · 2 · 2 = 24. Two nodes with self
loops 12 and 24, degrees 12 and 24, 2m = 36.
Level 2. Neither node has a neighbour (self loops are skipped), so nothing
moves: stop with levels = 2, communities = 2.
Q.
Q_leiden = 12/36 − (12/36)² + 24/36 − (24/36)² = 1 − (144 + 576)/1296 = 0.444
Q_modules : a = {a1,a2,a3,stray}: Σ_in 12, Σ_tot 18; b = {b1,b2,b3}: Σ_in 12, Σ_tot 18
= 24/36 − 2 · (18/36)² = 0.667 − 0.5 = 0.167
Q_crates : one crate: 36/36 − 1² = 0
MoJoFM. Modules → communities: o(a, c0) = 3, o(a, c1) = 1,
o(b, c1) = 3; tags a → c0, b → c1; moves = 7 − 6 = 1, joins = 2 − 2 = 0, mno = 1; max_mno = max(7 − 2, 7 − 4, 1) = 5; MoJoFM = 1 − 1/5 = 0.80. Crates → communities: one cluster tagged c1 (overlap 4), moves = 3,
joins = 0; MoJoFM = 1 − 3/5 = 0.40.
Rows. maj(c0) = app::a, maj(c1) = app::b (3 of 4 fns, share 0.75). stray
has home app::a, w(stray, app::a) = 0, w(stray, app::b) = 6 > 0: MISPLACED
app a/stray(). home app::a → app::b (community 75 % that module, edges 0
home vs 6 there); MOVE app::a → app::b ×1. No module has 5 fns, so there is
no SPLIT. The test asserts communities = 2, the one MISPLACED row, the MOVE
row and q_leiden > q_modules (crates/mimir-lens/tests/graph_lenses_study.rs:533-598). The
values 0.444, 0.167, 80 %, 40 % and levels = 2 are computed here by hand;
the module doc states the two Q values (crates/mimir-lens/src/lenses/misplaced.rs:66-71),
and the test does not pin the MoJoFM values.
Worked example 2: Holger (plan §F.2)
Holger h32/next 9dbe5cbf, prod code, γ = 1, the defaults:
| quantity | value |
|---|---|
| nodes | 6,766 fns + 5,735 types = 12,501 |
| communities / levels | 186 / 6 |
Q_leiden / Q_modules / Q_crates | 0.828 / 0.588 / 0.461 |
| MoJoFM modules → communities / crates → communities | 81 % / 36 % |
| MISPLACED fns / SPLIT modules | 407 / 6 |
Top MOVE rows: grpc::services → grpc 38 (the RepositoryService /
PromotionService impls: put_artifact, stream_artifact, fetch_artifact,
promote_artifact have 0–3 edges home vs 25–47 there), plugin_abi::v2::payload → v2 16, plugin_abi::codec → v2 12, pull_intake::ecosystems → pull_intake
9, grow → grpc 7, scanner::attestation → attest 7, sync::downstreams → sync 7, pull_intake → promote::intake 6, oci_push → promote::intake 5,
sandbox → scanner 5. SPLIT: holger_traits::backend (73 fns) into
promote::intake 22, offsite 20, art_probe 15 (each ≥ 0.2 · 73 = 14.6);
scanner::provenance (34) into scanner 17, provenance::policy 9,
pull_intake 7 (each ≥ 6.8). The modules agree with the communities far more
than the crates do (81 % vs 36 %). The format code stays in the communities of
the chain, not of the format crates: the lens does not show the review's
"format knowledge outside the format crates" as moves into those crates
(plan §F.2).
The plan's F-3 row estimates "≈ 25k fns + types, 300k edges" for Holger; the recorded run has 12,501 nodes (plan §F.2, and §G "Leiden over 12.5k nodes").
Cost
Plan §F.1 (the lens-study rows stand in the plan's §G table), build-1, debug build, wall time / peak cgroup memory, dump read included:
| dump | misplaced | graph read (lens cycles) |
|---|---|---|
| Holger (12.5k nodes, 6–7 levels) | 75 s / 3.7 GB | 77 s / 5.2 GB |
| gunnar (15.9k nodes) | 93 s / 4.4 GB | 94 s / 6.4 GB |
| dwarves (8.9k nodes) | 44 s / 2.1 GB | 43 s / 3.0 GB |
The lens's own work is below the noise of the dump read.
Limits
- Communities are found at one γ. A large cycle comes out as one blob at low
γ; the γ sweep is a manual knob (
--gamma), and the Holger run used γ = 1 only. - The graph uses plain calls. Calls through
dyn Trait, closures and fn pointers end at the trait method (Fn*or the trait's own method), and overloaded operators have no CALLS edge (section 3.2). - Type nodes pull fns together: two fns that use the same type gain weight
even when they never call each other. Types are in
Qand MoJoFM but never in a row. - MISPLACED says where a fn's edges go. It cannot say whether a fn is meant to be a bridge (an adapter belongs to both sides by design).
- MoJoFM is a lower bound (heuristic
mno), and itsmax_mnois the code's own expression (Math §5). - Refinement is greedy, so it can miss a sub-community that the randomised paper version would find. The result is stable, not optimal.
- The SPLIT rule has a fixed 5-fn floor and a fixed 3-community rule.
needs
The lens names cargo mimir-dump as the producer: "CALLS edges" when the dump
has none (:415-420), and "USES_TYPE / READS_FIELD edges (fn → type
coupling)" when both are missing, in which case the communities are drawn from
calls only (:421-426).
Tests
crates/mimir-lens/tests/graph_lenses_study.rs:533misplaced_sees_the_fn_that_belongs_to_the_other_module(fixture:381-407).- Unit tests
leiden_splits_two_cliques_joined_by_one_edge(crates/mimir-lens/src/lenses/misplaced.rs:688) andmojofm_is_one_for_the_same_partition_and_lower_for_a_wrong_one(:710). crates/mimir-lens/tests/graph_lenses_study.rs:680the_study_lenses_are_listed.- There is no Holger golden for this lens; the Holger numbers above are plan §F.2 only.
How it relates to the other lenses
cycles (4.4) and dsm (4.12) both use strongly connected components: a set
of modules where each one reaches each other one. That is a statement about
direction (a knot), not about density. misplaced ignores direction and asks
where the edge weight clusters. A 90-module SCC can hold several dense
communities joined by a few back edges, and two modules with no cycle between
them can form one community. The three readings answer different questions:
cycles names the knot, dsm prices it (how much of the system a change can
reach), and misplaced says how the knot would split if the boundaries
followed the edges. dispatch-erosion (4.3) finds format knowledge outside its
home by looking at match ladders over another module's enum; misplaced finds
the same symptom from the other side, as a fn whose edges go to another
module. A ladder fn that is also MISPLACED to the enum's module is the
strongest MOVE case. verdict (section 5) does not fold misplaced: it runs
twins, size, dispatch-erosion, cycles and gate-bypass only
(crates/mimir-lens/src/lenses/verdict.rs:258-340), and its "not folded" note does not name
this lens either (crates/mimir-lens/src/lenses/verdict.rs:19-22).
4.12 dsm
dsm builds the design structure matrix of the module graph (or the crate
graph) and computes two measures from the management-science literature: the
propagation cost (how much of the system a change in one unit can reach,
on average) and the core–periphery classes (which units form the largest
cyclic group, which ones the core uses, which ones drive it). It prints one
row for the whole dump and, for modules, one row per crate. The code is
crates/mimir-lens/src/lenses/dsm.rs; the closure is in crates/mimir-lens/src/lenses/grb2.rs.
Sources: MacCormack, Rusnak and Baldwin, Exploring the Structure of Complex
Software Designs: An Empirical Study of Open Source and Proprietary Code,
Management Science 52(7), 2006 (propagation cost); Baldwin, MacCormack and
Rusnak, Hidden Structure: Using Network Methods to Map System Architecture,
Research Policy 43(8), 2014 (cores, VFI / VFO classes, architecture types)
(crates/mimir-lens/src/lenses/dsm.rs:1-21). The plan notes that the full text of the 2014
paper could not be fetched, and that the thresholds 5 %, 1.5× and 4 % are
from the paper as remembered; they are knobs here, not checked values
(plan §F, reading list).
Run
cargo lens dsm [--unit module|crate] [--min-units 5] [--core-min 0.05] \
[--core-floor 0.04] [--dominant 1.5] \
[--mimir-dump DIR] [--crate NAME…] [--include-tests] [--include-deps] [--json] [--limit N]
# the Holger run of plan §F.2:
dwarves lens dsm --mimir-dump DIR --json --limit 0
The flags are in dw:src/cli.rs:456-475, the shared flags in
dw:src/cli.rs:553-583, the call in dw:src/cli.rs:809-828. The CLI
defaults equal DsmArgs::default() (crates/mimir-lens/src/lenses/dsm.rs:78-89).
| knob | default | meaning | file:line |
|---|---|---|---|
unit | module | crate makes crates the units; any other value means modules | crates/mimir-lens/src/lenses/dsm.rs:237-243 |
min_units | 5 | a crate needs this many modules for its own row | :83, :422 |
core_min | 0.05 | core share needed for core-periphery / multi-core | :84 |
core_floor | 0.04 | below this core share the system is hierarchical | :85 |
dominant | 1.5 | the core must be this many times the second cyclic group | :86 |
Per-crate rows are made only when unit is exactly module (:416).
Reads (from the compiler to the graph)
| fact | from the dump | used for |
|---|---|---|
| fns | fn / method nodes that Scope::holds | the sources of dependencies |
| calls | CALLS, deduplicated per caller, plain (no dyn expansion; the same choice as cycles) | unit → unit edges |
| type coupling | USES_TYPE / READS_FIELD, source mapped to its enclosing fn | unit → unit edges |
| units | module_path over the CONTAINS chain (mod.rs:314-334), or the crate column | rows and columns of A |
Math
1. The DSM (unit_edges, crates/mimir-lens/src/lenses/dsm.rs:247-307).
keep(x) = Scope.holds(x) (per-crate row: and crate(x) = that crate)
u(x) = module_path(x) or crate(x)
units U = { u(f) : f fn, keep(f) }
∪ { u(t) : t ∈ calls(f), keep(f), keep(t) }
∪ { u(f'), u(t) : USES_TYPE / READS_FIELD x → t, f' = enclosing_fn(x), keep(f'), keep(t) }
sorted by name; N = |U|
A[i][j] = 1 iff i ≠ j and some kept fn of unit i has a CALLS edge to a kept node of unit j,
or a USES_TYPE / READS_FIELD edge (from it or its children) to a kept node of unit j
A is boolean. The code counts the edges per unit pair (raw, :254), but
the counts are never used: only the key set enters the matrix (:134-139).
Unlike misplaced, dsm does not check that a USES_TYPE target is a non-fn
node (:288). Units are only modules that hold a kept fn or are the target of
a kept edge; a module of types that nothing uses is not a unit.
2. Visibility by repeated squaring (grb2::closure, crates/mimir-lens/src/lenses/grb2.rs:72-82).
V₀ = A
Vₖ₊₁ = Vₖ ∨ Vₖ·Vₖ (Vₖ·Vₖ on GxB_ANY_PAIR_BOOL, ∨ = GrB_LOR eWiseAdd)
stop: return Vₖ₊₁ as soon as nnz(Vₖ₊₁) = nnz(Vₖ)
Vₖ holds every path of length 1 … 2ᵏ, so V = A ∨ A² ∨ … ∨ A^N, the
transitive closure (without I, but with V[i][i] = 1 for every i on a
cycle). ANY.PAIR on bool is the structural OR-AND (LOR.LAND) product: an entry
exists when any witness k has V[i][k] and V[k][j]. The last round always
adds nothing and only confirms the stop, so the closure takes up to
⌈log₂ N⌉ + 1 products. The module doc says "⌈log₂ n⌉ rounds at most"
(grb2.rs:71); that counts the growing rounds only.
3. Propagation cost and fan (crates/mimir-lens/src/lenses/dsm.rs:141-156).
R = edges_of(V) ∪ { (i, i) : i ∈ U } (V ∨ I, built in a Rust BTreeSet)
visible = |R| = nnz(V ∨ I)
PC = visible / N²
VFO(i) = |{ j : (i, j) ∈ R }| (row sum: units i can reach, itself included)
VFI(j) = |{ i : (i, j) ∈ R }| (column sum: units that reach j, itself included)
I is included, as in MacCormack et al. 2006, so PC ≥ 1/N and a unit with
no dependency has VFI = VFO = 1. The module header's bullet says "VFI(i) =
units that reach i (column sum of V)" (dsm.rs:13); the code and the "math
in full" paragraph (:36-37) use V ∨ I. The text output prints visible
twice, once as "visibility … (x %)" and once as "PROPAGATION COST x %"; both
percentages are PC (:311-317).
4. Cyclic groups and the core (:157-176).
SCC = order::sccs (Tarjan) on the directed graph of A (not GraphBLAS)
groups = { S ∈ SCC : |S| ≥ 2 }, each sorted, ordered by |S| descending, then by the
sorted member index list (= name order) ascending
core = groups[0] (empty when there is no group)
core_share = |core| / N
second = |groups[1]| (0 when there is none)
On a tie for the largest group, the core is the one whose sorted member list comes first, and the 1.5× test then fails, so the system is multi-core.
5. Classes (:186-218). Every member of one SCC reaches the same set and
is reached by the same set, so the code's min over the core is the common
value:
VFI_c = min_{i ∈ core} VFI(i), VFO_c = min_{i ∈ core} VFO(i)
class(i) = core if i ∈ core
= periphery if core = ∅
= core-like if VFI(i) ≥ VFI_c ∧ VFO(i) ≥ VFO_c
= shared if VFI(i) ≥ VFI_c ∧ VFO(i) < VFO_c (used by the core, uses little)
= control if VFI(i) < VFI_c ∧ VFO(i) ≥ VFO_c (uses the core, little uses it)
= periphery otherwise (both below)
These match the module doc. Rows are sorted by unit name.
6. Architecture (:177-185), tested in this order:
hierarchical if core = ∅ or core_share < core_floor (0.04)
borderline else if core_share < core_min (0.05)
core-periphery else if second = 0 or |core| ≥ dominant · second (1.5)
multi-core otherwise
So borderline is the band 4 % ≤ core_share < 5 %. This matches the module
doc and the plan. The thresholds are the knobs named above; the plan says they
are the 2014 paper's values as remembered.
7. Per-crate rows (:416-439). For every crate the dump defines
(g.crates(), defined_here nodes, mod.rs:471-478), the same analysis runs
with keep(x) = Scope.holds(x) ∧ crate(x) = that crate: only the crate's own
modules, and edges that leave the crate are dropped. A crate with fewer than
min_units modules gets no row. Rows are sorted by PC descending, then
name. Each crate row re-walks every fn and every dump edge, so the cost grows
with (crates × dump size).
GraphBLAS steps
| step | operation | semiring / op as named in code | computes | file:line |
|---|---|---|---|---|
| 1 | A (N×N bool) from unit pairs | Matrix::<bool>::from_edges (pattern, duplicates collapse) | the DSM | crates/mimir-lens/src/lenses/dsm.rs:134-139, crates/mimir-lens/src/lenses/grb.rs:21-30 |
| 2 | S = V·V | GrB_mxm with GxB_ANY_PAIR_BOOL, no descriptor (structural OR-AND) | paths of twice the length | crates/mimir-lens/src/lenses/grb2.rs:75, crates/mimir-lens/src/lenses/grb.rs:33-54 |
| 3 | V' = V ∨ S | GrB_Matrix_eWiseAdd_BinaryOp with GrB_LOR | the next closure step | crates/mimir-lens/src/lenses/grb2.rs:76, crates/mimir-lens/src/lenses/grb.rs:139-141, nornir crates/mimir-graph/src/ops.rs:624-641 |
| 4 | stop test | GrB_Matrix_nvals on V' and V | fixpoint | crates/mimir-lens/src/lenses/grb2.rs:77 |
| 5 | V ∨ I, PC, VFI, VFO | not GraphBLAS (tuples into a BTreeSet, counts) | the cost and the fans | crates/mimir-lens/src/lenses/dsm.rs:141-156, crates/mimir-lens/src/lenses/grb.rs:158 |
| 6 | cyclic groups | not GraphBLAS (order::sccs) | SCCs of A | crates/mimir-lens/src/lenses/dsm.rs:158-172 |
| 7 | core, classes, architecture | not GraphBLAS | the rows | crates/mimir-lens/src/lenses/dsm.rs:173-218 |
Dataflow
Mímir v2 dump units matrices / products output
───────────── ───── ─────────────────── ──────
CALLS (dedup, plain) ──────────┐
USES_TYPE / READS_FIELD ───────┼─► (u(f), u(t)), u(f) ≠ u(t) ─► A N×N bool ─┬─► V ← V ∨ V·V (ANY.PAIR, LOR)
+ enclosing_fn │ (counts dropped) │ until nnz stable
CONTAINS chain ─► module_path ─┤ │ │
crate column ─► (--unit crate)─┘ │ ▼
Scope (prod, scoped crates) │ V ∨ I ─► PC = nnz/N²
│ ├─► VFO (rows), VFI (cols)
└─► tarjan_scc(A) ─► groups |S| ≥ 2
core = largest, second
│
classes (core, core-like, shared, control,
periphery) + architecture (hierarchical,
borderline, core-periphery, multi-core)
│
per crate: keep ∧ crate(x) = c, ≥ min_units modules ─► same pipeline ─► CRATE rows sorted by PC desc
▼
summary {unit, units, direct_deps, propagation_cost, core_units,
core_share, architecture, cyclic_groups, crates_with_rows}
findings {kind, name, units, direct, visible, propagation_cost,
cyclic_groups [sizes], core, core_share, architecture,
rows [{unit, vfi, vfo, class}]}
Output
Summary keys (:404-433): unit, units (N), direct_deps (nnz of A),
propagation_cost (a fraction), core_units, core_share, architecture,
cyclic_groups (the number of groups), and crates_with_rows when the unit
is module. The headline is N modules: propagation cost x %, core c (y %), architecture … (:395-403).
The first finding is the WORKSPACE row, then one CRATE row per crate
(:412-439). The text (describe, :309-360) gives units, direct deps,
visibility, PC, number of cyclic groups, core size and share, architecture,
the counts of shared / control / periphery (and core-like when non-zero), the
first 12 core members and the first 6 members of the second group. The JSON
(json_of, :362-376) carries cyclic_groups as a list of sizes and every
unit's {unit, vfi, vfo, class}.
Worked example 1: the fixture, by hand
Fixture gate_fixture (crates/mimir-lens/tests/graph_lenses_study.rs:229-261), crate app.
The fns http_handle, router, gate_ingest, mirror_run, fs_write,
lone, Store#put and the impl method put sit directly under the crate, so
their unit is app. m1::f1 and m2::f2 sit in modules m1 and m2 and
call each other. All other calls stay inside app. There are no USES_TYPE
edges.
U = [app, app::m1, app::m2] (sorted; indices 0, 1, 2), N = 3
A = { (1,2), (2,1) } direct_deps = 2
V₀ = A nnz 2
V₀·V₀ = { (1,1), (2,2) }
V₁ = V₀ ∨ V₀·V₀ = { (1,1), (1,2), (2,1), (2,2) } nnz 4 (grew)
V₁·V₁ = { (1,1), (1,2), (2,1), (2,2) }
V₂ = V₁ nnz 4 (stop; 2 products)
V ∨ I = V₁ ∪ { (0,0) } visible = 5
PC = 5 / 3² = 5/9 = 55.6 %
VFO: app 1, app::m1 2, app::m2 2 VFI: app 1, app::m1 2, app::m2 2
groups = [ {app::m1, app::m2} ] → core, core_share = 2/3 = 66.7 %, second = 0
architecture: 0.667 ≥ 0.04, 0.667 ≥ 0.05, second = 0 → core-periphery
VFI_c = VFO_c = 2; app: VFI 1 < 2, VFO 1 < 2 → periphery
The module doc's worked example says "V = A (already closed)"
(dsm.rs:51). As coded, V = A ∨ A² also holds the diagonal entries of m1
and m2 (each reaches itself through the cycle), so V has 4 entries, not 2.
V ∨ I has 5 entries either way, and PC = 5/9 is right.
The test asserts the core ["app::m1", "app::m2"], core-periphery,
units = 3 and PC = 5/9; with min_units = 5 there is no CRATE row (app
has 3 modules), and with min_units = 2 there is one
(crates/mimir-lens/tests/graph_lenses_study.rs:489-530).
Worked example 2: a four-unit DSM with every class (constructed)
The fixture has no shared or control unit. This hand-made example is not a
repo fixture; it only shows the class rules. Units c, s, x, y (indices 0–3),
A = { c→x, x→y, y→x, y→s }.
V₀ = {(0,2), (2,3), (3,2), (3,1)} nnz 4
V₁ = V₀ ∪ V₀² = V₀ ∪ {(0,3), (2,2), (2,1), (3,3)} nnz 8
V₂ = V₁ ∪ V₁² = V₁ ∪ {(0,1)} nnz 9
V₃ = V₂ stop (3 products)
V ∨ I adds (0,0), (1,1) visible = 11
PC = 11/16 = 68.75 %
VFO VFI
c 4 1 c reaches c, x, y, s; only c reaches c
s 1 4 s reaches only s; c, x, y, s reach s
x 3 3 core
y 3 3 core
core = {x, y}, VFI_c = VFO_c = 3, core_share = 2/4 = 50 %, second = 0 → core-periphery
c: VFI 1 < 3, VFO 4 ≥ 3 → control s: VFI 4 ≥ 3, VFO 1 < 3 → shared
Worked example 3: Holger (plan §F.2)
Holger h32/next 9dbe5cbf, prod code, modules:
| system | units | PC | cyclic groups | core | architecture | classes |
|---|---|---|---|---|---|---|
| workspace | 363 | 28.9 % | 12 | 97 (26.7 %) | core-periphery | — |
holger-server-lib | 171 | 58.7 % | — | 97 (56.7 %) | core-periphery | shared 58, control 12, periphery 4 |
holger-index-rows | 8 | 78 % | — | 7 of 8 | — | root ↔ 6 format modules |
holger-traits | — | 38.9 % | — | — | — | root ↔ backend |
For holger-server-lib the plan records nnz(V ∨ I) = 17,159 (plan §F.3):
17,159 / 171² = 17,159 / 29,241 = 0.587. The core share is
97 / 171 = 0.567, and 97 + 58 + 12 + 4 = 171, so no unit is core-like. The
workspace core share is 97 / 363 = 0.267. The crates cli, provenance, sync,
plugin-host, update-bundle, intel and job-runner are hierarchical (no cycle).
For scale: gunnar's gunnar-server has 87 modules, PC 56 %, core 47 (54 %);
the dwarves workspace has PC 6.6 % and is multi-core (plan §F.2). The cycles
lens counts 90 modules in the server-lib SCC with calls only; dsm adds
USES_TYPE / READS_FIELD and finds a core of 97.
Cost
Plan §F.1 (the lens-study rows stand in the plan's §G table), build-1, debug build, wall time / peak cgroup memory, dump read included:
| dump | dsm | graph read (lens cycles) |
|---|---|---|
| Holger (363 modules) | 115 s / 3.7 GB | 77 s / 5.2 GB |
| gunnar (432 modules) | 112 s / 4.4 GB | 94 s / 6.4 GB |
| dwarves (232 modules) | 48 s / 2.1 GB | 43 s / 3.0 GB |
The closure on a few hundred units is small. The extra time over the read on Holger (about 38 s) fits the per-crate rows, each of which re-walks all fns and dump edges (Math §7); this is an inference from the code, not a measured split. The plan's F-4 row estimated "< 1 s".
Limits
Ais binary: one call and a thousand calls between two modules weigh the same. That is the published method.- The lens cannot say which edge to cut to shrink the core; that needs the intended layering.
- Calls through
dyn Trait, closures and fn pointers end at the trait method, and overloaded operators have no CALLS edge (section 3.2), so a trait seam lowers PC on purpose. - Per-crate rows drop the edges that leave the crate, so a crate's PC is internal only.
- The thresholds 5 %, 1.5× and 4 % are knobs from the paper as remembered.
- Units are modules with fns or with used types; an empty or unused module is
not counted in
N, which raises PC slightly.
needs
"CALLS edges" when the dump has none (:380-385); "USES_TYPE / READS_FIELD
edges (type coupling between units)" when both are missing, and then the DSM
holds calls only (:386-391). Both name cargo mimir-dump.
Tests
crates/mimir-lens/tests/graph_lenses_study.rs:489dsm_finds_the_module_cycle_as_the_core_and_prices_the_propagation(fixture:229-261).crates/mimir-lens/tests/graph_lenses_study.rs:680the_study_lenses_are_listed.- No unit tests in
crates/mimir-lens/src/lenses/dsm.rsand no Holger golden; the Holger numbers above are plan §F.2 only.
How it relates to the other lenses
cycles (4.4) and dsm compute the same kind of object, SCCs of a module
graph with Tarjan (order::sccs), but on different edges: cycles uses plain
CALLS (and DEPENDS_ON at weight 0 for the crate graph), dsm adds USES_TYPE /
READS_FIELD and has no DEPENDS_ON. So dsm's core can be larger than the
largest cycles knot (97 vs 90 modules in holger-server-lib). cycles
names the knot; dsm puts a cost on it (PC) and classifies the rest of the
system by how it relates to the core. misplaced (4.11) reads the same code
as undirected clusters and suggests where the boundaries could go; a smaller
core after a MOVE round is the effect the 2014 paper links to a more modular
design. dispatch-erosion (4.3) finds format knowledge outside its home; in
dsm that shows only as extra dependency edges into the enum's module (a
higher VFI there), not as a named row. verdict (section 5) does not fold
dsm: it runs twins, size, dispatch-erosion, cycles and
gate-bypass only (crates/mimir-lens/src/lenses/verdict.rs:258-340), and its "not folded"
note does not name this lens (crates/mimir-lens/src/lenses/verdict.rs:19-22).
4.13 runtime-guard-erosion (plan H6)
The question. gate-bypass says whether every entry → sink path passes the
choke. This lens asks the review's sharper question about the paths that DO
pass something: is the barrier an invariant (a type only the gate can make, a
check whose verdict is read) or a setting (an if self.flag around the
write)? The review's §7(a)/(c) #1 — "the gate is a configuration toggle" — is
exactly the case the fn-level graph cannot see, because the conditional edges
live inside the MIR blocks.
Run. cargo lens runtime-guard-erosion --sink PAT… [--entry PAT…] [--gate-type PAT…] [--toggle-min N] (door: dwarves RepoLens::RuntimeGuardErosion).
--sink is required and expands to the trait family through IMPLEMENTS as
gate-bypass does (test impls left out). --entry keeps only the doors an entry
reaches (reverse BFS over callers and dyn sources). --gate-type names the
gate's value type (Admitted): a sink call that takes it is TYPED.
--toggle-min (default 3) is the flag-only door count from which a sink is a
CONFIG-TOGGLE finding.
Inputs. mir_block nodes (the ty payload: terminator, statements, the
block's local:type list), CFG_SUCC, CALLS (the doors = direct callers of
a sink in scope), IMPLEMENTS (the sink family). Nothing else.
Dataflow.
sinks (patterns + the trait family) → doors = in-scope fns that call a sink
door fn D → its CFG (mir_block nodes + CFG_SUCC) → post-dominators (reverse CFG + one exit E)
→ the sink call block S → the switch blocks S is control-dependent on (CD)
+ S in a nested body (closure, spawned block): CD of the block that MAKES the body
(`_k = closure(..)`, matched by position), up through every enclosing body
+ the switches further up (CD*) when they read configuration (FLAG, or a VERDICT
whose check reads a flag)
→ each switch operand traced to its definition inside its body (≤ 8 hops; `x?` =
`Try::branch(x)` is looked through to the call that made x):
a field place of _1 (`_1*.3`, `_1.3`, through discr(…) / deref_copy) → FLAG (field, type)
a dependency's predicate / accessor on such a field
(`Option::is_none(&self.f)`, `OnceLock::get(&self.f)`) → FLAG (field, via callee)
the destination of a `call` terminator → VERDICT (callee, type,
+ the flags the callee's own switches read, ≤ 2 own-code calls down)
anything else (a parameter, a constant, arithmetic) → OTHER (operand)
→ the sink call's argument types against --gate-type → TYPE
door class = TYPED ▷ VERDICT ▷ FLAG-ONLY ▷ UNGUARDED (no switch, or OTHER switches only)
sink finding: flag-only doors ≥ toggle_min → CONFIG-TOGGLE
The math. Over D's CFG with a virtual exit E (every return-like block →
E; return, unreachable, resume, terminate, coroutine_drop, or no
successor), the post-dominator tree is the dominator tree of the reversed CFG
rooted at E (lenses/order.rs immediate_dominators, Cooper–Harvey–Kennedy —
the one dominator engine of the crate, no second graph library). The sink block
S is control-dependent on a switch block B (Ferrante–Ottenstein–Warren) iff
∃ x ∈ succ(B): S ∈ pdom(x) ∪ {x} — taking that arm commits to S
and S ∉ pdom(B) \ {B} — B itself could still avoid S
pdom(x) = the ancestors of x in the post-dominator tree. Every such B is a
barrier of the write; its kind is what the switch reads.
Round 2 (lane LENS-POLISH, 2026-11-05, re-compare 2's wrong results 1 and 8):
CD(S) = { B : S control-dependent on B } (as above)
CD*(S) = CD(S) ∪ ⋃_{B ∈ CD(S)} CD*(B) (iterated: every condition S runs under)
make(β) = the block holding `_k = closure(..)` / `coroutine(..)` with the greatest
position ≤ the first position of body β, outside β
up(S) = S, make(body(S)), make(body(make(body(S)))), … (≤ 8, to the top body)
barriers = { classify(B) : B ∈ ⋃_{u ∈ up(S)} CD(u) }
∪ { classify(B) : B ∈ ⋃_{u ∈ up(S)} CD*(u), classify(B) is configuration }
configuration(b) ⇔ b = FLAG ∨ (b = VERDICT ∧ flags(b) ≠ ∅)
flags(verdict of own fn C, depth d) = { FLAG of C's switches } ∪ ⋃ flags(C's own verdicts, d − 1), d ≤ 2
A loop's next() or an earlier lookup up the CD* chain is no barrier of the write
(taking every CD* switch made 17 of Holger's 21 calls VERDICT, on iterator steps);
configuration up there is. A flag inside a check is reported on the VERDICT
(verdict of raw_write_refusal: …, which reads flag _1*.30 via OnceLock::get in write_refusal), so the class order TYPED ▷ VERDICT ▷ FLAG-ONLY ▷ UNGUARDED is kept and
verdict_reads_flag counts the doors whose check is a setting. The grammar is
name-free (mimir_schema::mir): a place is _<local> with projections *
(deref), .N (field N), ~V (variant); a field of self is therefore a field
projection rooted at _1 (MIR local 1 = the first parameter). A verdict is a
local a call terminator writes (call … -> _k). The lens follows the switch
operand for at most four hops through the reads a place goes through in real
MIR — use(…) (an operand read: _3 = use(_1*.2)), copy / move, ref /
refmut and a deref'd local (if let Some(_) = &self.cfg is _4 = ref(_1*.3); _5 = discr(_4*)), discr(…), deref_copy(…) — and only inside the switch's
own body: a nested body (a closure, an async fn's coroutine) numbers its locals
on its own, and there _1 is the environment, so _1.K* is a captured
variable (OTHER) and only _1.K*.N (a field of what capture K points at, self
for an async method) is a FLAG. Until 2026-11-05 the walk followed only
discr(…) / deref_copy(…) across all bodies: it could never see a real-MIR
FLAG (use(…)), and one Holger door was read as a VERDICT through a call of
another body that wrote the same local number.
Worked example (the fixture, tests/graph_lenses.rs). Sink Vault#write()
(trait + impl Vault for Mem). flag_a/b/c: _3 = _1*.2; switch(^_3; 0, 1)
around the write → FLAG-ONLY (flag _1*.2: bool), three of them → the sink is
a CONFIG-TOGGLE. verdict_door: _2 = check(_1); _3 = discr(_2); switch(^_3)
→ VERDICT (verdict of app check().: Result<(),E>). typed_door: _2: Admitted = check(_1); write(^_2) → TYPED with --gate-type Admitted, UNGUARDED without
(no switch controls the write; the type does). open_door: the write alone →
UNGUARDED. Doors: 6; flag-only 3, verdict 1, typed 1, unguarded 1; toggles 1.
Round-2 fixture (runtime_guard_erosion_sees_closures_flags_one_call_down_and_option_predicates,
sink safe_put): closure_door = check()? in the door, the write inside a closure
handed to spawn_blocking → VERDICT (in a closure passed to tokio … spawn_blocking;
was UNGUARDED); deep_door = if let Some(e) = side_door_check(), which does
self.side_doors.get()? → VERDICT that reads flag _1*.4 via OnceLock::get;
none_door = if self.scanner.is_none() → FLAG-ONLY (was VERDICT of is_none);
nested_door = if self.raw_ok { if req.raw { write } } → FLAG-ONLY through CD*.
On the lossless graph (schema 8, lens upgrade batch 2). A primitive compare of
a field of self with a constant (if self.mode == 2 { write }, MIR
_3 = Eq(_1*.0, k:u8)) is a FLAG, and the row names the value:
FLAG(sw) ⇐ operand(sw) ⇝ _k, _k = op(_1*.f, k) ∨ op(k, _1*.f), op ∈ { Eq, Ne, Lt, Le, Gt, Ge }
via = "<op> <value>" value = k_lo of the constant operand (schema 8), else the constant's type
Not(…) is followed like a read (if !self.open). The derived-text walk alone
knew PartialEq::eq calls (an enum compare) but not the primitive BinOp, so
such a door read UNGUARDED. Fixture
guard_erosion_reads_a_primitive_compare_of_a_self_field_as_a_flag_with_its_value.
What it cannot say. A flag behind a non-trivial helper is reported on the helper's
VERDICT (which reads flag …), up to two own-code calls down; deeper it is not seen. A door without MIR (--mir-blocks cap, a trait method) is
counted in doors_without_mir and named in a needs row. Which flag value the
deployment runs with is a configuration fact, not a graph fact.
Golden (Holger h32/next 9dbe5cbf, schema 6, the gate's five sinks and nine
entries, no gate type). See tests/golden/holger-h32-9dbe5cbf.json
runtime-guard-erosion. Round 2: 17 doors, 21 sink calls, 0 TYPED, 14 VERDICT
(was 5), 7 UNGUARDED (was 16), 0 FLAG-ONLY, 24 controlling switches (was 11),
verdict_reads_flag 7, in_closure 11, 0 toggles. The gRPC put_artifact raw
write in spawn_blocking (services.rs:894) is now VERDICT of
IntakeChain::raw_write_refusal, which reads flag _1*.30 (side_doors) via
OnceLock::get in write_refusal: the review's toggle (hole 1). With the sink
AuditLog#record() from advance_inner, if self.scanner.is_none() is read as
flag _1*.12 via Option::is_none (re-compare 2 wrong result 8). Before round 2:
17 doors, 21 sink calls, 0 TYPED, 5 VERDICT, 0
FLAG-ONLY, 16 UNGUARDED, 11 controlling switches, 0 toggles (before the read /
body fix: 6 VERDICT, 15 UNGUARDED — handle → put's barrier is a captured
variable of the coroutine, _1.0*: Option<String>, not a check). With the
reads followed, no write on these paths is guarded by a flag of self at
this commit. The fixture pins both real-MIR spellings (flag_use,
flag_ref) and both coroutine cases (async_self FLAG, async_capture
UNGUARDED).
4.14 taxonomy-drift (plan H9)
The question. Which enums name the same things twice, and what keeps them in
step? dispatch-erosion sees the symptom (ladders over a foreign enum, PARALLEL
pairs by variant names); this lens adds the review's second signal, the
translation fn (Format::of, IntakeVerb → Action), tells a translation that
renames one taxonomy (an EQUIVALENCE) from one that derives a coarser property
(a PROJECTION: Job → JobWrites), groups the enums into taxonomy families,
reads the alias pairs off the equivalences (Kind::D ↔ Kind2::E: the names
differ, the mapping says they are one), and names a family a TAXONOMY-DRIFT
when it holds a translation with more than min_arms arms — one canonical
taxonomy / registry instead of N enums and the fns between them.
Run. cargo lens taxonomy-drift [--min-variants N] [--min-arms N] [--jaccard F] (door: dwarves RepoLens::TaxonomyDrift); defaults 3 / 5 / 0.5.
No pattern is needed: every enum the dump defines in scope is a candidate.
Inputs. enum nodes and their variant children (CONTAINS; their
start_line gives the declaration order), mir_block nodes (the switch
terminator with its discr(…) statement, the adt~V(…) aggregate rvalue, the
block's local:type list), CFG_SUCC (the arms in target order), USES_TYPE
(which enum a bare type name in the MIR means).
Dataflow.
enums (defined here, in scope, ≥ min_variants) → variants sorted by start_line
fn with MIR → block b: switch(^_k; v0,…,v_{k-1}) with _k = discr(_p), _p: E1 (plain, not Option<E1>)
→ succ(b) in target order (the otherwise arm last)
→ arm a_i and its single-successor chain (≤ 3 blocks: false_edge, goto)
→ the first `_x = adt~V(…)` with _x: E2 (plain), E2 ≠ E1 → the arm builds E2::V
→ E2 = the enum most arms build; ≥ 2 building arms = a translation fn t: E1 → E2
→ positions trusted iff |succ(b)| = k + 1 (one successor per value) and both enums'
variants have distinct lines: value v_i ↦ E1 variant v_i, the otherwise arm ↦ the one
variant no value names
bare type names: USES_TYPE of the fn decides (matched by id: an edge end can be another
crate's reference stub); a used same-named type that is no local enum = not ours;
no USES_TYPE hit → the enum of that name in the fn's crate whose module shares the
longest prefix with the fn's module; a tie → skipped (ambiguous_types)
classify t: EQUIVALENCE | UNPROVEN (collapsed switch) | PROJECTION
parallel pairs P: J ≥ jaccard ∧ shared ≥ 3; equivalence edges Q
families = connected components over P ∪ Q (union-find)
aliases = the (E1 variant, E2 variant) pairs of an equivalence whose names differ
TAXONOMY-DRIFT = a family holding a translation of arms > min_arms
The math. Variant sets V1, V2; Jaccard J(E1,E2) = |V1 ∩ V2| / |V1 ∪ V2|.
A translation t: E1 → E2 reads a partial map μ_t : V1 ⇀ V2 off its arms
(μ_t(v_i) = V_i when arm i builds E2::V_i). With n_b = the number of
building arms and B = the set of variants they build:
injective(t) ⇔ |succ(b)| = k + 1 ∧ |B| = n_b ∧ every building arm has a known source
EQUIVALENCE ⇔ injective(t) ∧ n_b ≥ 3 ∧ n_b ≥ 0.8 · min(|V1|, |V2|)
UNPROVEN ⇔ |succ(b)| < k + 1 ∧ |B| = n_b ∧ n_b ≥ 3 ∧ n_b ≥ 0.8 · min(|V1|, |V2|)
PROJECTION ⇔ otherwise
P = { (E1,E2) : J ≥ j₀ ∧ |V1 ∩ V2| ≥ 3 }, Q = { (E1,E2) : ∃ t equivalence }
families = the connected components of (Enums, P ∪ Q) with ≥ 2 members
Why UNPROVEN exists: the driver drops a repeated adjacent switch target
(mimir-driver walk/mir.rs, succ.dedup()), so on a switch with fewer
successors than values one arm block may serve several values (Scanned | PolicyChecked => Quarantine), and "distinct blocks build distinct variants"
no longer proves injectivity. Such a translation is named, takes no family
edge and gives no aliases; a needs row names the missing fact (the target
list per value). The min-coverage is against the smaller enum: a wider enum
may map its extra variants to None (ArtifactFormat has 15 variants,
Format 11).
No GraphBLAS step: the enum graph has a few hundred nodes, union-find with path halving is the whole of the component step (semiring: none).
Worked example (the fixture, tests/graph_lenses.rs). Crate fmt: Kind {A,B,C,D} (written to the dump as D, A, B, C — the lens sorts by line),
Kind2 {A,B,C,E} (J = 3/5 = 0.6, 3 shared: a parallel pair), Level {Lo, Mid, Hi}. to_kind2: switch(^_2;0,1,2) over discr(_1: Kind), four successors,
arms _0 = adt~0..3() of type Kind2 → injective, n_b = 4 ≥ 0.8·4 →
EQUIVALENCE, 1:1. to_kind: the reverse, each arm reached through a
false_edge block → EQUIVALENCE. level_of: A, B ↦ Lo, C ↦ Mid, D ↦ Hi
(|B| = 3 < n_b = 4) → PROJECTION, so Level joins no family. squash: three
successors for four targets (collapsed), distinct variants built → UNPROVEN,
plus the needs per-value switch targets row. Result: 1 family Kind ≈ Kind2,
4 translations (2 equivalences, 1 unproven, 1 projection), 1 alias Kind::D ↔ Kind2::E (read both ways, one pair), 0 drift at min_arms 5 (4 arms), 1 at
--min-arms 3.
On Holger h32/next 9dbe5cbf (golden taxonomy-drift). 195 enums, 36
parallel pairs, 16 families, 19 translations: 4 equivalences, 3 UNPROVEN, 12
projections; 0 drift families, 0 aliases; 5 collapsed switches, 29 ambiguous
type sites skipped. The UNPROVEN three are the review's case and its two
neighbours: Format::of (ArtifactFormat → Format, 16 arms; in the source an
injective alias table — Rust ↦ Cargo, Pip ↦ Pypi, Docker ↦ Oci — with
Nuget | Gem | Conda | Composer | Znippy ↦ None), ArtifactState::repository
(11 arms) and intake_slot_kind (ScannerKind → EvidenceKind, 9 arms); the
source shows the last two are many-to-one, which is exactly why a collapsed
switch must not be read as 1:1. The families include the review's Verb ≡ IntakeVerb ≡ IntakeVerb ≡ Action (two 1:1 translations), RemoteKind ≈ SourceKind (0.67) and the credential family (ConnectorAuth ×2 ≈
UpstreamAuth ≈ Credential). Hand check (10 rows against the source): 7 true
(the three UNPROVEN, the Verb family, RemoteKind ≈ SourceKind, the
credential family, ProvenanceGate ≈ Mode), 3 weak: the cli/server copies of
the generated proto enums TokenScope, MeterScope (real duplicates, but
prost-generated in two crates) and holger-agent-cli Source ≈ Target (the
same formats read and written: a duality, not drift). Two false
positives of earlier drafts are gone by construction: ClassWord::of maps
edda's admit::Class, not export_profile's Class (USES_TYPE), and the license
aliases read off alphabetical variant rows (declaration lines). Bench (build-1,
cached graph, release door): Holger 0.9 s / 2.4 GB, dwarves 0.4 s / 1.1 GB,
gunnar 0.9 s / 2.5 GB — the held graph is the RSS; the lens itself is a few ms.
What it cannot say. A translation written as a lookup table or a chain of
if let is not one switch and is missed; a match that builds the target
through a helper (Kind2::from_str(name)) builds no aggregate. Explicit
discriminants map by position only when they run 0..n. Collapsed switches
(above) need the driver's per-value target list.
Round 2 (lane LENS-POLISH, 2026-11-05; re-compare 2 wrong results 4 and 5).
enums = defined, not derived, not test, NOT GENERATED (file under generated/, OUT_DIR, build/*/out)
side(E1,E2) ⇔ ∃ item X ∈ {struct, union, variant, fn} : two fields / params of X have types E1 and E2
DUALITY = parallel pair (J ≥ 0.5, ≥ 3 shared) with side(E1, E2): one vocabulary in two roles
parallel' = parallel \ DUALITY (families are built over parallel' ∪ equivalences)
The proto enums Holger include!s (holger-cli pb/* ≈ server-lib grpc/holger_proto/*):
the driver files each item under the generated file (holger-cli/src/generated/ holger.token.v1.rs:82), but the enum filter read only the crate scope, so they were
families. agent-cli Source ≈ Target (J = 1.0) are the two fields of one struct and now
read DUALITY. Holger golden: enums 196 → 186, parallel pairs 36 → 26, families 16 → 11,
equivalences 4 → 2, translation fns 19 → 17, dualities 1. Still PARALLEL (no side-by-side
use in the graph): apk UpstreamIndex ≈ Package, UnknownLicense ≈ LicenseDecision.
Fixture taxonomy_drift_tells_a_duality_from_a_copy_and_leaves_generated_enums_out.
dispatch-erosion keeps its names-only parallel count (fixture 1 → 3 with the new enums).
On the lossless graph (schema 8, lens upgrade batch 1). The switch keeps every
(value, target) pair (tg_role, tg_lo), and an aggregate names its full type
(st_ty), so the translation map is read exactly:
T(f: E1 → E2) = { (v, w) : switch(discr(x: E1)) arm with value v runs (≤ 3 blocks, one successor each) into `_ = adt~w(…): E2` }
the otherwise arm → the one variant no value names, when there is one
one_to_one ⇔ |T| = |arms building E2| ∧ the values are distinct ∧ the variants are distinct
EQUIVALENCE ⇔ one_to_one ∧ |T| ≥ 3 ∧ |T| ≥ 0.8 · min(|V1|, |V2|); else PROJECTION
The schema-7 view deduped a switch's repeated targets, so A | B => X could
not be told from a 1:1 map: those were UNPROVEN. On the lossless rows an
UNPROVEN switch is a many-to-one map by construction, a PROJECTION. E1 and
E2 are found by the full type (local_ty, st_ty), not by the bare name.
Summary key translations_from_graph; the PROJECTION row carries map.
Fixture taxonomy_drift_proves_the_translation_map_from_the_switch_values.
On Holger h32 9dbe5cbf (golden re-pinned): translation fns 17 → 19 (16 from the
rows), equivalences 2 → 4, projections 12 → 14, UNPROVEN 3 → 1, alias pairs
0 → 6, families 8 → 9, 1 TAXONOMY-DRIFT family. Checked by hand:
promote::identity::Format::of (identity.rs:180) maps 10 ArtifactFormat
variants 1:1 (Rust ↦ Cargo, Pip ↦ Pypi, Maven3 ↦ Maven, Go ↦ Golang,
Docker ↦ Oci, Raw ↦ Generic, 4 same-named) and 5 to None: the proven
EQUIVALENCE, its 6 renamed pairs the ALIAS rows, and with 16 arms the family
Format ≈ ArtifactFormat is the TAXONOMY-DRIFT the review's R2 names. Lossy it
was UNPROVEN. scanner::intake_slot_kind (mod.rs:1110) is many-to-one
(Malware | Inspection ↦ Malware, Secrets | Sbom | Model ↦ Other): a proven
PROJECTION. promote::airgap::judge_evidence (ScanVerdict → AirgapError,
6 arms) is new: its enums are found by the full type. The one UNPROVEN left is
a switch whose enum the full type does not place (the lossy reading stays for
it). Because Format and ArtifactFormat are one family now, concept-scatter
and arch-report count one concept less (713 → 712) and one family less
(8 → 7): re-pinned with this cause. On Gunnar c17a6ef4: translation fns 20 → 21,
UNPROVEN 1 → 0 (OpClass::lane is a proven PROJECTION), no golden row moved.
Round 3 (lane LENS-FIX4, 2026-11-06). Two more DUALITY signals, for two enums of one module only, and never for a pair a translation fn connects:
role_split(E₁,E₂) ⇔ (E₁ is only a field type ∧ E₂ only a return type) ∨ the reverse
(field = a struct / union field; types peeled through Option / Result)
payload_split(E₁,E₂) ⇔ ∃ v ∈ V₁ ∩ V₂: fields(E₁::v) ≠ fields(E₂::v)
DUALITY ⇔ side_by_side ∨ (module(E₁) = module(E₂) ∧ ¬translated(E₁,E₂)
∧ (role_split ∨ payload_split))
Two enums kept side by side in one file are two roles by design; the same
vocabulary across modules or crates is drift whatever its roles (the credential
family, Gunnar's Scope ≈ Access ≈ ReplicationScope). A translation fn turning
one into the other (policy::effective: ProvenanceGate → Mode) says the code
treats them as one taxonomy. Fixture
taxonomy_drift_reads_role_split_and_payload_split_pairs_as_dualities: Policy
(field only) ≈ Decision (return only) and Index ≈ Pkg (Verified(Vec<u8>, Idx) vs
Verified(Vec<u8>)) are DUALITY; Left ≈ Right (both fields) and other::Verdict
(another module) stay PARALLEL. Holger h32: families 11 → 8, dualities 1 → 5 —
apk::UpstreamIndex ≈ Package (payload), creds::CredentialRef ≈ Credential,
mirror_cycle::MirrorCredential ≈ LoadedCredential, replication::Prepared ≈ EstatePrepared (a declared or reference form against a loaded one). Gunnar
c17a6ef4: families 18 → 13; the six same-file CLI command pairs (TokenCommand ≈ KeyCommand, user vs Control* commands) are DUALITY, the cross-crate families
stay. UnknownLicense ≈ LicenseDecision stays PARALLEL on purpose: both are
record fields with the same unit variants, and UnknownLicense::decision()
translates one into the other — one taxonomy twice.
4.15 dyn-to-enum (plan H1)
The question. Should this trait be an enum, or this enum a trait? Three
lenses talk about dispatch and could give opposite advice: single-impl (a trait
with ≤ 1 impl), dispatch-erosion (an enum matched on all over) and this one (a
closed set of local impls held as dyn values). H1 is ONE rule
(dyn_to_enum::advise_trait / advise_enum), and every row names the branch
that fired.
Run. cargo lens dyn-to-enum [--max-enum N] [--trait-crates N] (door:
dwarves RepoLens::DynToEnum); defaults 8 / 3.
Inputs. trait nodes, IMPLEMENTS (impl → trait), FOR_TYPE (impl → the type
it is for), the ty text of every node (dyn <Name> occurrences), CALLS into
the trait's method nodes, and dispatch-erosion's own enum rows (ladders over
discr(E) and the crates they sit in — one source of ladders for both lenses).
Dataflow.
trait T in scope → IMPLEMENTS⁻¹ → impls, deduped; test impls apart (T_test)
→ per prod impl: crate ≠ crate(T) → FOREIGN; FOR_TYPE → a generic, or no local
struct / enum / union (`for Arc<G>`, `for &mut T`, `for &[u8]`) → BLANKET / forwarding
every node's ty → each `dyn a::b::Name …` → `Name`:
declaration sites (param / field / ret / local …) once each,
+ one per fn whose MIR block locals hold it (a block lists its locals: no repeats)
advise_trait(|I|, |F|, |B|, |T_test|, D) → DELETE | INLINE | TRAIT | KEEP | ENUM
dispatch-erosion enum rows (generated enums out) → |C(E)| ladder crates ≠ home
advise_enum(|C(E)|) → TRAIT | ENUM
The rule.
|I| = 0 → DELETE no production impl
|I| = 1 → INLINE single-impl: the seam has one side
|F| > 0 → TRAIT open set: implemented in another crate
|B| > 0 → TRAIT open set: a blanket or forwarding impl
|T_test| > 0 → KEEP a test seam: test code implements it
|I| > max_enum → TRAIT more impls than one enum should hold
D = 0 → KEEP generic bounds only: static dispatch already
otherwise → ENUM 2 ≤ |I| ≤ max_enum local impls held as dyn
enum E: |C(E)| ≥ trait_crates → TRAIT (the variants' behaviour lives in the callers)
The branches are ordered: the first that holds decides, so INLINE / DELETE
are exactly single-impl's SINGLE / ZERO rows (the fixture test asserts the
equality, and on Holger 19 / 2 = single-impl's golden 19 / 2).
No GraphBLAS step: one pass over the traits, their impls and the type texts; the enum side reuses dispatch-erosion's MIR ladder scan.
Worked example (the fixture). Shape with impls Circle, Square, Tri (all in
app) and fn draw(s: &dyn Shape): |I| = 3, D = 1 → ENUM. Codec (Json, Cbor
in app, Xml in fmt) and Box<dyn crate::Codec + Send>: |F| = 1 → TRAIT.
Bound (Lo, Hi), no dyn use → KEEP. Store (Disk; Fake is a test impl) and
Vault → INLINE, Lonely → DELETE — the single-impl lens's own counts. The
enum fmt::Kind is matched on in one other crate (app): ENUM at the default,
TRAIT with --trait-crates 1.
On Holger h32/next 9dbe5cbf (golden dyn-to-enum). 43 traits: 3 ENUM, 7
KEEP, 12 TRAIT, 19 INLINE, 2 DELETE; 42 enums with ladders, none matched on in
≥ 3 other crates. ENUM: offsite::Remote {DirRemote, S3Remote},
pull_intake::http::Fetch {UreqFetch, Staged}, grow::Releasable (6 local
impls, Box<dyn Releasable>). Hand check against the source (10 rows): the 3
ENUM rows true (closed, local, dyn-held); promote::gate::Gate TRAIT (its
impl<G: Gate> Gate for Arc<G> forwards: true); sandbox::Engine KEEP ("a
test substitutes its own", doc line 707: true); audit::AuditLog KEEP (test
impls: true); 4 INLINE rows = single-impl's (true by construction). Earlier
drafts called Gate and Engine ENUM (forwarding impl counted as a variant;
test doubles ignored) and counted dyn uses per MIR block (AuditLog 1,550 → 260
declaration sites + fns). Bench: Holger 0.9 s / 2.4 GB, dwarves 0.4 s / 1.1 GB,
gunnar 1.0 s / 2.5 GB (cached graph; the lens runs dispatch-erosion inside).
The union of builds (round 4, Gunnar re-compare row 11). With
--also-dump DIR the facts of advise_trait are read over the union of the
builds, impls matched by label: |I|, |F| and |B| count the union,
|T_test| and D take the largest build's value. On Gunnar's default dump
with the metered dump as --also-dump: 6 traits widen, and GoAheadPolicy
moves from INLINE to ENUM (2 impls [OperatorPolicy, MonetizePolicy], 6 dyn uses). World (2 impls in the union) is KEEP: the default build has a test
impl, so the rule reads a test seam. Run on the metered dump alone, the lens
gives 5 ENUM rows (ControlStore, FrameSink, World, Postman,
GoAheadPolicy): the delivery build holds them all, which is why the review's
"one Postman impl" holds only for the default build. Summary keys builds and
traits_widened.
What it cannot say. A pub trait of a library may have implementors
outside the workspace: the dump writes effective_visibility but the lens
loader does not read it yet (a needs row). Generic-bound uses (T: Trait)
are not counted (only dyn); the cost argument for an enum is allocation,
locality and inlining, not a measured speed-up.
4.16 foreign-helper-sprawl (plan H3)
The question. Which free fns grew up beside a type they do not own? A fn
whose first parameter is a foreign type (&str, &[u8], &Path, u64, a
dependency's type), that touches no local state and is called from many
modules is a method of that type in all but name: an extension trait (or a
newtype with methods) puts it in one place. Such helpers breed copies — the
twins lens's sha256_hex / tls_roots / now_unix rows — so the rank
multiplies the caller spread by the name twins. A helper that RETURNS a local
type is a smart constructor (fn not_found(msg: &str) -> Response): its home
is Response::not_found, so it is a CONSTRUCTOR row.
Run. cargo lens foreign-helper-sprawl [--min-modules N] (door: dwarves
RepoLens::ForeignHelperSprawl); default 3.
Inputs. fn nodes with their parent (a mod / crate parent = a free
fn), the first param child's ty, the ret child's ty, CALLS reversed,
CONTAINS up to the module, READS_FIELD / WRITES_FIELD, USES_TYPE.
Dataflow.
free fn f in scope → param[0] ty → type_name (refs, mut, lifetimes, Option/Box/Arc/Rc/Cow
peeled; the path's last segment) → FOREIGN when: a slice / tuple / impl / dyn, a
primitive or std / common type, or no local type of that name that f USES (USES_TYPE,
by id; without USES_TYPE edges: the bare name)
→ out when READS_FIELD ∪ WRITES_FIELD names a field of a local type (local state)
callers of f (prod fns) → module_path → compact indices
A ∈ 𝔹^{h×c} (helper × caller), P ∈ 𝔹^{c×m} (caller × module, one-hot)
M = A ANY.PAIR P [GraphBLAS mxm]
spread(f) = nvals(M[f, :]) the distinct caller modules
twins(f) = free fns with the same name AND the same param[0] type_name in other files
ret[0] = type_name with Result<T,_> peeled → CONSTRUCTOR when a local type f uses
finding: spread ≥ min_modules; rank = spread · (1 + twins)
The math. On the boolean semiring (ANY, PAIR) — ⊕ = ANY (any true),
⊗ = PAIR(x, y) = 1 (the structure only):
M[i][k] = ⊕_j A[i][j] ⊗ P[j][k] = 1 ⇔ ∃ caller j of helper i with module(j) = k
spread(i) = Σ_k M[i][k]
rank(i) = spread(i) · (1 + twins(i))
ANY.PAIR is the cheapest boolean product GraphBLAS has (no values are read);
the row counts are read off the result's tuples.
Worked example (the fixture). Modules hx, ua, ub, uc; use_a,
use_b, use_c (one per module) each call four fns of hx:
hex_of(b: &[u8]) (foreign, stateless: HELPER, 3 modules, 3 call sites; the
uc::hex_of(b: &[u8]) -> Vec<u8> in src/uc.rs is its name twin, rank 3·2 =
6), peek(s: &str) (READS_FIELD Repo::port: out), of_repo(r: &crate::Repo) (a local first parameter: out), repo_named(name: &str) -> Repo (CONSTRUCTOR → Repo). --min-modules 4 → no row.
On Holger h32/next 9dbe5cbf (golden foreign-helper-sprawl). 2,452 free
fns, 1,749 with a foreign first parameter, 1,353 stateless; 83 called from ≥ 3
modules: 77 HELPER, 6 CONSTRUCTOR, 15 with name twins. Top rows:
update-bundle digest::sha256_hex(&[u8]) (24 modules, 54 call sites, 14 name
twins — the review's sha256-hex family), traits znippy_base::sha256_hex,
grpc::functional_status(&str) (9 modules, 128 sites), promote::npm:: compare_versions(&str) (3 twins: the review's version orderings),
sold::read(&Path), peers::load(&Path) (4 twins), keyfile::write_private
(3 twins), blake3_hex ×2; CONSTRUCTOR twin_tls::mtls_off::load(&Path) → Credential, audit::chain::verify_dir(&Path) → VerifyReport. Hand check (10
rows against the source): 7 true, 3 weak (functional_status / not_found
build a dependency's response type: a helper, but no extension trait to make;
backend::head_of takes a response tuple). Two false-positive classes of an
earlier draft are gone: bare-name twins (load, read, ok paired across
unrelated fns: now name + first-parameter type) and bare-name "local" returns
(a local type Result alias and a local Status made tonic's Status look
local: now USES_TYPE by id). Since H4 a slice is as foreign as its element
(&[Source] is local). Bench: Holger 0.7 s / 2.3 GB, dwarves 0.3 s /
1.1 GB, gunnar 0.7 s / 2.5 GB (cached graph).
What it cannot say. A helper whose first parameter is a generic T is
counted as foreign (it serves any type). Name twins are the cheap stand-in for
the twins lens (name + first-parameter type); the twins lens proves the copies
by callee sets and MIR shape.
Round 2 (lane LENS-POLISH, 2026-11-05; re-compare 2 wrong result 6). Two more classes, both out of the HELPER (extension-trait candidate) count:
BUILDER(f) ⇔ ¬CONSTRUCTOR ∧ ret(f) = T, T ∉ std (primitives, String, Vec, Option, Result, Box,
Arc, Ordering, IpAddr, …), |T| > 1 (no generic), T ≠ type(param₀), T not a
workspace type name, T capitalised (`internal_err(e) -> tonic::Status`)
LEAF(f) ⇔ ret(f) = () ∧ type(param₀) ∈ {str, String, bool, char, integers} by value or `&`
(`functional_status(component: &str, …)`)
A builder makes a dependency's value from its arguments and a leaf reports; neither
operates on its first parameter, so an extension trait on &str is the wrong move.
Holger h32: HELPER 79 → 74, BUILDER 3 (internal_err → Status, build_agent → Agent,
worker → JobKind), LEAF 2 (functional_status, refusals::record). Not reclassified:
znippy_base::{ok, not_found, ok_bytes, not_found_bytes} return the workspace alias
HttpReply, which has no USES_TYPE edge of its own; reading every workspace type NAME as
local would bring back the Status / Result name collision fixed in lane LENS-SEQ
(internal_err would read CONSTRUCTOR). They stay HELPER until the dump carries alias
targets. Fixture: hx::reply(&str) -> web::Reply BUILDER, hx::status(&str) LEAF.
Round 3 (lane LENS-FIX4, 2026-11-06). A helper whose return is a TUPLE it builds
is a BUILDER: znippy_base::{ok, not_found, ok_bytes, not_found_bytes} return
HttpReply, a workspace alias the dump prints erased as (u16, Vec<(String, String)>, Vec<u8>). BUILDER ⇔ ret ∉ local ∧ (ret is a dependency's non-std type ∨ ret = (t₁, t₂, …)). Fixture foreign_helper_reads_a_tuple_reply_as_a_builder. Holger h32: HELPER 73 →
67, BUILDER 3 → 9 (the four znippy_base replies and backend::head_of /
head_of_bytes); Gunnar unchanged.
4.17 late-validation (plan H4, "parse, don't validate")
The question. Which checks run on raw values deep inside the system? A
validator over a primitive (fn valid_name(s: &str) -> bool, fn check(host: &str) -> Result<(), E>) that is called many calls below the entries and from
several subsystems re-checks a value that crossed the boundary unparsed. The
cure is a newtype built once at the entry (Name::parse(s)?): after it the
type carries the proof and the deep calls go. The review's §4 row 11
(sanitize_seg ×6) is the shape; R1's gate is its big brother.
Run. cargo lens late-validation [--entry PAT…] [--min-depth N] [--min-subsystems N] (door: dwarves RepoLens::LateValidation); defaults:
the roots, 4, 2. A server's real entries sit behind a framework (hyper, tonic)
and look like roots-with-callers; give --entry for them, as gate-bypass does.
Inputs. fn nodes with their param / ret children (ty), the
mir_block call terminators (callee monikers), CALLS, the dyn expansion
(IMPLEMENTS + CONTAINS), USES_TYPE (which bare names are local types).
Dataflow.
fn f in scope → VALIDATOR when
1 ≤ |params| ≤ 2, param[0] foreign (foreign-helper-sprawl's rule: primitive / std /
dependency type, a slice of one; USES_TYPE decides a bare name),
ret = bool | Result<(), _>,
its MIR calls ≥ 1 text test: str::eq / starts_with / ends_with / contains / strip_* /
eq_ignore_ascii_case (dispatch-erosion's `is_str_compare`), str::is_* / chars / bytes,
char::is_*
entries E = --entry patterns, else the roots (prod fns no prod code calls, no dyn source)
A = CALLS ∪ DYN over fn nodes (n × n, boolean)
dist = min_hops(E, A) level-synchronous MIN.PLUS BFS [GraphBLAS vxm]
per validator v: d(v) = dist[v]; S(v) = { subsystem(c) : c prod caller of v }
LATE ⇔ d(v) ≥ min_depth ∧ |S(v)| ≥ min_subsystems; rank = |S(v)| · d(v)
subsystem(c) = crate::mod₁ of c (Graph::subsystem)
The math. On the tropical semiring (MIN, PLUS) over ℕ ∪ {∞} — ⊕ = min,
⊗ = + — with unit edges (A[i][j] = 1 hop):
d₀(x) = 0 for x ∈ E, ∞ else
f₀ = d₀ restricted to E the first frontier
f_{k+1}⟨¬d, replace⟩ = f_k ⊕.⊗ A (vxm; every value of f_k is k, so the result is k+1)
d⟨f_{k+1}⟩ = f_{k+1} record the first visit
d(v) = min { k : v ∈ f_k } = the fewest calls from any entry to v
The mask ¬d (complement of the visited set) makes each node enter one
frontier only; with unit weights the first visit IS the minimum, so no
Bellman–Ford relaxation is needed (nornir_mimir_graph::ops::min_hops).
Worked example (the fixture). entry_e → ua::l1 → ua::l2 → {ua::l3, ub::l3b} → vx::seg_ok(s: &str) -> bool (a starts_with call with a k:&str
argument inside); entry_e → vx::short_ok. With --entry "app entry_e().":
d(seg_ok) = 4, callers l3 (app::ua) and l3b (app::ub) → 2 subsystems → LATE
(&str) -> bool, depth 4, 2 callers in 2 subsystems [app::ua, app::ub];
d(short_ok) = 1 → in time. --min-depth 5 → no row.
On Holger h32/next 9dbe5cbf (golden late-validation, the gate's nine entry
patterns = 219 entry fns). 115 validators, 21 not reached from those
entries, 4 LATE: egress::check(&str) -> Result<(), EgressRefused> (depth 4,
14 callers in 11 subsystems — the egress host is re-checked everywhere it is
used: an EgressHost newtype), repository::format_feature_compiled (depth
7), nuspec::is_root_nuspec (depth 6), trust::apk_key_name_ok (depth 4).
In roots mode (753 entries) 5 LATE, and they are the classic shape:
promote::cargo::valid_name, promote::npm::valid_package_name,
pull_intake::oci::is_digest (11 callers) at depth 9,
scanner::advisories_warn_only, is_root_nuspec. Hand check (10 rows over
both runs and gunnar's two): 7 true (egress::check, the three package-name /
digest validators, apk_key_name_ok, gunnar is_reserved_principal_name,
is_loopback_target), 3 weak (format_feature_compiled and
advisories_warn_only test configuration words, not input; is_root_nuspec
a path predicate inside one archive walk). One false positive of a draft is
gone: sources::validate_sources(&[Source]) — a slice is now as foreign as
its element. Bench: Holger 0.8–1.3 s / 2.1–2.4 GB, dwarves 0.35 s / 1.1 GB,
gunnar 0.9 s / 2.5 GB (cached graph; one MIN.PLUS BFS over all fns).
What it cannot say. A validator that tests bytes by arithmetic (no str / char call) is not seen; the depth is call depth, not data-flow distance (a value parsed at the entry and passed down raw looks the same as one read deep from a file).
4.18 asymmetric-seam (plan H5)
The question. Which trait methods are a capability of a few impls dressed
as a method of all? When most impls refuse a method — through a default body
that returns Err(Unsupported) / None / panics and that they do not
override, or through an override that refuses — every caller must handle
"unsupported" at run time and the trait says nothing true about its members.
The review's §1 / §6 "Clever" (Caps bitset + the Unsupported defaults of
RepositoryBackendTrait). The fix: split the trait (a sub-trait the capable
few implement) or move the capability into the impls.
Run. cargo lens asymmetric-seam [--min-impls N] [--max-ratio F] [--max-blocks N] (door: dwarves RepoLens::AsymmetricSeam); defaults 5 /
0.2 / 6.
Inputs. trait nodes and their method children, the methods' mir_block
nodes (a trait method WITH blocks has a provided default), the ret child's
ty, IMPLEMENTS + CONTAINS (each impl's own methods), FOR_TYPE (blanket
impls), CALLS (callers; the one-frame helper).
Dataflow.
trait T in scope → I(T) = prod impls, blanket / forwarding impls out (FOR_TYPE a generic,
or a self type named like an impl generic, read off `impl#[D][T]` when FOR_TYPE is absent)
method m of T → default(m) = m's own MIR (if any)
refuses(f) ⇔ |blocks(f)| ≤ max_blocks ∧ ( a call into `panicking/…` / begin_panic / panic_fmt
∨ `_0 = adt~1(…)` with ret Result (Err) ∨ `_0 = adt~0(…)` with ret Option (None)
∨ a CALLS target g with refuses(g), one frame: `unsupported(self.name(), Cap::X)` )
eff(i, m) = i's own child fn named m, else default(m)
Capable(T, m) = { i ∈ I(T) : eff(i, m) exists ∧ ¬refuses(eff(i, m)) }
ASYMMETRIC ⇔ |I| ≥ min_impls ∧ |Capable| / |I| ≤ max_ratio ∧ |Capable| < |I|
beside it: m's prod callers, the free fns among them
The math.
eff(i, m) = own(i, m) if i defines m
= default(m) otherwise
CapRatio(T, m) = |{ i ∈ I(T) : ¬refuses(eff(i, m)) }| / |I(T)|
ASYMMETRIC(T, m) ⇔ |I(T)| ≥ min_impls ∧ CapRatio(T, m) ≤ max_ratio ∧ CapRatio < 1
The two shapes of the plan row are one formula: a refusing default few
override (the Unsupported defaults) and refusing overrides (a put that says
"read-only"). refuses is name-free (Err = variant 1 of Result, None =
variant 0 of Option, in the MIR grammar's adt~V), bounded by max_blocks
so a default with real logic is never a refusal. No GraphBLAS step: one pass
over the trait methods, their impls' children and the callers.
Worked example (the fixture). trait Shelf with impls B1…B5. put() -> Result<(), E> defaults to _0 = adt~1(move _2) (Err); only B1 overrides →
capable 1/5 → ASYMMETRIC default Err; capable 1/5 impls [B1], its free-fn
caller shelve_all named. drop_all() has no default; B2…B5 override it with
Err → 4 impls override it with a refusal; capable 1/5. tag() defaults to
a call of refuse(), which builds Err → default Err via a refusing helper; capable 0/5. len() defaults to a constant → no refusal. get() is required
and every impl defines it → capable 5/5. With --max-ratio 0.1 only tag
stays.
On Holger h32/next 9dbe5cbf (golden asymmetric-seam). 163 trait methods,
49 provided defaults, 20 refuse; 6 ASYMMETRIC in 2 traits:
RepositoryBackendTrait delete_artifact (2 of 26 impls capable),
group_members (2/26: GroupBackend and the conformance probe), put_from_reader
and handle_http2_upload (3/26), fetch_chunks (4/26) — all through
unsupported(self.name(), Cap::…), the helper one frame down — and
scanner::Scanner::intel_sequence (default None, 2/16). Hand check
(the 6 rows + put, list, archive at --max-ratio 1, + the blanket impl):
the 6 true; put is capable in 21/26 at this commit (the backends check
is_writable and then write: logic, not a refusal — the review's "15 of 17"
counted an older tree), list 23/26, archive 18/26 (not findings: true
negatives); Everything (the pub mod conformance probe backend in prod
code) counts as capable — weak. A draft counted the forwarding
impl<D: …> RepositoryBackendTrait for D as capable (no FOR_TYPE edge: now
read off the moniker) and missed every unsupported(…) default (the one-frame
helper). Bench: Holger 0.7 s / 2.4 GB, dwarves 0.3 s / 1.0 GB, gunnar 0.7 s /
2.5 GB (cached graph).
Round 4: the trait as a whole at 2–4 impls (Gunnar re-compare row 8).
The method rule needs ≥ 5 impls, so Gunnar's ControlPlane (2 impls) was
silent. Under min_impls the lens now judges the trait as a whole:
R(T) = { m ∈ M(T) : refuses(default(m)) }
REFUSING-SEAM(T) ⇔ 2 ≤ |I(T)| < min_impls ∧ |R(T)| ≥ max(seam_min_refusing = 4, seam_share = 0.3 · |M(T)|)
working(i, T) = |{ m ∈ R(T) : i overrides m with a body that does not refuse }|
dead(T) = |{ m ∈ R(T) : working(i, m) = 0 for every i }|
On the Gunnar metered dump (golden
golden_gunnar_asymmetric_seam_control_plane_is_a_seam_of_refusals):
REFUSING-SEAM gunnar-grpc seam/ControlPlane# (crates/gunnar-grpc/src/seam.rs:1242) — 98/182 methods default to a refusal (54 %) across 2 impls, 3 refused by every impl; overridden by CoreControlPlane 95, MemoryControlPlane 16
The default dump gives 91/174 (52 %), 0 dead. Hand check: the 3 dead methods
are webhook_add, webhook_list and webhook_remove (seam.rs:3102–3116,
Err defaults, no impl overrides them). Holger has no 2–4-impl trait over the
bar (AuditLog, 3 impls, has 2 refusing defaults): its golden is unchanged.
Summary key refusing_seams; finding {class: REFUSING-SEAM, trait, loc, methods, refusing_defaults, impls, refused_by_all, overrides[{impl, working}]}.
What it cannot say. A refusal decided at run time (if !self.writable { bail!() } then work) is logic, not a refusal: the lens counts it capable (the
Caps bitset is that kind of switch; runtime-guard-erosion reads such flags).
A helper two frames down is not followed.
4.19 deferred-init-sprawl (plan H10)
The question. Which structs are built in steps? A struct whose fields are
mostly "filled in later" — Option<_>, OnceLock<_> / OnceCell<_> /
LazyLock<_>, ArcSwap<Option<_>>, Mutex<Option<_>>, and the doubly
deferred OnceLock<Option<Arc<_>>> — and that several subsystems read makes
every reader handle "not yet", and its construction order is a convention, not
a type. The review's R12 (Holger: 74 fields, 36 Option) and §6 "Clever":
sectioned, eager config instead — one struct per subsystem, built complete.
Run. cargo lens deferred-init-sprawl [--min-fields N] [--share F] [--min-subsystems N] (door: dwarves RepoLens::DeferredInitSprawl);
defaults 5 / 0.4 / 2.
Inputs. struct nodes and their field children (CONTAINS) with the
printed ty; READS_FIELD and WRITES_FIELD (field → its readers / writers);
IMPLEMENTS + FOR_TYPE + EXPANDS (derive-written serde / clap impls).
Dataflow.
struct S in scope → Φ(S) = its `field` children
→ per field: classify(ty) through Arc / Box / Rc:
Option<_> → OPTION
OnceLock / OnceCell / LazyLock / Lazy / ArcSwap(…)<T> → DEFERRED
OnceLock<Option<_>>, ArcSwapOption, Mutex / RwLock / (Ref)Cell<Option<_>> → DEFERRED+OPTION
else → eager
→ readers(φ) = READS_FIELD⁻¹ ∪ WRITES_FIELD⁻¹ (prod fns) → subsystem = crate::mod₁
OPTIONAL DATA = S has a derive-written Serialize / Deserialize / Parser / Args /
FromArgMatches / Subcommand impl (EXPANDS) and no once-cell field: a record's optional
keys, counted apart
DEFERRED-INIT ⇔ |Φ| ≥ min_fields ∧ share ≥ share₀ ∧ |subsystems| ≥ min_subsystems ∧ ¬OPTIONAL DATA
rank = (deferred fields) · |subsystems|
The math. With δ(φ) = 1 when φ is OPTION, DEFERRED or DEFERRED+OPTION:
share(S) = Σ_{φ ∈ Φ(S)} δ(φ) / |Φ(S)|
subsystems(S) = | { sub(f) : f ∈ readers(φ), φ ∈ Φ(S) } |
DEFERRED-INIT(S) ⇔ |Φ(S)| ≥ min_fields ∧ share(S) ≥ share₀ ∧ subsystems(S) ≥ min_subsystems
rank(S) = Σ δ · subsystems(S)
No GraphBLAS step: one pass over the fields and their reader lists.
Worked example (the fixture). Hub with six fields — a: Option<String>,
b: Option<u64>, c: std::option::Option<PathBuf>, d: std::sync::OnceLock<Option<Arc<Db>>>, e: u32, g: Vec<u8> — read by
ua::l1 and ub::l3b: share 4/6 = 67 %, 2 subsystems → DEFERRED-INIT 4/6 fields (67 %: 3 Option, 0 once-cell, 1 both) with d named as doubly
deferred. Calm (one Option of five) stays eager. Rec (five Options, a
derive-written Deserialize) is OPTIONAL DATA. --min-subsystems 3 → none.
On Holger h32/next 9dbe5cbf (golden deferred-init-sprawl). 1,066
structs with fields, 4,936 fields, 867 deferred or nullable, 34 doubly
deferred; 12 DEFERRED-INIT, 29 OPTIONAL DATA. Top: Holger (lib.rs:1256)
42 / 60 fields (27 Option, 14 once-cells, repo_key_cell: OnceLock<Option<Arc<RepoKey>>>), read from 12 subsystems — the review's R12;
grpc::HolgerGrpc 20 / 42 (Options set by 39 with_* builder methods),
exposed::http::GatewaySettings 16 / 31, topology::Sources,
mirror_cycle::SourcePlan. Hand check (10 rows against the source): 5 true
(those five), 5 weak — parsed records without a serde derive (AuditFilter,
rpm::RpmHeader, nuspec::Nuspec, rpmdb::Rpm, scanner::ScanSubject) whose
Options are optional data, which the graph cannot tell from late state
without a derive. A draft listed 41 rows; 29 were serde / clap records
(#[derive(Deserialize)] configs and #[derive(Parser)] CLIs), now counted
apart. Bench: Holger 0.7 s / 2.4 GB, dwarves 0.3 s / 1.1 GB, gunnar 0.6 s /
2.5 GB (cached graph).
What it cannot say. Whether an Option is set after construction:
WRITES_FIELD counts a struct literal as a write too, so "filled in later" is
read from the type, not from the order of writes.
Round 2 (lane LENS-POLISH, 2026-11-05; re-compare 2 wrong result 7, Gunnar's "patch and config records"). A record built WHOLE is optional data, serde derive or not:
builds(fun, S) ⇔ derived(fun) (Clone, Default: a whole copy)
∨ writes(fun) ⊇ fields(S) (a struct literal: build_primary)
∨ (ret(fun) names S or Self ∧ ¬ step(fun, S)) (fn parse(b) -> Result<Rpm, E>)
step(fun, S) ⇔ fun has `self` ∧ fun is a method of S (fn with_a(self, …) -> Self)
PARSED(S) ⇔ no once-cell field ∧ ∃ writer of a nullable field ∧
every writer of every nullable field builds(fun, S)
OPTIONAL DATA = serde / clap derive ∨ PARSED (counted apart, no finding)
Holger h32: DEFERRED-INIT 12 → 4 (Holger, HolgerGrpc, AuditFilter, ScanSubject),
optional data 29 → 37, parsed records 29 (some also serde). RpmHeader, Nuspec,
rpmdb::Rpm, GatewaySettings, Sources, Facts, SourcePlan are built whole and read
as records now. Still flagged: AuditFilter (its literals sit in closures of fns that
return something else and do not name every field) and ScanSubject. Fixture:
parse_rpm(&[u8]) -> Result<Rpm, E> makes Rpm a parsed record; Hub::with_a(self) -> Hub writes Hub#a after construction, so Hub stays DEFERRED-INIT.
Round 3 (lane LENS-FIX4, 2026-11-06). Three more ways a record is built whole, each from a Holger row:
aggregates(f, S) ⇔ f's MIR holds `_k = adt~0(…)` with _k : S (a struct literal,
functional update `..base` included: AuditFilter)
built_here(f, S) ⇔ f calls a fn returning S (or Self on S's impl), then sets fields
(`let mut s = ScanSubject::new(..); s.repo = ..`)
builder_step(f, S) ⇔ f is `fn(mut self, ..) -> Self` on S ∧ |steps(S)| ≤ 2
borrowed_view(S) ⇔ every optional field of S is `Option<&…>` (a borrowed argument
bundle cannot outlive the call that built it: ScanSubject<'a>)
A struct with three or more builder steps IS built in steps (HolgerGrpc: 39 with_*).
Fixture deferred_init_reads_a_functional_update_literal_as_a_whole_record (Filter by a
functional-update literal, Subject by a constructor then a tweak, View with three steps
over borrows: records; Hub, with a setter and three steps: DEFERRED-INIT). Holger h32:
DEFERRED-INIT 4 → 2 (Holger, HolgerGrpc); AuditFilter and ScanSubject are optional
data. Gunnar: 1 (ServiceRequest) unchanged.
4.20 typestate-candidate (plan H2)
The question. Which structs keep a state machine in a field? When several
methods of one struct begin by checking the same field of self and bail out
(if !self.open { return Err(..) }, let Some(f) = &self.inner else { bail!() }), the state is re-asked at run time on every call. Typestate moves it into
the type (Conn<Open> has send, Conn<Closed> does not): the guards and
their error paths disappear, and the methods that write the field become the
transitions. The review's §6 "Clever instead of clear" and R12.
Run. cargo lens typestate-candidate [--min-methods N] [--early N] [--bail-blocks N] (door: dwarves RepoLens::TypestateCandidate); defaults
3 / 3 / 6.
Inputs. struct nodes and their field children (start_line = the
declaration order, which is the MIR field index; ty), FOR_TYPE without
IMPLEMENTS (inherent impls, matched by id: the edge end can be a stub), the
methods' mir_block nodes and CFG_SUCC, the ret child, WRITES_FIELD (schema 6).
Dataflow.
struct S in scope → fields φ_0…φ_{k-1} by start_line
inherent impls of S (FOR_TYPE → S, no IMPLEMENTS) → methods M(S), prod
method m → CFG → early(m) = the first `early` blocks of the main body (bb0, bb1, …)
and of the first nested body (bbc1.0, …: an async method's coroutine)
→ each `switch` there: its operand traced (≤ 5 hops) through use / copy / move / not /
ref / refmut / discr / deref_copy and deref'd locals (`_5=discr(_4*)`, `_4=ref(_1*.0)`),
through one trivial accessor of the workspace (`fn is_open(&self) -> bool { self.open
}`, memoised), or through one predicate call on a field (`self.inner.is_none()`: the
call's first argument) → a field place of _1 (`_1*.N`) → field N
→ GUARD(m, N) when one successor bails within bail_blocks blocks: `_0 = Err(…)` (ret
Result), `_0 = None` (ret Option), a panic call
G(S, N) = guarded methods; T(S, N) = methods of S with WRITES_FIELD(m, φ_N)
TYPESTATE ⇔ |G(S, N)| ≥ min_methods ∧ φ_N holds a STATE (bool, Option, an atomic, a
once-cell, a local enum — through Arc / Box / a lock; not a map one looks things up in)
The math.
guard(m, N) ⇔ ∃ b ∈ early(m), term(b) = switch: operand(b) ⇝ _1.φ_N
∧ ∃ x ∈ succ(b): bails(x) within bail_blocks blocks
G(S, N) = { m ∈ M(S) : guard(m, N) }
T(S, N) = { m ∈ M(S) : WRITES_FIELD(m, φ_N) }
TYPESTATE(S, N) ⇔ |G(S, N)| ≥ min_methods ∧ state(ty(φ_N))
⇝ is the hop-bounded def-use walk over the name-free MIR grammar (_k=<rhs>
statements and call …⟨args⟩->_k terminators). No GraphBLAS step: one CFG
walk per inherent method, the early blocks only.
Worked example (the fixture). struct Conn { open: bool, peer: String }
with an inherent impl. send / recv: bb0: _3 = use(_1*.0); switch(^_3; 0),
bb1: _0 = adt~1(move _4) (Err) — guard on field 0. ping: bb0: call impl#[Conn]is_open()⟨_1⟩->_3, bb1: switch(^_3; 0), bb2: Err — the accessor
is_open is _0 = use(_1*.0), so the guard is on field 0 too. open() has a
WRITES_FIELD edge to Conn::open. → TYPESTATE app Conn# — field open: bool guards 3 methods [ping, recv, send]; transitions (write it): [open].
--min-methods 4 → none.
On Holger h32/next 9dbe5cbf (golden typestate-candidate). 479 structs
with inherent methods, 2,751 methods, 154 early switches on a field of self, 22
guards, 0 candidates at three methods. At --min-methods 2: holger-intel net::IntelNet (field inner: Option<Box<dyn Fetch>> guards get,
get_feed; transitions offline, online — a true typestate case: the two
constructors are the two states) and import_door::ImportDoor (intel: Option<IntelConfig> guards import_intel_as, refresh_intel); gunnar's
repair::Entitlements likewise. Hand check: the 3 two-method rows true (each
an Option field set by a constructor, re-checked by every method). The
tracer was widened twice on the real dump (18 → 154 early field switches):
real MIR reads a field as _3=use(_1*.1) and let Some(x) = &self.f as
_4=ref(_1*.0); _5=discr(_4*); async methods keep their check in the
coroutine body. Bench: Holger 1.2 s / 2.5 GB, dwarves 0.7 s / 1.1 GB, gunnar
1.1 s / 2.4 GB (cached graph).
On the lossless graph (schema 8, lens upgrade batch 1). The row names the state a guard lets through: the switch values of the arms that do not bail, read with how the switched local was made:
pass(b) = { v : arm v of the switch does not bail }
state = `== k` / `!= k` when the local is Eq(field, const k) / Ne(…) and the equal arm passes / bails
= `variant v` … when the local is discr(field)
= `true` / `false` when the local is the bool field itself
The TYPESTATE row ends with passes when it is <states> (JSON states).
Fixture typestate_names_the_state_the_guards_let_through.
What it cannot say. A check deeper than the first early blocks, or a
state read through two helpers, is not seen; a guard that logs and continues is
no bail. (The same use(…) spelling is why runtime-guard-erosion, whose
operand walk follows only discr(…) / deref_copy(…), reports 0 FLAG doors on
Holger — reported to the H6 owner, not changed here.)
4.21 pipeline-subclass-leak (plan H7)
The question. Which core structs carry the private state of their leaves?
A pipeline struct whose fields are touched by almost none of its methods and
are named after the kinds the pipeline switches on (IntakeChain holding
npm_tags, cargo_yanks, oci_push) has subclasses leaked into its base:
each leaf's state belongs in the leaf (its own struct behind the seam the
pipeline calls), and the core keeps what every stage uses. The review's §6
naming and §5 hint 14.
Run. cargo lens pipeline-subclass-leak [--min-fields N] [--min-methods N] [--max-share F] [--min-leaks N] [--min-ladders N] [--min-sources N] (door: dwarves
RepoLens::PipelineSubclassLeak); defaults 8 / 10 / 0.10 / 2 / 3 / 2.
Inputs. struct nodes and their field children; every impl for the
struct (FOR_TYPE by id, inherent and trait, derive-written ones out) and their
methods; READS_FIELD and WRITES_FIELD (schema 6); the leaf vocabulary from four
sources: local mod nodes, the local crate names, local enum nodes and their
variant children, and dispatch-erosion's ladder counts (MIR switch on discr
of the enum).
Dataflow.
the leaf vocabulary (none of the four sources needs a ladder):
Module mod m under module p (p not a crate root) → taxonomy `mod p`, leaf m
Crate local crates a-x-b, a-y-b (≥ 2 fixed words) → taxonomy `crates a-*-b`, leaves x, y
Variant local enum E → taxonomy E, leaves its variants
Dispatched E with ≥ min_ladders ladders (dispatch-erosion) → the same leaves, a second source
src(w) = the sources that name word w (lower case, ≥ 3 chars, generic words out)
W = { w : |src(w)| ≥ min_sources }, T(w) = the taxonomies that name w
struct S in scope with |Φ(S)| ≥ min_fields and |M(S)| ≥ min_methods (a core struct)
field φ → users(φ) = enclosing fns of READS_FIELD⁻¹(φ) ∪ WRITES_FIELD⁻¹(φ), ∩ M(S)
share(φ) = |users(φ)| / |M(S)|; stem(φ) = the first `_` segment of φ's name
LEAK(φ, T) ⇔ share(φ) < max_share ∧ stem(φ) ∈ W ∧ T ∈ T(stem(φ)) ∧ stem(φ) ∉ S's module path
LEAKY(S) ⇔ ∃ T: the narrow fields name ≥ min_leaks distinct leaves of T
The math. With the field × method incidence U ∈ 𝔹^{|Φ|×|M|}
(U[φ][m] = 1 iff m reads or writes φ):
share(φ) = (Σ_m U[φ][m]) / |M(S)|
leaf(w) ⇔ Σ_s V[w][s] ≥ min_sources (V ∈ 𝔹^{|words|×4}: word × source)
LEAK(φ, T) ⇔ share(φ) < max_share ∧ leaf(stem(φ)) ∧ stem(φ) ∈ L(T)
LEAKY(S) ⇔ |Φ(S)| ≥ min_fields ∧ |M(S)| ≥ min_methods
∧ ∃T: |{ v ∈ L(T) : ∃φ: LEAK(φ, T) ∧ stem(φ) = v }| ≥ min_leaks
The "≥ 2 leaves of ONE taxonomy, each named by ≥ 2 sources" clause is what
makes it a subclass leak and not a name coincidence. Until lane LENS-FIX3 the
only source was the dispatched enums (≥ 3 ladders); Holger's R2 refactor
(c36dee4b) removed the format ladders, and IntakeChain's npm_* / cargo_*
fields were no longer flagged. The modules (promote::{npm, cargo},
holger_traits::format::*), the crate family (holger-znippy-*-repository) and
the enums' variants (identity::Format) still name them. A crate's top modules
are its subsystems (audit, jobs) and holger-* is the whole workspace, so
neither is a taxonomy. No GraphBLAS step: the row sums are read off the field →
reader lists.
Worked example (vocabulary, vocab_fixture). No ladder at all. Modules
promote::{npm, cargo, intake}, crates app-npm-store / app-oci-store, enum Format { Npm, Cargo, Oci, Zip }; struct Chain in promote::intake (10 fields,
11 methods) with narrow npm_tags, cargo_yanks, oci_push, zip_cache. npm
has 3 sources, cargo 2 (module, variant), oci 2 (crate, variant), zip 1 →
mod app::promote: npm_tags → npm, cargo_yanks → cargo, crates app-*-store: npm_tags → npm, oci_push → oci, app Format#: npm_tags → Npm, cargo_yanks → Cargo, oci_push → Oci; zip_cache is not listed. --min-sources 4 → none.
Worked example (the fixture). enum Pkg { Npm, Cargo, Oci, Deb } with three
ladders (switch(^_2; 0,1,2) over discr(_1: Pkg) in module npm). struct Chain (8 fields, 11 methods in one impl): f1…f5 are read by every method;
npm_tags by one (1/11 < 10 %), cargo_yanks written by one, oci_push by
none → LEAK app Chain# — 8 fields, 11 methods, 3 leaf fields: app Pkg#: npm_tags → Npm (1 methods), cargo_yanks → Cargo (1 methods), oci_push → Oci (0 methods). --min-leaks 4 → none.
On Holger h32/next 9dbe5cbf (golden pipeline-subclass-leak). 19 core
structs; 3 LEAKY, 16 leaf fields: promote::intake::IntakeChain (36 fields,
122 methods) holds npm_tags, npm_meta, npm_base, cargo_yanks,
cargo_base, oci_push → promote::identity::Format — the review's list;
pull_intake::PullIntake holds apk_keys, apk_signer, npm_meta →
Format and crate_index, crate_records → index_rows::IndexRow;
metrics::Metrics holds one counter per Verb (put, delete, promote,
search, sbom). Hand check: IntakeChain and PullIntake true (11 fields);
Metrics weak (per-verb counters are a table, not leaf state). gunnar: 2 rows
matching an xtask enum (setup_svg::Role) from an unrelated crate — weak;
that dump predates WRITES_FIELD (a needs row). A draft took every module and
variant name as a leaf and listed 136 fields on 11 structs (audit, door,
key, write): the dispatched-enum clause took it to 16. Bench: Holger 1.1 s
/ 2.3 GB, dwarves 0.6 s / 1.1 GB, gunnar 1.1 s / 2.4 GB (cached graph; the lens
runs dispatch-erosion inside).
Lane LENS-FIX3 (the vocabulary without ladders). h32: 3 → 5 LEAKY, 16 → 24
leaf fields (Holger and HolgerGrpc join through modules and CLI variants).
Holger main c36dee4b: before, 7 LEAKY and IntakeChain flagged only through the
CLI's Command (audit, jobs); after, IntakeChain holds npm_tags,
npm_meta, npm_base, cargo_yanks, cargo_base, oci_push →
identity::Format, promote::{npm, cargo}, holger_traits::format again.
CLI command enums are no taxonomy (H7 noise). A clap derive (Subcommand,
Parser, Args, ValueEnum; by the trait node, or by the derive-written impl's
moniker when the trait is outside the dump) or an enum defined in a CLI place
(main.rs, a cli file or module, a *_cli module, a *-cli crate) gives no
leaves. On Holger these were holger-cli cli::Command, the holger bin's
Commands, BundleCmd and IntelOp: c36dee4b 7 → 6 LEAKY, 61 → 32 leaf fields
(AppsHub drops out; IntakeChain keeps its 6 format fields and gate_policy);
h32 5 → 5 LEAKY, 24 → 21 fields. Fixture test
pipeline_subclass_leak_leaves_cli_command_enums_out_of_the_vocabulary.
What it cannot say. A leaf field whose name does not start with the leaf's
name (tags_by_registry) is missed; an enum dispatched through a trait instead
of a match has no ladders and gives no leaves.
4.22 monolithic-router (plan H8)
The question. Which long fns are a route table written as code? A fn of
hundreds of lines that decides what to do by comparing one or two values (the
method, the path) against many string literals holds every route, its
handler and its checks in one body. The cure (review R3, §5 hint 7) is a table
((method, pattern) → handler) the router walks: the routes become data, each
handler a fn, and "every route has an auth check" a property of the table.
Run. cargo lens monolithic-router [--min-lines N] [--min-compares N] [--min-share F] [--key-values N] [--src DIR] (door: dwarves
RepoLens::MonolithicRouter); defaults 500 / 20 / 0.5 / 2.
Inputs. fn start_line / end_line, the fn's param children, its
mir_block nodes (call terminators with callee and arguments, statements,
the switch terminators); with --src, the text of the fn's lines when it has
no MIR.
Dataflow.
fn f in scope, lines(f) = end − start + 1 ≥ min_lines
MIR → string tests C(f): calls whose callee is dispatch-erosion's is_str_compare
(str::eq / starts_with / ends_with / contains / strip_* / eq_ignore_ascii_case), or
any PartialEq eq / ne with a `k:&str` constant argument
→ per test: its first non-constant argument traced back (≤ 12 hops) INSIDE ITS BODY
(bb… = main, bbcK.… = nested body K: each numbers its locals on its own) through
ref / refmut / use / copy / move / deref_copy, deref'd locals, a field of a tuple /
adt aggregate (`match (method, path)` → `_K = tuple(a, b)`, `_K.1` → b), and call
destinations (the call's first argument: `path(_17)`, `deref(_16)`) → root (body, local)
→ n_r = tests per root, ranked; key = the top key_values roots
ROUTER ⇔ |C(f)| ≥ min_compares ∧ Σ_key n_r / |C(f)| ≥ min_share
no MIR + --src → the fn's lines: `== "…"`, `"…" =>` arms, .starts_with / strip_prefix /
ends_with("…") → TEXT-ROUTER when ≥ min_compares (no root: the text cannot say which
value is compared); no MIR, no --src → a needs row
The math. For f with string tests C(f) and the root map ρ : C(f) → (body, local):
n_r(f) = |{ c ∈ C(f) : ρ(c) = r }|, r₁, r₂, … by n_r descending
key(f) = Σ_{i ≤ key_values} n_{r_i}(f)
share(f) = key(f) / |C(f)|
ROUTER(f) ⇔ lines(f) ≥ min_lines ∧ |C(f)| ≥ min_compares ∧ share(f) ≥ min_share
Two deviations from the plan row's wording, both forced by real MIR: the
share is taken over the string tests, not over every switch (each ? adds a
switch on a Try discriminant — handle has 705), and the key is two values
(a router keys on method and path). No GraphBLAS step: one def-use walk per
long fn.
Worked example (its own fixture, tests/graph_lenses.rs). Crate web:
fn route(path: &str) spans lines 1–600 and holds 30 str::eq calls, 24 with
argument _1 (the parameter path) and 6 with _7; huge() (900 lines) has
no MIR. → ROUTER web route() — 600 lines, 30 string compares, 30 (100 %) on the top 2 values [param path ×24, local _7 ×6], without_mir 1 and the
needs the MIR of the long fns without it row. With --key-values 1 --min-share 0.9 (24/30 = 80 %) no row.
On Holger h32/next 9dbe5cbf (golden monolithic-router). 7 fns ≥ 500
lines, all with MIR (the schema-6 dump has no block cap: the plan's "handle has
no MIR" no longer holds); 1 ROUTER: exposed::http::handle (http.rs:883,
4,071 lines) — 85 string compares, 55 (65 %) on the top two values: 50 on
the async body's captures (c1._1, the request: path == ×31 and method == ×29 in the source, through match (method, path) tuples) and 5 on a nested
body's. Hand check: true — the review's R3 router; the other six long fns
(start_grpc_endpoint, run_main, Holger::start …) compare ≤ 1 literal.
Three tracer fixes on the real dump took handle from 36 % to 65 %: tuple
projections, the PartialEq<&str> compares, and — the decisive one — keeping
the def-use walk inside one body (an async fn's coroutine re-uses local
numbers, so a walk across bodies read Instant::now as the path's source).
Bench: Holger 1.8 s / 2.2 GB, dwarves 0.4 s / 1.1 GB, gunnar 0.9 s / 2.3 GB
(cached graph).
What it cannot say. A route table spread over several mid-size fns (none ≥ min_lines) is not one finding; a router matching on an enum of parsed routes is already the cure.
4.23 reach-set (plan B10)
The question. What can this entry reach, and does it pass the check? The
generic engine behind gate-bypass: from a set of entry fns, the closure over
CALLS with every dyn Trait call expanded to every impl, the crates it spans,
the tables it touches (TOUCHES_TABLE) and the syscalls it makes
(SYSCALLS). With --must-reach P it answers the review's R3 as a question:
every entry that reaches NO fn matching P is MISSING ("this handler never
reaches the auth check").
Run. cargo lens reach-set --entry PAT… [--must-reach PAT…] [--max-hops N]
(door: dwarves RepoLens::ReachSet).
Inputs. fn nodes, CALLS, IMPLEMENTS + CONTAINS (the dyn expansion of the index: trait method → every impl's same-named method), TOUCHES_TABLE, SYSCALLS.
Dataflow.
fn nodes (dependencies included; test code out unless --include-tests) → compact 0..n
A ∈ 𝔹^{n×n} = CALLS ∪ DYN (trait method → impl methods)
--entry patterns → entry fns (in scope; one per label: a crate dumped as lib and test target
carries a fn twice)
reach_each(entries, A) — batched BFS: up to 256 entries per frontier matrix [GraphBLAS]
→ R(e) per entry → |R(e)|, local fns, crates, TOUCHES_TABLE / SYSCALLS targets of R(e)
--must-reach → targets T; MISSING(e) ⇔ R(e) ∩ T = ∅; REACHES otherwise
The math. On the boolean semiring (LOR, LAND) — ⊕ = ∨, ⊗ = ∧:
R₀ = E (the entries' one-hot rows, a b × n frontier F₀)
F_{k+1}⟨¬V_k⟩ = F_k ⊕.⊗ A (masked mxm: only not-yet-visited columns)
V_{k+1} = V_k ∨ F_{k+1}; R(e) = V_∞[e, :] (or V_{max_hops})
tables(e) = { t : ∃ f ∈ R(e), TOUCHES_TABLE(f, t) }
syscalls(e) = { s : ∃ f ∈ R(e), SYSCALLS(f, s) }
MISSING(e) ⇔ R(e) ∩ T = ∅
The dyn expansion over-approximates (every impl of the trait), which is the safe side for "can it write there" and the optimistic side for "does it reach the check" when the check sits behind a trait; the run prints how many dyn edges the walk used.
Worked example (the fixture). http_handle → gate_ingest → Store::put →(dyn) Disk::put → fs_write, and fs_write TOUCHES_TABLE artifacts;
mirror_run → Store::put directly; Fake::put (a test impl) is left out. With
--entry "app http_handle()" --entry "app mirror_run()": REACH app http_handle() — 5 fns (5 local) in 1 crates, tables [artifacts]; no SYSCALLS
edges → a needs row. With --must-reach "app gate_ingest()": MISSING app mirror_run(), REACHES app http_handle(). An entry pattern matching nothing
is an error, never an empty answer.
On Holger h32/next 9dbe5cbf (golden reach-set, the gate's nine entry
patterns, must-reach auth::authorize_read_identity). 217 entry fns reach
5,958 fns (692 dyn edges); 11 REACH the read-auth check — exposed::http:: handle (4,435 fns in 104 crates) and the gRPC read doors (search,
fetch_artifact, stream_artifact, list_artifacts, list_repositories,
archive_info, list_archive_files, get_properties, fetch_sbom) — 206 are
MISSING (job runner, import door, replication, admin and promotion RPCs: they
authorize through other checks or none — a question for the owner's target
set, not a lens fault). No TOUCHES_TABLE / SYSCALLS edges in the Holger,
dwarves or gunnar dumps (named). From main(): Holger 11 mains reach 7,251
fns, dwarves 11 → 4,164, gunnar 21 → 7,166. Hand check: the 11 REACHES rows
true (each read door calls the read gate), 3 MISSING rows checked (JobRunner:: run, become_twin, replicate_one_pair: none calls it) true. Bench: Holger
1.0 s / 2.5 GB, dwarves 0.4 s / 1.1 GB, gunnar 0.7 s / 2.4 GB (cached graph).
On the lossless graph (schema 8, lens upgrade batch 2). The call pairs come from every call site of the rows as well as from the CALLS edges:
S(f) = { callee(s) : s a call site of f's bodies } st_callee, resolved by label (crate + short id; the defining node over a stub)
extra(f) = { c ∈ S(f) : c a fn node, ¬derived(c), c ∉ calls(f) }
A = pattern(calls ∪ extra ∪ dyn) LOR.LAND reach as before; A_new ⊇ A_old
Measured on the schema-8 dumps (one pass over every body in scope: Holger 0.59 s,
Gunnar 0.51 s): Holger h32 has 1,668 call sites whose callee has no CALLS edge,
Gunnar 1,158; of those 1 (Holger) and 28 (Gunnar) name a fn defined in the
workspace, the rest are implicit trait calls into dependency stubs
(Iterator::next of a for, Deref::deref). The review's miss, Holger
attach_sbom → authorize_write<T> (services.rs:4270), HAS a CALLS edge in the
schema-8 dump: the driver closed that gap before this lens did, so no golden row
moves on either dump. Summary key calls_from_sites. Fixture
reach_set_reaches_through_a_generic_call_that_has_no_calls_edge.
What it cannot say. A call through a fn pointer or a closure stored in a field has no CALLS edge (not followed); tables and syscalls need their producers (the driver's SQL resolution, the binary layer).
Round 2 (lane LENS-POLISH, 2026-11-05; re-compare 2 wrong results 2 and 3).
Why attach_sbom was MISSING. Not a lost edge: the dump has attach_sbom → authorize_write
(CALLS and the MIR call block). authorize_write carries #[cfg(any(feature = "intake", test))], and Graph::is_test_code took any cfg that NAMES test as test-only, so the walk
dropped it (and with it the way to auth::validate_request). The rule is now
cfg_test_only (lenses/mod.rs), a three-valued evaluation with test false:
v(test) = F v(any other atom, `key = "value"`) = ? v(not p) = ¬v(p), ¬? = ?
v(all(p…)) = F if some F, T if all T, else ? v(any(p…)) = T if some T, F if all F, else ?
test_only(cfg) ⇔ v(cfg) = F
cfg(test), cfg(all(test, unix)) → test-only; cfg(any(feature = "intake", test)), cfg(not(test)) → not
It is the crate's one test rule, so every lens's production scope moved a little on Holger:
51 more production fns (size fns 6,463 → 6,514, twins fns 6,398 → 6,449, verdict "2989
tests reach 5622 of 6449 fns"); the golden diff lists each row before → after in the commit.
Holger B10 auth (58 gRPC + HTTP entries, auth/authorize, auth/validate_request()):
MISSING 10 → 8; attach_sbom REACHES auth::validate_request in 3 hops.
The hit and its path. A REACHES row now names the nearest target and one shortest path:
d(e) = 0, d(v) = min_{u : A(u,v)} d(u) + 1 (BFS from e, parents kept, plain calls before dyn)
t* = argmin_{t ∈ T ∩ R(e)} (d(t), label(t))
path = e → … → t*, each hop {fn, file:line, dyn: the edge into it is a dyn expansion}
JSON: first_target, path[{fn, loc, dyn}], through_dyn; text: → <t*> in k hops (through dyn: …). =<label> in --must-reach takes that one fn exactly. On Holger,
ProxyBackend::fetch and replicate_one_pair "reach" IntakeChain at
available_under, through dyn ChainView::fetch: a read of the chain, which the row now
says. Fixture reach_set_passes_cfg_any_test_code_names_the_hit_and_its_path.
Must-pass (lane LENS-FIX4, 2026-11-06; Gunnar re-compare FP 5). "Can reach" is
not "must pass". Gunnar's require_log_reader (mirror.rs:60) returns Ok(()) for a
member of group twin (return Ok(()), mirror.rs:81) before it calls require, so
log_head / log_fetch / snapshot_fetch REACH may_in but need not pass it. Every
REACHES row's path is now checked hop by hop on the hop fn's MIR CFG:
path(e) = p₀ → p₁ → … → t
C(pᵢ) = { b ∈ blocks(pᵢ) : term(b) calls pᵢ₊₁, or the trait method pᵢ₊₁ implements }
skip(pᵢ) ⇔ a walk bb0 → … → b with term(b) = return exists that never enters C(pᵢ) and
writes `_0 = adt~0(…)` (Ok) on the way — any such return, for a `()` fn
SKIPPABLE(e) ⇔ ∃ i: skip(pᵢ); else REACHES (must-pass along this path)
That is "C(pᵢ) post-dominates bb0 on the success returns", decided by one DFS over
(block, Ok-written) states — no GraphBLAS step. An early ? writes _0 through
FromResidual::from_residual (a call, not an Ok aggregate), so an early Err is no
skip. A hop is undecided (hops_undecided) when the fn has no MIR, the call to the
next hop sits in a nested body (a closure, an async fn's coroutine: its _0 is not the
fn's), or the return type is neither Result nor (). The row names the hop fn, the
file and the line of the Ok (MIR positions are relative to the fn's start line):
SKIPPABLE … — SKIPPABLE: <fn> returns Ok at <file>:<line> without calling <next>;
--json adds skip {fn, file, line, before} and hops_undecided.
Fixture (reach_set_calls_a_path_skippable_when_a_hop_returns_ok_before_the_next):
rpc_lax → reader → check, where reader is if is_member() { return Ok(()) } check() → SKIPPABLE at src/lib.rs:80; rpc_strict → strict → check → REACHES;
rpc_try → tried → check, where tried is parse()?; check() (the early arm through
from_residual) → REACHES. Gunnar c17a6ef4 (golden
golden_gunnar_reach_set_the_log_readers_can_skip_may_in): the three log readers are
SKIPPABLE at require_log_reader, mirror.rs:81 (before: REACHES). Beyond: the mirror
import's Protection::open returns Ok(None) before BranchProtectionGate::new when a
repo has no protection rules (mirror_fetch.rs:508), so the import is SKIPPABLE for that
gate — by design, and now visible.
4.24 history-comments (plan B4)
The question. How much of the comment text narrates how the code changed
instead of what it does? Dates (2026-11-04), lane names (lane lens-build), the owner, ticket tags (H32-P0, S2-3), change words
(ADDED, REMOVED, no longer, used to, previously). History belongs in
git; a comment that tells it rots the day the code moves again. The review's
R13 (2,653 lines on Holger, 1,005 dated, 1,676 lane).
Run. cargo lens history-comments --src DIR [--top N] (door: dwarves
RepoLens::HistoryComments); default 25 files listed.
Inputs. The graph names the files: every file of a node in scope (the
workspace's own hand-written code: no tests, no generated, no derive output,
the usual Scope rules). The text comes from --src — the one TEXT lens of
the crate; the graph carries no comments, and without --src the lens says so
in a needs row and prints only the file count.
Dataflow.
nodes in scope → their `file` (*.rs, relative) → the file set
each file from --src → a line scanner:
`//`, `///`, `//!` to the end of the line; `/* … */` across lines (nested);
string literals skipped ("// in a string" is no comment), char literals ('"') too
→ each comment line tested against the categories:
dated `20dd-dd` (not inside a longer number)
lane `lane <id>` where <id> has a `-`, a digit, or is upper case (`the lane count` is not)
owner the word `owner` / `owner's`
ticket `[A-Z][A-Z0-9]{1,5}-<alnum>` with a digit before the dash (`H32-P0`, `S2-3`;
`UTF-8`, `SHA-256` have none)
change ADDED / REMOVED / MEASURED / FIXED / CHANGED / RENAMED / MOVED, `no longer`,
`legacy`, `used to`, `previously`, `formerly`, `was renamed / moved / removed`
per file: lines, comment lines, history lines (a line counts once), per category;
totals; the files ranked by history lines
The math. For file f with lines L(f), comment lines C(f) ⊆ L(f) and
the category predicates p_k:
H(f) = { l ∈ C(f) : ∃ k p_k(l) } (a line counts once)
n_k = Σ_f |{ l ∈ C(f) : p_k(l) }| (a line may count in several categories)
ratio = Σ_f |H(f)| / Σ_f |C(f)|
comments = Σ_f |C(f)| / Σ_f |L(f)|
No GraphBLAS step: the graph only names the files.
Worked example (its own fixture). One fn in src/hist.rs; the file has 8
lines: a //! doc line, /// ADDED 2026-11-04 (lane lens-build): the cap, a
string holding // not a comment followed by // the owner asked for H32-P0,
// reads up to 2048 bytes, and a two-line block comment no longer reads the cache / UTF-8 and SHA-256. → 6 comment lines, 3 history lines (dated 1, lane
1, owner 1, ticket 1, change 2: ADDED and no longer); 2048, UTF-8,
SHA-256 and the string count nowhere. Without --src: needs the comment text.
On Holger h32/next 9dbe5cbf (golden history-comments, the 9dbe5cbf
checkout as --src). 353 files, 265,679 lines, 44,151 comment lines
(16.6 %), 1,775 history lines (4.0 %): 364 dated, 946 lane, 203 owner, 220
ticket, 247 change. Top files: server/lib/src/lib.rs 156 of 2,489,
exposed/http.rs 95, server/cli/src/paired.rs 87, promote/intake.rs 64,
errcode 63, auth.rs 50. The review's 2,653 counted every .rs (tests
included); this run is production code only. Hand check (10 sampled lines): 8
true ((lane M4), S2-3., it used to hand writes …, owner rule, 2026-10-23, SEC-REVIEW 2026-11-01 P3 …), 2 weak (a bundle id previously accepted was presented again describes behaviour; the owner's YubiKey key
names the owner without telling history). dwarves 1,144 / 43,285 comment
lines, gunnar 2,934 / 85,585 (both against the current checkouts, not the
dumps' commits). Bench: Holger 0.8 s / 2.4 GB, dwarves 0.4 s / 1.1 GB, gunnar
0.8 s / 2.6 GB (cached graph; the text scan is the rest).
What it cannot say. Which comments are wrong — only which ones narrate. The categories are word rules, configurable only in the code.
4.25 export-refactor-envelope (plan H11, "scout to carpenter")
The question. What does a coding agent need to act on ONE finding, and no
more? The finding itself, the full signatures of the items it names, and the
exact source lines of each — a bounded slice, not the repo. Every lens of this
crate names nodes by label (<crate> <short moniker>) in its --json
findings, so any of them can be enveloped: gate-bypass doors, the H1–H10 rows,
twins, dispatch ladders.
Run. cargo lens export-refactor-envelope --report FILE --finding N [--src DIR] [--commit REV] [--max-items N] [--max-lines N] (door: dwarves
RepoLens::ExportRefactorEnvelope); defaults 40 / 200. --report is a lens's
--json output; --finding is the position in its findings (stable: every
report is byte-stable). With --json the envelope is findings[0].
Inputs. The report file; from the graph: the label of every item node, its
file / start_line / end_line, param / ret children, CONTAINS,
IMPLEMENTS, FOR_TYPE; the source text from --src — at --commit through
gix (commit → tree → blob, a library: no git process), else the working
tree. The envelope feature (in lenses) carries the gix dependency.
Dataflow.
--report FILE → findings[N] (kept verbatim)
→ every string in it with its JSON path (`door`, `path[1]`, `sites[0].fn`)
→ the label index: label → node (fn / method / struct / enum / union / trait / impl /
field / variant / type / mod / const / static; the defining row wins over a stub)
→ N(F) = the nodes named, first-seen order, ≤ max_items (the rest counted: items_dropped)
→ per node: kind, file, start_line … end_line, named_at (the JSON paths), signature:
fn / method name(param types) -> ret, + the trait method it implements and the
impl's self type, or the trait it is declared in
trait every method's signature + the impl count
struct/union every field `name: ty` enum every variant impl trait + type
→ text = lines start … end (≤ max_lines, `truncated` says so) from the source;
resolvable ⇔ the file is there (at the commit) and end ≤ its line count
envelope = { finding, source {commit | tree}, items[], spans, resolvable, items_dropped, limits }
The math. An envelope is the finding projected onto the graph:
L(F) = { (p, s) : s a string at JSON path p of finding F }
N(F) = { ℓ⁻¹(s) : (p, s) ∈ L(F), s ∈ dom ℓ } (ℓ = the label map)
item(n) = (kind(n), span(n), sig(n), text(span(n))) for the first max_items n ∈ N(F)
|envelope| ≤ max_items · (max_lines · line + |sig|) + |F|
No GraphBLAS step: one label index, one lookup per string.
Worked example (the fixture). gate-bypass on the fixture reports BYPASS app mirror_run() (mirror_run → Store::put). The envelope of that finding,
with --src a tree whose src/lib.rs holds line 1 … line 70: the item app mirror_run() (fn, named at door, entry, path[0]) with the text line 22 … line 30 (its span) and resolvable: true; the item app Store#put()
(method, signature.trait = "app Store#"); resolvable = spans. With
--max-items 1 one item and items_dropped ≥ 1; --finding 999 is an error
naming the report's finding count.
On Holger h32/next 9dbe5cbf (golden export-refactor-envelope). The
gate-bypass report from the gate's sinks, choke and entries; its first BYPASS
row (finding 1: exposed::http::handle → RepositoryBackendTrait:: handle_http2_request_bytes), enveloped with --src the Holger checkout and
--commit 9dbe5cbf: 2 items, 2 spans, 2 resolvable at the dump's commit
(9dbe5cbfcfbaa64dc65f3be7ccd76c242d850893, read through gix). handle's
signature handle(Request<Incoming>, Arc<GatewaySettings>, Option<String>, String) -> impl Future<…>, its 4,071 lines cut to 200 (truncated); the
sink's handle_http2_request_bytes(&Self, &str, &str, &[u8]) -> Result<(u16, Vec<(String, String)>, Bytes), Error> with its trait. 14 KB of JSON. The
golden test asserts every local span is resolvable at the commit. Bench: 7.0 s
/ 1.9 GB (the label index over the whole dump plus the git open; cached graph).
What it cannot say. Strings in a finding that are not labels (a loc, a
class word) stay in the finding and resolve nothing; a node without a span
(a reference stub, a trait method declared in a dependency) has its signature
but no text. Kani harnesses / proptest invariants generated FROM an envelope
are the owner's call (plan H addendum).
4.26 package-graph views (viz::package_graph: Holger's metro and arch)
Not a dump lens: a renderer. It takes a package graph as JSON and returns the
SVG of Holger's two Security-tab subviews, "metro" and "arch", in the skade
house style of nornir's architecture board. Holger runs it inside a Heimdall job,
so it is a pure library call (crates/mimir-lens/src/viz/package_graph.rs):
use mimir_lens::viz::package_graph::{Options, PackageGraph, View, render};
let g = PackageGraph::from_json(&json)?; // parse + validate (package-graph/1)
let metro: String = render(&g, View::Metro, &Options::default());
let arch: String = render(&g, View::Arch, &Options::default());
- Feature
viz(indefault; no dependency beyond serde, no GraphBLAS). Holger takesnornir-mimir-lens = { default-features = false, features = ["viz"] }. - Pure and deterministic: no file or network I/O, no hash-map iteration, integer coordinates. The same graph gives the same bytes whatever the order of its nodes and edges (tested with a reversed and rotated copy).
- One renderer. The palette, fonts, chips, traces and the layering rule are
crates/mimir-lens/src/viz/skade.rs; nornir's own architecture board (src/arch/mod.rs,ArchGraph::to_svg) draws through the same functions (skade::open,board,trace,chip,kahn_layers), so the two boards cannot drift apart.
Input: schema package-graph/1
{
"schema": "package-graph/1",
"nodes": [
{ "purl": "pkg:cargo/hyper@1.7.0", "name": "hyper", "version": "1.7.0", "ecosystem": "cargo", "reached": true },
{ "purl": "pkg:cargo/h2@0.4.12", "name": "h2", "version": "0.4.12", "ecosystem": "cargo", "hit": "RUSTSEC-…" }
],
"edges": [
{ "from": "pkg:cargo/hyper@1.7.0", "to": "pkg:cargo/h2@0.4.12", "scope": "normal", "reached": true }
]
}
| field | type | rule |
|---|---|---|
schema | string | exactly package-graph/1 |
nodes[].purl | string | the identity; pkg:<type>/<name>…, unique |
nodes[].name, ecosystem | string | non-empty |
nodes[].version | string | optional (empty: the label is the bare name) |
nodes[].hit | string or list | advisory ids; none empty; a non-empty list marks the node hit |
nodes[].reached | bool | on a blast-radius path (it pulls a hit package in) |
edges[].from, to | purl | both are nodes, from ≠ to; from depends on to |
edges[].scope | normal | dev | build | default normal |
edges[].reached | bool | on a blast-radius path: from is reached, to is hit or reached |
Unknown fields are refused (deny_unknown_fields). PackageGraph::validate
returns every error in input order (SchemaError: Schema, EmptyField,
NotAPurl, DuplicatePurl, EmptyAdvisory, UnknownEndpoint, SelfEdge,
DuplicateEdge on the same (from, to, scope), ReachedEdgeOffPath);
from_json joins them into one message. render itself is total: on an invalid
graph it skips unknown endpoints and self edges.
Math
Canonical order. Nodes are sorted by purl, index i ∈ 0..n; edges are
deduplicated on (from, to, scope) (reached = OR over the duplicates) and sorted.
Every later choice breaks ties by this index, so input order never shows.
Depth. Let E′ = { (u, v) ∈ E : scope(u, v) ≠ dev } (dev edges close cycles
in cargo graphs and do not ship). The layer is level-synchronous Kahn over E′
(skade::kahn_layers, the arch board's rule):
L(v) = 0 if v has no in-edge in E′
L(v) = 1 + max { L(u) : (u, v) ∈ E′ } otherwise (the longest path from a source)
When no node is ready but some remain (a cycle in E′), every remaining node
takes the current level. With D = max L + 1 layers and at most C columns
(max_columns, default 32), a deep graph folds layers into bands:
b(v) = L(v) if D ≤ C
b(v) = ⌊ L(v) · C / D ⌋ otherwise
Arch: one column per band, folded when crowded. Column B_b = { v : b(v) = b }
is sorted by (rank(v), i) with rank = 0 hit, 1 reached, 2 unmarked. With
k = max_per_column (default 40):
fold(b) = |B_b| > k (Cluster::Auto; Never: false; Always: true)
keep(b) = |B_b| if not fold(b)
= min(|{v ∈ B_b : rank(v) < 2}|, ⌊k/2⌋) if fold(b)
The first keep(b) packages get a chip each; the rest fold into one chip per
ecosystem of the column (cargo ×412 · 2 hit, a stacked card). So a column
holds at most max(k, ⌊k/2⌋ + E_b) chips (E_b ecosystems in the fold), the
board at most C · max(k, ⌊k/2⌋ + E), whatever n is; hits and reached packages
are the last to fold.
Chips U, V are joined by one trace per scope s:
w(U, V, s) = |{ (u, v) ∈ E : unit(u) = U, unit(v) = V, scope = s }| (U ≠ V)
stroke(U, V, s) = min(1.2 + 0.6 · ⌊log₂ w⌋, 4.2) (+1 when any of them is reached)
Edges inside one fold are counted in its tooltip. Placement is the arch board's:
x = M + b · 240, y = H + M + (R − |col|) · 56 / 2 + r · 56 (M = 28 margin,
H = 76 header, R the tallest column, r the row), chip 188 × 34, and a trace
is the cubic M x₁,y₁ C xₘ,y₁ xₘ,y₂ x₂,y₂ with xₘ = (x₁ + x₂) / 2 from the right
edge of U to the left edge of V.
Metro: the blast radius as lines. Let R ⊆ E be the reached edges and
S = { v : (hit(v) ∨ reached(v)) ∧ in_R(v) = ∅ ∧ (out_R(v) ≠ ∅ ∨ hit(v)) }
the packages on a path that nothing reached depends on (the apps that pull a hit
in). For each s ∈ S in index order, the line is the BFS shortest path along R
from s to its nearest hit (neighbours in index order). A hit no line covers then
gets the reverse BFS path from its own nearest source. Identical lines are drawn
once; at most m = max_lines (default 24) lines are drawn and the rest are
counted (+N lines not drawn). Without any mark the view shows the spine
instead: h(v) = 1 + max { h(w) : (v, w) ∈ E′, L(w) > L(v) }, one line per root
(L = 0) along the argmax successor, roots by (−h, i).
Stations sit in the columns of the bands that hold one (c = the rank of b(v)
among the used bands). Line j claims track j; a station keeps the track of the
first line through it, and a slot (c, t) that is taken pushes it to t + 1.
x = M + 48 + 100 + c · 200, y = H + M + 30 + t · 64. A hop is octilinear:
horizontal, then a 45° run of d = |y₂ − y₁| into the station (straight when
|x₂ − x₁| < d). A station on two lines or more is an interchange.
Drawing.
| mark | arch | metro |
|---|---|---|
| hit | red chip (#d1344e stroke, #3f1420 fill, 2.6 px), tooltip with the advisory ids | red station r = 9, ids under it |
| reached node | amber stroke (#e0973a, 2.2 px) | amber ring r = 7 |
| reached edge | amber trace, +1 px, amber arrow pkg-arrow-hot, drawn last; other traces fade to 0.4 | amber halo (11 px, 0.35) under the hop |
| normal | cyan #3aa0d0, solid | solid line |
| dev | steel #5a8fb0, dashed 6 4 | dashed 10 6 |
| build | violet #9a6cd0, dotted 2 3 | dotted 2 6 |
Ecosystems take the arch board's kind chips (cargo steel, npm green, pypi blue,
oci cyan, maven violet, golang teal, deb/rpm/apk magenta); amber and red stay free
for the marks. Classes and data attributes (g.pkg.hit[data-purl],
path.dep.dev.reached, g.station.interchange, g.pkg.cluster[data-members])
let a console style or pick elements without parsing labels.
Dataflow
Holger (Heimdall job)
package graph rows ──serde──▶ PackageGraph (package-graph/1) ──validate──▶ errors | ok
│
▼ Prep: sort by purl, dedup edges, E′ = non-dev
skade::kahn_layers(E′) ─▶ L(v) ─▶ bands b(v) (≤ C columns)
│ │
View::Arch ▼ ▼ View::Metro
columns B_b, fold > k → ecosystem chips S (sources on R) ─BFS─▶ lines (≤ m)
unit edges w(U,V,s) → traces stations: band column, first track
│ │
└──── skade::open / board / trace / chip / legend ────▶ SVG String
Worked example: the tiny fixture by hand
crates/mimir-lens/tests/golden/package_graph/tiny.json: 12 packages (9 cargo,
3 npm), 15 edges, hits h2 (FIXTURE-0001) and lodash (FIXTURE-0002,
FIXTURE-0003), reached holger-server, axum, hyper, holger-console, vite.
E′drops the two dev edges (holger-server → proptest,holger-console → vite).- Layers:
L = 0holger-server, proptest, holger-console, vite (no in-edge inE′);L = 1axum, ring, serde (from holger-server and proptest), lodash;L = 2hyper, cc (ring → cc, build);L = 3h2;L = 4tokio (its deepest consumer is h2 at 3).D = 5 ≤ 32, sob = L. - Arch: 5 columns, the tallest has 4 chips, nothing folds (
4 ≤ 40). Column 0 by(rank, purl): holger-server, holger-console, vite (reached), then proptest. Board56 + 5 · 240 = 1256×76 + 56 + 4 · 56 = 356. 15 traces: 5 amber (the reached edges), 7 cyan, 2 dashed steel, 1 dotted violet. - Metro:
R= holger-server → axum → hyper → h2, holger-console → lodash, vite → lodash (build).S= holger-server, holger-console, vite (in purl order). Three lines, seven stations; lodash is on lines 2 and 3 (an interchange). Bands used 0..3 → 4 columns. Line 1 on track 0; line 2: holger-console (0, 1), lodash (1, 1); line 3: vite (0, 2), and its hop to lodash runs horizontally, then 45° up one track. Board56 + 48 + 4 · 200 = 904×76 + 56 + 3 · 64 + 24 = 348.
The rendered fixture (these are the goldens, byte for byte; a test keeps them equal):
Bench
crates/mimir-lens/examples/package_graph_bench.rs on fixture::synthetic(n, 7)
(release, feature viz only, build-1 under load 16–21 on 16 cores, 2026-11-05;
median of 7 renders; RSS = the process peak VmHWM: the graph, its JSON round
trip and the SVGs):
| n | edges | hits / reached | JSON | parse + validate | metro | arch | SVG metro / arch | peak RSS |
|---|---|---|---|---|---|---|---|---|
| 2,000 | 3,646 | 4 / 197 | 0.79 MB | 2.4 ms | 1.0 ms | 2.3 ms | 62 KB / 175 KB | 6.0 MB |
| 10,000 | 18,328 | 8 / 702 | 3.9 MB | 12.9 ms | 4.2 ms | 7.0 ms | 81 KB / 238 KB | 19.3 MB |
The SVG size is bounded by the folding, not by n (5× the packages, 1.3× the
bytes); render time is linear in V + E (layering, folding and unit edges are
single passes, the metro does at most m BFS walks over the reached edges).
Tests
crates/mimir-lens/tests/package_graph.rs (cargo test -p nornir-mimir-lens --no-default-features --features viz --test package_graph): the schema round
trip and both hit spellings; the validator, one case per error; the generated
2,000-package graph is valid and deterministic; golden SVGs, byte-stable (tiny:
the SVG files; 2,000 packages: length + FNV-1a 64 in gen2k-*.txt); input order
does not change a byte; hit and reached marks appear (2 hit chips, 5 reached
chips, 5 amber traces, 3 lines, 5 halos); the scopes are distinct (dash and colour
per scope, also on a reached edge); 2,000 packages stay readable (every package
drawn once, alone or folded, every hit on its own chip, at most 24 lines); deep
graphs fold into 32 bands and an unmarked graph draws spines; the docs images are
the tiny goldens. UPDATE_GOLDEN=1 rewrites the goldens and the docs images.
skade.rs pins the Kahn layering and the trace and chip bytes of the arch board.
Limits
- The metro draws at most
max_lineslines; with many sources the rest is a count, not a picture. A station's column is its depth band, so a line that steps back (a reached dev edge, a cycle) runs right to left for that hop. - Folding is per column and per ecosystem; a column with more than
⌊k/2⌋marked packages folds the surplus marked ones too (the fold chip then carries the hit count in red). - Labels are clipped at 26 characters (
name@version); the full purl and the advisory ids are in the<title>tooltip.
4.27 reflexion (plan M6, the macro core)
The question. Given the architecture we want, where does the code agree
with it, where does it break it, and what is missing? This is the reflexion
model of Murphy, Notkin & Sullivan (Software reflexion models: bridging the
gap between source and high-level models, SIGSOFT FSE 1995), computed as
masks on the room's matrices (macro-lenses-plan.md
§2 M6 and §5b).
Run. cargo lens reflexion [--spec FILE] [--no-infer] [--edges-per-pair N] [--src DIR] (door: dwarves RepoLens::Reflexion). The spec is --spec,
else <--src>/.nornir/architecture.ron; with neither, every crate is a
component and the layers come from the Cargo DAG.
The spec (.nornir/architecture.ron). Ordered layers (top to bottom),
components (name → globs), allow, forbidden, expected, and the edge
exceptions allow_edges and via. The rules are read in order: layers gives
the default T (a component may depend on every layer below its own; a layer
entry with *, such as "svc-*", is a glob over component names, and the
components it matches share that layer), allow adds pairs, forbidden
removes pairs and always wins over allow, expected names the pairs whose
absence is a finding; a component may always use itself. Edge exceptions
beat every pair rule: an item edge whose source matches the first and target
the second fqn glob of an allow_edges pair, or whose source matches a via
glob (a decorator's forwarding methods, fqn:*[Decorator]*), counts as
convergent and is listed as EXCEPTED with its rule and file:line
(excepted_edges, excepted_weight in the summary; M7 and arch-report
count it the same way). The validator refuses a spec with no components, a
component without globs, an unknown glob kind, the same glob in two
components, an unknown or repeated layer, a layer glob that matches no
component, a component in two layers, a crate: / mod: glob or an empty
glob in allow_edges / via, a rule naming an unknown component or a
component and itself, and an expected pair that is forbidden; it warns when a
pair is both allowed and forbidden and when a component is in no layer.
Listing caps: --edges-per-pair 0 lists every edge of a divergent pair;
the door passes 0 when --limit 0 is given (the same for the unmapped and
inferred module lists).
The mapping M. A node goes to the component of its most specific
matching glob: fqn: over mod: over crate:, and the glob with more
literal characters within a kind. crate: matches the package name (- and
_ the same), mod: the module path crate::a::b by segments (* inside a
segment, ** any number of segments, mod:x::** holds x), fqn: the
string crate::path::Item#member() (the SCIP descriptor with / as ::;
* any run, ? one character). An unmapped item in a crate that holds items
of two or more components takes the component most of its Leiden community's
mapped members have, marked inferred with that share α (--no-infer turns
this off). Unmapped items are listed by module, never dropped.
Inputs. The item graph A: fns / methods, types (struct, enum, union,
trait, impl, type alias, const, static) and enum variants; every edge end
rolled up to its item (a param, local or MIR block to its fn, a field to its
struct). Edges: CALLS (weighted by call sites), USES_TYPE, READS_FIELD,
WRITES_FIELD, IMPLEMENTS (one per dump edge). The source must be in scope
(hand-written production code); a dependency target stays only when a
crate: glob names its crate. CONTAINS gives the module paths; DEPENDS_ON
between crate nodes (dev targets left out) gives the no-spec layers; the MIR
call terminators give the call sites' lines.
The math. A ∈ ℕ^{n×n}, M ∈ 𝔹^{n×k} (one 1 per mapped row),
T ∈ 𝔹^{k×k} with the diagonal set, E ∈ 𝔹^{k×k} the expected pairs.
| step | formula | semiring / op |
|---|---|---|
| volume | H = Mᵀ · (A · M) | PLUS.TIMES on UINT64, two GrB_mxm |
| convergence | H⟨T⟩ | GrB_apply(IDENTITY_UINT64) under the structural mask T (GrB_DESC_S) |
| divergence | H⟨¬T⟩ | the same under the complemented mask (GrB_DESC_SC) |
| absence | E⟨¬H⟩ | GrB_apply(IDENTITY_BOOL) of E under the complement of H's structure |
| concrete edges | A_{a→b} = diag(M[:,a]) · A · diag(M[:,b]) | read off the edge list (rows mapped to a, columns to b) |
| conformance | κ = Σ H⟨T⟩_{a≠b} / (Σ H⟨T⟩_{a≠b} + Σ H⟨¬T⟩) | sums |
A divergence carries its rule: forbidden (the pair is in forbidden),
layer (it points up the layer order) or not-allowed (no rule allows it).
The line of a call is the caller's first line plus the MIR position's line
(a MIR @L:C counts from the fn's own first line: Holger's handle starts at
http.rs:883 and its handle_http2_upload call is @3534, file line 4417).
Dataflow.
dump ─▶ items + A (n×n u64: CALLS sites, USES_TYPE, READS/WRITES_FIELD, IMPLEMENTS) ─┐
architecture.ron ─▶ globs (fqn > mod > crate, longer wins) ─▶ M (n×k) ──────────────┼─▶ H = Mᵀ·A·M
└─▶ layers → T, + allow, − forbidden ─▶ T (k×k bool) ──────────────┼─▶ H⟨T⟩ convergent
└─▶ expected ─▶ E (k×k bool) ──────────────────────────────────────┤ H⟨¬T⟩ divergent ─▶ A_{a→b} ─▶ file:line
Leiden(sym A) ─▶ the unmapped items of a multi-component crate ─▶ M (inferred, α) ──┘ E⟨¬H⟩ absent ─▶ κ
no spec: crates ─▶ M; DEPENDS_ON ─▶ longest-path layers ─▶ T; direct deps ─▶ E
Worked example (the fixture macro_fixture, tests/graph_lenses.rs; the
plan's §4). Crate app, modules http intake gate store formats misc, one
fn each; calls http→intake, intake→gate, gate→store, http→store, intake→formats, formats→intake, misc→gate. Spec: one mod: glob per module
except misc, allowed http→intake, intake→gate, gate→formats, gate→store,
expected intake→gate. Then H has six off-diagonal entries of weight 1:
convergent 3, divergent 3 (http→store, intake→formats,
formats→intake), no absence, κ = 3/6 = 0.50; the divergent http→store
is listed as app http/handle(). → app store/put(). (CALLS ×1) at src/http.rs:13 (fn line 1 + MIR @12), and app::misc ×1 is unmapped. With
inference on, the helper takes its Leiden community's component. Adding
fqn:app::formats::parse* to intake (an fqn glob beats the module glob)
folds both intake⇄formats edges into intake: κ = 3/4; an expected
formats→store is then reported ABSENT. A spec naming an unknown component
is refused with the validator's reason. With no spec, the crates app, lib,
unused (app depends on lib and unused) give two layers: app→lib
converges, lib g() → app store/put() diverges [layer], and the declared
app → unused with no code edge is ABSENT.
On Holger h32/next 9dbe5cbf (goldens reflexion, reflexion-crates). The
spec is the fixture tests/golden/holger-architecture.ron, written from the
review's "As it should be" (§ONE PAGE, §7(b)): layers cli boot doors server gate checks formats format-api store-write base; store-write = the write
half of RepositoryBackendTrait (put, put_from_reader, handle_http2_upload,
handle_http2_request_bytes, delete_artifact, the dyn write_from_reader),
which only gate may call (R1); formats = the format crates, inspect, index
rows and the handlers, which nobody above them may name (R2). 8,365 items,
28,848 edges (weight 50,512); κ 0.861 (convergent 8,443 / divergent 1,360
in 15 pairs), 0 absences, 110 unmapped items (all in holger-mannequin, a test
double the spec leaves out).
- R1,
doors → store-write[forbidden], 12 edges, each at the review's line:exposed::http::handle→handle_http2_upload(http.rs:4417),handle_http2_request_bytes(http.rs:4856) andput; gRPCput_artifact→putandput_artifact_stream→write_from_reader;Job::run→put(jobs.rs:1620, 1692);mirror_cycle::run_pair_skipping,replication:: replicate_one_pair(replication.rs:348),offsite::restore(offsite.rs:417), the proxy's cache fill. Beside itserver → store-write(chain_view,purge,signed_index,LocalHolger::put) andboot → store-write(Holger::put, retention): the review's 12 raw paths plus its 4 latent ones, found by the mask with no list of doors given. - R2, format knowledge above the format crates [forbidden]:
checks → formatsw 209 (the scanners reaching into inspect and the nuget crate),server → formatsw 50 (the backend factorybackend_from_config_with,chain_view::buildconstructing five format readers),gate → formatsw 39 (IntakeChain::inspect→holger_inspect::Format::declared,row_at_ingest→index_rows::row_of),boot → formatsw 22,doors → formatsw 10 (promote::npm→index_rows::npm). AnArtifactFormat::Xmatch is a name-free MIR switch, not an edge, so the 41 ladders the review counted are the dispatch-erosion / taxonomy-drift lenses' rows, not these. - The rest is the server tangle:
checks → serverw 323,server → doorsw 282,gate → server153,gate → doors127 (all[layer]).
With no spec: 51 crate components in 5 Cargo layers, κ 1.000 (code cannot
call a crate it does not depend on), and 9 absences: declared dependencies
no code edge uses (holger-server-lib → holger-handler-skidbladnir and
→ holger-handler-rust-toolchain, holger-front → holger-traits, the wasm
handler builds → holger-plugin-abi, holger-agent-lib → holger-world-connector).
The same lens on Gunnar (gunnar-oden dump, no spec): 21 crate
components in 8 Cargo layers, 9,008 items, κ 1.000, 5 absences
(gunnar-common → gunnar-proto, gunnar-grpc → gunnar-common,
gunnar-pairing-door → gunnar-proto, gunnar-repair → gunnar-proto,
gunnar-server → gunnar-pairing-door). A Gunnar spec waits for the Gunnar
architecture review's target.
Bench (release door dwarves --no-default-features --features cli,lenses,mimir-fat, graph cache warm, timeit wall + peak RSS; the RSS is
the held graph): Holger with the spec 1.2 s / 2.4 GB, Holger no spec 1.2 s /
2.3 GB, Gunnar no spec 1.2 s / 2.4 GB.
What it cannot say. It checks the spec it is given: a wrong spec gives honest numbers about the wrong target. Variant-level format matches need the MIR switch lenses. A call through a generic bound is the monomorphised callee the dump records, not the trait edge.
4.28 arch-recover (plan M1, the layout the code wants)
The question. If the code were grouped purely by how it talks to itself,
which crates and modules would it have, and how far is that from today's
layout? Leiden (Traag, Waltman & van Eck 2019, the code of [misplaced]
§4.11) on a unit graph, at two resolutions, made stable over seeds, and
compared with the crates and modules the code has now.
Run. cargo lens arch-recover [--seeds N] [--gamma-crate G] [--gamma-module G] [--min-group N] [--compare SPEC] (door: dwarves
RepoLens::ArchRecover); defaults 10 seeds, γ from the sweep, groups of ≥ 5
units.
Inputs. The item graph of reflexion (§4.27: fns, types, variants;
CALLS by call sites, USES_TYPE, READS_FIELD, WRITES_FIELD, IMPLEMENTS), the
CONTAINS parents and FOR_TYPE of the impls (the units), the module paths.
The math.
| step | formula | semiring / op |
|---|---|---|
| units | C ∈ 𝔹^{n×u}: a type with its inherent and trait-impl methods (an impl's methods go to its FOR_TYPE when that type is local) and its fields; a trait with its methods; an enum with its variants; a free fn alone | CONTAINS, FOR_TYPE |
| unit graph | W = Cᵀ · (Σ_k w_k · sym A_k) · C, sym A = A + Aᵀ, calls capped at 20 sites, weights ×10: calls 10, uses 5, reads 7, writes 7, implements 15 (the plan's 1.0 / 0.5 / 0.7 / 0.7 / 1.5); the diagonal dropped | PLUS.TIMES on UINT64, two GrB_mxm |
| modularity | Q_γ = (1/2m) Σ_ij (W_ij − γ k_i k_j / 2m) δ(c_i, c_j) | — |
| one run | Leiden (local move → refinement → aggregation Pᵀ·W·P) repeated on its own result until Q_γ stops rising (the paper's iteration); seed s permutes the node order (splitmix64 Fisher–Yates; seed 0 = as is) | aggregation PLUS.TIMES |
| γ sweep | grid 0.005 … 3; k(γ) = groups ≥ min_group; crate level = argmin ` | ln k(γ)/c |
| ARI | (Σ_ij C(n_ij,2) − E) / (½(Σ_i C(a_i,2) + Σ_j C(b_j,2)) − E), E = Σ_i C(a_i,2) Σ_j C(b_j,2) / C(n,2) (sums in sorted order: byte-stable) | — |
| consensus | Lancichinetti & Fortunato (Consensus clustering in complex networks, Sci. Rep. 2:336, 2012): D_ij = share of the 10 runs with i, j together over the edges of W, D_ij < τ = ½ dropped, the 10 seeds run again on D (weights ×1000), until all runs agree (≤ 5 rounds); the consensus = the medoid run | — |
| reproducible? | the whole consensus again from 10 fresh seeds (1000 … 1009); ARI(consensus, replicate) | — |
| soft | a consensus group whose mean best Jaccard against the RAW seed runs is < 0.8: never proposed as a crate | — |
| module level | Leiden at γ_module, each run nested in the crate consensus (a module group = a (crate group, community) pair), the same consensus | — |
| overlap | O = P_proposedᵀ · P_current (groups × crates) | PLUS.TIMES on UINT64 |
| matching | Hungarian (Kuhn–Munkres, O(n²m) potentials) maximising Σ O one-to-one; a group keeps its crate only when that crate holds ≥ ½ of it, else it is a new crate (named after its heaviest module, Σ unit degree) | — |
| moves | a unit whose crate is not its group's crate; weight = its degree in W; rows per (module → group) by weight | — |
| scores | MoJoFM(today → proposed) by the Tzerpos–Holt heuristic (a lower bound), Q_{γ=1} of both partitions | — |
| compare | --compare SPEC: per component S_c, the group of best `F1 = 2 | L∩S_c |
Dataflow.
items + A_k ─ w_k, sym ─▶ S (n×n) ─Cᵀ·S·C (GraphBLAS)─▶ W (u×u)
W ─ γ sweep: Leiden(γ) ×3 seeds ─▶ k(γ), ARI(γ) ─▶ γ_crate, γ_module
W ─ 10 seeds (permuted order, iterated Leiden) ─▶ raw runs ─ raw ARI
└─ D (co-assignment ≥ ½) ─ 10 seeds again … until they agree ─▶ consensus ─ soft groups vs raw runs
└─ 10 fresh seeds, the same ─▶ replicate ─ ARI(consensus, replicate)
consensus ─ Pᵀ·P_cur (GraphBLAS) ─▶ O ─ Hungarian, ≥ ½ ─▶ group → crate ─▶ MOVE rows; MoJoFM, Q
spec (--compare) ─▶ S_c ─▶ best group F1, aligned groups (α ≥ ½)
Worked example (its own fixture, tests/graph_lenses.rs). Crate core:
module store (struct Disk with methods read, write; fn open) and
module fmt (parse_a, parse_b, parse_c, calling each other, the disk and
open); crate web: module http (route, serve, auth in a triangle)
and sniff, which calls the three parsers (3 sites each) and route once.
Disk and its methods are one unit, so 9 units. Every seed finds 2 groups
(ARI 1.0, no soft group): {Disk, open, parse_*, sniff} and {route, serve, auth}; the Hungarian match gives the first to core, the second to web;
sniff is the one MOVE (web::http → group core::fmt [core]), and
Q_proposed > Q_crates. With --compare on a spec mapping core::fmt, the
best group holds all of fmt (recall 1.00).
On Holger h32/next 9dbe5cbf (golden arch-recover). 3,870 units, 10,482
unit edges. The sweep (crate counts) picks γ_crate = 1 (40 groups against 39
crates) and γ_module = 3. Crate level: the raw runs agree at mean ARI 0.796
(min 0.716); three consensus rounds bring the 10 runs to one partition, and
the consensus reproduces from 10 fresh seeds at ARI 0.893 (the plan's
acceptance, ≥ 0.8, read as: the proposal is the same whichever seeds made
it). 47 groups, 18 of them soft. Module level: raw ARI 0.894, 95 groups. Q at
γ = 1: crates 0.402 → proposed 0.781, modules 0.471 → 0.754; MoJoFM crates
37 %, modules 72 % (the module layout is much closer to what the code wants
than the crate layout is). 1,808 units would change crate: server/lib falls
apart into its own groups (import_door 301 units with
holger-update-bundle, scanner, grpc, mirror_cycle, flow,
audit::siem, passkey …), and holger-traits shares a group with the
format crates' readers and a third of server/lib (no crate holds half of it).
The format group against the review's proposed format crate
(--compare tests/golden/holger-format-crate.ron: the format crates, inspect,
index rows, the handlers, and the ~19k lines of server/lib format logic the
review moves behind trait Format; 714 units). No single group is that
crate. The best one is the pull_intake group (194 units, precision 0.92,
recall 0.25, F1 0.39). 12 groups that are mostly format code (α ≥ ½) hold
76 % of it at precision 0.79: pull_intake 179/194, index_rows 44/45,
inspect::image 56/56, inspect::yara_lite 16/16, pull_intake::pom 46/46,
pull_intake::closures, promote::identity 25/25, nuget registration
29/29, holger_traits 133/260 (the trait crate shares its group with the
format readers). So the code wants the format knowledge as several
per-concern pieces (the pull doors, index rows, inspection, the Maven pull,
identity), not one crate; the promote::intake group (191 units) keeps the
rest of the format code beside the gate, the tie R1/R2 cut.
On Gunnar (gunnar-oden, no compare): 4,264 units; γ_crate = 0.2 (16
groups against 21 crates); raw ARI 0.821, two consensus rounds, replicate
ARI 0.985; Q crates 0.666 → 0.691 (Gunnar's crates are already close to what
its code wants), modules 0.474 → 0.770; MoJoFM crates 73 %, modules 73 %. The
largest group (1,577 units) is gunnar-server 57 % with gunnar-core 23 % and
gunnar-ssh 7 %; gunnar-cli and gunnar-crypt form one group (a new crate),
and gunnar-policy + gunnar-store objects one soft group.
Bench (release door, graph cache warm, timeit): Holger with --compare
11.6 s / 1.9 GB, Gunnar 7.3 s / 2.4 GB. The time is the Leiden runs: a sweep of
16 γ × 3 seeds, then 10 seeds + consensus rounds per level, and the replicate
(≈ 120 iterated Leiden runs on ~4k units).
What it cannot say. The groups follow the coupling the code has, so a tangle like server/lib is proposed as split along its present seams, not along the target's: the target is M6's job, and the loop of §4.30 joins them. Weights are fixed by hand (open question 1 of the plan).
4.29 refactor-price (plan M7, the value of a refactoring before doing it)
The question. If we did refactoring row R2 (or any move list), how much would the structure improve, and what would it cost? A plan's rows are applied to the mapping and the item graph in memory (no git, no cargo) and the structure is recomputed, so each row earns its ticket before anyone opens an editor.
Run. cargo lens refactor-price --plan FILE [--spec FILE] [--no-infer]
(door: dwarves RepoLens::RefactorPrice); the spec as reflexion (§4.27).
The plan (JSON). { "rows": [ { "id", "title", "review_rank", "moves": [{select, to}], "cuts": [{from, to}], "interfaces": [{name, from, to, in, invert}], "not_expressible" } ] }. A selector is a spec glob (crate: /
mod: / fqn:), comp:<name> (what the current mapping puts there), *, or
a list of those (the union); selectors read the mapping before the row.
| edit | M′, A′ |
|---|---|
| move | the selected items get component to (module @to in the module DSM) |
| cut | A′ = A⟨¬Cuts⟩: the edges from → to dropped |
| interface (mediator) | every edge from → to becomes from → X (its sites kept) and one X → callee per callee; X a new item in in (a gate) |
interface (invert, a port) | from → X and callee → X (the callees implement the trait: dependency inversion) |
The math. For (A, M) and (A′, M′):
| measure | formula | semiring / op |
|---|---|---|
| κ, divergent weight / pairs | reflexion: H = Mᵀ·A·M, H⟨T⟩, H⟨¬T⟩ | PLUS.TIMES + masked GrB_apply |
| propagation cost | PC = nnz(V ∨ I) / N² over the module DSM, V by repeated squaring (dsm::analyse) | ANY.PAIR closure |
| ΔSCC | cyclic module groups (Tarjan) and the core size | — |
| Q | modularity of the component partition of M′ on sym(A′), γ = 1 (an unmapped item is its own group) | — |
| gain | δ = div_w(A, M) − div_w(A′, M′) | — |
| cost | s = the sites of the cut and rerouted edges, plus those of every edge at a moved item whose component pair changes and crosses a boundary before or after | — |
| rank | by δ / max(s, 1), then Δκ, then ΔPC; ranking_by_gain by Δκ alone | — |
Dataflow.
plan.json ─ selectors (globs / comp: / * / lists) on the items ─▶ M′ (moves), A′ (cuts, interfaces)
(A, M) and (A′, M′) ─▶ reflect: Mᵀ·A·M, masks ─▶ κ, div_w, pairs ──┐
─▶ module DSM ─ closure ─▶ PC, SCC, core ────┼─▶ Δ, s ─▶ δ / s ─▶ ranked rows
─▶ sym(A′) + M′ ─▶ Q ─────────────────────────┘
Worked example (the macro fixture of §4.27, the plan's §4). Today
κ = 3/6. Row via-intake routes http → store through an intake mediator:
http → X is allowed but X → store (intake → store) is not, so δ = 0 over
one site. Row move-parse moves fqn:app::formats::parse* into intake: both
intake ⇄ formats divergences close, δ = 2 over s = 2 sites, κ = 3/4.
Row cut-http-store closes one (κ = 3/5, δ/s = 1). Ranking:
move-parse cut-http-store via-intake (the tie at 1.0 per site goes to the
higher κ); a row with not_expressible is listed, not priced.
On Holger h32/next 9dbe5cbf (golden refactor-price). The plan is
tests/golden/holger-review-rows.json: the review's §6 rows R1–R16, against
the M6 target. Today: κ 0.861, divergent weight 1,360 in 15 pairs, PC 29.1 %
over the module DSM, 10 cyclic module groups (core 100), Q 0.629.
| row (review rank) | as | Δκ | δ (div. w closed) | ΔPC | ΔSCC | sites | δ / site | rank |
|---|---|---|---|---|---|---|---|---|
| R1 typestate gate (1) | mediators Admitted in gate for doors / server / boot → store-write | +0.002 | 21 | −0.19 pp | 0 | 21 | 1.000 | 1 |
| R14 single-impl traits (14) | cuts into the 9 traits | +0.000 | 3 | 0 | 0 | 33 | 0.091 | 2 |
| R5 registry (5) | mediators from the two factories → formats, in format-api | +0.001 | 1 | −0.07 pp | 0 | 30 | 0.033 | 3 |
| R2+ports (2) | R2's moves + the two ports in format-api (format code → server through a port the server implements; server → format code through Format) | +0.059 | 169 | −3.10 pp | 0 | 6,316 | 0.027 | 4 |
| R8 twins (8) | 4 helper families moved to base | +0.002 | 3 | +19.29 pp | −2 | 153 | 0.020 | 5 |
| R16 one wasm runtime (15) | cuts into plugin-host's wasm / registry / verify / limits | +0.000 | 1 | −0.82 pp | 0 | 161 | 0.006 | 6 |
| R10+ports (10) | the 9 pull-source modules moved + the ports | +0.024 | 5 | −0.88 pp | 0 | 2,697 | 0.002 | 7 |
| R4 Heimdall only (4) | cuts into holger-sandbox and the server's sandbox modules (media kept) | −0.001 | 0 | −1.52 pp | −1 | 1,866 | 0.000 | 8 |
| R2 moves only (2) | the server/lib format modules moved to formats | −0.079 | −814 | −2.48 pp | 0 | 1,333 | −0.611 | 9 |
| R10 moves only (10) | the pull sources moved | −0.068 | −797 | −0.48 pp | 0 | 878 | −0.908 | 10 |
Not priced, with the reason in the report: R3 (one component: the router is
H8's), R6 (cfg features are not edges), R7 (macro inside formats), R9 (MIR
ladders inside jobs), R11 (error types), R12 (a struct split inside boot),
R13 (comments), R15 (tests out of scope).
Against the review's ranking. The review's top three are R1, R2, R3.
- R1 agrees: it ranks first per changed site (every one of its 21 sites closes a forbidden write).
- R2 is the largest gain, and only when done the review's way. Moving the
format code alone makes things worse (κ −0.079): the moved code still calls
the server, the gate and the doors, which a format crate may not. With the
Formattrait on both sides it is the best row by what it buys (ranking_by_gain: R2+ports, R10+ports, R1, …; Δκ +0.059, PC −3.1 pp), but it touches 6,316 call sites, so per site it is fourth. - R3 cannot be priced here (the router is inside one component); the H8 lens is its measure.
- R14 and R5 rank above R2 per site because they are small and cheap; the review ranks by lines removed per effort, which is a different cost.
- R8 moves helpers that still call their old neighbours: the module
@basebecomes a hub and PC rises by 19 pp. Twins should move with their callees cut, which is a second row.
On Gunnar (gunnar-oden, no spec: the crate model) with a two-row plan
from M1's groups (logs/gunnar-plan.json on build-1): moving
gunnar_cli::crypt into gunnar-crypt and gunnar_policy beside
gunnar-store each open 12 weight of divergence (κ 1.000 → 0.998): the moved
code still calls the crates above it, so both need a port first.
Bench (release door, graph cache warm): Holger, 10 priced rows 3.0 s / 2.4 GB; Gunnar, 2 rows 1.5 s / 2.4 GB (one reflexion, one module DSM closure and one Q per row).
What it cannot say. A row is priced alone on today's base; rows that depend on each other (R2 then R5) need a combined row. Effort is call sites, not hours.
4.30 arch-report (the macro engine: M6 + M1 + the loop + M7, architecture-report/1)
The question. One run that says how far the code is from the target, what layout the code wants, which moves bring the two together, and what each row of a plan buys: the plan's §5b engine (Leiden proposes, reflexion judges, M7 prices).
Run. cargo lens arch-report [--spec FILE] [--plan FILE] [--alpha 0.85] [--seeds 10] [--top N] (door: dwarves RepoLens::ArchReport). --top caps the
rows listed per section (default 60, 0 = all; --limit 0 without --top
means all too).
The loop. For each crate-level consensus group L of M1 (§4.28) and each
component c: α(L, c) = |L ∩ c| / |L mapped|. If α(L, c) ≥ α_min (0.85),
every member of L the mapping places elsewhere is a candidate relocation to
c. Each candidate is priced alone by its local change of the divergent
weight (the edges at the item, before and after, by T) and kept only when
that change is negative (a no-op or a worse move is dropped: the plan's "kept
only if it does not increase the divergences", without the no-ops). The kept
candidates are grouped into rows (module, from → c) and each row is priced
by M7 (§4.29); a --plan adds its rows beside them, and all rows are ranked
by divergent weight closed per changed site. The after-score applies every
loop row with a positive gain to one state.
The contract (architecture-report/1).
| part | where | what |
|---|---|---|
contract, score | summary | score = [before, after] of PC, cyclic module groups (scc), Q, divergent weight, divergent pairs, κ |
| counts | summary | components, groups, aligned groups, moves, seams, cuts, violations, unmapped, inferred, rows, ranking |
move | findings | symbol, file:line, from module + component, to component, weight closed, α, reason leiden |
seam | findings | M3 (§4.32) over the divergent pairs: from / to components (merged when the cuts share a target set), call sites, ν, the share behind a trait, the proposed trait <To>Port with its methods (by name, with H11 signatures and impls) and callers |
cut | findings | M2 (§4.31) on the module graph: a back edge of the minimum back-weight order, its two layers, its weight and its heaviest item edges with file:line |
concept | findings | M4 (§4.33): the top 20 concepts by entropy, with owner, home, touches per component and leaks |
violation | findings | rule forbidden / layer / not-allowed / absence, the edge and file:line |
unmapped, inferred | findings | the symbol (and component + α) |
row | findings | each loop row L<n>, plan row, seam row S<n> (M3's trait as a port the callees implement, homed in the lowest component both sides may use), cut row C<n> (M2's cuts into one target module with overlapping callees as one port), concept row K<n> (a concept whose owner is not its home moved to the owner), with Δκ, divergent weight, ΔPC, ΔSCC, ΔQ, Δβ and the back-edge weight, changed sites, items moved, score |
score.beta, cuts, cut_weight, layers, seams, proposed_seams, concepts, top_concepts | summary | M2's β before / after and the module stack, M3's seam counts, M4's concept count and top 5 |
The ranking (lane MACRO-LENS-2). Every row ranks by the structural weight it
closes per changed call site: (Δ divergent weight + Δ back-edge weight) / s,
the divergences of M6 and the back edges of M2 in one unit (call sites);
refactor-price alone keeps its divergence-only score.
Dataflow.
spec ─▶ M6 model (A, M, T) ─▶ reflect ─▶ divergences ─┬─▶ violations, cuts (up a layer)
└─▶ seams: Σ callees of A_calls a→b, signatures
dump ─▶ M1 recover ─▶ consensus groups L ─▶ α(L, c) ≥ 0.85 ─▶ candidates ─ local Δdiv < 0 ─▶ rows L1…Ln
plan.json ─▶ M7 rows R… ────────────────────────────────────────────────────────────────▶ priced, ranked
every positive loop row applied together ─▶ score [before, after]
Worked example (its fixture, tests/graph_lenses.rs). The macro
fixture plus three intake helpers in a clique with intake::take, which
formats::parse calls (3 sites each). M1 finds {take, h1, h2, h3, parse}:
4 of its 5 items are intake, α = 0.8. With --alpha 0.75 the loop moves
parse → intake (local Δ = −5: take ⇄ parse and the three parse → h
calls stop diverging), row L1 ranks first: divergences 6 → 1, κ 1/3 → 3/4,
5 changed sites. http → store stays a violation, with the seam http → store: 1 call site into 1 fn — trait StorePort { put() }.
On Holger h32/next 9dbe5cbf (golden arch-report, the M6 target and the
review rows). 64 consensus groups, 47 aligned (α ≥ 0.85) with one
component; 9 relocations pass the price, grouped into 6 rows, e.g.
scanner::image_sbom::purl_of and image_sbom (checks → formats, closes 16 +
12: the review's image_sbom.rs:111 purl_of ladder, R2), handlers:: builtin_catalog (server → formats), topology::ConsumerCount (server →
gate), three job_runner types (doors → checks). Score with the loop rows
applied: divergences 1,360 → 1,301, κ 0.861 → 0.868, cyclic module groups 10
→ 9; PC 29.1 % → 32.3 % (the moved items sit in one synthetic module per
component, which joins what they touched: a modelling cost, not a real one).
15 seams: checks → server (106 sites into 38 fns: TrustPolicy::from_ron,
now_unix_ms, sha256_hex … — the helper twins of R8 behind one port),
server → doors (95 sites, the Jobs handle), checks → formats (53 sites:
control_from_deb, read_manifest, row_of …, the FormatsPort R2 wants
as trait Format), gate → server (104), gate → doors (56). 634 cuts
(divergent edges up a layer), 885 violations. Ranking of all 16 rows: R1 first,
then the six loop rows L4 L6 L2 L1 L3 L5 (cheap single moves), then
R14 R5 R2+ports R8 R16 R10+ports R4 R2 R10 as in §4.29.
With M2–M5 in the report (Holger, the same target and plan). 12 seams (7
proposed traits), 108 M2 cuts (weight 495, 29 module layers), 715 concepts, 41
rows. Ranking: S5 S3 S2 S1 S4 S6 R1 S7 C1 C10 …: the seam ports close the
most per site (S5 DoorsPort for gate → doors; S3 DoorsPort for server → doors, 282 of 1,360 divergent weight; S1 ServerPort for checks → server,
323; all homed in base, the one layer both sides may use), R1 next; the cut ports (C1 ImportDoorPort for jobs → import_door, …) each take 0.07–0.12 pp off β; the concept moves mostly open
divergences (K2 moving ArtifactId to doors +538): a shared value type needs
its users moved with it, which is a plan row, not a single move.
On Gunnar master c17a6ef4 (dump gunnar-default; golden
golden_gunnar_arch_report_against_the_review_target, the target
tests/golden/gunnar-architecture.ron written from the Gunnar review's "As it
should be" and §6 / §7, the plan tests/golden/gunnar-review-rows.json). 37
components in 15 layers (the 21 gRPC service modules plus two handler modules
share the layer svc-*), every item mapped: κ 0.949, divergent weight 597
in 50 pairs, 4 absences.
- R1, the by-hand asks:
svc-* → access[forbidden] holds every handler that calls the raw check instead ofServiceContext::require:WatchScope::{decide, admits, mine_admits}→Identity::may(service/events.rs:162, 201, 250, 268),refuse_the_last_admin_passkey(access.rs:249),list_namespacesandrecipient_keys(repos.rs:346, 1025),break_glass_open(bootstrap.rs:648), andrequire_log_reader→is_group_member(mirror.rs:66, the review's 7(c) #3); beside themdoors → accessholds the SSH control verbs (exec_request→may_server, server.rs:2705, 2918;recipient_keys_answer, server.rs:1162;Control:: push_create_spec, control.rs:568, 576: R6). The four absencessvc-{billing, entitlement, meters, sealed} → requireare services that never reachrequireat all. (accounts.rs:323 holds_server_grant(..).unwrap_or(true)sits in a closure the dump records no call edge for.) - R3, the monolith:
svc-* → control-plane[forbidden] from 17 of the 21 services, 206 call sites (access 42, repos 31, mirror 29, admin 25, bootstrap 16, accounts 12, …), each into aControlPlane#…method. - R4, three gate-set builders:
doors → gatesat transport.rs:2840 (EncryptedGate::with_retired) and mirror_fetch.rs:511 (BranchProtectionGate::new),control-plane → gatesat control_plane.rs:9596, 9603, 9611 (merge_attemptbuilds protection, signed and trailer gates). - R2, the engine:
server / doors / control-plane → git-engine[forbidden] w 91 / 52 / 49 (OpenRepo::git_dir,Engine::maintain,Engine::name,MaintenanceReport…).
M7 on the review's rows (6 expressible, 9 listed with a reason): ranking
R3 R2 R1 R4 R6 L1 R8 (before M2–M5 joined the report; with them, L1 R3 R2 S1 S2 R1 S3 S4 R4 S5 R6 C1 …: S1 is M3's GitEnginePort homed in git-api, 140 of
597 closed, beside the review's R2 row (192); 28 seams, 5 proposed; 61 M2 cuts,
20 layers; 704 concepts). R3 as 21 per-service ports in planes closes 206 of
597 (κ 0.949 → 0.964, 206 sites); R2 as one GitEngine port in git-api
closes 192 (κ → 0.962); R1 as Allowed mediators in require closes 15; R4
(one GateSet mediator in policy) 5; R6 (ControlVerb) 2; R8 (the clock /
hex twins moved to core) opens 1. Against the review's own order (R1, R2,
R3): the same three on top, with R3 and R2 ahead of R1 by weight because R1's
by-hand asks are few call sites (15) while the monolith and the engine are
reached from everywhere; R1's real gain is the type the compiler checks, which
no edge count measures. The loop finds one relocation (functional_status,
boot → server).
Bench (release door, graph cache warm): Holger 8.4 s / 2.2 GB, Gunnar 7.8 s / 2.3 GB (M1's Leiden runs are most of it). With M2–M5 and their rows priced (lane MACRO-LENS-2, the review targets and plans): Holger 8.5 s / 2.4 GB (41 rows), Gunnar 12.7 s / 2.0 GB (30 rows over 37 components).
What it cannot say. The loop only moves single items into the component
their group already belongs to; it does not invent a component (M1's "new
crate" groups are that list) and it does not cut edges (the cut and seam
rows are the candidates for that).
4.31 layer-stack (plan M2, the dependency direction)
The question. Can the code be stacked into layers where every dependency points down, and which edges stop that? Eades, Lin & Smyth (A fast and effective heuristic for the feedback arc set problem, IPL 47(6), 1993) on the SCCs of the module graph, with an exact solver for the small ones.
Run. cargo lens layer-stack [--level module|crate|component] [--spec FILE] [--exact-max 20] [--edges-per-cut 3] (door: dwarves
RepoLens::LayerStack; --limit 0 lists every cut). --crate X stacks the
modules of one crate.
Inputs. The item graph A of reflexion (§4.27: CALLS by call sites,
USES_TYPE, READS_FIELD, WRITES_FIELD, IMPLEMENTS), each item's module path,
crate, or component of the target.
The math.
| step | formula | semiring / op |
|---|---|---|
| roll-up | W = Pᵀ · A · P (P items → modules), the diagonal dropped | PLUS.TIMES on the edge list |
| condensation | Tarjan's SCCs of W (order::sccs), the SCC DAG top first | a walk |
exact order (SCC of s ≤ exact_max nodes) | D(∅) = 0, D(S ∪ {v}) = min D(S) + Σ_{u∈S} W[v][u] (v placed below all of S), O(2^s · s); the optimum equal to the heuristic keeps the heuristic order | MIN.PLUS over subsets |
| heuristic order (larger SCC) | ELS: sinks to the bottom, sources to the top, else argmax δ(u) = w_out(u) − w_in(u) to the top (ties: the larger out weight in the whole graph, then the lower index); then single-node insertion moves until none lowers c(π) = Σ_{π(u)>π(v)} W[u][v] (≤ 50 passes) | — |
| cut list | Cuts = {(u, v) : π(u) > π(v)} | — |
| layers | on W⟨¬Cuts⟩: ℓ(v) = 0 without out-edges, else 1 + max_{v→w} ℓ(w) | MAX.PLUS (longest path) |
| score | β = Σ w(Cuts) / Σ w(W), L = 1 + max ℓ | sums |
The same stack runs inside M7 (refactor-price, §4.29) on the module graph of
every priced state, so each row now carries Δβ and the layer count, and
arch-report's score has beta: [before, after].
Dataflow.
dump ─▶ A (items) ─ P (module / crate / component) ─▶ W = Pᵀ·A·P ─▶ Tarjan ─▶ SCC DAG (top first)
each SCC: s ≤ exact_max ─▶ subset DP (exact) │ else ELS ─▶ insertion moves ─▶ order π
π ─▶ Cuts (back edges) ─▶ β; W⟨¬Cuts⟩ ─▶ longest path ─▶ layers ℓ
Cuts ─▶ the item edges under each cut ─▶ file:line
Worked example (the macro fixture, the plan's §4). Modules http intake gate store formats misc, seven calls of weight 1, one SCC {intake, formats}.
Both orders of it cost 1; the tie goes to intake (out weight 2 against 1),
so the cut is app::formats → app::intake w 1 (layer 0 → 2) at
src/formats.rs:16, β = 1/7 = 0.142857, and the stack is store, formats 0;
gate 1; intake, misc 2; http 3 (L = 4). The SCC has 2 nodes, so it is
solved exactly (exact_groups 1).
On Holger h32/next 9dbe5cbf, server-lib (golden layer-stack). 171
modules, 888 module edges (weight 10,100), 3 cyclic groups: one of 100
modules (heuristic) and two small ones (exact). Cut 92 edges (weight 440,
β = 4.4 %) and server-lib is 26 layers. The heaviest cuts: jobs → import_door w 23 (the job runner reads IntelImported, ImportError::tagged),
exposed::http → flow w 21 (handle writes flow::Sources, marks the console
seen), promote::intake → promote::duties w 21 (the chain builds DutyError
variants), grpc → import_door 16, grpc → flow 15, audit::chain → audit 14,
promote::intake → promote::catalog 14, scanner → scanner::store 13,
sources::policy → (root) 13 (Steps::lower writes PromotionSettings), sync → promote::intake 13, grpc → topology 12, mirror_cycle → (root) 12
(Holger::data_root, backend_named: R12's config root).
Checked by hand (10 cuts, source at 9dbe5cbf): jobs.rs:1716 (e.tagged()),
http.rs:1154 (settings.console.mark()), intake.rs:2810 (vote_locked builds
DuplicateApprover), grpc.rs:1065 (kind.as_str()), grpc.rs:2177
(flow_sources builds flow::Sources), chain.rs:287 (event.to_json_map()),
intake.rs:990 (Catalog::new), scanner/mod.rs:888 (self.store.record),
sync.rs:539 (chain.available), mirror_cycle.rs:2073 (holger.data_root()):
each is the edge the cut names, each runs from a lower layer up, and each sits
in the 100-module SCC (for 4 of them the reverse direction is a direct import:
duties, catalog and store use super::intake / super::{Evidence, …}, audit
uses chain::).
On Gunnar, gunnar-server (golden layer-stack). 83 modules, 277 edges
(weight 3,415), 2 cyclic groups (43 modules heuristic, one exact): cut 25
edges (weight 122, β = 3.6 %) and the server is 14 layers. On the whole
spec model (arch-report, §4.30) the module β is 1.1 % (Holger 2.6 %).
Whole workspaces: Holger 352 modules, 10 cyclic groups (9 exact), cut 108 edges (β 2.6 %), 29 layers; Gunnar 390 modules, 14 cyclic groups (13 exact), cut 61 edges (β 1.2 %), 20 layers.
Bench (release door, graph cache warm, timeit wall + peak RSS; the RSS is
the held graph): Holger server-lib 0.87 s / 2.3 GB, gunnar-server 0.82 s / 2.6
GB, whole workspaces 0.77 s / 2.4 GB and 0.80 s / 2.5 GB.
What it cannot say. An optimal stack for the 100-module SCC is not proven (the subset DP stops at 24 nodes; an ILP is the plan's open question 6); the insertion moves give a local optimum. A cut says where the direction breaks, not which side should move: M3 turns cuts that share a target set into one trait, and M7 prices it.
4.32 seam-interface (plan M3, which trait is missing between two groups)
The question. Where do two components talk through many concrete calls, so that an interface (a trait) is missing between them? The trait is read off the cut (plan §5b): its methods are the functions the cut calls.
Run. cargo lens seam-interface [--spec FILE] [--from SEL… --to SEL… [--name N]] [--k-min 5] [--nu-max 0.5] [--no-infer] (door: dwarves
RepoLens::SeamInterface). With --from / --to (spec globs, comp:<name>,
as refactor-price selectors) it judges that one seam; else every ordered
component pair of the target.
Inputs. The item graph's CALLS (call sites) and the mapping M of
reflexion (§4.27); IMPLEMENTS / CONTAINS for "behind a trait"; the H11
envelope code (§4.25) for signatures.
The math.
| step | formula | semiring / op |
|---|---|---|
| the cut | A_{a→b} = diag(M[:,a]) · A_calls · diag(M[:,b]), B[a][b] = Σ A_{a→b} | PLUS.TIMES, read off the edge list |
| callee set | S_ab = the column projection 1ᵀ · A_{a→b} > 0, with the sites per callee | LOR.LAND |
| merge | cuts into one b with ` | S_ab ∩ S_cb |
| methods | the callees grouped by name (with_archive on 13 format readers is one polymorphic method: the role a trait gives them) | — |
| narrowness | `ν = | names(S) |
| behind a trait | `τ = | {s ∈ S : s is a trait method or a trait-impl method} |
| proposal | `B ≥ k_min ∧ | names(S) |
Dataflow.
A_calls ─ diag(M_a)·A·diag(M_b) ─▶ A_{a→b} ─ 1ᵀ· ─▶ S_ab (callees, sites)
cuts into one b, |S ∩ S′| ≥ ½ min ─ union–find ─▶ one seam, many callers
callees by name ─▶ methods ─▶ ν = |names| / B; IMPLEMENTS ─▶ τ
ν ≤ ν_max ∧ τ < ½ ─▶ trait <B>Port { methods } ─ H11 ─▶ signatures, impls, callers
Worked example (its fixture, tests/graph_lenses.rs). The macro fixture
plus store::{get, del}: http::handle calls get 3 times, del 2 and put
once (edge counts), so http → store is B = 6 sites into 3 methods, ν = 0.5, nothing behind a trait: trait StorePort { get; del; put } (methods by
sites), each with its H11 signature. Let misc::helper call get 3 and put
2 times: its cut shares 2 of its 2 callees with http's, so the two are ONE
seam http + misc → store ("2 cut pairs share the target set: one trait"). A
--from mod:app::http::** --to mod:app::store::** --name Store seam gives
trait Store { get; del; put }.
On Holger h32/next 9dbe5cbf (goldens seam-interface,
seam-interface-pairs). The server/lib → format-crate seam (--from crate:holger-server-lib --to comp:formats --name Format, the M6 target): 122
call sites into 78 fns, 48 methods, ν 0.410, 4 % behind a trait (only
MalwareScanner::{name, scan} and Limits::default): proposed trait Format { with_archive ×13; new ×8; from_config ×3; Format::declared; Inspector::inspect; Findings::summary; integrity_of; read_manifest; with_base_url ×3; with_reader_and_rows ×3; with_reader ×2; row_of; … }. That is
the review's R2 trait read off the code: reader(base) is the readers'
constructors (with_archive / with_reader* / with_base_url over the 17
format crates: the pull side), identify / inspect_kind are
Format::declared / Inspector::inspect, index_row is row_of, and the write
side (put …) is not in it because it is already behind RepositoryBackendTrait
(τ). Over every pair of the target: 27 seams of ≥ 5 sites, 13 proposed, 6
of them merged from cut pairs sharing a target set (checks + doors + gate → server 1,036 sites, 265 methods; cli + doors + server → gate 524;
formats → format-api 455, ν 0.10, the ZnippyBase helpers; server → formats 38 sites into 13 methods: the backend factory, with_archive ×13).
On Gunnar c17a6ef4 (golden seam-interface, the review's target). 52 seams,
15 proposed (6 merged), 3,567 concrete call sites behind them. The cut into
git-engine reads off the review's trait GitEngine: control-plane + server → git-engine, 82 sites into 30 methods, ν 0.366, nothing behind a
trait: trait GitEnginePort { OpenRepo::serve; OpenRepo::git_dir; OpenRepo::hash_kind; Engine::open_object_counts; Engine::index_progress; Engine::name; Engine::maintain; Engine::hold_twin_memory_bound; … } (the
review: name; open(dir, hash) -> Arc<dyn GitServe>; maintain; index_progress; memory_bounds), and doors → git-engine (26 sites, 9
methods: the push / mirror ingest) beside it. The R1 road is a seam too: 14
services → require is 422 sites into 21 methods, ν 0.050 (ServiceContext:: {caller, require, authorize, require_self, …}: the Verb / Allowed<L> API
of 7(b)).
Bench (release door, graph cache warm, timeit): Holger format seam 1.10 s
/ 2.4 GB, Holger all pairs 0.80 s / 2.5 GB, Gunnar all pairs 1.32 s / 2.6 GB.
What it cannot say. Grouping by name joins two unrelated methods that
happen to share a name (new, get): the impls list says which types are
behind each. The trait's home (which component owns it) is M7's question
(arch-report prices each seam as a port, §4.30).
4.33 concept-scatter (plan M4, who owns a concept)
The question. Which concepts (a type family, an enum family) are touched by many components, so that no component owns them? And who should?
Run. cargo lens concept-scatter [--spec FILE] [--min-touches 10] [--no-structs] [--no-infer] (door: dwarves RepoLens::ConceptScatter;
--limit 0 lists every concept).
Concepts. A concept κ is a node set 1_κ: an enum with its variants and
the methods of its impls (FOR_TYPE); a struct with its fields and impl methods;
a taxonomy family: the enums H9 (taxonomy-drift, §4.14) joins (parallel
pairs, equivalence translations), its UNPROVEN translations as well (a switch
over E1 whose arms build distinct E2 variants: Format::of(ArtifactFormat),
16 arms; only the 1:1 proof is missing, which matters for alias pairs, not for
ownership), and enums that share ≥ 5 variant names (one list spelled in two
crates), joined by union–find.
The math.
| step | formula | semiring / op |
|---|---|---|
| touches | T = A_uses + A_reads + A_writes + A_calls + A_switch (USES_TYPE, READS_FIELD, WRITES_FIELD by dump edge, CALLS by sites, a MIR switch on the discriminant of one of κ's enums per ladder) | — |
| per group | t_κ = Pᵀ · T · 1_κ (touches from inside κ dropped) | PLUS.TIMES |
| interface / direct | t_κ = t^if + t^dir: CALLS into κ's methods go through its interface; uses, field reads / writes and switches are direct knowledge | — |
| scatter | p = t_κ / Σ t_κ, H(κ) = −Σ p_i log₂ p_i, `g(κ) = | {i : t_κ[i] > 0} |
| owner | argmax_i t_κ[i] (a tie goes to the group κ is defined in) | — |
| leaks | the direct touches from every other group, with file:line | — |
| rank | H, then Σ t_κ, over the concepts with Σ t_κ ≥ min_touches | — |
Dataflow.
enums / structs (+ variants, fields, impl methods) ──────────────────┬─▶ 1_κ
H9 families ∪ UNPROVEN translations ∪ ≥ 5 shared names ─ union–find ─┘
USES_TYPE, READS/WRITES_FIELD, CALLS, MIR switch ─▶ T ─ Pᵀ·T·1_κ ─▶ t_κ ─▶ p, H, g, owner = argmax
direct touches outside the owner ─▶ leaks (file:line)
Worked example (its fixture). Enum formats::Fmt {A, B, C}: http names it
twice, gate once and switches on it once, formats::parse switches on it
twice: t = (formats 2, gate 2, http 2), p = ⅓ each, H = log₂ 3 = 1.584963,
g = 3; the owner tie goes to formats (where it is defined), and gate and
http leak (LEAK gate ×2 direct: app gate/admit() (src/gate.rs:1)). A
parallel store::Fmt2 {A, B, C} that intake names makes the family
Fmt + Fmt2 (g = 4), which ranks first.
On Holger h32/next 9dbe5cbf (golden concept-scatter, the M6 target). 715
concepts touched ≥ 10 times, 10 families, 620 switches. Top 5 by entropy:
ErrCode (H 2.51, g 8: the error-code struct every layer builds by hand),
job_runner::Jobs (2.40), ArtifactId (2.35, g 9, 929 touches), the format
family ArtifactFormat + promote::identity::Format + inspect::Format + index_rows::IndexRow (H 2.23, g 7, 145 touches, owner formats; it touches
gate 42 times, server 19, checks 12 — the R2 ladders), and
ArtifactFormat alone (H 2.20, g 7). The plan's acceptance (ArtifactFormat
and its siblings in the top 5) holds, and the proposed owner is the format
crates the review moves them to.
On Gunnar c17a6ef4 (golden concept-scatter, the review's target). 704
concepts, 12 families. The top 10: the family Scope + Access + ReplicationScope (H 3.99, g 20: the access scope every service builds by hand,
the review's "the Access × GrantLevel pair passed by hand per RPC"),
Access (3.81, g 18), ServiceContext (3.80, g 20, 821 touches),
Caller (3.41, g 14), AccountAuditEvent (3.24, g 12), EventBus (2.82),
the family LevelArg + Level (2.44), forge::Event (2.37), ServerConfig
(2.34), model::Repo (2.33). The first four are R1's material: the proof
Allowed<L> of 7(b) is what would own them.
Bench (release door, graph cache warm, timeit): Holger 1.31 s / 2.2 GB,
Gunnar 1.29 s / 2.6 GB (one taxonomy-drift run inside).
What it cannot say. High entropy is not always a fault: a value type every
layer passes (ArtifactId, Repo) is meant to be shared; the leak list
(direct touches) and the interface share tell a shared value from scattered
knowledge. The family widening (UNPROVEN, ≥ 5 shared names) can join two lists
that only look alike.
4.34 ports-adapters (plan M5, where domain code does I/O)
The question. Is the I/O (files, network, processes, databases, git) at the edges, or mixed into the domain logic? Built, as plan §5b decided, as a shipped target template on the reflexion core (driving adapters → domain ← driven adapters) plus the I/O seed set.
Run. cargo lens ports-adapters [--spec FILE] [--entry PAT…] [--io PAT…] [--domain-share 0.7] [--groups module|crate|component] (door: dwarves
RepoLens::PortsAdapters; --limit 0 lists every violation).
Inputs. M1's units (§4.28: a type with its methods and fields, a trait with its methods, a free fn) over the item graph; CALLS to dependency nodes (the I/O API set), SYSCALLS and TOUCHES_TABLE (B10's facts; neither Holger's nor Gunnar's schema-6 dump carries them, so the API set does the work there); the entry patterns of B10.
The math.
| step | formula | semiring / op |
|---|---|---|
| units | C (items → units), the directed unit call graph U = Cᵀ · A_calls · C | PLUS.TIMES |
| I/O seeds | S_io = units holding a fn with a SYSCALLS / TOUCHES_TABLE edge or a call into std::{fs, net, process} (not process::id), the fs Path probes (exists, is_file, metadata, read_dir, canonicalize …), tokio / rustix::{fs, net, process}, socket2, hyper, reqwest, ureq, redb, rusqlite, memmap2, tempfile, gix, gix-{odb, ref, transport, protocol}, --io patterns | — |
| entries | E = units matching --entry, else the units no other unit calls | — |
| reach | R = reach(E, U): r ← r ∨ r·U from the seed SET in one walk (ops::reach_sets) | LOR.LAND (ANY.PAIR) |
| classes | adapter = S_io ∧ called from another group; domain = R \ S_io; glue = the rest | — |
| shares | per group `domain_share = | domain |
| violation | u ∈ S_io in a group with domain_share ≥ 0.7, ranked by ` | reach({u}, Uᵀ) ∩ domain |
| template | M_t: entries → driving, S_io → driven, other reached units → domain; T = {driving → domain, driven → domain}; H = M_tᵀ·A·M_t, H⟨¬T⟩ | the reflexion masks (§4.27) |
Dataflow.
items ─ C ─▶ units ─ CALLS ─▶ U ──────────────────────────────────┐
SYSCALLS / TOUCHES_TABLE / calls into the I/O API set ─▶ S_io │
--entry (or no caller) ─▶ E ─ reach_sets(E, U) ─▶ R ───────────────┼─▶ adapter / domain / glue, shares per group
S_io in a group ≥ 70 % domain ─ reach_each(·, Uᵀ) ∩ domain ─▶ violations ranked, file:line of the I/O call
template M_t (driving / domain / driven) ─ Mᵀ·A·M, H⟨¬T⟩ ─▶ domain → driven divergences, κ_hex
Worked example (its fixture). The macro fixture with store::put writing a
file and gate::admit reading one (both called from another component:
adapters) and a pure chain intake::take → take2 → take3 → take4 whose last
link reads a file. intake is 3 of 4 domain (75 %), so take4 is the one
violation: VIOLATION app intake/take4() (src/intake.rs:80) in intake (domain 75 %): direct fs I/O … → std fs/read() at src/intake.rs:80; 5 domain units above (take3, take2, take, http::handle, and formats::parse, which calls
take). The template diverges domain → driven with weight 2 (take → admit, take3 → take4) and driving → driven 2: κ_hex 0.2.
On Holger h32/next 9dbe5cbf (golden ports-adapters, module groups, the
gate-bypass entries). 3,870 units: 424 do direct I/O, 177 of them
adapters (called from another module), 2,006 domain, 1,687 glue; 40 entries
reach 2,232 units. 44 violations. The heaviest: sandbox::worker::input
(std::fs::read, worker.rs:144, 57 domain units above), sandbox::scan_program
(mod.rs:662), ScanWorker::with_jobs_in (fs::canonicalize, mod.rs:236),
upstream_rust::build_agent (ureq agent, upstream_rust.rs:89), JobQueue::open
(create_dir_all, jobs.rs:819), PropertyStore::get (fs::read,
properties.rs:124), PromotionEngine::is_promoted (Path::exists,
promote/mod.rs:742), YankLog::append (cargo.rs:345), TagLog::records
(npm.rs:621), UreqFetch::get (ureq call, pull_intake/http.rs:184).
Checked by hand (those 10, source at 9dbe5cbf): each line is the direct
I/O call the row names, inside a module that is otherwise domain logic (the
sandbox, the job queue, the property store, the promotion engine and its
npm / cargo logs write and read files themselves, with no port between). The
template: κ_hex 0.888, domain → driven weight 586.
On Gunnar c17a6ef4 (golden ports-adapters, the re-compare entries). 4,296
units, 545 direct I/O (182 adapters), 1,379 domain; 18 violations. The
heaviest: RedbSecretStore::rows (redb open_table, secrets.rs:199, 77 domain
units above), limits::raise (rustix getrlimit, limits.rs:118),
PrincipalActivity::import_once (fs::read, activity.rs:226),
layout_config::load_from (layout_config.rs:368), legacy_secret_log
(Path::is_file, secrets.rs:516), control_plane::repo_bytes and its walk
(fs::read_dir, control_plane.rs:8036, 8015: the 182-method plane sizing
repositories by walking the disk), ssh::resolve_identity_in (gunnar-client
ssh.rs:252), mirror_fetch::dial_ssh (set_permissions, mirror_fetch.rs:1843),
strict_host_keys (mirror_fetch.rs:124). Checked by hand (those 10, source
at c17a6ef4): each is the I/O call at that line. The template: κ_hex 0.754,
domain → driven 873.
Bench (release door, graph cache warm, timeit): Holger 0.84 s / 2.5 GB,
Gunnar 0.87 s / 2.6 GB.
What it cannot say. "Domain" is reached-and-no-direct-I/O, not a judgement of
the code's purpose; an I/O call behind a port the domain owns is the clean
shape, and the lens sees it as such only when the port is a trait (a call to the
trait method is not an I/O call). Without SYSCALLS / TOUCHES_TABLE in the dump,
I/O behind a dependency that is not in the API set (an HTTP client wrapped in a
dependency crate) is not seen; --io adds it.
4.35 fail-open (Gunnar re-compare row 10, a decision that answers yes on an error)
The question. Where does a yes/no decision that can fail (Result<bool, _>,
Option<bool>) get collapsed with unwrap_or(true), so that an I/O error
answers yes? holds_server_grant(ns).unwrap_or(true) says "seller" when the
identity table cannot be read. The fix is to make the error refuse (?, or
Err ⇒ refuse).
Run. cargo lens fail-open --mimir-dump DIR --src CHECKOUT [--limit 0]
(door: dwarves RepoLens::FailOpen). Without --src the rows are DEFAULTED and
the lens names what it needs: the MIR keeps a constant's type (k:bool), not
its value.
The math. One pass over the call blocks of the fns in scope; no GraphBLAS step.
| step | formula | op |
|---|---|---|
| decisions | D = { c : ret(c) ∈ { Result<bool, E>, Option<bool> } } (path prefix and aliases erased) | — |
| defaults | S(f) = { b ∈ blocks(f) : term(b) = call unwrap_or(r, k:bool) } | — |
| source | root(r): follow use / copy / move / ref / refmut back ≤ 4 hops inside one body to the destination of a call c; keep b when c ∈ D | — |
| literal | lit(b) = the literal after unwrap_or( in the source, from the call's line (line(f) + rel) down to the first unwrap_or( (≤ 12 lines); that line is reported | text |
| answer | ans(b) = ¬lit(b) when dest(b) is read by Not(…) in the body; false when lit(b) and the source is if …unwrap_or(true) { return; }; else lit(b) | — |
| class | ans = true → FAIL-OPEN, ans = false → fail-closed, no literal → DEFAULTED | — |
Dataflow.
fns in scope ─▶ MIR call blocks ─ callee = Result/Option::unwrap_or, arg 2 = k:bool ─▶ S
S ─ receiver traced ≤ 4 reads ─▶ call c ─ ret(c) is Result<bool>/Option<bool> ─▶ decision defaults
─ --src: literal at the `unwrap_or(` line ─▶ lit
─ MIR: dest read by Not(…)? source: `if … { return; }`? ─▶ ans ─▶ FAIL-OPEN / fail-closed / DEFAULTED
Worked example (its fixture). room() has holds_grant(a).unwrap_or(true)
at src/acc.rs:11: FAIL-OPEN. closed() spreads holds_grant(a) /
.unwrap_or(false) over lines 21–22: fail-closed at :22. admits() returns
!shielded(a).unwrap_or(true) (MIR _0=Not(^_6)): the error answers no,
fail-closed, , negated. guard() has if holds_grant(a).unwrap_or(true) { return; }: the yes only skips the work, fail-closed, , returns early.
tally() collapses a Result<u64, _>: not a decision, not a row.
On Gunnar c17a6ef4, the metered dump (golden fail_open). 36 constant-bool
unwrap_ors, 17 on decisions: 3 FAIL-OPEN, 14 fail-closed, 0 unread.
FAIL-OPEN gunnar-grpc service/accounts/account_room_at(). (crates/gunnar-grpc/src/service/accounts.rs:323) — gunnar-grpc seam/ControlPlane#holds_server_grant().(..).unwrap_or(true)
FAIL-OPEN gunnar-server control_plane/impl#[CoreControlPlane]namespace_out(). (crates/gunnar-server/src/control_plane.rs:1995) — gunnar-core auth/impl#[AuthTable]holds_server_grant().(..).unwrap_or(true)
FAIL-OPEN gunnar-server entitlement/impl#[EntitlementSlot]no_fact_quota(). (crates/gunnar-server/src/entitlement.rs:1655) — gunnar-core auth/impl#[AuthTable]holds_server_grant().(..).unwrap_or(true)
fail-closed gunnar-grpc service/events/impl#[WatchScope]server_admits(). (crates/gunnar-grpc/src/service/events.rs:227) — gunnar-grpc service/impl#[ServiceContext]shielded().(..).unwrap_or(true), negated
fail-closed gunnar-server installer_listener/impl#[Installed]primary_takes_late_principals(). (crates/gunnar-server/src/installer_listener.rs:280) — gunnar-core auth/impl#[AuthTable]is_empty().(..).unwrap_or(true), returns early
Checked by hand (source at c17a6ef4): the three FAIL-OPEN lines are the
review's three holds_server_grant(..).unwrap_or(true) sites ("an unreadable
identity table says seller"). events.rs:227 is !ctx.shielded(..).unwrap_or(true):
negated, so an error refuses. installer_listener.rs:280 is if stores.auth().is_empty().unwrap_or(true) { return; }: an error skips the
hand-over. The default dump has only control_plane.rs:1995: accounts.rs:323 and
entitlement.rs:1655 sit behind #[cfg(feature = "metered")], so dump the
delivery row (§1, the driver note on -p and --features) to see them.
Holger h32 9dbe5cbf: 42 constant-bool defaults, 1 on a decision
(verify_keyless, issued_by(..).unwrap_or(false), verify.rs:420): fail-closed.
Bench (release door, graph cache warm): Gunnar metered 0.69 s / 1.3 GB, Gunnar default 1.98 s / 4.6 GB, Holger 1.09 s / 2.5 GB.
On the lossless graph (schema 8, lens upgrade batch 1). The default's value
is a fact of the dump, so the lens needs no --src:
| step | formula | read from |
|---|---|---|
| forms | unwrap_or(r, k), map_or(r, k, _), unwrap_or_else(r, c) → bool, unwrap_or_default(r) → bool | the call site's callee and its destination type |
| literal | lit = k_lo(k) ≠ 0; a closure default: the one k with _0 = use(k) in every write of _0, when the closure body has no call; unwrap_or_default: false | code_mir: the constant pool; the closure's own row (sub = {closure#n}) |
| negation | dest (or a plain copy of it) is the operand of a Not assign | the sites of the same body |
| early return | the fn returns (), a switch reads dest, and its true arm (not the value 0) runs to return through goto / drop only | tg_role, tg_lo, the terminators |
*.mir.arrow ── owner column ─▶ index fn → rows (Gunnar metered: 27,133 rows, 25 ms)
default call found in the derived blocks ─▶ Lossless::bodies(fn) ─▶ the call's site ─▶ lit, Not, switch arm ─▶ ans
On the Gunnar metered dump with no --src (golden
golden_gunnar_fail_open_needs_no_source_on_the_lossless_graph): 3 FAIL-OPEN,
14 fail-closed, 0 unread, the same classes as the --src run. With schema 7
the same run gave 17 DEFAULTED. events.rs:227 is negated and
installer_listener.rs:280 is returns early, both now from the graph. Without
--src the line is the first line of the call expression
(control_plane.rs:1991), not the .unwrap_or( line (1995): a site has one
position, its start. The lens takes 0.34 s on that dump.
What it cannot say. Only a constant default on a decision whose type the
dump carries; matches!(r, Ok(true) | Err(_)) and if let Ok(false) = … are
not read, and a closure default that computes its value is DEFAULTED. The negation is
seen inside one body only; the early-return rule reads a bare return; on the
if line or the next.
5. The super lens verdict
The question. Which crate and which module should be fixed first, and why? The verdict folds the graph lenses into one band per unit and prints every term beside its raw numbers. The score is a sort key; it is never shown alone.
Run. cargo lens verdict [--sink PAT… --choke PAT… --entry PAT…] [--spec FILE] [--min-fns 5] (dw:src/cli.rs:376-477; --min-fns defaults to 5). Without
--sink and a choke the gate term is silent and named.
The chokes from the target architecture (0.2.2, Holger hpro-metrics note 2).
With --spec FILE (a .nornir/architecture.ron), gate-bypass and verdict add to
the --choke patterns every production fn the spec maps to its gate component:
gate(spec) = spec.gate, else the component named "gate" (no such component: an error)
C_spec = { f : is_fn(f) ∧ in scope ∧ M(f) = gate(spec) } (M = the reflexion mapping, §4.27)
chokes = --choke patterns ∪ { label(f) : f ∈ C_spec }
Why: a hand list goes stale when the gate moves. On Holger main c2220d76 R1 moved
the store writes into admitted::Admitted; the old list (IntakeChain only) made
the four Admitted writers BYPASS doors and painted 7 crates RED (the gate's own
module and the format crates whose sinks they call). The in-repo spec's gate
component (590 fns: admitted, promote, admission, refusals, IntakeChain)
gives 0 RED, the same as the hand list with Admitted and Gate added: the four
writes are the gate's own (golden golden_holger_main_verdict_takes_its_chokes_from_the_spec).
One source for the gate (lane LENS-FIX3). VerdictArgs::gate is an
Option<GateArgs>: the very value a gate-bypass run takes (sinks, chokes,
entries and the gate's own crate scope), and the verdict runs gate-bypass on
it. The term counts the fn-level door rows of that run only (BYPASS,
CHECKED-INLINE, through-gate, gate-door, latent); an UNCHECKED-CHOKE row names a
caller of the choke, not a door, and is no longer counted as one. The summary
echoes gate_crates, gate_doors (= the run's doors) and gate_bypass (=
its bypass), so a verdict and a gate-bypass report given the same inputs cannot
disagree. On Holger c36dee4b they did: gate-bypass with the Admitted chokes
said 0 bypass doors, while the verdict, run with the golden IntakeChain choke
over the whole workspace, painted admitted RED.
Units
P = { f ∈ F : scope(f), f not a trait method } (the production fns,
crates/mimir-lens/src/lenses/verdict.rs:106-109). Every f ∈ P belongs to two units: its
crate crate(f) and its module module_path(f) (the CONTAINS parent chain).
A unit exists only when it holds at least one fn of P (:115-128). For a
crate, mods = the number of distinct module units of its fns (:126-128).
|u| below is the number of fns of P in unit u, and ÷|u| uses
max(|u|, 1).
The terms, as coded
Inputs shared by several terms:
Tests = { f ∈ F : is_test(f) } (node flag only, verdict.rs:132-137)
Reached = Tests · (CALLS ∪ DYN)* BFS over the whole graph, not scoped (:139-149)
Gated = fns at or below a node that has a non-partial GATED_BY edge,
found by a DFS down CONTAINS that stops after 20,000 popped nodes per edge (:156-178)
BodyGates(f) = Σ max(count, 1) over partial GATED_BY edges whose enclosing fn is f (:160-163)
Nobody(f) = callers(f) = ∅ ∧ dyn_sources(f) = ∅ (:190)
FanIn(f) = |{ c ∈ callers(f) ∪ dyn_sources(f) : crate(c) ≠ crate(f) ∨ module(c) ≠ module(f) }| (:183-189)
Anyhow(f) = Σ call_sites(f, c) over callees c with anyhow_kind(label(c)) = "create"
(anyhow `format_err`, `]msg()`, `]new()`; the `must_use` wrapper is not counted; :193-197, error_shape.rs:44-58)
Per unit u (:420-456):
| term | formula | w | present |
|---|---|---|---|
| cov | ` | {f ∈ u : f ∉ Reached} | ÷ |
| risk | `min(1, (Σ_{f∈u} FanIn(f) ÷ | u | ) ÷ maxfan_kind) × cov` |
| zomb | `( | {f ∈ u : f ∈ Gated ∧ Nobody(f)} | ÷ |
| dup | ` | {f ∈ u : f is a member of any twins group} | ÷ |
| size | ` | {f ∈ u : end_line − start_line + 1 > 150} | ÷ |
| disp | `min(1, ladder sites in u ÷ | u | )` |
| cyc | crate: modules of u in a module cycle ÷ mods; module: 1 if in a cycle, else 0 | 0.05 | always |
| impl | ` | {traits t in u : prod impls(t) ≤ 1} | ÷ |
| err | Σ Anyhow ÷ (Σ Anyhow + typed) | 0.05 | when the sum > 0 |
| gate | bypass rows ÷ door rows of the door fns in u | 0.25 | when --sink and --choke are given AND u has at least one door; absent otherwise |
The details behind each row:
- risk.
maxfan_kind = max(10⁻⁹, max { Σ FanIn ÷ |u′| : u′ of the same kind (crate or module), |u′| ≥ min_fns })(:402-410). The busiest banded unit of each kind therefore has a fan ratio of 1. A small unit can exceed it, which themin(1, ·)caps.FanIncounts distinct caller fns, test callers and dyn sources included, and "outside" means outside the fn's module, also for a crate unit: a call from another module of the same crate counts. The module doc says "fan-in from outside the unit" (verdict.rs:9); for a crate unit the code counts outside the module. There is no churn factor (named as missing). - zomb.
Gatedcovers any non-partial GATED_BY (acfgon the fn, itsimplor itsmod), whatever the feature. The module doc says "fns gated by a feature that nobody calls" (:10); the code takes any item-level gate and theNobodytest, and does not ask whether the feature is ever built. - dup.
twins::runwithTwinsArgs::default()and the same scope (:258-275); everymemberslabel of every finding counts once. - size.
size::runwith its defaults (fn_lines150;crates/mimir-lens/src/lenses/size.rs:25-29); everykind = fnfinding counts (:276-287). Lines areend − start + 1(size.rs:63). - disp.
dispatch::rundefaults; every site of everykind = enumfinding counts one ladder for its fn (:288-301). A site is a ladder whoseatis notHome: over an enum of another crate or of another module of the same crate (crates/mimir-lens/src/lenses/dispatch.rs:316-319). The module doc says "foreign-enum ladders" (:13). - cyc.
cycles::run; the members of everykind = modulefinding form the cycle set (:302-327). - impl. Computed inline, not by the
single-impllens: every trait node in scope counts in the units of its crate and module,prod impls= its IMPLEMENTS sources that are not test code; 0 or 1 counts as single (:208-224). The module doc namessingle-implas the source (:14). - err. Computed inline, not by the
error-shapelens (only itsanyhow_kindhelper is shared).typed= the impls of theErrortrait ofcoreorstdwhose FOR_TYPE type is in scope, counted in the type's crate and module units (:225-247). The module doc nameserror-shape(:15). - gate.
gate_bypass::runwith the same sinks, chokes, entries and scope (:328-351). Only the Phase-1 rows count;CFPQ-BYPASSanddroppedrows are skipped (:343-345). Each row adds 1 todoorsof its door fn's crate and module, and 1 tobypasswhen the class is BYPASS. A unit with no door has no gate term:gate = Some(bypass ÷ doors)only when the gate is on anddoors > 0, elseNone, and its weight renormalises away (:438-444). (Before dwarves f36754d a door-free unit gotgate = 0at weight 0.25, which lowered its score.)
Score, bands, order
score(u) = Σ_{i ∈ present(u)} wᵢ · Eᵢ(u) ÷ Σ_{i ∈ present(u)} wᵢ (0 when Σ w = 0; verdict.rs:502-513)
Σ w over the always-present terms = 0.25 + 0.25 + 0.15 + 0.15 + 0.10 + 0.10 + 0.05 = 1.05
Σ w can be 1.05, 1.10 (+impl or +err), 1.15 (+both), 1.30 (+gate), 1.35, 1.40
band(score) = RED if score ≥ 0.45
AMBER if score ≥ 0.20
GREEN otherwise (Band::of, :45-53)
RED override: gate(u) > 0 ⇒ band = RED, whatever the score (:514-517)
small: |u| < min_fns ⇒ band = none (text "small", JSON null); the score is still computed (:518-526)
Order (:528-535): banded units first, then RED, AMBER, GREEN, then score
descending, then the unit key (crate <name> / module <path>) ascending.
The text row prints the band, the score (2 decimals), the unit, its fn count
and the top three terms by w·E (ties by term name) with their raw numbers
(:563-593).
Silent terms, named in every run (:355-387): the gate term when --sink /
--choke are missing, binary bytes (cargo mimir-dump --build), churn
(nornir's git-heat feed), line coverage (llvm-cov coverage_fn) and the
package surface (graph-package is not folded yet). A dump with no is_test
fn is named too (:150-152). The needs lines of the sub-lenses are passed
through (:352-354, :361-371).
GraphBLAS steps
| step | operation | semiring / op | computes | file:line |
|---|---|---|---|---|
| 1 | test reach | none: BFS over calls ∪ dyn_targets | Reached | crates/mimir-lens/src/lenses/verdict.rs:129-149 |
| 2 | gated fns | none: DFS down CONTAINS, 20,000-node cap | Gated | :153-179 |
| 3 | per-fn facts, traits, typed errors | none: counting and hash joins | FanIn, Anyhow, traits, typed | :180-247 |
| 4 | twins | the twins lens's own products (ANY.PAIR reach, PLUS.TIMES FP64 cosine, PLUS.MIN, components; see §4.2) | dup | :258-275 |
| 5 | size, dispatch-erosion, cycles | the sub-lenses (§4.6, §4.3, §4.4) | size, disp, cyc | :276-327 |
| 6 | gate-bypass | §4.1: BFS, then ANY.PAIR reach under mask_by | gate | :328-351 |
| 7 | fold | none: weighted mean in f64 | score, band | :389-535 |
The verdict itself runs no GraphBLAS product; its GraphBLAS work is inside
twins and gate-bypass.
Dataflow
code_nodes / code_edges (one dump) ──► Graph::load
│
├─ is_test fns ── BFS over CALLS ∪ DYN ──────────────────────► Reached ─────► cov ──┐
├─ callers ∪ dyn_sources, other module ── Σ ÷ |u| ÷ maxfan ──► fan ── × cov ► risk ─┤
├─ GATED_BY (item) ─ DFS down CONTAINS ─ ∧ nobody calls ─┐ │
├─ GATED_BY (partial) ─ enclosing fn ─ count ────────────┴──────────────► zomb ─────┤
├─ twins::run ─────────── members of every group ───────────────────────► dup ──────┤
├─ size::run ──────────── fns > 150 lines ──────────────────────────────► size ─────┤
├─ dispatch::run ──────── ladder sites not at home ─────────────────────► disp ─────┤ Σ wᵢEᵢ ÷ Σ wᵢ
├─ cycles::run ────────── modules in a module SCC ──────────────────────► cyc ──────┼─► score ─► band
├─ trait nodes + IMPLEMENTS (non-test impls ≤ 1) ───────────────────────► impl? ────┤ │
├─ CALLS into anyhow "create" + Error impls (FOR_TYPE) ─────────────────► err? ─────┤ │
└─ gate_bypass::run (fn-level rows only) ─ bypass ÷ doors ──────────────► gate? ────┘ │
gate > 0 ─► RED override ───┤
|u| < min_fns ─► "small" ────────┘
? = present only under the condition in the term table
Worked example: holger-server-lib (plan §E.2, Holger h32/next 9dbe5cbf)
Recorded: score 0.25, band RED by the gate override, evidence
gate 13 bypass doors of 21 doors, cov 899 of 4,025 fns unreached by tests,
err 509 anyhow sites vs 21 typed error types. Workspace-wide: 1,796 tests
reach 5,327 of 6,603 fns of P (golden verdict headline), so 1,276 fns of
P are unreached, 899 of them in this crate.
The recorded terms, computed from those numbers:
cov = 899 ÷ 4025 = 0.2234 w·E = 0.25 × 0.2234 = 0.0558
err = 509 ÷ (509 + 21) = 0.9604 w·E = 0.05 × 0.9604 = 0.0480
gate = 13 ÷ 21 = 0.6190 w·E = 0.25 × 0.6190 = 0.1548
sum of the recorded w·E = 0.2586
Not recorded for this crate: risk, zomb, dup, size, disp, cyc, and whether the crate holds a trait (the impl term). What the numbers still bound:
Σ w = 1.05 + 0.05 (err) + 0.25 (gate) = 1.35, or 1.40 with the impl term
score printed 0.25 ⇒ 0.245 ≤ score < 0.255
Σ w = 1.35: Σ w·E ∈ [0.3308, 0.3443) ⇒ the unrecorded terms add 0.072 to 0.086
Σ w = 1.40: Σ w·E ∈ [0.3430, 0.3570) ⇒ the unrecorded terms add 0.084 to 0.098
risk ≤ cov ⇒ w·E(risk) ≤ 0.0558
size: 66 fns > 150 lines in the whole run (golden `size.fns_long`) ⇒ size ≤ 66 ÷ 4025 = 0.0164, w·E ≤ 0.0016
Without the override, 0.25 is AMBER. The band is RED because gate = 0.619 > 0.
A second recorded unit shows the AMBER side: the crate holger (server/cli),
score 0.27: cov 83 of 109 = 0.7615 (w·E 0.1904); 103 gates inside bodies,
so the body half of zomb is min(1, 103 ÷ 109) = 0.9450 and
zomb ≥ 0.9450 ÷ 2 = 0.4725 (w·E ≥ 0.0709; its gated-unreached count is not
recorded); err = 132 ÷ (132 + 0) = 1 (w·E 0.05). The printed 0.27 was
measured when a door-free unit still had gate = 0 at weight 0.25; then the
recorded part alone was 0.3113 ÷ 1.35 = 0.231 (or ÷ 1.40 = 0.222). Since dwarves
f36754d the crate has no gate term, so Σ w is 1.10 (or 1.15), and the recorded
part alone is 0.3113 ÷ 1.10 = 0.283 (or ÷ 1.15 = 0.271). The new score is not
recorded; the golden headline moved from 1 AMBER to 2 AMBER crates.
Modules (plan §E.2, before dwarves f36754d): 9 RED, 18 AMBER, 268 GREEN, 68
small. The golden now reads 9 RED, 32 AMBER, 254 GREEN: door-free modules lost
the 0-valued gate term and scored higher. Every RED module holds a
BYPASS door, and several are RED only by the override: proxy scores 0.17
(GREEN by score) with 1 bypass of 3 doors, and offsite 0.26 (AMBER by
score). grpc::services scores 0.42 with 2 doors and 72 of 74 fns untested
(cov 0.973). promote::intake is GREEN: its doors are gate-doors, so its gate
term is 0.
Output
Summary units, gate_term, weights (:554-562). Findings
{unit, kind, fns, band, score, terms[{term, value, evidence}]}
(:594-598). The headline: crates: R RED / A AMBER / G GREEN; modules: … (S small ones not banded); T tests reach X of Y fns (:541-553).
Tests
crates/mimir-lens/tests/graph_lenses.rs:923
the_verdict_bands_every_unit_and_names_the_silent_terms (min_fns 1): the
silent-term lines are printed, the crate row has a band and a cov term, and
with sink Store#put(), choke gate_ingest and entries http_handle(),
mirror_run() the crate app goes RED (the mirror_run bypass) and the
"needs gate term" line disappears.
6. The dwarves folds on the dump
The folds dwarves mimir zombie|risk|bloat and dwarves coverage were written
against the v1 warehouse tables (symbol_facts, call_edges,
feature_gate_facts, test_inventory, symbol_sizes). Those tables have had
no writer since Mímir v2. The adapter in 6.1 reads a dump directory and
presents the same facts in the old column shapes, so the folds run unchanged on
a dump. dwarves dupes (6.6) does not read the dump at all.
None of the folds uses GraphBLAS. Every step below is a hash join, a set operation, a breadth-first search or a sort, and each table says so.
Flags of dwarves mimir <bloat|zombie|risk> (dw:src/cli.rs:194-258):
--wh DIR (repeatable: a dump directory or an Iceberg root, in priority
order), --namespace nornir, --checkouts DIR (manifests for zombie),
--json, --limit 25, --fail-on-findings (exit 1 when the fold lists
anything, dw:src/cli.rs:14729-14732). Without --wh, a dump in the local
data directory is folded offline (dw:src/cli.rs:14979-15011); with a remote
data source the verb asks nornir's lens instead (dw:src/cli.rs:15024-15032).
6.1 The adapter: a dump as a warehouse root
crates/mimir-lens/src/coverage/v2.rs. Entry points: is_dump(root) = the directory holds at
least one *.nodes.arrow (:141-143); read(root) / read_with(root, symbols) (:277-363); Dump::legacy_tables() (:528-716);
Dump::graph_feed() (:720-817). The graph lenses of section 4 use the same
reader with symbols = false (crates/mimir-lens/src/lenses/mod.rs:124-127).
Step 1: read. Every *.nodes.arrow, *.edges.arrow and (when asked)
*.symbols.arrow directly in the directory, in sorted path order. Columns are
read by name; an unknown column is ignored and a missing optional one reads as
null. Defaults: defined_here = true when the column is absent (an old dump),
count = max(count, 1) with null read as 1 (:283-346, :306, :319).
Step 2: normalize the ids in place. A package is dumped once per cargo
target, and the target is a backticked segment of the moniker: none for the
lib, `bin=name`: for a bin, `test=name`: for a test harness. The
rule (:250-275):
seg(id) = the first `kind=name` segment of id (plus a trailing ':'), if any
bins = { name : some node id has seg kind = bin } (:232-237)
drop seg ⇔ kind ∈ {bin, example, bench, build-script}
∨ (kind = test ∧ (name ∈ bins ∨ name = package(id) with '-' → '_'))
normalize_id(id) = id with seg removed when "drop seg", else id unchanged
A tests/*.rs crate keeps its segment (its name is neither the lib nor a
bin), and feature= segments stay. normalize_in_place does the same edit on
the existing String with replace_range, without an allocation when nothing
is dropped (:271-275). It runs over node ids, node parents, edge ends and
symbol ids (:347-360). Limit: two bins of one package that define the same
item name fold into one node (:249).
Step 3: dedupe (:389-445).
nodes: for each id keep ONE row: the first row with defined_here = true, else the first row
(a later defining row never replaces an earlier defining row)
edges: for each key (from, to, kind) keep the first row's slot, with
count = max over the rows of that key
negated = the first non-null value, partial = the first non-null value
So the counts are per crate, not per build: a lib call seen in the lib dump and in its test-harness dump counts once.
Step 4: the legacy tables (legacy_tables, :528-716). repo_of(n) = the
dump's repo column, else the node's crate (:463-465). module_path(n) =
the mod / crate ancestors through parent (:468-488), so a method in an
impl block gets the module of the impl, not the type.
| v1 table | rows built from the dump | columns | file:line |
|---|---|---|---|
symbol_facts | nodes with defined_here and kind ∈ {fn, method, struct, enum, union, trait, trait_alias, impl, const, static, type, macro, mod, ffi}; fn/method → item_kind = fn | repo, crate_name, module_path, item_name, item_kind, file, line, fqn = module_path::name | v2.rs:536-573 |
call_edges | CALLS edges whose caller is defined_here (the callee may be a reference stub); one row per site: the row is repeated count times | repo (of the caller), caller_ident, callee_ident, caller_fqn, callee_fqn (= module_path::name of each end), file, resolution = "mimir-driver (rustc-resolved CALLS edges of a Mímir v2 dump)" | v2.rs:504-528, :576-590 |
feature_gate_facts | (a) every GATED_BY edge with partial ≠ true: the item, with cfg_expr = the item's own cfg column, else feature = "φ" (or not(feature = "φ") when negated); (b) every fn/method reached from that item by a BFS over CONTAINS, under the gate's feature = "φ"; (c) every cfg_gated_out node with its cfg. One row per (id, cfg_expr) | repo, crate_name, module_path, item_name, cfg_expr, file, line | v2.rs:597-675 (BFS :639-655) |
symbol_sizes | code_symbols rows with a size and a binary (only after cargo mimir-dump --build); kind = Text when the section name contains text; is_monomorphization = (role = mono); the name with generics erased by strip_generics (drop every <…>, nesting-aware) | crate_name, symbol_demangled_no_generics, symbol_demangled, size_bytes, kind, is_monomorphization, artifact_path | v2.rs:366-378, :678-723 |
binary_facts, git_heat_facts, coverage_fn, semver_diffs, embeddings | not produced | — | v2.rs:27 |
Step 5: the test-reachability feed (graph_feed, :729-826).
succ(x) = the CALLS targets of x (after dedupe; any node, stub or defined)
fan_in(x) = Σ_{edges e = (·, x), kind CALLS} count(e) (sites, after dedupe)
is_fn(n) = defined_here ∧ kind ∈ {fn, method}
test_code(n) = is_test(n) ∨ cfg(n) contains "test" ∨ file starts "tests/" or contains "/tests/"
roots = { t : is_fn(t) ∧ is_test(t) } (is_test only, not test_code)
for each root t: R(t) = BFS closure of {t} over succ, minus t itself
reachable = ∪_t R(t)
test_reachable(n) = n ∈ reachable for every is_fn node n
TestFact(t).reached = |{ n ∈ R(t) : is_fn(n) ∧ ¬test_code(n) }|, .files = their files,
.edges_out = |succ(t)|
GraphFeed { state = "measured", room = "mimir-dump", via = the dump path,
fns = every is_fn node with test_code, test_reachable, fan_in }
The BFS runs once per test (cost O(tests × reachable edges)), and a test
reached from another test is itself in reachable. cfg contains "test"
is a substring test: a cfg naming a feature such as testing also makes a fn
test code here (:742).
Step 6: roots and provenance (crates/mimir-lens/src/mimir/mod.rs:82-117, :204-214).
read_roots reads every root in the given order; a dump root contributes its
legacy tables, an Iceberg root its tables by name. For each table the first
root holding rows wins, and the winning root is recorded
(Sources::root_of); tables are never concatenated. A dump is read first only
when it is the first --wh; nothing in the code puts it first
(dw:src/cli.rs:14501-14508). producer_of names who writes each missing
table (crates/mimir-lens/src/mimir/mod.rs:257-289), and evidence_note lists every empty
input as "MISSING FEEDS … any count below is a LOWER bound" plus "NEEDS t:
producer" (:293-311).
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | Arrow IPC read | none | nodes, edges, symbols | crates/mimir-lens/src/coverage/v2.rs:283-346 |
| 2 | id normalize | none (string edit) | one id per item across targets | crates/mimir-lens/src/coverage/v2.rs:250-275, :347-360 |
| 3 | dedupe | none (hash map by id / by (from,to,kind), max count) | one node per id, one edge per key | crates/mimir-lens/src/coverage/v2.rs:389-448 |
| 4 | legacy tables | none (joins by id, BFS over CONTAINS for gates) | v1 shapes | crates/mimir-lens/src/coverage/v2.rs:531-725 |
| 5 | test reach | none (BFS over CALLS, one per test) | test_reachable, fan_in, reached | crates/mimir-lens/src/coverage/v2.rs:729-826 |
Dataflow.
<crate>@<ver>-<hash>.nodes.arrow ┐
<crate>@<ver>-<hash>.edges.arrow ├─read─▶ Node / Edge / Symbol rows ─normalize_in_place─▶ ids without bin=/test= segments
<crate>@<ver>-<hash>.symbols.arrow┘ (--build only) │
dedupe (id; (from,to,kind) max count)
│
┌──────────────────────────────┬────────────────────────────┬───────────────┴─────────────┬───────────────────┐
▼ ▼ ▼ ▼ ▼
symbol_facts (items, call_edges (CALLS × count, feature_gate_facts (GATED_BY symbol_sizes graph_feed (BFS from
fqn = module_path::name) callee_fqn, resolution) item + CONTAINS-BFS fns + (code_symbols every is_test fn over
cfg_gated_out) with size) CALLS; fan_in = Σ count)
└───────────── mimir::read_roots: first root holding a table wins ─────────────┘ │
▼ coverage::build, risk
zombie · risk · bloat folds
Worked example (fixture). crates/mimir-lens/tests/mimir_v2_dump.rs:1-20, :46-220:
crate app with fns a, b, c (in mod gated), test t, struct S,
param n, the stub Vec::push, features x and y. Edges: a → b count 2,
t → a count 1, a → push count 1; GATED_BY c → x and gated → y;
CONTAINS gated → c.
symbol_facts={S, a, b, c, gated, t}(6 rows: no param, no stub, no feature, no crate node).chas module pathapp::gated; its repo falls back to the crate,app(:227-288).call_edges=a → btwice andt → a,a → pushonce: callees[a, b, b, push](:290-306).feature_gate_facts= 3 rows:gatedunderfeature = "y"(its own gate),cunderfeature = "y"(the CONTAINS BFS fromgated),cunderfeature = "x"(its owncfg) (:308-350).- Dedupe: with a second copy of the same two files under another hash
(
…-fedcba9876543210.*),symbol_factsstill has 6 rows andcall_edges4, andgraph_feed4 fns;files = 4(:353-377). graph_feed:R(t) = BFS({t}) − {t} = {a, b, push}; the fns in it that are not test code areaandb, soreached(t) = 2.fan_in(b) = 2(one edge of count 2).cis not reachable;tis test code (:455-499).
6.2 dwarves mimir zombie
- Run.
dwarves mimir --wh <dump> [--checkouts DIR] [--json] [--limit 25] zombie. - Finds. Cargo features that are zombies in one of three directions:
unreached(the feature gates code nothing calls),never-built(declared, and no build turns it on),both. The never-built directions are severe (crates/mimir-lens/src/mimir/zombie.rs:45-72). - Reads.
feature_gate_facts,call_edges,scip_occurrences(only "is it non-empty"),symbol_sizes, and everyCargo.tomlplusnornir.toml/.nornir/build-variants.toml/build-variants.tomlof each checkout (crates/mimir-lens/src/mimir/fold.rs:16-99manifest_features,:103-117manifests_in).
Math.
by_fqn = call_edges has a callee_fqn column (true on a dump)
key(row) = module_path::item_name if by_fqn, item_name otherwise
features_in_cfg(s) = sorted, deduplicated names x of every feature = "x" in s
(scan for the substring "feature", then '=', then a quoted name;
not(…), all(…), any(…) are not evaluated)
Gates: for each feature_gate_facts row (repo ρ, cfg c):
for x ∈ features_in_cfg(c): items(ρ, x) ∪= { key(row) }; seen ∪= {(ρ, x)}
Inbound: in(ρ, k) = |{ call_edges rows with repo ρ and callee key k }| (rows = sites)
calls(ρ, x) = Σ_{k ∈ items(ρ, x)} in(ρ, k)
Manifests, per checkout directory ρ (its directory name = the repo):
declared₀ = every [features] key except "default", over every Cargo.toml of ρ
built₀ = {"default"} (when a [features] table has it)
∪ every feature listed in a `features = [...]` of a dependency entry in
[dependencies], [dev-dependencies], [build-dependencies] or [target.*.<those>]
∪ every feature of every [[build.variant]] / [[variant]] in the build-space files
implies(f) = the entries of [features].f (merged across the repo's manifests)
built = close_built(built₀, implies): the least fixpoint
B = built₀ ∪ { strip(n) : f ∈ B, n ∈ implies(f), n does not start with "dep:" }
strip(n) = the part after the last '/' ("server-lib/gateway" → "gateway",
"pkg?/feat" → "feat")
(computed with a work stack; a name is pushed once)
build_space_known(ρ) = one of the three build-space files exists
never_built(ρ, x) = build_space_known(ρ) ∧ x ∈ declared(ρ) ∧ x ∉ built(ρ)
unreached(ρ, x) = calls(ρ, x) = 0
direction(ρ, x) = both if unreached ∧ never_built
= never-built if never_built ∧ ¬unreached
= unreached if unreached ∧ ¬never_built
= (not a zombie) otherwise
Plus: for every ρ with build_space_known, every x ∈ declared(ρ) \ built(ρ) with no gate row:
a never-built zombie with gated_items = 0, inbound_calls = 0.
As sets, per repo ρ: Gated = { x : items(ρ,x) ≠ ∅ }, Called = { x ∈ Gated : calls(ρ,x) > 0 }
never-built-only = (declared \ built) ∩ Called (needs build_space_known)
both = (declared \ built) ∩ (Gated \ Called) (needs build_space_known)
unreached-only = (Gated \ Called) \ ((declared \ built) if build_space_known else ∅)
never-built, no gates = (declared \ built) \ Gated (needs build_space_known)
features_seen = |seen| (gate features plus the no-gate never-built ones)
sort: severe first, then gated_items descending, then repo, then feature; truncate to --limit
bytes_of(key) = Σ size_bytes of the Text symbols whose last "::" segment equals key's last
segment, and, when the key is an FQN (`module::path::item`, a v2 dump),
whose demangled name starts with key's module path
bytes_in_binary = Σ over the feature's gated items of bytes_of(item key)
(None when symbol_sizes has no Text row)
bytes_of is crates/mimir-lens/src/mimir/zombie.rs:385-400 (fixed in dwarves b364012;
before it, the bare-name map never matched an FQN key and every finding read 0).
Code: features_in_cfg crates/mimir-lens/src/mimir/zombie.rs:146-167; close_built
:197-220; manifest_facts_of :230-278; the build-variant scan and
build_space_known in crates/mimir-lens/src/mimir/fold.rs:16-99; analyze crates/mimir-lens/src/mimir/zombie.rs:285-486
(gates :319-351, inbound :354-362, bytes :365-378, direction
:381-419, no-gate never-built :424-443, sort :445-453).
Where the code and its comments differ:
- The comment says the callers come from
call_edges"within a repo" (:26-32), and that holds:in(ρ, k)uses the caller's repo. When the dump has norepocolumn, the repo is the crate, so a call from crate A into a gated item of crate B in the same workspace does not count, and the item can readunreached. Holger's golden run hasrepos_with_gates = 1, so this did not apply there. features_in_cfglooks for the substringfeature, so atarget_feature = "x"in an item'scfgwould also yieldx; the unit test coverstarget_osonly (:149,:541-544).- The docs call
scip_occurrencesthe designed input; the fold only checks whether it is empty to word theresolutionline, and on a dump the line is the edges' ownresolutionvalue (:296-312).
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | parse cfg_expr | none (substring scan) | features per gate row | crates/mimir-lens/src/mimir/zombie.rs:146-167 |
| 2 | gate rows → items(ρ, x) | none (map insert) | gated item keys per feature | crates/mimir-lens/src/mimir/zombie.rs:319-351 |
| 3 | count call rows by (repo, callee key) | none (hash join) | inbound calls | crates/mimir-lens/src/mimir/zombie.rs:354-362, :404-412 |
| 4 | manifest scan + close_built | none (fixpoint with a work stack) | built set | crates/mimir-lens/src/mimir/zombie.rs:197-278, crates/mimir-lens/src/mimir/fold.rs:16-99 |
| 5 | direction, sort | none | findings | crates/mimir-lens/src/mimir/zombie.rs:413-470 |
Dataflow.
feature_gate_facts (repo, module_path, item_name, cfg_expr) ──features_in_cfg──▶ items(ρ,x) ──┐
call_edges (repo, callee_fqn) ──count rows──▶ in(ρ, k) ─────────────────────────────────────────┼─▶ calls(ρ,x) ─▶ unreached?
<checkouts>/<ρ>/**/Cargo.toml ──[features], dep features──▶ declared, built₀, implies ──┐ │
<checkouts>/<ρ>/nornir.toml | build-variants ──variant features──▶ built₀ ──────────────┴─close_built─▶ built ─▶ never_built?
symbol_sizes (Text) ──tail + module──▶ bytes_in_binary │
▼
direction ─▶ {zombies, severe, features_seen, repos_with_gates, build_space_unknown, resolution, worst[]}
Worked example (fixture). crates/mimir-lens/tests/mimir_v2_dump.rs:380-410 on the
fixture of 6.1, with no manifests: features_in_cfg("feature = \"x\"") = [x]
and features_in_cfg("feature = \"y\"") = [y]. items(app, x) = {app::gated::c}, items(app, y) = {app::gated, app::gated::c}. No
call_edges row has callee app::gated::c or app::gated, so
calls(app, x) = 0 and calls(app, y) = 0 + 0 = 0. No manifest, so
never_built is false: both are unreached, with gated_items 1 and 2. The
closure is pinned by crates/mimir-lens/src/mimir/zombie.rs:805-851: with
built₀ = {gateway}, implies(gateway) = {intake, jobs, dep:tar},
implies(intake) = {bundle-import, modgunn/scip}, the fixpoint is
{gateway, intake, jobs, bundle-import, scip}; dep:tar is skipped and
modgunn/scip enters as scip.
Worked example (Holger). Golden: features_seen = 78, zombies = 24,
severe = 0, repos_with_gates = 1. Since severe counts never-built and
both, all 24 zombies are unreached: 24 / 78 ≈ 31 % of the features seen
gate code that no recorded call reaches. Round 4 of the plan (§E, an earlier
build of the fold) listed front (27 items), meter-guests (14) and mirror
(9) among them, and warned that unreached over-reports items reached only
through dyn Trait, because rustc's CALLS edge points at the trait method.
- Output. JSON
{kind: zombie, features_seen, repos_with_gates, build_space_unknown, zombies, severe, resolution, evidence_note, feeds, worst[{repo, feature, direction, gated_items, inbound_calls, bytes_in_binary}]}(crates/mimir-lens/src/mimir/zombie.rs:120-137). - Limits. A repo with no build-space file gets no never-built verdict and
is named in
build_space_unknown(:472-493). Calls throughdyn Trait, closures and fn pointers reach the trait method (Fn*::call*), not the impl, so they are not callers of the impl (section 3.2). Bytes needcargo mimir-dump --build. - Tests. 9 unit tests in
crates/mimir-lens/src/mimir/zombie.rs:551-851;crates/mimir-lens/tests/mimir_v2_dump.rs:380zombie_finds_the_feature_nobody_reaches_in_a_v2_dump; goldengolden_zombie_risk_bloat_coverage_on_holger(crates/mimir-lens/tests/golden_holger.rs:246).
6.3 dwarves mimir risk
- Run.
dwarves mimir --wh <dump> [--wh <iceberg root with git heat>] [--json] [--limit 25] risk. - Finds. Files ranked by Adam Tornhill's behavioural risk model (as in
CodeScene): callers × churn × uncovered (
crates/mimir-lens/src/mimir/risk.rs:1-36). The three inputs ride beside the score in every row. - Reads.
symbol_facts,call_edges,git_heat_facts,coverage_fn,semver_diffs, and the dump'sgraph_feedthrough the coverage index (crates/mimir-lens/src/mimir/fold.rs:121-154,risk_coverage).
Math. For a file u = (repo ρ, path p):
fns(u) = { (fqn, name) : symbol_facts rows with repo ρ, file p, item_kind = fn }
(test fns of the file included; fqn = module_path::item_name)
callers(u) = Σ_{(fqn, name) ∈ fns(u)} |{ call_edges rows : repo = ρ, callee_fqn = fqn }|
(by bare callee_ident when the table has no callee_fqn column; one row = one
call site, so this is a count of call SITES into the file, from any caller,
the file itself and tests included)
churn(u) = commits_90d of git_heat_facts (ρ, p), 0 when the row is missing
covered(ρ) = { fqn of fns with test_reachable ∧ ¬test_code } (coverage::build from graph_feed)
coverage_frac(u) = |{ (fqn,·) ∈ fns(u) : fqn ∈ covered(ρ) }| / |fns(u)| (0 when fns(u) = ∅)
risk(u) = callers(u) × c(u) × (1 − coverage_frac(u)),
c(u) = 1 when git_heat_facts is empty (no heat table at all), churn(u) otherwise
files_scored = |{ u }| (every file with ≥ 1 fn)
not ranked: "generated" if a path segment equals "generated"
"test code" if a directory segment (not the file name) equals "tests"
(counted in files_scored and in files_not_ranked, then removed)
units = { u ranked : risk(u) > 0 }, sorted by risk descending, then path; truncated to --limit
Code: crates/mimir-lens/src/mimir/risk.rs:137-146 (not_ranked_class), :173-182 (churn),
:185-199 (fns per file), :204-213 (inbound), :215-255 (per-file score;
the churn factor at :248-250), :257-273 (rank). The covered set is built
in crates/mimir-lens/src/coverage/mod.rs:636-641.
Where the code and the docs differ:
- The field is called
downstream_callersand documented as "call sites landing on symbols defined in this file" (:50). It is a count of sites, and it includes calls from inside the same file and from tests. coverage_fracis documented as "fraction of this file's functions a test can reach" (:54). The denominator includes the file's own test fns (they arefnrows insymbol_facts), and the numerator never does (test code is left out ofcovered). An inline#[cfg(test)] mod teststherefore lowers a file's coverage.semver_diffsis read for presence only:breaks_published_apiisSome(false)when the table has rows andNoneotherwise; it is nevertrue(:252). The module doc lists it as a join input (:3).- fqn =
module_path::nameignores theimpllevel, so twonewmethods of two types in one module share one key: their callers are summed and both count as covered when either is.
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | group symbol_facts by (repo, file) | none (map) | fns(u) | crates/mimir-lens/src/mimir/risk.rs:185-199 |
| 2 | count call_edges rows by (repo, callee_fqn) | none (hash join) | callers(u) | crates/mimir-lens/src/mimir/risk.rs:204-227 |
| 3 | test reach | none (BFS in graph_feed, 6.1 step 5) | covered(ρ) | crates/mimir-lens/src/coverage/v2.rs:748-790, crates/mimir-lens/src/coverage/mod.rs:636-641 |
| 4 | score, filter, sort | none | units | crates/mimir-lens/src/mimir/risk.rs:228-273 |
Dataflow.
symbol_facts (repo, file, module_path, item_name, kind=fn) ──group by (repo,file)──▶ fns(u) ──┬──▶ coverage_frac(u)
call_edges (repo, callee_fqn) ──count rows──▶ in(ρ, fqn) ──Σ over fns(u)──▶ callers(u) ─────────┤
dump graph_feed ──coverage::build──▶ covered(ρ) = test_reachable ∧ ¬test_code ──────────────────┘
git_heat_facts (repo, file, commits_90d) | absent ⇒ factor 1 ──▶ churn(u)
risk(u) = callers × churn × (1 − coverage_frac) ──▶ drop generated / tests/ / risk = 0
──▶ sort desc ──▶ {files_scored, files_not_ranked, units, worst[]}
Worked example (fixture). crates/mimir-lens/tests/mimir_v2_dump.rs:413-434 on the
fixture of 6.1, called with an empty covered set and no heat:
src/lib.rs: fnsa,b,t. Rows with calleeapp::a: 1 (fromt); withapp::b: 2 (one edge of count 2); withapp::t: 0.callers = 1 + 2 + 0 = 3;coverage_frac = 0 / 3 = 0;risk = 3 × 1 × (1 − 0) = 3.src/gated.rs: fnc, no rows:risk = 0, dropped from the rank.files_scored = 2, one unit,src/lib.rsfirst; the note says "RANK" and namesgit_heat_factsas not produced by Mímir v2.
With the covered set the CLI passes (covered(app) = {app::a, app::b}, from
6.1 step 5), the same file gives coverage_frac = 2 / 3 (the test fn t is in
the denominator) and risk = 3 × 1 × (1 − 2/3) = 1.
Worked example (Holger). Plan §D round 4 ranks holger-cli/cli.rs first
with 4,127 callers and 12 % coverage. Mímir v2 writes no git heat, so the churn
factor is 1:
risk(holger-cli/cli.rs) = 4,127 × 1 × (1 − 0.12) = 4,127 × 0.88 ≈ 3,631.8
The plan prints the coverage as a rounded percentage, so 12 % stands for
coverage_frac ∈ [0.115, 0.125) and the exact score lies between
4,127 × 0.875 = 3,611.1 and 4,127 × 0.885 = 3,652.4. The second file,
server/cli/lib.rs (3,661 callers, 5 %), scores 3,661 × 0.95 ≈ 3,478, below the
first file even though its coverage is lower. Golden: files_scored = 616,
units = 324 (with --limit 500), semver_available = false. Round 4 names
12 generated and 251 test-code files as not ranked.
- Output. JSON
{kind: risk, files_scored, files_not_ranked, units, coverage_basis, semver_available, evidence_note, model_attribution, feeds, worst[{repo, file, risk, downstream_callers, churn_90d, coverage_frac, explain}]};explain= "N callers × C commits/90d × (1 − F covered) = R" (crates/mimir-lens/src/mimir/risk.rs:67-72,:107-127). - Limits. With no heat every
churn_90dreads 0 and the note says the rank is callers × (1 − coverage) (:287-292). Test reach is more generous than line coverage, so risk is under-reported;coverage_basissays which sense fed it (:162-168). The folds have no scope filter: on a full dump, dependency files are scored too. The CLI reads the dump twice (once as a root, once forgraph_feed,crates/mimir-lens/src/mimir/fold.rs:136-139), which is whyriskcosts about twice the load (section 8). - Tests. 7 unit tests in
crates/mimir-lens/src/mimir/risk.rs:349-529(the arithmetic10 × 20 × (1 − 0) = 200at:349);crates/mimir-lens/tests/mimir_v2_dump.rs:413; golden atcrates/mimir-lens/tests/golden_holger.rs:246.
6.4 dwarves mimir bloat
- Run.
dwarves mimir --wh <dump made with --build> [--wh <root with binary_facts>] [--json] [--limit 25] bloat [--min-instantiations 2]. - Finds. The monomorphisation tax per generic: the bytes above what one
instantiation would cost (
crates/mimir-lens/src/mimir/bloat.rs:1-49). - Reads.
symbol_sizes,binary_facts,scip_occurrences(presence only).
Math.
rows = symbol_sizes rows with kind = "Text" and is_monomorphization = "true"
group g = (content_sha256, crate_name, symbol_demangled_no_generics)
n(g) = number of rows in g; bytes(g) = Σ size_bytes; example = the first symbol_demangled
mono_bytes(a) = Σ bytes(g) over every group of artifact a (before the floor)
keep g only when n(g) ≥ max(min_instantiations, 2) min_instantiations = 2
tax(g) = bytes(g) − ⌊bytes(g) / n(g)⌋ (integer division, saturating)
class(g) = glue if the name contains "drop_in_place" or "drop_glue"
= trait-method if the name starts with "::", contains "_::::", or ends with "::"
= named otherwise (only "named" is actionable)
tax(a) = Σ_g tax(g); tax_by_class(a)[c] = Σ_{class(g)=c} tax(g)
total_bytes(a) = binary_facts.total_bytes of content a (0 when absent)
tax_pct(a) = 100 × tax(a) / total_bytes(a) (0 when total_bytes = 0)
groups sorted by tax descending, then symbol; artifacts by tax descending
Code: crates/mimir-lens/src/mimir/bloat.rs:103-114 (classify), :238-351 (analyze;
the floor at :276, the tax at :280, tax_pct at :331-335).
On a dump the adapter writes the binary path into content_sha256 as the
artifact's identity (crates/mimir-lens/src/coverage/v2.rs:690-702), so two binaries stay two
artifacts. With no binary_facts row the artifact is named after the last
segment of that path (crates/mimir-lens/src/mimir/bloat.rs:305-315). Mímir v2 writes no
binary_facts, so total_bytes = 0 and tax_pct = 0 on a dump, and the
evidence note names binary_facts as the missing producer. (Before dwarves
b364012 the column was missing and every binary merged into one artifact "".)
GraphBLAS steps. None.
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | group by (content, crate, erased name) | none (map sum) | n(g), bytes(g) | crates/mimir-lens/src/mimir/bloat.rs:244-270 |
| 2 | tax, class | none (arithmetic, substring test) | tax(g), class(g) | crates/mimir-lens/src/mimir/bloat.rs:272-290 |
| 3 | per artifact sums, join binary_facts | none (hash join on content) | tax(a), tax_pct(a) | crates/mimir-lens/src/mimir/bloat.rs:292-342 |
Dataflow.
code_symbols (--build) ──adapter──▶ symbol_sizes (kind=Text, is_mono=true) ──group (content, crate, erased)──▶ n, bytes
└─▶ tax = bytes − bytes/n, class
binary_facts (content_sha256, total_bytes) [no producer in Mímir v2] ──────────────────▶ tax_pct
──▶ {artifacts, total_tax_bytes, actionable_tax_bytes, worst_artifacts[top_named]}
Worked example (unit test). crates/mimir-lens/src/mimir/bloat.rs:404-468: one artifact
of 1,000 bytes, znippy::parse_stream instantiated 4 times at 100 bytes:
bytes = 400, tax = 400 − 400 / 4 = 300, tax_pct = 100 × 300 / 1,000 = 30 %. crates/mimir-lens/tests/mimir_v2_dump.rs:626
bloat_keeps_two_binaries_apart_on_a_build_dump: a two-binary symbols layer;
bloat reports bin-a alone with tax 400 (bin-b's 100 bytes are not merged in).
crates/mimir-lens/tests/mimir_v2_dump.rs:648
zombie_reports_the_bytes_of_the_dead_feature_on_a_build_dump: zombie reports
300 bytes for feature x. On the source-only fixture dump the fold lists nothing and says
"needs symbol_sizes: … cargo mimir-dump --build"
(crates/mimir-lens/tests/mimir_v2_dump.rs:437-452).
Worked example (Holger). Golden: artifacts = 0, total_tax_bytes = 0,
actionable_tax_bytes = 0, scip_available = false. The Holger dump is
source-only, so there is nothing to fold.
- Output. JSON
{kind: bloat, artifacts, total_tax_bytes, actionable_tax_bytes, scip_available, evidence_note, feeds, worst_artifacts[{repo, artifact, total_bytes, tax_bytes, tax_pct, actionable_tax_bytes, top_named[≤ 5]}]}(crates/mimir-lens/src/mimir/bloat.rs:192-211). - Limits.
symbol_sizesonly aftercargo mimir-dump --build;binary_factsnever in Mímir v2. Data symbols are left out on purpose (the question is duplicated code). - Tests. 5 unit tests in
crates/mimir-lens/src/mimir/bloat.rs:404-629;crates/mimir-lens/tests/mimir_v2_dump.rs:437; golden atcrates/mimir-lens/tests/golden_holger.rs:246.
6.5 dwarves coverage --mimir-dump
- Run.
dwarves coverage --mimir-dump DIR [--warehouse DIR] [--checkouts DIR] [--worst 20] [--hot 25] [--json](dw:src/cli.rs:151-192). Without--mimir-dump,target/dwarves/mimirunder the cwd is used when it is a dump. - Finds. Per repo: which fns a test reaches, the untested fns with the
most call sites, and the hot modules without a bench
(dw:
src/cli.rs:15204-15305). - Reads. The dump's
graph_feed(6.1 step 5); from the warehouse, when present,git_heat_facts,bench_runs,bench_results,test_outcomes.
Math.
Test reach (6.1 step 5): reachable = ∪_{t : is_fn ∧ is_test} BFS-closure({t} over CALLS) − {t}
fan_in(f) = Σ_{CALLS e into f} count(e)
Per repo ρ (crates/mimir-lens/src/coverage/testcov.rs):
Surface(ρ) = { fqn(f) : f ∈ fns(ρ), ¬test_code(f) } (a set keyed by fqn)
Covered(ρ) = { fqn(f) ∈ Surface : test_reachable(f) }
Gap = Surface − Covered − Allowlist (autonom::compute_gap)
covered_frac = |Covered| / |Surface| (0 when Surface = ∅)
worst = Gap members sorted by fan_in descending, then fqn; first --worst
Hot paths per module m (crates/mimir-lens/src/coverage/benchcov.rs:260-358), test modules left out:
fan(m) = Σ fan_in of its fns; churn(m) = Σ commits_90d of its files; cov(m) = covered fns / fns
score(m) = 0.40·fan(m)/max fan + 0.25·churn(m)/max churn + 0.25·cov(m) + 0.10·[perf-name hint]
first --hot modules by score descending, then name; a module is benched when a bench arm covers it
Code: crates/mimir-lens/src/coverage/testcov.rs:103 (repo_coverage), weights
crates/mimir-lens/src/coverage/benchcov.rs:85-92, ranking :319-358, index assembly
crates/mimir-lens/src/coverage/mod.rs:577-745.
GraphBLAS steps. None. The reach is a plain BFS per test over CALLS
(crates/mimir-lens/src/coverage/v2.rs:750-760).
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | BFS from each is_test fn | none (queue over succ) | test_reachable | crates/mimir-lens/src/coverage/v2.rs:750-775 |
| 2 | Σ edge counts into each fn | none (map sum) | fan_in | crates/mimir-lens/src/coverage/v2.rs:734-737 |
| 3 | surface − covered − allowlist | none (set difference, autonom) | gap, worst | crates/mimir-lens/src/coverage/testcov.rs:103 |
| 4 | weighted module score | none (arithmetic) | hot paths | crates/mimir-lens/src/coverage/benchcov.rs:319-358 |
Dataflow.
dump CALLS (count) ──▶ succ, fan_in ──▶ BFS from every is_test fn ──▶ test_reachable per fn ──┐
dump nodes (fn, is_test, cfg, file) ──▶ test_code per fn ─────────────────────────────────────┼─▶ Surface, Covered, Gap ──▶ worst (by fan_in)
warehouse git_heat_facts / bench_runs / bench_results (optional) ──▶ churn, bench arms ────────┴─▶ hot paths (score) ──▶ un-benched list
graph {state: measured, room: mimir-dump, fns, tests}
Worked example (fixture). crates/mimir-lens/tests/mimir_v2_dump.rs:455-499:
reachable = {a, b, push}, so a and b are test-reachable and c is not;
t is test code. Surface = {a, b, c} (3), Covered = {a, b} (2),
covered_frac = 2 / 3. fan_in(b) = 2. One test, which reaches 2 fns.
Worked example (Holger). Golden coverage.graph: fns = 27,340,
tests = 1,796, state = measured. Plan §E (round 4): 23,471 production fns,
6,888 test-reachable, so 6,888 / 23,471 ≈ 29.3 % and
23,471 − 6,888 = 16,583 unreached. The worst untested fns by fan-in include
grpc::record_audit (67 sites) and scanner::model::malformed (38).
- Output. The coverage index:
graph {surface, state, measured, evidence, room, via, fns, tests},feeds, per repotest {surface, covered, allowlisted, missing, tests, tests_without_edges, worst[{fqn, file, line, fan_in, …}]}andbench {…},weak,integrity,unscanned_repos(crates/mimir-lens/src/coverage/mod.rs:720-745). - Limits. Overloaded operators have no CALLS edge, and calls through
closures and fn pointers stop at the
Fn*trait method (section 3.2), so the reach is a lower bound. Git heat, bench runs and results, test outcomes and line coverage are not in the dump; each is named when missing. - Tests.
crates/mimir-lens/tests/mimir_v2_dump.rs:455coverage_reaches_through_the_v2_dump,:502the_cli_folds_a_v2_dump_offline,:549the_local_data_dir_dump_is_folded_without_wh; golden atcrates/mimir-lens/tests/golden_holger.rs:246.
6.6 dwarves dupes
- Run.
dwarves dupes [--warehouse DIR] [--checkouts DIR] [--json] [--all-scopes] [--min-lines 6] [--min-code-tokens 30] [--min-shape-tokens 55] [--cosine 0.95] [--max-per-tier 100] [--no-vectors] [--fail-on-findings](dw:src/cli.rs:102-149). - Finds. Duplicate code in four tiers, cheapest first
(dw:
src/dupes/mod.rs:23-34). - Reads.
embedding_manifest,embeddings,embedding_snapshotsfrom nornir's warehouse (dw:src/dupes/local.rs:39), and the checkouts to hydrate span bodies. It does not read the dump. Mímir v2 has no embedding producer, so the verb exits 1 and says "NOT produced by Mímir v2" (dw:src/cli.rs:15637-15643), or, on the fat leg with an empty warehouse, "EMPTY INDEX, not a clean result" (dw:src/cli.rs:15444-15451).
Math.
span s: a warehouse chunk (repo, file, start_line, end_line, content_hash), body from the checkout
gate (every tier): lines(s) ≥ min_lines ∧ code_tokens(body) ≥ min_code_tokens
Tier 1 EXACT key = content_hash (byte-identical)
Tier 2 NORMALISED key = SHA-256(normalise(body)) comments stripped, every token joined by one space
Tier 3 STRUCTURAL key = SHA-256(shape(body)) identifiers → i, literals → their kind,
only when shape tokens ≥ min_shape_tokens
Tier 4 SEMANTIC cos(a, b) = a·b / (‖a‖ ‖b‖) over the span embeddings, pair kept when cos ≥ cosine;
only spans of ≥ 12 lines (SEMANTIC_MIN_LINES), compared within a span-height
bucket of ±2 lines, cross-repo only, at most 40,000,000 pairs (reported when hit)
A bucket is a finding when it has ≥ 2 distinct sites, spans > 1 repo (unless --all-scopes) and
its median line count ≥ min_lines. A span is claimed by the cheapest tier that matches it; a
later tier drops a bucket whose sites are all claimed.
idiom = sites > max_sites (12)
damage = median_lines × (min(sites, 12) − 1) × w_tier × (1.5 if drifted) × (1.4 if cross-repo)
w_tier = 1.0 exact / normalised, 0.7 structural, 0.4 semantic
drifted = the sites' content hashes are not all equal
Defaults: the CLI passes min_lines 6, min_code_tokens 30,
min_shape_tokens 55, cosine 0.95 (dw:src/cli.rs:130-141,
:15291-15300). The library defaults in TierOptions::default are higher:
min_lines 8, min_code_tokens 60, min_shape_tokens 120, max_sites 12,
cosine 0.95, height slack 2, pair budget 40,000,000
(dw:src/dupes/tiers.rs:270-290). The semantic tier ignores min_lines and
uses its own floor of 12 (:226-230). Tiers 1–3 are evidence; tier 4 is a
suggestion and carries its cosine (:60).
Code: damage dw:src/dupes/tiers.rs:200-214; bucket rules :355-400; tiers
:508, :535, :561, :674; cosine :616-631; normalisers
dw:src/dupes/normalize.rs:51 (normalise), :257 (shape).
GraphBLAS steps. None. Tiers 1–3 are hash buckets; tier 4 is a pairwise
cosine sweep over height buckets, fanned out over the gatling worker pool
(dw:src/dupes/tiers.rs:650-672).
| step | operation | semiring | what it computes | file:line |
|---|---|---|---|---|
| 1 | bucket by content hash | none (hash map) | EXACT | dw:src/dupes/tiers.rs:508 |
| 2 | bucket by SHA-256 of normalise | none | NORMALISED | dw:src/dupes/tiers.rs:535 |
| 3 | bucket by SHA-256 of shape | none | STRUCTURAL | dw:src/dupes/tiers.rs:561 |
| 4 | cosine per pair in height buckets | none (dot product loop) | SEMANTIC | dw:src/dupes/tiers.rs:616-631, :674 |
Dataflow.
warehouse embedding_manifest (spans, content_hash) ──┬──▶ tier 1 hash buckets
checkouts (span bodies) ──hydrate──▶ normalise / shape ┼──▶ tier 2 / tier 3 hash buckets ──claim──▶ evidence findings (damage)
warehouse embeddings (vectors) ──height buckets ±2────┴──▶ tier 4 cosine ≥ 0.95 ──────────▶ suggestions (cosine)
Mímir v2 dump: not read no embedding producer ⇒ exit 1, "NOT produced by Mímir v2"
Worked example. No Holger run: the verb refuses in about 10 ms (plan §G).
The tier arithmetic is pinned by the unit tests in dw:src/dupes/tiers.rs
(the tests module from :911); the graph twins lens (section 4.2) finds twins on the dump
without embeddings.
- Tests.
crates/mimir-lens/tests/graph_lenses.rs:1022dupes_names_the_producer_it_needs_instead_of_answering_empty(exit not 0, text "NOT produced by Mímir v2"); unit tests in dw:src/dupes/mod.rsand dw:src/dupes/tiers.rs.
7. Worked examples on Holger
These runs are recorded in the lens plan ~/plan/LENS-PLAN-2026-11-04.md
(sections D, E, E.2 and F). Unless a section says otherwise, the input was the
Holger dump of h32/next 9dbe5cbf, hand-written production code only (no
tests, no generated/, no derive-written impls). The comparison is against the
Holger architecture review. The goldens in crates/mimir-lens/tests/golden_holger.rs pin the
numbers of this dump (section 7.6).
7.1 Risk: the top 25 against the review's top 15 (plan §D round 4)
After four fix rounds the risk fold ranks hand-written production files only.
Generated (12) and test-code (251) files are scored and named, not ranked.
The top 25 by callers × uncovered starts with holger-cli/cli.rs (4,127
callers, 12 %), server/cli/lib.rs (3,661, 5 %), traits/lib.rs, flow.rs
(2,523, 54 %) and promote/intake.rs (2,074, 51 %). 8 of the review's top-15
files to split are in the lens top 25: flow.rs, promote/intake.rs,
lib.rs, scanner/mod.rs, pull_intake/mod.rs, grpc.rs, mirror_cycle.rs
and import_door.rs. The same run also matches review findings R4
(server/sandbox, 1,037 callers, 40 %) and R10 (pull_intake/mod.rs,
sources.rs). Churn is not in the score: git_heat_facts was not fed, so churn
was taken as 1 (section 6). The review's longest fn, handle (4,071 lines), is
not in the risk top 25 because its fan-in is low; the size lens finds it.
7.2 Gate-bypass: 13 realizable doors before, 0 after the R1 gate
Before: h32/next 9dbe5cbf (plan §D rounds 5–6, §E.2; golden
gate-bypass). Sinks: RepositoryBackendTrait::{put, put_from_reader, handle_http2_upload, handle_http2_request_bytes} and dyn ….write_from_reader.
Choke: impl#[IntakeChain]. Entries: the HTTP handle, the gRPC services,
JobRunner::run, mirror run_cycle/run_pair, replication, import door,
migration and offsite restore.
- Phase 1 (fn level): 13 write paths through 11 doors bypass the chain, 4 gate-doors, 0 through-gate, 4 latent (21 doors in all).
- Phase 2 (CFPQ): the MIR ICFG of
holger-server-libhas 194,731 blocks, 199,569 intra edges, 11,386 call edges and 30,354 return edges. The Dyck closure took 2 rounds and gave 10,230 matched call → successor edges (both recorded before dwarves f36754d; the door counts are unchanged). The insensitive walk reaches 19 doors; the matched walk reaches 13. Six impossible call/return paths are dropped:Holger::put,LocalHolger::put,ProxyBackend::{fetch, put, put_from_reader}andSignedIndexes::handle_http2_request_bytesare reached only by returning into another call site. - The 13 realizable doors:
handle×3,ChainView::handle_http2_request_bytesandIndexSigner::answer(both fromhandle), gRPCput_artifact,put_artifact_streamand itsdyn ….write_from_reader×2,ServerJobRunner::run,mirror_cycle::run_pair_skipping,offsite::restoreandreplication::replicate_one_pair. Complete mediation: NO.doors_without_miris 0. - The async doors were found only after coroutine bodies (
bbcN.0) were entered frombb0(dwarves 6f881c0).
After: Holger main with the R1 Admitted gate (plan §D round 9; holger main
91768f29 / bb80377b → 8fd5d586, the holger-refactor lane's dump). Chokes:
impl#[IntakeChain], admitted/impl#[Admitted] and admitted/impl#[Gate].
Sinks: the four write verbs put, put_from_reader, handle_http2_upload and
dyn ….write_from_reader.
- 0 bypass doors, 3 gate-doors, 2 latent (
ProxyBackend::{put, put_from_reader}). - The matched walk reaches 0 doors; the insensitive walk reaches 2; 2 impossible paths are dropped. Complete mediation: YES.
- With
handle_http2_request_bytesas a sink too, 4 fn-level doors stay and the matched walk reaches 3 of them. That verb serves the whole collected request, and it writes only on PUT, DELETE or PATCH. The method is a runtime value, so the lens cannot call that door mediated. Splitting it into a read and a write verb is a Holger change, not a lens change.
The lens cannot see the *_refusal() config toggles: they are runtime ifs,
not graph edges.
7.3 Twins: 327 groups, 41 STRONG and 71 DIVERGENT pairs at the defaults
At the lens_defaults (rare_max 12, min_blocks 8), the golden twins row
reads: 6,603 fns, 4,243 with MIR; 473 api pairs, 312 name+sig, 179 shape, 149
NEAR, 71 DIVERGENT, 10 inlined; 327 groups, 41 STRONG; 889 fns have a
twin.
dwarves 27eaa1d changed two of these numbers. STRONG now needs two independent classes (api, name+sig, shape, NEAR), so 78 STRONG became 41: a group whose second class was only DIVERGENT or inlined no longer counts. The deduped type entries keep the type cosine at or below 1, which removed 2 name+sig pairs (314 → 312) and 1 DIVERGENT pair (72 → 71); 890 → 889 fns have a twin.
The plan's earlier figures were measured with the CLI's old defaults,
rare_max 50 and min_blocks 4 (plan §D rounds 6 and 7). Those runs found
562 groups and 92 STRONG (191 NEAR, 24 inlined, 1,625 fns with a twin). With the
DIVERGENT class added, the same run found 107 DIVERGENT pairs and 118 STRONG.
The defaults became one table (lens_defaults) in round 9.
The first GraphBLAS run was one component of 5,473 fns (4-block shapes, loose inlined edges). The fix was a minimum block count, two shared rare APIs, the multiset Jaccard, skipped constructor stems, and inlined edges kept out of the components (plan §D round 6).
Matches with the review's §4 twin table: the connectors artifactory_load ≈
nexus_load (api + shape), plugin-abi decode_* / from_bytes (the same shape
but for a branch), and tls_roots STRONG across crates. These were found in the
round-6 run; the golden pins only the summary counts.
The DIVERGENT golden row is the PEP 503 normaliser, normalize_name ×3
(golden twins.normalize_name: 3 fns, 1 group, 2 DIVERGENT pairs). pip.rs and
znippy-python have the same 5-block body (api, name+sig and shape agree);
wheel.rs (30 blocks) is DIVERGENT from both. The review's "PEP 503 ×5 with 2
wrong" is these 3 normalize_name fns; the other two normalisers have other
names (normalize, normalise_*) and are found with --name normal (plan §D
round 7). Which body is wrong is a question for a test.
7.4 Verdict (plan §E.2, golden verdict)
The golden headline at dwarves main abbe6fd: crates 1 RED / 2 AMBER / 43
GREEN; modules 9 RED / 32 AMBER / 254 GREEN (68 small ones not banded); 1,796
tests reach 5,327 of 6,603 fns. The RED crate is holger-server-lib, RED by
the gate override (13 bypass doors of 21). The plan's §E.2 run, before the
verdict stopped giving door-free units a 0-valued gate term (dwarves f36754d),
read 1 RED (holger-server-lib, score 0.25), 1 AMBER (holger = server/cli,
0.27), 44 GREEN; modules 9 RED, 18 AMBER, 268 GREEN. The per-unit scores of
the new run are not recorded.
Every RED module holds a bypass door, for example grpc::services (2 doors, 72
of 74 fns untested) and exposed::http. promote::intake is GREEN: its doors
are gate-doors, so the chain itself is not the problem; the doors around it are.
Lane LENS-FIX3 (one gate source): the golden verdict now runs its gate term on the
gate-bypass golden's own GateArgs (crate holger-server-lib) and counts only the
fn-level door rows. Modules 8 RED / 16 AMBER / 262 GREEN → 8 RED / 17 AMBER / 261
GREEN; crates unchanged; gate_bypass = 11 = the gate-bypass golden's bypass. On
Holger c36dee4b with the golden IntakeChain choke the verdict now reads exactly the
gate-bypass run (4 bypass doors of 7; RED: holger-server-lib and admitted, where
the whole-workspace run of the metrics lane had 7 RED crates and 7 RED modules); with
the Admitted chokes both say 0 (verdict 0 RED).
7.4b The Gunnar re-compare rows (lane LENS-FIX3) and the Gunnar golden
The 12 rows of plan/handback/gunnar-recompare.md, as built (test first, a fixture each in
tests/graph_lenses.rs):
| row | rule now | formula / test |
|---|---|---|
| 1 module labels | a crate node shared by a package's lib and its examples carries the example's name (fake_idp); for an item whose file is not under examples/ / benches/, the root of its module path is the package (- → _) | root(i) = name(c) unless file(c) ∈ examples ∧ file(i) ∉ examples → pkg(c); module_paths_name_the_lib_not_the_example_that_shares_its_crate_node |
| 2 bare enum names | the MIR local's type is printed bare (1:Role); a candidate enum must sit in the fn's crate or a crate it depends on (any edge A → B, closed transitively) | cands(f) = {e : krate(e) = krate(f) ∨ krate(e) ∈ deps⁺(krate(f))}; dispatch_erosion_resolves_a_bare_enum_name_only_among_the_crates_the_fn_depends_on |
| 3 check-then-write | when choke calls are opaque and BYPASS doors remain, gate-bypass says those doors may be check-then-write | note; summary choke_calls_opaque |
| 5 / 7 entry patterns | each --entry pattern that matches nothing is named (entry_patterns_unmatched, a needs row: the version is no part of a label) | gate_bypass_names_each_entry_pattern_that_matches_nothing |
| 6 derive entries | B10 takes no derive-written fn as an entry | E = fns_matching(P) ∖ derived; reach_set_takes_no_derive_written_fn_as_an_entry |
| 7 FORWARD | a fn under 12 lines that calls its twin wraps it: the pair leaves the twin graph (pairs_forward, a FORWARD note) | fwd(x, y) ⇔ y ∈ calls(x) ∧ lines(x) < 12; twins_read_a_small_fn_calling_its_twin_as_a_forward_wrapper |
| 8 examples / suites | a file under examples/ / benches/ and a module named conformance, testkit, test_support, test_util(s) are test code | examples_and_library_test_suites_are_not_production_code |
| 9 config compare | a PartialEq compare (or a derive-written predicate) whose first argument is a field of self is a FLAG | runtime_guard_erosion_reads_a_compare_of_a_self_field_as_a_flag |
| 10 patches / configs | *Patch / *Update / *Delta / *Changes and *Config / *Settings / *Options / *Opts that no method of its own writes later (a builder step -> Self constructs) are optional data (patch_records, config_records) | deferred_init_reads_patches_and_config_records_as_optional_data |
Not built here: row 4 (B10 docs: name the constructors of a Vec<Box<dyn Gate>> set — the
Gunnar golden does exactly that), the early return Ok arm of FP 5 (a "must pass" walk), and
rows 8-12 of the re-compare's plan (H5 two-impl traits, const &str literals, the fail-open
lens, the delivery dump union, the driver note).
On Gunnar c17a6ef4 (gunnar-default). Module labels fake_idp::serve → gunnar_server::serve;
dispatch-erosion no longer credits mirror::Role switches to xtask setup_svg::Role; H6 reads
ServiceContext::require as FLAG-ONLY (both barriers _1*.2.3, the AuthMode compare) and
Identity#may is a CONFIG-TOGGLE; twins 25 → 21 STRONG, 154 FORWARD pairs, the backoff group is
api-only (no longer STRONG); H10 1 finding (ServiceRequest), 2 patches, 5 config records
(ServeConfig is one of them); size no longer lists examples/git-store-serve.rs.
Gunnar golden (tests/golden_gunnar.rs, 12 tests). The 20 hand-checked candidates of the
re-compare, on /work/gunnar-recompare/dumps/gunnar-default (+ gunnar-metered for the zombie
row): size (Server::start 1,721, exec_request 1,044, six files > 5,000 with control_plane.rs
9,822 first), B10 with the gate constructors as targets (push 7, mirror import BranchProtection
only, merge BranchProtection + SignedCommits + AiTrailer), gate-bypass BYPASS of
mirror_fetch::apply and the clan copy_repository, H6 require FLAG-ONLY, B10 7 Bootstrap
RPCs and merge_signing_key MISSING, Clans stop at require_self, exactly log_head /
log_fetch / snapshot_fetch reach require_log_reader, H9 PrincipalKind and
MirrorCredentialKind ≈ Credential, twins now_ms ×9, zombie heimdall never-built on both
dumps, single-impl AsTwinHead / AsMainStatus one impl each, cfg-sprawl features per crate,
H1 ForgeDriver a closed set of 3.
Holger h32 re-pin (before → after). Examples / benches out of production: size fns 6,514 → 6,452, files 387 → 384; derive-written B10 entries out: reach-set entries 217 → 138, missing 206 → 127; FORWARD: twins groups 323 → 271, STRONG 42 → 39, api pairs 486 → 358; bare-name enums by dependency: dispatch foreign ladders 30 → 28; verdict modules 8 / 17 / 261 → 8 / 17 / 259 (69 small); reflexion items 8,435 → 8,365 (κ 0.861324 → 0.861267). The full list is in the commit message.
7.5 The other graph lenses (golden rows)
| lens | golden row on h32/next 9dbe5cbf |
|---|---|
size | 6,668 fns, 66 over 150 lines, 19 over 300; 370 files, 5 over 5,000 lines; 273,777 lines (exact) |
dispatch-erosion | 119 enums; 31 ladders / 136 arms over another crate's enum, 73 / 389 over another module's, 187 at home; 1,066 == and 495 sniffs in 187 files; 14 parallel enum pairs |
cycles | 9 module cycles holding 115 modules; 1 crate cycle |
single-impl | 43 traits, 19 with one production impl, 2 with none; 176 impls |
error-shape | 936 anyhow sites vs 37 typed error types; 378 stringly results; 279 .context() sites |
cfg-sprawl | 155 features; 830 item gates, 1,099 body gates; 29 never referenced; 435 gated-out stubs |
zombie | 78 features seen, 24 zombies, 0 severe |
risk | 616 files scored, 324 units |
bloat | 0 artifacts (a source-only dump) |
| coverage graph | 27,340 fns, 1,796 tests, measured |
7.6 The tests that pin these numbers
crates/mimir-lens/tests/golden_holger.rs: 6 golden tests againstcrates/mimir-lens/tests/golden/holger-h32-9dbe5cbf.json. Each lens runs through thedwarvesbinary with--json. They are#[ignore]because the dump is 1.6 GB on build-1; run them withDWARVES_HOLGER_DUMP,DWARVES_HOLGER_SRCandDWARVES_HOLGER_CHECKOUTSset andcargo test --features cli,lenses,mimir-fat --test golden_holger -- --ignored(golden_holger.rs:1-16). Without the env a test fails and says what it needs. The tests:golden_gate_bypass_on_holger(:114),golden_twins_on_holger(:124),golden_dispatch_erosion_on_holger(:160),golden_size_cycles_single_impl_error_shape_cfg_sprawl_on_holger(:183),golden_verdict_on_holger(:233),golden_zombie_risk_bloat_coverage_on_holger(:246).crates/mimir-lens/tests/graph_lenses.rs: 15 fixture tests, one or more per graph lens, the graph index, anddupes_names_the_producer_it_needs_instead_of_answering_empty(:1022), andthe_matched_walk_does_not_fall_through_a_callee_that_never_returns(:986; fn-level bypass 1, matched walk 0). They need no dump: the fixture is written with the v2 schema.
8. Cost
The lossless graph (lens upgrade batch 1, nornir-mimir-lens 0.3.0). The
lenses that read code_mir rows on demand (fail-open, dispatch-erosion,
taxonomy-drift, twins, typestate-candidate) add to lens all, measured on
build-1 with one door binary (NORNIR_LENS_LOSSLESS=off = the lossy reading),
graph cache warm, --src given:
| dump | before | after | peak RSS |
|---|---|---|---|
| Holger h32 9dbe5cbf | 10.1 s | 14.4 s | 2.9 GB → 2.9 GB |
| Gunnar c17a6ef4 default | 10.3 s | 12.1 s | 3.0 GB → 3.0 GB |
The cost is the reads: one record batch of a unit's *.mir.arrow is decoded
at a time and dropped, nothing is held for the dump, so memory does not move.
Watched per batch of the upgrade.
The bench in the lens plan (§G, the lens-build benches, 2026-11-04, build-1: 32 cores, 128 GB, debug
build of dwarves --features cli,lenses,mimir-fat, one run each, wall time and
peak RSS). Loading the dump dominates every lens. The graph-load row is
read and index only, with no lens work. These runs used --workspace-only
dumps. A full dump costs more per lens (about 8 GB on Rusternetes, section 1.5).
| step | Holger s / GB | dwarves s / GB | Gunnar s / GB |
|---|---|---|---|
dump (cargo mimir-dump --workspace-only) | 127.2 / 4.29 | 126.9 / 2.39 | 111.9 / 2.48 |
graph-load | 32.2 / 2.16 | 18.6 / 1.35 | 40.3 / 2.76 |
twins | 39.0 / 3.12 | 22.6 / 1.73 | 48.0 / 4.01 |
zombie | 28.9 / 2.17 | 17.5 / 1.36 | 38.0 / 2.77 |
risk | 62.6 / 2.35 | 33.9 / 1.46 | 79.4 / 3.02 |
These rows were measured after the loader fix (dwarves 354f668: ids normalised in
place, dedupe by index, the binary layer not read by the graph lenses). Before
it, graph-load was 35.8 s / 3.44 GB on Holger. The plan measured the other
lenses only before the fix. Then they cost the load plus 1–7 s (for example
size 37.7 s, cycles 34.9 s on Holger); gate-bypass with the CFPQ walk
53.6 s; verdict 74.1 s, since it runs twins, size, dispatch, cycles and the
gate inside. dupes refuses in about 10 ms (no producer).
After lane graph-fast (section 4.0; plan §G block GRAPH-FAST, build-1: 24 cores in
the CI image, 94 GB, warm page cache), graph-load, wall s / peak RSS GB:
| build, path | Holger | dwarves | Gunnar |
|---|---|---|---|
| debug, before | 32.8 / 2.24 | 19.1 / 1.40 | 43.5 / 2.87 |
| release, before | 9.1 / 2.22 | 5.3 / 1.38 | 11.5 / 2.85 |
| release, no cache | 1.4 / 3.4 | 0.85 / 2.2 | 3.0 / 4.3 |
| release, cached | 0.78 / 2.1 | 0.43 / 1.1 | 0.85 / 2.4 |
| debug, cached | 3.0 / 2.2 | 1.8 / 1.0 | 3.6 / 2.5 |
Loading no longer dominates: dwarves lens all in release costs 11.8 s on Holger
(13 lenses), 3.2 s on dwarves and 7.2 s on Gunnar (11 lenses each), one load
included. With the 20-lens default set (2026-11-05, "The default set" above):
7.5 s / 2.75 GB on Holger against 6.7 s / 2.73 GB for the old eleven on the same box. On Holger the rest is lens compute, mostly single-threaded Rust: verdict
3.8 s (it reruns twins and the gate), gate-bypass 3.2 s, dsm 2.1 s, twins 1.2 s.
GraphBLAS is a small share: one OpenMP thread instead of 24 adds about 1 s to the run.
The study lenses (taint, chokepoints, misplaced, dsm) have their own bench in plan §F.1; each section's Cost part quotes it.
nornir lens costs stated in the code: a room build for the whole workspace was
28 s on dwarf (2026-10-16, src/mimir_lens/queued.rs:3); folding the
dependency universe for data_reach and system_boundary was 8.2 M nodes in 951 s
(room_feed/lenses.rs:1169, room_feed/boundary.rs:51-53).
9. What is not wired
No nornir lens is not_wired today: WAITING is empty
(src/mimir_lens/mod.rs:185; test no_lens_waits_for_a_producer_any_more,
room_feed/room_tests.rs:803). These inputs still have no producer, and each
lens that needs one names it:
| missing fact | producer that is missing | lenses that name it |
|---|---|---|
| per-commit file sets | none (git heat holds counts only) | anomaly (coupling is a proxy) |
| semver diffs, callers outside the workspace | none (P5) | change_risk, world_impact |
| binary call edges | none (P4) | binary_drift |
calls through overloaded operators (a + b on an Add impl, Index, Deref) | the driver's HIR walk writes no CALLS edge for them (P6); MIR call terminators do carry them | every caller and reach figure built on CALLS |
| the sensitive-data term in trust | trust does not read TOUCHES_TABLE | trust |
| binary bytes on a source-only dump | cargo mimir-dump --build | dwarves mimir bloat, zombie bytes, verdict |
binary_facts (artifact totals) | none in Mímir v2 | dwarves mimir bloat |
| churn in the dwarves folds | nornir's git-heat feed | dwarves mimir risk, verdict |
| line coverage | an llvm-cov import (coverage_fn) | change_risk, dwarves mimir risk, verdict |
embeddings for dwarves dupes | an embedding producer over the driver's spans | dwarves dupes |
| the package surface in the verdict | graph-package output is not folded | verdict |
| MIR for bodies over 1,024 blocks | given since schema 6 (no cap; dumps older than 6 cut them) | gate-bypass, twins |
| literal values | name-free MIR; pass --src | dispatch-erosion, error-shape |