.nornir/mimir-analyze-duplicates.md

Mimir `analyze duplicates` — copied-not-moved detection

Class: design · A warehouse-native code analysis that finds the same code living in two places, so the move-means-delete + REUSE laws can be enforced by a query instead of by hand. This is the capability that would have found the release/brokkr duplication the extermination wave cleaned manually.

What it does

nornir mimir analyze duplicates clusters symbols that are the same code copied into more than one crate/repo, and prints a keep/kill recommendation per cluster as JSON (the CLI twin of the Knowledge/Mimir state_json).

A vector index alone is noisy — a trait impl and its default method embed alike — so the detector keeps a pair only when both signals agree:

semantic: embedding cosine ≥ --cosine (default 0.92), AND
structural: signature-token + callee-set agreement ≥ --structural (default 0.5).

Cross-scope survivors (different crates) are unioned into clusters, ranked by avg_cosine × size, and each cluster gets a canonical keep (preferring a shared reuse home — edda, skidbladnir, znippy-zoomies, facett, plus any --canonical <crate>) with the rest marked repoint (a home exists) or kill (no home).

Shape (two layers)

Pure core — src/mimir_analyze.rs. Takes materialised SymbolVec rows (embedding + structural fingerprint) → DuplicateReport. No warehouse/GPU dependency, exhaustively unit-tested (cosine, structural gate, union-find clustering, ranking, canonical pick, state_json).
- assemble_symbol_vecs(repo, symbols, calls, chunks, vectors) is the PURE join that turns raw warehouse rows into SymbolVecs: a symbol inherits the vector of the smallest embedding_manifest chunk that covers its file:line; its callees come from call_edges grouped by (file, caller last segment); a symbol with no covering chunk is dropped.
Warehouse read — src/mimir_dup.rs (analyze_warehouse_duplicates). The thin wiring: per repo, load the latest-snapshot symbols + call-edges via the tested knowledge::query::load_preferred; read every embedding_manifest span + its embeddings vector; assemble_symbol_vecs each repo; run the core over all repos at once. Defensive — an empty/embedding-less warehouse yields an empty (green) report, never an error.

Join keys

symbol_facts(repo, crate, module, item_name, file, line, signature)
   ⋈ (file, line ∈ [start,end])  embedding_manifest(content_hash, file, start_line, end_line)
   ⋈ content_hash                embeddings(content_hash, vector: list<f32>)
   ⋈ (file, last_seg(caller_path)==item_name)  call_edges(caller_path, callee_ident)

CLI

nornir mimir analyze duplicates \
    [--cosine 0.92] [--structural 0.5] [--all-scopes] [--canonical <crate> ...]

--all-scopes also reports same-crate near-duplicates (default: cross-scope only).
Warehouse-native / fat mode: reads the persisted knowledge map directly, needs no cargo metadata dep graph and has no gRPC twin, so it runs locally regardless of NORNIR_SERVER.

Tests

src/mimir_analyze.rs — the detector + the join (assemble_joins_vector_and_callees_and_drops_uncovered is red-when-broken on span containment; assembled_rows_feed_the_duplicate_detector is the end-to-end assemble→analyze path).
src/mimir_dup.rs — default_opts_name_the_shared_reuse_homes pins the reuse-home set.

Remaining

The pure core + join + CLI are wired and unit-tested; an end-to-end run against a freshly nornir populated + embedded warehouse (real clusters surfacing) is the live-drive follow-up, plus a facett cluster-list view over DuplicateReport.

Mimir analyze duplicates — copied-not-moved detection