src/mimir_lens/subclone.rs
//! **Lens — `subclone`: SUB-ITEM clone regions in one repo's source.**
//!
//! The shipping path for [`crate::mimir_subclone`], which until 2026-08-26 had
//! none: `detect` worked, twenty tests asserted it worked, and no CLI, no lens
//! and no job called it. `grep -rn mimir_subclone src/` outside the module
//! returned two doc comments.
//!
//! ## Why this lens is unlike every other one
//!
//! Every other Mímir lens folds the code graph or a warehouse table. This one
//! folds SOURCE TEXT, because the thing it detects — a shared run *smaller
//! than the item it sits in* — has no representation in a graph of items or in
//! `embeddings`. A chunk-keyed table can only ever express a whole item; that
//! is precisely the blindness the detector exists to remove, so the fold reads
//! the materialised checkout at `<ws>/git/<repo>`, the sibling of the warehouse
//! root that [`crate::feed_chain`] already resolves against.
//!
//! That is stated in the feed record as a source root rather than a table, so
//! nothing pretends this reading came out of Iceberg.
//!
//! The ITEM SPANS it judges with — which item encloses a site, whether a site
//! lies in a callable body (R6) — are an input ([`Spans`]): the code room's
//! node spans for the repo. A file the room holds no span of goes in span-less
//! and is counted, never dropped.
//!
//! ## Why there is NO workspace scope
//!
//! MEASURED (`faf0517`): **65 findings on `nornir/src` at the shipped
//! defaults; 347 across three repos.** Re-measured through this lens's own
//! corpus: **70 on nornir**. 70 is a list a person works through. 347 is a list
//! they close, and the ledger already carries the same lesson one size up —
//! *"10 152 clusters at cosine ≥0.80 — far too many to act on"*.
//!
//! One-repo-at-a-time was a convention nothing enforced. Here it is
//! STRUCTURAL: `repo = ""` does not scan the constellation, it REFUSES and
//! names the repos you can ask for. That also keeps the boot chain cheap — the
//! workspace-scope job the monitor submits per populate costs a directory
//! listing, not a full lex of every checkout.
//!
//! ## The three states, which must never draw the same
//!
//! * **not scanned — no source tree.** `scanned: false`, `files_scanned: 0`,
//! the feed marked `missing`, and `NO SOURCE TREE` in the note.
//! * **scanned, nothing found.** `scanned: true` with the file count it looked
//! at. This is an ANSWER.
//! * **no fold has run at all.** Not this fold's business — `lens_runs` has no
//! row and [`crate::warehouse::lens_cache::not_computed`] renders it. Three
//! states, three renderings; the `read_ledger_jobs` bug (`Ok(vec![])` for both
//! absent and empty, and a deleted workspace painting invented jobs) is the
//! one this repo has already paid for.
//!
//! ## What is NOT exposed
//!
//! The overlap threshold. MEASURED: 1057 / 1001 / 979 item pairs at ≥0.70 /
//! 0.80 / 0.90 — 8 % across the whole usable range. It is the wrong knob, and
//! putting it in the lens surface would invite people to turn it. The lever is
//! normalisation, which stays ON by default ([`SubcloneOptions`] documents the
//! fixture that goes red if OFF ever regains the verified top find) and is
//! reachable through `NORNIR_SUBCLONE_NORMALIZE=0` with no code edit.
use std::path::{Path, PathBuf};
use serde_json::{Value, json};
use super::sources::Feed;
use crate::mimir_subclone::{self, SubcloneOptions, SubcloneReport};
use crate::vector::spans::ItemSpan;
/// Where the item spans of a repo's files come from.
pub struct Spans<'a> {
/// The spans of one file (path relative to the checkout root); `None` =
/// nothing is known about the file.
pub of: &'a dyn Fn(&str) -> Option<Vec<ItemSpan>>,
/// What supplied them — or why there are none. Said in the note whenever
/// a file went in span-less.
pub source: String,
}
impl Spans<'_> {
/// No span source at all, with the reason.
pub fn none(why: impl Into<String>) -> Spans<'static> {
Spans { of: &|_| None, source: why.into() }
}
}
/// Directories never worth lexing: build output, git internals, and vendored
/// third-party source (a clone between two vendored copies of the same crate is
/// true and useless).
const SKIP_DIRS: &[&str] = &["target", ".git", "vendor", "node_modules"];
/// **The corpus is every `src` tree, at any depth — and nothing else.**
///
/// MEASURED on this repo (2026-08-26, shipped defaults, Type-2 ON), which is
/// the whole argument — reproduce it with `subclone_volume_on_this_repo`:
///
/// | corpus | files | findings |
/// |---------------------------------|-------|----------|
/// | every `src` tree — **the lens** | 278 | **70** |
/// | top-level `src/` only | 220 | 65 |
/// | whole checkout | 407 | 120 |
///
/// 120 is the wrong side of the volume rule. The 50 extra findings are almost
/// all `tests/`, `examples/` and `benches/` scaffolding — real duplication, and
/// a different job from this one: a shared 30-line fixture between two
/// integration tests is usually the *point* of the fixture. Including them
/// nearly doubles the list a person is asked to work through, in order to add a
/// class of finding they will skip.
///
/// A `src` COMPONENT rather than a top-level `src/`, so a multi-crate repo's
/// `crates/<x>/src` is in — it is library source by every meaning of the word.
/// That is +58 files and +5 findings here, and two of the top five come from
/// it, so excluding it would be hiding real duplication to make a number
/// prettier.
///
/// **70 is the number a person sees, and 25 is the page.** The `default_limit`
/// in `LENSES` is 25, ranked by `damage` = lines × (sites − 1), with
/// `findings_total` always stating the full 70 so the truncation is visible.
const CORPUS_DIR: &str = "src";
/// The feed name for the source tree. Deliberately not a table name — this lens
/// does not read Iceberg, and a feed record that implied it did would be the
/// same lie in the other direction.
pub const SOURCE_FEED: &str = "<source tree>";
/// The materialised checkout for `repo`: `<ws>/git/<repo>`, the sibling of the
/// warehouse root. Same resolution `feed_chain`'s coverage producer uses, so
/// there is one answer to "where is the source" (LAW 5).
pub fn source_root(warehouse_root: &Path, repo: &str) -> PathBuf {
warehouse_root
.parent()
.map(|p| p.join("git").join(repo))
.unwrap_or_else(|| PathBuf::from(repo))
}
/// The repos that HAVE a checkout, sorted — what a refusal names, so "you must
/// pick one" is never a dead end.
pub fn repos_available(warehouse_root: &Path) -> Vec<String> {
let Some(git_dir) = warehouse_root.parent().map(|p| p.join("git")) else {
return Vec::new();
};
let Ok(rd) = std::fs::read_dir(&git_dir) else {
return Vec::new();
};
let mut out: Vec<String> = rd
.filter_map(|e| e.ok())
.filter(|e| e.path().is_dir())
.filter_map(|e| e.file_name().into_string().ok())
.collect();
out.sort();
out
}
/// Every `.rs` file inside a [`CORPUS_DIR`] tree under `root`, as
/// `(path-relative-to-root, source)`.
///
/// Unreadable files are SKIPPED, never silently substituted with an empty
/// string — an empty source tokenises to nothing and would read as a clean file.
pub(crate) fn rust_sources(root: &Path) -> Vec<(String, String)> {
let mut out: Vec<(String, String)> = Vec::new();
// Pointing the walker AT a `src` tree makes everything under it source —
// otherwise the component test below could never match and the corpus would
// be silently empty, which is the exact failure this lens refuses to make
// elsewhere.
let root_is_src = root.file_name() == Some(std::ffi::OsStr::new(CORPUS_DIR));
for entry in walkdir::WalkDir::new(root)
.into_iter()
.filter_entry(|e| {
!e.file_type().is_dir()
|| !e
.file_name()
.to_str()
.is_some_and(|n| SKIP_DIRS.contains(&n))
})
.filter_map(|e| e.ok())
{
if !entry.file_type().is_file() {
continue;
}
if entry.path().extension().and_then(|e| e.to_str()) != Some("rs") {
continue;
}
let Ok(text) = std::fs::read_to_string(entry.path()) else {
continue;
};
let rel = entry.path().strip_prefix(root).unwrap_or(entry.path());
// Inside a `src` tree, at any depth. See CORPUS_DIR.
if !root_is_src
&& !rel
.components()
.any(|c| c.as_os_str() == std::ffi::OsStr::new(CORPUS_DIR))
{
continue;
}
out.push((rel.to_string_lossy().into_owned(), text));
}
out.sort_by(|a, b| a.0.cmp(&b.0));
out
}
/// **The lens.** `repo` is REQUIRED — see the module docs on why there is no
/// workspace scope. `limit` truncates the ranked `worst` list only; the totals
/// beside it always describe the whole scan.
pub fn analyze(warehouse_root: &Path, repo: &str, limit: usize, spans: &Spans<'_>) -> Value {
if repo.is_empty() {
return refused(warehouse_root);
}
let root = source_root(warehouse_root, repo);
if !root.is_dir() {
return not_scannable(repo, &root, "the directory does not exist");
}
let files = rust_sources(&root);
if files.is_empty() {
return not_scannable(repo, &root, "it holds no readable `.rs` files");
}
// The environment door, with its own line in the envelope: a knob that was
// read is stated, and a knob whose value did not parse is named LOUDLY
// rather than falling back in silence.
let env = mimir_subclone::options_from_env();
let files = files
.into_iter()
.map(|(p, t)| {
let sp = (spans.of)(&p);
(p, t, sp)
})
.collect();
let report = mimir_subclone::detect(files, &env.opts);
envelope(repo, &root, &report, &env.opts, &env.line(), limit, &spans.source)
}
/// The refusal. `scanned: false` and `scope_required: true`, because a person
/// reading this must not mistake it for "the constellation is clean".
fn refused(warehouse_root: &Path) -> Value {
let repos = repos_available(warehouse_root);
json!({
"kind": "subclone",
"scanned": false,
"scope_required": true,
"repo": "",
"files_scanned": 0,
"findings_total": 0,
"worst": [],
"repos_available": repos,
"feeds": [Feed {
table: SOURCE_FEED.to_string(),
rows: 0,
root: String::new(),
missing: true,
}],
"evidence_note":
"subclone: this lens is PER-REPO by construction and nothing was scanned. \
65 findings on one repo is a list you work through; 347 across three is a \
list you close, so there is no workspace scope to ask for. Name a repo. \
This is a REFUSAL, not a clean constellation.",
})
}
/// Could not scan. Distinct from a scan that found nothing, and it says which.
fn not_scannable(repo: &str, root: &Path, why: &str) -> Value {
json!({
"kind": "subclone",
"scanned": false,
"scope_required": false,
"repo": repo,
"files_scanned": 0,
"findings_total": 0,
"worst": [],
"source_root": root.display().to_string(),
"feeds": [Feed {
table: SOURCE_FEED.to_string(),
rows: 0,
root: root.display().to_string(),
missing: true,
}],
"evidence_note": format!(
"subclone: NO SOURCE TREE for `{repo}` at {} — {why}. Nothing was scanned, so \
this is the absence of a READING, not an absence of clones.",
root.display()
),
})
}
/// The scanned answer — findings first, and every count the scan could not
/// reach stated beside them.
fn envelope(
repo: &str,
root: &Path,
r: &SubcloneReport,
opts: &SubcloneOptions,
env_line: &str,
limit: usize,
span_source: &str,
) -> Value {
let worst: Vec<Value> = r
.findings
.iter()
.take(limit)
.map(|f| {
let lead = r.groups.get(f.groups[0]);
let location = lead
.and_then(|g| g.sites.first())
.map(|s| s.location())
.unwrap_or_default();
let sub_item = lead
.map(|g| g.sites.iter().any(|s| s.sub_item))
.unwrap_or(false);
json!({
"key": f.key,
// The `subject` the typed projection ranks on — one region, one
// name, and the file it leads from.
"lead_file": f.files.first().cloned().unwrap_or_default(),
"files": f.files,
"file_count": f.files.len(),
"sites": f.sites,
"lines": f.lines,
"damage": f.damage,
// How many GROUPS restate this one region. The largest region on
// nornir's own src/ is restated by 38 of them, which is why the
// headline count is regions and not groups.
"restated_by": f.groups.len(),
"location": location,
"sub_item": sub_item,
"explain": format!(
"{} lines shared across {} files at {} sites (damage {:.0}); \
{} group(s) restate this one region",
f.lines,
f.files.len(),
f.sites,
f.damage,
f.groups.len()
),
})
})
.collect();
let feeds = vec![Feed {
table: SOURCE_FEED.to_string(),
rows: r.files_scanned,
root: root.display().to_string(),
missing: r.files_scanned == 0,
}];
// The note carries what the scan COULD NOT do, in the same sentence as what
// it did. A skipped bucket or an unparsed file makes every count below a
// lower bound, and saying so is the difference between a reading and a
// claim.
let mut note = format!(
"subclone: scanned {} file{} of `{repo}` — {} finding(s) from {} group(s) \
(before subsumption {}), guarantee {} shape tokens.",
r.files_scanned,
if r.files_scanned == 1 { "" } else { "s" },
r.findings.len(),
r.groups.len(),
r.groups_before_subsumption,
r.guarantee_tokens,
);
if r.files_unparsed > 0 {
note.push_str(&format!(
" ⚠ {} file(s) did NOT tokenise, so every count is a LOWER bound.",
r.files_unparsed
));
}
if r.buckets_skipped > 0 {
note.push_str(&format!(
" ⚠ {} fingerprint bucket(s) exceeded max_bucket and were SKIPPED — the sweep \
is INCOMPLETE.",
r.buckets_skipped
));
}
if r.files_without_spans > 0 {
note.push_str(&format!(
" ⚠ {} file(s) carried no item spans ({span_source}) — their sites name no \
enclosing item and R6 left {} group(s) UNJUDGED.",
r.files_without_spans, r.groups_r6_unjudged
));
}
if r.findings.is_empty() && r.files_unparsed == 0 && r.buckets_skipped == 0 && r.files_without_spans == 0 {
note.push_str(" Nothing shared clears the gate — this is a clean READING.");
}
note.push(' ');
note.push_str(env_line);
json!({
"kind": "subclone",
"scanned": true,
"scope_required": false,
"repo": repo,
"source_root": root.display().to_string(),
"files_scanned": r.files_scanned,
"files_unparsed": r.files_unparsed,
"tokens_total": r.tokens_total,
// BOTH counts, always: the reduction from groups to regions is the whole
// reason the headline is actionable, and a reduction nobody can see is a
// reduction nobody can check.
"findings_total": r.findings.len(),
"groups": r.groups.len(),
"groups_before_subsumption": r.groups_before_subsumption,
"groups_non_callable": r.groups_non_callable,
"groups_r6_unjudged": r.groups_r6_unjudged,
"files_without_spans": r.files_without_spans,
"idiom_groups": r.idiom_groups,
"buckets_skipped": r.buckets_skipped,
"runs_rejected": r.runs_rejected,
"guarantee_tokens": r.guarantee_tokens,
"options": opts,
"environment": env_line,
"span_source": span_source,
"worst": worst,
"feeds": feeds,
"evidence_note": note,
})
}
#[cfg(test)]
mod tests {
use super::*;
fn tree(files: &[(&str, &str)]) -> tempfile::TempDir {
let tmp = tempfile::tempdir().unwrap();
for (rel, body) in files {
let p = tmp.path().join("git").join("korp").join(rel);
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, body).unwrap();
}
std::fs::create_dir_all(tmp.path().join("warehouse-root")).unwrap();
tmp
}
/// **The corpus, as a test.** Every `src` tree at any depth is IN — a
/// multi-crate repo's `crates/<x>/src` is source. `tests/`, `examples/` and
/// `benches/` are OUT, and that is the volume decision, not tidiness: on
/// this repo they take 65 findings to 120. `target/` and `vendor/` are out
/// because a clone between two vendored copies of one crate is true and
/// worthless, and build output would dwarf the real tree.
#[test]
fn the_corpus_is_every_src_tree_and_nothing_else() {
let tmp = tree(&[
("src/lib.rs", "pub fn a() {}\n"),
("crates/inner/src/lib.rs", "pub fn b() {}\n"),
("target/debug/src/build.rs", "pub fn generated() {}\n"),
("vendor/other/src/lib.rs", "pub fn vendored() {}\n"),
("tests/integration.rs", "pub fn t() {}\n"),
("examples/demo.rs", "pub fn d() {}\n"),
("benches/bench.rs", "pub fn bm() {}\n"),
("build.rs", "pub fn build() {}\n"),
("src/notes.md", "not rust\n"),
]);
let files = rust_sources(&tmp.path().join("git").join("korp"));
let names: Vec<&str> = files.iter().map(|(p, _)| p.as_str()).collect();
for want in ["src/lib.rs", "crates/inner/src/lib.rs"] {
assert!(
names.iter().any(|n| n.replace('\\', "/") == want),
"{want} must be in the corpus: {names:?}"
);
}
for out in ["target", "vendor", "tests", "examples", "benches", ".md"] {
assert!(!names.iter().any(|n| n.contains(out)), "{out}: {names:?}");
}
assert!(!names.iter().any(|n| *n == "build.rs"), "{names:?}");
assert_eq!(files.len(), 2);
}
/// The source root is the warehouse root's SIBLING `git/<repo>` — the same
/// resolution the coverage feed uses. Getting this wrong reads as "no
/// source tree", which is honest but permanently wrong.
#[test]
fn the_source_root_is_the_sibling_checkout() {
let root = source_root(Path::new("/srv/ws/warehouse-root"), "korp");
assert_eq!(root, Path::new("/srv/ws/git/korp"));
}
/// **The volume instrument.** `#[ignore]`d because it reads the LIVE tree
/// and its number therefore moves with the code — an assertion on it would
/// go red for reasons that have nothing to do with this lens. But "measure
/// what a person actually sees and defend that number" needs something to
/// run, so here it is, reproducible:
///
/// ```text
/// cargo test -p nornir --lib --features … subclone_volume -- --ignored --nocapture
/// ```
///
/// It prints findings, groups, the truncated page and the top region, so the
/// `default_limit` in `LENSES` can be re-argued against a fresh reading
/// rather than against this comment.
#[test]
#[ignore = "reads the live source tree; run with --ignored to re-measure the volume"]
fn subclone_volume_on_this_repo() {
let manifest = Path::new(env!("CARGO_MANIFEST_DIR"));
// BOTH corpora, because they are different numbers and only one of them
// is what the lens actually reports: `src/` alone is the corpus
// `faf0517` measured 65 on, and the WHOLE checkout is what the lens
// walks (it cannot know which sub-tree a repo calls its own).
for (label, root) in [
("the lens corpus (every src tree)", manifest.to_path_buf()),
("top-level src/ only", manifest.join("src")),
] {
let files = rust_sources(&root);
let env = mimir_subclone::options_from_env();
let files = files.into_iter().map(|(p, t)| (p, t, None)).collect();
let report = mimir_subclone::detect(files, &env.opts);
let v = envelope("nornir", &root, &report, &env.opts, &env.line(), 25, "no span source: the volume instrument reads the tree alone");
println!(
"subclone volume [{label}] — files {} findings {} groups {} \
(before subsumption {}) page {}",
v["files_scanned"], v["findings_total"], v["groups"],
v["groups_before_subsumption"],
v["worst"].as_array().unwrap().len()
);
for f in v["worst"].as_array().unwrap().iter().take(5) {
println!(
" {:>8.0} {:>2} files restated_by {:>2} {}",
f["damage"].as_f64().unwrap_or(0.0),
f["file_count"],
f["restated_by"],
f["location"].as_str().unwrap_or("")
);
}
println!(" note: {}", v["evidence_note"].as_str().unwrap_or(""));
}
}
/// A refusal must be usable: it names the repos that DO have a checkout.
#[test]
fn the_refusal_names_the_repos_you_can_ask_for() {
let tmp = tree(&[("src/lib.rs", "pub fn a() {}\n")]);
std::fs::create_dir_all(tmp.path().join("git").join("dwarves")).unwrap();
let v = analyze(&tmp.path().join("warehouse-root"), "", 50, &Spans::none("test"));
assert_eq!(v["scanned"], false);
assert_eq!(v["scope_required"], true);
let repos: Vec<&str> = v["repos_available"]
.as_array()
.unwrap()
.iter()
.map(|r| r.as_str().unwrap())
.collect();
assert_eq!(repos, vec!["dwarves", "korp"]);
assert!(v["evidence_note"].as_str().unwrap().contains("PER-REPO"));
assert!(
v["evidence_note"].as_str().unwrap().contains("REFUSAL"),
"a refusal must not read as a clean constellation"
);
}
/// An EMPTY tree and an ABSENT tree are both "not scanned", and both say
/// which — but neither may read as "scanned, found nothing".
#[test]
fn an_empty_tree_and_an_absent_tree_are_both_unscannable_and_say_which() {
let tmp = tree(&[("src/lib.rs", "pub fn a() {}\n")]);
let wh_root = tmp.path().join("warehouse-root");
let absent = analyze(&wh_root, "no-such-repo", 50, &Spans::none("test"));
assert_eq!(absent["scanned"], false);
assert!(
absent["evidence_note"].as_str().unwrap().contains("does not exist"),
"{absent}"
);
std::fs::create_dir_all(tmp.path().join("git").join("docs-only")).unwrap();
std::fs::write(
tmp.path().join("git").join("docs-only").join("README.md"),
"hi",
)
.unwrap();
let empty = analyze(&wh_root, "docs-only", 50, &Spans::none("test"));
assert_eq!(empty["scanned"], false);
assert!(
empty["evidence_note"]
.as_str()
.unwrap()
.contains("no readable `.rs` files"),
"{empty}"
);
// Both are NOT SCANNED, and neither claims a clean reading.
for v in [&absent, &empty] {
assert!(v["evidence_note"].as_str().unwrap().contains("NO SOURCE TREE"));
assert!(!v["evidence_note"].as_str().unwrap().contains("clean READING"));
}
}
}