src/mimir_groupclone.rs
//! Mimir **group twins** — the same semantic GROUP of statements inside two
//! DIFFERENT functions, found even when the copy is interleaved with noise.
//!
//! # The hole this fills
//!
//! [`crate::mimir_subclone`] removed the item boundary: winnowing finds a
//! shared run *inside* two different functions. But winnowing's unit is a
//! **contiguous token run**, and its own first guarantee is the blindness this
//! module targets: *nothing shorter than `k` tokens can ever match*. Interleave
//! a copied block with two or three unrelated lines (a log line, an unrelated
//! local) and the shared text falls apart into fragments each shorter than `k`
//! — unreportable **by construction**, at every threshold. The oracle in this
//! file plants exactly that and asserts the subclone detector reports nothing
//! for it at the settings that find the contiguous copy whole
//! (`winnowing_is_blind_to_the_interleaved_copy_and_this_module_is_not`).
//!
//! # Semantic groups (v1: a line/statement approximation, said plainly)
//!
//! A function body is segmented into **semantic groups** by def-use
//! clustering over its statements:
//!
//! * a statement joins a group when it READS a name the group has written, or
//! WRITES a name the group already uses (`let` / `for` bindings and
//! assignments define; every other workspace-chosen lower-case identifier in
//! value position reads);
//! * a control-flow header binds through its own condition's def-use links;
//! a header with none (`loop {`) binds to its block's dominant group, and a
//! closing brace to its header;
//! * a statement sharing no name with any group — a log line, an unrelated
//! local chain — falls into its own group. That is what makes the
//! interleaved copy recoverable: the copy's statements re-connect through
//! their own names, the noise stays outside.
//!
//! This is deliberately NOT a dataflow analysis: it is deterministic,
//! identifier-driven and line/statement-level, built on the one lexer this
//! repo allows (`proc-macro2`, through [`crate::mimir_subclone::FlatTok`]'s
//! walk with source lines kept). Reads of a name no statement in the body
//! defined (a parameter, a global) link nothing — otherwise every user of a
//! parameter would collapse into one group.
//!
//! # Normalisation and the vector
//!
//! Each group of at least [`MIN_GROUP_TOKENS`] normalised tokens is normalised
//! **by the same rule as `semantic_clones`'s bodies**
//! ([`crate::clone_signals::norm::normalize_tokens`]: names → numbered roles,
//! literals → kinds, visibility gone), numbered per GROUP so the same group
//! inside two different functions renders the same text. Its SHA-256 is the
//! group's **exact key**, and the key of its vector in `embeddings_groups` —
//! written through the SAME seam as `embeddings_norm`
//! ([`crate::clone_signals::feed::embed_groups`], the `clone_shapes` feed's
//! deferred half). Tests embed through a deterministic bag; no worker is ever
//! called here.
//!
//! # Pairing
//!
//! Two groups of DIFFERENT functions are a **group twin** when
//!
//! * their vectors' cosine ≥ [`COSINE_FLOOR`] (the same exact key is one
//! normalised body: cosine 1.0 by construction, no vector needed), AND
//! * their structure agrees: at least [`MIN_SHARED_KGRAMS`] shared
//! [`KGRAM`]-grams over the normalised ROLE-CLASS alphabet (the same
//! alphabet `near_clones`' MinHash shingles use — the role-aware cousin of
//! the subclone shape alphabet), of which at least one is RARE
//! ([`RARE_DF`]). A model that merely reads two groups alike never pairs
//! them on its own.
//!
//! A pair whose structure agrees but whose vector is missing on either side is
//! COUNTED (`candidates_unjudged`), never silently dropped — the lens turns
//! that count into an `evidence_note` sentence.
//!
//! # The decided constants — MEASURED on this file's own fixtures, 2026-10-19
//! (`the_constants_are_the_measured_ones` re-measures them on every run)
//!
//! * the planted 8-statement group normalises to **102 tokens** (the tail
//! statement that reads `acc` rides into the group — def-use, not
//! adjacency); the noise groups the oracle interleaves it with are **7–19**;
//! [`MIN_GROUP_TOKENS`] = 30 sits between with margin on both sides.
//! * bag-embedder cosines on the fixtures: the same group with one statement
//! reworded **0.991**; the planted group vs. an unrelated control group
//! **0.796**. [`COSINE_FLOOR`] = 0.90 matches `semantic_clones`'
//! `NORM_FLOOR` and splits them — but the margin under the floor is thin,
//! because role normalisation makes even unrelated groups read alike to a
//! bag (`let` / `for` / `V1` everywhere). The floor is model-relative: the
//! fixture value is the FALLBACK for unmeasured spaces, and the real
//! embedder's own operating point is MEASURED and RECORDED
//! ([`GROUP_RECORDED`], v2 — jina-v2-base-code runs at 0.967, μ−1σ,
//! verdict CALIBRATED). The lens states which basis it ran with.
//! * shared role-class k-grams at k = [`KGRAM`] = 8: **81** with the reworded
//! copy, **4** with the control — and those 4 are generic runs a two-item
//! scan even counts as rare. Neither gate stands alone: the cosine floor
//! blocks the control, the rare-k-gram gate blocks model-only lookalikes.
use std::collections::{BTreeMap, BTreeSet, HashMap};
use crate::clone_signals::norm::{self, NormBody};
use crate::mimir_subclone::FlatTok;
/// Minimum normalised tokens a group must have to be reported or embedded.
/// Below it a group is a statement or two — the same false-positive class
/// every clone floor in this repo guards (`MIN_TOKENS` 40 for whole bodies;
/// a group is smaller than a body, so its floor is lower). Measured against
/// the fixtures: planted 102, noise 7–19.
pub const MIN_GROUP_TOKENS: usize = 30;
/// Cosine floor for a pair whose exact keys differ — the same operating point
/// as `semantic_clones`' normalised-model floor (`NORM_FLOOR` 0.90).
pub const COSINE_FLOOR: f32 = 0.90;
/// k-gram length over the normalised role-class alphabet (the MinHash shingle
/// alphabet of `near_clones`), for the structure-agreement signal.
pub const KGRAM: usize = 8;
/// A pair must share at least this many k-grams …
pub const MIN_SHARED_KGRAMS: usize = 2;
/// … of which at least one occurs in at most this many DISTINCT normalised
/// group bodies across the scan (a rare k-gram: shared boilerplate shapes
/// never carry a pair alone).
pub const RARE_DF: usize = 2;
/// A k-gram occurring in more than this many groups is boilerplate: its
/// candidate bucket is SKIPPED and COUNTED, never silently dropped.
pub const MAX_BUCKET: usize = 100;
/// The gates, in one struct so a report can state what it ran with.
#[derive(Debug, Clone, Copy, PartialEq, serde::Serialize)]
pub struct TwinOptions {
pub cosine_floor: f32,
pub min_group_tokens: usize,
pub kgram: usize,
pub min_shared_kgrams: usize,
pub rare_df: usize,
pub max_bucket: usize,
}
impl Default for TwinOptions {
fn default() -> Self {
Self {
cosine_floor: COSINE_FLOOR,
min_group_tokens: MIN_GROUP_TOKENS,
kgram: KGRAM,
min_shared_kgrams: MIN_SHARED_KGRAMS,
rare_df: RARE_DF,
max_bucket: MAX_BUCKET,
}
}
}
// ---------------------------------------------------------------------------
// Lexing with lines — the one lexer, with the source line kept per token
// ---------------------------------------------------------------------------
/// One flat token with its 1-based source line (relative to the text handed in).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LineTok {
pub tok: FlatTok,
pub line: u32,
}
/// Lex `text` into flat tokens with lines, attributes (`#[…]`) left out — the
/// same stream [`crate::mimir_subclone::flat_tokens`] yields, line-attributed.
/// `None` when the text does not tokenise.
pub fn flat_tokens_lines(text: &str) -> Option<Vec<LineTok>> {
let ts: proc_macro2::TokenStream = text.parse().ok()?;
let mut out = Vec::new();
walk(ts, &mut out);
Some(out)
}
fn walk(ts: proc_macro2::TokenStream, out: &mut Vec<LineTok>) {
let mut it = ts.into_iter().peekable();
while let Some(t) = it.next() {
match t {
proc_macro2::TokenTree::Punct(p) if p.as_char() == '#' => {
// `#[attr]`, `#![attr]`: an attribute is not behaviour
let line = p.span().start().line as u32;
let bang = matches!(it.peek(), Some(proc_macro2::TokenTree::Punct(q)) if q.as_char() == '!');
if bang {
let _ = it.next();
}
match it.peek() {
Some(proc_macro2::TokenTree::Group(g)) if g.delimiter() == proc_macro2::Delimiter::Bracket => {
let _ = it.next();
}
_ => {
out.push(LineTok { tok: FlatTok::Punct('#', false), line });
if bang {
out.push(LineTok { tok: FlatTok::Punct('!', false), line });
}
}
}
}
proc_macro2::TokenTree::Punct(p) => out.push(LineTok {
tok: FlatTok::Punct(p.as_char(), p.spacing() == proc_macro2::Spacing::Joint),
line: p.span().start().line as u32,
}),
proc_macro2::TokenTree::Ident(id) => out.push(LineTok { tok: FlatTok::Ident(id.to_string()), line: id.span().start().line as u32 }),
proc_macro2::TokenTree::Literal(l) => {
let s = l.to_string();
let kind = match s.as_bytes() {
[b'"', ..] | [b'r', b'"' | b'#', ..] | [b'b', b'"', ..] | [b'b', b'r', ..] | [b'c', b'"', ..] => 's',
[b'\'', ..] | [b'b', b'\'', ..] => 'c',
_ => 'n',
};
out.push(LineTok { tok: FlatTok::Lit(kind), line: l.span().start().line as u32 });
}
proc_macro2::TokenTree::Group(g) => {
let (o, c) = match g.delimiter() {
proc_macro2::Delimiter::Parenthesis => ('(', ')'),
proc_macro2::Delimiter::Brace => ('{', '}'),
proc_macro2::Delimiter::Bracket => ('[', ']'),
proc_macro2::Delimiter::None => {
walk(g.stream(), out);
continue;
}
};
out.push(LineTok { tok: FlatTok::Open(o), line: g.span_open().start().line as u32 });
walk(g.stream(), out);
out.push(LineTok { tok: FlatTok::Close(c), line: g.span_close().start().line as u32 });
}
}
}
}
// ---------------------------------------------------------------------------
// Segmentation
// ---------------------------------------------------------------------------
/// Rust keywords (the subclone / norm list): never a variable.
const KEYWORDS: &[&str] = &[
"as", "async", "await", "break", "const", "continue", "crate", "dyn", "else", "enum", "extern", "false", "fn", "for", "if", "impl", "in", "let", "loop",
"match", "mod", "move", "mut", "pub", "ref", "return", "self", "Self", "static", "struct", "super", "trait", "true", "type", "union", "unsafe", "use",
"where", "while", "yield", "_",
];
/// One semantic group of a function body.
#[derive(Debug, Clone, PartialEq)]
pub struct Group {
/// The source lines of the group, 1-based within the text handed to
/// [`segment`], sorted — NOT necessarily contiguous (the whole point).
pub lines: Vec<u32>,
/// The group's normalised body (names → roles numbered per group,
/// literals → kinds): `norm.exact` is its key everywhere.
pub norm: NormBody,
/// FNV k-gram hashes over `norm.classes` ([`KGRAM`]) — the structure signal.
pub kgrams: BTreeSet<u64>,
}
impl Group {
pub fn start_line(&self) -> u32 {
self.lines.first().copied().unwrap_or(0)
}
pub fn end_line(&self) -> u32 {
self.lines.last().copied().unwrap_or(0)
}
}
/// Is the identifier at `i` a workspace-chosen VARIABLE name (role `V`)?
/// Types, fields after `.`, calls, macros, lifetimes, keywords and the
/// language's own vocabulary ([`norm::KEEP`]) are not.
fn var_at(toks: &[LineTok], i: usize) -> Option<&str> {
let FlatTok::Ident(name) = &toks[i].tok else { return None };
let bare = name.trim_start_matches("r#");
if KEYWORDS.contains(&bare) || norm::KEEP.contains(&bare) {
return None;
}
if bare.chars().next().is_some_and(char::is_uppercase) {
return None; // a type / variant
}
let prev = i.checked_sub(1).map(|p| &toks[p].tok);
if matches!(prev, Some(FlatTok::Punct('.', _))) || matches!(prev, Some(FlatTok::Punct('\'', _))) {
return None; // a field / a lifetime
}
match toks.get(i + 1).map(|t| &t.tok) {
Some(FlatTok::Open('(')) => return None, // a call
Some(FlatTok::Punct('!', _)) if matches!(toks.get(i + 2).map(|t| &t.tok), Some(FlatTok::Open(_))) => return None, // a macro
_ => {}
}
// a path segment (`mod::item`): not a value read
let colon = |t: Option<&FlatTok>| matches!(t, Some(FlatTok::Punct(':', _)));
if colon(toks.get(i + 1).map(|t| &t.tok)) && colon(toks.get(i + 2).map(|t| &t.tok)) {
return None;
}
Some(bare)
}
/// Is the token at `j` an ASSIGNMENT `=` (not `==`, `=>`, `<=`, `>=`, `!=`,
/// and not the second half of a compound `+=`)?
fn assign_eq_at(toks: &[LineTok], j: usize) -> bool {
let Some(FlatTok::Punct('=', _)) = toks.get(j).map(|t| &t.tok) else { return false };
if matches!(toks.get(j + 1).map(|t| &t.tok), Some(FlatTok::Punct('=' | '>', _))) {
return false;
}
match j.checked_sub(1).map(|p| &toks[p].tok) {
Some(FlatTok::Punct(c, true)) if "=<>!".contains(*c) => false,
_ => true,
}
}
/// One statement, as segmentation sees it.
struct Stmt {
/// Token range `toks[from..to]`.
from: usize,
to: usize,
/// Depth-first id of the brace block whose HEADER this statement is
/// (it ends with `{`), if any.
opens_block: bool,
/// The statement is a bare `}` (possibly `} else {`): bound to its header.
closes_block: bool,
/// Excluded from every group: the signature, and the body's outer braces.
signature: bool,
}
/// Split `toks` into statements: a statement ends at a `;` outside any nested
/// delimiter, at a `{` opening a brace block (the header is its own
/// statement), or at a `}` closing one. The function signature — everything
/// up to and including the body's opening `{` — and the body's closing `}`
/// are marked and never grouped.
fn statements(toks: &[LineTok]) -> Vec<Stmt> {
let mut out: Vec<Stmt> = Vec::new();
// the body: the outermost brace group
let body_open = toks.iter().position(|t| matches!(t.tok, FlatTok::Open('{')));
let Some(body_open) = body_open else {
return out;
};
if body_open > 0 {
out.push(Stmt { from: 0, to: body_open + 1, opens_block: false, closes_block: false, signature: true });
} else {
out.push(Stmt { from: 0, to: 1, opens_block: false, closes_block: false, signature: true });
}
let mut depth_paren = 0i32; // ( and [
let mut start = body_open + 1;
let mut i = body_open + 1;
let body_close = toks.len().saturating_sub(1); // the matching outer `}` is last by construction
while i < body_close {
match &toks[i].tok {
FlatTok::Open('(') | FlatTok::Open('[') => depth_paren += 1,
FlatTok::Close(')') | FlatTok::Close(']') => depth_paren -= 1,
FlatTok::Open('{') if depth_paren == 0 => {
// header statement, `{` included
out.push(Stmt { from: start, to: i + 1, opens_block: true, closes_block: false, signature: false });
start = i + 1;
}
FlatTok::Open('{') => {
// a brace group inside parens (a closure body, a struct literal
// in an argument): stays inside the statement — skip to its close
let mut d = 1;
let mut j = i + 1;
while j < body_close && d > 0 {
match &toks[j].tok {
FlatTok::Open('{') => d += 1,
FlatTok::Close('}') => d -= 1,
_ => {}
}
j += 1;
}
i = j - 1;
}
FlatTok::Close('}') => {
if start < i {
out.push(Stmt { from: start, to: i, opens_block: false, closes_block: false, signature: false });
}
out.push(Stmt { from: i, to: i + 1, opens_block: false, closes_block: true, signature: false });
start = i + 1;
}
FlatTok::Punct(';', _) if depth_paren == 0 => {
out.push(Stmt { from: start, to: i + 1, opens_block: false, closes_block: false, signature: false });
start = i + 1;
}
_ => {}
}
i += 1;
}
if start < body_close {
out.push(Stmt { from: start, to: body_close, opens_block: false, closes_block: false, signature: false });
}
out.push(Stmt { from: body_close, to: toks.len(), opens_block: false, closes_block: true, signature: true });
out
}
/// The variables a statement READS and WRITES (workspace-chosen value names).
fn reads_writes(toks: &[LineTok], s: &Stmt) -> (BTreeSet<String>, BTreeSet<String>) {
let (mut reads, mut writes) = (BTreeSet::new(), BTreeSet::new());
let t = &toks[s.from..s.to];
let mut binding_until: Option<usize> = None; // `let`/`for` pattern extent
for (k, lt) in t.iter().enumerate() {
if let FlatTok::Ident(w) = <.tok {
if w == "let" {
// bindings run to the `=` (or the end: `let x;`)
binding_until = Some((k..t.len()).find(|j| assign_eq_at(t, *j)).unwrap_or(t.len()));
continue;
}
if w == "for" {
binding_until = Some((k..t.len()).find(|j| matches!(&t[*j].tok, FlatTok::Ident(x) if x == "in")).unwrap_or(t.len()));
continue;
}
}
let Some(v) = var_at(t, k) else { continue };
if binding_until.is_some_and(|u| k < u) {
writes.insert(v.to_string());
continue;
}
// `v = …` / `v += …`: a write (compound also reads)
let eq_next = assign_eq_at(t, k + 1)
|| (matches!(t.get(k + 1).map(|x| &x.tok), Some(FlatTok::Punct(c, true)) if "+-*/%^&|".contains(*c)) && k + 2 < t.len() && assign_eq_at(t, k + 2));
if eq_next {
writes.insert(v.to_string());
}
reads.insert(v.to_string());
}
(reads, writes)
}
struct Dsu(Vec<usize>);
impl Dsu {
fn new(n: usize) -> Self {
Self((0..n).collect())
}
fn find(&mut self, mut x: usize) -> usize {
while self.0[x] != x {
self.0[x] = self.0[self.0[x]];
x = self.0[x];
}
x
}
fn union(&mut self, a: usize, b: usize) {
let (ra, rb) = (self.find(a), self.find(b));
if ra != rb {
// deterministic: the smaller root wins
let (lo, hi) = if ra < rb { (ra, rb) } else { (rb, ra) };
self.0[hi] = lo;
}
}
}
/// **Segment a function's text into semantic groups** (see the module docs for
/// the rule). `None` when the text does not tokenise. Line numbers in the
/// groups are 1-based within `text`; the caller offsets them to the file.
pub fn segment(text: &str) -> Option<Vec<Group>> {
let toks = flat_tokens_lines(text)?;
let stmts = statements(&toks);
if stmts.is_empty() {
return Some(Vec::new());
}
let n = stmts.len();
let mut dsu = Dsu::new(n);
// def-use pass, in statement order
let mut owner: HashMap<String, usize> = HashMap::new();
let rw: Vec<(BTreeSet<String>, BTreeSet<String>)> = stmts.iter().map(|s| reads_writes(&toks, s)).collect();
for (si, s) in stmts.iter().enumerate() {
if s.signature {
continue; // parameters own nothing: reads of a param link nothing
}
let (reads, writes) = &rw[si];
for v in reads.iter().chain(writes.iter()) {
if let Some(&o) = owner.get(v) {
dsu.union(si, o);
}
}
for v in writes {
owner.insert(v.clone(), si);
}
}
// blocks: a `}` binds to its header; a header with no def-use link of its
// own binds to its block's dominant group
let mut stack: Vec<(usize, Vec<usize>)> = Vec::new(); // (header stmt, inner stmts)
let mut blocks: Vec<(usize, Vec<usize>, usize)> = Vec::new(); // (header, inner, close)
for (si, s) in stmts.iter().enumerate() {
if s.closes_block {
if let Some((h, inner)) = stack.pop() {
blocks.push((h, inner, si));
}
continue;
}
if let Some((_, inner)) = stack.last_mut() {
inner.push(si);
}
if s.opens_block {
stack.push((si, Vec::new()));
}
}
for (h, inner, close) in &blocks {
if stmts[*h].signature {
continue;
}
dsu.union(*close, *h);
// isolated header (`loop {`, a header reading only parameters): bind
// to the dominant group of its block
let header_alone = (0..n).filter(|x| !stmts[*x].closes_block && dsu.find(*x) == dsu.find(*h)).count() <= 1;
if header_alone && !inner.is_empty() {
let mut count: BTreeMap<usize, usize> = BTreeMap::new();
for s in inner.iter().filter(|s| !stmts[**s].signature && !stmts[**s].closes_block) {
*count.entry(dsu.find(*s)).or_default() += 1;
}
if let Some((dom, _)) = count.into_iter().max_by_key(|(root, c)| (*c, usize::MAX - *root)) {
dsu.union(*h, dom);
}
}
}
// assemble: component → statements in order → lines + normalised tokens
let mut comps: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
for si in 0..n {
if stmts[si].signature {
continue;
}
comps.entry(dsu.find(si)).or_default().push(si);
}
let mut out: Vec<Group> = Vec::new();
for (_, members) in comps {
let mut lines: BTreeSet<u32> = BTreeSet::new();
let mut flat: Vec<FlatTok> = Vec::new();
for si in &members {
let s = &stmts[*si];
for lt in &toks[s.from..s.to] {
lines.insert(lt.line);
flat.push(lt.tok.clone());
}
}
if flat.is_empty() {
continue;
}
let norm = norm::normalize_tokens(&flat);
let kgrams = kgrams_of(&norm.classes, KGRAM);
out.push(Group { lines: lines.into_iter().collect(), norm, kgrams });
}
out.sort_by_key(|g| g.start_line());
Some(out)
}
/// FNV-1a hashes of every `k`-gram over the role-class sequence.
pub fn kgrams_of(classes: &[u64], k: usize) -> BTreeSet<u64> {
let mut out = BTreeSet::new();
if k == 0 || classes.len() < k {
return out;
}
for w in classes.windows(k) {
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
for c in w {
for b in c.to_le_bytes() {
h ^= b as u64;
h = h.wrapping_mul(0x0000_0100_0000_01b3);
}
}
out.insert(h);
}
out
}
// ---------------------------------------------------------------------------
// Pairing
// ---------------------------------------------------------------------------
/// One function handed to the pairer, already segmented.
#[derive(Debug, Clone)]
pub struct GroupedItem {
pub file: String,
/// `module::ident` — the room span's qualified name (a READING label).
pub ident: String,
/// The item's own span in the file, 1-based inclusive.
pub item_start: u32,
pub item_end: u32,
pub groups: Vec<Group>,
}
/// One side of a twin.
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
pub struct TwinSite {
pub file: String,
pub ident: String,
pub item_start: u32,
pub item_end: u32,
/// The group's lines in the FILE (offset applied), sorted, possibly
/// non-contiguous.
pub group_lines: Vec<u32>,
pub tokens: usize,
/// The group's exact key (SHA-256 of its normalised text).
pub key: String,
}
/// One group twin: the same semantic group in two different functions.
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
pub struct GroupTwin {
pub a: TwinSite,
pub b: TwinSite,
pub cosine: f32,
/// The two groups are ONE normalised body (cosine 1.0 by construction).
pub exact: bool,
pub shared_kgrams: usize,
pub rare_shared: usize,
/// What found it, one line per signal — the clones-family convention.
pub found_by: Vec<String>,
}
/// The whole answer, including what it could NOT judge.
#[derive(Debug, Clone, Default, serde::Serialize)]
pub struct TwinReport {
pub twins: Vec<GroupTwin>,
pub items: usize,
pub groups: usize,
pub groups_below_weight: usize,
/// Groups at or above the weight floor whose exact key has NO vector:
/// their non-exact pairs were not judged.
pub groups_without_vector: usize,
/// Structure-agreeing candidate pairs skipped for a missing vector.
pub candidates_unjudged: usize,
/// k-gram buckets skipped for exceeding `max_bucket` (boilerplate shapes).
pub buckets_skipped: usize,
pub options: TwinOptions,
}
/// **Pair semantic groups across functions.** `vectors` maps a group's exact
/// key to its embedding (`embeddings_groups`); a pair of EQUAL keys needs no
/// vector (one normalised body). Same-function pairs are never reported.
pub fn pair(items: &[GroupedItem], vectors: &HashMap<String, Vec<f32>>, opts: &TwinOptions) -> TwinReport {
let mut report = TwinReport { items: items.len(), options: *opts, ..Default::default() };
// the eligible groups, flat
struct G<'a> {
item: usize,
group: &'a Group,
}
let mut gs: Vec<G<'_>> = Vec::new();
for (ii, it) in items.iter().enumerate() {
for g in &it.groups {
report.groups += 1;
if g.norm.tokens < opts.min_group_tokens {
report.groups_below_weight += 1;
continue;
}
gs.push(G { item: ii, group: g });
}
}
// document frequency per k-gram, over DISTINCT normalised bodies
let mut df: HashMap<u64, BTreeSet<&str>> = HashMap::new();
for g in &gs {
for k in &g.group.kgrams {
df.entry(*k).or_default().insert(g.group.norm.exact.as_str());
}
}
report.groups_without_vector = gs.iter().filter(|g| !vectors.contains_key(&g.group.norm.exact)).count();
// candidates through the k-gram index
let mut index: HashMap<u64, Vec<usize>> = HashMap::new();
for (gi, g) in gs.iter().enumerate() {
for k in &g.group.kgrams {
index.entry(*k).or_default().push(gi);
}
}
let mut cands: BTreeSet<(usize, usize)> = BTreeSet::new();
for (_, bucket) in index {
if bucket.len() > opts.max_bucket {
report.buckets_skipped += 1;
continue;
}
for x in 0..bucket.len() {
for y in (x + 1)..bucket.len() {
let (a, b) = (bucket[x].min(bucket[y]), bucket[x].max(bucket[y]));
if gs[a].item != gs[b].item {
cands.insert((a, b));
}
}
}
}
let site = |g: &G<'_>| -> TwinSite {
let it = &items[g.item];
TwinSite {
file: it.file.clone(),
ident: it.ident.clone(),
item_start: it.item_start,
item_end: it.item_end,
group_lines: g.group.lines.clone(),
tokens: g.group.norm.tokens,
key: g.group.norm.exact.clone(),
}
};
for (a, b) in cands {
let (ga, gb) = (&gs[a], &gs[b]);
let shared: Vec<u64> = ga.group.kgrams.intersection(&gb.group.kgrams).copied().collect();
let rare = shared.iter().filter(|k| df.get(*k).is_some_and(|d| d.len() <= opts.rare_df)).count();
if shared.len() < opts.min_shared_kgrams || rare == 0 {
continue;
}
let exact = ga.group.norm.exact == gb.group.norm.exact;
let cosine = if exact {
1.0
} else {
match (vectors.get(&ga.group.norm.exact), vectors.get(&gb.group.norm.exact)) {
(Some(va), Some(vb)) if va.len() == vb.len() => crate::vector::pair_index::dot(va, vb),
_ => {
report.candidates_unjudged += 1;
continue;
}
}
};
if cosine < opts.cosine_floor {
continue;
}
let mut found_by = vec![if exact {
"exact: one normalised group body (names → roles, literals → kinds) — cosine 1.0 by construction".to_string()
} else {
format!("normalised model over the group: cosine {cosine:.3} ≥ {:.2}", opts.cosine_floor)
}];
found_by.push(format!(
"structure: {} shared role-class {}-gram(s), {rare} of them rare (≤ {} distinct bodies)",
shared.len(),
opts.kgram,
opts.rare_df
));
report.twins.push(GroupTwin { a: site(ga), b: site(gb), cosine, exact, shared_kgrams: shared.len(), rare_shared: rare, found_by });
}
report.twins.sort_by(|x, y| {
y.cosine
.total_cmp(&x.cosine)
.then(y.shared_kgrams.cmp(&x.shared_kgrams))
.then((x.a.file.as_str(), x.a.group_lines.first()).cmp(&(y.a.file.as_str(), y.a.group_lines.first())))
.then((x.b.file.as_str(), x.b.group_lines.first()).cmp(&(y.b.file.as_str(), y.b.group_lines.first())))
});
report
}
// ---------------------------------------------------------------------------
// The RECORDED group calibration — what the lens reads at runtime (v2)
// ---------------------------------------------------------------------------
/// One MEASURED operating point for the group gates in ONE embedding space —
/// [`crate::mimir_calibrate::RecordedCalibration`]'s idea one level down. The
/// fixture-measured [`COSINE_FLOOR`] stays the fallback for every space nobody
/// has measured; a row here replaces it for the space it names, and the lens
/// says which of the two it ran with (`floor_basis`).
#[derive(Debug, Clone, Copy, PartialEq, serde::Serialize)]
pub struct RecordedGroupCalibration {
/// Registry id of the model (`jina-v2-base-code`, …).
pub model_id: &'static str,
/// **The `model_profile` id stored on every `embeddings_groups` row in
/// this space** — exactly what the lens resolves from the request or the
/// newest green embed run, so matching is equality, not inference.
pub model_profile: &'static str,
pub weights_sha: &'static str,
pub weights_bytes: u64,
pub dim: usize,
pub corpus: &'static str,
/// `YYYY-MM-DD`.
pub measured_at: &'static str,
/// The cosine floor the lens runs with in this space.
pub cosine_floor: f32,
/// How the floor was derived, in one clause.
pub floor_method: &'static str,
pub auc: f32,
/// Fraction of constructed group twins the floor keeps.
pub recall: f32,
/// Fraction of structure-gate candidates (the hard negatives) it admits.
pub hard_negative_survival: f32,
}
impl RecordedGroupCalibration {
/// The gates the lens runs with in this space: the measured floor, every
/// other gate at the shipped (fixture-measured, embedder-independent)
/// value — the k-gram gates are over the normalised role-class alphabet,
/// which no model touches.
pub fn options(&self) -> TwinOptions {
TwinOptions { cosine_floor: self.cosine_floor, ..TwinOptions::default() }
}
/// The provenance sentence the lens note carries instead of "fixtures".
pub fn note(&self) -> String {
format!(
"cosine floor {:.3} is CALIBRATED for this space, not fixture-set: MEASURED {} against \
`{}` ({}-dim, weights sha256 {}…, {} bytes) over {}. Derivation: {}. ROC-AUC {:.3}; \
at this floor {:.1}% of constructed group twins survive and {:.1}% of structure-gate \
candidates that are NOT twins do. Vectors from any other model_profile are a \
different space and this number does not transfer",
self.cosine_floor,
self.measured_at,
self.model_id,
self.dim,
&self.weights_sha[..self.weights_sha.len().min(12)],
self.weights_bytes,
self.corpus,
self.floor_method,
self.auc,
self.recall * 100.0,
self.hard_negative_survival * 100.0,
)
}
}
/// Every operating point ever MEASURED for the group gates, one row per
/// embedding space. Empty is the honest state for a space nobody measured;
/// adding a row is a deliberate act backed by a run of
/// `examples/calibrate_group_space.rs`.
pub const GROUP_RECORDED: &[RecordedGroupCalibration] = &[
// ── jina-embeddings-v2-base-code, fp32 ONNX export ──────────────────────
//
// The space every box with `/opt/nornir/models` (or the build cache) embeds
// `embeddings_groups` in — the same weights file as `mimir_calibrate`'s
// function-level RECORDED row. Produced by
// `calibrate_group_space nornir/src --sample 200 --negatives 500` on
// dwarf (tract-onnx CPU, container), 2026-10-20, 3 100 s for 2 061 bodies.
//
// WHAT THE RUN SAID (full log: the merge commit of group_twins v2):
// corpus 270 files, 6 701 functions, 6 282 groups ≥ 30 norm tokens
// positives n=360 constructed single-statement edits of real groups
// (reword μ=0.9948 σ=0.0090 · drop μ=0.9809 σ=0.0369 ·
// swap μ=0.9946 σ=0.0082; pooled μ=0.9900 σ=0.0234, UNIMODAL;
// 113 further edits normalised away — those pairs are EXACT
// twins, never the vector path's to judge)
// hard-neg n=500 REAL structure-gate candidates (≥2 shared role-class
// 8-grams, ≥1 rare, different body): μ=0.6951 σ=0.1383
// p95=0.9052 max=0.9895 — sampled from 1 655 301 candidates
// random-neg n=500 cross-file pairs: μ=0.5095 p95=0.7641
// ROC-AUC 0.997, Cohen's d 3.24
//
// ⚠ THE GROUP SPACE ACCEPTS μ_pos − k·σ_pos — the opposite of the
// function-level measurement, and worth saying why: normalising per GROUP
// (names → roles numbered per group, literals → kinds) collapses the
// same-shape noise that put function-level hard negatives at p95 0.9693.
// Here they stop at 0.9052, under the k=1 floor:
// k=0.0 → 0.990 recall 78.6% hard-neg 0.0%
// k=1.0 → 0.967 recall 93.6% hard-neg 0.6% ← shipped (the verdict's own)
// k=2.0 → 0.943 recall 96.9% hard-neg 1.8%
// FP≤5% quantile → 0.905 recall 98.3% (the fixture 0.90 ≈ this point)
// Survivors on the corpus (the real pairer): floor 0.90 → 657 twins
// (286 by vector), 0.95 → 417 (46), 0.967 → 383 (12); 371 exact-key twins
// are floor-independent. The vector path exists for REWORDED statements,
// and those sit at μ≈0.995 — 0.967 keeps them and refuses all but 0.6% of
// the structure-gate noise. The weakest positive type is drop-1stmt
// (p05 0.9124): a copy missing a statement below the floor is the accepted
// cost of a report a person can act on.
//
// THE K-GRAM GATES STAY at the fixture values (KGRAM=8, MIN_SHARED_KGRAMS=2,
// RARE_DF=2, MAX_BUCKET=100), now with corpus numbers: 311/360 constructed
// twins pass the gate (shared 8-grams p05=16 med=91), hard negatives sit at
// med=13 — the gate is load-bearing (it kills ~92% of all cross-function
// pairs) but 1.66M candidates still pass on one repo, so PRECISION is the
// cosine floor's job, which is why the floor is read at k=1 and not at a
// looser FP quantile. The gates are over the normalised role-class
// alphabet: no model touches them, so they are embedder-independent.
RecordedGroupCalibration {
model_id: "jina-v2-base-code",
model_profile: "c51625221e8ce764e080673844f9858c4a05ed5f5dc4e2d268ce331d80b60534",
weights_sha: "63363fc178428b74620c6f3780cbc7191883fa5c7f84c0945c45eb5c4256733b",
weights_bytes: 641_517_466,
dim: 768,
corpus: "nornir/src — 6 701 functions, 6 282 semantic groups ≥ 30 normalised tokens, embedded per NORMALISED GROUP BODY",
measured_at: "2026-10-20",
cosine_floor: 0.967,
floor_method: "mu_pos - 1*sigma_pos over 360 constructed single-statement edits of real groups, \
verdict CALIBRATED — the group space accepts the formula (hard negatives, i.e. real \
structure-gate candidates, reach only p95 0.905)",
auc: 0.997,
recall: 0.936,
hard_negative_survival: 0.006,
},
];
/// The recorded operating point of `model_profile` (the id stored in
/// `embeddings_groups`), when one was measured.
pub fn recorded_for_profile(model_profile: &str) -> Option<&'static RecordedGroupCalibration> {
GROUP_RECORDED.iter().find(|r| r.model_profile == model_profile)
}
// ---------------------------------------------------------------------------
// Tests — the oracle first (LAW: shown red before the detector existed)
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
/// The deterministic test embedder: a 64-bucket normalised bag of the
/// text's identifier-like tokens — the same scheme as the room fixtures'
/// `Bag`, local so these tests need no fixture features. Identical text →
/// identical vector; cosine behaves (shared tokens raise it).
fn bag(text: &str) -> Vec<f32> {
let mut v = vec![0f32; 64];
for tok in text.split(|c: char| !(c.is_alphanumeric() || c == '_')).filter(|s| !s.is_empty()) {
let h = tok.bytes().fold(0xcbf2_9ce4_8422_2325u64, |h, b| (h ^ b as u64).wrapping_mul(0x0100_0000_01b3));
v[(h % 64) as usize] += 1.0;
}
let n = v.iter().map(|x| x * x).sum::<f32>().sqrt().max(f32::MIN_POSITIVE);
v.iter().map(|x| x / n).collect()
}
fn vectors_of(items: &[GroupedItem]) -> HashMap<String, Vec<f32>> {
let mut out = HashMap::new();
for it in items {
for g in &it.groups {
out.entry(g.norm.exact.clone()).or_insert_with(|| bag(&g.norm.text));
}
}
out
}
fn item(file: &str, ident: &str, text: &str) -> GroupedItem {
let groups = segment(text).unwrap_or_else(|| panic!("{ident} does not tokenise"));
GroupedItem { file: file.into(), ident: ident.into(), item_start: 1, item_end: text.lines().count() as u32, groups }
}
/// The planted semantic group: 8 statements, one def-use chain
/// (`acc` ← `carry` ← `w` ← `raw`).
const PLANTED: &str = "\
let mut acc: Vec<(String, u64)> = Vec::new();
let mut carry: u64 = 7;
for raw in input {
let w = raw.len() as u64 + carry;
carry = carry.wrapping_mul(31) ^ w;
acc.push((raw.clone(), w));
}
acc.sort_by(|x, y| y.1.cmp(&x.1));";
/// PLANTED consistently renamed, literals retuned — the Type-2 copy.
fn planted_renamed() -> String {
PLANTED
.replace("acc", "bucket")
.replace("carry", "roll")
.replace(" w ", " weight ")
.replace("(w)", "(weight)")
.replace(", w)", ", weight)")
.replace("^ w;", "^ weight;")
.replace("let w =", "let weight =")
.replace("= 7;", "= 9;")
.replace("(31)", "(37)")
}
/// PLANTED with ONE statement rewritten (push → insert at the front, ^ → +):
/// another normalised body — the cosine path, not the exact one.
fn planted_reworded() -> String {
PLANTED.replace("acc.push((raw.clone(), w));", "acc.insert(0, (raw.clone(), w));").replace("^ w;", "+ w;")
}
/// An unrelated group of comparable size — the control that must NOT pair.
const CONTROL: &str = "\
let mut seen = std::collections::BTreeMap::new();
for name in names {
let trimmed = name.trim().to_string();
if trimmed.is_empty() {
continue;
}
*seen.entry(trimmed).or_insert(0usize) += 1;
}
let busiest = seen.iter().max_by_key(|(_, n)| **n).map(|(k, _)| k.clone());";
fn fn_a() -> String {
format!("fn summarise(input: &[String], route: &str) -> u64 {{\n{PLANTED}\n acc.len() as u64\n}}\n")
}
/// Case (a): PLANTED verbatim, contiguous, in a different function. The
/// tail reads `acc`, so it RIDES INTO the group — correct def-use
/// behaviour, so the fixture keeps the tail in the same ROLE SHAPE as
/// `fn_a`'s (a `+ window` here made the two groups 102 vs 104 tokens and
/// broke exactness: measured, not guessed).
fn fn_b_verbatim() -> String {
format!("fn rollup(input: &[String], window: u64) -> u64 {{\n{PLANTED}\n acc.len() as u64\n}}\n")
}
/// Case (b): renamed + retuned, contiguous.
fn fn_b_renamed() -> String {
format!("fn rollup(input: &[String], window: u64) -> u64 {{\n{}\n bucket.len() as u64\n}}\n", planted_renamed())
}
/// Case (c): renamed + retuned AND interleaved with three noise lines —
/// a log line and an unrelated local chain on ANOTHER parameter.
fn fn_b_interleaved() -> String {
let p = planted_renamed();
let lines: Vec<&str> = p.lines().collect();
// noise after statement 1, after statement 2, and after the loop
let (head, s2, rest, tail) = (lines[0], lines[1], lines[2..7].join("\n"), lines[7]);
format!(
"fn rollup(input: &[String], route: &str) -> u64 {{\n\
{head}\n\
\x20 eprintln!(\"rollup {{}}\", route);\n\
{s2}\n\
\x20 let banner = route.to_ascii_uppercase();\n\
{rest}\n\
\x20 let _ = banner.len();\n\
{tail}\n\
\x20 bucket.len() as u64\n\
}}\n"
)
}
/// Case (d): one statement rewritten — pairs through the VECTOR, not the key.
fn fn_b_reworded() -> String {
format!("fn rollup(input: &[String], window: u64) -> u64 {{\n{}\n acc.len() as u64\n}}\n", planted_reworded())
}
fn fn_control() -> String {
format!("fn roster(names: &[String]) -> Option<String> {{\n{CONTROL}\n busiest\n}}\n")
}
fn planted_group(it: &GroupedItem) -> &Group {
it.groups.iter().max_by_key(|g| g.norm.tokens).unwrap()
}
// ── segmentation ────────────────────────────────────────────────────────
#[test]
fn segmentation_is_deterministic_and_isolates_noise() {
let text = fn_b_interleaved();
let g1 = segment(&text).unwrap();
let g2 = segment(&text).unwrap();
assert_eq!(g1, g2, "deterministic");
// the planted chain is ONE group; the noise lines are NOT in it
let planted = g1.iter().max_by_key(|g| g.norm.tokens).unwrap();
let text_lines: Vec<&str> = text.lines().collect();
for l in &planted.lines {
let line = text_lines[(*l - 1) as usize];
assert!(!line.contains("eprintln") && !line.contains("banner"), "noise in the planted group: {line}");
}
// every noise line landed in SOME group — its own, not the planted one
let noise_lines: Vec<u32> = text_lines
.iter()
.enumerate()
.filter(|(_, l)| l.contains("eprintln") || l.contains("banner"))
.map(|(i, _)| i as u32 + 1)
.collect();
assert_eq!(noise_lines.len(), 3, "{text}");
for nl in &noise_lines {
assert!(g1.iter().any(|g| g.lines.contains(nl)), "noise line {nl} fell out of every group");
}
// the two banner lines chain together (write → read), apart from eprintln
let banner_group = g1.iter().find(|g| g.lines.contains(&noise_lines[1])).unwrap();
assert!(banner_group.lines.contains(&noise_lines[2]), "banner's writer and reader are one group");
assert!(!banner_group.lines.contains(&noise_lines[0]), "eprintln shares no name with banner");
}
#[test]
fn a_control_flow_header_binds_to_its_blocks_group() {
// `for raw in input {` writes `raw`, which the block reads: one group,
// and the closing brace rides with the header.
let g = segment(&fn_a()).unwrap();
let planted = g.iter().max_by_key(|g| g.norm.tokens).unwrap();
let text = fn_a();
let for_line = text.lines().position(|l| l.contains("for raw in")).unwrap() as u32 + 1;
let close_line = text.lines().position(|l| l.trim() == "}").unwrap() as u32 + 1;
assert!(planted.lines.contains(&for_line), "{planted:?}");
assert!(planted.lines.contains(&close_line), "{planted:?}");
}
#[test]
fn the_signature_owns_nothing_and_is_in_no_group() {
// two statements reading only the parameter stay SEPARATE groups: a
// parameter read links nothing (else every body would be one group).
// No tail reads BOTH (`a + b` would — correctly — union them).
let text = "fn f(input: &[u8], out: &mut Vec<usize>) {\n let a = input.len() * 3;\n let b = input.first().copied().unwrap_or(0) as usize;\n out.push(a);\n out.push(b);\n}\n";
let g = segment(text).unwrap();
assert!(g.iter().all(|gr| !gr.lines.contains(&1)), "the signature line is grouped: {g:?}");
let a_line = 2u32;
let b_line = 3u32;
let ga = g.iter().find(|gr| gr.lines.contains(&a_line)).unwrap();
assert!(!ga.lines.contains(&b_line), "two independent param readers merged: {g:?}");
}
#[test]
fn text_that_does_not_tokenise_is_none() {
assert!(segment("fn broken( {").is_none());
}
// ── the oracle ──────────────────────────────────────────────────────────
#[test]
fn oracle_a_the_verbatim_group_pairs_across_two_functions() {
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_verbatim()), item("c.rs", "roster", &fn_control())];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
let hit: Vec<&GroupTwin> = r.twins.iter().filter(|t| t.a.ident == "summarise" && t.b.ident == "rollup").collect();
assert_eq!(hit.len(), 1, "exactly one twin between the two functions: {r:#?}");
assert!(hit[0].exact, "{:#?}", hit[0]);
assert!((hit[0].cosine - 1.0).abs() < f32::EPSILON);
assert!(hit[0].tokens_both_sides() >= MIN_GROUP_TOKENS);
}
#[test]
fn oracle_b_renamed_vars_and_retuned_literals_still_pair() {
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_renamed())];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
let hit: Vec<&GroupTwin> = r.twins.iter().filter(|t| t.a.ident != t.b.ident).collect();
assert_eq!(hit.len(), 1, "{r:#?}");
assert!(hit[0].exact, "a consistent rename + retuned literals is ONE normalised body: {:#?}", hit[0]);
}
#[test]
fn oracle_c_the_interleaved_copy_pairs_and_names_only_its_own_lines() {
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_interleaved()), item("c.rs", "roster", &fn_control())];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
let hit: Vec<&GroupTwin> = r.twins.iter().filter(|t| t.a.ident == "summarise" && t.b.ident == "rollup").collect();
assert_eq!(hit.len(), 1, "the interleaved copy is THE finding this module exists for: {r:#?}");
let t = hit[0];
assert!(t.exact, "the noise stripped away, the group is the same normalised body: {t:#?}");
// the finding's B side skips the noise lines
let text = fn_b_interleaved();
let lines: Vec<&str> = text.lines().collect();
for l in &t.b.group_lines {
let line = lines[(*l - 1) as usize];
assert!(!line.contains("eprintln") && !line.contains("banner"), "a noise line in the finding: {line}");
}
// and it is NON-CONTIGUOUS — the thing a contiguous-run detector cannot state
let contiguous = t.b.group_lines.windows(2).all(|w| w[1] == w[0] + 1);
assert!(!contiguous, "the interleave must show in the finding: {:?}", t.b.group_lines);
}
#[test]
fn oracle_d_a_reworded_statement_pairs_through_the_vector() {
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_reworded())];
let vectors = vectors_of(&items);
let r = pair(&items, &vectors, &TwinOptions::default());
let hit: Vec<&GroupTwin> = r.twins.iter().filter(|t| t.a.ident != t.b.ident).collect();
assert_eq!(hit.len(), 1, "{r:#?}");
let t = hit[0];
assert!(!t.exact, "one statement rewritten is another normalised body: {t:#?}");
assert!(t.cosine >= COSINE_FLOOR && t.cosine < 1.0, "found by the model, not the key: {}", t.cosine);
assert!(t.rare_shared >= 1, "and held by shared rare structure: {t:#?}");
}
#[test]
fn oracle_control_unrelated_groups_do_not_pair() {
let items = [item("a.rs", "summarise", &fn_a()), item("c.rs", "roster", &fn_control())];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
assert!(r.twins.is_empty(), "unrelated groups are not twins: {r:#?}");
}
#[test]
fn same_function_pairs_are_never_reported() {
// the SAME group twice inside one function: two groups, zero twins
let text = format!("fn twice(input: &[String]) -> u64 {{\n{PLANTED}\n{}\n 0\n}}\n", planted_renamed());
let items = [item("a.rs", "twice", &text)];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
assert!(r.twins.is_empty(), "{r:#?}");
}
#[test]
fn a_group_below_the_weight_floor_is_counted_not_paired() {
let a = "fn one(input: &[u8]) -> usize {\n let n = input.len();\n n + 1\n}\n";
let b = "fn two(input: &[u8]) -> usize {\n let n = input.len();\n n + 1\n}\n";
let items = [item("a.rs", "one", a), item("b.rs", "two", b)];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
assert!(r.twins.is_empty(), "a two-statement group is Rust, not a twin: {r:#?}");
assert!(r.groups_below_weight >= 2, "and it was COUNTED: {r:?}");
}
#[test]
fn a_missing_vector_is_counted_never_silently_dropped() {
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_reworded())];
let r = pair(&items, &HashMap::new(), &TwinOptions::default());
assert!(r.twins.is_empty(), "{r:#?}");
assert!(r.candidates_unjudged >= 1, "the unjudged pair is a COUNT: {r:?}");
assert!(r.groups_without_vector >= 2, "{r:?}");
}
// ── the measured constants ──────────────────────────────────────────────
#[test]
fn the_constants_are_the_measured_ones() {
// the numbers the module docs claim, re-measured on every run
let a = item("a.rs", "summarise", &fn_a());
let planted = planted_group(&a);
assert!((90..=110).contains(&planted.norm.tokens), "the planted group is ~97 normalised tokens: {}", planted.norm.tokens);
let inter = item("b.rs", "rollup", &fn_b_interleaved());
for g in inter.groups.iter().filter(|g| g.norm.tokens < MIN_GROUP_TOKENS) {
assert!(g.norm.tokens <= 19, "a noise group is 7–19 tokens: {}", g.norm.tokens);
}
// the cosine margin: reworded ≥ floor, control far below it
let d = item("b.rs", "rollup", &fn_b_reworded());
let c = item("c.rs", "roster", &fn_control());
let cos = |x: &Group, y: &Group| crate::vector::pair_index::dot(&bag(&x.norm.text), &bag(&y.norm.text));
let reworded = cos(planted, planted_group(&d));
let control = cos(planted, planted_group(&c));
assert!(reworded >= COSINE_FLOOR, "the reworded copy stays above the floor: {reworded}");
// MEASURED 0.796: role normalisation makes even unrelated groups read
// somewhat alike to a bag model (let / for / V1 / u64 everywhere), so
// the cosine floor alone would be a thin wall — the RARE-k-gram gate
// below is load-bearing, not decoration.
assert!(control < COSINE_FLOOR, "the control stays under the floor: {control}");
// the structure margin
let shared_reworded = planted.kgrams.intersection(&planted_group(&d).kgrams).count();
let shared_control = planted.kgrams.intersection(&planted_group(&c).kgrams).count();
assert!(shared_reworded >= MIN_SHARED_KGRAMS, "{shared_reworded}");
// MEASURED 4 of ~95: a handful of GENERIC role runs (`; } } let V =`)
// overlap even between unrelated groups — and in a two-item scan they
// even count as rare, so structure alone would pass this pair. It is
// the COSINE floor that blocks the control (0.796 < 0.90), and the
// rare k-grams that block model-only lookalikes: two gates, each one
// load-bearing for the half the other cannot hold.
assert!(shared_control * 10 <= shared_reworded, "the control's overlap is generic noise, an order under the copy's: {shared_control} vs {shared_reworded}");
println!(
"measured: planted {} tokens; cosine reworded {reworded:.3}, control {control:.3}; shared kgrams reworded {shared_reworded}, control {shared_control}",
planted.norm.tokens
);
}
// ── the recorded (real-embedder) operating point ─────────────────────────
/// RED FIRST (2026-10-20): ran against an EMPTY `GROUP_RECORDED` — failed
/// at "a measured operating point is RECORDED" — and went green when the
/// `calibrate_group_space` run's row landed. Pins, parametrically in the
/// recorded floor: the calibrated gates ACCEPT a known-twin pair sitting
/// just above the measured floor (with the rare shared structure the gate
/// also demands) and REFUSE a known-noise pair just below it.
#[test]
fn the_recorded_space_drives_the_gates_accepting_twins_and_refusing_noise() {
let r = GROUP_RECORDED.first().expect("a measured operating point is RECORDED (the v2 real-embedder calibration run)");
assert_eq!(recorded_for_profile(r.model_profile).map(|x| x.cosine_floor), Some(r.cosine_floor));
let opts = r.options();
assert!((opts.cosine_floor - r.cosine_floor).abs() < f32::EPSILON);
assert_eq!(opts.min_group_tokens, MIN_GROUP_TOKENS, "the weight floor is embedder-independent and stays");
assert!(r.note().contains("CALIBRATED"), "{}", r.note());
// synthetic unit vectors pin the gate numerically at the measured floor
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_reworded())];
let key_a = planted_group(&items[0]).norm.exact.clone();
let key_b = planted_group(&items[1]).norm.exact.clone();
let v = |c: f32| vec![c, (1.0f32 - c * c).max(0.0).sqrt()];
let mut vectors = HashMap::new();
vectors.insert(key_a.clone(), vec![1.0f32, 0.0]);
vectors.insert(key_b.clone(), v(r.cosine_floor + 0.005));
let accepted = pair(&items, &vectors, &opts);
assert_eq!(accepted.twins.len(), 1, "a known twin just above the calibrated floor is ACCEPTED: {accepted:#?}");
assert!(!accepted.twins[0].exact && accepted.twins[0].rare_shared >= 1, "{:#?}", accepted.twins[0]);
vectors.insert(key_b, v(r.cosine_floor - 0.005));
let refused = pair(&items, &vectors, &opts);
assert!(refused.twins.is_empty(), "known noise just below the calibrated floor is REFUSED: {refused:#?}");
}
// ── the measured blindness this module closes ───────────────────────────
/// Winnowing finds the CONTIGUOUS copy whole and the INTERLEAVED one not at
/// all — same options, same planted group. The noise lines break the shared
/// text into fragments shorter than `k = 25` shape tokens, which winnowing's
/// own first guarantee makes unreportable at ANY threshold; and no
/// contiguous-run detector can name the group as ONE finding either way.
#[test]
fn winnowing_is_blind_to_the_interleaved_copy_and_this_module_is_not() {
use crate::mimir_subclone::{SubcloneOptions, detect};
// gates low enough that the ~97-token, 8-line planted group is in reach
// of the CONTIGUOUS detector — everything else at the shipped defaults
let opts = SubcloneOptions { min_run_tokens: 60, min_run_lines: 5, require_callable: false, ..Default::default() };
let contiguous = detect(vec![("a.rs".into(), fn_a(), None), ("b.rs".into(), fn_b_verbatim(), None)], &opts);
assert!(!contiguous.groups.is_empty(), "RED HALF: the contiguous copy IS in winnowing's reach at these gates: {contiguous:#?}");
let interleaved = detect(vec![("a.rs".into(), fn_a(), None), ("b.rs".into(), fn_b_interleaved(), None)], &opts);
assert!(
interleaved.groups.is_empty(),
"the measured gap: three noise lines put the same copy out of winnowing's reach entirely: {:#?}",
interleaved.groups
);
// …and this module finds exactly that case (oracle_c asserts the details)
let items = [item("a.rs", "summarise", &fn_a()), item("b.rs", "rollup", &fn_b_interleaved())];
let r = pair(&items, &vectors_of(&items), &TwinOptions::default());
assert_eq!(r.twins.len(), 1, "{r:#?}");
}
impl GroupTwin {
fn tokens_both_sides(&self) -> usize {
self.a.tokens.min(self.b.tokens)
}
}
}