src/mimir_calibrate.rs
//! **The calibration oracle** — what the duplicate/clone lenses' cosine
//! thresholds are actually WORTH in the embedding space that is loaded.
//!
//! ## The problem this exists to answer
//!
//! [`crate::mimir_analyze::DupThresholds::default`] says `cosine: 0.92` and
//! [`crate::mimir_clones::CloneThresholds::default`] says `cosine_floor: 0.80`.
//! Both numbers were *chosen*, never *measured*: no run has ever asked what a
//! cosine of 0.80 means for the embedding model that ships. The consequence is
//! recorded in the lens itself — the 2026-08-19 full-corpus run produced
//! **10 795 clusters at cosine 0.93** and **10 152 at 0.80**, and
//! [`crate::mimir_clones::MAX_RANKED_CLUSTER_SIZE`] names a 600–1 800-member
//! mega-blob at cosine ~0.83 as *"evidence the cosine floor is too loose for the
//! model profile"*. That is a claim about the THRESHOLD, and a threshold claim
//! can only be settled by measuring the space.
//!
//! ## What a calibration IS here
//!
//! Two populations of pairwise cosines, measured through the SAME embedder the
//! lens will use:
//!
//! * **positives** — pairs that are clones *by construction*: a byte-identical
//! copy (Type-1), the same function with renamed identifiers and literals
//! (Type-2), the same function with statements inserted and comments removed
//! (Type-3). No warehouse is needed to know the truth label, because the
//! label is how the pair was built.
//! * **negatives** — pairs that are NOT clones: random cross-file pairs, and —
//! the ones that actually decide the floor — **hard negatives**, two real
//! functions with the same signature and a different body. Random pairs are
//! too easy; calibrating on them alone yields a flatteringly small σ and a
//! floor that lets every same-shaped function through.
//!
//! From those, [`Calibration::compute`] derives `cosine_floor = μ_pos − k·σ_pos`
//! and then — this is the part that makes it a measurement rather than a formula
//! — states what that floor DOES: the fraction of true clones it keeps
//! ([`Calibration::recall`]) and the fraction of hard negatives it lets through
//! ([`Calibration::hard_negative_survival`]).
//!
//! ## Three traps this module is built to fall into loudly
//!
//! 1. **The identity value.** Cosine between a vector and itself is 1.0 whether
//! the embedder works or is a `vec![0.1; 768]` constant. An oracle that
//! measures only Type-1 pairs sits on 1.0 and cannot tell those apart. So the
//! positive population is never Type-1 alone, and [`Verdict::NoSeparation`]
//! fires on a degenerate σ *and* on a ROC-AUC that a coin could reach. The
//! tests prove both go RED against a constant-vector embedder and against a
//! hash embedder that returns 1.0 for Type-1 and noise for everything else.
//! 2. **The chunk artefact.** Two symbols inside one embedding chunk carry the
//! SAME vector, so their cosine is exactly 1.0 *by construction*
//! ([`crate::mimir_analyze::shares_one_chunk_vector`]) — and 58.6 % of
//! functions in this constellation share a chunk. Calibrating on those would
//! put μ_pos at ~1.0 for reasons that have nothing to do with similarity.
//! [`filter_same_chunk`] routes through the SAME predicate the gates use
//! (LAW 5 — one writer, not two agreeing copies).
//! 3. **The shape.** `μ − kσ` is only meaningful on a roughly unimodal
//! distribution. Type-1 pairs pile up at 1.0 and Type-3 pairs sit far below,
//! which is textbook bimodal — a single μ is then the wrong answer and this
//! module says so ([`Verdict::PerTypeRequired`]) instead of reporting a tidy
//! number. [`CosineDistribution::modes`] counts the peaks; the histogram is
//! carried on the result so the shape is auditable, not asserted.
//!
//! ## Provenance
//!
//! A calibrated threshold with no provenance is the same kind of unfounded claim
//! as the heuristic it replaces. [`Provenance`] records the model id, the
//! weights SHA, the dimensionality, the dtype, the corpus and the date, and
//! [`Calibration::note`] renders all of it — so the evidence note can stop
//! saying "UNCALIBRATED" and start saying what it was calibrated *against*.
//!
//! ## What it MEASURED — oden, 2026-08-25
//!
//! Full numbers and their derivation live on the [`RECORDED`] row. The three
//! results worth reading first:
//!
//! **1. The shipped space is fp32, not q8.** `/opt/nornir/models/model.onnx` is
//! 641 517 466 bytes (611.8 MiB) — the fp32 export — and no `NORNIR_EMBED_MODEL`
//! is set anywhere on the box, so `selected()` resolves to the registry default.
//! The `jina-v2-base-code-q8` entry is registry data whose cache subdir does not
//! exist. So "calibrate the q8 space" had a prior question in front of it, and
//! the answer was that q8 is not the space anything embeds in.
//!
//! **2. `μ_pos − k·σ_pos` is the wrong instrument, and the measurement says so.**
//! In the fp32 space the positives are unimodal (μ=0.9058 σ=0.0936, ROC-AUC
//! 0.920 against the negatives) — so the formula is well-defined, and its answer
//! at k=1 is 0.812, which is essentially the 0.80 that already shipped. The
//! problem is not the positives at all: the HARD negatives (same signature,
//! different body) run to p95 = 0.9693, and the formula cannot see them because
//! it only reads the positive moments. Every k from 0 to 3 lands the floor
//! inside them. [`Calibration::floor_for_hard_negative_rate`] is the instrument
//! that can: read the floor off the negative quantile, at a stated
//! false-positive rate.
//!
//! **3. The survivor number.** Through the real
//! [`crate::mimir_clones::analyze_semantic_clones`] on 3 222 per-function
//! vectors of `nornir/src`: **138 clusters at the shipped 0.80 → 11 at the
//! calibrated 0.882** (largest cluster 7 → 2). Not from the shipped k=1 floor —
//! 0.812 gives 107, which is no help — but from the hard-negative 80th
//! percentile. At FP ≤ 10 % (0.952) the count is 0, so 0.882 is where the lens
//! is both quiet and alive.
//!
//! ⚠ That 138 is **not** the 10 152 of the 2026-08-19 constellation run and the
//! ratio does not transfer: this corpus is one repo, and its vectors are
//! per-function, so it does not carry the `impl`-block chunk sharing that made
//! the mega-blob. The number measured here is the number measured here.
//!
//! Two things fell out of the run that are not about thresholds at all, and are
//! recorded because they were measured:
//!
//! * **The int8 export is not batch-invariant.** `vector::embed` documents its
//! batched forward as "bit-identical to the one-at-a-time path". For fp32 it
//! is (Type-1 μ=1.0000 σ=0.0000). For the int8 export it is NOT: identical
//! bytes embed to μ=0.9965, worst pair 0.9632, depending on what shares the
//! batch. And on oden's ort-CUDA path the int8 graph ran the same oracle in
//! 845 s against fp32's 7.4 s — 114× SLOWER, not faster.
//! * **`$NORNIR_EMBED_MODEL` alone selects the wrong weights.** `model_dir()`
//! returns `/opt/nornir/models` whenever that directory holds a tokenizer,
//! without consulting which model was selected — so
//! `NORNIR_EMBED_MODEL=jina-v2-base-code-q8` loads the **fp32** weights while
//! every label downstream says q8. Measured both ways: with the env var alone
//! the run reports 641 517 466 bytes; with `$NORNIR_MODEL_DIR` also set it
//! reports 161 895 621. A per-model subdir under `/opt` would close it.
use std::collections::{BTreeMap, BTreeSet};
use serde::Serialize;
// ---------------------------------------------------------------------------
// Distribution shape
// ---------------------------------------------------------------------------
/// Histogram bin width over the cosine range `[-1, 1]` → 40 bins.
pub const HIST_BIN_WIDTH: f32 = 0.05;
/// A peak in the histogram counts as a MODE only when it reaches this fraction
/// of the tallest bin — otherwise single-sample noise in a tail would read as a
/// second mode and every distribution would look multi-modal.
const MODE_PEAK_FRACTION: f32 = 0.15;
/// Two peaks are separate modes only when the lowest bin BETWEEN them falls to
/// this fraction of the shorter peak — a shoulder on one hill is not two hills.
const MODE_VALLEY_FRACTION: f32 = 0.60;
/// σ below this counts as "no spread at all" — the signature of an embedder
/// that returns the same vector for every input.
pub const DEGENERATE_SD: f32 = 1e-4;
/// One measured population of pairwise cosines, with everything needed to
/// audit it: the sorted samples, the moments, the quantiles and the histogram.
#[derive(Debug, Clone, Serialize)]
pub struct CosineDistribution {
/// What this population IS (`"type-1"`, `"hard-negative"`, …).
pub label: String,
pub n: usize,
pub mean: f32,
/// Population standard deviation (`n`, not `n-1` — this is the whole
/// measured population, not a sample of a larger one).
pub sd: f32,
pub min: f32,
pub max: f32,
pub p05: f32,
pub median: f32,
pub p95: f32,
/// `(bin_lower_bound, count)` over `[-1, 1]` in [`HIST_BIN_WIDTH`] steps.
/// Empty bins are kept so the shape reads correctly.
pub histogram: Vec<(f32, usize)>,
/// Number of separate peaks in [`Self::histogram`]. `1` is unimodal; `>= 2`
/// means a single μ describes nothing.
pub modes: usize,
/// Ascending samples — kept so a floor's survival rate is a MEASUREMENT
/// against the real values rather than a normal-distribution assumption.
pub samples: Vec<f32>,
}
impl CosineDistribution {
/// Measure a population. An empty input yields an all-zero distribution with
/// `n == 0`, which every verdict rule treats as unusable rather than as
/// "perfectly tight".
pub fn measure(label: impl Into<String>, xs: &[f32]) -> Self {
let label = label.into();
let mut samples: Vec<f32> = xs.iter().copied().filter(|x| x.is_finite()).collect();
samples.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
let n = samples.len();
if n == 0 {
return CosineDistribution {
label,
n: 0,
mean: 0.0,
sd: 0.0,
min: 0.0,
max: 0.0,
p05: 0.0,
median: 0.0,
p95: 0.0,
histogram: Vec::new(),
modes: 0,
samples,
};
}
let mean = samples.iter().copied().sum::<f32>() / n as f32;
let var = samples.iter().map(|x| (x - mean) * (x - mean)).sum::<f32>() / n as f32;
let sd = var.max(0.0).sqrt();
let histogram = histogram(&samples);
let modes = count_modes(&histogram);
CosineDistribution {
label,
n,
mean,
sd,
min: samples[0],
max: samples[n - 1],
p05: quantile(&samples, 0.05),
median: quantile(&samples, 0.50),
p95: quantile(&samples, 0.95),
histogram,
modes,
samples,
}
}
/// Every sample is (near enough) the same value — the shape of an embedder
/// that has stopped depending on its input.
pub fn is_degenerate(&self) -> bool {
self.n == 0 || self.sd < DEGENERATE_SD
}
/// One peak, so `μ ± kσ` describes the population.
pub fn is_unimodal(&self) -> bool {
self.modes <= 1
}
/// Fraction of this population at or above `floor` — the APPLIED meaning of
/// a threshold (recall for a positive population, false-positive rate for a
/// negative one), read off the real samples.
pub fn survival(&self, floor: f32) -> f32 {
if self.samples.is_empty() {
return 0.0;
}
let kept = self.samples.iter().filter(|&&x| x >= floor).count();
kept as f32 / self.samples.len() as f32
}
/// `μ − k·σ` for THIS population, clamped into the cosine range.
pub fn floor_at(&self, k: f32) -> f32 {
(self.mean - k * self.sd).clamp(-1.0, 1.0)
}
/// One compact auditable line: moments, quantiles and the peak count.
pub fn line(&self) -> String {
format!(
"{:<16} n={:<6} μ={:.4} σ={:.4} min={:.4} p05={:.4} med={:.4} p95={:.4} max={:.4} modes={}",
self.label, self.n, self.mean, self.sd, self.min, self.p05, self.median, self.p95,
self.max, self.modes,
)
}
/// ASCII histogram — the shape evidence. `μ − kσ` is a claim about a shape,
/// so the shape travels with the claim.
pub fn sparkline(&self) -> String {
let peak = self.histogram.iter().map(|(_, c)| *c).max().unwrap_or(0);
if peak == 0 {
return String::new();
}
let mut out = String::new();
for (lo, c) in &self.histogram {
if *c == 0 && *lo < 0.0 {
continue; // the negative half is almost always empty for code
}
let bars = (*c as f32 / peak as f32 * 40.0).round() as usize;
out.push_str(&format!(
" [{lo:>5.2}) {:>6} {}\n",
c,
"#".repeat(bars)
));
}
out
}
}
/// Linear-interpolation quantile over an ASCENDING slice.
fn quantile(sorted: &[f32], q: f32) -> f32 {
if sorted.is_empty() {
return 0.0;
}
if sorted.len() == 1 {
return sorted[0];
}
let pos = q.clamp(0.0, 1.0) * (sorted.len() - 1) as f32;
let lo = pos.floor() as usize;
let hi = pos.ceil() as usize;
let frac = pos - lo as f32;
sorted[lo] * (1.0 - frac) + sorted[hi] * frac
}
/// Fixed-width histogram over `[-1, 1]`; bins are `[lo, lo + HIST_BIN_WIDTH)`
/// with the final bin closed so an exact `1.0` (a Type-1 pair, or a constant
/// embedder) lands somewhere rather than being dropped.
fn histogram(sorted: &[f32]) -> Vec<(f32, usize)> {
let bins = (2.0 / HIST_BIN_WIDTH).round() as usize;
let mut counts = vec![0usize; bins];
for &x in sorted {
let idx = (((x + 1.0) / HIST_BIN_WIDTH).floor() as isize).clamp(0, bins as isize - 1);
counts[idx as usize] += 1;
}
counts
.into_iter()
.enumerate()
.map(|(i, c)| (-1.0 + i as f32 * HIST_BIN_WIDTH, c))
.collect()
}
/// Count separate peaks: a bin is a peak candidate when it is a local maximum
/// reaching [`MODE_PEAK_FRACTION`] of the tallest bin, and two consecutive
/// candidates count as two modes only when the valley between them drops to
/// [`MODE_VALLEY_FRACTION`] of the shorter of the pair.
fn count_modes(hist: &[(f32, usize)]) -> usize {
let counts: Vec<usize> = hist.iter().map(|(_, c)| *c).collect();
let peak = counts.iter().copied().max().unwrap_or(0);
if peak == 0 {
return 0;
}
let min_peak = (peak as f32 * MODE_PEAK_FRACTION).ceil() as usize;
// Local maxima (plateaus count once, at their first bin).
let mut candidates: Vec<usize> = Vec::new();
for i in 0..counts.len() {
if counts[i] < min_peak {
continue;
}
let left = if i == 0 { 0 } else { counts[i - 1] };
let right = if i + 1 == counts.len() { 0 } else { counts[i + 1] };
if counts[i] >= left && counts[i] > right {
candidates.push(i);
}
}
if candidates.len() <= 1 {
return candidates.len();
}
let mut modes = 1;
let mut last = candidates[0];
for &c in &candidates[1..] {
let valley = counts[last..=c].iter().copied().min().unwrap_or(0);
let shorter = counts[last].min(counts[c]);
if (valley as f32) <= shorter as f32 * MODE_VALLEY_FRACTION {
modes += 1;
last = c;
} else if counts[c] > counts[last] {
last = c;
}
}
modes
}
/// Rank-based ROC-AUC: the probability that a randomly drawn positive scores
/// above a randomly drawn negative, with ties counting a half.
///
/// This is the separation statistic that survives a bimodal population — unlike
/// Cohen's *d*, it makes no assumption about shape, and unlike "μ_pos > μ_neg"
/// it cannot be satisfied by a distribution that overlaps completely. `0.5` is
/// a coin, which is exactly what a constant-vector embedder scores.
pub fn roc_auc(pos: &[f32], neg: &[f32]) -> f32 {
if pos.is_empty() || neg.is_empty() {
return 0.5;
}
let mut all: Vec<(f32, bool)> = pos
.iter()
.map(|&x| (x, true))
.chain(neg.iter().map(|&x| (x, false)))
.collect();
all.sort_by(|a, b| a.0.partial_cmp(&b.0).unwrap_or(std::cmp::Ordering::Equal));
// Mid-ranks for ties, then Mann-Whitney U.
let mut rank_sum_pos = 0.0f64;
let mut i = 0usize;
while i < all.len() {
let mut j = i;
while j + 1 < all.len() && all[j + 1].0 == all[i].0 {
j += 1;
}
// ranks are 1-based: i+1 ..= j+1
let mid = ((i + 1) + (j + 1)) as f64 / 2.0;
for e in &all[i..=j] {
if e.1 {
rank_sum_pos += mid;
}
}
i = j + 1;
}
let np = pos.len() as f64;
let nn = neg.len() as f64;
let u = rank_sum_pos - np * (np + 1.0) / 2.0;
(u / (np * nn)) as f32
}
/// Standardised mean difference (pooled σ). Reported alongside the AUC because
/// it is the number `μ − kσ` implicitly assumes is large — and saying it out
/// loud makes it checkable. Meaningless on a multi-modal population; the
/// verdict never keys on it alone.
pub fn cohens_d(pos: &CosineDistribution, neg: &CosineDistribution) -> f32 {
let pooled = ((pos.sd * pos.sd + neg.sd * neg.sd) / 2.0).sqrt();
if pooled < DEGENERATE_SD {
return 0.0;
}
(pos.mean - neg.mean) / pooled
}
// ---------------------------------------------------------------------------
// The calibration
// ---------------------------------------------------------------------------
/// A labelled family of populations plus their pooled distribution.
#[derive(Debug, Clone, Serialize)]
pub struct PopulationSet {
pub by_type: Vec<CosineDistribution>,
pub pooled: CosineDistribution,
}
impl PopulationSet {
/// Pool the per-type samples into one distribution. `label` names the pool.
pub fn new(label: &str, by_type: Vec<CosineDistribution>) -> Self {
let mut all: Vec<f32> = Vec::new();
for d in &by_type {
all.extend_from_slice(&d.samples);
}
let pooled = CosineDistribution::measure(label, &all);
PopulationSet { by_type, pooled }
}
/// Look one population up by label.
pub fn get(&self, label: &str) -> Option<&CosineDistribution> {
self.by_type.iter().find(|d| d.label == label)
}
}
/// Everything that makes a measured threshold attributable: WHICH space, WHICH
/// corpus, WHEN. A number without these is the same unfounded claim as the
/// heuristic it replaces.
#[derive(Debug, Clone, Serialize)]
pub struct Provenance {
/// Registry id — `jina-v2-base-code`, `jina-v2-base-code-q8`, …
pub model_id: String,
/// Fully qualified model name from the registry.
pub model_name: String,
/// SHA-256 of the weights file actually loaded. This — not the id — is what
/// makes two "same model" spaces distinguishable (fp32 vs int8 differ here
/// and nowhere else in the registry row).
pub weights_sha: String,
pub dim: usize,
/// `f32` / `int8` — the QUANTIZATION of the loaded graph, which is a
/// property of the weights file, not of the pooled output element type.
pub dtype: String,
/// What was embedded (`"nornir/src, 215 files, 1 842 functions"`).
pub corpus: String,
/// `YYYY-MM-DD` — passed in, never read from the clock, so a recorded
/// calibration and a re-run of it are comparable.
pub measured_at: String,
}
/// What the measurement licenses.
#[derive(Debug, Clone, PartialEq, Serialize)]
#[serde(tag = "verdict", rename_all = "snake_case")]
pub enum Verdict {
/// Positives are unimodal and separated from the negatives; `μ − kσ` is a
/// number the lens may ship.
Calibrated,
/// Positives and negatives cannot be told apart in this space — a coin-flip
/// AUC, or no spread at all. **This is a finding about the EMBEDDER, not
/// about the threshold**, and no floor derived here would mean anything.
NoSeparation { auc: f32, reason: String },
/// The positives have more than one peak, so a single μ describes none of
/// them. Carries the per-type floors, which are the honest answer.
PerTypeRequired {
modes: usize,
per_type_floor: Vec<(String, f32)>,
},
/// Separation exists and the shape is fine, but `μ − kσ` lands INSIDE the
/// negative distribution, so the floor would admit hard negatives at the
/// stated rate. `k` is too large for this space.
FloorInsideNegatives { hard_negative_survival: f32 },
}
impl Verdict {
/// True only for [`Verdict::Calibrated`] — the one state a shipped floor may
/// be taken from without a caveat.
pub fn is_shippable(&self) -> bool {
matches!(self, Verdict::Calibrated)
}
}
/// Minimum ROC-AUC before the two populations count as distinguishable at all.
/// 0.75 is deliberately LOW: this gate exists to catch a dead embedder, not to
/// judge a good one, and setting it near 0.5 would let a coin through.
pub const MIN_AUC: f32 = 0.75;
/// A floor may admit at most this fraction of HARD negatives (same signature,
/// different body). Above it, `k` is too generous for the space.
pub const MAX_HARD_NEGATIVE_SURVIVAL: f32 = 0.10;
/// A measured operating point for one embedding space.
#[derive(Debug, Clone, Serialize)]
pub struct Calibration {
pub provenance: Provenance,
pub positives: PopulationSet,
pub negatives: PopulationSet,
/// The `k` in `μ_pos − k·σ_pos`.
pub k: f32,
/// `μ_pos − k·σ_pos` over the POOLED positives.
pub cosine_floor: f32,
/// The most conservative per-type floor — `min` over the positive types.
/// When the positives are multi-modal this, not [`Self::cosine_floor`], is
/// the number that keeps the weakest clone type.
pub conservative_floor: f32,
pub auc: f32,
pub cohens_d: f32,
/// Fraction of TRUE clones the floor keeps.
pub recall: f32,
/// Fraction of HARD negatives the floor lets through — the number that
/// decides whether a report is actionable.
pub hard_negative_survival: f32,
/// Fraction of RANDOM negatives the floor lets through.
pub random_negative_survival: f32,
pub verdict: Verdict,
}
/// Label of the hard-negative population — same signature, different body.
pub const HARD_NEGATIVE: &str = "hard-negative";
/// Label of the random cross-file negative population.
pub const RANDOM_NEGATIVE: &str = "random-negative";
impl Calibration {
/// Derive the operating point. Everything here is arithmetic over the two
/// population sets; nothing embeds, so this is testable against a synthetic
/// embedder at CI speed (which is how the RED proofs run).
pub fn compute(
provenance: Provenance,
positives: PopulationSet,
negatives: PopulationSet,
k: f32,
) -> Self {
let cosine_floor = positives.pooled.floor_at(k);
let conservative_floor = positives
.by_type
.iter()
.filter(|d| d.n > 0)
.map(|d| d.floor_at(k))
.fold(f32::INFINITY, f32::min);
let conservative_floor = if conservative_floor.is_finite() {
conservative_floor
} else {
cosine_floor
};
let auc = roc_auc(&positives.pooled.samples, &negatives.pooled.samples);
let d = cohens_d(&positives.pooled, &negatives.pooled);
let recall = positives.pooled.survival(cosine_floor);
let hard = negatives
.get(HARD_NEGATIVE)
.map(|d| d.survival(cosine_floor))
.unwrap_or(0.0);
let random = negatives
.get(RANDOM_NEGATIVE)
.map(|d| d.survival(cosine_floor))
.unwrap_or(0.0);
let verdict = if positives.pooled.n == 0 || negatives.pooled.n == 0 {
Verdict::NoSeparation {
auc,
reason: "one of the populations is EMPTY — nothing was measured".to_string(),
}
} else if positives.pooled.is_degenerate() && negatives.pooled.is_degenerate() {
// The constant-vector embedder lands exactly here: every cosine is
// the same value, so positives and negatives are indistinguishable
// and the identity value 1.0 proves nothing.
Verdict::NoSeparation {
auc,
reason: format!(
"both populations are DEGENERATE (σ_pos={:.2e}, σ_neg={:.2e} < {DEGENERATE_SD:.0e}) \
— the embedder returns the same vector regardless of input; a cosine of \
{:.4} here is the identity value, not a similarity",
positives.pooled.sd, negatives.pooled.sd, positives.pooled.mean,
),
}
} else if auc < MIN_AUC {
Verdict::NoSeparation {
auc,
reason: format!(
"ROC-AUC {auc:.3} < {MIN_AUC:.2}: true clones do not score above \
non-clones in this space. This is a finding about the EMBEDDER, not \
about the threshold — no floor derived from it would mean anything"
),
}
} else if !positives.pooled.is_unimodal() {
Verdict::PerTypeRequired {
modes: positives.pooled.modes,
per_type_floor: positives
.by_type
.iter()
.filter(|d| d.n > 0)
.map(|d| (d.label.clone(), d.floor_at(k)))
.collect(),
}
} else if hard > MAX_HARD_NEGATIVE_SURVIVAL {
Verdict::FloorInsideNegatives {
hard_negative_survival: hard,
}
} else {
Verdict::Calibrated
};
Calibration {
provenance,
positives,
negatives,
k,
cosine_floor,
conservative_floor,
auc,
cohens_d: d,
recall,
hard_negative_survival: hard,
random_negative_survival: random,
verdict,
}
}
/// The provenance sentence that replaces "HEURISTIC and UNCALIBRATED" on the
/// evidence note. States the space, the corpus, the date, the moments, the
/// shape and — because a threshold is only worth what it does — the recall
/// and hard-negative survival at the floor.
pub fn note(&self) -> String {
let p = &self.provenance;
let shape = if self.positives.pooled.is_unimodal() {
"unimodal".to_string()
} else {
format!("{}-MODAL (a single μ describes none of them)", self.positives.pooled.modes)
};
let verdict = match &self.verdict {
Verdict::Calibrated => "CALIBRATED".to_string(),
Verdict::NoSeparation { reason, .. } => format!("NOT CALIBRATED — {reason}"),
Verdict::PerTypeRequired { per_type_floor, .. } => format!(
"NOT a single threshold — per clone type: {}",
per_type_floor
.iter()
.map(|(l, f)| format!("{l}={f:.3}"))
.collect::<Vec<_>>()
.join(", ")
),
Verdict::FloorInsideNegatives {
hard_negative_survival,
} => format!(
"FLOOR INSIDE THE NEGATIVES — {:.1}% of hard negatives survive it (k={} is too large here)",
hard_negative_survival * 100.0,
self.k
),
};
format!(
"Cosine floor {:.3} = μ_pos {:.4} − {}·σ_pos {:.4}, MEASURED {} against embedding space \
`{}` ({}, weights sha256 {}, {}-dim, {}) over corpus {}. Positive population: {} \
constructed clone pair(s) ({}), negatives: {} pair(s) of which {} hard (same \
signature, different body). ROC-AUC {:.3}, Cohen's d {:.2}. At this floor {:.1}% of \
true clones survive and {:.1}% of hard negatives do. Verdict: {}. Pairs whose two \
members share ONE embedding chunk carry a cosine of 1.0 by construction and no \
similarity signal; a source-fed corpus is per-FUNCTION and cannot produce them, \
a warehouse-fed one must drop them first (`filter_same_chunk`).",
self.cosine_floor,
self.positives.pooled.mean,
self.k,
self.positives.pooled.sd,
p.measured_at,
p.model_id,
p.model_name,
&p.weights_sha,
p.dim,
p.dtype,
p.corpus,
self.positives.pooled.n,
shape,
self.negatives.pooled.n,
self.negatives
.get(HARD_NEGATIVE)
.map(|d| d.n)
.unwrap_or(0),
self.auc,
self.cohens_d,
self.recall * 100.0,
self.hard_negative_survival * 100.0,
verdict,
)
}
/// Full auditable dump — every population line plus the positive histogram.
pub fn report(&self) -> String {
let mut s = String::new();
s.push_str(&format!("{}\n\n", self.note()));
s.push_str("POSITIVES\n");
for d in &self.positives.by_type {
s.push_str(&format!(" {}\n", d.line()));
}
s.push_str(&format!(" {}\n", self.positives.pooled.line()));
s.push_str(&self.positives.pooled.sparkline());
s.push_str("NEGATIVES\n");
for d in &self.negatives.by_type {
s.push_str(&format!(" {}\n", d.line()));
}
s.push_str(&format!(" {}\n", self.negatives.pooled.line()));
s.push_str(&self.negatives.pooled.sparkline());
s
}
/// **The floor that admits at most `rate` of the HARD negatives**, and the
/// recall it costs.
///
/// This is the operating point `μ_pos − k·σ_pos` cannot express. That
/// formula only looks at the POSITIVES, so it is blind to how far the
/// negatives reach — and when the two distributions overlap (which is what
/// hard negatives are FOR), every k trades recall for precision along a
/// curve the formula never sees. Reading the floor off the negative
/// distribution's quantile targets the quantity that actually decides
/// whether a report is actionable: how much noise survives.
///
/// Returns `None` when there is no hard-negative population to read.
pub fn floor_for_hard_negative_rate(&self, rate: f32) -> Option<(f32, f32)> {
let hard = self.negatives.get(HARD_NEGATIVE)?;
if hard.samples.is_empty() {
return None;
}
// The smallest floor whose survival is <= rate: the (1-rate) quantile of
// the hard negatives, nudged past the sample sitting exactly on it.
let floor = quantile(&hard.samples, (1.0 - rate).clamp(0.0, 1.0));
let floor = (floor + 1e-4).min(1.0);
Some((floor, self.positives.pooled.survival(floor)))
}
/// Recall / hard-negative survival as `k` varies — the table that shows
/// whether the chosen `k` sits on a cliff or a plateau. Reported rather than
/// optimised: picking `k` to maximise a number measured on the same data is
/// how a calibration becomes a fit.
pub fn k_sweep(&self, ks: &[f32]) -> Vec<(f32, f32, f32, f32)> {
ks.iter()
.map(|&k| {
let floor = self.positives.pooled.floor_at(k);
let recall = self.positives.pooled.survival(floor);
let hard = self
.negatives
.get(HARD_NEGATIVE)
.map(|d| d.survival(floor))
.unwrap_or(0.0);
(k, floor, recall, hard)
})
.collect()
}
}
// ---------------------------------------------------------------------------
// Building the populations
// ---------------------------------------------------------------------------
/// A function lifted out of real source, with everything the pair builders need.
/// Built from the code room's item spans and the checkout's text
/// ([`crate::mimir_lens::room_feed::corpus::from_room`]).
#[derive(Debug, Clone)]
pub struct CorpusFn {
/// Repo-relative path.
pub file: String,
/// Enclosing `impl`/`trait` name, or `""` for a free function.
pub container: String,
pub name: String,
/// 1-based inclusive first line.
pub start_line: usize,
pub end_line: usize,
/// The function's own source text, doc comments and attributes included.
pub text: String,
/// The signature with the function's NAME removed — the key hard negatives
/// are grouped on ("same signature, different body").
pub signature_key: String,
/// The full signature text, as `symbol_facts.signature` would carry it.
pub signature: String,
/// Callee identifiers this function invokes — the OTHER half of
/// [`crate::mimir_analyze::structural_agreement`]. Without these the
/// structural gate degenerates to a signature comparison, which would make
/// a survivor count measured through the real lens meaningless.
pub callees: BTreeSet<String>,
}
impl CorpusFn {
/// Stable identity for a corpus row.
pub fn id(&self) -> String {
if self.container.is_empty() {
format!("{}::{}", self.file, self.name)
} else {
format!("{}::{}::{}", self.file, self.container, self.name)
}
}
/// Line count of the function's own body — the honest size signal
/// (`symbol_facts` has none; see stinky mimir-2).
pub fn span_lines(&self) -> usize {
self.end_line.saturating_sub(self.start_line) + 1
}
/// The warehouse row this function WOULD be, given its embedding — so a
/// calibration can be pushed straight through the real lens
/// ([`crate::mimir_clones::analyze_semantic_clones`]) instead of through a
/// re-implementation of it (LAW 5).
///
/// ⚠ `chunk_hash` is the function's OWN identity, so
/// [`crate::mimir_analyze::shares_one_chunk_vector`] never fires on these
/// rows. That is deliberate and it is a REAL DIFFERENCE from the warehouse,
/// where an `impl` block is one chunk and 58.6 % of functions therefore
/// share a vector. A survivor count taken here is measured on a strictly
/// CLEANER corpus than the live one, and must be reported as such.
pub fn to_symbol_vec(&self, repo: &str, vector: Vec<f32>) -> crate::mimir_analyze::SymbolVec {
const SEP: char = '\u{1}';
let module_path = self
.file
.trim_end_matches(".rs")
.replace(['/', '\\'], "::");
crate::mimir_analyze::SymbolVec {
id: format!("{repo}::{}", self.id()),
repo: repo.to_string(),
crate_name: repo.to_string(),
module_path,
item_kind: if self.container.is_empty() { "fn" } else { "method" }.to_string(),
item_name: self.name.clone(),
file: self.file.clone(),
line: self.start_line as u32,
signature: Some(self.signature.clone()),
callees: self.callees.clone(),
span_lines: self.span_lines() as u32,
// The item's own extent — known exactly here, because the corpus is
// built from the code room's item spans
// ([`crate::mimir_lens::room_feed::corpus`]).
item_end_line: self.end_line as u32,
// Same shape as `mimir_analyze::chunk_key` writes: repo<SEP>file<SEP>start-end.
chunk_hash: format!(
"{repo}{SEP}{}{SEP}{}-{}",
self.file, self.start_line, self.end_line
),
impl_header: None,
vector,
}
}
}
// ---------------------------------------------------------------------------
// Clone synthesis — where the ground truth comes from
// ---------------------------------------------------------------------------
/// Rust keywords + the handful of names that must survive a rename for the text
/// to keep meaning anything.
const RESERVED: &[&str] = &[
"as", "async", "await", "break", "const", "continue", "crate", "dyn", "else", "enum",
"extern", "false", "fn", "for", "if", "impl", "in", "let", "loop", "match", "mod", "move",
"mut", "pub", "ref", "return", "self", "Self", "static", "struct", "super", "trait", "true",
"type", "unsafe", "use", "where", "while", "yield", "union", "macro_rules",
];
/// **Type-2 clone**: the same function with every LOCAL identifier and every
/// literal renamed.
///
/// Renames an identifier only when it is not a keyword, not a path segment
/// (`::`-adjacent), not a field or method access (`.`-preceded), not immediately
/// followed by `(` or `!` (a call or macro), and does not start with an
/// uppercase letter (a type). What is left is variables, parameters and the
/// function's own name — exactly the Type-2 surface. Integer literals are
/// incremented and string literals get a suffix, so literal substitution is
/// exercised too.
///
/// Deliberately a TEXT transform, not an AST one: the vector is computed from
/// text, so the perturbation must live where the model sees it.
fn is_ident_start(c: char) -> bool {
c.is_alphabetic() || c == '_'
}
fn is_ident_char(c: char) -> bool {
c.is_alphanumeric() || c == '_'
}
/// The last non-whitespace character before `at`, or `None` at the start.
fn prev_significant(chars: &[char], at: usize) -> Option<char> {
chars[..at].iter().rev().find(|c| !c.is_whitespace()).copied()
}
pub fn synth_type2(src: &str) -> String {
let bytes: Vec<char> = src.chars().collect();
let mut out = String::with_capacity(src.len());
let mut i = 0usize;
let mut names: BTreeMap<String, String> = BTreeMap::new();
let mut next = 0usize;
while i < bytes.len() {
let c = bytes[i];
if c == '"' {
// string literal — copy through, then append a suffix inside it
let start = i;
i += 1;
while i < bytes.len() {
if bytes[i] == '\\' {
i += 2;
continue;
}
if bytes[i] == '"' {
break;
}
i += 1;
}
let end = i.min(bytes.len());
let lit: String = bytes[start..end].iter().collect();
out.push_str(&lit);
out.push_str("_z");
if i < bytes.len() {
out.push('"');
i += 1;
}
continue;
}
if c.is_ascii_digit() && (i == 0 || !is_ident_char(bytes[i - 1])) {
let start = i;
while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == '_') {
i += 1;
}
let lit: String = bytes[start..i].iter().collect();
match lit.parse::<u64>() {
Ok(v) => out.push_str(&(v.wrapping_add(1)).to_string()),
Err(_) => out.push_str(&lit),
}
continue;
}
if is_ident_start(c) {
let start = i;
while i < bytes.len() && is_ident_char(bytes[i]) {
i += 1;
}
let word: String = bytes[start..i].iter().collect();
let prev = prev_significant(&bytes, start);
let nextc = bytes.get(i).copied().unwrap_or(' ');
let is_path = prev == Some(':') || (nextc == ':' && bytes.get(i + 1) == Some(&':'));
let is_field = prev == Some('.');
let is_call = nextc == '(' || nextc == '!';
let is_type = word.chars().next().is_some_and(|c| c.is_uppercase());
if RESERVED.contains(&word.as_str()) || is_path || is_field || is_call || is_type {
out.push_str(&word);
} else {
let n = names.entry(word).or_insert_with(|| {
next += 1;
format!("q{next}")
});
out.push_str(n);
}
continue;
}
out.push(c);
i += 1;
}
out
}
/// **Type-3 clone**: the same function with statements inserted and comments
/// removed.
///
/// Strips every whole-line `//` comment (a removal that a real refactor makes
/// constantly) and inserts `inserts` inert `let` bindings after the opening
/// brace of the body. Both directions of the Type-3 definition — added and
/// removed lines — are therefore present in one sample.
pub fn synth_type3(src: &str, inserts: usize) -> String {
let kept: Vec<&str> = src
.lines()
.filter(|l| !l.trim_start().starts_with("//"))
.collect();
let mut out: Vec<String> = Vec::with_capacity(kept.len() + inserts);
let mut done = false;
for l in kept {
out.push(l.to_string());
if !done && l.trim_end().ends_with('{') {
for n in 0..inserts {
out.push(format!(" let _scratch_{n} = {n} + 1;"));
}
done = true;
}
}
if !done {
// No brace found (an expression-bodied or macro-generated fn) — append
// instead of silently returning a Type-1 copy under a Type-3 label.
for n in 0..inserts {
out.push(format!("// pad {n}"));
}
}
out.join("\n")
}
/// Drop pairs whose two members would carry the SAME embedding chunk's vector.
///
/// Routes through [`crate::mimir_analyze::shares_one_chunk_vector`] — the ONE
/// predicate both duplicate lenses already use — rather than re-deriving the
/// rule here (LAW 5). A calibration that skipped this would put μ_pos at ~1.0
/// on a corpus where 58.6 % of functions share their chunk, and would then
/// "prove" that the floor should be 0.99.
///
/// **Where it is needed, and where the artefact cannot arise.**
/// [`measure_populations`] builds its corpus per FUNCTION from source, embedding
/// each function's own text, so no two rows can be handed one chunk's vector and
/// the artefact is structurally absent — that is a property of how the corpus is
/// built, and it is also why a survivor count taken this way is on a strictly
/// cleaner corpus than the warehouse's. This filter is for the other direction:
/// a calibration fed from warehouse rows, where an `impl` block is one chunk and
/// 58.6 % of functions share one.
pub fn filter_same_chunk(
pairs: &[(crate::mimir_analyze::SymbolVec, crate::mimir_analyze::SymbolVec)],
) -> (Vec<usize>, usize) {
let mut keep = Vec::new();
let mut dropped = 0usize;
for (i, (a, b)) in pairs.iter().enumerate() {
if crate::mimir_analyze::shares_one_chunk_vector(a, b) {
dropped += 1;
} else {
keep.push(i);
}
}
(keep, dropped)
}
/// Deterministic 64-bit LCG — pair sampling must be reproducible, and pulling
/// in an RNG crate for `next % n` would be the wrong trade.
pub struct Lcg(pub u64);
impl Lcg {
pub fn next(&mut self) -> u64 {
self.0 = self
.0
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
self.0 >> 11
}
pub fn below(&mut self, n: usize) -> usize {
if n == 0 { 0 } else { (self.next() % n as u64) as usize }
}
}
/// Index pairs of functions with the SAME signature key and a DIFFERENT body —
/// the hard negatives.
///
/// These, not random pairs, are what a floor has to beat. Two functions of the
/// same shape land close in any code-embedding space (both are `fn(&self, &mut
/// Formatter) -> Result`, both open with a `match`), so a floor tuned on random
/// pairs sails past them — and a 600-member mega-blob is what that looks like
/// on a real corpus.
///
/// ⚠ HONEST CAVEAT: some same-signature pairs in real code genuinely ARE
/// re-implementations of each other. This population is therefore a slight
/// UNDER-estimate of separability, never an over-estimate — a floor that clears
/// it clears the easier cases too.
pub fn hard_negative_pairs(fns: &[CorpusFn], max_pairs: usize, seed: u64) -> Vec<(usize, usize)> {
let mut by_sig: BTreeMap<&str, Vec<usize>> = BTreeMap::new();
for (i, f) in fns.iter().enumerate() {
by_sig.entry(f.signature_key.as_str()).or_default().push(i);
}
let mut rng = Lcg(seed);
let mut out = Vec::new();
let mut seen: BTreeSet<(usize, usize)> = BTreeSet::new();
let groups: Vec<&Vec<usize>> = by_sig.values().filter(|v| v.len() >= 2).collect();
if groups.is_empty() {
return out;
}
let mut guard = 0usize;
while out.len() < max_pairs && guard < max_pairs * 50 {
guard += 1;
let g = groups[rng.below(groups.len())];
let a = g[rng.below(g.len())];
let b = g[rng.below(g.len())];
if a == b || fns[a].text == fns[b].text {
continue;
}
let key = (a.min(b), a.max(b));
if seen.insert(key) {
out.push(key);
}
}
out
}
/// Index pairs of functions from DIFFERENT files — the easy negatives. Kept
/// only so the report can show how much easier they are than the hard ones;
/// calibrating on these alone is the mistake this module names.
pub fn random_negative_pairs(fns: &[CorpusFn], max_pairs: usize, seed: u64) -> Vec<(usize, usize)> {
let mut rng = Lcg(seed);
let mut out = Vec::new();
let mut seen: BTreeSet<(usize, usize)> = BTreeSet::new();
let mut guard = 0usize;
while out.len() < max_pairs && guard < max_pairs * 50 && fns.len() > 1 {
guard += 1;
let a = rng.below(fns.len());
let b = rng.below(fns.len());
if a == b || fns[a].file == fns[b].file {
continue;
}
let key = (a.min(b), a.max(b));
if seen.insert(key) {
out.push(key);
}
}
out
}
/// The three positive clone types, as `(label, transform)`.
///
/// Type-1 is present but NEVER alone: it is the identity value, and the whole
/// point of the other two is that a dead embedder cannot fake them.
///
/// ⚠ Type-1 is nonetheless NOT a tautology in this oracle, because the copy is
/// embedded as its own text rather than compared with itself. Its cosine
/// therefore measures **batch invariance**: `vector::embed` documents the
/// batched forward as "bit-identical to the one-at-a-time path", and a Type-1 μ
/// below 1.0 falsifies that for the loaded weights. MEASURED 2026-08-25 on
/// oden's ort-CUDA path: jina fp32 gives μ=1.0000 σ=0.0000 (invariant), the int8
/// export gives μ=0.9965 with a worst pair at 0.9632 (NOT invariant — the same
/// bytes embed differently depending on what shares their batch).
pub fn positive_transforms() -> Vec<(&'static str, fn(&str) -> String)> {
fn t1(s: &str) -> String {
s.to_string()
}
fn t2(s: &str) -> String {
synth_type2(s)
}
fn t3(s: &str) -> String {
synth_type3(s, 3)
}
vec![("type-1", t1 as fn(&str) -> String), ("type-2", t2), ("type-3", t3)]
}
/// Run the whole oracle over a corpus, given a way to embed text.
///
/// `embed` takes the texts and returns one L2-comparable vector each, in order —
/// the same contract as [`crate::vector::store::Embedder::embed`], expressed as
/// a closure so this core has no feature gate and the RED proofs can drive it
/// with a deliberately broken embedder.
///
/// Returns `(positives, negatives)` ready for [`Calibration::compute`].
pub fn measure_populations<E>(
fns: &[CorpusFn],
sample: usize,
negatives_per_class: usize,
seed: u64,
mut embed: E,
) -> anyhow::Result<(PopulationSet, PopulationSet)>
where
E: FnMut(&[&str]) -> anyhow::Result<Vec<Vec<f32>>>,
{
// Which originals get cloned. Spread over the corpus rather than taking a
// prefix, so the sample is not "whatever is alphabetically first".
let mut rng = Lcg(seed);
let mut chosen: Vec<usize> = Vec::new();
let mut seen: BTreeSet<usize> = BTreeSet::new();
let want = sample.min(fns.len());
let mut guard = 0usize;
while chosen.len() < want && guard < want * 100 {
guard += 1;
let i = rng.below(fns.len());
if seen.insert(i) {
chosen.push(i);
}
}
let transforms = positive_transforms();
// One flat text list: every original, then every transform of it, then the
// negative members. ONE embed call keeps batching identical for every
// population, so a per-population difference cannot come from batch shape.
let mut texts: Vec<String> = Vec::new();
let mut orig_at: BTreeMap<usize, usize> = BTreeMap::new();
for &i in &chosen {
orig_at.insert(i, texts.len());
texts.push(fns[i].text.clone());
}
let mut clone_at: Vec<(usize, &'static str, usize)> = Vec::new();
for &i in &chosen {
for (label, f) in &transforms {
clone_at.push((i, label, texts.len()));
texts.push(f(&fns[i].text));
}
}
let hard = hard_negative_pairs(fns, negatives_per_class, seed ^ 0x51ed);
let rand = random_negative_pairs(fns, negatives_per_class, seed ^ 0xa17e);
let mut neg_at: BTreeMap<usize, usize> = BTreeMap::new();
for &(a, b) in hard.iter().chain(rand.iter()) {
for i in [a, b] {
if !orig_at.contains_key(&i) && !neg_at.contains_key(&i) {
neg_at.insert(i, texts.len());
texts.push(fns[i].text.clone());
}
}
}
let at = |i: usize| -> usize {
*orig_at.get(&i).or_else(|| neg_at.get(&i)).expect("indexed")
};
let refs: Vec<&str> = texts.iter().map(|s| s.as_str()).collect();
let vecs = embed(&refs)?;
anyhow::ensure!(
vecs.len() == texts.len(),
"embedder returned {} vectors for {} texts",
vecs.len(),
texts.len()
);
let cos = |a: usize, b: usize| crate::mimir_analyze::cosine(&vecs[a], &vecs[b]);
let mut by_type: BTreeMap<&'static str, Vec<f32>> = BTreeMap::new();
for (i, label, slot) in &clone_at {
by_type.entry(label).or_default().push(cos(orig_at[i], *slot));
}
let positives = PopulationSet::new(
"positives",
transforms
.iter()
.map(|(l, _)| CosineDistribution::measure(*l, by_type.get(l).map(|v| v.as_slice()).unwrap_or(&[])))
.collect(),
);
let hard_c: Vec<f32> = hard.iter().map(|&(a, b)| cos(at(a), at(b))).collect();
let rand_c: Vec<f32> = rand.iter().map(|&(a, b)| cos(at(a), at(b))).collect();
let negatives = PopulationSet::new(
"negatives",
vec![
CosineDistribution::measure(HARD_NEGATIVE, &hard_c),
CosineDistribution::measure(RANDOM_NEGATIVE, &rand_c),
],
);
Ok((positives, negatives))
}
/// **Is a zero a finding or a silent empty?** — count the RAW pairs above each
/// floor, before any structural gate, same-kind gate or lens partition touches
/// them.
///
/// A lens that returns zero clusters at cosine 0.93 is saying one of two very
/// different things: "no pair in this corpus is that close" or "plenty are, and
/// every one of them was routed to the other lens / refused by the structural
/// ceiling". Only the census can tell them apart, and reporting the cluster
/// count without it is exactly the shape of a check that cannot fail.
///
/// `O(n²/2)` cosines through [`crate::mimir_analyze::cosine`] — the same
/// function the lenses use, so the census and the gate cannot disagree.
pub fn cosine_pair_census(
rows: &[crate::mimir_analyze::SymbolVec],
floors: &[f32],
) -> Vec<(f32, usize)> {
let mut counts = vec![0usize; floors.len()];
for i in 0..rows.len() {
for j in (i + 1)..rows.len() {
let c = crate::mimir_analyze::cosine(&rows[i].vector, &rows[j].vector);
for (k, &f) in floors.iter().enumerate() {
if c >= f {
counts[k] += 1;
}
}
}
}
floors.iter().copied().zip(counts).collect()
}
// ---------------------------------------------------------------------------
// The RECORDED operating points — what the lens reads at runtime
// ---------------------------------------------------------------------------
/// One measurement, frozen into the binary so the lens can look up an operating
/// point without re-running an embedder.
///
/// Every field is provenance except the two floors. A recorded calibration that
/// could not say WHICH space, WHICH corpus and WHEN would be the same unfounded
/// claim as the heuristic it replaces — so the struct makes those non-optional
/// and [`Self::note`] prints all of them.
#[derive(Debug, Clone, Copy, PartialEq, Serialize)]
pub struct RecordedCalibration {
pub model_id: &'static str,
/// SHA-256 of the weights file the measurement ran through.
pub weights_sha: &'static str,
/// Length of that weights file in bytes — the CHEAP identity, and the one
/// [`active`] matches on. Reading 611 MB to hash it on every lens call is
/// not acceptable, and the two jina exports differ here by 4× (641 517 466
/// vs 161 895 621), so length alone separates every space in the registry.
/// The SHA above is what a human verifies when it matters.
pub weights_bytes: u64,
/// `f32` / `int8` — read off the weights, NOT off `ModelProfile.dtype`,
/// which is a hardcoded `"f32"` for every model in the registry.
pub dtype: &'static str,
pub dim: usize,
pub corpus: &'static str,
/// `YYYY-MM-DD`.
pub measured_at: &'static str,
pub k: f32,
/// What [`Calibration::compute`] returned for the pooled positives.
pub pooled_floor: f32,
/// The weakest clone TYPE's floor — the number to ship when the positives
/// are multi-modal, because it is the one that still keeps Type-3 clones.
pub conservative_floor: f32,
/// The floor this record RECOMMENDS for `mimir_clones`, chosen by the
/// measurement's own verdict rather than by taste.
pub clone_cosine_floor: f32,
/// The floor this record recommends for `mimir_analyze`'s copied-not-moved
/// lens, whose positives are Type-1/Type-2 (a literal copy), not Type-3.
pub dup_cosine: f32,
pub auc: f32,
/// Fraction of constructed clones the recommended floor keeps.
pub recall: f32,
/// Fraction of HARD negatives (same signature, different body) it admits.
pub hard_negative_survival: f32,
/// Peak count of the pooled positive distribution. `> 1` means the single
/// pooled floor is NOT the honest answer and per-type floors are recorded.
pub positive_modes: usize,
/// `(clone type, its own μ−kσ)`.
pub per_type_floor: &'static [(&'static str, f32)],
/// What the run measured on the real corpus, at the shipped floor and at
/// this one: `(floor, clusters)`. The SURVIVOR count — a threshold that does
/// not move it is not a fix.
pub survivors: &'static [(f32, usize)],
/// What [`Calibration::compute`] said about `mu_pos - k*sigma_pos` in this
/// space, in one machine-readable word. **`"floor_inside_negatives"` is not
/// a footnote** — it means the formula was measured and REFUSED here, and
/// [`Self::clone_cosine_floor`] therefore comes from somewhere else. A
/// record that hid that would claim a derivation it does not have.
pub verdict: &'static str,
/// How [`Self::clone_cosine_floor`] was derived, in one clause, for the note.
pub floor_method: &'static str,
}
impl RecordedCalibration {
/// The provenance sentence the evidence note carries INSTEAD of "HEURISTIC
/// and UNCALIBRATED".
pub fn note(&self) -> String {
let shape = if self.positive_modes <= 1 {
"unimodal".to_string()
} else {
format!(
"{}-MODAL, so the pooled μ−kσ ({:.3}) describes no single clone type and the WEAKEST type's floor is what ships; per type: {}",
self.positive_modes,
self.pooled_floor,
self.per_type_floor
.iter()
.map(|(l, f)| format!("{l}={f:.3}"))
.collect::<Vec<_>>()
.join(", ")
)
};
let survivors = self
.survivors
.iter()
.map(|(f, c)| format!("{f:.2}→{c}"))
.collect::<Vec<_>>()
.join(", ");
format!(
"Cosine floor {:.3} is CALIBRATED, not heuristic: MEASURED {} against the \
embedding space `{}` ({} weights, {}-dim, sha256 {}…, {} bytes) over corpus {}, \
using constructed Type-1/2/3 clone pairs against hard negatives (same signature, \
different body). Derivation: {}. The mu_pos-k*sigma_pos form WAS measured, at \
k={} (floor {:.3}), and the verdict on it was `{}`. ROC-AUC {:.3}; at the floor \
above, {:.1}% of constructed clones survive and {:.1}% of hard negatives do. \
Positive distribution is {}. Cluster survivors on that corpus: {}. Vectors from \
any other model_profile are a DIFFERENT space and this number does not transfer \
to them.",
self.clone_cosine_floor,
self.measured_at,
self.model_id,
self.dtype,
self.dim,
&self.weights_sha[..self.weights_sha.len().min(12)],
self.weights_bytes,
self.corpus,
self.floor_method,
self.k,
self.pooled_floor,
self.verdict,
self.auc,
self.recall * 100.0,
self.hard_negative_survival * 100.0,
shape,
survivors,
)
}
}
/// Every operating point that has ever been MEASURED, one row per embedding
/// space. Empty is the honest state for a space nobody has measured; adding a
/// row is a deliberate act backed by a run of
/// `examples/calibrate_embedding_space.rs`.
pub const RECORDED: &[RecordedCalibration] = &[
// ── jina-embeddings-v2-base-code, fp32 ONNX export ─────────────────────
//
// The space `/opt/nornir/models` actually holds on oden, and therefore the
// one every nornir on this box embeds in. Produced by
// `examples/calibrate_embedding_space src 300 600` on the ort-CUDA path,
// 2026-08-25.
//
// WHAT THE RUN SAID:
// positives n=900 μ=0.9058 σ=0.0936 UNIMODAL (modes=1)
// type-1 μ=1.0000 σ=0.0000 ← batch-invariant: identity IS 1.0 here
// type-2 μ=0.8333 σ=0.0781
// type-3 μ=0.8841 σ=0.0748
// hard-neg n=600 μ=0.6798 σ=0.2115 p95=0.9693 BIMODAL (modes=2)
// random-neg n=600 μ=0.2127 σ=0.1195 p95=0.4304
// ROC-AUC 0.920, Cohen's d 2.13
//
// ⚠ `μ_pos − k·σ_pos` DOES NOT WORK IN THIS SPACE, and that is the single
// most useful thing the measurement produced. The formula reads only the
// POSITIVES, so it cannot see how far the hard negatives reach — and here
// they reach to p95 = 0.9693, straight up through the positive mass. Every
// k from 0 to 3 puts the floor inside them:
// k=0.0 → 0.906 recall 55.3% hard-neg 15.3%
// k=1.0 → 0.812 recall 84.3% hard-neg 31.8%
// k=3.0 → 0.625 recall 99.2% hard-neg 63.7%
// The shipped floor is therefore read off the HARD-NEGATIVE quantile, which
// targets the quantity that decides whether a report can be acted on.
//
// ⚠ AND THE HARD NEGATIVES ARE PARTLY MISLABELLED, honestly: 98 of 600 sit
// at cosine ≥ 0.90, and the top of that list is `col_str`/`col_i64`/`col_ts`
// repeated across `airgap_events`, `saga_events` and `workspace_events`, and
// `tokenize` in `vector/embed.rs` vs `vector/embed_ort.rs` — real duplicated
// helpers, i.e. TRUE positives wearing a negative label. 16.3 % is an upper
// bound on how much of the measured false-positive rate is not real.
RecordedCalibration {
model_id: "jina-v2-base-code",
weights_sha: "63363fc178428b74620c6f3780cbc7191883fa5c7f84c0945c45eb5c4256733b",
weights_bytes: 641_517_466,
dtype: "f32",
dim: 768,
corpus: "nornir/src — 3 222 functions over 199 files, 5..=120 lines each (the \
warehouse's own 120-line chunk window), embedded PER FUNCTION",
measured_at: "2026-08-25",
k: 1.0,
pooled_floor: 0.812,
conservative_floor: 0.755,
// FP ≤ 20 % on the hard negatives. FP ≤ 10 % would be 0.952, and 0.952
// takes the lens to ZERO clusters on this corpus — a threshold that
// finds nothing has not been calibrated, it has been switched off.
clone_cosine_floor: 0.882,
// UNCHANGED, and measured rather than assumed: at 0.92 the copied-not-
// moved lens recovers type-1 100 %, type-2 11.7 %, type-3 39.7 % — it is
// a LITERAL-COPY gate and nothing else, which is exactly what that lens
// is for. Its AND with structural ≥ 0.50 is what keeps it precise.
dup_cosine: 0.92,
auc: 0.920,
recall: 0.633,
hard_negative_survival: 0.20,
positive_modes: 1,
per_type_floor: &[("type-1", 1.000), ("type-2", 0.755), ("type-3", 0.809)],
// MEASURED through `mimir_clones::analyze_semantic_clones` on those rows.
survivors: &[(0.755, 255), (0.800, 138), (0.812, 107), (0.882, 11), (0.952, 0)],
verdict: "floor_inside_negatives",
floor_method: "the cosine at which at most 20% of HARD negatives survive (their 80th \
percentile), NOT mu_pos - k*sigma_pos, which this space refuses",
},
];
/// Look a recorded operating point up by the weights file's byte length.
pub fn recorded_for_bytes(bytes: u64) -> Option<&'static RecordedCalibration> {
RECORDED.iter().find(|r| r.weights_bytes == bytes)
}
/// The space this process would embed in, and whether anybody has measured it.
#[derive(Debug, Clone, PartialEq, Serialize)]
pub struct ActiveSpace {
/// Registry id resolved from `$NORNIR_EMBED_MODEL`.
pub model_id: String,
/// The weights file the loader would actually open, and its size.
pub weights_path: String,
pub weights_bytes: u64,
pub calibration: Option<RecordedCalibration>,
}
impl ActiveSpace {
/// The sentence a lens carries about its thresholds — calibrated provenance
/// when this space has been measured, and an explicit, SPACE-NAMING refusal
/// when it has not. "Uncalibrated" without saying which space is uncalibrated
/// is what the old note did.
pub fn note(&self) -> String {
match &self.calibration {
Some(c) => c.note(),
None => format!(
"Thresholds are HEURISTIC and UNCALIBRATED for the space this process embeds in: model `{}`, weights {} ({} bytes) — no measured operating point is recorded for it. A new measurement needs a function corpus, whose producer (the syn extractor) was removed with Mímir v2 (lane 3 restores it); a threshold inherited from another model is a guess about a different space.",
self.model_id, self.weights_path, self.weights_bytes,
),
}
}
}
/// Resolve the loaded embedding space and its recorded operating point, WITHOUT
/// loading the model — `stat` on the weights file, not a 611 MB read.
///
/// Requires the vector feature because that is where the registry and the model
/// directory resolution live; callers compiled without it have no embedder and
/// therefore no space to be calibrated for.
#[cfg(any(feature = "embed-tract", feature = "embed-ort"))]
pub fn active() -> ActiveSpace {
let model_id = crate::vector::embed_registry::selected_model_id();
let path = crate::vector::embed_support::model_dir().join("model.onnx");
let bytes = std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0);
ActiveSpace {
model_id,
weights_path: path.display().to_string(),
weights_bytes: bytes,
calibration: recorded_for_bytes(bytes).copied(),
}
}
// ---------------------------------------------------------------------------
// Tests — the oracle, and the RED proofs that it is not hollow
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
fn provenance(model: &str) -> Provenance {
Provenance {
model_id: model.to_string(),
model_name: format!("test/{model}"),
weights_sha: "0".repeat(64),
dim: 64,
dtype: "f32".to_string(),
corpus: "nornir/src/vector".to_string(),
measured_at: "2026-08-25".to_string(),
}
}
// ---- the corpus: functions lifted out of a code room ----
/// The oracle run end to end on a corpus the code room supplies: the
/// driver's item spans say where each function is, the checkout holds its
/// lines ([`crate::mimir_lens::room_feed::corpus`]). The tree is the
/// generated one of the lens fixture — written so every extent is known by
/// construction — and the same run is repeated on a REAL lake by the
/// ignored `real_lake_calibration_corpus`.
#[cfg(feature = "code-room")]
mod on_a_room_corpus {
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
use super::super::*;
use crate::mimir_lens::room_feed::corpus;
use crate::mimir_lens::room_feed::fixture::{self, calibration_tree};
/// The generated tree's functions, as the room + checkout give them.
fn small_corpus() -> (Vec<CorpusFn>, calibration_tree::Tree) {
let tree = calibration_tree::build();
let room = tree.ws.room.room(&mimir_room::Scope::current()).unwrap();
let c = corpus::from_room(&room, calibration_tree::REPO, &tree.checkout, 5, 120).unwrap();
assert_eq!((c.macro_generated, c.without_span, c.unreadable, c.outside_size), (0, 0, 0, 0), "{c:?}");
let mut fns = c.fns;
fns.sort_by_key(|f| f.id());
assert!(fns.len() > 80, "corpus too small to calibrate on: {} fns", fns.len());
(fns, tree)
}
fn provenance(model: &str) -> Provenance {
Provenance {
model_id: model.to_string(),
model_name: format!("test/{model}"),
weights_sha: "0".repeat(64),
dim: 64,
dtype: "f32".to_string(),
corpus: "plant (generated tree), 120 fns over 24 files".to_string(),
measured_at: "2026-10-13".to_string(),
}
}
fn l2(v: &mut [f32]) {
let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();
if n > 0.0 {
for x in v.iter_mut() {
*x /= n;
}
}
}
/// A WORKING (if weak) embedder: hashed bag-of-tokens. Genuinely semantic in
/// the only sense that matters here — its output depends on the input's
/// content, similar texts land close, unrelated texts do not. This is what
/// gives the oracle a GREEN it is allowed to trust at CI speed.
fn bag_embed(texts: &[&str]) -> anyhow::Result<Vec<Vec<f32>>> {
const D: usize = 512;
Ok(texts
.iter()
.map(|t| {
let mut v = vec![0f32; D];
for tok in t.split(|c: char| !c.is_alphanumeric() && c != '_') {
if tok.is_empty() {
continue;
}
let mut h = DefaultHasher::new();
tok.hash(&mut h);
v[(h.finish() % D as u64) as usize] += 1.0;
}
l2(&mut v);
v
})
.collect())
}
/// THE DEAD EMBEDDER: one constant vector, whatever the input. Every cosine
/// it produces is exactly 1.0 — the identity value — and an oracle that
/// cannot see through that is measuring nothing.
fn constant_embed(texts: &[&str]) -> anyhow::Result<Vec<Vec<f32>>> {
Ok(texts.iter().map(|_| vec![0.125f32; 64]).collect())
}
/// THE SUBTLER DEAD EMBEDDER: hash the WHOLE text to a random direction.
/// Byte-identical texts collide exactly (Type-1 → cosine 1.0, looks
/// perfect!) while everything else is orthogonal noise. This is precisely
/// the embedder a Type-1-only oracle would certify.
fn whole_text_hash_embed(texts: &[&str]) -> anyhow::Result<Vec<Vec<f32>>> {
const D: usize = 64;
Ok(texts
.iter()
.map(|t| {
let mut h = DefaultHasher::new();
t.hash(&mut h);
let mut s = Lcg(h.finish() | 1);
let mut v: Vec<f32> = (0..D).map(|_| (s.next() % 2000) as f32 / 1000.0 - 1.0).collect();
l2(&mut v);
v
})
.collect())
}
fn run(embed: fn(&[&str]) -> anyhow::Result<Vec<Vec<f32>>>, model: &str) -> Calibration {
let _g = fixture::serial();
let (fns, _tree) = small_corpus();
let (pos, neg) = measure_populations(&fns, 60, 120, 0x9e37, embed).unwrap();
Calibration::compute(provenance(model), pos, neg, 1.0)
}
/// Every extracted text is the WHOLE function and nothing else: from its
/// doc line to its closing brace, exactly as it was written — which a
/// module-wide slice or a 120-line window would not be. And what the
/// room knows about it came along: its parameters, its callees.
#[test]
fn corpus_functions_are_whole_functions_not_chunks() {
let _g = fixture::serial();
let (fns, tree) = small_corpus();
assert_eq!(fns.len(), tree.written.len(), "every generated function, once");
for (file, start, end, generated) in &tree.written {
let f = fns.iter().find(|f| &f.file == file && f.name == generated.name).unwrap_or_else(|| panic!("{file}::{} is not in the corpus", generated.name));
assert_eq!((f.start_line, f.end_line), (*start, *end), "{}", f.id());
assert_eq!(f.text, generated.lines.join("\n"), "{} is not the function's own text", f.id());
assert!(f.text.starts_with("/// ") && f.text.ends_with("\n}"), "{}:\n{}", f.id(), f.text);
assert_eq!(f.text.matches("pub fn ").count(), 1, "{} holds more than one function", f.id());
assert_eq!(f.container, "", "a free function");
assert_eq!(f.callees.iter().cloned().collect::<Vec<_>>(), generated.callees, "{}", f.id());
let ty = &generated.params[0].1;
assert!(f.signature.starts_with(&format!("fn {}(batch: {ty}, ", generated.name)), "{}", f.signature);
}
// same shape, different body: the three wide functions of a module share a key, no two modules do
let key_of = |name: &str| fns.iter().find(|f| f.name == name).map(|f| f.signature_key.clone()).unwrap();
assert_eq!(key_of("settle_ledger"), key_of("merge_ledger"));
assert_ne!(key_of("settle_ledger"), key_of("audit_ledger"));
assert_ne!(key_of("settle_ledger"), key_of("settle_parcel"));
}
// ---- GREEN, and only against something that could have been RED ----
/// The honest green: a weak-but-real embedder separates clones from hard
/// negatives. `auc > 0.5` would be a meaningless bar — it is what a
/// constant returns — so the bar is `> 0.80`, the level the two RED tests
/// below miss by scoring 0.50 and a refusal against the same oracle.
#[test]
fn a_working_embedder_separates_clones_from_hard_negatives() {
let c = run(bag_embed, "bag-of-tokens");
eprintln!("{}", c.report());
assert!(c.auc > 0.80, "a working embedder should separate clones from hard negatives, AUC={:.3}\n{}", c.auc, c.report());
// APPLIED output, not a state round-trip: clones must score above
// same-signature-different-body pairs, on the real samples.
let hard = c.negatives.get(HARD_NEGATIVE).unwrap();
assert!(c.positives.pooled.mean > hard.median, "positive mean {:.4} is not above the hard-negative median {:.4}", c.positives.pooled.mean, hard.median);
// ⚠ and the hard negatives really are HARD: their p95 reaches far
// above the random pairs', which is the whole reason random pairs are
// not enough to calibrate on.
assert!(
hard.p95 > c.negatives.get(RANDOM_NEGATIVE).unwrap().p95 + 0.2,
"hard negatives ({:.4} p95) are no harder than random ones ({:.4}) — \
the hard-negative construction is not doing anything",
hard.p95,
c.negatives.get(RANDOM_NEGATIVE).unwrap().p95
);
// The positives are multi-modal (Type-1 pinned at 1.0, Type-2/3
// below), so the oracle must REFUSE a single number rather than report
// the pooled μ−kσ as if it described them.
assert!(
matches!(c.verdict, Verdict::PerTypeRequired { .. }),
"positives spanning 1.000/{:.3}/{:.3} were accepted as ONE threshold: {:?}",
c.positives.get("type-2").unwrap().mean,
c.positives.get("type-3").unwrap().mean,
c.verdict
);
// Type-1 MUST be the identity value; if it is not, the embedder is broken.
let t1 = c.positives.get("type-1").unwrap();
assert!((t1.mean - 1.0).abs() < 1e-3, "byte-identical copies embed at {:.4}, not 1.0 — the embedder is not deterministic", t1.mean);
// …and Type-2/Type-3 must NOT be, or the oracle is measuring the identity.
for l in ["type-2", "type-3"] {
let d = c.positives.get(l).unwrap();
assert!(d.mean < 0.999, "{l} pairs embed at {:.4} — the transform did not change the vector, so this population is Type-1 under another name", d.mean);
}
}
// ---- RED: the dead embedders must be caught ----
#[test]
fn constant_vector_embedder_is_refused_not_certified() {
let c = run(constant_embed, "constant-vector");
eprintln!("{}", c.note());
// Every cosine is exactly the identity value.
assert_eq!(c.positives.pooled.mean, 1.0);
assert_eq!(c.negatives.pooled.mean, 1.0);
// A naive "μ_pos - kσ_pos" would hand back a beautiful 1.000 floor.
assert_eq!(c.cosine_floor, 1.0, "the naive formula really does return 1.0 here");
// The oracle must not.
assert!(!c.verdict.is_shippable(), "a constant-vector embedder was CERTIFIED — the oracle is hollow: {:?}", c.verdict);
assert!(matches!(c.verdict, Verdict::NoSeparation { .. }), "wrong refusal reason: {:?}", c.verdict);
assert!((c.auc - 0.5).abs() < 1e-6, "AUC {:.4} is not a coin on an all-ties population", c.auc);
assert!(c.note().contains("identity value"), "note hides the cause: {}", c.note());
}
#[test]
fn whole_text_hash_embedder_fools_a_type1_only_oracle_but_not_this_one() {
let c = run(whole_text_hash_embed, "whole-text-hash");
eprintln!("{}", c.report());
// The trap: measured on Type-1 ALONE this embedder looks flawless.
let t1 = c.positives.get("type-1").unwrap();
assert_eq!(t1.mean, 1.0, "Type-1 identity should be perfect here");
assert!(t1.is_degenerate(), "Type-1 alone has zero spread — μ−kσ = 1.000");
// But it has no semantics at all: a renamed copy is orthogonal noise.
assert!(c.positives.get("type-2").unwrap().mean.abs() < 0.30, "type-2 mean {:.4} — the hash embedder should carry no semantics", c.positives.get("type-2").unwrap().mean);
// So the full oracle must refuse it.
assert!(!c.verdict.is_shippable(), "a semantics-free embedder was CERTIFIED from its Type-1 identity: {:?}", c.verdict);
}
#[test]
fn note_carries_full_provenance_or_it_is_not_a_calibration() {
let c = run(bag_embed, "bag-of-tokens");
let n = c.note();
for needle in ["bag-of-tokens", "2026-10-13", "plant (generated tree)", "sha256", "ROC-AUC", "hard negatives"] {
assert!(n.contains(needle), "note is missing `{needle}`:\n{n}");
}
}
#[test]
fn k_sweep_moves_the_floor_and_the_rates_with_it() {
let c = run(bag_embed, "bag-of-tokens");
let sweep = c.k_sweep(&[0.5, 1.0, 2.0, 3.0]);
for w in sweep.windows(2) {
assert!(w[1].1 <= w[0].1, "floor did not fall as k grew: {sweep:?}");
assert!(w[1].2 >= w[0].2 - 1e-6, "recall did not rise as the floor fell: {sweep:?}");
}
assert!(sweep[0].1 > sweep[3].1, "k had no effect at all on the floor — the sweep is a no-op: {sweep:?}");
}
/// **The same oracle on a REAL corpus**: every function of a repo a real
/// consumer wrote a lake for, lifted out with the driver's spans.
/// `#[ignore]`d — it reads a lake and a checkout outside the tree:
///
/// ```text
/// MIMIR_PROOF_BUILDS=<builds dir> MIMIR_PROOF_REPO=<repo> MIMIR_PROOF_CHECKOUT=<its checkout> \\
/// cargo test --lib real_lake_calibration_corpus -- --ignored --nocapture
/// ```
#[test]
#[ignore = "reads a real lake: set MIMIR_PROOF_BUILDS / MIMIR_PROOF_REPO / MIMIR_PROOF_CHECKOUT"]
fn real_lake_calibration_corpus() {
let builds = std::path::PathBuf::from(std::env::var("MIMIR_PROOF_BUILDS").expect("MIMIR_PROOF_BUILDS"));
let repo = std::env::var("MIMIR_PROOF_REPO").expect("MIMIR_PROOF_REPO");
let checkout = std::path::PathBuf::from(std::env::var("MIMIR_PROOF_CHECKOUT").expect("MIMIR_PROOF_CHECKOUT"));
let handle = crate::code_room::CodeRoom::new("proof", &builds);
let room = handle.room(&mimir_room::Scope::current()).unwrap();
let t = std::time::Instant::now();
let c = corpus::from_room(&room, &repo, &checkout, 5, 120).unwrap();
println!(
"corpus of `{repo}`: {} function(s) of {} callable(s) in {} ms — left out: {} macro-generated, {} without a span, {} unreadable, {} outside 5..=120 lines",
c.fns.len(),
c.callables,
t.elapsed().as_millis(),
c.macro_generated,
c.without_span,
c.unreadable,
c.outside_size
);
let mut lost: Vec<(&String, &usize)> = c.unreadable_files.iter().collect();
lost.sort_by(|a, b| b.1.cmp(a.1));
println!("unreadable, by file (top 8 of {}): {:?}", lost.len(), &lost[..lost.len().min(8)]);
let fns = c.fns;
assert!(fns.len() > 80, "corpus too small to calibrate on: {} fns", fns.len());
// whole functions: the text holds the item's own name, and it is one item
let named = fns.iter().filter(|f| f.text.contains(&format!("fn {}", f.name))).count();
println!("functions whose text holds their own `fn <name>`: {named} of {}", fns.len());
let (pos, neg) = measure_populations(&fns, 200, 400, 0x9e37, bag_embed).unwrap();
let mut p = provenance("bag-of-tokens");
p.corpus = format!("{repo}, {} fns (code room spans)", fns.len());
let cal = Calibration::compute(p, pos, neg, 1.0);
println!("{}", cal.report());
}
}
// ---- the synthesis itself has to actually change the text ----
#[test]
fn type2_renames_locals_and_literals_but_keeps_calls_and_types() {
let src = "fn add_one(value: u32) -> u32 {\n let total = value + 1;\n helper(total);\n String::from(\"hi\")\n}";
let out = synth_type2(src);
assert_ne!(out, src, "Type-2 transform was a no-op — the population would be Type-1 under a false label");
assert!(!out.contains("value"), "parameter not renamed: {out}");
assert!(!out.contains("total"), "local not renamed: {out}");
assert!(out.contains("helper("), "call target must survive: {out}");
assert!(out.contains("String::from"), "path must survive: {out}");
assert!(out.contains("\"hi_z\""), "string literal not perturbed: {out}");
assert!(out.contains("+ 2"), "integer literal not perturbed: {out}");
}
#[test]
fn type3_inserts_statements_and_removes_comments() {
let src = "fn f() {\n // a comment\n let x = 1;\n}";
let out = synth_type3(src, 3);
assert!(!out.contains("// a comment"), "comment not removed: {out}");
assert_eq!(
out.lines().count(),
src.lines().count() - 1 + 3,
"wrong line delta:\n{out}"
);
}
// ---- mode detection: red before green ----
#[test]
fn mode_detector_separates_unimodal_from_bimodal() {
let uni: Vec<f32> = (0..400).map(|i| 0.70 + (i % 20) as f32 * 0.002).collect();
let d = CosineDistribution::measure("uni", &uni);
assert_eq!(d.modes, 1, "unimodal population read as {} modes", d.modes);
assert!(d.is_unimodal());
// Type-1 at 1.0, Type-3 near 0.65 — the exact shape the brief warns of.
let mut bi: Vec<f32> = (0..200).map(|_| 1.0f32).collect();
bi.extend((0..200).map(|i| 0.63 + (i % 10) as f32 * 0.002));
let d = CosineDistribution::measure("bi", &bi);
assert_eq!(d.modes, 2, "bimodal population read as {} modes:\n{}", d.modes, d.sparkline());
assert!(!d.is_unimodal());
}
#[test]
fn degenerate_population_is_not_mistaken_for_a_tight_one() {
let d = CosineDistribution::measure("const", &vec![1.0f32; 500]);
assert!(d.is_degenerate(), "σ={} read as real spread", d.sd);
assert_eq!(d.mean, 1.0);
// A tight-but-real population must NOT trip the same wire.
let real: Vec<f32> = (0..500).map(|i| 0.95 + (i % 50) as f32 * 0.0005).collect();
assert!(!CosineDistribution::measure("real", &real).is_degenerate());
}
#[test]
fn roc_auc_is_a_coin_on_identical_populations_and_one_on_separated_ones() {
let a: Vec<f32> = (0..200).map(|i| i as f32 / 200.0).collect();
assert!((roc_auc(&a, &a) - 0.5).abs() < 0.01, "{}", roc_auc(&a, &a));
let hi: Vec<f32> = (0..200).map(|i| 0.9 + i as f32 / 4000.0).collect();
let lo: Vec<f32> = (0..200).map(|i| 0.1 + i as f32 / 4000.0).collect();
assert!((roc_auc(&hi, &lo) - 1.0).abs() < 1e-6);
// all-ties (the constant embedder) must score exactly a coin
assert!((roc_auc(&vec![1.0; 100], &vec![1.0; 100]) - 0.5).abs() < 1e-6);
}
// ---- the chunk artefact ----
#[test]
fn same_chunk_pairs_are_refused_through_the_shared_predicate() {
use crate::mimir_analyze::SymbolVec;
let mk = |id: &str, chunk: &str| SymbolVec {
id: id.into(),
repo: "r".into(),
crate_name: "c".into(),
module_path: "m".into(),
item_kind: "fn".into(),
item_name: id.into(),
file: "f.rs".into(),
line: 1,
signature: None,
callees: Default::default(),
span_lines: 40,
chunk_hash: chunk.into(),
impl_header: None,
item_end_line: 40,
vector: vec![1.0, 0.0],
};
let pairs = vec![
(mk("a", "r\u{1}f.rs\u{1}1-120"), mk("b", "r\u{1}f.rs\u{1}1-120")), // one chunk
(mk("c", "r\u{1}f.rs\u{1}1-120"), mk("d", "r\u{1}g.rs\u{1}1-120")), // different
(mk("e", ""), mk("f", "")), // no provenance
];
let (keep, dropped) = filter_same_chunk(&pairs);
assert_eq!(dropped, 1, "same-chunk pair not dropped");
assert_eq!(keep, vec![1, 2], "wrong survivors: {keep:?}");
}
// ---- GREEN: a working embedder calibrates ----
// ---- RED: the dead embedders must be caught ----
#[test]
fn empty_populations_are_refused_rather_than_read_as_perfect() {
let empty = PopulationSet::new("positives", vec![CosineDistribution::measure("type-1", &[])]);
let neg = PopulationSet::new(
"negatives",
vec![CosineDistribution::measure(HARD_NEGATIVE, &[0.3, 0.4])],
);
let c = Calibration::compute(provenance("empty"), empty, neg, 1.0);
assert!(!c.verdict.is_shippable(), "an EMPTY measurement was certified: {:?}", c.verdict);
assert!(c.note().contains("EMPTY"), "silent empty: {}", c.note());
}
#[test]
fn a_floor_inside_the_negatives_is_refused() {
// Unimodal, well separated on the means, but k is large enough to drag
// the floor down into the hard negatives.
let pos: Vec<f32> = (0..400).map(|i| 0.80 + (i % 40) as f32 * 0.005).collect();
let hard: Vec<f32> = (0..400).map(|i| 0.60 + (i % 40) as f32 * 0.005).collect();
let p = PopulationSet::new("positives", vec![CosineDistribution::measure("type-2", &pos)]);
let n = PopulationSet::new("negatives", vec![CosineDistribution::measure(HARD_NEGATIVE, &hard)]);
let tight = Calibration::compute(provenance("m"), p.clone_for_test(), n.clone_for_test(), 0.5);
assert!(tight.verdict.is_shippable(), "k=0.5 should be fine here: {:?}", tight.verdict);
let loose = Calibration::compute(provenance("m"), p, n, 4.0);
assert!(
matches!(loose.verdict, Verdict::FloorInsideNegatives { .. }),
"k=4.0 puts the floor at {:.3} inside a negative population topping out at {:.3}, \
and that was accepted: {:?}",
loose.cosine_floor,
hard.iter().cloned().fold(f32::MIN, f32::max),
loose.verdict
);
}
#[test]
fn bimodal_positives_get_per_type_floors_not_one_number() {
let t1: Vec<f32> = (0..200).map(|_| 1.0f32).collect();
let t3: Vec<f32> = (0..200).map(|i| 0.60 + (i % 20) as f32 * 0.002).collect();
let p = PopulationSet::new(
"positives",
vec![
CosineDistribution::measure("type-1", &t1),
CosineDistribution::measure("type-3", &t3),
],
);
let n = PopulationSet::new(
"negatives",
vec![CosineDistribution::measure(
HARD_NEGATIVE,
&(0..200).map(|i| 0.20 + (i % 20) as f32 * 0.002).collect::<Vec<f32>>(),
)],
);
// k = 0.5 puts the pooled floor squarely in the VALLEY between the two
// peaks — a stretch of the cosine axis where not one measured clone
// lives. (At k = 1.0 it would land exactly ON the lower peak, but that
// is arithmetic, not calibration: for two equal-mass point clouds at a
// and b, μ − 1σ == b identically. A number that is right by algebraic
// accident is exactly the kind this module refuses to ship.)
let c = Calibration::compute(provenance("m"), p, n, 0.5);
match &c.verdict {
Verdict::PerTypeRequired { modes, per_type_floor } => {
assert_eq!(*modes, 2);
assert_eq!(per_type_floor.len(), 2);
// APPLIED: the pooled floor keeps ALL of one type and NONE of
// the other, so it describes neither.
let t1 = c.positives.get("type-1").unwrap().survival(c.cosine_floor);
let t3 = c.positives.get("type-3").unwrap().survival(c.cosine_floor);
assert_eq!(t1, 1.0, "pooled floor {:.3} dropped Type-1", c.cosine_floor);
assert_eq!(
t3, 0.0,
"pooled floor {:.3} was supposed to sit ABOVE every Type-3 pair (max {:.3})",
c.cosine_floor,
c.positives.get("type-3").unwrap().max
);
// The conservative (weakest-type) floor is the one that keeps them.
let t3c = c.positives.get("type-3").unwrap().survival(c.conservative_floor);
assert!(
t3c > 0.6,
"conservative floor {:.3} keeps only {:.0}% of Type-3",
c.conservative_floor,
t3c * 100.0
);
}
v => panic!("bimodal positives accepted as one threshold: {v:?}"),
}
assert!(c.note().contains("MODAL"), "note hides the shape: {}", c.note());
}
/// A stand-in record with every field filled — the shape a real run writes
/// into [`RECORDED`].
static REC: RecordedCalibration = RecordedCalibration {
model_id: "jina-v2-base-code",
weights_sha: "63363fc178428b74620c6f3780cbc7191883fa5c7f84c0945c45eb5c4256733b",
weights_bytes: 641_517_466,
dtype: "f32",
dim: 768,
corpus: "nornir/src, 3215 fns over 199 files",
measured_at: "2026-08-25",
k: 1.0,
pooled_floor: 0.811,
conservative_floor: 0.761,
clone_cosine_floor: 0.903,
dup_cosine: 0.96,
auc: 0.918,
recall: 0.519,
hard_negative_survival: 0.16,
positive_modes: 1,
per_type_floor: &[("type-1", 1.0), ("type-2", 0.761), ("type-3", 0.804)],
survivors: &[(0.80, 138), (0.903, 21)],
verdict: "floor_inside_negatives",
floor_method: "hard-negative 80th percentile",
};
/// A record is only reachable through the identity of the weights on disk,
/// and a space nobody measured must resolve to NOTHING rather than to the
/// nearest neighbour's number.
#[test]
fn an_unmeasured_space_gets_no_operating_point() {
const TABLE: &[RecordedCalibration] = &[];
assert!(TABLE.iter().find(|r| r.weights_bytes == 641_517_466).is_none());
// and the real lookup over the real table never invents one
assert!(recorded_for_bytes(0).is_none());
assert!(
recorded_for_bytes(161_895_621).is_none() || RECORDED.iter().any(|r| r.weights_bytes == 161_895_621),
"recorded_for_bytes returned a record that is not in RECORDED"
);
}
/// THE POINT OF THE WHOLE EXERCISE: when a floor has been measured, the
/// evidence note must stop saying it is a guess — and say what it was
/// measured against instead.
#[test]
fn a_recorded_calibration_replaces_the_uncalibrated_sentence() {
use crate::mimir_clones::{CloneThresholds, analyze_semantic_clones};
// RED half — the shipped default still calls itself a guess, in words.
let d = analyze_semantic_clones(&[], &CloneThresholds::default(), "", 8, Vec::new());
assert!(
d.evidence_note.contains("HEURISTIC and UNCALIBRATED"),
"the default note stopped admitting it is uncalibrated: {}",
d.evidence_note
);
// GREEN half — with a record attached it names the space, the corpus,
// the date, the separation and the survivor counts.
let t = CloneThresholds {
cosine_floor: REC.clone_cosine_floor,
calibration: Some(&REC),
..Default::default()
};
let c = analyze_semantic_clones(&[], &t, "", 8, Vec::new());
assert!(
!c.evidence_note.contains("UNCALIBRATED"),
"a MEASURED floor is still advertised as uncalibrated: {}",
c.evidence_note
);
for needle in [
"is CALIBRATED, not heuristic",
"2026-08-25",
"jina-v2-base-code",
"63363fc17842",
"641517466",
"nornir/src",
"ROC-AUC",
"hard negatives",
"0.80\u{2192}138",
] {
assert!(
c.evidence_note.contains(needle),
"calibrated note is missing `{needle}`:\n{}",
c.evidence_note
);
}
// The floor the note quotes must be the floor the lens ran with — a note
// that describes a different number than the gate used is worse than none.
assert!(
c.evidence_note.contains("Cosine floor 0.903"),
"note quotes a floor the gate did not use: {}",
c.evidence_note
);
}
/// The one row the table actually ships must resolve, and its note must
/// carry the numbers a reader would need to disbelieve it.
#[test]
fn the_shipped_fp32_operating_point_resolves_and_states_its_own_refusal() {
let r = recorded_for_bytes(641_517_466)
.expect("the fp32 jina space has a recorded operating point");
assert_eq!(r.model_id, "jina-v2-base-code");
assert_eq!(r.dtype, "f32");
// The floor is NOT mu-k*sigma, and the record says so rather than
// implying a derivation it does not have.
assert_eq!(r.verdict, "floor_inside_negatives");
assert!(
(r.clone_cosine_floor - r.pooled_floor).abs() > 0.05,
"the shipped floor {:.3} is suspiciously close to the refused mu-k*sigma floor {:.3}",
r.clone_cosine_floor,
r.pooled_floor
);
let n = r.note();
assert!(n.contains("floor_inside_negatives"), "refusal hidden: {n}");
assert!(n.contains("80th"), "derivation hidden: {n}");
// The SURVIVOR count is the number that decides whether this is a fix.
let at_shipped = r.survivors.iter().find(|(f, _)| (*f - 0.80).abs() < 1e-6).unwrap().1;
let at_calibrated = r
.survivors
.iter()
.find(|(f, _)| (*f - r.clone_cosine_floor).abs() < 1e-6)
.expect("the recorded floor must have a measured survivor count")
.1;
assert!(
at_calibrated * 10 < at_shipped,
"the calibrated floor left {at_calibrated} clusters against {at_shipped} at 0.80 — \
less than an order of magnitude is not an actionable change"
);
assert!(
at_calibrated > 0,
"a floor that leaves ZERO clusters has not been calibrated, it has been switched off"
);
}
/// **The wiring, end to end.** `CloneThresholds::calibrated()` must actually reach
/// the loaded weights, pick the recorded floor, and put the provenance on
/// the note the lens returns — not merely compile.
///
/// Conditional on the box: when the weights this build would load are a
/// space with a recorded operating point, the calibrated path MUST use it;
/// when they are not, it must fall back to the labelled heuristic. Both arms
/// are asserted, so the test is not silently vacuous on a box without the
/// model.
#[cfg(any(feature = "embed-tract", feature = "embed-ort"))]
#[test]
fn calibrated_opts_reach_the_loaded_weights_and_carry_the_provenance() {
use crate::mimir_clones::{CloneThresholds, analyze_semantic_clones};
let space = active();
let thresholds = CloneThresholds::calibrated();
let rep = analyze_semantic_clones(&[], &thresholds, "", 8, Vec::new());
match recorded_for_bytes(space.weights_bytes) {
Some(rec) => {
assert_eq!(
thresholds.cosine_floor, rec.clone_cosine_floor,
"the calibrated opts did not take the recorded floor for {} ({} bytes)",
space.model_id, space.weights_bytes
);
assert!(
thresholds.cosine_floor != CloneThresholds::default().cosine_floor,
"the calibrated floor is identical to the heuristic one — this test could \
not tell a working lookup from a dead one"
);
assert!(
!rep.evidence_note.contains("UNCALIBRATED"),
"calibrated run still advertises itself as uncalibrated: {}",
rep.evidence_note
);
assert!(
rep.evidence_note.contains(rec.measured_at),
"note carries no measurement date: {}",
rep.evidence_note
);
}
None => {
assert_eq!(
thresholds.cosine_floor,
CloneThresholds::default().cosine_floor,
"an UNMEASURED space ({}, {} bytes) was handed a calibrated floor",
space.model_id,
space.weights_bytes
);
assert!(
rep.evidence_note.contains("HEURISTIC and UNCALIBRATED"),
"an unmeasured space stopped saying so: {}",
rep.evidence_note
);
}
}
}
/// The lookup key must be unique, or two spaces silently share one floor.
#[test]
fn no_two_recorded_spaces_share_a_lookup_key() {
let mut seen = BTreeSet::new();
for r in RECORDED {
assert!(
seen.insert(r.weights_bytes),
"two records share weights_bytes {} — recorded_for_bytes would return \
whichever comes first",
r.weights_bytes
);
assert_eq!(r.weights_sha.len(), 64, "{} has no full sha256", r.model_id);
}
}
/// A multi-modal record must SAY so in the note, and name the per-type
/// floors, instead of printing one number that describes no clone type.
#[test]
fn a_multimodal_record_refuses_to_print_a_single_number_silently() {
let mut rec = REC;
rec.positive_modes = 3;
let n = rec.note();
assert!(n.contains("3-MODAL"), "shape hidden: {n}");
assert!(n.contains("type-3=0.804"), "per-type floors hidden: {n}");
assert!(!REC.note().contains("MODAL"), "unimodal record should not claim modality");
}
// small helper so the two-arm test can reuse a population set
impl PopulationSet {
fn clone_for_test(&self) -> PopulationSet {
PopulationSet {
by_type: self.by_type.clone(),
pooled: self.pooled.clone(),
}
}
}
}