crates/gunnar-store/src/conformance/red.rs
//! Measure, never assert, made a test: **the conformance suite is seen to go red.**
//!
//! Every backend passes the suite. That is worth exactly nothing until the suite
//! has been shown to catch a backend that is wrong, so the wrappers below break
//! one in five specific, realistic ways and run the *same* suite over the
//! result. Each test names the checks that must fail and, just as importantly,
//! the checks that must still pass, because a suite that fails everything is as
//! blind as one that passes everything.
//!
//! The five breaks are not arbitrary. Each is a bug someone would plausibly
//! ship, and three of them are bugs that *have* shipped in this tree's history:
//!
//! | break | the plausible bug |
//! |---|---|
//! | `Cas` | `update_ref` that applies without consulting `old` — every push wins |
//! | `UnderSend` | `reachable` that contributes only the tips: the clone exits zero and is short by everything beneath them |
//! | `StoredSize` | `size` that reports the stored extent instead of the post-resolution size — the quota gate that can never fire |
//! | `Truncate` | `get` that returns not quite the bytes it stored |
//! | `Regenerate` | `get_or_create` that always creates: the SSH host key changes on every boot |
//! | `HeadIsARow` | `HEAD` advertised in the ref stream, `head()` answering nothing — **the disagreement the two arms actually had**, and the one the suite could not see until 2026-08-10 |
//! | `PackCloses` | `emit_pack` treating its object set as tips and closing over the graph — Agent K found this live: one commit in, four objects out on a `--filter=tree:0` fetch |
//!
//! **`UnderSend` is the one that matters most.** It passes every object check,
//! every ref check, `gc` and the restart, because nothing else in the suite asks
//! the graph a question. It is also the failure mode with no external symptom —
//! *"the clone ends with a pack the client cannot close, or worse, closes against
//! objects it happened to have"* — so a suite without a genuine closure
//! assertion would call it green and a fetch would call it done.
//!
//! **`PackCloses` is the sharp one's serving twin.** The over-sent pack passes
//! `git index-pack --strict` **and** `git fsck`, the clone succeeds, and the
//! client receives objects it explicitly asked not to have — so no exit code and
//! no format check anywhere in this tree can see it. Only an assertion on what
//! came out can, which is why `emit_pack_emits_the_set_it_was_given` asserts
//! membership rather than validity.
//!
//! **`HeadIsARow` is the one that proves the newest check is live.** Before it,
//! the two `GitOps` arms disagreed about whether `HEAD` is a row for as long as
//! there were two arms, and this suite was **incapable of noticing**, because it
//! never exercised `HEAD` at all. A check added for a defect nobody can
//! reproduce is a check nobody can trust; this reproduces it.
//!
//! **`StoredSize` is the quiet one.** The number it corrupts is read by exactly
//! one caller, an object-size ceiling, and on every corpus in this repository
//! that ceiling never fires — so the bug is invisible to every wire-byte and
//! object-count test and shows up in production as an oversized object served.
//! It takes a fixture with a delta in it to see at all.
//!
//! The wrappers are generic over any [`Harness`], which buys the other half of
//! the argument: breaking a *real* harness and watching the same checks go red
//! proves that green run is live code and not a harness that quietly does
//! nothing. Since 2026-08-10 the only harness in this crate is the control, and
//! the real ones are elsewhere — `gunnar-core`'s Arrow-log harness reconstructs
//! `Regenerate` locally, and both `GitOps` backends should do the same in their
//! own repositories.
use std::collections::BTreeSet;
use git_storage_trait::{
Caps, Extent, GcReport, GitOps, GitServe, ObjType, Oid, PackStats, ReachSet, RefCas, RefRow,
RefUpdate, Stored, TxId,
};
use super::{run_all, Backend, Harness, Outcome};
use crate::error::Result;
use crate::secret::SecretStore;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Break {
/// Nothing broken. Proves the wrappers themselves are transparent.
None,
/// `update_ref` ignores the `old` it was given.
Cas,
/// `reachable` contributes the tips and nothing beneath them.
UnderSend,
/// `size` and `Stored::uncompressed_size` report the stored entry length.
StoredSize,
/// `get` returns one byte less than was stored.
Truncate,
/// `get_or_create` always creates.
Regenerate,
/// `HEAD` is a row in `refs()` and `head()` cannot answer.
HeadIsARow,
/// `emit_pack` closes over the graph instead of emitting the set it was
/// given.
PackCloses,
/// `reachable` answers a non-commit `want` with its own id and stops, so an
/// annotated tag arrives without the commit it names.
TagStopsAtItself,
/// `put_pack_observed` declares that it publishes a live object count and
/// then publishes `(0, N)` and `(N, N)` — the two endpoints, and nothing
/// between them.
TwoValueCount,
}
struct BrokenGit<G> {
inner: G,
brk: Break,
}
// `GitServe` and not `GitOps`, because two of the breaks are about the reading
// half and one of them — `HeadIsARow` — has to reach across: the bug it models
// moves `HEAD` from the accessor into the row stream, so the `refs()` override
// asks `head()`.
impl<G: GitServe> GitOps for BrokenGit<G> {
fn put(&self, pack: &[u8], refs: &[RefUpdate]) -> anyhow::Result<TxId> {
self.inner.put(pack, refs)
}
fn put_pack(&self, bytes: &[u8]) -> anyhow::Result<TxId> {
self.inner.put_pack(bytes)
}
/// **The gap that was shipped, wrapped in a struct.**
///
/// This is not a hypothetical break: it is the literal behaviour every
/// engine had until 2026-09-08 — `(0, N)` at the packfile header, `(N, N)`
/// when the ingest returned, nothing in between — dressed in a
/// `publishes_pack_progress()` that says `true`. A console watching it sees
/// a bar that jumps from empty to full and learns nothing during the
/// minutes that matter.
///
/// It reads the object count out of the pack's own header (bytes 8..12)
/// rather than counting anything, which is the point: **a fabricated series
/// costs nothing to produce and is indistinguishable from a real one at
/// every seam except this check.** Twelve bytes is also all it may read —
/// a break that walked the pack to fake the numbers would be a second pack
/// decoder, which is the thing the real implementation exists to avoid.
fn put_pack_observed(
&self,
bytes: &[u8],
progress: &dyn git_storage_trait::PackProgress,
) -> anyhow::Result<TxId> {
if self.brk != Break::TwoValueCount {
return self.inner.put_pack_observed(bytes, progress);
}
let total = u64::from(u32::from_be_bytes([
bytes[8], bytes[9], bytes[10], bytes[11],
]));
progress.indexed(0, total);
let tx = self.inner.put_pack(bytes)?;
progress.indexed(total, total);
Ok(tx)
}
fn publishes_pack_progress(&self) -> bool {
self.brk == Break::TwoValueCount || self.inner.publishes_pack_progress()
}
fn put_refs(&self, updates: &[RefUpdate]) -> anyhow::Result<TxId> {
self.inner.put_refs(updates)
}
fn get(&self, oid: Oid<'_>) -> anyhow::Result<Option<Stored>> {
let Some(mut stored) = self.inner.get(oid)? else {
return Ok(None);
};
match self.brk {
Break::Truncate => {
stored.bytes.pop();
}
// The bug: the *stored* length, which for a delta is the length of
// the instructions rather than of what they resolve to.
Break::StoredSize => stored.uncompressed_size = stored.bytes.len() as u64,
_ => {}
}
Ok(Some(stored))
}
fn has(&self, oid: Oid<'_>) -> anyhow::Result<bool> {
self.inner.has(oid)
}
fn size(&self, oid: Oid<'_>) -> anyhow::Result<Option<u64>> {
if self.brk == Break::StoredSize {
return Ok(self.inner.get(oid)?.map(|s| s.bytes.len() as u64));
}
self.inner.size(oid)
}
fn extents(&self, oids: &[Oid<'_>]) -> anyhow::Result<Vec<Option<Extent>>> {
self.inner.extents(oids)
}
fn refs(&self) -> anyhow::Result<Vec<RefRow>> {
let mut rows = self.inner.refs()?;
if self.brk == Break::HeadIsARow {
// The bug: the pseudo-ref in the row stream. It is the shape the
// Arrow arm actually had — `HEAD` was written to the ref log like
// any other row and read back out with the rest of them.
if let Some(head) = self.inner.head()? {
rows.push(head);
rows.sort_by(|a, b| a.name.cmp(&b.name));
}
}
Ok(rows)
}
fn update_ref(
&self,
name: &str,
old: Option<Oid<'_>>,
new: Option<Oid<'_>>,
) -> anyhow::Result<TxId> {
if self.brk == Break::Cas {
// The bug: `old` is never consulted. Whatever is there is accepted
// as what was expected, so every push wins.
let current = self
.inner
.refs()?
.into_iter()
.find(|r| r.name == name)
.and_then(|r| r.oid);
return self.inner.update_ref(name, current.as_deref(), new);
}
self.inner.update_ref(name, old, new)
}
fn reachable(&self, want: &[Oid<'_>], have: &[Oid<'_>]) -> anyhow::Result<Vec<Vec<u8>>> {
if self.brk == Break::UnderSend {
// The bug: a tip whose closure the index does not actually hold,
// contributing only itself. Deduplicated, because an answer with a
// repeat in it would be caught by something other than the bug.
let tips: BTreeSet<Vec<u8>> = want.iter().map(|id| id.to_vec()).collect();
return Ok(tips.into_iter().collect());
}
if self.brk == Break::TagStopsAtItself {
// **The bug the appliance shipped**, modelled exactly: the store's
// reachability index is built over the commit graph, so a `want`
// that is a commit gets its whole closure and a `want` that is not
// — an annotated tag — contributes its own id and nothing else. A
// tree never points at a tag, so no other tip picks it up either.
//
// Every commit still answers correctly, which is the whole reason
// this went unnoticed: the four checks around it stay green, six of
// the reference appliance's 25 repositories could not obtain a second copy,
// and `git clone` worked throughout.
let mut commits: Vec<Oid<'_>> = Vec::new();
let mut out: BTreeSet<Vec<u8>> = BTreeSet::new();
for id in want {
match self.inner.header(id)? {
Some((ObjType::Commit, _)) => commits.push(id),
_ => {
out.insert(id.to_vec());
}
}
}
out.extend(self.inner.reachable(&commits, have)?);
return Ok(out.into_iter().collect());
}
self.inner.reachable(want, have)
}
fn put_refs_cas(&self, edits: &[RefCas<'_>]) -> anyhow::Result<TxId> {
if self.brk == Break::Cas {
// The same bug as `update_ref`'s, in the batch shape: the caller's
// expectation is replaced by whatever is there, so no edit can lose.
let held = self.inner.refs()?;
let current: Vec<Option<Vec<u8>>> = edits
.iter()
.map(|e| {
held.iter()
.find(|r| r.name == e.name)
.and_then(|r| r.oid.clone())
})
.collect();
let rewritten: Vec<RefCas<'_>> = edits
.iter()
.zip(¤t)
.map(|(e, current)| RefCas {
name: e.name.clone(),
old: current.as_deref(),
new: e.new,
peeled: e.peeled.clone(),
})
.collect();
return self.inner.put_refs_cas(&rewritten);
}
self.inner.put_refs_cas(edits)
}
fn gc(&self) -> anyhow::Result<GcReport> {
self.inner.gc()
}
}
impl<G: GitServe> GitServe for BrokenGit<G> {
fn read(&self, oid: Oid<'_>) -> anyhow::Result<Option<(ObjType, Vec<u8>)>> {
self.inner.read(oid)
}
fn header(&self, oid: Oid<'_>) -> anyhow::Result<Option<(ObjType, u64)>> {
self.inner.header(oid)
}
fn sizes(&self, oids: &[Oid<'_>]) -> anyhow::Result<Vec<Option<u64>>> {
self.inner.sizes(oids)
}
fn head(&self) -> anyhow::Result<Option<RefRow>> {
if self.brk == Break::HeadIsARow {
// The other half of the same bug, and the half that makes it fatal:
// a backend that publishes HEAD as a row has no reason to answer
// the accessor, so the one caller that needs HEAD — the
// advertisement — gets nothing.
return Ok(None);
}
self.inner.head()
}
fn set_head(&self, target: &str) -> anyhow::Result<TxId> {
self.inner.set_head(target)
}
fn emit_pack(
&self,
objects: &[Oid<'_>],
have: &[Oid<'_>],
caps: &Caps,
out: &mut dyn std::io::Write,
) -> anyhow::Result<PackStats> {
if self.brk == Break::PackCloses {
// The bug: treat the exact set as tips and close over it. Every
// object emitted is reachable, the pack is valid, `fsck` is happy,
// and the client receives what its filter excluded.
let closure = self.inner.reachable(objects, have)?;
let refs: Vec<Oid<'_>> = closure.iter().map(|id| id.as_slice()).collect();
return self.inner.emit_pack(&refs, have, caps, out);
}
self.inner.emit_pack(objects, have, caps, out)
}
fn select(&self, want: &[Oid<'_>], have: &[Oid<'_>]) -> anyhow::Result<Option<ReachSet>> {
self.inner.select(want, have)
}
}
struct BrokenSecrets<S> {
inner: S,
brk: Break,
}
impl<S: SecretStore> SecretStore for BrokenSecrets<S> {
fn get(&self, key: &str) -> Result<Option<Vec<u8>>> {
self.inner.get(key)
}
fn put(&self, key: &str, value: &[u8]) -> Result<()> {
self.inner.put(key, value)
}
fn delete(&self, key: &str) -> Result<()> {
self.inner.delete(key)
}
fn get_or_create(&self, key: &str, mk: &mut dyn FnMut() -> Result<Vec<u8>>) -> Result<Vec<u8>> {
if self.brk == Break::Regenerate {
// The bug: a fresh host key on every call, and therefore on every
// boot. This is the exact failure the secret-store seam exists to prevent.
let value = mk()?;
self.inner.put(key, &value)?;
return Ok(value);
}
self.inner.get_or_create(key, mk)
}
}
/// Any harness, with one thing about it broken.
struct BrokenHarness<H> {
inner: H,
brk: Break,
}
impl<H: Harness> Harness for BrokenHarness<H> {
type Git = BrokenGit<H::Git>;
type Secrets = BrokenSecrets<H::Secrets>;
fn name(&self) -> &str {
self.inner.name()
}
fn open(
&self,
slot: &str,
hash_kind: gix_hash::Kind,
) -> anyhow::Result<Backend<Self::Git, Self::Secrets>> {
let backend = self.inner.open(slot, hash_kind)?;
Ok(Backend {
git: BrokenGit {
inner: backend.git,
brk: self.brk,
},
secrets: BrokenSecrets {
inner: backend.secrets,
brk: self.brk,
},
})
}
}
fn red_checks<H: Harness>(inner: H, brk: Break) -> (BTreeSet<&'static str>, Vec<Outcome>) {
let outcomes = run_all(&BrokenHarness { inner, brk });
let failed = outcomes
.iter()
.filter(|o| !o.passed())
.map(|o| o.check)
.collect();
(failed, outcomes)
}
#[track_caller]
fn assert_red(
backend: &str,
failed: &BTreeSet<&'static str>,
must_fail: &[&str],
must_pass: &[&str],
) {
for check in must_fail {
assert!(
failed.contains(check),
"[{backend}] the conformance suite did NOT catch a deliberately broken backend: \
`{check}` passed when it had to fail. Failing checks were {failed:?}"
);
}
for check in must_pass {
assert!(
!failed.contains(check),
"[{backend}] `{check}` failed against a backend that is not broken in that way. \
the suite is failing indiscriminately, which is its own kind of blindness. \
Failing checks were {failed:?}"
);
}
}
const CAS_MUST_FAIL: &[&str] = &[
"update_ref_creates_only_when_absent",
"update_ref_is_a_real_compare_and_swap",
"update_ref_delete_checks_its_old_value",
"put_refs_cas_applies_every_edit_or_none",
];
const CAS_MUST_PASS: &[&str] = &[
"pack_roundtrip_every_entry",
"refs_round_trip",
"reachable_returns_the_whole_closure",
"secret_get_or_create_survives_restart",
];
#[test]
fn the_wrappers_themselves_are_transparent() {
// Without this, a red result below could be an artefact of the wrapper
// rather than of the break it is supposed to model.
let (failed, outcomes) = red_checks(super::MemHarness::default(), Break::None);
assert!(
failed.is_empty(),
"the pass-through broken-backend wrapper does not itself conform: {failed:?}"
);
assert!(!outcomes.is_empty(), "the suite ran nothing");
}
#[test]
fn the_suite_catches_a_broken_compare_and_swap() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::Cas);
assert_red("mem", &failed, CAS_MUST_FAIL, CAS_MUST_PASS);
}
/// **The sharp one.** Every object check, every ref check, `gc` and the restart
/// pass: nothing but the closure assertions can see this bug, and a clone served
/// by it exits zero.
#[test]
fn the_suite_catches_a_reachability_index_that_under_sends() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::UnderSend);
assert_red(
"mem",
&failed,
&[
"reachable_returns_the_whole_closure",
"reachable_subtracts_what_the_client_has",
"reachable_tolerates_a_have_it_does_not_know",
"reachable_does_not_follow_a_submodule",
],
&[
"pack_roundtrip_every_entry",
"get_returns_the_verbatim_entry_bytes",
"update_ref_is_a_real_compare_and_swap",
"gc_keeps_everything_a_ref_points_at",
"the_store_survives_a_restart",
"secret_put_get_delete",
],
);
}
/// **The one that was shipping.** `reachable` is right about every commit and
/// wrong about exactly one edge, and the other three closure checks cannot see
/// it because none of them puts a tag in a `want`.
///
/// 🔴 This is the reference appliance on 2026-08-31, in the suite: six of 25
/// repositories refused replication with `missing necessary objects: <commit>
/// is not in this repository (peeling a tag)`, while clones of the same
/// repositories succeeded. `reachable_follows_an_annotated_tag_to_its_target`
/// is the check that had to exist for a suite to call that backend broken, and
/// this is the proof it can.
#[test]
fn the_suite_catches_a_reachability_that_stops_at_a_tag() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::TagStopsAtItself);
assert_red(
"mem",
&failed,
&["reachable_follows_an_annotated_tag_to_its_target"],
&[
// The three that were green on the appliance the whole time it was
// failing. If any of these goes red the new check is not what
// caught the break, and the suite is failing indiscriminately.
"reachable_returns_the_whole_closure",
"reachable_subtracts_what_the_client_has",
"reachable_does_not_follow_a_submodule",
"pack_roundtrip_every_entry",
"update_ref_is_a_real_compare_and_swap",
"gc_keeps_everything_a_ref_points_at",
"the_store_survives_a_restart",
],
);
}
/// **The quiet one.** Only a fixture with a delta in it can see this, which is
/// why the fixture has one.
#[test]
fn the_suite_catches_a_size_that_is_not_post_resolution() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::StoredSize);
assert_red(
"mem",
&failed,
&["size_is_post_resolution_not_stored_length"],
&[
"get_returns_the_verbatim_entry_bytes",
"update_ref_is_a_real_compare_and_swap",
"reachable_returns_the_whole_closure",
"secret_put_get_delete",
],
);
}
#[test]
fn the_suite_catches_a_store_that_loses_bytes() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::Truncate);
assert_red(
"mem",
&failed,
&["get_returns_the_verbatim_entry_bytes"],
&[
"update_ref_is_a_real_compare_and_swap",
"reachable_returns_the_whole_closure",
"secret_put_get_delete",
],
);
}
/// **The check that did not exist until 2026-08-10, seen red.**
///
/// `HEAD` in the row stream and `head()` answering nothing is not a
/// hypothetical: it is what one of the two arms did, and the reason it survived
/// is that the suite never asked. This is the proof that asking works.
#[test]
fn the_suite_catches_a_backend_that_calls_head_a_row() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::HeadIsARow);
assert_red(
"mem",
&failed,
&["head_is_not_a_row_and_has_its_own_accessor"],
&[
"refs_round_trip",
"update_ref_is_a_real_compare_and_swap",
"pack_roundtrip_every_entry",
"reachable_returns_the_whole_closure",
"emit_pack_emits_the_set_it_was_given",
],
);
}
/// **An `emit_pack` that closes over the graph, seen red.**
///
/// The pack it produces is valid in every mechanical sense — `index-pack
/// --strict` and `fsck` both pass it — so `pack_roundtrip_every_entry` and the
/// closure checks stay green and only the membership assertion fails. That is
/// the point: this bug has no other detector.
#[test]
fn the_suite_catches_an_emit_pack_that_closes_over_its_input() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::PackCloses);
assert_red(
"mem",
&failed,
&["emit_pack_emits_the_set_it_was_given"],
&[
"read_is_inflated_and_delta_resolved",
"header_and_sizes_agree_with_read",
"select_agrees_with_the_graph_or_declines",
"reachable_returns_the_whole_closure",
"pack_roundtrip_every_entry",
"head_is_not_a_row_and_has_its_own_accessor",
],
);
}
/// **The gap of 2026-09-08, seen red.**
///
/// 🔴 `mirror_queue`'s module doc said it in as many words: *"the progress pair
/// goes `(0, N) → (N, N)`"*. A mirror of a large repository published its
/// denominator, then nothing, then its numerator — and every other check in
/// this suite was green throughout, because storing a pack correctly and
/// **saying how far you have got while you store it** are different claims.
///
/// This break declares the second claim and delivers the first, which is the
/// only shape of this bug anybody would actually ship: nobody writes a progress
/// sink that lies about the numbers, they write one that is technically correct
/// at both ends and useless in between.
#[test]
fn the_suite_catches_a_count_that_only_has_two_values() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::TwoValueCount);
assert_red(
"mem",
&failed,
&["put_pack_observed_publishes_a_count_that_moves"],
&[
// The pack still lands, and every reading is inside the contract:
// monotone, right denominator, ending at N. Only "it moved" fails,
// which is the whole point — nothing else in this suite can see it.
"put_pack_is_idempotent",
"pack_roundtrip_every_entry",
"get_returns_the_verbatim_entry_bytes",
"update_ref_is_a_real_compare_and_swap",
"reachable_returns_the_whole_closure",
"the_store_survives_a_restart",
],
);
}
#[test]
fn the_suite_catches_a_regenerated_host_key() {
let (failed, _) = red_checks(super::MemHarness::default(), Break::Regenerate);
assert_red(
"mem",
&failed,
&[
"secret_get_or_create_survives_restart",
"secret_get_or_create_is_atomic",
],
&[
"secret_put_get_delete",
"update_ref_is_a_real_compare_and_swap",
"pack_roundtrip_every_entry",
],
);
}
// The two Arrow-log red tests moved to `gunnar-core` with the harnesses they
// break (`crates/gunnar-core/tests/arrow_log_conformance.rs`). They asserted
// that the Arrow log's green run is live code rather than a harness that
// quietly does nothing — which matters more there than it would for a database,
// because the log's atomicity comes from the IPC framing rather than from a
// commit, so "the frame makes it atomic" is exactly the claim that goes
// untested. That argument survived the move; `Break::Regenerate` is
// reconstructed there in twenty lines because the decorators here are private
// to this crate and widening them is a decision about the conformance suite's
// public surface, not about where the Arrow log lives.