crates/storage-git-gix/parked-gix-2/coldtier-archive/overlay.rs
//! The sealed namespace, **plus what has happened since**.
//!
//! [`crate::refstore::ArchiveRefStore`] answers `ls-remote` out of
//! `__gunnar_refs__` about ten times faster than `gix-ref` does, and it answers
//! it *as of the last `gc`*. A repository pushed to since then has branches the
//! archive has never heard of, tips it holds an older value for, and branches it
//! still advertises that were deleted. That is not a performance bug: a client
//! that fetches a tip the server no longer has gets a wrong answer, and a
//! `receive-pack` handed a stale old-oid in a compare-and-swap loses an update.
//!
//! This module is what makes the archive answerable. It is a **read** over two
//! tiers:
//!
//! | tier | what it is | who writes it |
//! |---|---|---|
//! | base | `__gunnar_refs__` in the archive, folded and sorted once at open | `gunnar gc`, through `refs_reserved_builder` |
//! | delta | the reference transactions the live log has recorded since the seal | `FsRefStore::transaction`, through `ArrowLogRefStore::journal` |
//!
//! **No write path is added to the archive.** The delta is a replay of a log
//! this module does not own and cannot append to, so [`RefStore::transaction`]
//! still refuses exactly as it did — see [`OverlayRefStore::transaction`]. That
//! boundary is deliberate and unchanged: fetch needs reads, push needs a
//! transaction, and the transaction belongs to the live store.
//!
//! # Which log the delta comes from, and why it is the right one
//!
//! gunnar's authority for a reference is `gix-ref`. `FsRefStore::transaction`
//! commits there and then calls `ArrowLogRefStore::journal` with the edits gix
//! **applied** — resolved previous value and all. So `gunnar-store/refs.arrow`
//! is a complete record of every reference move the server has performed, in
//! order, and it is already fsynced before the client is told the push
//! succeeded.
//!
//! Reading it here is O(the tail), not O(the log):
//! [`gunnar_store::history::read_live_since`] splices the stream's leading
//! schema message onto the bytes after the watermark, so the sealed prefix is
//! never parsed.
//!
//! # The watermark, and why the overlay refuses rather than guesses
//!
//! Applying *the whole journal* on top of the archive is nearly right and
//! sometimes catastrophically wrong. The journal's last entry for a reference is
//! older than the archive's value whenever a move did not reach the journal —
//! and `FsRefStore` deliberately allows exactly that: a journal write that fails
//! is logged and **does not fail the push**, because "history is worth less than
//! state". Replaying that journal over the archive would take the older value
//! and serve it.
//!
//! So the delta is bounded below by a watermark, and the watermark is
//! **verified at seal, not trusted**: `gunnar gc` takes a [`LiveMark`] on the
//! log, replays the log *up to* it, and writes the mark out only if that replay
//! agrees, reference for reference, with the namespace it sealed. A mismatch —
//! a push that raced the fold, a journal entry that was lost — leaves no
//! watermark, and no watermark means no overlay.
//!
//! Every failure mode of this module is the same failure mode: [`open`] returns
//! [`NotCurrent`], the caller falls back to the authoritative store, and the
//! only thing lost is speed. There is no path here that answers with a stale
//! reference, which is the property that had to hold before any of this could
//! be a default.
//!
//! # Asserting which tier answered
//!
//! Byte-comparing two advertisements cannot tell you which code produced them —
//! that is the whole point of a drop-in backend, and on 2026-08-04 it is what
//! let a lane measure the wrong path for a round. So the overlay counts, per
//! call, whether the answer came from the base, from the delta, or from a
//! delta tombstone: [`OverlayRefStore::from_base`],
//! [`OverlayRefStore::from_delta`], [`OverlayRefStore::tombstoned`]. Every test
//! below asserts on those as well as on the value, because "the ref has the
//! right value" and "the overlay is what gave it to you" are two claims and a
//! value comparison only makes one of them.
use std::collections::BTreeMap;
use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering};
use anyhow::{anyhow, Result};
use gunnar_store::history;
use gunnar_store::refs::{FullName, RefEdit, RefStore, Target};
use gunnar_store::RefTransaction;
use crate::refmark::RefMark;
use crate::refstore::ArchiveRefStore;
/// Why an overlay could not be built, when nothing is actually broken.
///
/// Every variant means the same thing to a caller — **fall back to the
/// authoritative store** — and they are distinguished only so that an operator
/// (and a test) can tell a repository that has never been sealed from one whose
/// log was compacted out from under its watermark. Returning the archive alone
/// in any of these cases would be serving a stale namespace, which is the
/// defect this module exists to close.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NotCurrent {
/// The archive carries no `__gunnar_refs__` at all — it predates refs being
/// sealed, or it was written by `append` rather than by `gc`.
NoRefsSection,
/// No watermark beside the archive. Either `gc` has not run since this
/// module existed, or it ran and **refused** to write one because the log
/// it replayed disagreed with the namespace it sealed.
NoWatermark,
/// A watermark is there and it describes a different archive. A `gc` that
/// resealed and then died before rewriting the mark leaves exactly this.
WatermarkIsForAnotherArchive,
/// The live log no longer contains the marked prefix: it was compacted
/// (a rename, so a new inode), replaced, or truncated below the mark.
///
/// Correct, and **silent** unless someone counts it: the default policy
/// compacts at 1 024 frames, so a repository busy enough to reach that
/// loses the fast path until its next `gc` with nothing to show for it.
/// [`OverlayCache::refusals`] is what makes it observable.
JournalRewritten,
/// A transaction is **missing from the journal**: some reference's recorded
/// `old` value is not what the tier below it says was there.
///
/// `FsRefStore` commits through `gix-ref` and journals afterwards, and a
/// journal write that fails is logged rather than fatal. That was harmless
/// while the journal was only history. It is not harmless when the journal
/// is half of an advertisement, so the gap is **detected from the data
/// itself** rather than trusted not to have happened: every row carries the
/// resolved previous value, so a complete journal is a chain and a lost
/// transaction breaks it.
///
/// This needs no I/O in an error path, which is why it is the mechanism
/// rather than the mitigation — a repair that writes to the filesystem is
/// being asked to fix a fault whose likeliest causes will fail it too.
JournalGap,
}
impl std::fmt::Display for NotCurrent {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Self::NoRefsSection => "the archive carries no __gunnar_refs__ section",
Self::NoWatermark => "there is no reference watermark beside the archive",
Self::WatermarkIsForAnotherArchive => "the watermark describes a different archive",
Self::JournalRewritten => "the live reference log no longer holds the marked prefix",
Self::JournalGap => "a transaction is missing from the live reference log",
})
}
}
/// What [`open`] produced.
#[derive(Debug)]
pub enum Opened {
/// An overlay that is current with the live log as of this call.
Ready(OverlayRefStore),
/// Not usable; use the authoritative store. Never a wrong answer.
NotCurrent(NotCurrent),
}
impl Opened {
/// The store, or `None`. For the callers that only want the fast path and
/// have their own fallback.
pub fn ready(self) -> Option<OverlayRefStore> {
match self {
Self::Ready(store) => Some(store),
Self::NotCurrent(_) => None,
}
}
}
/// Open `archive`'s sealed namespace overlaid with everything `store_dir`'s
/// reference log has recorded since it was sealed.
///
/// `store_dir` is the repository's `gunnar-store/` directory — the one holding
/// `refs.arrow`. **No lock is taken on it**, so this is safe against the
/// running server that is writing it.
///
/// # `Err` is a fourth `NotCurrent`, for a serving caller
///
/// [`Opened::NotCurrent`] is "nothing is wrong, the fast path does not apply".
/// `Err` is "the archive or the log is malformed" — a ref name the journal
/// should never have been able to write, an oid that is not one, a section that
/// will not decode. It is returned rather than swallowed because corruption
/// that presents as a merely-missing branch is worse than corruption that
/// presents as an error (the same reason `ArchiveRefStore::from_folded` refuses
/// a ref that points nowhere instead of dropping the row).
///
/// **A caller on the serving path must still fall back rather than propagate**:
/// `ls-remote` failing outright is a worse outcome than `ls-remote` served by
/// `gix-ref`, and the authority is right there. Log it, do not return it.
pub fn open(archive: &Path, store_dir: &Path, hash_kind: gix_hash::Kind) -> Result<Opened> {
let Some(base) = ArchiveRefStore::open(archive)? else {
return Ok(Opened::NotCurrent(NotCurrent::NoRefsSection));
};
let Some(mark) = RefMark::read_for(archive)? else {
return Ok(Opened::NotCurrent(match RefMark::path(archive).exists() {
true => NotCurrent::WatermarkIsForAnotherArchive,
false => NotCurrent::NoWatermark,
}));
};
let Some(since) = history::read_live_since(store_dir, hash_kind, &mark)? else {
return Ok(Opened::NotCurrent(NotCurrent::JournalRewritten));
};
match OverlayRefStore::from_parts(base, &since) {
Ok(overlay) => Ok(Opened::Ready(overlay)),
Err(BuildError::Gap(why)) => {
tracing::warn!(
target: crate::STORE_TARGET,
reason = %why,
"the reference journal has a gap, so `ls-remote` will answer from the \
authoritative store until the next `gc`"
);
Ok(Opened::NotCurrent(NotCurrent::JournalGap))
}
Err(BuildError::Malformed(err)) => Err(err),
}
}
/// Why an overlay could not be built from a base and a delta.
#[derive(Debug)]
pub enum BuildError {
/// A transaction is missing from the journal. See [`NotCurrent::JournalGap`]
/// — this is a fall-back, not a fault.
Gap(String),
/// The archive or the journal holds something that is not a reference.
Malformed(anyhow::Error),
}
impl std::fmt::Display for BuildError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Gap(why) => write!(f, "{why}"),
Self::Malformed(err) => write!(f, "{err}"),
}
}
}
impl std::error::Error for BuildError {}
/// The sealed namespace with the live log's tail applied over it.
///
/// Read-only, and read-only for the same reason [`ArchiveRefStore`] is: the
/// archive is the batch tier. See the module docs.
pub struct OverlayRefStore {
base: ArchiveRefStore,
/// Reference moves since the seal. `None` is a **tombstone** — the
/// reference was deleted after the seal and must not be advertised even
/// though the base still carries it. That is the classic log-structured
/// bug, and it is a distinct state rather than a missing key on purpose.
delta: BTreeMap<FullName, Option<Target>>,
/// `HEAD`, built once. See [`RefStore::head`].
head_name: FullName,
served: AtomicU64,
from_base: AtomicU64,
from_delta: AtomicU64,
tombstoned: AtomicU64,
}
impl std::fmt::Debug for OverlayRefStore {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("OverlayRefStore")
.field("sealed", &self.base.len())
.field("delta", &self.delta.len())
.finish()
}
}
impl OverlayRefStore {
/// Build from a base and the transactions that came after it.
///
/// Split out so a test can drive the overlay over a delta it wrote by hand,
/// and so [`open`] has exactly one body.
pub fn from_parts(
base: ArchiveRefStore,
since: &[RefTransaction],
) -> std::result::Result<Self, BuildError> {
let mut delta: BTreeMap<FullName, Option<Target>> = BTreeMap::new();
let bad = |e: anyhow::Error| BuildError::Malformed(e);
for transaction in since {
// A compaction's snapshot row can only appear here if the log was
// rewritten after the mark was taken, and that is exactly what
// `read_live_since` refuses to return. Reaching this means the
// invalidation is not working, and a snapshot row applied as a
// delta would republish the whole namespace as of the compaction —
// silently correct on a quiet repository and silently stale on a
// busy one. Refused rather than applied.
if transaction.is_snapshot() {
return Err(bad(anyhow!(
"a compaction snapshot row for '{}' reached the overlay delta; the live log \
was rewritten and the watermark did not catch it",
transaction.refname
)));
}
let name = FullName::new(transaction.refname.as_bytes()).map_err(|e| {
bad(anyhow!(
"the journal holds '{}', which is not a ref name: {e}",
transaction.refname
))
})?;
let value = target_of(&transaction.new_symref, transaction.new_oid).map_err(bad)?;
// ── the chain ────────────────────────────────────────────────────
//
// Every row records the value the reference held **before** it
// moved, resolved by `gix-ref` rather than claimed by the client
// (`rows_from_applied` takes it from `MustExistAndMatch`). So a
// complete journal for one reference is a chain: the first row
// since the seal starts from what was sealed, and each row after it
// starts where the previous one ended. A transaction that never
// reached the log breaks that link — and breaking it is the ONLY
// evidence of the gap that exists, because the write that would
// have recorded it is the write that failed.
let was = target_of(&transaction.old_symref, transaction.old_oid).map_err(bad)?;
let expected = match delta.get(&name) {
Some(previous) => previous.clone(),
None => base.peek(&name),
};
if was != expected {
return Err(BuildError::Gap(format!(
"'{}' moved from {} in the journal, but the tier below it holds {}; a \
transaction between the two never reached the log",
transaction.refname,
describe(&was),
describe(&expected)
)));
}
delta.insert(name, value);
}
Ok(Self {
base,
delta,
head_name: FullName::new(gunnar_store::refs::HEAD)
.expect("`HEAD` is a valid reference name"),
served: AtomicU64::new(0),
from_base: AtomicU64::new(0),
from_delta: AtomicU64::new(0),
tombstoned: AtomicU64::new(0),
})
}
/// The sealed base, for a caller that wants to ask what the archive alone
/// says — which is what a test asserting "this ref came from the delta"
/// needs in order to prove the base does *not* have it.
pub fn base(&self) -> &ArchiveRefStore {
&self.base
}
/// References the delta touches, deletions included.
pub fn delta_len(&self) -> usize {
self.delta.len()
}
/// Whether the delta is empty — the repository has not been pushed to since
/// the seal, and this overlay is answering exactly what the archive holds.
pub fn is_current_with_seal(&self) -> bool {
self.delta.is_empty()
}
/// Calls this store answered, of any kind. The counter
/// `refs_from_archive.rs` reads either side of an advertisement.
pub fn served(&self) -> u64 {
self.served.load(Ordering::Relaxed)
}
/// References answered out of the **sealed** namespace.
pub fn from_base(&self) -> u64 {
self.from_base.load(Ordering::Relaxed)
}
/// References answered out of the **live** delta.
pub fn from_delta(&self) -> u64 {
self.from_delta.load(Ordering::Relaxed)
}
/// References the delta **withheld** because they were deleted after the
/// seal. Counted separately from [`Self::from_delta`] because "the ref is
/// gone" and "the ref moved" are different facts, and a test that could
/// only see a total could not tell a working tombstone from a lookup that
/// missed.
pub fn tombstoned(&self) -> u64 {
self.tombstoned.load(Ordering::Relaxed)
}
/// The peeled target the **seal** recorded, if any.
///
/// `None` for anything the delta touches, and that is the load-bearing
/// half. `peeled` has been a dead column since `__gunnar_refs__` was
/// defined — `ref_update` has never written it — so today every answer here
/// is `None` and this looks like a formality. It is not: the day the column
/// is filled, a tag that *moved* after the seal would otherwise be peeled to
/// the commit its **old** target pointed at, which is a wrong `^{}` line in
/// the advertisement rather than a missing one. Peeling stays the object
/// database's job either way; this is an accelerator, and an accelerator
/// that is stale is worse than one that is absent.
pub fn peeled(&self, name: &FullName) -> Option<gix_hash::ObjectId> {
match self.delta.contains_key(name) {
true => None,
false => self.base.peeled(name),
}
}
/// The delta's answer for `name`, without touching the counters.
fn delta_of(&self, name: &FullName) -> Option<&Option<Target>> {
self.delta.get(name)
}
}
/// One journal row's value, either side of a move. `None` is "the reference was
/// not there" — before a creation and after a deletion alike.
///
/// Symbolic first, matching `ArchiveRefStore::from_folded`: a reference that is
/// symbolic is symbolic, and flattening `HEAD` to the oid it happens to resolve
/// to would seal a detached head.
fn target_of(symbolic: &Option<String>, oid: Option<gix_hash::ObjectId>) -> Result<Option<Target>> {
Ok(match (symbolic, oid) {
(Some(symbolic), _) => Some(Target::Symbolic(
FullName::new(symbolic.as_bytes())
.map_err(|e| anyhow!("the journal holds symref target '{symbolic}': {e}"))?,
)),
(None, Some(oid)) => Some(Target::Object(oid)),
(None, None) => None,
})
}
fn describe(target: &Option<Target>) -> String {
match target {
None => "nothing".to_string(),
Some(Target::Object(oid)) => oid.to_string(),
Some(Target::Symbolic(name)) => name.to_str_lossy().into_owned(),
}
}
impl RefStore for OverlayRefStore {
fn read(&self, name: &FullName) -> gunnar_store::Result<Option<Target>> {
self.served.fetch_add(1, Ordering::Relaxed);
match self.delta_of(name) {
Some(Some(target)) => {
self.from_delta.fetch_add(1, Ordering::Relaxed);
Ok(Some(target.clone()))
}
Some(None) => {
self.tombstoned.fetch_add(1, Ordering::Relaxed);
Ok(None)
}
None => {
let found = self.base.read(name)?;
if found.is_some() {
self.from_base.fetch_add(1, Ordering::Relaxed);
}
Ok(found)
}
}
}
fn iter(&self, prefix: Option<&str>) -> gunnar_store::Result<Vec<(FullName, Target)>> {
self.served.fetch_add(1, Ordering::Relaxed);
// Both sides are already in ascending name order — the base by
// construction (`assert_sorted` at open) and the delta because it is a
// `BTreeMap`. So this is a merge and never a sort, which is what keeps
// the advertisement's byte order the same one `gix-ref` produces without
// anything downstream re-sorting.
let base = self.base.iter(prefix)?;
let matches = |name: &FullName| match prefix {
None => true,
Some(p) => name.as_bytes().starts_with(p.as_bytes()),
};
let mut delta = self
.delta
.iter()
.filter(|(name, _)| matches(name) && !name.is_head())
.peekable();
let mut out: Vec<(FullName, Target)> = Vec::with_capacity(base.len() + self.delta.len());
let mut from_base = 0u64;
let mut from_delta = 0u64;
let mut tombstoned = 0u64;
let mut emit = |out: &mut Vec<(FullName, Target)>,
name: &FullName,
value: &Option<Target>| match value {
Some(target) => {
from_delta += 1;
out.push((name.clone(), target.clone()));
}
None => tombstoned += 1,
};
for (name, target) in base {
while delta.peek().is_some_and(|(d, _)| **d < name) {
let (d, value) = delta.next().expect("peeked");
emit(&mut out, d, value);
}
match delta.peek() {
// The delta overrides this row: consume it and emit the delta's
// value (or nothing at all, for a deletion).
Some((d, _)) if **d == name => {
let (d, value) = delta.next().expect("peeked");
emit(&mut out, d, value);
}
_ => {
from_base += 1;
out.push((name, target));
}
}
}
for (name, value) in delta {
emit(&mut out, name, value);
}
self.from_base.fetch_add(from_base, Ordering::Relaxed);
self.from_delta.fetch_add(from_delta, Ordering::Relaxed);
self.tombstoned.fetch_add(tombstoned, Ordering::Relaxed);
Ok(out)
}
fn head(&self) -> gunnar_store::Result<Option<Target>> {
// `self.head_name`, not a fresh `FullName::new`: the advertisement calls
// this once per request and `FullName::new` validates, so building it
// here would put a byte walk on the hot path for a constant.
self.read(&self.head_name)
}
fn transaction(&self, _edits: Vec<RefEdit>) -> gunnar_store::Result<Vec<RefEdit>> {
Err(read_only("a reference transaction"))
}
fn set_head(&self, _target: Target) -> gunnar_store::Result<()> {
Err(read_only("setting HEAD"))
}
}
/// One refusal, worded once, and worded the same as
/// [`crate::refstore`]'s — the overlay adds a *read* over the live log and adds
/// nothing writable. Layering a live tier under the archive does not make a
/// transaction expressible here: the delta is a replay of a log this crate has
/// no handle on and no lock over, and a compare-and-swap needs the writer mutex
/// that `ArrowLogRefStore` holds.
fn read_only(what: &str) -> gunnar_store::Error {
gunnar_store::Error::Backend(anyhow!(
"{what} was attempted against a sealed archive overlaid on the live log, which is a read \
and is read-only; the write path is the live store, and `gunnar gc` is what folds it back \
into an archive"
))
}
#[cfg(test)]
mod tests {
use super::*;
use znippy_plugin_git::RefState;
/// One row of the journal, shaped by hand. Every test below builds its
/// deltas from this and nothing else, so there is one place a column moves.
fn row(refname: &str) -> RefTransaction {
RefTransaction {
push_seq: 0,
refname: refname.to_string(),
old_oid: None,
new_oid: None,
old_symref: None,
new_symref: None,
actor: "test".to_string(),
at_unix_ms: 0,
}
}
/// A recognisable oid, one repeated hex digit.
///
/// `c` must BE a hex digit: `oid('z')` parses to nothing and surfaces as an
/// opaque `Invalid` forty lines from the fixture that wrote it, which cost
/// a debugging round here. Named rather than left to `parse`'s error.
fn oid(c: char) -> gix_hash::ObjectId {
assert!(
c.is_ascii_hexdigit(),
"oid('{c}') is not a hex digit, so it is not an oid"
);
std::iter::repeat_n(c, 40)
.collect::<String>()
.parse()
.unwrap()
}
fn name(s: &str) -> FullName {
FullName::new(s.as_bytes()).unwrap()
}
/// A sealed namespace, straight into a base store.
fn sealed(refs: &[(&str, char)]) -> ArchiveRefStore {
let mut folded = BTreeMap::new();
for (n, c) in refs {
folded.insert(
(*n).to_string(),
RefState {
target: Some(oid(*c).to_string()),
peeled: None,
symref_target: None,
push_seq: 0,
updated_ms: 0,
},
);
}
ArchiveRefStore::from_folded(folded).unwrap()
}
/// One journalled move: `from` is the value the reference held before it,
/// `to` the value after. `None` on either side is "the reference was not
/// there" — a creation before, a deletion after.
///
/// **Both halves are real columns and the chain reads them**, so a fixture
/// that leaves `from` empty is not a lazy fixture, it is an invalid journal.
/// These helpers took only `to` until chain validation landed and every
/// test below then failed as a gap — which is the fixtures being wrong,
/// caught by the guard, rather than the guard being wrong.
fn moved(refname: &str, from: Option<char>, to: Option<char>) -> RefTransaction {
RefTransaction {
old_oid: from.map(oid),
new_oid: to.map(oid),
..row(refname)
}
}
fn symbolic(refname: &str, from: Option<&str>, to: &str) -> RefTransaction {
RefTransaction {
old_symref: from.map(|f| f.to_string()),
new_symref: Some(to.to_string()),
..row(refname)
}
}
fn names(store: &OverlayRefStore) -> Vec<String> {
store
.iter(None)
.unwrap()
.into_iter()
.map(|(n, _)| n.to_str_lossy().into_owned())
.collect()
}
/// **Case 1.** A reference created after the seal appears.
///
/// Seen red against `ArchiveRefStore` alone, which is the whole defect: the
/// base returns `None` for `refs/heads/new` and omits it from the scan, and
/// both assertions below fail. The `from_delta` assertion is what stops this
/// passing for the wrong reason — a base that somehow held the ref would
/// satisfy the value check and not this one.
#[test]
fn a_ref_created_after_the_seal_appears() {
let base = sealed(&[("refs/heads/main", 'a')]);
assert_eq!(
base.read(&name("refs/heads/new")).unwrap(),
None,
"the base must not have it"
);
let store =
OverlayRefStore::from_parts(base, &[moved("refs/heads/new", None, Some('b'))]).unwrap();
assert_eq!(
store.read(&name("refs/heads/new")).unwrap(),
Some(Target::Object(oid('b'))),
"a ref created after the seal was not advertised"
);
assert_eq!(
store.from_delta(),
1,
"the answer did not come from the delta"
);
assert_eq!(
names(&store),
vec!["refs/heads/main".to_string(), "refs/heads/new".to_string()],
"the scan disagrees with the lookup"
);
}
/// **Case 2.** A reference deleted after the seal does not appear.
///
/// The classic log-structured bug, from the other side: here the sealed
/// snapshot is the *older* record and it must not resurrect a ref that a
/// later push removed. Seen red against the base alone, which advertises
/// `refs/heads/doomed` because that is what was sealed.
#[test]
fn a_ref_deleted_after_the_seal_does_not_appear() {
let base = sealed(&[("refs/heads/main", 'a'), ("refs/heads/doomed", 'b')]);
assert!(
base.read(&name("refs/heads/doomed")).unwrap().is_some(),
"the base must still hold it, or this test proves nothing"
);
let store =
OverlayRefStore::from_parts(base, &[moved("refs/heads/doomed", Some('b'), None)])
.unwrap();
assert_eq!(
store.read(&name("refs/heads/doomed")).unwrap(),
None,
"a ref deleted after the seal was resurrected by the sealed snapshot"
);
assert_eq!(
store.tombstoned(),
1,
"the deletion was not answered by a tombstone"
);
assert_eq!(
names(&store),
vec!["refs/heads/main".to_string()],
"the deleted ref is still in the advertisement"
);
assert_eq!(
store.tombstoned(),
2,
"the scan did not go through the tombstone either"
);
}
/// **Case 3.** A force-update after the seal takes the newer value,
/// including one that moves the reference **backwards**.
///
/// `'0' < 'f'`, so a store that resolved a conflict by taking the larger oid
/// — or the first one, or the sealed one — passes a forward-only test and
/// fails this.
#[test]
fn a_force_update_after_the_seal_takes_the_newer_value_even_backwards() {
let base = sealed(&[("refs/heads/main", 'f')]);
let store =
OverlayRefStore::from_parts(base, &[moved("refs/heads/main", Some('f'), Some('0'))])
.unwrap();
assert_eq!(
store.read(&name("refs/heads/main")).unwrap(),
Some(Target::Object(oid('0'))),
"a force-push that moved the ref backwards was ignored in favour of the sealed value"
);
assert_eq!(store.from_delta(), 1);
assert_eq!(
store.from_base(),
0,
"the sealed value answered a ref the delta overrides"
);
let advertised = store.iter(None).unwrap();
assert_eq!(
advertised,
vec![(name("refs/heads/main"), Target::Object(oid('0')))],
"the scan advertised the sealed value while the lookup gave the live one"
);
}
/// **Case 4.** A reference both created and deleted after the seal appears
/// nowhere.
///
/// Two deltas for one name, and the later must win even though the earlier
/// one is the only one that carries a value. A fold that took the first
/// non-null it saw would advertise a branch that does not exist.
#[test]
fn a_ref_created_and_then_deleted_after_the_seal_appears_nowhere() {
let base = sealed(&[("refs/heads/main", 'a')]);
let store = OverlayRefStore::from_parts(
base,
&[
moved("refs/heads/ghost", None, Some('c')),
moved("refs/heads/ghost", Some('c'), None),
],
)
.unwrap();
assert_eq!(store.read(&name("refs/heads/ghost")).unwrap(), None);
assert_eq!(names(&store), vec!["refs/heads/main".to_string()]);
// And the ordering is what did it, not luck: reversing the two makes the
// ref exist, which is asserted rather than asserted-about.
let reversed = OverlayRefStore::from_parts(
sealed(&[("refs/heads/main", 'a')]),
&[
moved("refs/heads/ghost", None, None),
moved("refs/heads/ghost", None, Some('c')),
],
)
.unwrap();
assert_eq!(
reversed.read(&name("refs/heads/ghost")).unwrap(),
Some(Target::Object(oid('c'))),
"order does not decide the fold, so the create-then-delete test proves nothing"
);
}
/// **Case 5.** `HEAD` and symref resolution stay correct across the overlay.
///
/// Three separate claims, because they fail separately: `HEAD` is answered
/// from the delta when the delta moved it; `HEAD` never leaks into `iter()`,
/// which would put an extra line in the advertisement; and `resolve` chases
/// the symref through the overlay to the branch's **live** oid rather than
/// the sealed one.
#[test]
fn head_and_symref_resolution_survive_the_overlay() {
let mut folded = BTreeMap::new();
folded.insert(
"HEAD".to_string(),
RefState {
target: None,
peeled: None,
symref_target: Some("refs/heads/old".to_string()),
push_seq: 0,
updated_ms: 0,
},
);
folded.insert(
"refs/heads/old".to_string(),
RefState {
target: Some(oid('a').to_string()),
peeled: None,
symref_target: None,
push_seq: 0,
updated_ms: 0,
},
);
folded.insert(
"refs/heads/new".to_string(),
RefState {
target: Some(oid('b').to_string()),
peeled: None,
symref_target: None,
push_seq: 0,
updated_ms: 0,
},
);
let base = ArchiveRefStore::from_folded(folded).unwrap();
assert_eq!(
base.head().unwrap(),
Some(Target::Symbolic(name("refs/heads/old"))),
"the base must start pointing at the old branch"
);
let store = OverlayRefStore::from_parts(
base,
&[
symbolic("HEAD", Some("refs/heads/old"), "refs/heads/new"),
moved("refs/heads/new", Some('b'), Some('9')),
],
)
.unwrap();
assert_eq!(
store.head().unwrap(),
Some(Target::Symbolic(name("refs/heads/new"))),
"HEAD was answered from the sealed snapshot after the delta moved it"
);
assert!(
store.iter(None).unwrap().iter().all(|(n, _)| !n.is_head()),
"HEAD leaked into iter(), which adds a line to the advertisement"
);
assert_eq!(
store.resolve(&name("HEAD")).unwrap(),
Some(oid('9')),
"the symref chase resolved to the SEALED tip of the branch HEAD now points at"
);
}
/// **Case 6.** Peeled tags stay correct.
///
/// `peeled` is a dead column — `ref_update` has never written it — so the
/// base answers `None` for everything today and this test would be vacuous
/// if it only checked that. It fills the column by hand instead, so it
/// asserts the *rule*: a sealed peel survives for a tag the delta did not
/// touch, and is withheld for one it did. Serving the old peel for a moved
/// tag would be a wrong `^{}` line rather than a missing one.
#[test]
fn a_sealed_peel_survives_an_untouched_tag_and_is_withheld_from_a_moved_one() {
let mut folded = BTreeMap::new();
for (n, target, peel) in [("refs/tags/still", 'a', 'e'), ("refs/tags/moved", 'b', 'f')] {
folded.insert(
n.to_string(),
RefState {
target: Some(oid(target).to_string()),
peeled: Some(oid(peel).to_string()),
symref_target: None,
push_seq: 0,
updated_ms: 0,
},
);
}
let base = ArchiveRefStore::from_folded(folded).unwrap();
assert_eq!(
base.peeled(&name("refs/tags/moved")),
Some(oid('f')),
"the base must carry a peel, or the withholding below proves nothing"
);
let store =
OverlayRefStore::from_parts(base, &[moved("refs/tags/moved", Some('b'), Some('c'))])
.unwrap();
assert_eq!(
store.peeled(&name("refs/tags/still")),
Some(oid('e')),
"a sealed peel was dropped for a tag nothing touched"
);
assert_eq!(
store.peeled(&name("refs/tags/moved")),
None,
"the peel sealed for the OLD target was served for a tag that has since moved"
);
}
/// The scan and the lookup must agree on every name, in both directions.
///
/// Two implementations of one question is how a store starts answering
/// differently depending on which one you ask, and the merge in `iter` is a
/// second implementation of `read`.
#[test]
fn the_scan_and_the_lookup_agree_on_every_name() {
let base = sealed(&[
("refs/heads/a", '1'),
("refs/heads/c", '2'),
("refs/heads/e", '3'),
("refs/tags/v1", '4'),
]);
let store = OverlayRefStore::from_parts(
base,
&[
// before the first base row, between two, after the last, an
// override and a deletion: every position the merge has.
moved("refs/heads/A", None, Some('5')),
moved("refs/heads/b", None, Some('6')),
moved("refs/heads/c", Some('2'), Some('7')),
moved("refs/heads/e", Some('3'), None),
moved("refs/tags/v2", None, Some('8')),
],
)
.unwrap();
let scanned = store.iter(None).unwrap();
assert_eq!(
scanned
.iter()
.map(|(n, _)| n.to_str_lossy().into_owned())
.collect::<Vec<_>>(),
vec![
"refs/heads/A",
"refs/heads/a",
"refs/heads/b",
"refs/heads/c",
"refs/tags/v1",
"refs/tags/v2"
],
"the merge lost, duplicated or misordered a row"
);
for (n, target) in &scanned {
assert_eq!(
store.read(n).unwrap().as_ref(),
Some(target),
"the scan and the lookup disagree on {}",
n.to_str_lossy()
);
}
assert_eq!(
store.read(&name("refs/heads/e")).unwrap(),
None,
"the deletion did not stick"
);
// Ascending, which is what the advertisement's byte-identity rests on.
for pair in scanned.windows(2) {
assert!(
pair[0].0.as_bytes() < pair[1].0.as_bytes(),
"the merged scan is not in ascending name order at {:?}",
pair[0].0
);
}
}
/// A prefix scan must apply to both tiers, and stay bounded on both sides.
#[test]
fn a_prefix_scan_covers_the_delta_too() {
let base = sealed(&[("refs/heads/a", '1'), ("refs/tags/v1", '2')]);
let store = OverlayRefStore::from_parts(
base,
&[
moved("refs/heads/z", None, Some('3')),
moved("refs/tags/v2", None, Some('4')),
],
)
.unwrap();
let heads: Vec<String> = store
.iter(Some("refs/heads/"))
.unwrap()
.into_iter()
.map(|(n, _)| n.to_str_lossy().into_owned())
.collect();
assert_eq!(
heads,
vec!["refs/heads/a".to_string(), "refs/heads/z".to_string()]
);
assert_eq!(store.iter(Some("refs/tags/")).unwrap().len(), 2);
assert_eq!(store.iter(Some("refs/nope/")).unwrap().len(), 0);
assert_eq!(store.iter(None).unwrap().len(), 4);
}
/// The write refusal is unchanged. Layering a live tier underneath does not
/// make a transaction expressible — see [`read_only`].
#[test]
fn the_overlay_still_refuses_a_write() {
let store = OverlayRefStore::from_parts(sealed(&[("refs/heads/main", 'a')]), &[]).unwrap();
assert!(
store.transaction(vec![]).is_err(),
"a transaction was accepted"
);
assert!(
store
.set_head(Target::Symbolic(name("refs/heads/main")))
.is_err(),
"set_head was accepted"
);
}
/// A compaction snapshot row must never be applied as a delta.
///
/// It cannot arrive through `read_live_since`, which invalidates the mark
/// when the log is rewritten. This asserts the second guard *is* a guard:
/// hand one over directly and the overlay refuses to build rather than
/// republishing a whole namespace as of the compaction.
#[test]
fn a_compaction_snapshot_row_is_refused_rather_than_applied() {
let compactor = gunnar_store::arrow_log::COMPACTION_ACTOR.to_string();
let snapshot = RefTransaction {
new_oid: Some(oid('a')),
actor: compactor.clone(),
..row("refs/heads/main")
};
assert!(snapshot.is_snapshot(), "the fixture is not a snapshot row");
let err = OverlayRefStore::from_parts(sealed(&[("refs/heads/main", 'b')]), &[snapshot])
.unwrap_err()
.to_string();
assert!(err.contains("snapshot"), "unexpected refusal: {err}");
// The same row with a previous value is a genuine move by a client that
// happens to be named like the compactor, and must be applied.
let genuine = RefTransaction {
old_oid: Some(oid('b')),
new_oid: Some(oid('a')),
actor: compactor,
..row("refs/heads/main")
};
assert!(!genuine.is_snapshot());
let store =
OverlayRefStore::from_parts(sealed(&[("refs/heads/main", 'b')]), &[genuine]).unwrap();
assert_eq!(
store.read(&name("refs/heads/main")).unwrap(),
Some(Target::Object(oid('a')))
);
}
/// **The chain, at the unit level.** A journal that skips a transaction is
/// refused; the same journal with the missing row put back is accepted.
///
/// Stated as a pair on purpose. "A gap is refused" alone is satisfied by a
/// store that refuses everything, and the corpus test would then be the
/// only thing standing between that and a permanently-disabled fast path.
#[test]
fn a_journal_that_skips_a_transaction_is_refused_and_a_complete_one_is_not() {
let sealed_at = || sealed(&[("refs/heads/main", 'a')]);
// a -> b -> c, whole.
let whole = [
moved("refs/heads/main", Some('a'), Some('b')),
moved("refs/heads/main", Some('b'), Some('c')),
];
let store = OverlayRefStore::from_parts(sealed_at(), &whole).unwrap();
assert_eq!(
store.read(&name("refs/heads/main")).unwrap(),
Some(Target::Object(oid('c'))),
"a complete chain was not applied"
);
// The middle transaction never reached the log: a -> [lost] -> c.
let gapped = [moved("refs/heads/main", Some('b'), Some('c'))];
match OverlayRefStore::from_parts(sealed_at(), &gapped) {
Err(BuildError::Gap(why)) => {
assert!(why.contains("refs/heads/main"), "the refusal does not name it: {why}")
}
Err(other) => panic!("expected a gap, got {other}"),
Ok(_) => panic!(
"a journal missing a transaction was accepted; the overlay would advertise refs/heads/main at whatever the archive holds"
),
}
// A creation whose reference the archive already carries is the same
// fault from the other side, and must not be waved through because
// `None` is the easy case.
let resurrect = [moved("refs/heads/main", None, Some('e'))];
assert!(
matches!(
OverlayRefStore::from_parts(sealed_at(), &resurrect),
Err(BuildError::Gap(_))
),
"a journal that creates a reference the archive already holds was accepted"
);
}
/// An empty delta must answer exactly what the archive does, counter for
/// counter — otherwise every measurement of the overlay against the base is
/// measuring the overlay's own bookkeeping.
#[test]
fn an_empty_delta_answers_exactly_the_sealed_namespace() {
let refs: Vec<(&str, char)> = vec![
("refs/heads/a", '1'),
("refs/heads/b", '2'),
("refs/tags/v1", '3'),
];
let alone = sealed(&refs);
let store = OverlayRefStore::from_parts(sealed(&refs), &[]).unwrap();
assert!(store.is_current_with_seal());
assert_eq!(store.iter(None).unwrap(), alone.iter(None).unwrap());
assert_eq!(store.from_delta(), 0, "an empty delta answered something");
assert_eq!(
store.from_base(),
3,
"the sealed rows were not counted as sealed"
);
assert_eq!(store.tombstoned(), 0);
}
}
// ─────────────────────────── staying current ─────────────────────────────────
/// Everything that can make an opened overlay stale, as three `stat`s.
///
/// An [`OverlayRefStore`] is a **snapshot**: it folds the archive and reads the
/// journal's tail once, at open. Cache one and it goes stale exactly the way the
/// bare archive did — more slowly, but the same class of wrong. This is what
/// makes caching it safe.
///
/// | what moved | what shows it |
/// |---|---|
/// | a push appended to the journal | `journal.2`, the length — the log is append-only within its inode |
/// | the journal was compacted, replaced or truncated | `journal.0`/`journal.1`, device and inode: compaction renames a replacement over it |
/// | `gc` resealed the archive | the archive inode: `seal` stages under a scratch name and renames |
/// | a push appended to the archive | the archive length and mtime |
/// | a failed journal write destroyed the watermark | the watermark file |
///
/// **The last row was missing and the cache's own test found it.** A journal
/// write that fails fires `FsStore`'s gap hook, which removes
/// `RefMark::path(archive)` — and nothing else on disk changes. A token that
/// watched only the archive and the journal therefore kept serving the cached
/// overlay straight through the one event the hook exists to signal, which
/// would have made the cache *defeat* the protection rather than carry it. It
/// is a third `stat`, and it is not optional.
///
/// The archive row is not redundant either. `cold_tier::append_push` writes the archive
/// **in place** on any push that carries objects, and `meta_sink_append`'s
/// `open_existing` re-supplies no reserved section it was not handed — so an
/// object-carrying push **drops `__gunnar_refs__` entirely** (MEASURED, oden
/// 2026-08-04: 144 928 -> 146 396 bytes and the section gone). A cache that
/// watched only the inode would keep serving a namespace whose archive no
/// longer carries one.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct Validity {
archive: (u64, u64, u64, i64),
journal: (u64, u64, u64),
watermark: (u64, u64, u64, i64),
}
impl Validity {
/// Read the two `stat`s. A missing file is a distinct, stable value rather
/// than an error: a repository can legitimately have no archive yet, and a
/// cache must be able to notice one appearing.
pub fn read(archive: &Path, store_dir: &Path) -> Self {
use std::os::unix::fs::MetadataExt as _;
let four = |path: std::path::PathBuf| {
std::fs::metadata(path)
.map(|m| (m.dev(), m.ino(), m.size(), m.mtime_nsec()))
.unwrap_or((0, 0, 0, 0))
};
let j = std::fs::metadata(store_dir.join("refs.arrow"))
.map(|m| (m.dev(), m.ino(), m.size()))
.unwrap_or((0, 0, 0));
Self {
archive: four(archive.to_path_buf()),
journal: j,
watermark: four(crate::refmark::RefMark::path(archive)),
}
}
}
/// An [`OverlayRefStore`] that stays current, re-opened only when something
/// actually moved.
///
/// # Why validate rather than invalidate
///
/// There is no write path this process can hook to be told. `gunnar gc` is a
/// CLI verb as well as a server call, and a push can land from a connection this
/// handle knows nothing about. Asking the filesystem is the only answer that is
/// correct for a writer this process never saw — the same reasoning, and the
/// same conclusion, as `cold_tier.rs`'s opened-archive cache, which validates a
/// `(dev, ino, len, mtime)` tuple on every lookup for exactly this reason.
///
/// It costs **three `stat`s** per advertisement against a re-open that costs
/// 43 µs at 8 references and 10.4 ms at 10 000.
///
/// # A refusal is cached too
///
/// [`Opened::NotCurrent`] is cached against the same [`Validity`] as a success.
/// Without that, the common steady state after an object-carrying push — the
/// archive has no `__gunnar_refs__` at all — would re-read and re-fold the
/// archive on **every** advertisement only to refuse it again, which is slower
/// than the `gix-ref` path it falls back to.
pub struct OverlayCache {
archive: std::path::PathBuf,
store_dir: std::path::PathBuf,
hash_kind: gix_hash::Kind,
cached: std::sync::Mutex<Option<(Validity, Cached)>>,
opens: AtomicU64,
validations: AtomicU64,
refusals: std::sync::Mutex<BTreeMap<&'static str, u64>>,
}
/// What the cache is holding.
#[derive(Clone)]
enum Cached {
Ready(std::sync::Arc<OverlayRefStore>),
NotCurrent(NotCurrent),
}
/// What [`OverlayCache::get`] answered.
pub enum Current {
/// Current as of this call. Answer from it.
Ready(std::sync::Arc<OverlayRefStore>),
/// Not usable; use the authoritative store.
NotCurrent(NotCurrent),
}
impl OverlayCache {
pub fn new(archive: &Path, store_dir: &Path, hash_kind: gix_hash::Kind) -> Self {
Self {
archive: archive.to_path_buf(),
store_dir: store_dir.to_path_buf(),
hash_kind,
cached: std::sync::Mutex::new(None),
opens: AtomicU64::new(0),
validations: AtomicU64::new(0),
refusals: std::sync::Mutex::new(BTreeMap::new()),
}
}
/// The overlay, current as of this call.
pub fn get(&self) -> Result<Current> {
self.validations.fetch_add(1, Ordering::Relaxed);
let now = Validity::read(&self.archive, &self.store_dir);
let mut cached = self
.cached
.lock()
.map_err(|_| anyhow!("the overlay cache is poisoned"))?;
if let Some((when, held)) = cached.as_ref() {
if *when == now {
return Ok(match held {
Cached::Ready(store) => Current::Ready(store.clone()),
Cached::NotCurrent(why) => Current::NotCurrent(*why),
});
}
}
// Something moved. Re-open — and re-read the validity is NOT repeated
// here on purpose: a push landing between the `stat` and the open makes
// the entry stale the moment it is stored, which the next `get` sees
// and repairs. Re-reading after the open would instead record a
// validity NEWER than the bytes that were read, which is the one
// ordering that would make a stale entry look current.
self.opens.fetch_add(1, Ordering::Relaxed);
let held = match open(&self.archive, &self.store_dir, self.hash_kind)? {
Opened::Ready(store) => Cached::Ready(std::sync::Arc::new(store)),
Opened::NotCurrent(why) => {
*self
.refusals
.lock()
.map_err(|_| anyhow!("the overlay cache is poisoned"))?
.entry(why.reason())
.or_insert(0) += 1;
Cached::NotCurrent(why)
}
};
*cached = Some((now, held.clone()));
Ok(match held {
Cached::Ready(store) => Current::Ready(store),
Cached::NotCurrent(why) => Current::NotCurrent(why),
})
}
/// How many times the overlay was actually re-opened.
///
/// The counter that says the cache is a cache. A `get` that re-opened every
/// time would satisfy every correctness test in this file and none of the
/// reason for it to exist.
pub fn opens(&self) -> u64 {
self.opens.load(Ordering::Relaxed)
}
/// How many times [`OverlayCache::get`] was called.
pub fn validations(&self) -> u64 {
self.validations.load(Ordering::Relaxed)
}
/// Refusals since this cache was created, by reason.
///
/// **The metric for the silent case.** A repository busy enough to compact
/// its reference log (1 024 frames by default) loses the fast path until its
/// next `gc`, and does so correctly and without saying anything. So does one
/// whose archive an object-carrying push has stripped of its refs section.
/// Neither is a fault and both are worth seeing, and an operator can only
/// see them if something counts them.
pub fn refusals(&self) -> BTreeMap<&'static str, u64> {
self.refusals.lock().map(|m| m.clone()).unwrap_or_default()
}
}
impl NotCurrent {
/// A short, stable key for counting and for a log field.
pub fn reason(&self) -> &'static str {
match self {
Self::NoRefsSection => "no_refs_section",
Self::NoWatermark => "no_watermark",
Self::WatermarkIsForAnotherArchive => "watermark_for_another_archive",
Self::JournalRewritten => "journal_rewritten",
Self::JournalGap => "journal_gap",
}
}
}