crates/storage-git-gix/parked-gix-2/coldtier-archive/refmark.rs
//! The reference watermark: **where in the live log this archive was sealed**.
//!
//! [`crate::overlay`] answers from the sealed namespace plus the reference
//! transactions that came after it. "After" needs a position, and a position in
//! a log that is appended to by a running server, compacted behind its own back
//! and rebuilt by `gc` needs to be *verifiable* rather than merely recorded.
//! That is what this file is.
//!
//! # Why it is beside the archive and not inside it
//!
//! Inside would be atomic for free — one rename publishes the archive and its
//! watermark together — and it was the first design. It needs a new
//! `__gunnar_*` reserved module in `znippy-common`, which is a **different
//! repository**, shared by every other lane building against the same checkout,
//! and it would put a format constant in znippy for a fact that is entirely
//! gunnar's. The cost of the sidecar is that it can be missing or stale where a
//! section could not be; the answer to that is not to hope, it is
//! [`RefMark::read_for`], which binds the mark to the archive's `(device,
//! inode, length, mtime)` and refuses it when they do not match.
//!
//! **Every failure mode here is "no overlay", never "a wrong reference".** A
//! missing mark, a stale mark, a mark for a different archive, a truncated mark,
//! a `gc` that died between the rename and this file — all of them end at
//! [`crate::overlay::NotCurrent`] and a fall-back to the authoritative store.
//! That asymmetry is the whole reason a sidecar is acceptable at all, and it is
//! why the archive identity is carried rather than assumed.
//!
//! # Why the mark is verified at seal rather than trusted
//!
//! `repack()` takes its reference snapshot and then calls the cold tier, so a
//! push can land between the two. The watermark taken inside the fold would
//! then sit *after* that push, which would put it in the sealed prefix — hidden
//! from the delta — while the archive, snapshotted before it, does not carry it
//! either. The branch would vanish from the advertisement, which is the exact
//! defect this module exists to remove.
//!
//! So [`verify`] replays the log **up to** the mark and requires it to agree,
//! reference for reference, with the namespace that was sealed. A push that
//! raced the fold makes them disagree and no mark is written, so the archive
//! serves through the authority until the next `gc`. The same check catches the
//! other way a journal can be behind: `FsRefStore` logs a failed journal write
//! and does **not** fail the push, so a hole in the log is a state gunnar
//! already tolerates — and one that would make an unverified overlay serve the
//! older value.
use std::io::Write as _;
use std::path::{Path, PathBuf};
use anyhow::{anyhow, Result};
use gunnar_store::history::{History, LiveMark};
use gunnar_store::refs::Target;
/// `b"gunnarRM"`, so a file that is not one of these is refused rather than
/// decoded into a plausible offset.
const MAGIC: [u8; 8] = *b"gunnarRM";
/// Bumped if the layout ever changes. An older or newer version reads as
/// "no watermark", which costs the overlay and never correctness.
const VERSION: u32 = 1;
/// What makes one archive a *different* archive from another at the same path:
/// `(device, inode, length, mtime_ns)`.
///
/// The same tuple `cold_tier.rs`'s opened-archive cache uses, and for the same
/// reason: `seal` stages under a scratch name and **renames over** the old
/// archive, so a reseal always lands a new inode even when the length is
/// identical and the clock has not moved, while an `append` keeps the inode and
/// changes length and mtime.
type ArchiveId = (u64, u64, u64, i64);
fn archive_id(path: &Path) -> Option<ArchiveId> {
use std::os::unix::fs::MetadataExt as _;
let meta = std::fs::metadata(path).ok()?;
Some((meta.dev(), meta.ino(), meta.size(), meta.mtime_nsec()))
}
/// The watermark file for one archive.
pub struct RefMark;
impl RefMark {
/// Where the watermark for `archive` lives: the archive's own path with
/// `.refmark` appended, so it moves, backs up and is deleted with the
/// repository exactly as the archive is.
pub fn path(archive: &Path) -> PathBuf {
let mut name = archive.as_os_str().to_os_string();
name.push(".refmark");
PathBuf::from(name)
}
/// Write the watermark for `archive`, atomically.
///
/// Staged under a `.writing` name beside the destination, fsynced, renamed,
/// and the parent directory fsynced so the rename itself is durable — the
/// same discipline the archive and the ref log are written with. A crash
/// anywhere before the rename leaves the previous mark, which the identity
/// check then rejects; it never leaves a half-written one that decodes.
pub fn write(archive: &Path, mark: &LiveMark) -> Result<()> {
let id = archive_id(archive).ok_or_else(|| {
anyhow!(
"{}: cannot stat the archive to bind a watermark to it",
archive.display()
)
})?;
let mut bytes = Vec::with_capacity(MAGIC.len() + 4 + 32 + LiveMark::ENCODED_LEN);
bytes.extend_from_slice(&MAGIC);
bytes.extend_from_slice(&VERSION.to_le_bytes());
bytes.extend_from_slice(&id.0.to_le_bytes());
bytes.extend_from_slice(&id.1.to_le_bytes());
bytes.extend_from_slice(&id.2.to_le_bytes());
bytes.extend_from_slice(&id.3.to_le_bytes());
bytes.extend_from_slice(&mark.encode());
let path = Self::path(archive);
let staged = path.with_extension("refmark.writing");
{
let mut file = std::fs::File::create(&staged)?;
file.write_all(&bytes)?;
file.sync_all()?;
}
std::fs::rename(&staged, &path)?;
if let Some(parent) = path.parent() {
if let Ok(dir) = std::fs::File::open(parent) {
let _ = dir.sync_all();
}
}
Ok(())
}
/// The watermark for `archive`, or `None` when there is not one that
/// describes **this** archive.
///
/// `None` covers: no file, a short or malformed file, a version this build
/// does not know, and — the one that matters — a mark whose recorded
/// archive identity is not the identity of the archive on disk. A `gc` that
/// resealed and died before rewriting the mark leaves exactly that, and a
/// reader that trusted the offset in it would apply a delta measured from a
/// point in the log that has nothing to do with what it is overlaying.
pub fn read_for(archive: &Path) -> Result<Option<LiveMark>> {
let bytes = match std::fs::read(Self::path(archive)) {
Ok(bytes) => bytes,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(None),
Err(err) => return Err(err.into()),
};
const HEAD: usize = 12 + 32;
if bytes.len() != HEAD + LiveMark::ENCODED_LEN
|| bytes[..8] != MAGIC
|| u32::from_le_bytes(bytes[8..12].try_into().expect("4 bytes")) != VERSION
{
return Ok(None);
}
let u = |at: usize| u64::from_le_bytes(bytes[at..at + 8].try_into().expect("8 bytes"));
let recorded: ArchiveId = (
u(12),
u(20),
u(28),
i64::from_le_bytes(bytes[36..44].try_into().expect("8 bytes")),
);
if archive_id(archive) != Some(recorded) {
return Ok(None);
}
Ok(LiveMark::decode(&bytes[HEAD..]))
}
/// Remove the watermark, so the overlay refuses until the next `gc`.
///
/// The lever for anything that learns the log and the archive have stopped
/// agreeing. Removing a file that is not there is not an error.
pub fn remove(archive: &Path) -> Result<()> {
match std::fs::remove_file(Self::path(archive)) {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(err.into()),
}
}
}
/// Take a watermark on `store_dir`'s reference log, or `None` if it cannot be
/// taken at all.
///
/// `None` rather than an error: a repository with no `gunnar-store/` directory
/// is one the cold tier can still seal, it just cannot be overlaid, and a fold
/// must not fail over that.
pub fn mark_for(store_dir: &Path) -> Option<LiveMark> {
gunnar_store::history::live_mark(store_dir).ok()
}
/// Whether `mark` is a watermark the sealed `refs` can be overlaid from.
///
/// Replays `store_dir`'s reference history up to `mark` and compares it with
/// `refs`, the namespace that was sealed. Every reference the replay knows about
/// must be at the value the seal recorded; a reference the replay has never seen
/// is not a disagreement (a repository can have references older than its
/// journal, and `HEAD` is written by `init_bare` and never by a transaction).
///
/// `Ok(None)` when they agree. `Ok(Some(reason))` when they do not, worded for a
/// log line, because "the watermark was refused" is something an operator has to
/// be able to see the cause of.
pub fn verify(
store_dir: &Path,
hash_kind: gix_hash::Kind,
mark: &LiveMark,
refs: &[(String, Target)],
) -> Result<Option<String>> {
let sealed: std::collections::BTreeMap<&str, &Target> = refs
.iter()
.map(|(name, target)| (name.as_str(), target))
.collect();
let history = History::read_upto(store_dir, hash_kind, mark.len())?;
let mut replayed: std::collections::BTreeMap<String, Option<Target>> = Default::default();
for transaction in history.transactions()? {
let value = match (&transaction.new_symref, transaction.new_oid) {
(Some(symbolic), _) => Some(Target::Symbolic(
gunnar_store::refs::FullName::new(symbolic.as_bytes())
.map_err(|e| anyhow!("the journal holds symref target '{symbolic}': {e}"))?,
)),
(None, Some(oid)) => Some(Target::Object(oid)),
(None, None) => None,
};
replayed.insert(transaction.refname, value);
}
for (name, value) in &replayed {
match (value, sealed.get(name.as_str())) {
(Some(replayed), Some(sealed)) if &replayed == sealed => {}
(None, None) => {}
(replayed, sealed) => {
return Ok(Some(format!(
"the reference log replayed to {} = {replayed:?} at the watermark, but the \
archive sealed {sealed:?}",
name
)))
}
}
}
Ok(None)
}
#[cfg(test)]
mod tests {
use super::*;
fn tmpdir(tag: &str) -> PathBuf {
let ns = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
// Scratch on md1, never /tmp — /tmp is RAM on this box.
let base = std::env::var("GUNNAR_TEST_SCRATCH")
.unwrap_or_else(|_| "/home/rickard/scratch/cargo/gunnar-refs/t".to_string());
let dir = PathBuf::from(base).join(format!("refmark_{tag}_{ns}"));
std::fs::create_dir_all(&dir).unwrap();
dir
}
/// A watermark round-trips, and is refused the moment the archive it is
/// bound to is replaced.
///
/// The second half is the one that matters: a `gc` that reseals and dies
/// before rewriting the mark leaves a mark whose offset is measured from a
/// point in the log that has nothing to do with the new archive. Seen red by
/// dropping the `archive_id(archive) != Some(recorded)` check, which makes
/// the reseal below read back as a valid watermark.
#[test]
fn a_watermark_is_refused_once_the_archive_it_names_is_replaced() {
let dir = tmpdir("bind");
let archive = dir.join("cold.znippy");
std::fs::write(&archive, b"first").unwrap();
let mark = gunnar_store::history::live_mark(&dir).unwrap();
RefMark::write(&archive, &mark).unwrap();
assert_eq!(
RefMark::read_for(&archive).unwrap(),
Some(mark),
"the watermark did not survive its own round trip"
);
// A reseal: staged elsewhere, renamed over. A new inode.
let staged = dir.join("cold.staging");
std::fs::write(&staged, b"second, resealed").unwrap();
std::fs::rename(&staged, &archive).unwrap();
assert!(
RefMark::read_for(&archive).unwrap().is_none(),
"a watermark written for the PREVIOUS archive was accepted for the new one"
);
std::fs::remove_dir_all(&dir).ok();
}
/// Garbage, truncation and a wrong version all read as "no watermark", and
/// none of them as a decoded offset.
#[test]
fn a_damaged_watermark_reads_as_absent_rather_than_as_an_offset() {
let dir = tmpdir("damaged");
let archive = dir.join("cold.znippy");
std::fs::write(&archive, b"archive").unwrap();
let mark = gunnar_store::history::live_mark(&dir).unwrap();
RefMark::write(&archive, &mark).unwrap();
let good = std::fs::read(RefMark::path(&archive)).unwrap();
assert!(
RefMark::read_for(&archive).unwrap().is_some(),
"the fixture must start valid"
);
for (what, bytes) in [
("empty", Vec::new()),
("truncated", good[..good.len() - 1].to_vec()),
("overlong", [good.clone(), vec![0u8]].concat()),
(
"wrong magic",
[b"XXXXXXXX".to_vec(), good[8..].to_vec()].concat(),
),
(
"wrong version",
[
good[..8].to_vec(),
99u32.to_le_bytes().to_vec(),
good[12..].to_vec(),
]
.concat(),
),
("garbage", vec![0xABu8; good.len()]),
] {
std::fs::write(RefMark::path(&archive), &bytes).unwrap();
assert!(
RefMark::read_for(&archive).unwrap().is_none(),
"a {what} watermark decoded to an offset instead of reading as absent"
);
}
// And the good bytes still read, so the loop above was rejecting damage
// rather than rejecting everything.
std::fs::write(RefMark::path(&archive), &good).unwrap();
assert!(RefMark::read_for(&archive).unwrap().is_some());
RefMark::remove(&archive).unwrap();
assert!(RefMark::read_for(&archive).unwrap().is_none());
RefMark::remove(&archive).unwrap();
std::fs::remove_dir_all(&dir).ok();
}
fn oid(step: u8) -> gix_hash::ObjectId {
let mut bytes = [0u8; 20];
bytes[0] = step;
bytes[19] = step.wrapping_mul(7);
gix_hash::ObjectId::from_bytes_or_panic(&bytes)
}
fn refname(s: &str) -> gunnar_store::refs::FullName {
gunnar_store::refs::FullName::new(s).unwrap()
}
/// **The guard that closes the race and the hole.**
///
/// `repack()` takes its reference snapshot and *then* calls the fold, so a
/// push can land in between; and `FsRefStore` logs a failed journal write
/// rather than failing the push, so the log can be behind the references it
/// is supposed to record. Both come out the same way here — the log
/// replayed to the watermark says one thing and the archive sealed another
/// — and both must end at "no watermark", because a watermark published
/// over that disagreement is exactly how the overlay would serve a stale
/// reference.
///
/// Seen red by making `verify` return `Ok(None)` unconditionally, which is
/// the "trust the offset" design this rejected: the disagreeing case below
/// then reports agreement.
#[test]
fn a_watermark_over_a_log_that_disagrees_with_the_seal_is_refused() {
use gunnar_store::arrow_log::ArrowLogStore;
use gunnar_store::refs::{RefEdit, RefStore as _};
let dir = tmpdir("verify");
let store_dir = dir.join("gunnar-store");
let kind = gix_hash::Kind::Sha1;
let store = ArrowLogStore::open(&store_dir, kind).unwrap();
store
.refs()
.transaction(vec![RefEdit::create(
refname("refs/heads/main"),
Target::Object(oid(1)),
)])
.unwrap();
let mark = gunnar_store::history::live_mark(&store_dir).unwrap();
// The namespace the seal actually recorded, agreeing with the log.
let agreeing = vec![("refs/heads/main".to_string(), Target::Object(oid(1)))];
assert_eq!(
verify(&store_dir, kind, &mark, &agreeing).unwrap(),
None,
"a log that matches the sealed namespace was refused, so nothing could ever verify"
);
// A push that landed after the snapshot and before the mark: the log
// holds it, the archive does not.
let raced = vec![("refs/heads/main".to_string(), Target::Object(oid(2)))];
let why = verify(&store_dir, kind, &mark, &raced)
.unwrap()
.expect("a log ahead of the sealed namespace must be refused");
assert!(
why.contains("refs/heads/main"),
"the refusal does not name the ref: {why}"
);
// A journal hole: the reference moved, gix committed it, the journal
// write failed. The archive sealed a ref the log has never heard of at
// a value the log cannot confirm — and for the ref the log DOES know,
// the two disagree.
let hole = vec![
("refs/heads/main".to_string(), Target::Object(oid(1))),
(
"refs/heads/unjournalled".to_string(),
Target::Object(oid(3)),
),
];
assert_eq!(
verify(&store_dir, kind, &mark, &hole).unwrap(),
None,
"a reference the log has never seen is not a disagreement; a repository can hold \
references older than its journal, and HEAD is written by init_bare"
);
// But a reference the log DID see and the archive dropped is.
let dropped: Vec<(String, Target)> = Vec::new();
assert!(
verify(&store_dir, kind, &mark, &dropped).unwrap().is_some(),
"the archive sealed no references at all while the log holds one, and that passed"
);
// And a deletion that both agree on is not a disagreement.
store
.refs()
.transaction(vec![RefEdit::delete(
refname("refs/heads/main"),
Target::Object(oid(1)),
)])
.unwrap();
let after_delete = gunnar_store::history::live_mark(&store_dir).unwrap();
assert_eq!(
verify(&store_dir, kind, &after_delete, &dropped).unwrap(),
None,
"a reference deleted before the seal must be absent from both, not a mismatch"
);
drop(store);
std::fs::remove_dir_all(&dir).ok();
}
}