crates/storage-git-gix/parked-gix-2/gunnar-store-gix/src/fs.rs
//! The `fs` backend: a real bare repository on a real filesystem, served by
//! git's own **plumbing**, `gix-odb` for objects and `gix-ref` for references.
//!
//! **This backend assumes a filesystem**, which is why it is behind the `fs`
//! feature. What it does *not* assume is a worktree, a shell, or a `.git/config`
//! worth parsing, so it links none of `gix`'s porcelain. That matters:
//! `gix::Repository` reaches its object and ref stores through `gix-discover`,
//! `gix-config`, `gix-worktree`, `gix-index`, `gix-filter`, `gix-attributes`,
//! `gix-ignore` and, via `blob-diff`, `gix-command`, which exists to spawn
//! subprocesses. Spec §0 forbids every one of those in a build that has to run
//! as PID 1 in a distroless image, and `xtask/src/guard.rs`
//! asserts their absence from `cargo tree`. This module used to open
//! `gix::ThreadSafeRepository`, and that was the sole reason all eight were
//! linked into the appliance build.
//!
//! Nothing here is reimplemented that git already implements. Objects go
//! through `gix_odb`'s implementation of the very same
//! `gix_object::{Find, Exists, FindHeader, Write}` traits the
//! [`crate::GixObjects`] adapter targets, and references go through `gix-ref`'s
//! loose + `packed-refs` file store with its `gix-lock` rename atomicity,
//! exactly what `gix::Repository` delegated to. gunnar's job here is
//! translation, not invention (LAW 5).
//!
//! The one thing the porcelain did that had to be replaced rather than called
//! is **creating** a bare repository ([`init_bare`]): `gix::create` builds one
//! out of `gix-config` and `gix-discover`. A bare repository is four
//! directories and two files, so this module writes them, and
//! `tests/real_git_reads_what_we_wrote.rs` checks the result with the real
//! `git` binary, including `git rev-parse --show-object-format`, which is the
//! only honest proof that the `config` written here says what it must.
//!
//! Secrets have no gix equivalent, there being no such thing as a git secret, so
//! this backend reuses [`crate::arrow_log::ArrowLogSecretStore`] in a small
//! Arrow IPC log inside the git directory. Reusing it means the atomic
//! get-or-create is proven by the same conformance run rather than written
//! twice, and it keeps the crate on **one** storage technology (D18).
//!
//! That log brings its exclusive lock with it, so two `FsStore` handles on the
//! same git directory cannot exist at once. That is a real narrowing relative
//! to git's own loose-ref model, and it is the right one: gunnar's secret store
//! must be single-writer whatever the objects and refs underneath it are doing.
use std::os::unix::ffi::OsStrExt;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use gix_hash::ObjectId;
use gix_object::{Data, Header, Kind};
use crate::arrow_log::{ArrowLogRefStore, ArrowLogSecretStore, ArrowLogStore};
use crate::error::{Error, Observed, Result};
use crate::object::ObjectStore;
use crate::odb::OdbPool;
use crate::refs::{
matches_prefix, Change, FullName, PreviousValue, RefEdit, RefStore, Target, HEAD,
};
/// Directory inside the git dir holding this repository's gunnar-owned logs.
///
/// Public because `gunnar_store::history` reads this directory **without**
/// opening the store — the O_EXCL lock an open takes would refuse every
/// repository a running server is serving — and a caller that had to spell
/// `"gunnar-store"` itself would be a second definition of the layout (LAW 5).
pub const STORE_DIR: &str = "gunnar-store";
/// A bare git repository on disk.
///
/// Cheap to clone; every clone shares the same object and reference stores.
#[derive(Clone)]
pub struct FsStore {
git_dir: Arc<PathBuf>,
/// Shared across every [`FsObjects`] this store hands out, which is what
/// makes it worth anything. See [`crate::odb`] and `P-012`.
odb: Arc<OdbPool>,
refs: Arc<gix_ref::file::Store>,
hash_kind: gix_hash::Kind,
secrets: ArrowLogSecretStore,
/// The reflog journal. `gix-ref` owns the references; this records what it
/// did, so the history `gunnar reflog` reads and `History::seal` seals
/// exists on the backend the server actually runs.
///
/// The same [`ArrowLogStore`] handle the secrets come from — ONE handle,
/// ONE `O_EXCL` lock, one writer. Splitting them would take a second lock on
/// a file this process already holds and deadlock every open.
journal: ArrowLogRefStore,
/// The reachability bitmap this repository is currently served by, if it
/// has one. Shared with every [`FsObjects`] handed out, so one fetch's
/// parse is every fetch's parse. See [`crate::bitmap::BitmapCache`].
bitmaps: Arc<crate::bitmap::BitmapCache>,
/// One repack at a time within this process.
///
/// Across processes the Arrow log's exclusive lock already refuses a second
/// handle on the same git directory, so this closes the only remaining
/// window: two threads holding clones of the same `FsStore`. See
/// [`crate::repack`] for why one writer is required at all.
pub(crate) repack_lock: Arc<std::sync::Mutex<()>>,
/// Called when a reference **moved** and its journal entry did **not**
/// land. See [`JournalGapHook`].
journal_gap: Option<JournalGapHook>,
}
/// What to do when the reference journal loses a transaction gix-ref already
/// committed.
///
/// # Why this is a hook and not a call
///
/// Until the cold tier began *answering* from the journal, a failed journal
/// write cost history and nothing else, and this store logged it and carried on
/// — "history is worth less than state", which was true while `gix-ref` was the
/// only thing anyone read. It stopped being true when
/// `gunnar_coldtier::overlay` started serving the sealed namespace overlaid
/// with the journal's tail: a lost transaction is then a **missing branch in an
/// advertisement**, not a hole in `gunnar reflog`.
///
/// The repair is to destroy the cold tier's reference watermark, so every later
/// advertisement falls back to `gix-ref` until the next `gc` republishes a
/// verified one. This crate cannot do that itself and must not learn how:
/// `P-006` keeps znippy out of `gunnar-store`'s resolved graph entirely, and a
/// path constant for `gunnar-cold.znippy.refmark` spelled here would be a second
/// definition of a layout the cold tier owns (LAW 5). So the knowledge stays
/// where it belongs and this is the seam it plugs into — `gunnar-cli`, which
/// already names both, installs it at [`crate::fs::FsStore::with_journal_gap_hook`].
///
/// **It is a mitigation, not the mechanism.** A hook that writes to a
/// filesystem is being asked to repair a fault whose likeliest causes — ENOSPC,
/// EIO, a read-only remount — will fail it too. The overlay therefore does not
/// depend on it: it validates the journal's own `old -> new` chain and refuses
/// on a break, which needs no I/O in an error path and catches gaps this hook
/// never saw. This closes the one case that chain validation structurally
/// cannot, a gap with no later transaction on the same reference to break the
/// chain against.
pub type JournalGapHook = Arc<dyn Fn() + Send + Sync>;
impl std::fmt::Debug for FsStore {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("FsStore")
.field("git_dir", &self.git_dir)
.field("hash_kind", &self.hash_kind)
.finish()
}
}
impl FsStore {
/// Open the bare repository at `git_dir`, creating it if it is not there.
///
/// `hash_kind` is only used when creating; if the repository already exists
/// with a different one, this fails with [`Error::HashKindMismatch`] rather
/// than writing sha1 objects into a sha256 repository.
pub fn open_or_init(git_dir: impl AsRef<Path>, hash_kind: gix_hash::Kind) -> Result<Self> {
let git_dir = git_dir.as_ref().to_path_buf();
if git_dir.join("HEAD").exists() {
let stored = stored_hash_kind(&git_dir)?;
if stored != hash_kind {
return Err(Error::HashKindMismatch {
stored,
requested: hash_kind,
});
}
} else {
init_bare(&git_dir, hash_kind)?;
}
let objects = gix_odb::Store::at_opts(
git_dir.join("objects"),
&mut std::iter::empty(),
gix_odb::store::init::Options {
object_hash: hash_kind,
..Default::default()
},
)?;
let refs = gix_ref::file::Store::at(
git_dir.clone(),
gix_ref::store::init::Options {
// A bare repository is created with `core.logallrefupdates =
// false`, and a git server has no use for a per-ref reflog.
// Stating it here rather than reading it back out of `config` is
// part of what keeps `gix-config` out of the build.
write_reflog: gix_ref::store::WriteReflog::Disable,
object_hash: hash_kind,
// `precompose_unicode: false`, from the default. `gix::open`
// took it from `core.precomposeunicode`, which its own `init`
// had written by probing the filesystem, so on APFS it could
// come out `true`. gunnar's server target is Linux, where the
// probe answers `false`; a macOS host serving a repository
// whose ref names are in decomposed form is the one case this
// narrows, and it is not a case gunnar supports.
..Default::default()
},
);
let store_dir: PathBuf = git_dir.join(STORE_DIR);
let logs = ArrowLogStore::open(store_dir, hash_kind)?;
let secrets = logs.secrets();
let journal = logs.refs();
// The pool takes ownership of the store from here on: it may REPLACE
// it (`OdbPool::grow_slots_if_full`), so a second handle onto this
// particular instance would be a snapshot that stops being updated.
let odb = Arc::new(OdbPool::new(Arc::new(objects), &git_dir, hash_kind));
let store = FsStore {
git_dir: Arc::new(git_dir),
odb,
refs: Arc::new(refs),
hash_kind,
secrets,
journal,
bitmaps: Arc::new(crate::bitmap::BitmapCache::default()),
repack_lock: Arc::new(std::sync::Mutex::new(())),
journal_gap: None,
};
// Maintenance, once, under the exclusive store lock taken just above.
// A repository whose packs have piled up between garbage collections is
// `P-012`'s case, and this is the last moment before it starts serving
// requests at which a hundred milliseconds is nobody's latency. Below
// `MULTI_PACK_INDEX_THRESHOLD` packs, or when an index is already there,
// it does nothing at all.
store.write_multi_pack_index_if_worthwhile();
Ok(store)
}
/// The `.git` directory this store serves.
pub fn git_dir(&self) -> &Path {
self.git_dir.as_path()
}
/// The hash kind of this repository.
pub fn hash_kind(&self) -> gix_hash::Kind {
self.hash_kind
}
/// A handle implementing [`ObjectStore`], backed by loose objects and packs.
///
/// Cheap, and every handle shares this store's pool of `gix-odb` handles,
/// so calling it per request is what it is for.
pub fn objects(&self) -> FsObjects {
FsObjects {
git_dir: self.git_dir.clone(),
odb: self.odb.clone(),
hash_kind: self.hash_kind,
bitmaps: self.bitmaps.clone(),
}
}
/// A handle implementing [`RefStore`], backed by `gix-ref`'s file store.
pub fn refs(&self) -> FsRefStore {
FsRefStore {
store: self.refs.clone(),
journal: self.journal.clone(),
journal_gap: self.journal_gap.clone(),
}
}
/// Install the [`JournalGapHook`] every [`FsRefStore`] this store hands out
/// will call if a reference moves and its journal entry does not land.
pub fn with_journal_gap_hook(mut self, hook: JournalGapHook) -> Self {
self.journal_gap = Some(hook);
self
}
/// Make the next reference transaction's **journal** write fail while the
/// reference itself still commits — the exact state a full disk leaves, and
/// the only way to produce it deliberately.
///
/// `gix-ref` is untouched, so this is not "a broken store": it is a store
/// whose references are correct and whose history has a hole, which is the
/// state `gunnar-store` has always tolerated and the cold tier's overlay
/// must not serve through.
#[doc(hidden)]
pub fn break_journal_for_test(&self, broken: bool) {
self.journal.break_journal_for_test(broken);
}
/// Fold the reference journal into a snapshot plus a window, right now.
///
/// Compaction is what a busy repository does on its own at 1 024 frames,
/// and it renames a replacement over the live log — which invalidates the
/// cold tier's watermark. A test cannot wait for 1 024 pushes, and a
/// property nobody has watched engage is not a property.
#[doc(hidden)]
pub fn compact_refs_for_test(&self) -> Result<()> {
self.journal.compact_for_test()
}
/// A handle implementing [`crate::SecretStore`].
pub fn secrets(&self) -> ArrowLogSecretStore {
self.secrets.clone()
}
/// The raw `gix-odb` handle, for the one caller that needs more than
/// [`ObjectStore`] gives: [`crate::repack`], which asks *where* an object
/// lives so it can copy its compressed bytes instead of decoding them.
///
/// `ObjectStore` deliberately has no such method. An oid goes in and bytes
/// come out is the whole contract, and a cold tier that is not a pack has no
/// pack location to report.
pub(crate) fn odb_handle(&self) -> gix_odb::store::Handle<Arc<gix_odb::Store>> {
// Through the pool, not through `self.objects`. The pool may have
// **replaced** the `gix_odb::Store` since this handle's field was set:
// its slot table cannot grow, so a repository that outgrows it is
// served by a rebuilt store (`OdbPool::grow_slots_if_full`). A handle
// taken off the original would be the frozen snapshot that defect is
// about.
self.odb.store().to_handle_arc()
}
/// `(slots in the object database's index table, times it has been grown)`.
///
/// Exposed so the condition behind `gunnar.endurance`'s cycle-7 defect is
/// **observable** rather than only inferable from objects going missing:
/// see [`crate::odb::OdbPool::grow_slots_if_full`]. A long-lived server
/// that never grows its table and a fixed one look identical from outside
/// until a pack lands past the table, at which point it is too late to ask.
pub fn slot_table(&self) -> (u64, u64) {
self.odb.slot_table()
}
/// `(staleness probes, of those, full re-scans)` since this store was
/// opened.
///
/// A *probe* is one `statx` on `objects/pack`, taken by a miss that wants to
/// know whether it is looking at a stale snapshot; a *re-scan* is the full
/// `consolidate_with_disk_state` it pays when the answer is yes. Both are
/// paid by [`ObjectStore::contains`] and by neither
/// [`ObjectStore::contains_without_rescan`] nor a hit.
///
/// This is the only observable the negotiation change has. It leaves every
/// answer, every wire byte and every `contains` call count exactly as they
/// were — by design, since the point is that the two paths already agreed on
/// the answer — so a test that compares outputs across it is green either
/// way and proves nothing. What changes is a syscall, so this counts the
/// syscall.
///
/// [`ObjectStore::contains`]: crate::ObjectStore::contains
/// [`ObjectStore::contains_without_rescan`]: crate::ObjectStore::contains_without_rescan
pub fn rescan_probes(&self) -> (u64, u64) {
self.odb.rescan_counts()
}
}
/// `HEAD` as a fresh repository carries it: the same bytes `git init` writes.
///
/// Built from [`crate::DEFAULT_BRANCH`] rather than spelled out, so this file
/// no longer holds an opinion of its own about which branch that is. A
/// repository whose default branch is *not* the default is corrected by
/// `gunnar_core::point_head_at`, the one writer of that projection — creating
/// the directory and deciding whose repository it is are two different jobs and
/// only the second one knows the catalog.
fn head_contents() -> Vec<u8> {
format!("ref: {}", crate::DEFAULT_BRANCH).into_bytes()
}
/// The `extensions.objectformat` key, which is what makes a repository sha256.
const OBJECT_FORMAT_KEY: &str = "objectformat";
/// Create the bare repository at `git_dir`.
///
/// This is what `gix::create` would have done, minus everything a server has no
/// use for: no `hooks/` (gunnar's policy gate is in-process, §7, and would never
/// run a script anyway), no `description`, and no `info/exclude`, since an
/// ignore file belongs to a worktree. What remains is what git requires, and
/// `tests/real_git_reads_what_we_wrote.rs` is what says so, against the real
/// binary rather than against this module's own opinion.
fn init_bare(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<()> {
for dir in [
Path::new("objects").join("info"),
Path::new("objects").join("pack"),
Path::new("refs").join("heads"),
Path::new("refs").join("tags"),
] {
std::fs::create_dir_all(git_dir.join(dir))?;
}
// sha256 is a format *extension*: git refuses to read it unless
// `repositoryformatversion` is raised to 1 alongside it.
let config = match hash_kind {
gix_hash::Kind::Sha256 => format!(
"[core]\n\trepositoryformatversion = 1\n\tbare = true\n\tlogallrefupdates = false\n\
[extensions]\n\t{OBJECT_FORMAT_KEY} = sha256\n"
),
// `gix_hash::Kind` is `#[non_exhaustive]`. sha1 is the format that needs
// no extension recorded, and it is also the reading git gives a
// repository whose format it does not recognise, so a hash gunnar
// learns about later lands here and is caught by `stored_hash_kind`
// disagreeing, rather than by a repository that silently claims sha1.
_ => "[core]\n\trepositoryformatversion = 0\n\tbare = true\n\tlogallrefupdates = false\n"
.to_owned(),
};
std::fs::write(git_dir.join("config"), config)?;
// Written last: `HEAD`'s presence is what `open_or_init` reads as "this
// repository already exists", so a half-created directory must not have one.
std::fs::write(git_dir.join("HEAD"), head_contents())?;
Ok(())
}
/// The hash kind an existing repository was created with.
///
/// One key decides it, `extensions.objectformat`, and its absence means sha1,
/// which is the reading git gives `repositoryformatversion = 0`. Reading that
/// one key is deliberate: linking `gix-config` to parse a file whose every other
/// key describes a worktree this server does not have is what put `gix-config`,
/// and through it the rest of the porcelain, into the appliance build. It is the
/// exact key [`init_bare`] writes, so the format is understood in one module
/// rather than implemented twice and kept in agreement (LAW 5).
///
/// Public because reading a repository's **history** must not open its store:
/// the store's `O_EXCL` lock is held by whatever server is serving it, and the
/// hash kind is the one thing a history reader needs that the log itself does
/// not carry. Reading it from the repository's own `config` keeps the reader
/// off the catalog too, which is a store of its own and is locked just as hard.
pub fn stored_hash_kind(git_dir: &Path) -> Result<gix_hash::Kind> {
let config_path = git_dir.join("config");
let text = match std::fs::read_to_string(&config_path) {
Ok(text) => text,
// A repository with no config at all is a sha1 repository, which is also
// how git reads it.
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(gix_hash::Kind::Sha1),
Err(err) => return Err(Error::Io(err)),
};
let mut in_extensions = false;
for line in text.lines() {
let line = line.trim();
if let Some(header) = line.strip_prefix('[').and_then(|l| l.strip_suffix(']')) {
in_extensions = header.trim().eq_ignore_ascii_case("extensions");
continue;
}
if !in_extensions {
continue;
}
let Some((key, value)) = line.split_once('=') else {
continue;
};
// Git's config keys are case-insensitive; the value here is not.
if !key.trim().eq_ignore_ascii_case(OBJECT_FORMAT_KEY) {
continue;
}
return match value.trim() {
"sha256" => Ok(gix_hash::Kind::Sha256),
"sha1" => Ok(gix_hash::Kind::Sha1),
other => Err(Error::backend(format!(
"{} declares an object format gunnar does not know: {other:?}",
config_path.display()
))),
};
}
Ok(gix_hash::Kind::Sha1)
}
/// Objects on disk: loose files and packs, read and written by `gix_odb`.
#[derive(Clone)]
pub struct FsObjects {
git_dir: Arc<PathBuf>,
odb: Arc<OdbPool>,
hash_kind: gix_hash::Kind,
bitmaps: Arc<crate::bitmap::BitmapCache>,
}
impl FsObjects {
/// Re-read `objects/` from disk, so packs another program has written
/// become visible.
///
/// **gunnar's own writes do not need this.** Every lookup that misses
/// already checks whether `objects/pack` has changed and re-scans if it has
/// (see [`crate::odb`]), which is what keeps a pack indexed by
/// `receive-pack` readable the moment it lands. This exists for the case
/// that check cannot see: a stock `git` binary, run by an operator against
/// the same directory within the same filesystem timestamp tick as gunnar's
/// last scan. [`FsStore::repack`] and [`FsStore::write_multi_pack_index`]
/// call it because they change the pack set themselves and should not wait
/// for a miss to notice.
pub fn refresh(&self) {
self.odb.refresh();
}
}
impl std::fmt::Debug for FsObjects {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("FsObjects")
.field("git_dir", &self.git_dir)
.finish()
}
}
impl ObjectStore for FsObjects {
fn hash_kind(&self) -> gix_hash::Kind {
self.hash_kind
}
fn contains(&self, id: &gix_hash::oid) -> bool {
use gix_object::Exists;
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_ODB_CONTAINS);
self.odb.lookup(|handle| handle.exists(id), |found| *found)
}
/// The same lookup with the staleness check taken out of the miss path.
///
/// `objects/pack` is not consulted, so a negotiation's misses — which are
/// the *normal* outcome there rather than an error — cost a binary search
/// and no syscall. The trait method carries the argument for why a stale
/// `false` is safe for that one caller and for no other; this is the two
/// lines that follow from it.
fn contains_without_rescan(&self, id: &gix_hash::oid) -> bool {
use gix_object::Exists;
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_ODB_CONTAINS_NO_RESCAN);
self.odb
.lookup_without_rescan(|handle| handle.exists(id), |found| *found)
}
fn read<'a>(&self, id: &gix_hash::oid, buf: &'a mut Vec<u8>) -> Result<Option<Data<'a>>> {
use gix_object::Find;
// `Data` borrows `buf`, so the odb handle must not be what it borrows
// from: read the bytes out, then rebuild the view over the caller's
// buffer once the handle is gone.
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_ODB_READ);
let found = self.odb.lookup(
|handle| {
handle
.try_find(id, buf)
.map_err(box_err)
.map(|found| found.map(|data| (data.kind, data.data.len())))
},
|outcome| matches!(outcome, Ok(Some(_))),
)?;
match found {
None => Ok(None),
Some((kind, len)) => {
buf.truncate(len);
Ok(Some(Data::new(buf, kind, self.hash_kind)))
}
}
}
fn header(&self, id: &gix_hash::oid) -> Result<Option<Header>> {
use gix_object::FindHeader;
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_ODB_HEADER);
self.odb.lookup(
|handle| handle.try_header(id).map_err(box_err),
|outcome| matches!(outcome, Ok(Some(_))),
)
}
/// `P-001`: yes, and this is the backend the whole entry is about.
///
/// A fresh handle per call, because `gix-odb`'s is `Send` but not `Sync`
/// and the generator wants one per worker. Two things are set on it and
/// both are load-bearing:
///
/// * `prevent_pack_unload()`, because `entry_by_location` returns bytes
/// read out of a mapped pack. `gix-odb` **panics** rather than serve it from a
/// handle which might unmap the pack underneath it, and a panic is the
/// right call there: the alternative is reading freed memory.
/// * the default refresh mode, deliberately **not** the pooled handles'
/// `RefreshMode::Never` (`crate::odb`). A pack that landed after this
/// store's last scan would otherwise report every one of its objects
/// missing, and this path treats a missing object as a failed clone. The
/// ids reaching here have already been found by the closure walk, so a
/// miss here means a stale snapshot and re-scanning is exactly right. It
/// costs nothing in the common case, where there are no misses.
fn packed(&self) -> Option<gunnar_pack::PackedObjects> {
let store = self.odb.store();
Some(gunnar_pack::PackedObjects::new(move || {
let mut handle = store.to_handle_arc();
handle.prevent_pack_unload();
Box::new(handle) as Box<dyn gix_pack::Find + Send>
}))
}
/// The `.bitmap` in `objects/pack`, if there is exactly one and it checks
/// out.
///
/// # Why `read_dir` on every call rather than once at open
///
/// A server opens a repository once and serves it for weeks. A `gc` in
/// between replaces the pack and its sidecar underneath it, and an
/// `FsObjects` that had cached the answer at open would keep answering
/// with a bitmap describing a pack that no longer exists — a *wrong* object
/// set, not a slow one. The directory read is a handful of entries and its
/// result feeds [`crate::bitmap::BitmapCache`]'s key, so the expensive half
/// still happens once per bitmap rather than once per fetch.
fn bitmap(&self) -> Option<Arc<crate::bitmap::PackBitmap>> {
self.bitmaps
.get(&self.git_dir.join("objects").join("pack"), self.hash_kind)
}
/// **`None`, and this is the backend for which that is the *right* answer.**
///
/// [`ObjectStore::reachable`] exists for a store whose reachability is not
/// git-shaped. This one's is: it serves a git object directory, the
/// `.bitmap` in it is the index git itself wrote, and
/// [`FsObjects::bitmap`] above hands it over. Answering here as well would
/// be a second, slower route to the same set — the twinning LAW 5 forbids —
/// and `select_objects` prefers this method, so it would also *replace* the
/// bitmap path on the one backend that has a real one.
///
/// Written out rather than left to the default so that the `None` reads as
/// a decision. `crate::object::NoPackCopy::bitmap` records what a `None`
/// nobody chose costs on this trait.
fn reachable(&self) -> Option<Arc<dyn crate::object::ReachIndex>> {
None
}
fn write(&self, kind: Kind, bytes: &[u8]) -> Result<ObjectId> {
use gix_object::Write;
// A write always "hits", so no staleness retry: this lands a loose
// object, and a loose object is found by stat'ing its path rather than
// through a pack index, so no snapshot goes stale because of it.
self.odb.lookup(
|handle| handle.write_buf(kind, bytes).map_err(box_err),
|_| true,
)
}
}
fn box_err(err: Box<dyn std::error::Error + Send + Sync + 'static>) -> Error {
Error::Backend(anyhow::Error::from_boxed(err))
}
fn anyhow_from(err: impl std::error::Error + Send + Sync + 'static) -> Error {
Error::Backend(anyhow::Error::new(err))
}
/// References on disk, via `gix-ref`'s loose + `packed-refs` store.
#[derive(Clone)]
pub struct FsRefStore {
store: Arc<gix_ref::file::Store>,
/// Where a completed transaction is recorded. See [`FsStore`]'s field of
/// the same name.
journal: ArrowLogRefStore,
/// See [`JournalGapHook`].
journal_gap: Option<JournalGapHook>,
}
impl std::fmt::Debug for FsRefStore {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("FsRefStore")
.field("git_dir", &self.store.git_dir())
.finish()
}
}
impl FsRefStore {
/// Create the directory a loose reference will live in, before `gix-ref`
/// tries to.
///
/// **`gix-ref` caps how deep a reference may nest, and the cap is 25.** The
/// lock it takes is created through `gix_fs::dir::create::all` with
/// `Retries::default()`, whose `on_create_directory_failure: 25` is
/// documented upstream as *"the maximum subdirectory depth we allow to be
/// created"*. It is not a filesystem limit and stock git does not have it:
/// MEASURED on oden 2026-08-06, ext4, git 2.53.0 created
/// `refs/heads/a/a/…/a` at depth **2000**, while this backend refused at
/// depth **30** with a bare IO error and accepted at 20. `mem` and both
/// Arrow-log backends accepted 2040.
///
/// So the `fs` backend was the only one of gunnar's four that could not
/// hold a name the other three, and git, hold — the mirror of the
/// over-long-component case, and equally invisible because every hostile
/// ref-name test in the tree ran on `mem`.
///
/// `std::fs::create_dir_all` has no depth budget, and creating the
/// directory up front is idempotent and harmless when it already exists.
/// Deletes are skipped: removing a reference needs no new directory, and
/// creating one for a name being deleted would leave litter behind.
fn create_parent_directories(&self, edits: &[RefEdit]) -> Result<()> {
let git_dir = self.store.git_dir();
for edit in edits {
if matches!(edit.change, Change::Delete { .. }) {
continue;
}
let relative = std::path::Path::new(std::ffi::OsStr::from_bytes(edit.name.as_bytes()));
let Some(parent) = git_dir.join(relative).parent().map(|p| p.to_path_buf()) else {
continue;
};
if parent.is_dir() {
continue;
}
std::fs::create_dir_all(&parent).map_err(Error::Io)?;
}
Ok(())
}
}
fn to_gix_name(name: &FullName) -> Result<gix_ref::FullName> {
gix_ref::FullName::try_from(gix_object::bstr::BStr::new(name.as_bytes())).map_err(|_| {
Error::InvalidRefName {
name: name.to_str_lossy().into_owned(),
reason: "rejected by gix-validate",
}
})
}
fn from_gix_name(name: &gix_ref::FullName) -> Result<FullName> {
FullName::new(name.as_bstr())
}
fn to_gix_target(target: &Target) -> Result<gix_ref::Target> {
Ok(match target {
Target::Object(id) => gix_ref::Target::Object(*id),
Target::Symbolic(name) => gix_ref::Target::Symbolic(to_gix_name(name)?),
})
}
/// What gix says was there, **including** the case where it cannot be decoded
/// (`S-030`).
///
/// The two call sites in [`map_prepare_error`] used to write
/// `from_gix_target(actual).ok()`, which turned a reference that existed and
/// could not be read into `None` — reported to the pushing client as *"nothing
/// was there"*. It is reachable rather than theoretical: gix's `FullName`
/// admits pseudo-refs such as `MERGE_HEAD`, and `gunnar-store`'s own
/// [`crate::refs::FullName`] admits only `HEAD` and `refs/…`, so a symbolic
/// target pointing at one decodes in gix and not here.
fn observed_from_gix(target: &gix_ref::Target) -> Observed {
match from_gix_target(target) {
Ok(target) => Observed::Value(target),
Err(err) => Observed::Unreadable(err.to_string()),
}
}
fn from_gix_target(target: &gix_ref::Target) -> Result<Target> {
Ok(match target {
gix_ref::Target::Object(id) => Target::Object(*id),
gix_ref::Target::Symbolic(name) => Target::Symbolic(from_gix_name(name)?),
})
}
fn to_gix_previous(expected: &PreviousValue) -> Result<gix_ref::transaction::PreviousValue> {
use gix_ref::transaction::PreviousValue as G;
Ok(match expected {
PreviousValue::Any => G::Any,
PreviousValue::MustExist => G::MustExist,
PreviousValue::MustNotExist => G::MustNotExist,
PreviousValue::MustExistAndMatch(t) => G::MustExistAndMatch(to_gix_target(t)?),
PreviousValue::ExistingMustMatch(t) => G::ExistingMustMatch(to_gix_target(t)?),
})
}
fn from_gix_previous(expected: &gix_ref::transaction::PreviousValue) -> Result<PreviousValue> {
use gix_ref::transaction::PreviousValue as G;
Ok(match expected {
G::Any => PreviousValue::Any,
G::MustExist => PreviousValue::MustExist,
G::MustNotExist => PreviousValue::MustNotExist,
G::MustExistAndMatch(t) => PreviousValue::MustExistAndMatch(from_gix_target(t)?),
G::ExistingMustMatch(t) => PreviousValue::ExistingMustMatch(from_gix_target(t)?),
})
}
fn to_gix_edit(edit: &RefEdit) -> Result<gix_ref::transaction::RefEdit> {
use gix_ref::transaction::{Change as G, LogChange, RefLog};
let change = match &edit.change {
Change::Update { expected, new } => G::Update {
log: LogChange {
mode: RefLog::AndReference,
force_create_reflog: false,
message: Default::default(),
},
expected: to_gix_previous(expected)?,
new: to_gix_target(new)?,
},
Change::Delete { expected } => G::Delete {
expected: to_gix_previous(expected)?,
log: RefLog::AndReference,
},
};
Ok(gix_ref::transaction::RefEdit {
change,
name: to_gix_name(&edit.name)?,
// Symbolic references are never dereferenced by a server-side edit: a
// push that names `refs/heads/main` must move `refs/heads/main` and
// nothing else.
deref: false,
})
}
fn from_gix_edit(edit: &gix_ref::transaction::RefEdit) -> Result<RefEdit> {
use gix_ref::transaction::Change as G;
let change = match &edit.change {
G::Update { expected, new, .. } => Change::Update {
expected: from_gix_previous(expected)?,
new: from_gix_target(new)?,
},
G::Delete { expected, .. } => Change::Delete {
expected: from_gix_previous(expected)?,
},
};
Ok(RefEdit {
name: from_gix_name(&edit.name)?,
change,
})
}
/// Translate gix's transaction failure into gunnar's, keeping compare-and-swap
/// rejections distinguishable from faults.
fn map_prepare_error(err: gix_ref::file::transaction::prepare::Error, edits: &[RefEdit]) -> Error {
use gix_ref::file::transaction::prepare::Error as P;
let cas = |full_name: &gix_object::bstr::BString| -> Option<(FullName, PreviousValue)> {
let name = FullName::new(full_name.as_slice()).ok()?;
let expected = edits
.iter()
.find(|e| e.name == name)
.map(|e| e.change.expected().clone())
.unwrap_or(PreviousValue::Any);
Some((name, expected))
};
match &err {
P::MustNotExist {
full_name, actual, ..
} => match cas(full_name) {
Some((name, expected)) => Error::RefCas {
name,
expected,
actual: observed_from_gix(actual),
},
None => Error::backend(err),
},
P::MustExist { full_name, .. } | P::DeleteReferenceMustExist { full_name } => {
match cas(full_name) {
// Not a swallow: gix reports no observed value for this
// variant, so `Nothing` is the honest answer rather than a
// dropped one. The audit counted three `.ok()` sites in this
// function and there are two; this is the line it counted third.
Some((name, expected)) => Error::RefCas {
name,
expected,
actual: Observed::Nothing,
},
None => Error::backend(err),
}
}
P::ReferenceOutOfDate {
full_name, actual, ..
} => match cas(full_name) {
Some((name, expected)) => Error::RefCas {
name,
expected,
actual: observed_from_gix(actual),
},
None => Error::backend(err),
},
// A lock this writer could not take. TYPED, not a `Backend`: it is
// transient and the one control-document writer retries it. Left as an
// opaque backend error it read to a caller as a broken disk.
P::LockAcquire { full_name, .. } => match FullName::new(full_name.as_slice()) {
Ok(name) => Error::RefLocked { name },
Err(_) => Error::backend(err),
},
_ => Error::backend(err),
}
}
impl FsRefStore {
fn head_name() -> FullName {
FullName::new(HEAD).expect("`HEAD` is a valid reference name")
}
}
impl RefStore for FsRefStore {
fn read(&self, name: &FullName) -> Result<Option<Target>> {
let gix_name = to_gix_name(name)?;
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_REF_READ);
match self.store.try_find(&gix_name).map_err(anyhow_from)? {
Some(reference) => Ok(Some(from_gix_target(&reference.target)?)),
None => Ok(None),
}
}
fn iter(&self, prefix: Option<&str>) -> Result<Vec<(FullName, Target)>> {
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_REF_READ);
let platform = self.store.iter().map_err(anyhow_from)?;
let mut out = Vec::new();
// `all()` walks `refs/` (loose and packed) and deliberately excludes the
// pseudo-refs such as `HEAD`, which is exactly the contract `RefStore`
// states and every other backend honours.
for reference in platform.all().map_err(Error::Io)? {
let reference = reference.map_err(anyhow_from)?;
let name = from_gix_name(&reference.name)?;
if !matches_prefix(&name, prefix) {
continue;
}
out.push((name, from_gix_target(&reference.target)?));
}
out.sort_by(|a, b| a.0.cmp(&b.0));
Ok(out)
}
fn transaction(&self, edits: Vec<RefEdit>) -> Result<Vec<RefEdit>> {
// S-023: the expectation is checked HERE, before gix sees the batch.
//
// gix's file ref store short-circuits an edit whose new value equals the
// value the reference already holds: it rewrites `expected` to
// `MustExistAndMatch(current)` and never evaluates the expectation the
// caller wrote. For `MustNotExist` that turns a create that must fail
// into a silent success, so "exactly one creator wins" — the property
// receive-pack relies on to arbitrate two racing pushes — was not true
// on the only backend that survives a restart.
//
// [`crate::refs::check_all`] is the same function mem and the Arrow log
// decide with (LAW 5), so there is one definition of a CAS failure and
// not one per backend. Its answer is discarded: gix still performs the
// apply, and its `applied` edits are what this returns.
//
// This is a pre-check, not a lock, so it can only *add* refusals; it
// never accepts anything gix would reject. The narrow race it does not
// close is a concurrent writer landing between this read and gix's
// prepare with exactly the value being proposed; closing that needs the
// check inside gix's own transaction, which is upstream work (S-038).
// The name-conflict rule too, and here it is a *pre*-check of something
// gix already enforces: gix refuses the pair with a lock error whose
// text is about a file and a directory. Running the shared check first
// means every backend returns the same typed `RefNameConflict`, so
// receive-pack can tell the class apart without reading a message.
crate::refs::check_no_name_conflict(
&edits,
|name| self.read(name),
|prefix| {
Ok(self
.iter(Some(prefix))?
.into_iter()
.map(|(n, _)| n)
.collect())
},
)?;
crate::refs::check_all(&edits, |name| self.read(name))?;
let gix_edits = edits.iter().map(to_gix_edit).collect::<Result<Vec<_>>>()?;
self.create_parent_directories(&edits)?;
// gix prepares every edit (taking a lock and checking the expectation
// for each) before committing any, and a preparation failure rolls the
// whole thing back. That is the all-or-nothing this trait promises,
// implemented by gix rather than by gunnar.
//
// Both locks fail immediately rather than waiting: a server that blocks
// on a contended reference turns a lost race into a stalled connection,
// and the caller already has to handle [`Error::RefCas`]. That is the
// same choice `gix::Repository` made from `core.filesRefLockTimeout`'s
// default, now stated here instead of read out of `config`.
//
// No committer: the store is opened with the reflog disabled (a bare
// repository's `core.logallrefupdates = false`), so gix writes none.
// gunnar's own reflog is the Arrow journal below, not gix's per-ref
// files — one log, queryable, rather than a file tree per reference.
gunnar_pack::oidprobe::bump(&gunnar_pack::oidprobe::GIX_REF_TXN);
let applied = self
.store
.transaction()
.prepare(
gix_edits,
gix_lock::acquire::Fail::Immediately,
gix_lock::acquire::Fail::Immediately,
)
.map_err(|err| map_prepare_error(err, &edits))?
.commit(None)
.map_err(anyhow_from)?;
let applied: Vec<RefEdit> = applied
.iter()
.map(from_gix_edit)
.collect::<Result<Vec<_>>>()?;
// The reflog. A bare repository is created with `core.logallrefupdates
// = false` and this store is opened with `WriteReflog::Disable`, so
// until this line the `fs` backend kept **no reference history at
// all** — §4.5 promises "the log of (ref, old, new, actor, at) IS the
// reflog" and that promise was only ever true on the backend nobody
// runs. `gunnar reflog` reads what this writes.
//
// `journal`, not `transaction`: `gix-ref` has already decided and
// committed these moves, and re-deciding them against this log's own
// replayed state would refuse every one of them on any repository gix
// has been serving. There is one authority for a reference and it is
// the file store.
//
// **A journal failure must not fail the push.** The reference has
// already moved and gix has already fsynced it; returning an error here
// would tell a client its push failed after it demonstrably succeeded,
// and a receive-pack that retried would then see the ref already at the
// new value. History is worth less than state, so it is the thing that
// is allowed to be lost — loudly (`S-030`: this is an `error!`, not a
// swallowed `let _ =`).
//
// # WHAT IT COSTS, measured rather than waved past
//
// A second fsync per transaction, and it is not free. MEASURED on oden,
// 2026-08-04, loadavg ~10, 400 sequential single-ref transactions
// through this function on md1 (xfs, RAID0 NVMe), the two arms differing
// only in whether this call is made:
//
// | | µs/transaction | transactions/s | refs.arrow after |
// |---|---|---|---|
// | journal off | 75.2 | 13 295 | 448 B (schema only) |
// | **journal on** | **198.1** | **5 049** | 273 128 B |
// | | **+122.9 µs, 2.63×** | 0.38× | |
//
// **2.63× on this path, and ~0.03% of a push.** The ratio is the honest
// headline for a ref-update microbenchmark and it is the wrong number to
// judge this by: a git push also negotiates, receives a pack, indexes it
// and walks connectivity. Against the 385 ms push the README reports,
// 123 µs is three parts in ten thousand. gunnar had no reflog at all
// before this; that is what the 123 µs buys.
//
// The lever, if it ever matters: this journal records *history*, and
// `gix-ref` has already fsynced the *state*, so losing the last entry to
// a power cut costs a row of history and never a reference. It could
// therefore skip its own barrier. That is a change to `ArrowLogStore`'s
// durability contract, which the secret store shares, so it is not made
// here on the strength of a microbenchmark nobody is blocked by.
if let Err(err) = self.journal.journal(&applied) {
tracing::error!(
event = "store.reflog.journal_failed",
%err,
"the references moved and are durable, but the journal entry for them was \
not written; this repository's history now has a hole, and anything reading \
state out of the journal must stop trusting it"
);
// The journal is load-bearing for STATE now, not only for history:
// `gunnar_coldtier::overlay` answers `ls-remote` from the sealed
// namespace plus this log's tail. See [`JournalGapHook`] for why
// the repair lives outside this crate and why it is a mitigation
// rather than the mechanism.
if let Some(hook) = &self.journal_gap {
hook();
}
}
Ok(applied)
}
fn head(&self) -> Result<Option<Target>> {
self.read(&Self::head_name())
}
fn set_head(&self, target: Target) -> Result<()> {
self.transaction(vec![RefEdit::force(Self::head_name(), target)])?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
/// `S-030`: a reference that is **there** and cannot be decoded must not be
/// reported as *"nothing was there"*.
///
/// Driven through [`observed_from_gix`] rather than through a real
/// transaction, because the failure needs gix to hand back a target this
/// crate rejects — and that is a property of the two `FullName` validators,
/// not something a push can be talked into producing on demand. The value
/// used is real, not forged: `MERGE_HEAD` is a legal gix `FullName` and is
/// legal nowhere in `gunnar_store::refs`, which admits `HEAD` and `refs/…`
/// only.
///
/// **Seen RED** against the pre-fix `from_gix_target(actual).ok()`, which
/// answers `None` here — the same value it answers when the reference
/// genuinely does not exist.
#[test]
fn a_reference_that_cannot_be_decoded_is_not_reported_as_absent() {
let undecodable = gix_ref::Target::Symbolic(
gix_ref::FullName::try_from("MERGE_HEAD").expect("a legal gix full name"),
);
// The premise: gix accepts this name and gunnar does not. If a future
// gix or a future `validate()` changes that, this test must say so
// rather than quietly exercising the decodable path.
assert!(
from_gix_target(&undecodable).is_err(),
"MERGE_HEAD now decodes into a gunnar Target, so this test no longer constructs an undecodable reference and proves nothing"
);
let observed = observed_from_gix(&undecodable);
assert!(
observed.is_unreadable(),
"a reference that could not be decoded was reported as {observed}"
);
assert_ne!(
observed,
Observed::Nothing,
"\"there was something I could not read\" collapsed back into \"nothing \
was there\", which is the whole finding"
);
assert!(
observed.to_string().contains("could not be decoded"),
"the message does not say what happened: {observed}"
);
// The other half. Without it, an `observed_from_gix` that answered
// `Unreadable` to everything would pass every assertion above, and
// every ordinary lost push would be reported as a corrupt repository.
let id = gix_hash::ObjectId::empty_blob(gix_hash::Kind::Sha1);
let ordinary = observed_from_gix(&gix_ref::Target::Object(id));
assert_eq!(ordinary, Observed::Value(Target::Object(id)));
assert_eq!(ordinary.to_string(), format!("{:?}", Target::Object(id)));
}
/// The object format survives a close and reopen, and a mismatch is refused.
///
/// [`init_bare`] writes `extensions.objectformat` and [`stored_hash_kind`]
/// reads it; before this module dropped `gix::Repository` that round-trip
/// belonged to `gix-config`, and this is the check that says the replacement
/// is equivalent where it has to be. `tests/real_git_reads_what_we_wrote.rs`
/// covers the other direction, that the real `git` binary agrees about the
/// format, so between them the written bytes are pinned from both ends.
///
/// The sha256 half is what fails if `stored_hash_kind` ever degenerates into
/// "assume sha1": a sha1 repository reopened as sha1 is green under a reader
/// that has stopped reading, and would prove nothing on its own.
#[test]
fn a_repository_remembers_its_object_format() {
let dir = tempfile::tempdir().expect("temporary directory");
for (created_as, other) in [
(gix_hash::Kind::Sha256, gix_hash::Kind::Sha1),
(gix_hash::Kind::Sha1, gix_hash::Kind::Sha256),
] {
let git_dir = dir.path().join(format!("{created_as}.git"));
let store = FsStore::open_or_init(&git_dir, created_as).expect("create");
drop(store);
assert_eq!(
stored_hash_kind(&git_dir).expect("read the object format"),
created_as,
"a {created_as} repository did not report {created_as} after being closed"
);
let store = FsStore::open_or_init(&git_dir, created_as).expect("reopen");
assert_eq!(store.hash_kind(), created_as);
drop(store);
let err = FsStore::open_or_init(&git_dir, other)
.err()
.unwrap_or_else(|| {
panic!("a {created_as} repository was opened as {other} without complaint")
});
assert!(
matches!(
err,
Error::HashKindMismatch { stored, requested }
if stored == created_as && requested == other
),
"expected a hash-kind mismatch, got {err:?}"
);
}
}
}