crates/gunnar-server/src/storage/gix_engine.rs
//! **The gix arm: `storage-git-gix`, `gix-odb` / `gix-ref` on a plain filesystem.**
//!
//! The second implementation of the same eleven. Not the default and not in the
//! appliance image: `--no-default-features --features server,engine-gix` builds
//! it alone, `--features engine-znippy,engine-gix` builds both and picks at
//! runtime. Its reason for existing is that an A/B needs two things to compare,
//! and until it landed there was one.
//!
//! The store is an ordinary **bare git repository**, so whatever this arm writes,
//! stock `git` can read — which is what makes it a usable oracle for the other.
//!
//! # Three narrowings, recorded as facts rather than as trait-change requests
//!
//! Two are the backend's own, from the two-engine record:
//!
//! 1. `extents()` returns a **pack-local** offset. With N packs an `Extent` does
//! not identify a byte range on its own, and there is nowhere in `(u64, u64)`
//! for a pack id. No synthetic global offset was invented, because it would
//! index nothing.
//! 2. `TxId.push_seq` is **in-process only**. This backend keeps no ref log, so
//! there is no durable ordering authority; it resets on restart.
//!
//! The third belongs to this file and is new with it:
//!
//! 3. **`want_objects` is refused.** Enumerating the ids one push introduced is
//! not on `GitOps` and has no analog here — the znippy arm answers it out of
//! its own extent index (`oids_in_extent`), and this arm has no extent index
//! to ask. [`ingest`] therefore *fails* rather than answering `None`: `None`
//! on that field means **not asked for**, and a policy gate handed `None`
//! refuses, so returning it here would turn "this engine cannot tell you"
//! into "your policy passed". A named, bounded absence, never a plausible
//! wrong answer.
//!
//! # `seal` has no analog either, and that is not a gap
//!
//! `seal()` folds znippy's live logs into reserved **Arrow** sections. This
//! backend has no live logs to fold — `gc()` reads the packs and the ref store
//! directly — so [`maintain`] reports `sections_sealed: 0`, which is the count
//! of what happened rather than a stub value.
use std::io::BufRead;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use anyhow::{bail, Context, Result};
use git_storage_trait::{GitOps, GitServe, Oid, TxId};
use storage_git_gix::GixStore;
use super::store_cache::{self, Cache};
use super::{BeforeStore, MaintenanceReport, OpenRepo, PushReceipt};
/// Every `GixStore` this process has opened, by store root.
static STORES: Cache<GixStore> = Cache::new();
/// **The one handle for `git_dir`** — the reading path and the ingest path get
/// the same object, so a `put_pack` refreshes the view the server serves from.
///
/// This is the fix for the defect written up in [`super::store_cache`]: `open`
/// and `ingest` used to call `GixStore::open_or_init` separately, and
/// `GixStore::reload_packs` — *"git's `reprepare_packed_git()`, called from the
/// writer, which is where git calls it"* — then refreshed a handle that was
/// about to be dropped. Everything pushed into a repository was invisible to
/// the reader for the life of the process.
///
/// It is by construction rather than by a guard (one implementation): there is no
/// "invalidate after write" step to forget, because there is nothing to
/// invalidate. One writer, one reader, one `PackSet`.
///
/// # This says nothing about two PROCESSES
///
/// `storage-git-gix`'s own docs record *"no writer lock — two processes writing
/// one store root is not defended against here"*, and that is still true and
/// still not defended against. This cache is per-process and fixes a
/// per-process defect. The file's original note — that one handle per open was
/// "the honest shape of a store that makes no such promise" — conflated the two:
/// declining an in-process cache never bought any cross-process safety, and it
/// cost correctness within the process.
fn open_shared(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<Arc<GixStore>> {
let root = PathBuf::from(git_dir);
store_cache::shared(&STORES, root, || {
GixStore::open_or_init(git_dir, hash_kind)
.with_context(|| format!("opening the gix store at {}", git_dir.display()))
})
}
/// Release every cached handle. See [`super::shutdown`].
pub fn shutdown() {
store_cache::clear(&STORES);
}
/// Open — creating if absent — the bare repository at `git_dir`.
///
/// Through [`open_shared`], so the handle this hands the serving path is the
/// same one [`ingest`] writes through. `ServerStores` also caches the
/// `Arc<OpenRepo>` per repository, but that cache is keyed on the repository
/// and is not what makes the write visible — the shared `GixStore` underneath
/// is.
///
/// # 🔴 This does not compile until edda implements `GitServe` for `GixStore`
///
/// [`OpenRepo::new`] requires the **reading** contract, because everything the
/// server does with a repository needs it (`UploadPack`, `ReceivePack`,
/// `gunnar-policy`'s ancestry walk, and `HEAD`, which is not a row in
/// [`GitOps::refs`]). `storage-git-gix` implements `GitOps` and not `GitServe`
/// — the arm is PAUSED — so this call is a compile
/// error in any build carrying `engine-gix`, and it is the right kind: a
/// missing half of the contract, named at the seam, rather than a repository
/// that opens and then refuses every clone at runtime.
///
/// Two things are owed by edda before this arm builds, and neither is gunnar's
/// (see `agentL-report.md`):
///
/// 1. `GitOps::put_refs_cas` — a **pre-existing** `E0046` on `GixStore`, which
/// is why `--features engine-gix` did not compile before this change either.
/// 2. `impl GitServe for GixStore`.
pub fn open(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<Arc<OpenRepo>> {
Ok(OpenRepo::new(
git_dir,
hash_kind,
Arc::new(Served(open_shared(git_dir, hash_kind)?)),
))
}
/// **The gix store as the server serves it**: every call goes straight
/// through, except one `select` the engine should have declined (lane
/// GUN-PERF-FETCH, 2026-10-31).
///
/// `GitServe::select`'s contract says `Ok(None)` means *"this engine cannot
/// answer cheaply — walk it yourself"*. Without a reachability bitmap,
/// `storage-git-gix` 0.3.0 answers a request with `have`s anyway, by closing
/// over the client's whole `have` side — every object the client holds.
/// MEASURED on oden, git.git (81 188 commits, 1 006 refs), one new commit
/// fetched: **23 s** of server CPU, where gunnar's own walk (git's `limit_list`
/// plus the edge's trees) serves the same request in **0.2 to 0.4 s**.
///
/// So such a request is declined here and the walk answers it. Two shapes stay
/// with the engine: a request with no `have` (a clone, where both cost the same
/// whole walk), and every request once [`GitOps::gc`] has written the one
/// `.bitmap` the engine reads (its bitmap cache uses a
/// pack directory only when it holds exactly one). The fix belongs in the
/// engine; this is the server keeping the contract on its behalf until it does.
struct Served(Arc<GixStore>);
impl Served {
/// The engine's own rule for "a bitmap is there to read": exactly one
/// `.bitmap` in `objects/pack`.
fn has_bitmap(&self) -> bool {
let Ok(entries) = std::fs::read_dir(self.0.git_dir().join("objects").join("pack")) else {
return false;
};
entries
.flatten()
.filter(|entry| entry.path().extension().is_some_and(|e| e == "bitmap"))
.count()
== 1
}
}
impl GitOps for Served {
fn put(&self, pack: &[u8], refs: &[git_storage_trait::RefUpdate]) -> Result<TxId> {
self.0.put(pack, refs)
}
fn put_pack(&self, bytes: &[u8]) -> Result<TxId> {
self.0.put_pack(bytes)
}
fn put_pack_observed(
&self,
bytes: &[u8],
progress: &dyn git_storage_trait::PackProgress,
) -> Result<TxId> {
self.0.put_pack_observed(bytes, progress)
}
fn publishes_pack_progress(&self) -> bool {
self.0.publishes_pack_progress()
}
fn put_refs(&self, updates: &[git_storage_trait::RefUpdate]) -> Result<TxId> {
self.0.put_refs(updates)
}
fn get(&self, oid: Oid<'_>) -> Result<Option<git_storage_trait::Stored>> {
self.0.get(oid)
}
fn has(&self, oid: Oid<'_>) -> Result<bool> {
self.0.has(oid)
}
fn size(&self, oid: Oid<'_>) -> Result<Option<u64>> {
self.0.size(oid)
}
fn extents(&self, oids: &[Oid<'_>]) -> Result<Vec<Option<git_storage_trait::Extent>>> {
self.0.extents(oids)
}
fn refs(&self) -> Result<Vec<git_storage_trait::RefRow>> {
self.0.refs()
}
fn update_ref(&self, name: &str, old: Option<Oid<'_>>, new: Option<Oid<'_>>) -> Result<TxId> {
self.0.update_ref(name, old, new)
}
fn put_refs_cas(&self, edits: &[git_storage_trait::RefCas<'_>]) -> Result<TxId> {
self.0.put_refs_cas(edits)
}
fn reachable(&self, want: &[Oid<'_>], have: &[Oid<'_>]) -> Result<Vec<Vec<u8>>> {
self.0.reachable(want, have)
}
fn gc(&self) -> Result<git_storage_trait::GcReport> {
self.0.gc()
}
}
impl GitServe for Served {
fn read(&self, oid: Oid<'_>) -> Result<Option<(git_storage_trait::ObjType, Vec<u8>)>> {
self.0.read(oid)
}
fn header(&self, oid: Oid<'_>) -> Result<Option<(git_storage_trait::ObjType, u64)>> {
self.0.header(oid)
}
fn sizes(&self, oids: &[Oid<'_>]) -> Result<Vec<Option<u64>>> {
self.0.sizes(oids)
}
fn index_now(&self) -> Result<usize> {
self.0.index_now()
}
fn head(&self) -> Result<Option<git_storage_trait::RefRow>> {
self.0.head()
}
fn set_head(&self, target: &str) -> Result<TxId> {
self.0.set_head(target)
}
fn set_head_projected(&self, target: &str) -> Result<TxId> {
self.0.set_head_projected(target)
}
fn emit_pack(
&self,
objects: &[Oid<'_>],
have: &[Oid<'_>],
caps: &git_storage_trait::Caps,
out: &mut dyn std::io::Write,
) -> Result<git_storage_trait::PackStats> {
self.0.emit_pack(objects, have, caps, out)
}
/// See [`Served`]: a request with `have`s on a store without a bitmap is
/// the walk's.
fn select(
&self,
want: &[Oid<'_>],
have: &[Oid<'_>],
) -> Result<Option<git_storage_trait::ReachSet>> {
if !have.is_empty() && !self.has_bitmap() {
return Ok(None);
}
self.0.select(want, have)
}
}
/// One push into the bare repository: `index_pack` writes `.pack` / `.idx`.
///
/// See narrowing 3 in the module docs for `want_objects`.
///
/// `progress` is carried and handed to
/// [`GitOps::put_pack_observed`](git_storage_trait::GitOps::put_pack_observed),
/// which on this arm is the trait's own default: `GixStore` has no entry loop
/// of its own — `gix_pack` owns the split — so it publishes nothing and says so
/// through `publishes_pack_progress()`. That is the right answer rather than a
/// gap: a fabricated `(0, N) → (N, N)` here would read to a console exactly
/// like the live count the znippy arm now produces.
pub fn ingest(
git_dir: &Path,
hash_kind: gix_hash::Kind,
pack: &mut dyn BufRead,
objects: super::ObjectSet,
progress: Option<&dyn git_storage_trait::PackProgress>,
before_store: Option<BeforeStore<'_>>,
) -> Result<PushReceipt> {
// **`Required` is refused; `Wanted` is answered with `None`.**
//
// It used to refuse on a bare `want_objects`, and when the push attestation
// started asking for the set that turned every push on this arm into
// "the repository is unavailable". The refusal is still right for a gate —
// a gate over objects cannot decide without them — and wrong for a record
// that has a field for saying the set is unknown.
if objects == super::ObjectSet::Required {
bail!(
"the gix storage engine cannot enumerate the objects a push introduced: that is not \
one of the eleven, and this backend keeps no extent index to answer it from. The \
policy on this repository has a gate over the pushed object set, so the push is \
refused rather than passed on an answer the engine did not give. Serve this \
repository from the znippy engine, or drop the gate."
);
}
let started = std::time::Instant::now();
// THE SAME handle the serving path holds. `put_pack` reloads the pack set
// on the instance it is called on, so writing through a throwaway here is
// what made every pushed object invisible until the next restart.
let store = open_shared(git_dir, hash_kind)?;
let pushed = super::read_pushed_pack_spooling(pack, hash_kind, Some(git_dir))?;
// The transaction log first (`Engine::ingest_push`'s `before_store`).
if let Some(hook) = before_store {
hook(&pushed.bytes).context("framing the pushed pack in the transaction log")?;
}
match progress {
Some(sink) => store.put_pack_observed(&pushed.bytes, sink),
None => store.put_pack(&pushed.bytes),
}
.context("storing the pushed pack")?;
let pack_id = super::pack_trailer(&pushed.bytes, hash_kind)?;
Ok(PushReceipt {
num_objects: pushed.num_objects,
version: pushed.version,
pack_id,
pack_bytes: pushed.bytes.len() as u64,
new_oids: None,
elapsed_ms: started.elapsed().as_millis() as u64,
})
}
/// `gc` alone. There is no seal on this arm — see the module docs.
pub fn maintain(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<MaintenanceReport> {
// Shared for the same reason ingest is, and it is the same bug in the other
// direction: `gc` UNLINKS packs, so a reader holding its own `PackSet`
// would go on answering out of an index naming files that are gone.
let store = open_shared(git_dir, hash_kind)?;
let report = store.gc().context("sweeping dead packs")?;
Ok(MaintenanceReport {
strategy: report.strategy.to_owned(),
bytes_before: report.bytes_before,
bytes_after: report.bytes_after,
rows: report.rows,
retired_packs: report.retired_packs,
verified: report.verified,
sections_sealed: 0,
})
}
#[cfg(test)]
mod tests {
use std::process::Command;
use super::*;
/// A one-commit pack and the oid of its commit, built by stock `git`.
///
/// Stock git rather than a hand-rolled writer: the bytes this arm has to
/// accept are the bytes a real client pushes, and a pack this file wrote
/// itself would prove that this file agrees with itself.
fn tiny_pack(work: &Path) -> (Vec<u8>, String) {
let run = |args: &[&str]| {
let out = Command::new("git")
.args(args)
.current_dir(work)
.output()
.expect("git is on PATH (xtask preflight asserts it)");
assert!(
out.status.success(),
"git {args:?}: {:?}",
out.status.code()
);
out.stdout
};
run(&["init", "-q", "--initial-branch=main", "."]);
run(&["config", "user.email", "t@t"]);
run(&["config", "user.name", "t"]);
std::fs::write(work.join("f"), b"hello").expect("fixture file");
run(&["add", "f"]);
run(&["commit", "-qm", "one"]);
let oid = String::from_utf8(run(&["rev-parse", "HEAD"]))
.expect("utf8")
.trim()
.to_owned();
// `--revs --stdout` takes the rev on stdin and writes the pack to
// stdout, which is the same shape `receive-pack` hands `ingest`.
let mut child = Command::new("git")
.args(["pack-objects", "--revs", "--stdout", "-q"])
.current_dir(work)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.spawn()
.expect("git pack-objects");
{
use std::io::Write as _;
let mut stdin = child.stdin.take().expect("stdin");
writeln!(stdin, "{oid}").expect("write rev");
}
let out = child.wait_with_output().expect("pack-objects output");
assert!(out.status.success(), "git pack-objects failed");
assert!(!out.stdout.is_empty(), "git pack-objects produced no pack");
(out.stdout, oid)
}
/// **A push must be visible to a reader that was opened BEFORE it.**
///
/// The regression test for the defect in [`super::super::store_cache`], and
/// the ordering is the whole test. `open` *after* `ingest` passes even on
/// the broken code, because a fresh handle loads a fresh `PackSet` off disk
/// — which is exactly why restarting the server "fixed" it and why this went
/// unnoticed. The server does not work that way: `ServerStores` caches an
/// `Arc<OpenRepo>` per repository, so the reader is almost always older than
/// the push.
///
/// Seen red on the pre-fix code, which answered
/// `<oid> is a LOOSE object: it is in this repository but has no pack entry,
/// so there are no stored bytes to copy` — with the object sitting in a pack
/// that `git verify-pack` called `ok`.
#[test]
fn a_push_is_visible_to_a_reader_opened_before_it() {
let dir = tempfile::tempdir().expect("tempdir");
let work = dir.path().join("work");
std::fs::create_dir_all(&work).expect("work dir");
let (pack, oid_hex) = tiny_pack(&work);
let git_dir = dir.path().join("repo.git");
let kind = gix_hash::Kind::Sha1;
// THE READER FIRST. This is the ordering the server has.
let reader = open(&git_dir, kind).expect("opening the repository");
let raw = gix_hash::ObjectId::from_hex(oid_hex.as_bytes()).expect("oid");
assert!(
!reader.serve().has(raw.as_slice()).expect("has() before"),
"the repository already held the commit before it was pushed, so this test would \
pass without the push and prove nothing"
);
let mut cursor: &[u8] = &pack;
ingest(
&git_dir,
kind,
&mut cursor,
crate::storage::ObjectSet::NotNeeded,
// `None`: this test is about the store handle a reader holds, not
// about progress reporting. RED before this argument was added
// here — `ingest` grew a fifth parameter and this call did not
// follow it, so the whole `gix_engine` unit-test target stopped
// compiling and took the crate's `--lib` tests with it.
None,
// `None`: no `before_store` tee either. `ingest` grew a sixth
// parameter (786bc63, the twin frames a pushed pack) and this call
// did not follow it, which again took the crate's `--lib` tests down.
None,
)
.expect("ingesting the pushed pack");
assert!(
reader.serve().has(raw.as_slice()).expect("has() after"),
"the reader opened before the push cannot see it: the ingest wrote through a \
different store handle and refreshed that one instead"
);
let got = reader
.serve()
.get(raw.as_slice())
.expect("get() must not error")
.expect("the commit must have stored bytes to copy");
assert!(
!got.bytes.is_empty(),
"the commit resolved to zero stored bytes, which is what `emit_pack` copies"
);
shutdown();
}
/// Two commits, the second on the first, as one stock-git pack.
fn two_commits(work: &Path) -> (Vec<u8>, String, String) {
let run = |args: &[&str]| {
let out = Command::new("git")
.args(args)
.current_dir(work)
.output()
.expect("git is on PATH (xtask preflight asserts it)");
assert!(
out.status.success(),
"git {args:?}: {:?}",
out.status.code()
);
String::from_utf8(out.stdout)
.expect("utf8")
.trim()
.to_owned()
};
run(&["init", "-q", "--initial-branch=main", "."]);
run(&["config", "user.email", "t@t"]);
run(&["config", "user.name", "t"]);
std::fs::write(work.join("f"), b"one").expect("fixture file");
run(&["add", "f"]);
run(&["commit", "-qm", "one"]);
let first = run(&["rev-parse", "HEAD"]);
std::fs::write(work.join("f"), b"two").expect("fixture file");
run(&["commit", "-qam", "two"]);
let second = run(&["rev-parse", "HEAD"]);
let mut child = Command::new("git")
.args(["pack-objects", "--all", "--revs", "--stdout", "-q"])
.current_dir(work)
.stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::piped())
.spawn()
.expect("git pack-objects");
let pack = std::io::Read::bytes(child.stdout.take().expect("stdout"))
.collect::<std::io::Result<Vec<u8>>>()
.expect("read the pack");
assert!(child.wait().expect("pack-objects").success());
(pack, first, second)
}
/// **An incremental fetch is the walk's to answer until the store has a
/// reachability bitmap** (lane GUN-PERF-FETCH, 2026-10-31).
///
/// `GitServe::select`'s contract: `Ok(None)` means *"this engine cannot
/// answer cheaply — walk it yourself"*. `storage-git-gix` 0.3.0 answers
/// anyway when it has no `.bitmap`, with a walk that closes over the
/// client's whole `have` side — every object it holds. MEASURED on oden,
/// git.git (81 188 commits, 1 006 refs), one new commit fetched: **23 s**,
/// where gunnar's own walk (git's `limit_list` plus the edge) serves the
/// same request in **0.2 to 0.4 s**. So the server's handle declines for
/// it: a request with `have`s, on a store with no bitmap, goes to the walk.
/// A clone (no `have`) costs the same either way and stays with the
/// engine, and so does every request once `gc` has written a bitmap.
#[test]
fn a_fetch_with_haves_on_a_store_without_a_bitmap_is_left_to_the_walk() {
let dir = tempfile::tempdir().expect("tempdir");
let work = dir.path().join("work");
std::fs::create_dir_all(&work).expect("work dir");
let (pack, first, second) = two_commits(&work);
let git_dir = dir.path().join("repo.git");
let kind = gix_hash::Kind::Sha1;
let reader = open(&git_dir, kind).expect("opening the repository");
ingest(
&git_dir,
kind,
&mut pack.as_slice(),
crate::storage::ObjectSet::NotNeeded,
None,
None,
)
.expect("ingesting the pack");
let raw = |hex: &str| {
gix_hash::ObjectId::from_hex(hex.as_bytes())
.expect("oid")
.as_slice()
.to_vec()
};
let (first, second) = (raw(&first), raw(&second));
assert!(
reader
.serve()
.select(&[&second], &[&first])
.expect("select")
.is_none(),
"no bitmap and a `have`: the engine's answer is a closure over everything the client \
holds, so the server must walk instead"
);
assert!(
reader
.serve()
.select(&[&second], &[])
.expect("select")
.is_some(),
"a clone has no `have` side to close over; the engine answers it as before"
);
let bitmap = open_shared(&git_dir, kind)
.expect("the shared handle")
.write_reachability_bitmap()
.expect("write a bitmap");
assert!(
bitmap.is_some(),
"one pack, closed: a bitmap must be written"
);
assert!(
reader
.serve()
.select(&[&second], &[&first])
.expect("select")
.is_some(),
"with a bitmap the engine answers a fetch cheaply, and it is asked"
);
shutdown();
}
}