crates/storage-git-gix/examples/git-store-serve.rs
//! **The gix arm of gunnar's head-to-head benchmarks: a server binary.**
//!
//! The twin of `gunnar/crates/gunnar-coldtier/examples/git-store-serve.rs`,
//! which is the znippy arm. Same three modes, same argv, same
//! `GUNNAR_BENCH_COPY_STATS` receipt, so `bench/` needs no change at all: its
//! whole seam is `GUNNAR_BENCH_GUNNAR_SERVER=<path>`, and it knows a path, not a
//! crate.
//!
//! ```text
//! stock git ──ext:: over a pipe──► bench meter ──► THIS BINARY
//! │
//! upload_pack / receive_pack
//! │
//! storage_git_gix::GixStore
//! │
//! gix-odb / gix-ref on a filesystem
//! ```
//!
//! # The three modes
//!
//! | argv | what it does |
//! |---|---|
//! | `prepare <repo>` | build the gix store for `<repo>`, **outside any timed region** |
//! | `git-upload-pack <repo>` | serve a fetch/clone out of that store |
//! | `git-receive-pack <repo>` | receive a push into that store |
//!
//! The store lives beside the repository at `<repo>.gix`, a **sibling**, for the
//! same reason the znippy arm's does: the bench deletes and re-copies repository
//! directories between rounds, and a store inside one would be copied along with
//! it.
//!
//! `prepare` is a separate mode on purpose — building the store is ingest work,
//! and ingest work inside `git-upload-pack` would be charged to the clone it
//! precedes. A serve that finds no store **fails** rather than quietly building
//! one.
//!
//! # ⚠ READ THIS BEFORE QUOTING A RATIO — the wire halves are not identical
//!
//! The znippy arm serves through `gunnar_wire::{UploadPack, ReceivePack}`. **This
//! arm cannot**: `gunnar-wire` does not exist in edda, and a path dependency on a
//! sibling gunnar checkout is read at cargo *resolve* time, which is `P-006` and
//! is the very coupling `bench/` is detached to avoid. So the protocol driver
//! here is written on `gunnar-pktline` and `gunnar-sendpack`, and it is **mine,
//! not gunnar-wire's**.
//!
//! What that does and does not confound:
//!
//! * **The expensive half is shared.** Pack generation — the thing a clone's
//! clock is nearly all of — goes through `gunnar_pack::write_pack_copy`, the
//! same `P-001` pack-copy path with the same `Some(1)` thread limit and the
//! same `TrailerHasher`. `objects_copied_from_pack` and
//! `objects_recompressed` therefore mean exactly what they mean on the other
//! arm, which is why the receipt is comparable even when the clock is not.
//! * **The cheap half is not shared.** Advertisement, negotiation and framing
//! are this file's. On a clone that is a few hundred microseconds against a
//! multi-second transfer; on `vs_forge_*` arms that count round trips it is
//! not negligible.
//! * **This arm speaks protocol v0 only** (see [`upload_pack`]). If the bench
//! sets `GIT_PROTOCOL=version=2` the other arm may serve v2 and this one will
//! serve v0, and the negotiation round counts are then not comparable. This
//! binary prints the version it served and warns when v2 was asked for, so the
//! condition is visible rather than inferred.
//!
//! **Prefer the counts over the clocks**, which is `locked_on_target.md` §6's
//! rule anyway, and treat a wall-clock ratio from these two binaries as
//! store-plus-wire until gunnar-wire is reachable from both.
//!
//! # What this binary deliberately does NOT implement
//!
//! * `GUNNAR_BENCH_NO_PACK_COPY` — the bench's red switch for `P-001`. There is
//! no `NoPackCopy` shim here, so the binary **refuses to start** when it is
//! set. A switch that is silently ignored is the hollow-guard shape the bench
//! has been burned by twice.
//! * **`--depth` / shallow, and `filter`.** Refused loudly on the wire rather
//! than served as a full clone under a shallow label.
//! * **Thin packs.** `thin-pack` is not advertised, so `write_pack_copy` is
//! called and never `write_pack_copy_thin`. A clone copies 100% of its entries
//! either way (`S-003`); a *fetch* pays for this, and the receipt says so by
//! showing a non-zero `recompressed`.
use std::io::{BufWriter, Read, Write};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::time::Instant;
use git_storage_trait::{GitOps, RefUpdate};
use gix_hash::ObjectId;
use gunnar_pack::{write_pack_copy, PackedObjects, WriteOptions};
use gunnar_pktline::{read_packet, Packet, BAND_PACK, SIDEBAND_CHUNK};
use gunnar_sendpack::{advertisement, capabilities, command, report, RefStatus, RemoteRef};
use storage_git_gix::GixStore;
type Res<T> = Result<T, Box<dyn std::error::Error>>;
/// The store that backs `repo`. A sibling, not a child — see the module doc.
fn store_root(repo: &Path) -> PathBuf {
let mut s = repo.as_os_str().to_os_string();
s.push(".gix");
PathBuf::from(s)
}
/// The repository's hash algorithm, read from its own config.
///
/// A repository served under the wrong hash is not a degraded repository, it is
/// a different one. Same discipline as the znippy arm's `hash_kind`.
fn hash_kind(repo: &Path) -> gix_hash::Kind {
let config = std::fs::read_to_string(repo.join("config")).unwrap_or_default();
if config.to_ascii_lowercase().contains("sha256") {
gix_hash::Kind::Sha256
} else {
gix_hash::Kind::Sha1
}
}
fn open(repo: &Path) -> Res<GixStore> {
Ok(GixStore::open_or_init(store_root(repo), hash_kind(repo))?)
}
// ─────────────────────────────────────────────────────────────── prepare ────
/// Build `<repo>.gix` from `<repo>`, or confirm the one that is there.
///
/// Idempotent: the clone scenario prepares one server repository once and the
/// push scenario prepares a fresh copy every round.
///
/// # What is ingested, and why it is the honest input
///
/// The repository's own `objects/pack/*.pack`, byte for byte, through
/// `GitOps::put_pack` — the same call a real push lands in. The bench fixtures
/// are `git repack -adq`-ed, so those bytes are what stock `git pack-objects`
/// wrote at git's own window and depth, and are what a forge would have
/// received.
///
/// # ⚠ One difference from the znippy arm, and it is forced
///
/// The znippy arm ingests each `.pack` verbatim. This arm cannot always: gitoxide
/// **refuses to index a pack whose `OBJ_REF_DELTA` bases live inside that same
/// pack** (its thin-pack resolver consults only the external base source), which
/// is the gap `tests/round_trip.rs::ref_delta_with_an_in_pack_base_is_a_known_gap`
/// pins. A fixture packed without `--delta-base-offset` is exactly that shape.
/// So the verbatim path is tried first and, if gitoxide refuses it, the
/// repository is re-packed **into a stream** with `--delta-base-offset`, without
/// touching the fixture directory. Which path was taken is printed, because a
/// re-pack is a different input and a ratio taken across two different inputs is
/// not a ratio.
fn prepare(repo: &Path) -> Res<()> {
let root = store_root(repo);
let started = Instant::now();
let store = open(repo)?;
// Already built? Ask the pack set, not the directory: `open_or_init` creates
// the directory itself, so its existence says nothing about ingest.
let held = store.refs()?.len();
if held > 0 {
eprintln!(
"gix-prepare: {} already holds {held} refs",
root.display()
);
return Ok(());
}
let pack_dir = repo.join("objects").join("pack");
let mut packs: Vec<PathBuf> = Vec::new();
if pack_dir.is_dir() {
for entry in std::fs::read_dir(&pack_dir)? {
let path = entry?.path();
if path.extension().is_some_and(|e| e == "pack") {
packs.push(path);
}
}
}
packs.sort();
let loose = loose_object_count(repo);
let mut pack_bytes = 0u64;
let ingested_from;
let verbatim = !packs.is_empty() && loose == 0 && ingest_verbatim(&store, &packs).is_ok();
if verbatim {
for p in &packs {
pack_bytes += std::fs::metadata(p)?.len();
}
ingested_from = format!("{} verbatim pack(s)", packs.len());
} else {
if packs.is_empty() && loose == 0 {
return Err(format!("{} holds no objects at all", repo.display()).into());
}
let why = if packs.is_empty() {
format!("{loose} loose object(s)")
} else {
format!(
"{} pack(s) gitoxide would not index verbatim (in-pack ref-delta bases)",
packs.len()
)
};
ingested_from = format!("{why}, re-packed into a stream with --delta-base-offset");
let bytes = pack_objects(repo)?;
pack_bytes += bytes.len() as u64;
store.put_pack(&bytes)?;
}
// ── refs, through `git for-each-ref` ────────────────────────────────────
//
// Not a walk of `refs/`: packed refs are the case a hand-rolled reader
// misses first, and git's own reader is the definition of what this
// repository contains. Identical to the znippy arm's `read_refs`.
let edits = read_refs(repo)?;
if edits.is_empty() {
return Err(format!("{} has no references under refs/", repo.display()).into());
}
let count = edits.len();
let updates: Vec<RefUpdate> = edits
.iter()
.map(|(name, oid)| RefUpdate::set(name, oid))
.collect();
store.put_refs(&updates)?;
write_head(&root, repo)?;
let after = store.refs()?.len();
if after == 0 {
return Err("the store holds no refs after ingest".into());
}
// ── the reachability bitmap ─────────────────────────────────────────────
//
// Here, because `prepare` is by construction the step that runs **outside
// any timed region**, and because a forge that has ingested a repository and
// not yet run maintenance over it is a forge mid-setup rather than a forge.
// The gitea and Forgejo columns of `vs_forge_*` serve out of stores that
// have bitmaps; a gix column serving without one is not the same question
// asked twice, it is a walk measured against two bitmaps.
//
// `Ok(None)` is a real answer and is printed as one: a pack whose closure
// leaves it gets no bitmap and is served by walking, exactly as before.
// Printed either way, because which path a clone will take is not visible in
// the clone's own output and a row that does not say it is a row about
// nothing (`P-027`).
let bitmap_started = Instant::now();
let bitmap = match store.write_reachability_bitmap() {
Ok(Some(path)) => format!(
", bitmap {} bytes in {:.1} ms",
std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0),
bitmap_started.elapsed().as_secs_f64() * 1000.0
),
Ok(None) => ", NO BITMAP (this store will be served by walking)".to_string(),
// Not fatal. A store without a bitmap serves correctly and slowly, and
// failing `prepare` over it would turn a lost optimisation into a lost
// benchmark arm.
Err(err) => format!(", bitmap FAILED: {err}"),
};
eprintln!(
"gix-prepare: {} <- {ingested_from}, {pack_bytes} bytes, {count} refs{bitmap}, {:.1} ms",
root.display(),
started.elapsed().as_secs_f64() * 1000.0
);
Ok(())
}
/// Try to land every `.pack` byte for byte. `Err` means gitoxide declined one.
///
/// All-or-nothing by intent: a partial ingest leaves a store that serves clones
/// which are quietly missing objects, and a clone that omits objects still exits
/// 0. The caller falls back to a whole re-pack, never to "the packs that worked".
fn ingest_verbatim(store: &GixStore, packs: &[PathBuf]) -> Res<()> {
for p in packs {
let bytes = std::fs::read(p)?;
store.put_pack(&bytes)?;
}
Ok(())
}
/// Every reference in `repo` as `(name, raw oid)`, symrefs resolved.
///
/// `git for-each-ref`, for the reason `prepare` gives.
fn read_refs(repo: &Path) -> Res<Vec<(String, String)>> {
let out = Command::new("git")
.arg("-C")
.arg(repo)
.args(["for-each-ref", "--format=%(refname)%00%(objectname)"])
.stderr(Stdio::inherit())
.output()?;
if !out.status.success() {
return Err(format!("git for-each-ref in {} failed", repo.display()).into());
}
let mut refs = Vec::new();
for line in String::from_utf8_lossy(&out.stdout).lines() {
let mut parts = line.split('\0');
if let (Some(name), Some(oid)) = (parts.next(), parts.next()) {
if !name.is_empty() && !oid.is_empty() {
refs.push((name.to_owned(), oid.to_owned()));
}
}
}
Ok(refs)
}
/// Mirror `<repo>`'s `HEAD` into the store.
///
/// Written as a file rather than through `GitOps`, because `HEAD` is a
/// pseudo-ref and the frozen contract's `refs()`/`update_ref` deliberately do
/// not carry one. A clone needs it: without a symbolic `HEAD` git picks the
/// default branch by guesswork.
fn write_head(root: &Path, repo: &Path) -> Res<()> {
let sym = Command::new("git")
.arg("-C")
.arg(repo)
.args(["symbolic-ref", "--quiet", "HEAD"])
.stderr(Stdio::null())
.output()?;
if sym.status.success() {
let name = String::from_utf8_lossy(&sym.stdout).trim().to_string();
std::fs::write(root.join("HEAD"), format!("ref: {name}\n"))?;
return Ok(());
}
let direct = Command::new("git")
.arg("-C")
.arg(repo)
.args(["rev-parse", "--verify", "--quiet", "HEAD"])
.stderr(Stdio::null())
.output()?;
if direct.status.success() {
let oid = String::from_utf8_lossy(&direct.stdout).trim().to_string();
std::fs::write(root.join("HEAD"), format!("{oid}\n"))?;
}
Ok(())
}
/// Every reachable object in `repo`, as one packfile.
///
/// `rev-list --objects --all` then `pack-objects --stdout
/// --delta-base-offset`, git's own two step. The flag is not optional here —
/// see [`prepare`].
fn pack_objects(repo: &Path) -> Res<Vec<u8>> {
let listed = Command::new("git")
.arg("-C")
.arg(repo)
.args(["rev-list", "--objects", "--all"])
.stderr(Stdio::inherit())
.output()?;
if !listed.status.success() {
return Err(format!("git rev-list in {} failed", repo.display()).into());
}
let mut oids = String::new();
for line in String::from_utf8_lossy(&listed.stdout).lines() {
if let Some(oid) = line.split_whitespace().next() {
oids.push_str(oid);
oids.push('\n');
}
}
// An empty list and a repository with no objects produce the same packfile,
// and only one of them is a thing to store.
if oids.is_empty() {
return Err(format!("git rev-list named no objects in {}", repo.display()).into());
}
let mut child = Command::new("git")
.arg("-C")
.arg(repo)
.args(["pack-objects", "--stdout", "--delta-base-offset", "-q"])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::inherit())
.spawn()?;
child
.stdin
.take()
.ok_or("git pack-objects has no stdin")?
.write_all(oids.as_bytes())?;
let out = child.wait_with_output()?;
if !out.status.success() {
return Err(format!(
"git pack-objects in {} exited {:?}",
repo.display(),
out.status.code()
)
.into());
}
if out.stdout.len() < 32 {
return Err(format!(
"git pack-objects in {} produced {} bytes, which is not a packfile",
repo.display(),
out.stdout.len()
)
.into());
}
Ok(out.stdout)
}
/// Loose objects under `objects/`, i.e. everything that is not a pack.
fn loose_object_count(repo: &Path) -> usize {
let dir = repo.join("objects");
let Ok(entries) = std::fs::read_dir(&dir) else {
return 0;
};
let mut n = 0;
for entry in entries.flatten() {
let name = entry.file_name();
let name = name.to_string_lossy();
if name.len() == 2 && name.chars().all(|c| c.is_ascii_hexdigit()) {
if let Ok(inner) = std::fs::read_dir(entry.path()) {
n += inner.flatten().count();
}
}
}
n
}
// ────────────────────────────────────────────────────────── the receipt ────
/// Append the `copied`/`recompressed` receipt, in the **byte-for-byte** shape
/// the znippy arm writes and the bench meter reads.
///
/// A differing receipt would make the two columns incomparable, which is the
/// whole point of the exercise, so this is a copy of the other arm's `writeln!`
/// and not a re-derivation of it.
fn copy_stats(copied: usize, recompressed: usize) -> Res<()> {
if let Ok(path) = std::env::var("GUNNAR_BENCH_COPY_STATS") {
let mut f = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(path)?;
writeln!(
f,
"{{\"copied\":{},\"recompressed\":{}}}",
copied as f64, recompressed as f64
)?;
}
Ok(())
}
// ───────────────────────────────────────────────────────── upload-pack ────
/// `git-upload-pack`, **protocol v0 only**.
///
/// # Why v0 and not v2
///
/// v0 is what git's `ext::` transport reaches by default — the transport passes
/// `GIT_EXT_SERVICE` and `GIT_DIR` but **not** `GIT_PROTOCOL` (measured on git
/// 2.53.0 by the quarantined `git_harness`), so a v2 grammar would sit unused
/// behind an advertisement the client never asked for. A server is free to
/// answer a v2 request with a v0 advertisement and every git client falls back
/// correctly, so this is a narrowing rather than a defect — but it is one that
/// changes round-trip counts, so it is printed and warned about.
///
/// # The conversation
///
/// ```text
/// S: <oid> HEAD\0<caps> S: <oid> refs/heads/main S: 0000
/// C: want <oid> <caps> C: want <oid> C: 0000
/// C: have <oid> … C: 0000 → S: ACK <oid> or NAK
/// C: done → S: ACK <oid> / NAK, then the pack
/// ```
///
/// Single-ack, because `multi_ack` is not advertised. Every git client speaks
/// it; it costs a fetch one extra round trip against `multi_ack_detailed` and
/// costs a **clone** — the arm that matters — nothing at all, because a clone
/// sends no `have` lines.
fn upload_pack(repo: &Path) -> Res<()> {
let store = open(repo)?;
let kind = hash_kind(repo);
let root = store_root(repo);
let rows = store.refs()?;
if rows.is_empty() {
return Err(format!(
"{} holds no refs; run `{} prepare {}` before measuring",
root.display(),
std::env::args().next().unwrap_or_default(),
repo.display()
)
.into());
}
if let Ok(v) = std::env::var("GIT_PROTOCOL") {
if v.contains("version=2") {
eprintln!(
"gix: WARNING — the client asked for {v:?} and this arm serves v0. Round-trip \
counts are not comparable with an arm that served v2."
);
}
}
let stdin = std::io::stdin();
let mut input = stdin.lock();
let stdout = std::io::stdout();
let mut out = BufWriter::new(stdout.lock());
// ── advertise ───────────────────────────────────────────────────────────
//
// `HEAD` first, because git clone reads the default branch off the first
// advertised ref when it cannot resolve one otherwise. Peeled annotated
// tags follow their tag as `<name>^{}`, which is how a client learns a tag's
// commit without fetching the tag object first.
let caps: Vec<String> = vec![
"side-band-64k".into(),
"ofs-delta".into(),
"no-progress".into(),
format!("object-format={}", advertisement::object_format_name(kind)?),
format!("agent=storage-git-gix/{}", env!("CARGO_PKG_VERSION")),
];
let mut adv: Vec<RemoteRef> = Vec::with_capacity(rows.len() + 1);
if let Some(head) = head_target(&root, &rows) {
adv.push(RemoteRef::new("HEAD", head));
}
for row in &rows {
let Some(raw) = &row.oid else { continue };
let Ok(id) = ObjectId::try_from(raw.as_slice()) else {
continue;
};
adv.push(RemoteRef::new(row.name.clone(), id));
if let Some(peeled) = &row.peeled {
if let Ok(p) = ObjectId::try_from(peeled.as_slice()) {
adv.push(RemoteRef::new(format!("{}^{{}}", row.name), p));
}
}
}
for line in advertisement::lines(&adv, &caps, kind) {
gunnar_pktline::write_text(&mut out, &line)?;
}
gunnar_pktline::write_flush(&mut out)?;
out.flush()?;
// ── wants ───────────────────────────────────────────────────────────────
let mut wants: Vec<ObjectId> = Vec::new();
let mut client_caps: Vec<String> = Vec::new();
// A hang-up and a flush are the same statement here: "I have read your
// advertisement and I am asking for nothing."
while let Some(pkt) = read_packet(&mut input)? {
let Packet::Data(line) = pkt else { break };
let line = trim_nl(&line);
if line.is_empty() {
continue;
}
if let Some(rest) = line.strip_prefix(b"want ") {
let (oid, caps) = split_once_space(rest);
if client_caps.is_empty() {
if let Some(c) = caps {
client_caps = String::from_utf8_lossy(c)
.split_whitespace()
.map(str::to_owned)
.collect();
}
}
wants.push(ObjectId::from_hex(oid)?);
} else if line.starts_with(b"deepen") || line.starts_with(b"shallow ") {
// Refused rather than served whole: a full clone reported under a
// shallow label is a wrong number that looks exactly like a right
// one.
return Err("this arm does not implement shallow (`deepen`/`shallow`)".into());
} else if line.starts_with(b"filter ") {
return Err("this arm does not implement `filter`".into());
}
}
if wants.is_empty() {
// **Not an error.** `git ls-remote` reads the advertisement and sends a
// bare flush; so does a `git fetch` that finds it is already up to date.
// Both are complete, successful conversations, and failing them was a
// real bug in the first draft of this file — `ls-remote` printed the
// right refs and exited 1, which the bench would have counted as a
// failed arm on a server that had just answered correctly.
eprintln!("gix: served git-upload-pack v0 advertisement only, no wants ({} refs)", adv.len());
return Ok(());
}
// ── haves, single-ack ───────────────────────────────────────────────────
let mut haves: Vec<ObjectId> = Vec::new();
let mut common: Option<ObjectId> = None;
let mut rounds = 0usize;
let mut done = false;
while !done {
rounds += 1;
let mut got_any = false;
loop {
let Some(pkt) = read_packet(&mut input)? else {
done = true;
break;
};
match pkt {
Packet::Data(line) => {
let line = trim_nl(&line);
if let Some(oid) = line.strip_prefix(b"have ") {
let id = ObjectId::from_hex(oid)?;
got_any = true;
if store.has(id.as_bytes())? {
haves.push(id);
if common.is_none() {
common = Some(id);
}
}
} else if line == b"done" {
done = true;
break;
}
}
// End of this batch of `have` lines.
Packet::Flush => break,
_ => {}
}
}
if done {
break;
}
if !got_any {
// A flush with no `have` in it is the client saying it has nothing.
break;
}
match &common {
Some(id) => writeln_pkt(&mut out, &format!("ACK {}", id.to_hex()))?,
None => writeln_pkt(&mut out, "NAK")?,
}
out.flush()?;
}
match &common {
Some(id) => writeln_pkt(&mut out, &format!("ACK {}", id.to_hex()))?,
None => writeln_pkt(&mut out, "NAK")?,
}
out.flush()?;
// ── selection, then the pack ────────────────────────────────────────────
let started = Instant::now();
let want_raw: Vec<&[u8]> = wants.iter().map(|id| id.as_bytes()).collect();
let have_raw: Vec<&[u8]> = haves.iter().map(|id| id.as_bytes()).collect();
let selected = store.reachable(&want_raw, &have_raw)?;
let ids: Vec<ObjectId> = selected
.iter()
.map(|b| ObjectId::try_from(b.as_slice()))
.collect::<std::result::Result<_, _>>()?;
// `P-001`, and the same call the other arm's pack generation reaches.
// `prevent_pack_unload` is mandatory: `entry_by_location` hands out slices
// into a mapped pack and gix panics rather than let a caller read memory it
// may have unmapped underneath them.
let store_arc = store.odb_store();
let packed = PackedObjects::new(move || {
let mut handle = store_arc.to_handle_arc();
handle.prevent_pack_unload();
Box::new(gix_odb::Cache::from(handle)) as Box<dyn gix_pack::Find + Send>
});
let ofs = client_caps.iter().any(|c| c == "ofs-delta");
let opts = WriteOptions::new(kind).ofs_delta(ofs);
let sideband = client_caps.iter().any(|c| c == "side-band-64k");
let outcome = if sideband {
let mut sink = Sideband::new(&mut out);
let o = write_pack_copy(&ids, &packed, &mut sink, &opts)?;
sink.finish()?;
gunnar_pktline::write_flush(&mut out)?;
o
} else {
write_pack_copy(&ids, &packed, &mut out, &opts)?
};
out.flush()?;
let (probes, rescans) = store.rescan_counts();
eprintln!(
"gix: served git-upload-pack v0 rounds={rounds} objects={} copied={} recompressed={} \
bytes={} ofs_delta={ofs} sideband={sideband} rescan_probes={probes} rescans={rescans} \
{:.1} ms",
outcome.num_objects,
outcome.objects_copied_from_pack,
outcome.objects_recompressed,
outcome.bytes_written,
started.elapsed().as_secs_f64() * 1000.0,
);
copy_stats(
outcome.objects_copied_from_pack,
outcome.objects_recompressed,
)?;
Ok(())
}
/// What `HEAD` points at, resolved through the advertised rows.
///
/// The store's `HEAD` file is written by [`write_head`]; a symbolic one is
/// resolved against `rows` rather than re-read through gix, because the rows are
/// already in hand and are exactly what the client is about to be told.
fn head_target(root: &Path, rows: &[git_storage_trait::RefRow]) -> Option<ObjectId> {
let text = std::fs::read_to_string(root.join("HEAD")).ok()?;
let text = text.trim();
if let Some(name) = text.strip_prefix("ref: ") {
let row = rows.iter().find(|r| r.name == name)?;
return ObjectId::try_from(row.oid.as_deref()?).ok();
}
ObjectId::from_hex(text.as_bytes()).ok()
}
/// A `Write` that frames everything through it as sideband band-1 pkt-lines.
///
/// The subtraction that makes a band byte fit is [`SIDEBAND_CHUNK`]'s, made once
/// in `gunnar-pktline` so no caller has to remember it.
struct Sideband<'a, W: Write> {
out: &'a mut W,
buf: Vec<u8>,
}
impl<'a, W: Write> Sideband<'a, W> {
fn new(out: &'a mut W) -> Self {
Sideband {
out,
buf: Vec::with_capacity(SIDEBAND_CHUNK),
}
}
fn emit(&mut self) -> std::io::Result<()> {
if !self.buf.is_empty() {
gunnar_pktline::write_sideband(self.out, BAND_PACK, &self.buf)
.map_err(std::io::Error::other)?;
self.buf.clear();
}
Ok(())
}
/// Flush the tail. **Not** `Drop`: a short write here truncates the pack and
/// a `Drop` that cannot return an error would swallow it.
fn finish(mut self) -> std::io::Result<()> {
self.emit()
}
}
impl<W: Write> Write for Sideband<'_, W> {
fn write(&mut self, data: &[u8]) -> std::io::Result<usize> {
let room = SIDEBAND_CHUNK - self.buf.len();
let take = room.min(data.len());
self.buf.extend_from_slice(&data[..take]);
if self.buf.len() == SIDEBAND_CHUNK {
self.emit()?;
}
Ok(take)
}
fn flush(&mut self) -> std::io::Result<()> {
self.emit()?;
self.out.flush()
}
}
// ──────────────────────────────────────────────────────── receive-pack ────
/// `git-receive-pack`, protocol v0.
///
/// The grammar is `gunnar-sendpack`'s server half — one implementation with two
/// entry points, so the encoder and decoder here cannot drift from the client
/// gunnar itself pushes with.
fn receive_pack(repo: &Path) -> Res<()> {
let store = open(repo)?;
let kind = hash_kind(repo);
let stdin = std::io::stdin();
let mut input = stdin.lock();
let stdout = std::io::stdout();
let mut out = BufWriter::new(stdout.lock());
// ── advertise ───────────────────────────────────────────────────────────
let caps: Vec<String> = vec![
"report-status".into(),
"delete-refs".into(),
"ofs-delta".into(),
format!("object-format={}", advertisement::object_format_name(kind)?),
format!("agent=storage-git-gix/{}", env!("CARGO_PKG_VERSION")),
];
let rows = store.refs()?;
let mut adv: Vec<RemoteRef> = Vec::with_capacity(rows.len());
for row in &rows {
if let Some(raw) = &row.oid {
if let Ok(id) = ObjectId::try_from(raw.as_slice()) {
adv.push(RemoteRef::new(row.name.clone(), id));
}
}
}
for line in advertisement::lines(&adv, &caps, kind) {
gunnar_pktline::write_text(&mut out, &line)?;
}
gunnar_pktline::write_flush(&mut out)?;
out.flush()?;
// ── commands ────────────────────────────────────────────────────────────
let mut commands = Vec::new();
let mut report_status = false;
let mut first = true;
while let Some(pkt) = read_packet(&mut input)? {
let Packet::Data(line) = pkt else { break };
let line = trim_nl(&line);
if line.is_empty() {
continue;
}
let (payload, caps) = if first {
capabilities::split(line)
} else {
(line, None)
};
if first {
if let Some(c) = caps {
report_status = String::from_utf8_lossy(c)
.split_whitespace()
.any(|w| w == "report-status" || w == "report-status-v2");
}
first = false;
}
if payload.starts_with(command::SHALLOW_PREFIX) {
return Err("this arm does not implement shallow push".into());
}
commands.push(command::parse_line(payload, kind)?);
}
if commands.is_empty() {
// A push with nothing in it. Legal, and a complete conversation.
eprintln!("gix: served git-receive-pack, no commands");
return Ok(());
}
// ── the pack, if any command creates or updates ─────────────────────────
//
// Raw bytes, not pkt-lines: the packfile follows the flush that ends the
// command list, unframed, to EOF. A delete-only push sends none.
let mut unpack = String::from("ok");
let mut indexed = 0u32;
if commands.iter().any(|c| !c.is_delete()) {
let mut pack = Vec::new();
input.read_to_end(&mut pack)?;
if pack.is_empty() {
unpack = "the client sent no packfile".into();
} else {
match store.put_pack(&pack) {
Ok(_) => indexed = 1,
// Verbatim, and through `{:#}`: gunnar-pack's top-level Display
// is "failed to index the pack" and every useful word is in its
// `source()` chain. This string reaches the pushing client, and
// a client told only "failed" cannot act.
Err(err) => unpack = format!("{err:#}"),
}
}
}
// ── apply the refs, one CAS each ────────────────────────────────────────
let mut statuses = Vec::with_capacity(commands.len());
for c in &commands {
if unpack != "ok" {
statuses.push(RefStatus::rejected(
c.name.clone(),
"the packfile was not indexed",
));
continue;
}
let name = String::from_utf8_lossy(c.name.as_slice()).into_owned();
let old = (!c.old.is_null()).then(|| c.old.as_bytes().to_vec());
let new = (!c.new.is_null()).then(|| c.new.as_bytes().to_vec());
match store.update_ref(&name, old.as_deref(), new.as_deref()) {
Ok(_) => statuses.push(RefStatus::ok(c.name.clone())),
Err(err) => statuses.push(RefStatus::rejected(c.name.clone(), format!("{err:#}"))),
}
}
if report_status {
let unpack_result = if unpack == "ok" {
Ok(())
} else {
Err(unpack.as_str())
};
for line in report::lines(unpack_result, &statuses) {
gunnar_pktline::write_text(&mut out, &line)?;
}
gunnar_pktline::write_flush(&mut out)?;
}
out.flush()?;
let ok = statuses.iter().filter(|s| s.accepted).count();
eprintln!(
"gix: served git-receive-pack commands={} accepted={ok} packs_indexed={indexed} \
unpack={unpack}",
commands.len()
);
Ok(())
}
// ─────────────────────────────────────────────────────────────── helpers ────
fn trim_nl(line: &[u8]) -> &[u8] {
line.strip_suffix(b"\n").unwrap_or(line)
}
fn split_once_space(s: &[u8]) -> (&[u8], Option<&[u8]>) {
match s.iter().position(|&b| b == b' ') {
Some(i) => (&s[..i], Some(&s[i + 1..])),
None => (s, None),
}
}
fn writeln_pkt<W: Write>(out: &mut W, text: &str) -> Res<()> {
gunnar_pktline::write_text(out, text.as_bytes())?;
Ok(())
}
/// Ask the store about one oid and print the **error**, not the `false` a
/// boolean is forced to collapse it into.
///
/// A diagnostic, never part of a measurement. The znippy arm has the same mode
/// and for the same reason: a push refused with "missing necessary objects" and
/// a swallowed lookup error look identical from the client's end.
fn has(repo: &Path, oid: &str) -> Res<()> {
let store = open(repo)?;
let id = ObjectId::from_hex(oid.as_bytes())?;
let (probes, rescans) = store.rescan_counts();
let (slots, regrows) = store.slot_table();
println!(
"refs={} slots={slots} regrows={regrows} rescan_probes={probes} rescans={rescans} \
has({oid})={:?}",
store.refs()?.len(),
store.has(id.as_bytes())
);
Ok(())
}
fn main() {
let args: Vec<String> = std::env::args().collect();
let mode = args.get(1).cloned().unwrap_or_default();
let repo = PathBuf::from(args.get(2).cloned().unwrap_or_default());
// See the module doc: a red switch this binary cannot honour must stop it,
// not be ignored by it.
if std::env::var_os("GUNNAR_BENCH_NO_PACK_COPY").is_some() && mode != "prepare" {
eprintln!(
"git-store-serve: GUNNAR_BENCH_NO_PACK_COPY is set, and this server has no \
NoPackCopy shim to honour it with. Serving anyway would report a switched run's \
label over an unswitched server."
);
std::process::exit(1);
}
let result = match mode.as_str() {
"prepare" => prepare(&repo),
"has" => has(&repo, args.get(3).map(String::as_str).unwrap_or_default()),
"git-upload-pack" => upload_pack(&repo),
"git-receive-pack" => receive_pack(&repo),
other => Err(format!(
"usage: {} prepare|has|git-upload-pack|git-receive-pack <repo> (got {other:?})",
args.first().map(String::as_str).unwrap_or("git-store-serve")
)
.into()),
};
if let Err(err) = result {
eprintln!("git-store-serve: {err}");
std::process::exit(1);
}
}