crates/gunnar-pack/src/copy.rs
//! **`P-001`: serving a pack by copying entries that are already deflated.**
//!
//! # What this deletes
//!
//! [`write_pack`](crate::write_pack) reads every object's *inflated* content
//! and deflates it again from scratch. For an object that is already sitting
//! deflated inside a packfile that is a complete inflate → deflate round trip
//! to reproduce bytes gunnar already had. Under `TOTAL-PROMPT` §4's design
//! that is every object in the incoming tier, because a push is stored as the
//! packfile the client sent. Worse, the pushing client's git ran a
//! full delta search before uploading, and re-deflating whole objects throws
//! every one of those deltas away.
//!
//! MEASURED head-to-head against stock `git 2.53.0`, oden, 2026-08-03, same
//! repository (528 MiB of blob content over 2048 churned commits, 36 172
//! objects): gunnar served **108.70 MB** for git's **5.93 MB**, 18.34×, and
//! spent **4.875 s** of server CPU for git's **0.022 s**, 219.96×. git wins
//! that by shipping deltas it already stored. **The 18.34× is the delta-reuse
//! gap, and this module is the whole of the fix.**
//!
//! # The fix was already written, in a crate gunnar already links
//!
//! `gix_pack::data::output::entry::iter_from_counts` has exactly one [`Mode`]
//! variant, `PackCopyAndBaseObjects`, which reads like a limitation and is the
//! opposite of one: `output::Entry::from_pack_entry` copies an existing pack
//! entry's compressed bytes verbatim and re-points its `OfsDelta` header, so
//! there is no inflate, no deflate and the delta survives. Only `RefDelta` is
//! downgraded to a base, and only objects with no pack residency are deflated
//! fresh. gunnar writes no delta encoder, gitoxide has none (`P-009`), and
//! none is needed, because the client already paid for the search.
//!
//! # ⚠ `P-020` — why gix produces the entries and gunnar emits the bytes
//!
//! `P-001`'s own Fix step says to finish the pipeline with
//! `output::bytes::FromEntriesIter`. **Do not, on the serve path.** That type
//! holds its sink as a private `output: gix_hash::io::Write<W>` field built
//! inside `new()`, and `gix_hash::io::Write::new` hardcodes
//! `gix_hash::hasher(object_hash)`, which is `sha1_checked` with UBC collision
//! detection on. There is no constructor, builder or field access that admits a
//! different hasher, so adopting it would revert `P-008`'s [`TrailerHasher`],
//! and revert it in exactly the shape where `P-008` measured **4.02×** rather than 1.06×,
//! because pack-copy removes the deflate that used to dominate the write.
//!
//! Nothing would go red. A pack written with either hasher is byte-identical
//! absent an attack, which is precisely what `tests/trailer_hash.rs` asserts.
//! Only the price would change.
//!
//! So: gix for **entry production**, which is the hard, generic, delta-aware
//! half; gunnar's own [`PackWriter`] for **byte emission**, because it already
//! owns the right trailer hasher. Every type crossing that seam is public and
//! no fork is involved. LAW 5 is satisfied: what is kept is a 60-line emitter
//! that exists for a measured reason.
//!
//! The **batch** tier makes the opposite call: a repack is one pass over a
//! repository, off the request path, so it accepts the sha1dc trailer rather
//! than carry a second emitter. Serve is a request, and cannot.
//!
//! # ⚠ `P-020`, second half — the serve path must not fan out
//!
//! `iter_from_counts` fans out through `gix_parallel::in_parallel_if`
//! whenever `counts.len() > 4_000`, a threshold nearly every real clone
//! exceeds, and `Options::thread_limit` decides how wide. Left at `None` it
//! takes the machine's core count **per request**, which `P-004` settled
//! against: gunnar has no shared pool, so per-request fan-out at forge
//! concurrency spawns tens of thousands of threads. This module passes
//! `Some(1)` and `xtask/tests/pack_pipeline.rs` fails the build if it ever
//! stops. The batch tier is where the pool belongs.
//!
//! [`Mode`]: gix_pack::data::output::entry::iter_from_counts::Mode
//! [`TrailerHasher`]: crate::checksum::TrailerHasher
//! [`PackWriter`]: crate::PackWriter
use std::collections::HashMap;
use std::io::Write;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use gix_hash::ObjectId;
use gix_pack::data::entry::Location;
use gix_pack::data::output;
use crate::write::prepare_ids;
use crate::{Error, PackWriter, Result, WriteOptions, WriteOutcome};
/// How many objects the generator hands over at a time.
///
/// This is the memory ceiling of the whole path and the reason it is named
/// rather than left at gix's default of 10. An in-flight chunk holds each
/// entry's **compressed** bytes, and with `thread_limit: Some(1)` at most three
/// chunks exist at once (one being built, one in the bounded channel, one being
/// written). gix then re-derives its own chunk size from the count anyway and
/// clamps it to 1000, so this is an upper bound rather than a request.
const CHUNK_SIZE: usize = 1000;
/// A packfile's header: `PACK`, a 4-byte version, a 4-byte object count.
///
/// Read to hold an index's per-pack offset list to what the pack itself says it
/// holds. See [`PackedObjects::objects_in_pack`].
const PACK_HEADER_BYTES: usize = 12;
// ── `P-025`: an entry's length comes from the index, not from inflating it ───
/// One pack's entry extents, read out of its **index** rather than inflated out
/// of its data.
///
/// # The inflate this deletes
///
/// `iter_from_counts` resolves every count through [`gix_pack::Find::location_by_oid`]
/// before it produces a single entry, and `gix-odb`'s implementation of that
/// method (`store_impls/dynamic/find.rs`) does this per object:
///
/// ```text
/// buf.resize(entry.decompressed_size, 0);
/// pack.decompress_entry(&entry, &mut inflate, buf) // ← the whole object
/// .map(|entry_size_past_header| Location { … }) // ← and only the LENGTH is kept
/// ```
///
/// It inflates the object in full, zeroing a buffer the size of the inflated
/// object first, and keeps nothing but *how many compressed bytes that took*.
/// On `write_pack_copy`'s path the inflated bytes are then never looked at
/// again — the entry is copied compressed. So a clone inflates the entire
/// repository in order to compute byte ranges it is about to `memcpy` around.
///
/// MEASURED oden 2026-08-04, `perf record -F 4999` over a whole-repository
/// clone of the 19-member corpus's `znippy.git` through `stage-probe fetch`:
/// `zlib_rs::inflate*` held **51.3%** of the profile (36.70 + 7.99 + 7.28) with
/// `sha1::compress` — the trailer, which is irreducible — a distant second at
/// 12.95%, and page faults from the zeroing resize another 15.19% inclusive.
///
/// # What replaces it, and why it is exact rather than approximate
///
/// Pack entries are contiguous: entry *n* ends where entry *n+1* begins. So an
/// entry's on-disk length is the distance to the **next offset in the pack**,
/// which the pack index already holds and which costs no inflate at all. That
/// is not a trick, it is what git's own `pack-revindex.c` exists for, and
/// `builtin/pack-objects.c:write_reuse_object` sizes a reused entry exactly this
/// way (`revidx[1].offset - offset`).
///
/// [`gix_pack::Find::pack_offsets_and_oid`] hands over every `(offset, oid)`
/// pair of a pack in one read of the `.idx`, which is all this needs.
///
/// # The three cases it declines
///
/// * **the last entry in a pack** has no successor, so its length is the
/// distance to the pack trailer — and the trait gives no way to ask how long
/// the pack file is. It falls back to `gix-odb`, at one inflate per pack.
/// * **a pack whose offsets are not strictly ascending** is not a pack this can
/// reason about, and `install` refuses it rather than emitting an entry of the
/// wrong length. `tests/pack_extents.rs` drives that refusal red.
/// * **a pack whose offset list is short of what the pack holds.** "The next
/// offset in the pack" is only the end of an entry if *every* offset is in
/// hand; a list with a hole measures one entry across its neighbour. See
/// [`PackedObjects::objects_in_pack`], which is what closes that.
/// * **any pack that is not the one `gix-odb` would have searched first.** See
/// [`PackedObjects::front`]; this is the condition that keeps the bytes
/// identical when an object lives in two packs, and it is the one that was
/// missing from the first version of this and produced a 53-byte difference
/// on a two-pack fixture.
///
/// Everything else — every object but one per pack, on every request — costs a
/// binary search.
struct PackExtents {
/// `(oid, pack_offset)`, ascending by oid.
by_oid: Vec<(ObjectId, u64)>,
/// `(pack_offset, oid)`, **strictly** ascending by offset. The successor of
/// an offset is where its entry ends.
///
/// One table, read in both directions: offset → length for `P-025`, and
/// offset → oid for [`PackedObjects::oid_at_offset`], which is how an
/// `OBJ_OFS_DELTA`'s base gets a name. Keeping a bare `Vec<u64>` here and
/// re-reading the `.idx` for the second direction would be a second
/// full-index read per pack per request for a table already in hand
/// (LAW 5).
by_offset: Vec<(u64, ObjectId)>,
}
impl PackExtents {
/// Build from the `(offset, oid)` pairs of one pack, or refuse.
///
/// `None` means the pack does not satisfy the contiguity this reads
/// lengths out of, and the caller must keep asking `gix-odb`.
fn install(mut pairs: Vec<(u64, ObjectId)>) -> Option<Self> {
if pairs.is_empty() {
return None;
}
let mut by_offset = pairs.clone();
by_offset.sort_unstable_by_key(|(offset, _)| *offset);
// STRICTLY ascending. A duplicate offset would mean two objects claiming
// the same entry, and `w[1].0 - w[0].0` would then be a zero-length
// entry written onto the wire as a truncated deflate stream.
if by_offset.windows(2).any(|w| w[0].0 >= w[1].0) {
return None;
}
let mut by_oid: Vec<(ObjectId, u64)> =
pairs.drain(..).map(|(offset, id)| (id, offset)).collect();
by_oid.sort_unstable_by_key(|entry| entry.0);
Some(PackExtents { by_oid, by_offset })
}
/// Where `id` lives in pack `pack_id`, and how long its entry is.
///
/// `None` for an object this pack does not hold, and for the one object per
/// pack that has no successor to measure against.
fn location(&self, pack_id: u32, id: &gix_hash::oid) -> Option<Location> {
let at = self
.by_oid
.binary_search_by(|(oid, _)| oid.as_ref().cmp(id))
.ok()?;
let pack_offset = self.by_oid[at].1;
let ordinal = self
.by_offset
.binary_search_by_key(&pack_offset, |e| e.0)
.ok()?;
let next = self.by_offset.get(ordinal + 1)?.0;
Some(Location {
pack_id,
pack_offset,
entry_size: usize::try_from(next - pack_offset).ok()?,
})
}
/// Which object starts at `offset`, and how long its entry is.
///
/// `None` when no entry starts there, and for the one entry per pack with
/// no successor to measure against — the same two refusals
/// [`PackExtents::location`] makes, for the same reason.
fn entry_at(&self, pack_id: u32, offset: u64) -> Option<(ObjectId, Location)> {
let at = self.by_offset.binary_search_by_key(&offset, |e| e.0).ok()?;
let next = self.by_offset.get(at + 1)?.0;
Some((
self.by_offset[at].1,
Location {
pack_id,
pack_offset: offset,
entry_size: usize::try_from(next - offset).ok()?,
},
))
}
}
/// How many object locations were answered out of [`PackExtents`], and how many
/// fell through to `gix-odb`'s inflate.
///
/// Process-wide and monotonic, which is what a diagnostic wants: it is read by
/// a bench arm and by `tests/pack_extents.rs` to assert **the path that ran**,
/// not merely that the bytes came out right. A pure cost optimisation cannot be
/// seen in its own output by construction — the pack is byte-identical either
/// way — so without this pair a run with the table silently disabled is shaped
/// exactly like a run with it working.
static EXTENT_HITS: AtomicU64 = AtomicU64::new(0);
static EXTENT_MISSES: AtomicU64 = AtomicU64::new(0);
/// `(answered from the index, fell back to an inflate)` since the process
/// started. See [`PackExtents`].
pub fn extent_lookups() -> (u64, u64) {
(
EXTENT_HITS.load(Ordering::Relaxed),
EXTENT_MISSES.load(Ordering::Relaxed),
)
}
/// `GUNNAR_PACK_EXTENTS=off` puts every location lookup back through
/// `gix-odb`'s inflate.
///
/// The A/B switch, and it exists so *one binary* can measure both arms against
/// the same repository in the same process — which is the only way to compare a
/// pack-copy path honestly, because the two arms are defined to emit identical
/// bytes and a second build could differ for reasons no digest would show.
fn extents_enabled() -> bool {
!matches!(
std::env::var("GUNNAR_PACK_EXTENTS").as_deref(),
Ok("off") | Ok("0")
)
}
/// A cloneable, sendable way of reaching a store's **pack-resident** view.
///
/// This is `P-001`'s "third read method", the seam the entry says
/// `ObjectStore` needs so a writer can ask *where an object lives* rather than
/// only *what it says*. It is expressed as [`gix_pack::Find`] rather than as a
/// hand-rolled `read_pack_entry`, because that is the trait
/// `output::entry::iter_from_counts` consumes and because re-deriving pack
/// locations and `OfsDelta` base re-pointing outside gix would be a second copy
/// of the one piece of this that is genuinely hard (LAW 5).
///
/// # Why a factory rather than a shared handle
///
/// `gix_odb`'s handle is `Send` but **not** `Sync`, its index snapshot and
/// inflate scratch being `RefCell`s, so it cannot simply be put behind an `Arc`
/// and shared. `iter_from_counts` requires `Find: Send + Clone + 'static` and
/// clones one per worker, so [`Clone`] here opens a *fresh* handle from the
/// same store. That is the shape gix-odb is built for, and it means a clone is
/// as expensive as a handle and no more.
pub struct PackedObjects {
open: Arc<dyn Fn() -> Box<dyn gix_pack::Find + Send> + Send + Sync>,
handle: Box<dyn gix_pack::Find + Send>,
/// The entry-extent tables, by pack id. **Shared across clones**, which is
/// the whole reason it is an `Arc`: `iter_from_counts` clones this type once
/// for its resolve phase and once per worker, and a per-clone table would
/// read every `.idx` two or three times per request and hold two or three
/// copies of it. See [`PackExtents`].
///
/// `None` for a pack that was **examined and declined** — the multi-pack
/// index case below, or offsets that are not strictly ascending. Remembering
/// the refusal is not tidiness: without it `learn_pack` retried the whole
/// examination on every lookup, because "already known" was `contains_key`
/// and a declined pack was never inserted.
///
/// MEASURED oden 2026-08-05, one whole-repository clone of `fx/knut` with a
/// multi-pack index over its 12 packs, 3273 objects: **2943** rebuilds of the
/// same offset list, each one a full `pack_offsets_and_oid` — and 2943
/// identical `debug!` lines saying so. The correct number is one per pack per
/// request.
extents: Arc<Mutex<HashMap<u32, Option<PackExtents>>>>,
/// The pack `gix-odb` would search **first**, mirrored.
///
/// # Why this field is what keeps the bytes identical
///
/// `gix-odb`'s `location_by_oid` returns the first index *in its current
/// order* that holds the object, and then does
/// `snapshot.indices.swap(0, idx)` — a move-to-front. So its answer for an
/// object that lives in **two** packs depends on which index it last used.
///
/// That is not academic. Answering such an object out of any table this type
/// happens to hold produced a **53-byte** difference on a 287 KB pack over a
/// two-pack fixture, MEASURED 2026-08-04: both packs held the object, the
/// two copies had different delta bases, and both packs were valid answers.
/// A pure cost optimisation that picks a different valid answer has changed
/// behaviour.
///
/// Mirroring the front index removes the choice entirely. `gix-odb` searches
/// index 0 first, so **if the object is in the front pack, gix returns the
/// front pack** — no MRU state can change that. If it is not, this declines
/// and `gix-odb` does its own search, whose result then becomes the new
/// front here exactly as it becomes index 0 there.
///
/// Per **clone**, not shared: each clone opens its own `gix-odb` handle with
/// its own snapshot, so a shared front would be one handle's order applied
/// to another's. The tables themselves are shared, because a pack's index is
/// a property of the pack and not of the handle that read it.
front: Mutex<Option<u32>>,
}
impl PackedObjects {
/// Build one from a function that opens a fresh pack-aware handle.
///
/// The handle **must** have been configured with
/// `prevent_pack_unload()`: `entry_by_location` hands out slices pointing
/// into a mapped pack, and `gix-odb` panics rather than let a caller read
/// memory it may have unmapped underneath them.
pub fn new<F>(open: F) -> Self
where
F: Fn() -> Box<dyn gix_pack::Find + Send> + Send + Sync + 'static,
{
let handle = open();
PackedObjects {
open: Arc::new(open),
handle,
extents: Arc::new(Mutex::new(HashMap::new())),
front: Mutex::new(None),
}
}
/// Where `id` lives, **if it lives in the pack `gix-odb` would search
/// first** and that pack's index has been read.
///
/// Every other case declines, including "it is in some other table we
/// hold": see [`PackedObjects::front`] for why that is the whole of the
/// correctness argument rather than a conservatism.
///
/// The search runs under the lock rather than cloning the table out of it:
/// this is the per-object path, and an `Arc` clone per object would be the
/// very allocation the table exists to remove.
fn extent_location(&self, id: &gix_hash::oid) -> Option<Location> {
let front = (*self.front.lock().ok()?)?;
let tables = self.extents.lock().ok()?;
tables.get(&front)?.as_ref()?.location(front, id)
}
/// Which object starts at `offset` in pack `pack_id`, and how long its
/// entry is, out of the extent table this request has already read.
///
/// The one direction of the table [`PackExtents::location`] does not use,
/// and the only way an `OBJ_OFS_DELTA`'s base — which the pack names by
/// *distance*, never by id — can be added to a selection by name. `None`
/// when the table for that pack has not been read, which leaves
/// [`base_closure`] declining rather than guessing.
///
/// [`PackedObjects::front`]'s "which pack would `gix-odb` have searched"
/// question does not arise here and this does not consult it: the caller
/// already holds the pack id, because the base lives in the same pack as
/// the delta that named it. There is no second valid answer to pick wrongly
/// from.
fn entry_at_offset(&self, pack_id: u32, offset: u64) -> Option<(ObjectId, Location)> {
let tables = self.extents.lock().ok()?;
tables.get(&pack_id)?.as_ref()?.entry_at(pack_id, offset)
}
/// How many objects pack `pack_id`'s **own header** says it holds.
///
/// # Why the pack and not the index it was found through
///
/// This is the completeness check the extent table cannot do without, and
/// the answer has to come from a source that cannot be filtered. A packfile
/// begins with 12 bytes — `PACK`, a version, and an object count — and that
/// count is a property of the file, so no index over it can disagree.
///
/// [`gix_pack::Find::entry_by_location`] hands back a raw slice of the
/// mapped pack at any offset, so offset 0 for 12 bytes is the header, and it
/// costs one `memcpy` of 12 bytes per pack per request. Nothing is parsed as
/// an entry.
///
/// `None` when the pack cannot be read at all, or when those bytes are not a
/// pack header, and the caller then declines the table rather than assuming
/// the list it holds is whole.
fn objects_in_pack(&self, pack_id: u32) -> Option<u32> {
let header = self.handle.entry_by_location(&Location {
pack_id,
pack_offset: 0,
entry_size: PACK_HEADER_BYTES,
})?;
let bytes: &[u8] = header.data.as_ref();
let header: [u8; PACK_HEADER_BYTES] = bytes.try_into().ok()?;
if &header[..4] != b"PACK" {
return None;
}
Some(u32::from_be_bytes([
header[8], header[9], header[10], header[11],
]))
}
/// Read pack `pack_id`'s index into an extent table, once.
fn learn_pack(&self, pack_id: u32) {
{
let Ok(tables) = self.extents.lock() else {
return;
};
if tables.contains_key(&pack_id) {
return;
}
}
// Outside the lock: this reads a whole `.idx`, and holding the mutex
// across it would serialise every other worker's lookups behind it.
let Ok(Some(pairs)) = self.handle.pack_offsets_and_oid(pack_id) else {
// Not remembered as a refusal: this is `gix-odb` declining to answer
// for a pack id it does not currently hold, which a later snapshot
// may well answer. The two refusals below are properties of the
// pack and do not change under this handle.
return;
};
let objects = pairs.len();
// ── the offset list has to be the WHOLE pack ────────────────────────
//
// `PackExtents` measures an entry as the distance to the next offset in
// the same pack, which is the end of that entry **only if every offset
// is in the list**. A missing offset does not shorten the table, it
// makes the entry before it measure across it, and the copied entry
// then carries its neighbour's bytes as a tail.
//
// That is not hypothetical and it is not a property of `.idx` files. It
// is what a **multi-pack index** does: the format holds one
// deduplicated oid table across every pack it spans, so an object living
// in two of them is recorded against exactly one, and `gix-odb` answers
// `pack_offsets_and_oid` for the *other* pack by filtering that table by
// pack — which leaves out the object it attributed elsewhere. MEASURED
// oden 2026-08-05 on two overlapping packs with an index over them: 48
// surplus bytes in a 287 691-byte pack, and
//
// ```text
// fatal: pack is corrupted (SHA1 mismatch)
// ```
//
// from stock git 2.53.0, which parsed all 72 entries and then read its
// trailer 48 bytes early. `tests/pack_extents.rs` drives it red, in
// **both** hashes: nothing in the trigger is hash-shaped, and the sha256
// corpus was clean because it had 12 packs against a threshold of 16,
// not because sha256 is immune. It holds 13 objects in two packs.
//
// The pack's own header is the one count no index can filter, so it is
// what the list is held to. A pack that lost entries to a multi-pack
// index keeps using `gix-odb`'s inflate, and every other pack keeps
// `P-025`.
//
// # What the refusal costs, MEASURED rather than feared
//
// oden 2026-08-05, `knut.git` from the production-shaped corpus (12
// pushed packs + 4 more, 16 in all, so `FsStore::open_or_init` writes a
// multi-pack index unasked), whole-repository serve of 3273 objects:
//
// | | wire bytes | index-answered | inflated | stock git |
// |---|---|---|---|---|
// | before | 8 066 803 | 2915 (89.1%) | 358 | **refused** |
// | after | 8 020 723 | 2685 (82.0%) | 588 | accepted |
//
// The 46 080-byte difference is the surplus, 0.57% of the pack. The
// refusal costs **230 of 2915 lookups**, 7.9% — it is per *pack*, not
// per store, so the packs the index left whole keep answering out of
// their own tables. `objects_copied_from_pack` and
// `objects_recompressed` are 1559/1714 on both arms: this changes how an
// entry's length is *learnt*, never whether it is copied, so `P-001`'s
// reuse and `ad30937`'s base-closure numbers are untouched by it.
match self.objects_in_pack(pack_id) {
Some(in_pack) if in_pack as usize == objects => {}
in_pack => {
tracing::debug!(
target: crate::GIT_TARGET,
pack_id,
listed = objects,
in_pack = in_pack.unwrap_or_default(),
header_read = in_pack.is_some(),
"the index lists fewer entries than the pack holds — a multi-pack index \
attributed some of them to another pack; entry lengths keep coming from \
gix-odb"
);
self.decline(pack_id);
return;
}
}
let Some(extents) = PackExtents::install(pairs) else {
tracing::debug!(
target: crate::GIT_TARGET,
pack_id,
objects,
"pack entry offsets are not strictly ascending; entry lengths keep coming from \
gix-odb"
);
self.decline(pack_id);
return;
};
if let Ok(mut tables) = self.extents.lock() {
tables.entry(pack_id).or_insert(Some(extents));
}
}
/// Remember that pack `pack_id` cannot answer entry lengths, so the
/// examination is not repeated on the next lookup. See
/// [`PackedObjects::extents`].
fn decline(&self, pack_id: u32) {
if let Ok(mut tables) = self.extents.lock() {
tables.entry(pack_id).or_insert(None);
}
}
}
impl Clone for PackedObjects {
fn clone(&self) -> Self {
PackedObjects {
open: Arc::clone(&self.open),
handle: (self.open)(),
extents: Arc::clone(&self.extents),
// A fresh handle has a fresh snapshot whose order this clone has not
// observed, so it starts with no front and learns one from its own
// first fall-through.
front: Mutex::new(None),
}
}
}
impl std::fmt::Debug for PackedObjects {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("PackedObjects")
}
}
impl gix_pack::Find for PackedObjects {
fn contains(&self, id: &gix_hash::oid) -> bool {
crate::oidprobe::bump(&crate::oidprobe::GIX_PACK_CONTAINS);
self.handle.contains(id)
}
fn try_find_cached<'a>(
&self,
id: &gix_hash::oid,
buffer: &'a mut Vec<u8>,
pack_cache: &mut dyn gix_pack::cache::DecodeEntry,
) -> gix_error::Result<
Option<(
gix_object::Data<'a>,
Option<gix_pack::data::entry::Location>,
)>,
> {
crate::oidprobe::bump(&crate::oidprobe::GIX_PACK_TRY_FIND_CACHED);
self.handle.try_find_cached(id, buffer, pack_cache)
}
/// **`P-025`.** The one override that is not a forward, and the reason this
/// type implements the trait rather than handing `gix-odb`'s handle
/// straight to `iter_from_counts`.
///
/// `gix-odb` answers this by inflating the whole object and keeping only
/// how long that took (see [`PackExtents`] for the measurement). Here the
/// answer comes out of the pack index, and `buf` is left untouched —
/// `iter_from_counts` uses it as scratch and reads nothing back out of it,
/// which is what makes that legal.
///
/// The fall-through is not a slow path being tolerated, it is how the table
/// gets built: a location `gix-odb` resolves names the pack it came out of,
/// and that is the only way this type can learn a pack id through the
/// trait. So the first object of each pack pays an inflate and teaches the
/// table about every other object in the same pack.
fn location_by_oid(
&self,
id: &gix_hash::oid,
buf: &mut Vec<u8>,
) -> gix_error::Result<Option<Location>> {
if extents_enabled() {
if let Some(location) = self.extent_location(id) {
EXTENT_HITS.fetch_add(1, Ordering::Relaxed);
return Ok(Some(location));
}
}
EXTENT_MISSES.fetch_add(1, Ordering::Relaxed);
// Counted on the fall-through only. A lookup answered out of the
// extent table never reaches `gix-odb`, so counting it here would
// report probes that did not happen — and `oidprobe` exists to
// measure what the seam actually costs.
crate::oidprobe::bump(&crate::oidprobe::GIX_PACK_LOCATION_BY_OID);
let Some(location) = self.handle.location_by_oid(id, buf)? else {
return Ok(None);
};
if extents_enabled() {
// `gix-odb` has just moved this pack's index to the front of its
// snapshot. Mirror that before reading the index, so the two never
// disagree about which pack is first.
if let Ok(mut front) = self.front.lock() {
*front = Some(location.pack_id);
}
self.learn_pack(location.pack_id);
}
Ok(Some(location))
}
fn pack_offsets_and_oid(
&self,
pack_id: u32,
) -> gix_error::Result<Option<Vec<(gix_pack::data::Offset, ObjectId)>>> {
self.handle.pack_offsets_and_oid(pack_id)
}
fn entry_by_location(
&self,
location: &gix_pack::data::entry::Location,
) -> Option<gix_pack::find::Entry> {
crate::oidprobe::bump(&crate::oidprobe::GIX_PACK_ENTRY_BY_LOCATION);
self.handle.entry_by_location(location)
}
}
// ── the unselected delta base ────────────────────────────────────────────────
/// How many bytes of an entry are read to decide whether it is a delta.
///
/// A V2 entry header is a type-and-size varint (at most 10 bytes for a `u64`
/// size) followed, for an `OBJ_OFS_DELTA`, by a base-distance varint (at most
/// 10) and, for an `OBJ_REF_DELTA`, by a raw object id (32 for SHA-256). 64 is
/// past every one of those with room to spare, and it is what keeps this pass
/// cheap: `entry_by_location` copies `Location::entry_size` bytes out of the
/// mapped pack, so asking for the whole entry would copy the repository to read
/// one byte per object.
const ENTRY_HEADER_PROBE: usize = 64;
/// What the pack will carry, in the **two** ways [`base_closure`] has to ask
/// about it — and the one place an object is added to it.
///
/// # Why two, and why they are not the same question
///
/// A pack may hold an object **once**. `git index-pack` says so by refusing the
/// pack outright (`The same object <id> appears twice in the pack`), so the
/// question "is this object already in the transfer" is asked of an **object
/// id**.
///
/// But the question the closure exists to answer is a different one: "will
/// `Entry::from_pack_entry` find this delta's base among `potential_bases`". It
/// searches by **pack offset**, inside the pack the delta was copied out of, so
/// that question is asked of a `(pack_id, pack_offset)`.
///
/// On a **single-pack** store the two are the same question, because an object
/// has exactly one location and a location has exactly one object. On a
/// multi-pack store they come apart: an object stored in two packs is one id at
/// two locations, `location_by_oid` picks one of them, and the other is a
/// location the selection does not hold *whose object the selection does*. That
/// gap is where the same blob got added to a pack that already carried it, and
/// where twelve of nineteen production-shaped repositories stopped cloning; see
/// [`base_closure`]'s own note.
///
/// So both are kept, both are updated by the one [`Carried::add`], and nothing
/// in this module inserts into either of them directly (LAW 5). A future reader
/// adding a third route into the pack has one function to go through.
struct Carried {
/// The locations `Entry::from_pack_entry` will find, by pack.
at: HashMap<u32, std::collections::HashSet<u64>>,
/// The object ids the pack carries, at **any** location — including the
/// selected objects with no pack residency at all, which are in no `at`
/// set and are still objects this pack must not be handed twice.
ids: std::collections::HashSet<ObjectId>,
}
impl Carried {
fn with_capacity(objects: usize) -> Self {
Carried {
at: HashMap::new(),
ids: std::collections::HashSet::with_capacity(objects),
}
}
/// Will a delta in `pack_id` naming `offset` find its base?
fn holds_at(&self, pack_id: u32, offset: u64) -> bool {
self.at
.get(&pack_id)
.is_some_and(|offsets| offsets.contains(&offset))
}
/// Is this object already going into the pack, by any route?
fn holds_id(&self, id: &ObjectId) -> bool {
self.ids.contains(id)
}
/// `location` is `None` for a selected object with no pack residency.
fn add(&mut self, id: ObjectId, location: Option<&Location>) {
self.ids.insert(id);
if let Some(location) = location {
self.at
.entry(location.pack_id)
.or_default()
.insert(location.pack_offset);
}
}
}
/// What [`base_closure`] found and what it added.
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
pub struct BaseClosure {
/// Selected objects that are an `OBJ_OFS_DELTA` on a base living in the
/// same pack but **outside** the selection. Each one of these is an object
/// the generator would otherwise inflate and deflate whole.
pub deltas_recovered: usize,
/// The distinct base objects added to the selection to recover them,
/// transitively — a base may itself be a delta on another unselected base.
pub bases_added: usize,
/// Selected objects whose entry header could not be read, or whose base
/// offset named no entry this could put a name to. They are left exactly as
/// they were and the generator decides them as it always did.
pub undecided: usize,
/// Selected objects with no pack residency at all — a loose object. Nothing
/// here can help one: it has no delta to preserve. Counted so a run where
/// *every* object is loose is visible as such rather than as a closure that
/// found nothing to do.
pub unpacked: usize,
/// Deltas declined because the base — or something further down its
/// chain — is an object the pack **already carries**, at a location in
/// another pack. Adding it would be a duplicate; see [`Carried`].
///
/// It is its own number rather than another `undecided` because it is the
/// one that says the store is multi-pack and the selection overlaps
/// itself. On the 19-repository production-shaped corpus it is non-zero on
/// 13 of them and zero on every single-pack fixture in the tree, which is
/// exactly why no guard here could see the defect it stands for.
pub already_carried: usize,
}
/// `GUNNAR_BASE_CLOSURE=off` puts the explosion back, for one process, so both
/// arms can be measured against the **same** pack.
///
/// The A/B switch and nothing else: the shipping decision is
/// [`WriteOptions::base_closure`], which a filtered request must be able to
/// turn off for correctness rather than for measurement. The env var can only
/// subtract, so a run with it set is never doing more than the default would.
///
/// Read once per pack, not per object.
fn base_closure_enabled(options: &WriteOptions) -> Option<u64> {
if matches!(
std::env::var("GUNNAR_BASE_CLOSURE").as_deref(),
Ok("off") | Ok("0")
) {
return None;
}
options.base_closure
}
/// Add to `ids` every pack-resident **delta base** that `ids` refers to and does
/// not contain, so the delta can be copied instead of expanded.
///
/// # The item this exists for
///
/// A clone's selection is the closure of its `want`s. A pack, in general, holds
/// more: what a deleted branch left behind, what a push carried in beyond the
/// ref it updated. MEASURED oden 2026-08-05 on the corpus's `nornir.git`, whose
/// single pack holds 12 328 objects against the clone's 12 186 — **142 pack
/// residents outside the selection**.
///
/// When a *selected* object is an `OBJ_OFS_DELTA` on one of those,
/// `gix_pack`'s `Entry::from_pack_entry` searches `potential_bases` — the
/// counts of that pack, which is the selection — finds nothing, and with
/// `allow_thin_pack` false returns `None`. The generator then falls back to
/// `try_find`, which inflates the whole delta chain, and to `Entry::from_data`,
/// which deflates the object **whole**. So the same object, in the same pack, is
/// copied when its base happens to be selected and expanded when it is not.
///
/// This names the base instead: it walks the selection's entry headers, and for
/// every unselected base offset it can put a name to, appends that object.
/// `Entry::from_pack_entry` then finds it among `potential_bases` on the very
/// next call and copies the delta verbatim.
///
/// # What it costs and what it buys
///
/// **It is not byte-neutral in either direction**, and the direction depends on
/// the repository. MEASURED oden 2026-08-05, 18 corpus repositories (`bifrost`
/// excluded: `git fsck` fails on the SOURCE repo), one process per arm against
/// one unchanged pack, `want` = the branch tip, no `have`s:
///
/// | | objects | wire bytes | server CPU | re-deflated |
/// |---|---|---|---|---|
/// | off | 64 445 | 94 575 097 | 355.0 ms | 448 |
/// | on | 64 533 | 91 903 865 | 169.7 ms | 320 |
/// | | **+0.14%** | **−2.82%** | **−52.19%** | **−128** |
///
/// Per repository the bytes range from **−16.74%** (`korp.git`) to **+0.08%**
/// (`znippy-zoomies.git`); nine repositories save, three pay at most 0.08%, six
/// are unchanged. The CPU falls on every repository that has anything to
/// recover and rises by 3–7% on the six that do not, which is inside the
/// measured instrument floor and inside those runs' own 35–42% spreads.
///
/// # Why an estimate said the opposite, and how to not repeat it
///
/// The first version of this comment said **+5.15% bytes**, and it was wrong by
/// a factor that mattered. It was computed from `git verify-pack -v`, whose
/// `size` column for a **delta** entry is the size of the *delta stream*, not
/// of the object it reconstructs. `nornir.git`'s 36 exploded objects read as
/// 20 809 bytes of content by that column and are **3 403 910** by
/// `git cat-file -s`. A 163× error, in the one number the whole trade turns on.
///
/// The measurement that settled it — both arms in one process against one pack,
/// counting what the server actually wrote — was right the first time.
///
/// # Only 128 of the 448 explosions are recoverable, and that is the ceiling
///
/// The rest are trees and commits, and their bases may not be added; see the
/// blob restriction below. `znippy.git`'s 129 are **all commits** deltified
/// against unselected commits, which is why it is the one large repository this
/// changes nothing about. What the 128 are worth out of proportion to their
/// number is that they are the *blobs*: 3.4 MB of content on `nornir.git`
/// alone, against a few hundred bytes for a commit.
///
/// # Why the base is looked up rather than resolved
///
/// A pack names an `OBJ_OFS_DELTA`'s base by **distance backwards**, never by
/// id, so an offset has to become a name before it can go in a selection.
/// [`PackedObjects::entry_at_offset`] answers that out of the extent table
/// `P-025` already reads for entry lengths — the same `.idx`, read once, in the
/// other direction. Nothing here inflates anything, and nothing computes a
/// delta.
///
/// # What it declines
///
/// * An `OBJ_REF_DELTA`. `Entry::from_pack_entry` maps that to `None`
/// unconditionally — "ref deltas are for thin packs or legacy, repack them as
/// base objects" — so its base being present would change nothing.
/// * An object with no pack residency, and the one object per pack whose entry
/// length the extent table cannot measure. Both are `undecided` and are left
/// to the generator.
/// * A base offset in a pack whose extent table has not been read. Guessing
/// would be worse than the explosion it avoids.
/// * **A base the pack already carries under another pack's location**, which
/// is `already_carried` and the next section.
///
/// # The multi-pack case, which cost twelve of nineteen repositories a clone
///
/// The paragraph above says "unselected", and on a single-pack store that is
/// exactly what an offset outside the selection means. On a multi-pack store it
/// does not. An object stored in **two** packs — whole in the pack whose push
/// introduced its dependant, spliced there by `index-pack --fix-thin`, and as a
/// delta in the pack whose push introduced it — is one object id at two
/// locations. `location_by_oid` answers with one of them, so the selection
/// holds that location and not the other, and a delta in the *other* pack names
/// an offset the selection does not hold whose **object it does**.
///
/// Naming it puts the object in the pack twice, and `git index-pack` refuses
/// such a pack:
///
/// ```text
/// fatal: The same object 00d6fd78e6c2b91b311f54f5401b6ca5eb635f33 appears twice in the pack
/// fatal: fetch-pack: invalid index-pack output
/// ```
///
/// MEASURED oden 2026-08-05, one real `git clone --single-branch` per
/// repository with `transfer.fsckObjects=true`, over the 19 workspace_nordisk
/// repositories rebuilt in the **production** shape (12 pushes, 12 packs, no
/// `.bitmap`): **13 of 18 refused** before this, 18 of 18 clean after. On the
/// shipped single-pack corpus the same 18 clones pass both ways, and that is
/// the whole reason the first version of this pass shipped: a `git clone
/// --bare` fixture repacks, and a repacked fixture cannot hold one object
/// twice.
///
/// So membership is asked twice, of two different keys, and [`Carried`] is the
/// one place that answers either.
///
/// # Why it hands back the counts
///
/// `iter_from_counts`'s first phase resolves every count's pack location, and
/// this pass has to resolve the same locations to ask its question at all.
/// Handing them on as `PackLocation::LookedUp` means the request pays for
/// **one** resolve pass rather than two — the answers are the same
/// [`PackedObjects::location_by_oid`] gix would have called, from the same
/// handle, in the same order — and the closure's own cost falls to a 64-byte
/// header read per object. Without this the pass showed up as **+18 to +23%**
/// of CPU on the four corpus repositories that have nothing to recover, which
/// is a real cost paid by every request that this item cannot help.
fn base_closure(
ids: &mut Vec<ObjectId>,
packed: &PackedObjects,
max_base_bytes: u64,
) -> (BaseClosure, Vec<output::Count>) {
use gix_pack::Find as _;
use output::count::PackLocation;
let mut stats = BaseClosure::default();
let Some(hash_kind) = ids.first().map(|id| id.kind()) else {
return (stats, Vec::new());
};
// The selection, by location and by id. See [`Carried`]: the first is the
// set `Entry::from_pack_entry` will search, so an offset in it is an offset
// that will be found and an offset outside it is an explosion; the second
// is what the *pack* may not be handed twice.
let mut carried = Carried::with_capacity(ids.len());
let mut counts: Vec<output::Count> = Vec::with_capacity(ids.len());
let mut buf = Vec::new();
for id in ids.iter() {
// A lookup that fails counts as a location not known, which is what the
// trait answered before gix-pack 0.77 made it a `Result` (gix-odb folded
// the error into `None` itself): the object goes in as unpacked.
let location = packed.location_by_oid(id, &mut buf).ok().flatten();
if location.is_none() {
stats.unpacked += 1;
}
carried.add(*id, location.as_ref());
counts.push(output::Count {
id: *id,
entry_pack_location: PackLocation::LookedUp(location),
});
}
// Read one entry's header, and nothing else. See `ENTRY_HEADER_PROBE`.
let header = |location: &Location| -> Option<(gix_pack::data::entry::Header, u64)> {
let probe = Location {
pack_id: location.pack_id,
pack_offset: location.pack_offset,
entry_size: location.entry_size.min(ENTRY_HEADER_PROBE),
};
let entry = packed.entry_by_location(&probe)?;
gix_pack::data::Entry::from_bytes(&entry.data, 0, hash_kind)
.ok()
.map(|parsed| (parsed.header, parsed.decompressed_size))
};
// `counts.len()` is read once, when the range is built, so the bases pushed
// inside the loop are not re-examined by it. They do not need to be: the
// chain walk below already followed each one to the bottom.
for at in 0..counts.len() {
let Some(location) = counts[at].entry_pack_location.as_ref().cloned() else {
continue;
};
let Some((entry_header, _)) = header(&location) else {
stats.undecided += 1;
continue;
};
let gix_pack::data::entry::Header::OfsDelta { base_distance } = entry_header else {
continue;
};
let Some(base_offset) = location.pack_offset.checked_sub(base_distance) else {
stats.undecided += 1;
continue;
};
if carried.holds_at(location.pack_id, base_offset) {
continue;
}
// ── follow the chain down to the entry that is not a delta ─────────
//
// Two things are wanted from it and neither is available any other way.
//
// **The type.** git deltifies only within a type, so every entry in a
// chain has the type of the entry the chain bottoms out at — and an
// `OBJ_OFS_DELTA`'s own header carries no type at all. The type is what
// decides whether the base may be added; see below.
//
// **The rest of the chain.** A base may itself be a delta on another
// unselected base, and adding only the first of them leaves the second
// explosion exactly where it was.
//
// Termination is structural rather than bounded: `base_distance` is
// positive, so the offset strictly decreases and the walk cannot
// revisit an entry.
let mut chain: Vec<(ObjectId, Location)> = Vec::new();
let mut at_offset = base_offset;
// The size of the entry the walk last read. When the walk breaks on a
// non-delta entry this is that entry's object size, which is the one
// size in a chain that a header carries. See
// `WriteOptions::base_closure`.
let mut root_size;
// Set when the chain runs into an object the pack already carries at a
// location in a **different** pack. See [`Carried`] and the note on
// `BaseClosure::already_carried`: adding it would put the object in the
// pack twice, and `git index-pack` refuses such a pack outright.
//
// The whole recovery is then declined rather than the chain truncated.
// The entry that collided is the one the delta above it names, so
// nothing above it can be copied either, and adding the residue below
// it would be bytes that recover nothing.
let mut collides = false;
let root = loop {
let Some((id, at_location)) = packed.entry_at_offset(location.pack_id, at_offset)
else {
break None;
};
let Some((entry_header, decompressed_size)) = header(&at_location) else {
break None;
};
root_size = decompressed_size;
if !carried.holds_at(location.pack_id, at_offset) {
if carried.holds_id(&id) {
collides = true;
break None;
}
chain.push((id, at_location.clone()));
}
match entry_header {
gix_pack::data::entry::Header::OfsDelta { base_distance } => {
match at_offset.checked_sub(base_distance) {
Some(next) => at_offset = next,
None => break None,
}
}
// An `OBJ_REF_DELTA` names its base by id, and this walk has
// only offsets. It is also the one entry kind
// `Entry::from_pack_entry` refuses outright, so a chain that
// runs into one was never going to be copied anyway.
gix_pack::data::entry::Header::RefDelta { .. } => break None,
// A non-delta entry: its header carries the object's own size,
// which is the one size in the chain that is knowable without
// reconstructing anything. See `WriteOptions::base_closure`.
other => break other.as_kind().map(|kind| (kind, root_size)),
}
};
// ── why only a blob ────────────────────────────────────────
//
// An added base is **unreachable** from anything the client asked for.
// That is not a risk of this change, it is the definition of the case:
// if it were reachable it would be in the selection already.
//
// `git index-pack --strict` — what `transfer.fsckObjects=true` turns on,
// and what this repository's bar requires — runs `fsck` over every
// object in the pack and then demands that every object those objects
// *refer to* be present. A blob refers to nothing, so adding one can
// never fail that. A tree refers to its entries, and an unselected tree
// is by construction a version of a directory the client is not being
// sent, so some of its entries are not in the pack.
//
// MEASURED oden 2026-08-05, which is why this restriction is here
// rather than in a paragraph about hypothetical risk: adding tree bases
// as well produced a pack that stock git 2.53.0 **refused**, on 14 of
// the 18 corpus repositories —
//
// ```text
// fatal: did not receive expected object 284ca166bd34b7441cf41f4b2a9eb3309846b848
// fatal: fetch-pack: invalid index-pack output
// ```
//
// — `284ca166` being a blob that one of the added trees names. The wire
// bytes, the CPU and the pack digest had all been measured before that
// clone was run, and all three looked like a win. Only stock git said
// otherwise.
if collides {
stats.already_carried += 1;
continue;
}
match root {
// The server's own ceiling on "bytes of a single object gunnar will
// put in a pack" applies to a base as much as to a selected object;
// `WriteOptions::base_closure` carries it here rather than leaving
// this to infer it.
Some((gix_object::Kind::Blob, size)) if size <= max_base_bytes => {}
_ => {
stats.undecided += 1;
continue;
}
}
if chain.is_empty() {
continue;
}
stats.deltas_recovered += 1;
for (base_id, base_location) in chain {
carried.add(base_id, Some(&base_location));
ids.push(base_id);
counts.push(output::Count {
id: base_id,
entry_pack_location: PackLocation::LookedUp(Some(base_location)),
});
stats.bases_added += 1;
}
}
(stats, counts)
}
/// Write a packfile containing exactly `ids`, **copying** every entry that
/// already lives in a pack instead of inflating and re-deflating it.
///
/// The drop-in counterpart of [`write_pack`](crate::write_pack): same ids, same
/// options, same guarantees about duplicates, hash kinds and empty packs, and
/// the same streaming property: entries are written as the generator produces
/// them, so nothing buffers the whole pack.
///
/// The returned [`WriteOutcome`]'s `objects_copied_from_pack` and
/// `objects_recompressed` are the applied output that distinguishes this from
/// [`write_pack`](crate::write_pack). A byte count and a wall clock cannot:
/// both paths produce a valid pack of the right shape.
///
/// # A note on [`WriteOptions::ofs_delta`]
///
/// When it is `false` — a client that did not advertise git's `ofs-delta`
/// capability — every copied delta is emitted as `OBJ_REF_DELTA`, naming its
/// base by object id rather than by distance backwards. **The delta bytes are
/// byte-for-byte the same** and the reuse survives in full; the pack is larger
/// only by `hash_kind.len_in_bytes()` minus a varint per delta entry.
///
/// Before 2026-08-03 the serve path simply did not copy for such a client
/// (`upload_pack.rs` gated `packed()` on the capability), which cost that client
/// the whole of `P-001`: MEASURED on the churned head-to-head fixture, **18.4×
/// the bytes**. `OBJ_REF_DELTA` is the older of the two forms and requires no
/// capability at all, so there was never a reason for it.
///
/// # A note on [`WriteOptions::compression_level`]
///
/// It applies only to the objects that still have to be deflated, the ones
/// with no pack residency. A copied entry keeps whatever level the pushing
/// client chose, which is the point. gix's generator takes its level from
/// `gix_zlib::Compression::DEFAULT`, git's own default and the same 6 that
/// [`WriteOptions::new`] picks, so today the two agree; plumbing a level
/// through would mean naming `gix-zlib` in this crate's manifest for a knob
/// nothing sets.
///
/// # Errors
///
/// * [`Error::ObjectVanished`] if the generator could not locate an object it
/// was told to pack. The pack is abandoned rather than silently shortened,
/// exactly as [`write_pack`](crate::write_pack) abandons it on a missing id.
/// * [`Error::HashKindMismatch`] if an id's hash kind is not
/// `options.hash_kind`.
pub fn write_pack_copy<W>(
ids: &[ObjectId],
packed: &PackedObjects,
out: W,
options: &WriteOptions,
) -> Result<WriteOutcome>
where
W: Write,
{
write_pack_copy_thin(ids, packed, None, out, options)
}
/// The same, but permitted to leave a delta pointing at a base the transfer
/// does **not** carry — a **thin pack** — whenever `client_has` vouches for it.
///
/// # `S-003`: what a clone hides and a fetch pays for
///
/// [`write_pack_copy`] copies **100%** of a clone's entries, and that number
/// concealed this for two sessions. A clone's object selection is the whole
/// history, so every `OBJ_OFS_DELTA`'s base travels with it and
/// `PackCopyAndBaseObjects` re-points it. A **fetch**'s selection is a suffix,
/// and the bases the pushing client's git chose sit on the far side of the
/// negotiation cut. `gix_pack`'s `Entry::from_pack_entry` then finds no base
/// among `potential_bases`, and with `allow_thin_pack` false it returns `None`
/// — whereupon the caller inflates the whole delta chain and deflates the
/// object **whole**.
///
/// So the same object, in the same pack, is copied on a clone and expanded on
/// a fetch. MEASURED on oden 2026-08-04, a 60-file churn fetched 8 commits
/// deep: **24 of 40 objects re-deflated**,
/// 31 355 bytes against stock git's 7 873-byte thin pack of the same object
/// set — **3.98×**. That is `P-009`'s own stated reopen condition ("if a
/// measurement ever shows a meaningful fraction of served entries
/// recompressed") and the whole of gunnar's 1.47× wire-byte loss to gitea.
///
/// # What this recovers, and what it leaves for an encoder
///
/// MEASURED oden 2026-08-04 on the bench's own `h2h-divergent-2000b-500d`
/// fixture, one negotiated fetch of 2912 objects across a 500-commit
/// divergence, server tallies from this function:
///
/// | | expanded | wire bytes |
/// |---|---|---|
/// | non-thin (before) | **71** | 419 834 |
/// | thin, `Exclusion::Edges` (shipping) | **8** | 348 671 |
/// | thin, `Exclusion::Full` | **0** | 341 485 |
/// | stock `git pack-objects --thin` | — | 292 336 |
///
/// 2841 of 2912 is 97.56%, which is `P-009`'s "97.6% of a fetch" to the digit:
/// **the residue that entry measured is exactly what this recovers**, and it
/// was never objects nothing could compress. It was objects gunnar had a delta
/// for and was not allowed to name the base of.
///
/// The last row is the part this does **not** touch and `S-003` still names
/// honestly. `git pack-objects --thin` does not only reuse, it re-runs a delta
/// search over the subset and can pick a base gunnar's pack never chose,
/// because the pack was deltified against the whole history rather than
/// against this transfer. Copying cannot reach that; only an encoder can. So
/// of the 127 498 bytes between the old server and stock git, **78 349 (61%)
/// were reuse** and are now recovered, and 49 149 remain for `S-003`'s encoder.
///
/// The 8-object gap between `Edges` and `Full` is the voucher being an
/// under-approximation: the frontier walk does not open up the client's whole
/// history, so a base deeper than the frontier cannot be vouched for and its
/// object is expanded. It costs 2.1% of the fetch's bytes and is the right
/// trade — `P-010` measured `Full` faulting in 377.23 MiB of pack against
/// `Edges`'s 10.17 MiB on a comparable fixture. `GUNNAR_EXCLUSION=full` buys
/// the last 2.1% for that read amplification.
///
/// # Why this is still `P-001`, and not `S-003`'s encoder
///
/// Nothing here computes a delta. The deltas being recovered were computed by
/// the pushing client's git and are already sitting in gunnar's pack; the only
/// change is that the server is now allowed to *name* a base it is not
/// shipping. `gunnar-delta` stays off the serve path and out of the appliance
/// graph, which is what `xtask`'s `FORBIDDEN_IN_SERVER_GRAPH` entry asserts.
///
/// # The soundness condition, which is the whole of the risk
///
/// A thin delta whose base the receiver lacks is an **unreadable pack**:
/// `index-pack --fix-thin` cannot complete it and the fetch fails. So the base
/// is not taken on trust. `client_has` is consulted for every one, and an
/// entry it does not vouch for falls back to the whole object — the same
/// bytes [`write_pack_copy`] would have written, through the same
/// [`PackWriter::add_object`] and so the same deflate scratch and the same
/// trailer hasher (LAW 5).
///
/// `None` disables thin packing entirely and is exactly [`write_pack_copy`].
/// A caller must pass `Some` only when the receiver advertised `thin-pack`.
///
/// **"The receiver lacks it" includes an object this very pack is carrying.**
/// gix hands back a `DeltaOid` whenever it could not re-point the delta inside
/// the pack it is writing, and on a multi-pack store that happens for bases
/// that *are* in the transfer — the base resolved to a copy in one pack while
/// the delta was copied out of another. Naming such a base produces a pack
/// stock git completes and **gitoxide refuses**: its thin-base resolver looks
/// only in the receiver's object database, never inside the pack it is
/// reading. So `client_has` must vouch for held objects and for nothing else;
/// `gunnar_wire::graph::Selection::client_has` carries the object that proved
/// it and the `gunnar.year` run that died on it.
///
/// # Composition: one pass, not two
///
/// The partition into "copy" and "expand" is **not** computed up front, and a
/// pre-pass that tried to would be both slower and wrong: whether an entry can
/// be copied depends on where its base landed in *this* pack, which is not
/// known until the entries have been generated. gix decides it per entry, this
/// loop overrides it per entry, and both lanes write through the one
/// [`PackWriter`] in the one streaming pass. Nothing buffers the pack.
///
/// The one extra cost is the fallback itself: an object whose base is not
/// vouched for is inflated and re-deflated, which is precisely what the
/// non-thin path did for *every* such object.
pub fn write_pack_copy_thin<W>(
ids: &[ObjectId],
packed: &PackedObjects,
client_has: Option<&dyn Fn(&gix_hash::oid) -> bool>,
out: W,
options: &WriteOptions,
) -> Result<WriteOutcome>
where
W: Write,
{
use gix_parallel::Finalize as _;
use gix_parallel::InOrderIter;
let (mut ids, mut count) = prepare_ids(ids, options)?;
// ── the unselected delta base ───────────────────────────────────────────
//
// Off by default because the trade is bad, and MEASURED bad rather than
// assumed so: see [`base_closure`]. It runs here rather than in
// `gunnar_wire::graph` because *this* is the one place both selection paths
// meet — the reachability walk and the bitmap answer reach the same
// `ids` — and because pack residency is this crate's knowledge and not the
// walk's (LAW 5). A copy in `graph` would need its own `location_by_oid`
// pass and its own idea of what a pack holds.
//
// Only when `client_has` is `None`, which is the clone and every fetch from
// a client that did not ask for a thin pack. With a voucher in hand gix
// already names the base without shipping it, and shipping it anyway would
// be strictly more bytes for a delta that was going to be copied regardless.
let mut declined = 0usize;
let closed = if let (None, Some(max_base_bytes)) = (client_has, base_closure_enabled(options)) {
let (stats, counts) = base_closure(&mut ids, packed, max_base_bytes);
declined = stats.already_carried;
count = u32::try_from(ids.len()).map_err(|_| Error::TooManyObjects(ids.len()))?;
tracing::debug!(
target: crate::GIT_TARGET,
deltas_recovered = stats.deltas_recovered,
bases_added = stats.bases_added,
undecided = stats.undecided,
unpacked = stats.unpacked,
already_carried = stats.already_carried,
"added unselected delta bases to the pack"
);
Some(counts)
} else {
None
};
let mut writer = PackWriter::new(out, count, options)?;
if count == 0 {
// gix's generator over an empty count list would still spawn its
// worker; a pack of no objects is a header and a trailer and is worth
// no threads at all.
return writer.finish();
}
// The locations are deliberately left `NotLookedUp`: `iter_from_counts`
// resolves them itself as its first phase, so running `count::objects`
// beforehand would be a second pass over the same ids. The reachability
// walk that decided this id set is `gunnar_wire::graph`, not gix's
// expansion, which is why `ObjectExpansion` never appears here.
//
// Unless [`base_closure`] already ran, in which case it resolved exactly
// these locations to ask its own question and hands them on rather than
// making gix resolve them a second time.
let counts: Vec<output::Count> = closed.unwrap_or_else(|| {
ids.iter()
.map(|id| output::Count {
id: *id,
entry_pack_location: output::count::PackLocation::NotLookedUp,
})
.collect()
});
// P-020: `thread_limit: Some(1)` is not a tuning choice. `iter_from_counts`
// fans out above 4000 counts, which nearly every real clone exceeds, and
// this is a per-request path. `P-004` settled that gunnar's serving tier
// stays serial until there is an admission gate that knows the box is idle.
// `xtask/tests/pack_pipeline.rs` fails the build if this line loses it.
crate::oidprobe::bump(&crate::oidprobe::GIX_PACK_ITER_FROM_COUNTS);
let entries = output::entry::iter_from_counts(
counts,
packed.clone(),
Box::new(gix_utils::progress::Discard),
output::entry::iter_from_counts::Options {
thread_limit: Some(1),
mode: output::entry::iter_from_counts::Mode::PackCopyAndBaseObjects,
// A thin pack lets a delta point at a base outside the pack. It is
// asked for only when the caller can vouch for what the receiver
// holds; gix then resolves such a base to its object id and hands
// back `Kind::DeltaOid`, and the loop below is what checks it.
// Without a voucher gix downgrades those deltas to whole objects,
// which is the pre-`S-003` behaviour and still the fallback.
allow_thin_pack: client_has.is_some(),
chunk_size: CHUNK_SIZE,
version: gix_pack::data::Version::V2,
..Default::default()
},
)
.map_err(Error::Generate)?;
let mut in_order = InOrderIter::from(entries);
// One `(offset, valid)` per entry written so far, in pack order. An
// `ofs-delta` names its base by *distance backwards*, so the base's offset
// has to be remembered until the delta that refers to it is emitted. This
// is the same table `FromEntriesIter` keeps, and the only piece of
// bookkeeping that emitter had which is not already in `PackWriter`.
let mut offsets: Vec<(u64, bool)> = Vec::with_capacity(ids.len());
// And, when the reader cannot decode `ofs-delta`, the same table by **id**.
// Kept only in that case: at a million objects it is 32 MB of object ids
// that the common path has no use for.
let mut written_ids: Vec<ObjectId> =
Vec::with_capacity(if options.ofs_delta { 0 } else { ids.len() });
// Entries gix copied as a thin delta whose base `client_has` would not
// vouch for, and which this loop therefore wrote out whole instead. gix
// has already counted them as copied, so the outcome is corrected by this
// at the end rather than left overstating the reuse.
let mut vetoed = 0usize;
let mut offered = 0usize;
let mut scratch = Vec::with_capacity(8 * 1024);
// ── `GUNNAR_THIN_VETO_LOG`: the refused bases, by name ─────────────────
//
// A count says how many deltas the voucher refused; it cannot say how many
// of those refusals were **right**. Only the receiving repository can, and
// only if it is told which bases to look for. So when this names a path,
// every refused base is written to it as `<object> <base>` and the question
// becomes one `git cat-file --batch-check` away.
//
// Read **once**, here, and not per entry: a `getenv` inside the copy loop
// would be one syscall per object served. Unset — which is every serve path
// there is — this is one `Option<PathBuf>` that is `None` and a branch the
// predictor never misses.
let veto_log = std::env::var_os("GUNNAR_THIN_VETO_LOG");
let mut veto_lines: Vec<u8> = Vec::new();
for chunk in in_order.by_ref() {
let chunk = chunk.map_err(Error::Generate)?;
for entry in chunk {
// gix marks an object it could not find with a null id and empty
// data, and would happily write it as a zero-length blob whose
// deflate stream is not a deflate stream. `write_pack` fails the
// whole pack on a missing object and so does this.
if entry.is_invalid() {
return Err(Error::ObjectVanished);
}
crate::oidprobe::bump_by(
&crate::oidprobe::GIX_PACK_ENTRY_BYTES,
entry.decompressed_size as u64,
);
crate::oidprobe::bump(match entry.kind {
output::entry::Kind::Base(_) => &crate::oidprobe::GIX_PACK_KIND_BASE,
output::entry::Kind::DeltaRef { .. } => &crate::oidprobe::GIX_PACK_KIND_DELTA_REF,
output::entry::Kind::DeltaOid { .. } => &crate::oidprobe::GIX_PACK_KIND_DELTA_OID,
});
// ── the thin-pack veto ──────────────────────────────────────────
//
// `DeltaOid` is the one kind that names a base this pack does not
// carry, and gix only produces it when `allow_thin_pack` asked it
// to. Sending one the receiver cannot resolve is not a smaller
// pack, it is a **failed fetch**, so an unvouched base costs the
// object its delta and nothing more: it is written whole, exactly
// as the non-thin path would have written it, through the same
// `add_object`.
if let output::entry::Kind::DeltaOid { id } = entry.kind {
offered += 1;
let vouched = client_has.is_some_and(|has| has(&id));
if !vouched {
if veto_log.is_some() {
veto_lines.extend_from_slice(format!("{} {}\n", entry.id, id).as_bytes());
}
let found = packed
.handle
.try_find_cached(&entry.id, &mut scratch, &mut gix_pack::cache::Never)
.map_err(|err| Error::Find {
id: entry.id,
source: err,
})?
.ok_or(Error::ObjectVanished)?;
let (data, _) = found;
let offset = writer.offset();
writer.add_object(data.kind, data.data)?;
offsets.push((offset, true));
if !options.ofs_delta {
written_ids.push(entry.id);
}
vetoed += 1;
crate::oidprobe::bump(&crate::oidprobe::GIX_PACK_THIN_VETOED);
continue;
}
}
let offset = writer.offset();
let header = if options.ofs_delta {
entry.to_entry_header(gix_pack::data::Version::V2, |index| {
let (base_offset, valid) = offsets[index];
debug_assert!(valid, "a delta base that was never written");
offset - base_offset
})
} else {
// ── the no-`ofs-delta` reader ───────────────────────────────
//
// `Entry::to_entry_header` maps `DeltaRef` to `OfsDelta`
// unconditionally, and gix offers no way to ask for the other
// form, so this is written out here rather than delegated. It is
// not a re-derivation of anything hard: the delta's **bytes** are
// untouched and only the base's *name* changes, from a distance
// backwards to the id of the entry at that index — which is the
// id gix already resolved and this loop has already written.
//
// `OBJ_REF_DELTA` whose base lies inside the same pack is what
// gix's own comment calls "a discontinued practice". Discontinued
// is not invalid: `git index-pack` resolves ref-deltas against
// the pack's own objects before looking outside it, which is
// exactly how it completes a thin pack, and
// `tests/ofs_delta_off.rs` holds stock `git` to that.
match entry.kind {
output::entry::Kind::DeltaRef { object_index } => {
let base_id =
*written_ids.get(object_index).ok_or(Error::ObjectVanished)?;
gix_pack::data::entry::Header::RefDelta { base_id }
}
// `Base` and `DeltaOid` do not name a base by distance, so
// gix's mapping is already the right one and the closure is
// never called.
_ => entry.to_entry_header(gix_pack::data::Version::V2, |_| {
unreachable!("only DeltaRef names its base by distance")
}),
}
};
offsets.push((offset, true));
if !options.ofs_delta {
written_ids.push(entry.id);
}
writer.add_entry(
&header,
entry.decompressed_size as u64,
&entry.compressed_data,
)?;
}
}
let stats = in_order
.inner
.finalize()
.map_err(Error::Generate)?;
if stats.missing_objects != 0 {
return Err(Error::ObjectVanished);
}
let mut outcome = writer.finish()?;
// The generator's own tally, not the writer's guess. These two numbers are
// the only thing that tells a pack-copy from a re-deflate, so they come
// from the code that made the choice.
//
// `vetoed` moves across the two, and moving it is not bookkeeping: gix
// counted a vetoed entry as copied because it *did* copy it, and this loop
// then threw that copy away and deflated the object instead. Leaving it on
// the copied side would make the one instrument that can see a re-deflate
// report the re-deflates it caused as reuse.
outcome.objects_copied_from_pack = stats.objects_copied_from_pack - vetoed;
outcome.objects_recompressed = stats.decoded_and_recompressed_objects + vetoed;
// The census reads the corrected pair, not gix's raw one, for the reason
// the correction exists: gix counts a vetoed entry as copied because it did
// copy it, and this function then threw that copy away.
crate::oidprobe::bump_by(
&crate::oidprobe::GIX_PACK_COPIED,
outcome.objects_copied_from_pack as u64,
);
crate::oidprobe::bump_by(
&crate::oidprobe::GIX_PACK_RECOMPRESSED,
outcome.objects_recompressed as u64,
);
outcome.thin_bases_offered = offered;
outcome.thin_bases_vetoed = vetoed;
outcome.closure_bases_declined = declined;
if let Some(path) = veto_log {
use std::io::Write as _;
// Best effort and deliberately so: this is a diagnostic, and a serve
// path that failed a fetch because a debug log could not be opened
// would be a worse bug than the one it was opened to find.
if let Ok(mut f) = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(path)
{
let _ = f.write_all(&veto_lines);
}
}
debug_assert_eq!(
outcome.objects_copied_from_pack + outcome.objects_recompressed,
outcome.num_objects as usize,
"every object is either copied or recompressed"
);
tracing::debug!(
target: crate::GIT_TARGET,
objects = outcome.num_objects,
copied = outcome.objects_copied_from_pack,
recompressed = outcome.objects_recompressed,
thin_vetoed = vetoed,
bytes = outcome.bytes_written,
"wrote a packfile by copying pack entries"
);
Ok(outcome)
}