crates/storage-git-gix/parked-gix-2/gunnar-store-gix/src/repack.rs
//! `repack()`: fold every pushed pack and every loose object into **one new
//! pack**, put it in place, and only then remove what it replaced.
//!
//! # What this is, and what it is not (P-005)
//!
//! Spec §4.4 describes `repack()` against a **znippy** cold tier. That tier is
//! not what this module builds, and the reason is recorded in
//! `.nornir/performance-design.md` **P-005**: §4 stores each git object in
//! znippy under its oid as *inflated content*, which permanently forecloses
//! pack-copy. Every object that ages into such a tier would be OpenZL-decoded
//! and then zlib-deflated again on **every** fetch, forever, which is the exact cost
//! `P-001` exists to delete. P-005's own fix is to store the *deflated pack
//! entry* instead, at which point the cold tier is a packfile with an archive
//! wrapped round it, which §4 explicitly rejects ("packs-in-znippy is
//! rejected"). Those two cannot both be satisfied, and choosing between them is
//! a design call, not an implementation detail. See this crate's `README` note
//! and the report accompanying this change.
//!
//! So the tier here is **a consolidated packfile**, which is what git's own cold
//! tier is. Nothing about that is a placeholder:
//!
//! * it reclaims: duplicate objects across packs, thin-pack completion bases
//! and loose objects collapse into one copy,
//! * it **preserves deltas and the client's own deflate**. The generator runs in
//! `PackCopyAndBaseObjects` mode, so an object that already sits in a pack is
//! copied as bytes rather than inflated and recompressed. [`RepackOutcome`]
//! reports `objects_copied_from_pack` against `objects_recompressed`, which is
//! `P-001`'s applied-output measure: a byte count and a wall clock cannot tell
//! a pack-copy from a re-deflate,
//! * it is exactly the "packed tier to reuse from" that `P-001` needs.
//!
//! # The znippy cold tier, and why it hangs off a trait (`P-005`, `P-006`)
//!
//! `P-005` is now decided in both halves. **What** the cold tier stores is the
//! *packed* representation, settled above and by construction. **Whether znippy
//! sits under it** is settled here, and the answer is yes, with the pack — not
//! the inflated object — as what znippy holds. One archive per repository; the
//! consolidated pack this function writes folds into it, keeping its `.pack` and
//! `.idx` names so znippy stores it raw instead of feeding already-deflated
//! bytes to a codec.
//!
//! It is reached through [`ColdTier`] rather than by a direct call, and that is
//! `P-006`, not taste. znippy's codec cmake-builds C and C++ from a GitHub
//! tarball fetched *during* `cargo build`. A `gunnar-store` that named znippy —
//! even behind an off-by-default feature — would put that path dependency in the
//! resolved graph of every gunnar build, and cargo reads a missing optional path
//! dependency's manifest at resolve time and fails, so the appliance build would
//! not merely get slower, it would stop existing. The implementation therefore
//! lives outside the workspace, in `crates/gunnar-coldtier`, and arrives here as
//! a trait object.
//!
//! What that costs is stated rather than hidden: a seam has to be *remembered*,
//! and [`ObjectStore::packed`](crate::ObjectStore::packed)'s own documentation
//! argues against exactly that. Two things keep it honest. [`RepackOutcome`]
//! carries [`folded`](RepackOutcome::folded), so "no cold tier was configured"
//! and "a cold tier ran" are different values a test can tell apart rather than
//! a silence. And a fold that fails **fails the repack**, before a single old
//! pack is unlinked — see the ordering note at step 3b.
//!
//! # The swap protocol (§4.4, D14)
//!
//! Never in place. In order:
//!
//! 1. **snapshot** which packs and which loose objects are being folded. Only
//! these are ever deleted, so a push that lands during the repack is safe by
//! construction rather than by a lock.
//! 2. **write** the new pack to a temporary file, then hand it to
//! `gunnar-pack`'s indexer, which persists `pack-<id>.pack` and then
//! `pack-<id>.idx` into `objects/pack` through `gix-tempfile`. `gix-odb`
//! discovers packs *by their `.idx`*, so the new pack becomes visible at the
//! instant the index rename lands. One atomic step, git's own trick.
//! 3. **verify** by opening the index that landed and looking up every object
//! that was meant to be in it.
//! 4. **fold into the cold tier**, if one is configured. Here and not after
//! step 5 on purpose: at this point the new pack is durable and *every old
//! pack is still on disk*, so a fold that fails costs nothing but a retry.
//! Folding after the deletions would mean a failed fold left a repository
//! with no archive and no way back to one.
//! 5. **only then** unlink the folded packs (`.idx` first) and the folded loose
//! objects.
//!
//! Readers serve from the old packs for the whole of 1–3, and from either during
//! 4, because every object is in both. A crash anywhere leaves the old packs
//! untouched and at worst a stale `.tmp` and a duplicate pack behind, which the
//! next run sweeps up; the operation is simply retried.
//!
//! # Consolidating and re-encoding are two different operations
//!
//! Everything above is **consolidation**: many containers become one, and every
//! delta that already existed is carried across as bytes. It reclaims
//! duplicates and it does not improve compression, because
//! `Mode::PackCopyAndBaseObjects` computes no deltas — by design, since
//! computing one for an object that already has a good one is pure cost.
//!
//! **Re-encoding** is the other operation, and until 2026-08-04 this module
//! could not do it at all. [`RepackOptions::delta_policy`] is the switch, and
//! [`DeltaOrder`] is what makes it worth switching. The two are separate
//! because they have opposite cost profiles and opposite triggers: consolidation
//! is cheap and is triggered by *how many* packs there are, re-encoding is
//! expensive and is triggered by *how bad* the deltas in them are. Conflating
//! them is what produced a `gc` that could only ever run the cheap half.
//!
//! # The branch that had to go, and the shape that replaces it
//!
//! A repository with one pack, no loose objects and nothing to prune used to
//! `return` from the middle of this function. Three separate features were then
//! found to be unreachable in practice because of it, each fixed by bolting one
//! more step onto that early return:
//!
//! * the reachability bitmap — 18 of 19 real repositories got none (2026-08-04);
//! * the znippy cold-tier fold, and with it `__gunnar_refs__`, the section the
//! whole tier exists for — same 18 of 19;
//! * delta re-encoding, which is this change, and which is worth the most of
//! the three because gunnar's serve path copies pack entries verbatim, so a
//! pack's deltas *are* the wire's deltas for the rest of that pack's life.
//!
//! Every one of those had green fixture tests, because **a fixture built by
//! three pushes holds three packs and never takes the branch**. A third patch
//! to the same early return would have been the third guarantee that a fourth
//! step would be forgotten, so there is no early return any more. The function
//! now decides *what to produce* — a new pack, or the one already there — and
//! then runs **one tail**: bitmap, cold-tier fold, retire what was replaced,
//! report. A step added to that tail cannot be added to only half the
//! repositories, because there is no other half.
//!
//! # Single writer
//!
//! [`FsStore`] already holds the Arrow log's exclusive lock for the lifetime of
//! the handle, so a second *process* cannot open the same git directory at all
//! (§4.5, and [`crate::Error::StoreLocked`]). Within one process a mutex
//! serialises repacks. That is the same one-writer rule `P-014` demands of the
//! znippy tier, obtained here without a new lock file, and `S-019`'s lesson
//! about locks that outlive their holder does not apply, because nothing new is
//! created on disk to go stale.
//!
//! It is also why a `gc` against a **running** server is refused rather than
//! serialised: the server process holds that lock for its lifetime, so
//! `gunnar gc` from a second process fails with [`crate::Error::StoreLocked`]
//! and maintenance means stopping the forge. That is a real limitation and it
//! is not fixed here; what it would take is recorded in this change's report.
use std::collections::BTreeSet;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use gix_hash::ObjectId;
use crate::connectivity;
use crate::error::{Error, Result};
use crate::fs::FsStore;
/// The archive a repack folds its consolidated pack into: one znippy archive per
/// repository.
///
/// Implemented outside this workspace, in `crates/gunnar-coldtier`. See the
/// [module docs](self) for why that boundary exists (`P-006`) and what it costs.
///
/// # It is handed the pack, not the objects
///
/// The two paths are the whole of `P-005`. Given the objects, an implementation
/// would store inflated content per oid, and every fetch from that tier would
/// pay an OpenZL decode plus a fresh zlib deflate, for ever, with the delta
/// chains already thrown away. Given the *pack*, it stores bytes that are
/// already deflated and already deltified, which znippy detects and stores raw.
/// So the signature carries `pack` and `index` and no object iterator, and that
/// is a design decision expressed as a type rather than as a comment somebody
/// has to obey.
pub trait ColdTier: Send + Sync {
/// Fold `request` into the archive. Returning `Err` aborts the repack with
/// nothing deleted.
fn fold(&self, request: &FoldRequest<'_>) -> std::result::Result<FoldReport, FoldError>;
}
/// Whatever went wrong inside a [`ColdTier`], boxed so this crate names none of
/// the archive format's error types.
pub type FoldError = Box<dyn std::error::Error + Send + Sync + 'static>;
/// What a repack hands its cold tier.
///
/// Refs travel with the pack because an archive of objects nobody can name is
/// not a repository. **Secrets deliberately do not.** The archive format
/// reserves `__gunnar_secrets__` for already-encrypted material, and gunnar's
/// secret store holds the SSH host key in the clear, so sealing it would move a
/// plaintext private key into a file whose whole purpose is to be copied
/// elsewhere. Encrypting on the way in is the fix, and it is not written.
#[derive(Debug)]
pub struct FoldRequest<'a> {
/// The repository being folded.
pub git_dir: &'a Path,
/// The consolidated `pack-<id>.pack` that just landed.
pub pack: &'a Path,
/// Its `pack-<id>.idx`.
pub index: &'a Path,
/// The repository's hash kind. sha1 and sha256 repositories never mix.
pub hash_kind: gix_hash::Kind,
/// How many objects the pack holds.
pub objects: u32,
/// Every reference, and `HEAD` under its own name, as of the fold.
pub refs: &'a [(String, crate::refs::Target)],
}
/// What one fold turned out to be.
///
/// Every field is an **applied output** — a file that exists, bytes that were
/// written — rather than a restatement of the request (LAW 2). In particular
/// `entries` is read back out of the sealed archive, so an archive that sealed
/// nothing cannot report that it sealed two files.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct FoldReport {
/// The archive that now holds this repository.
pub archive: PathBuf,
/// Data entries the sealed archive lists. Two for a folded pack tier: the
/// `.pack` and its `.idx`.
pub entries: usize,
/// Bytes of pack and index that went in.
pub bytes_in: u64,
/// Bytes of archive that came out.
pub bytes_out: u64,
/// Refs sealed into `__gunnar_refs__`, counted by reading the section back.
pub refs_sealed: usize,
}
/// What to do about the deltas that are already there.
///
/// # Why this is a policy and not a `bool`
///
/// The two obvious answers are both wrong for the case that matters.
///
/// *Never recompute* is what this module did until 2026-08-04, and it means a
/// repository can never get better than the client that pushed into it. gunnar's
/// own client pushed **0.0% deltified** packs before `write_pack_deltified`
/// landed (19 real repositories, 2026-08-03: 407.08 MiB stored against stock
/// git's 88.14 MiB), and `P-001`'s verbatim copy then propagates that to every
/// clone for ever. Consolidation cannot fix it, because consolidation copies.
///
/// *Always recompute* makes every `gc` a full window search over every object,
/// including the overwhelmingly common case where git or a modern gunnar client
/// already computed good deltas and the search will re-derive the same ones at
/// full price.
///
/// So the default is [`Self::Auto`]: **look at what is actually in the pack**
/// and recompute only when it is bad. The measurement is cheap — pack entry
/// headers, no inflation, no delta application — and it is an applied output,
/// not a guess about what the client might have been.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum DeltaPolicy {
/// Copy every delta that exists and compute none.
///
/// `git repack` without `-f`, near enough. The right answer when the input
/// is known-good and the only goal is to reclaim duplicates.
Preserve,
/// Recompute every delta with a full window search.
///
/// `git repack -adf`, and `git gc --aggressive`'s expensive half. Use it
/// after changing [`RepackOptions::delta_order`], since a better ordering
/// buys nothing until the search is re-run under it.
Recompute,
/// Recompute only if fewer than `min_delta_fraction` of the objects
/// currently in the repository's packs are deltas.
///
/// The default, at `0.20`. Healthy packs sit far above it — stock git
/// measures 74.0% on gunnar's own 19-repository corpus — and a pack written
/// by a client with no encoder sits at 0.0%, so the threshold separates the
/// two cases by a factor of three in either direction rather than splitting
/// a hair. Loose objects count as non-deltas, because that is what they are.
Auto {
/// Below this fraction, re-encode. `0.0` never fires; `1.0` always does.
min_delta_fraction: f32,
},
}
impl Default for DeltaPolicy {
fn default() -> Self {
DeltaPolicy::Auto {
min_delta_fraction: 0.20,
}
}
}
/// How a re-encoding repack decides which objects are alike, i.e. which of them
/// the delta window ever offers each other.
///
/// This is the single highest-value knob in the search, and it is not a tuning
/// parameter: a base that never shares a window with its target cannot be found
/// however good the encoder is. See [`crate::pathwalk`] for the walk that
/// produces it and for the published numbers behind each variant.
///
/// It has **no effect** under [`DeltaPolicy::Preserve`], and none on a repack
/// that turns out not to need re-encoding: copying a delta does not consult an
/// ordering.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum DeltaOrder {
/// `(kind, size descending)`. No tree walk, and no idea which objects are
/// versions of the same file.
Size,
/// git's original `pack_name_hash`: effectively the last sixteen
/// non-whitespace bytes of the path, so every `index.ts` in a monorepo
/// collides into one group.
NameHashV1,
/// git 2.49's `pack_name_hash_v2`, per path component with the parent
/// directories folded in.
///
/// MEASURED BY Jonathan Blau on microsoft/fluentui, `git repack -adf` with
/// nothing else changed: 439 MiB → 160 MiB *and* 96 s → 34 s.
///
/// It was the default until 2026-08-04. It is the right choice for a
/// repository that is mostly **cloned**: it keeps size-descending inside a
/// group, which is what compresses a whole history, and on gunnar's
/// 19-repository corpus it produces a 6.2% smaller archive than
/// [`Self::TransferV2`] — and a fetch 50 commits behind that is 1.85× the
/// size.
NameHashV2,
/// One dense number per distinct path: the grouping git's `--path-walk`
/// produces, with no collisions at all.
///
/// MEASURED BY Derrick Stolee, same command, same repository: 438 MB → 148
/// MB, and 130 049 MB → 4 432 MB on a private monorepo. Not strictly better
/// than [`Self::NameHashV2`] — a path with one single version gets no
/// candidates at all here, where a name hash would still have offered it
/// the similarly-named files — so which wins is measured, not assumed.
///
/// # It trades the fetch for the clone, MEASURED 2026-08-04
///
/// gunnar's own bake-off corpora, one `gunnar serve`, stock `git` as the
/// client, bytes counted as the pack the client wrote:
///
/// | corpus | operation | copied | path-walk |
/// |---|---|---:|---:|
/// | `small` | clone | 119 102 958 | **89 840 705** (0.754×) |
/// | `small` | fetch, 50 commits behind | 962 922 | 1 000 403 (1.039×) |
/// | `large` | clone | 9 705 923 | 9 705 923 (the search lost; copy kept) |
///
/// A quarter off a clone and four percent onto a fetch, and the reason is
/// the same fact in both directions: grouping strictly by path picks bases
/// from anywhere in the file's history, which is ideal when the whole
/// history is being sent and wrong when only the last fifty commits are.
/// **It is not the default for that reason**, and it is worth choosing
/// deliberately on a repository that is mostly cloned.
PathWalk,
/// [`Self::NameHashV2`]'s grouping, with the chains inside each group
/// pointed the way a **fetch** is sent: oldest revision as the base, newer
/// revisions as deltas of it.
///
/// # The regression this exists to remove, MEASURED 2026-08-04 on oden
///
/// gunnar's serve path copies pack entries verbatim (`P-001`), so the
/// direction the repacker chose is the direction on the wire. Every other
/// ordering here sorts a group by **size descending**, which is git's
/// `type_size_sort` and is blind to time — so roughly half the time the
/// newest revision of a file ends up as the base and the older ones as its
/// deltas. A clone does not care. A fetch pays for it twice: the newest
/// revision is exactly what a fetch is sending, and as a base it has to
/// travel whole.
///
/// The effect is not marginal. 19 real `workspace_nordisk` repositories
/// pushed into gunnar by stock git, then fetched 50 commits behind by stock
/// git 2.53.0, bytes counted on the wire by one relay:
///
/// | store | fetch bytes | deltas in the sent set with a client-side base |
/// |---|---:|---:|
/// | as pushed, never repacked | 6 843 048 | 47 of 207 (korp) |
/// | after `gunnar gc` | 13 235 040 | **0** of 159 (korp) |
///
/// `gc` did not lose the deltas — korp's delta fraction moved 67.3% → 66.0%
/// — it **turned them round**. Every delta in the fetch set whose base sat
/// on the client's side of the cut became a whole object.
///
/// # What it costs
///
/// Dropping size-descending inside a group is a real loss to whole-history
/// compression, and this variant is not the default for that reason. The
/// clone and the fetch are reported as separate rows in this change's
/// report, because they move in opposite directions and one number would
/// hide it.
///
/// # Why it is the default from 2026-08-04, and where it is the wrong one
///
/// MEASURED on oden, 19 real `workspace_nordisk` repositories pushed into a
/// gunnar store by stock git, `gc --aggressive` under each ordering, bytes
/// counted on the wire by one relay, stock git 2.53.0 as the client. Clone
/// and fetch as separate rows, because they move opposite ways:
///
/// | | name-hash-v2 | transfer-v2 | ratio |
/// |---|---:|---:|---:|
/// | fetch, 50 behind (16 repos) | 11 903 316 | **6 425 160** | **0.540×** |
/// | fetch, 100 behind (14) | 26 837 056 | 21 142 744 | 0.788× |
/// | fetch, 200 behind (12) | 42 129 724 | 38 130 496 | 0.905× |
/// | fetch, 400 behind (7) | 49 216 530 | 47 734 162 | 0.970× |
/// | fetch, 800 behind (2) | 14 807 444 | 15 825 156 | 1.069× |
/// | clone (19) | 92 563 934 | 98 269 038 | 1.062× |
///
/// The instrument's own noise floor, measured as the difference between two
/// arms serving byte-identical stores, is **0.071%** — so every ratio above
/// is a result and not a wobble.
///
/// **The crossover is the whole question and it was measured, not
/// extrapolated.** The win shrinks with depth and does eventually invert,
/// because a client far enough behind is doing a clone. Per repository it
/// inverts at 60–90% of the repository's own history for 15 of the 16, and
/// at ~15% for the one outlier (`workspace_nordisk`). A client that far
/// behind is not fetching, and that is why this is the default rather than
/// a flag.
///
/// It is the wrong default for a repository that is mostly **cloned** —
/// CI that clones fresh every run, a release mirror, an artefact store.
/// Those want [`Self::NameHashV2`] or [`Self::PathWalk`], and the 6.2%
/// archive difference is what they are choosing between.
#[default]
TransferV2,
}
impl DeltaOrder {
/// Whether this key needs the trees walked at all.
pub fn needs_paths(self) -> bool {
!matches!(self, DeltaOrder::Size)
}
/// Whether the walk has to date the objects as well as name them.
///
/// Separate from [`Self::needs_paths`] because it costs a full commit
/// decode and a sort of the root trees, and no other ordering reads the
/// answer. An ordering that returned `true` here and never consulted the
/// epoch would pay for it silently, which is what `Grouping::aged` is
/// reported for.
pub fn needs_epochs(self) -> bool {
matches!(self, DeltaOrder::TransferV2)
}
}
/// How recently an object may have been written and still survive
/// `--prune`: **two weeks**, stock git's `gc.pruneExpire`.
///
/// # The floor, swept 2026-08-05 on oden
///
/// The quantity this has to cover is *how long a push's objects can sit in the
/// object database with no reference pointing at them*. gunnar has no
/// quarantine, so that is the whole span from the first object write to the ref
/// transaction: the pack index, the policy gate (signature verification, a
/// protection rule, a pre-receive hook) and the connectivity check.
///
/// Swept against `measure_concurrent_prune_with_gate`, which is that span made
/// explicit. At grace `0` the loss is **1 of 1 overlapping pushes at every gate
/// width tried — 200 µs, 2 ms, 20 ms and 200 ms**; there is no width at which
/// reachability alone is safe, because the walk and the window do not race, the
/// walk simply sees a window that is open. At any grace **longer than the
/// fixture's own age** the loss is **0**, likewise at every width.
///
/// So the floor is not a duration this code can derive — it is "longer than the
/// slowest gate any deployment will ever run", and a gate is an operator's hook.
/// Two weeks is not a measured optimum, it is git's answer to the same
/// unanswerable question, and matching it means an operator who knows
/// `gc.pruneExpire` already knows this. What the sweep does establish is the
/// shape: **anything above zero fixes it and zero never does**, so the risk of
/// this constant is entirely in the direction of someone setting it to zero.
pub const DEFAULT_PRUNE_GRACE: std::time::Duration =
std::time::Duration::from_secs(14 * 24 * 60 * 60);
/// How to repack.
///
/// `Default` is hand-written rather than derived, because `write_bitmap`'s
/// default is `true` and `bool`'s is `false`. A derive here would ship the
/// bitmap writer switched off with nothing to say so.
#[derive(Clone)]
pub struct RepackOptions {
/// Drop objects no reference can reach, instead of carrying every object
/// across.
///
/// **Off by default, and that default is the safe one.** Pruning is what
/// finally reclaims a superseded control document (`S-001`/`S-002`) or the
/// objects of a force-pushed-away branch, and it is also the only thing here
/// that can *lose* data: an object written but not yet referenced (a push
/// whose pack has landed while its ref transaction has not) is unreachable
/// by definition.
///
/// That window is closed by [`prune_grace`](Self::prune_grace), not by this
/// flag. Until 2026-08-05 this doc said the window was "small, but not zero
/// and not closed by anything in this module", and that was an
/// understatement worth correcting: MEASURED, a pruning repack whose
/// reachability walk overlaps a push destroys that push **every time**, with
/// no interleaving to get lucky with, and the dangling reference it leaves
/// behind then makes every later `--prune` on that repository fail outright.
/// See `tests/repack_under_load.rs`.
pub prune_unreachable: bool,
/// How recently an object may have been written and still be safe from
/// [`prune_unreachable`](Self::prune_unreachable).
///
/// An unreachable object whose holding file — its pack, or its loose file —
/// was last written inside this window is **carried across rather than
/// dropped**. That is the mechanism stock git uses and the reason
/// `git gc`'s prune is `--prune=2.weeks.ago` by default rather than
/// `--prune=now`: reachability is not a safe answer for an object that
/// arrived a moment ago, because a push writes its objects before it moves
/// its reference, and in that gap its whole payload is indistinguishable
/// from garbage.
///
/// gunnar has **no receive-pack quarantine** — objects land in the ODB
/// before the policy gate runs, which is a written position guarded by
/// `a_refused_push_leaves_its_objects_in_the_object_database` — so this is
/// the only thing standing between a scheduled `gc --prune` and a live
/// push's payload.
///
/// [`Duration::ZERO`] means `--prune=now`: prune purely on reachability,
/// which is only safe when nothing can be writing to the repository.
///
/// The default is [`DEFAULT_PRUNE_GRACE`].
pub prune_grace: std::time::Duration,
/// Cap the threads **the pack generator** uses. `None` means all cores.
///
/// The parallelism is `gix-pack`'s own, over `gix-features`' std-thread
/// pool. gunnar writes no scheduler here and pulls in no rayon (LAW 3).
///
/// # It does not bound the repack — MEASURED 2026-08-03
///
/// It reaches `output::count::objects` and `output::entry::iter_from_counts`
/// and nothing else. [`FsStore::write_multi_pack_index`] also runs inside
/// [`FsStore::repack`], and `gix_pack::multi_index::write_from_index_paths`
/// takes **no thread limit at all** —
/// `gix_pack::multi_index::write::Options` carries only `object_hash`. So a
/// repack of a million objects over 64 packs peaked at **33 threads with
/// `Some(1)` and 33 with `None`** (`gunnar.repack_1m`, oden), i.e. an
/// operator who sets this to keep a repack off the serving cores does not get
/// that. What the setting does change is throughput: **247 052 objects/s
/// serial against 324 941/s unrestricted, 1.315×** in that run, 1.30–1.55×
/// across three.
///
/// Left as it is rather than papered over. Bounding the whole repack means a
/// thread knob upstream in `gix-pack`, or writing the multi-pack index in a
/// separate step the caller schedules; both are larger than a doc comment and
/// neither is worth guessing at. `P-011`.
pub thread_limit: Option<usize>,
/// Cancel a running repack.
///
/// Checked between phases and threaded into the object count. Setting it
/// makes [`FsStore::repack`] return [`Error::Interrupted`] with the old tier
/// untouched, which is also how a `gunnar gc` on a large repository is
/// meant to be stopped.
pub should_interrupt: Option<Arc<AtomicBool>>,
/// Write a `.bitmap` reachability index beside the new pack.
///
/// **On by default, which is what stock git does.** `repack.writeBitmaps`
/// defaults to true for a bare repository, and a forge is bare by
/// definition. A pack without one is served by walking the commit graph,
/// which is the cost `crates/gunnar-bitmap` exists to remove.
///
/// This is batch-tier work and it is where `P-020`'s two tiers differ: the
/// serve path must not fan out, but `repack` may, and the bitmap build is
/// the same one-at-a-time offline job the pack write already is.
///
/// Turning it off is for a caller that has measured the write cost and does
/// not want it, not for one that suspects the bitmap is wrong. A bitmap
/// that cannot be written honestly is not written at all — see
/// [`crate::bitmap`] — so `false` here and "no bitmap appeared" are
/// different situations, and [`RepackOutcome::bitmap`] tells them apart.
pub write_bitmap: bool,
/// Fold the consolidated pack into a znippy archive as well (`P-005`).
///
/// `None` means the repository's cold tier is the packfile and nothing
/// else, which is what every caller did before this existed and remains the
/// default. See [`ColdTier`].
pub cold_tier: Option<Arc<dyn ColdTier>>,
/// Whether to recompute the deltas rather than copy them. See
/// [`DeltaPolicy`]; the default is [`DeltaPolicy::Auto`].
///
/// **A repository with one pack is not exempt.** That is the whole point of
/// this field existing: one pack is exactly the state a repository seeded by
/// a single push is in, it is the state 18 of the 19 real repositories in
/// gunnar's corpus were in, and it is the state the old `already
/// consolidated` short-circuit returned from without doing anything.
///
/// Requires the `repack-delta` feature. Without it, anything but
/// [`DeltaPolicy::Preserve`] that actually decides to re-encode fails the
/// repack with a message saying so, rather than quietly copying — an
/// operator who asked for `-f` and got a silent no-op is how a forge ends up
/// believed to have been maintained.
pub delta_policy: DeltaPolicy,
/// Which objects a re-encoding repack considers alike. See [`DeltaOrder`].
///
/// Ignored unless the repack actually re-encodes.
pub delta_order: DeltaOrder,
/// How many earlier objects, in [`delta_order`](Self::delta_order), each
/// object is offered as a delta base. git's default is 10 and so is this.
///
/// It was a hardcoded `10` until 2026-08-04, which was not a decision: it
/// was `DeltaOptions::default()` reaching through a `..Default::default()`
/// and never being named. The window is the second half of the ordering
/// question and the two only make sense together — a base that shares a
/// group with its target but sits more than `window` positions away in that
/// group is exactly as unreachable as one that never shared a group at all.
///
/// Ignored unless the repack actually re-encodes.
pub delta_window: usize,
/// Longest delta chain the search may build. git's default is 50 and so is
/// this. A reader pays one object reconstruction per link; gunnar's *serve*
/// path pays none, because it copies pack entries (`P-001`) and never
/// applies a delta.
///
/// Ignored unless the repack actually re-encodes.
pub delta_depth: usize,
}
impl Default for RepackOptions {
fn default() -> Self {
RepackOptions {
prune_unreachable: false,
prune_grace: DEFAULT_PRUNE_GRACE,
thread_limit: None,
should_interrupt: None,
// Stock git's `repack.writeBitmaps` default for a bare repository,
// and a forge is bare.
write_bitmap: true,
cold_tier: None,
delta_policy: DeltaPolicy::default(),
delta_order: DeltaOrder::default(),
// git's own `pack.window` and `pack.depth`. Named here rather than
// inherited silently from `gunnar_pack::DeltaOptions::default()`,
// so that a reader of this struct can see what the search is
// actually run with.
delta_window: 10,
delta_depth: 50,
}
}
}
impl std::fmt::Debug for RepackOptions {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("RepackOptions")
.field("prune_unreachable", &self.prune_unreachable)
.field("prune_grace", &self.prune_grace)
.field("thread_limit", &self.thread_limit)
.field("should_interrupt", &self.should_interrupt)
.field("write_bitmap", &self.write_bitmap)
.field("delta_policy", &self.delta_policy)
.field("delta_order", &self.delta_order)
.field("delta_window", &self.delta_window)
.field("delta_depth", &self.delta_depth)
// A trait object has no useful Debug; whether one is configured is
// the whole of what a reader needs, and it is the thing that would
// otherwise be invisible.
.field("cold_tier", &self.cold_tier.is_some())
.finish()
}
}
impl RepackOptions {
/// The safe defaults: fold everything, prune nothing, use every core.
pub fn new() -> Self {
Self::default()
}
/// Whether to write a `.bitmap` beside the new pack. See
/// [`RepackOptions::write_bitmap`]; the default is on.
pub fn write_bitmap(mut self, write: bool) -> Self {
self.write_bitmap = write;
self
}
/// Also drop objects no reference can reach. See
/// [`RepackOptions::prune_unreachable`] before turning this on.
pub fn prune_unreachable(mut self, prune: bool) -> Self {
self.prune_unreachable = prune;
self
}
/// How young an unreachable object may be and still survive a prune. See
/// [`RepackOptions::prune_grace`]; the default is [`DEFAULT_PRUNE_GRACE`].
///
/// [`Duration::ZERO`](std::time::Duration::ZERO) is `--prune=now`, and is
/// only safe against a repository nothing is writing to.
pub fn prune_grace(mut self, grace: std::time::Duration) -> Self {
self.prune_grace = grace;
self
}
/// Cap the pack generator's threads.
pub fn thread_limit(mut self, limit: usize) -> Self {
self.thread_limit = Some(limit);
self
}
/// Give the repack a cancellation flag.
pub fn should_interrupt(mut self, flag: Arc<AtomicBool>) -> Self {
self.should_interrupt = Some(flag);
self
}
/// Also fold the consolidated pack into a znippy archive. See [`ColdTier`].
pub fn cold_tier(mut self, tier: Arc<dyn ColdTier>) -> Self {
self.cold_tier = Some(tier);
self
}
/// What to do about the deltas already in the packs. See [`DeltaPolicy`].
pub fn delta_policy(mut self, policy: DeltaPolicy) -> Self {
self.delta_policy = policy;
self
}
/// Which objects a re-encoding repack considers alike. See [`DeltaOrder`].
pub fn delta_order(mut self, order: DeltaOrder) -> Self {
self.delta_order = order;
self
}
/// How far back in the ordering the search looks for a base, and how long a
/// chain it may build. See [`RepackOptions::delta_window`] and
/// [`RepackOptions::delta_depth`]; git's defaults, 10 and 50.
///
/// One setter for both because they are one decision: a wider window with
/// no depth to spend it on finds bases it is then not allowed to use.
pub fn delta_search(mut self, window: usize, depth: usize) -> Self {
self.delta_window = window;
self.delta_depth = depth;
self
}
}
/// What a repack turned out to be.
///
/// Every field is an **applied output**, something that happened on disk or in
/// the generator, rather than a restatement of the request (LAW 2).
// `Eq` is gone, and deliberately: `delta_fraction_before`/`_after` are `f32`,
// because a fraction of objects is a fraction. Rounding them to integers to keep
// a derive would be tuning the measurement to fit the type.
#[derive(Debug, Clone, PartialEq, Default)]
pub struct RepackOutcome {
/// `false` when there was nothing to fold and no bytes were written.
///
/// A repack of an already-consolidated repository must be a **no-op**, not a
/// rewrite: rewriting would churn the pack id, invalidate every reader's
/// mmap and reclaim nothing.
pub rewrote: bool,
/// How many objects the new pack holds.
pub objects: u32,
/// The new pack's id, i.e. the `<id>` in `pack-<id>.pack`.
pub pack_id: Option<ObjectId>,
/// How many packs were folded and then removed.
pub packs_folded: usize,
/// How many loose objects were folded and then removed.
pub loose_folded: usize,
/// How many objects were dropped as unreachable. Always 0 unless
/// [`RepackOptions::prune_unreachable`] was set.
pub objects_pruned: usize,
/// Objects whose already-compressed pack entry was **copied**: no inflate,
/// no deflate, delta chains preserved.
///
/// This and the next field are `P-001`'s measure. A repack that silently
/// stopped copying would still produce a correct pack of the right size
/// class, and only these two numbers would show it.
pub objects_copied_from_pack: usize,
/// Objects that had to be decoded and deflated again: loose objects, and
/// deltas whose base did not come along.
pub objects_recompressed: usize,
/// Bytes of pack, index and loose files that were removed.
pub bytes_folded: u64,
/// Bytes of the pack and index that replaced them.
pub bytes_written: u64,
/// The `.bitmap` that was written, or `None` when none was.
///
/// A path rather than a bool, and `None` covers three different situations
/// on purpose — not asked for, nothing to write one over, or a closure that
/// left the pack. All three are "this pack will be served by walking", and
/// the distinction that matters to a caller is whether a file exists.
///
/// It is reported at all because a repack that silently stopped writing
/// bitmaps would produce a correct pack of the right size, pass every test
/// in this file, and quietly hand the serve path its slow route back. That
/// is the shape `objects_copied_from_pack` exists for one field up.
pub bitmap: Option<PathBuf>,
/// What the cold tier did, or `None` when none was configured.
///
/// A value rather than a silence, so that "the archive was not written" and
/// "no archive was asked for" are distinguishable at the call site. A test
/// asserting only on `rewrote` cannot tell a wired fold from an unwired one,
/// which is exactly how a cold tier ships absent behind a green suite.
pub folded: Option<FoldReport>,
/// Objects whose delta this run **computed**, as opposed to copied.
///
/// Zero on every consolidating repack, and that is not a defect: those copy.
/// It is non-zero only when [`RepackOptions::delta_policy`] decided to
/// re-encode, and it is the applied output that says so. `rewrote` cannot:
/// a repack that produced a new pack out of copied entries and one that
/// re-derived every delta from scratch both set it.
pub objects_deltified: usize,
/// The fraction of objects that were deltas **before** this run, and after
/// it.
///
/// `None` when it could not be read (no packs, or an index that would not
/// open). This is the number [`DeltaPolicy::Auto`] decides on, reported
/// rather than kept private, because "the policy declined to re-encode" and
/// "the policy never ran" look identical from every other field, and the
/// first of those is a legitimate outcome while the second is the bug this
/// whole change exists to remove.
pub delta_fraction_before: Option<f32>,
/// See [`Self::delta_fraction_before`]. Read back off the pack that is on
/// disk when the repack finishes, never predicted from the encoder's own
/// report.
pub delta_fraction_after: Option<f32>,
/// A re-encode ran, produced a pack that was **not smaller** than the one it
/// would have replaced, and was therefore thrown away.
///
/// The third thing `rewrote: false` would otherwise mean, and the one that
/// costs real time: "nothing to do", "no encoder in this build" and "the
/// search ran for a minute and its answer was worse" are three different
/// facts about a `gc`, and an operator watching a repository not shrink is
/// entitled to know which. It is also the only way a test can tell the guard
/// firing from the guard never being reached.
pub reencode_declined: bool,
}
/// The `.pack`/`.idx` pair of one pack, plus the sidecars git may have written.
#[derive(Debug, Clone)]
struct PackFiles {
index: PathBuf,
data: PathBuf,
sidecars: Vec<PathBuf>,
}
impl PackFiles {
fn all(&self) -> impl Iterator<Item = &Path> {
// `.idx` first at delete time: it is what `gix-odb` finds a pack
// through, so removing it is what makes the pack stop being used.
std::iter::once(self.index.as_path())
.chain(std::iter::once(self.data.as_path()))
.chain(self.sidecars.iter().map(PathBuf::as_path))
}
fn bytes(&self) -> u64 {
self.all()
.filter_map(|p| std::fs::metadata(p).ok())
.map(|m| m.len())
.sum()
}
}
/// The prefix every temporary file this module writes carries, so a run that
/// died can recognise and sweep its own leftovers.
pub(crate) const TMP_PREFIX: &str = "gunnar-repack-";
impl FsStore {
/// Write a `.bitmap` for the pack this repack is leaving behind, unless
/// one is already there.
///
/// **One writer for both producers** (LAW 5). The pack may have been
/// written by this run two steps ago, or may have been on disk since the
/// first push; the bitmap is a pure function of the pack either way. This
/// was two functions until 2026-08-04, and the second one had to be written
/// months after the first, once 18 of 19 real repositories were found to
/// have no bitmap at all because they never reached the first.
///
/// Best effort: a failure is logged and swallowed. The bitmap is an index,
/// not data — the pack is complete, verified and readable without it — so
/// failing a finished repack over one would turn a performance feature into
/// an availability risk. `None` is what a caller checks.
fn bitmap_for(&self, index_path: &Path, hash_kind: gix_hash::Kind) -> Option<PathBuf> {
// Never rewrite one that is already there. It may be stock git's, it is
// valid for this pack either way, and rewriting costs a full closure
// walk to produce a file that already exists. A pack this run wrote has
// a fresh id, so it reaches this only when the rewrite reproduced a pack
// that was already on disk byte for byte — in which case the existing
// bitmap is the right one.
if crate::bitmap::path_for(index_path).exists() {
return None;
}
match crate::bitmap::write::write_for_pack(index_path, hash_kind, &self.objects()) {
Ok(path) => path,
Err(err) => {
tracing::warn!(
error = %err,
pack = %index_path.display(),
"repack: the pack is complete but its reachability bitmap could not be \
written; this repository will be served by walking"
);
None
}
}
}
/// Generate the consolidated pack by **re-encoding** every delta, rather
/// than copying the ones that are already there.
///
/// This is the half `gix-pack` does not have: `iter_from_counts` has exactly
/// one `Mode` and it is a delta *copier*. The encoder is `gunnar-delta`,
/// reached through `gunnar-pack`'s `write_pack_deltified_ordered`, which is
/// the same writer, the same deflate scratch and the same trailer hasher
/// every other pack path uses.
///
/// The ordering is computed first and it is not a detail — see
/// [`crate::pathwalk`]. A window search under the wrong order is a full
/// window search that finds nothing.
#[cfg(feature = "repack-delta")]
fn write_recoded_pack_file(
&self,
path: &Path,
ids: &[ObjectId],
hash_kind: gix_hash::Kind,
options: &RepackOptions,
interrupt: &AtomicBool,
) -> Result<(ObjectId, Generated)> {
use std::io::Write as _;
let tips = connectivity::ref_tips(&self.refs())?;
let grouping = crate::pathwalk::group(&self.objects(), &tips, options.delta_order)?;
// Applied output, and it is the one that would be silent otherwise: a
// walk that reached no trees produces a `Grouping` that answers 0 for
// everything, which is a *valid* input to the search and looks exactly
// like "this repository has no paths".
tracing::info!(
target: gunnar_core_target::STORE,
order = ?options.delta_order,
window = options.delta_window,
depth = options.delta_depth,
paths = grouping.paths,
grouped = grouping.grouped,
aged = grouping.aged(),
roots = grouping.roots,
objects = ids.len(),
"repack: computing deltas under a path ordering"
);
// A `TransferV2` that dated nothing is a `NameHashV2` wearing another
// name: the epoch key would be `0` everywhere and the sort would fall
// straight back to size-descending, with every other number in the line
// above looking healthy. It is the exact shape of the silent green this
// tree has found five times, so it is a refusal and not a warning.
//
// `roots > 0` SHARPENS it, and the difference was found by making
// `TransferV2` the default on 2026-08-04: `aged() == 0` alone fires on
// TWO states and only one of them is a fault.
//
// * **No root at all.** No reference reaches a commit or a tree, so the
// walk had nothing to descend. Every ordering degenerates to
// size-descending there — `NameHashV2` included, since there are no
// paths either — so `TransferV2` is not masquerading as anything and
// nothing is hidden, which is the whole test of whether a silence is
// a lie. `paths=0 grouped=0 aged=0 roots=0` on the line above says so
// as an applied output. It is also a repository `gunnar gc` must
// still be able to consolidate: three tests build exactly it
// (`gc_repacks_every_repository_it_lists`,
// `a_repack_invalidates_the_multi_pack_index_it_made_stale`,
// `a_repacked_repository_reopens_without_a_stale_index`) and all
// three went red on the unqualified condition.
// * **Roots, and no dates.** There was history and the walk did not
// date it. Nothing in `pathwalk::group` can produce that — the epoch
// is `rank + 1`, never zero — so it is precisely the state a
// regression in that function would create, and it is what this
// refusal is for. `the_transfer_order_refuses_a_walk_that_dated_no_history`
// reaches it by stubbing the epoch to `0` and asserts the refusal.
//
// This is not a softening. The unqualified condition could only ever
// fire on the first case, which is legitimate; it could never fire on
// the second, which is the fault it names. The condition below fires on
// the fault and not on the legitimate case.
if options.delta_order.needs_epochs()
&& grouping.aged() == 0
&& grouping.roots > 0
&& !ids.is_empty()
{
return Err(Error::backend(
"the transfer ordering dated no object, so its epoch key is 0 everywhere and \
the search would silently run under the size ordering instead; the walk \
descended root trees and stamped none of them, which `pathwalk::group` \
cannot do, so this is a fault in the walk and not a property of the \
repository",
));
}
let source = self.odb_handle();
let mut file = std::fs::File::create(path)?;
let mut writer = std::io::BufWriter::new(&mut file);
let outcome = gunnar_pack::write_pack_deltified_ordered(
ids,
&|id| grouping.get(id),
interrupt,
&source,
&mut writer,
&gunnar_pack::WriteOptions::new(hash_kind),
&gunnar_pack::DeltaOptions {
// P-020: batch tier. A repack is offline and one at a time,
// which is what a machine's full core count is for; the serve
// path must never be here (`P-004`).
//
// P-020: batch tier
thread_limit: options.thread_limit,
// Named, not inherited. These were a `..Default::default()`
// until 2026-08-04 and so were a hardcoded 10/50 that no caller
// could reach; the ordering and the window are one question and
// an operator who can change one must be able to change the
// other.
window: options.delta_window,
depth: options.delta_depth,
..Default::default()
},
)
.map_err(|err| {
// The encoder reports a cancellation as whatever its inner stage
// failed with, so the flag is what decides — the same rule the
// indexer is read under, two functions down.
if interrupt.load(Ordering::Relaxed) {
Error::Interrupted
} else {
Error::Backend(anyhow::Error::new(err))
}
})?;
writer.flush()?;
drop(writer);
file.sync_all()?;
Ok((
outcome.pack_id,
Generated {
objects: outcome.num_objects,
// Nothing is copied on this path, by definition: that is what
// makes it the re-encoding one. Reporting a copy here would
// make `P-001`'s measure lie in the one place it is not meant
// to hold.
copied_from_pack: 0,
recompressed: outcome.objects_recompressed,
deltified: outcome.objects_deltified,
},
))
}
/// The same function, in a build without the encoder.
///
/// It refuses instead of falling back to a copy. An operator who asked for
/// `--aggressive` and got a silent consolidation has a forge they believe
/// has been re-encoded and has not, which is the exact failure mode this
/// whole change exists to remove — one level up, in a feature flag rather
/// than in a branch.
#[cfg(not(feature = "repack-delta"))]
fn write_recoded_pack_file(
&self,
_path: &Path,
_ids: &[ObjectId],
_hash_kind: gix_hash::Kind,
_options: &RepackOptions,
_interrupt: &AtomicBool,
) -> Result<(ObjectId, Generated)> {
Err(Error::backend(
"this build has no delta encoder, so the repack cannot recompute deltas; rebuild \
`gunnar-store` with the `repack-delta` feature, or set `DeltaPolicy::Preserve` to \
consolidate without re-encoding",
))
}
/// Fold this repository's packs and loose objects into one new pack.
///
/// See the [module docs](self) for the swap protocol and for why the cold
/// tier is a packfile rather than a znippy archive.
///
/// Returns a [`RepackOutcome`] whose [`rewrote`](RepackOutcome::rewrote) is
/// `false` when there was nothing to do.
///
/// # Errors
///
/// * [`Error::Interrupted`] if the flag in
/// [`RepackOptions::should_interrupt`] was raised. Nothing was changed.
/// * [`Error::MissingObject`] if the pack that was written does not contain
/// something it was meant to. **Nothing is deleted in that case**; the
/// verification runs before the first unlink.
pub fn repack(&self, options: &RepackOptions) -> Result<RepackOutcome> {
// Poisoning only means a previous repack panicked; the data is a `()`.
let _one_writer = self
.repack_lock
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let hash_kind = self.hash_kind();
let objects_dir = self.git_dir().join("objects");
let pack_dir = objects_dir.join("pack");
std::fs::create_dir_all(&pack_dir)?;
let interrupt = options
.should_interrupt
.clone()
.unwrap_or_else(|| Arc::new(AtomicBool::new(false)));
// A previous run may have been killed between creating its scratch file
// and unlinking it. Nothing else can be writing here (the store holds
// the exclusive lock), so its leftovers are ours to remove.
sweep_stale_scratch(&pack_dir);
check_interrupt(&interrupt)?;
// ── 1. snapshot ─────────────────────────────────────────────────────
//
// Everything after this point deletes only what is named here, so a
// pack that arrives mid-repack is not at risk.
let folded_packs = snapshot_packs(&pack_dir)?;
let folded_loose = snapshot_loose(&objects_dir, hash_kind)?;
// Re-scan from disk BEFORE the iteration below, and after the snapshot
// above. That order is the whole safety argument, so it is stated here
// rather than left to be reconstructed.
//
// `Handle::iter()` reaches `load_all_indices()`, which is documented to
// load every index the store already **knows about** and explicitly
// passes `RefreshMode::Never`; it never looks at the directory. So the
// set of objects this repack is about to carry across comes from the
// store's snapshot, while the set of packs it is about to **delete**
// comes from a fresh `read_dir` two lines up. If the snapshot is older
// than the directory, a pack gets deleted whose objects were never
// carried, which is data loss, not a slow lookup.
//
// That was not hypothetical: it is what
// `real_git_clones_a_repacked_repository` caught on 2026-08-03, the
// moment `P-012`'s pool began initialising the store's index eagerly at
// open. Before that it was latent, surviving only because nothing had
// touched the store between opening it and repacking it.
//
// Refreshing here and not earlier keeps the invariant one-directional:
// the object set is at least as new as the deletion set, so a pack that
// lands mid-repack is at worst copied twice and never unlinked
// unread.
self.objects().refresh();
let mut handle = self.odb_handle();
// Pack-copy hands out `entry_by_location` results that point *into* a
// mapped pack. `gix-odb` refuses that outright unless the handle has
// been told not to unload packs underneath it: a panic, not an error,
// because the alternative is reading freed memory.
handle.prevent_pack_unload();
let handle = handle;
let present: Vec<ObjectId> = {
use gix_odb::store::iter::Ordering;
let mut ids = Vec::new();
for id in handle
.iter()
.map_err(|err| Error::Backend(anyhow::Error::new(err)))?
.with_ordering(Ordering::PackAscendingOffsetThenLooseLexicographical)
{
ids.push(id.map_err(|err| Error::Backend(anyhow::Error::new(err)))?);
}
ids
};
let (keep, pruned): (Vec<ObjectId>, usize) = if options.prune_unreachable {
let reachable = connectivity::reachable(&self.objects(), &self.refs())?;
// The grace period. Reachability alone is not a safe answer for an
// object that arrived a moment ago: a push writes every object into
// the ODB and only then moves its reference, and gunnar has no
// quarantine to hide that gap in. An unreachable object whose
// holding file is younger than the grace is carried across.
//
// MEASURED 2026-08-05: without this, a repack whose walk overlaps a
// push destroys that push, at every gate width from 200 µs to
// 200 ms. See `DEFAULT_PRUNE_GRACE`.
let young = young_objects(&folded_packs, &folded_loose, options.prune_grace, hash_kind);
let spared = present
.iter()
.filter(|id| !reachable.contains(*id) && young.contains(*id))
.count();
// Keep the store's own iteration order. `iter_from_counts` sorts by
// pack and offset anyway, and a stable order makes a no-op repack of
// an unchanged repository reproducible.
let kept: Vec<ObjectId> = present
.iter()
.copied()
.filter(|id| reachable.contains(id) || young.contains(id))
.collect();
let pruned = present.len() - kept.len();
// Said whenever the grace did anything, because "gc --prune
// reclaimed less than I expected" and "gc --prune is not working"
// are different situations and only this line tells them apart.
if spared > 0 {
tracing::info!(
target: gunnar_core_target::STORE,
event = "store.repack.prune_grace",
spared,
pruned,
grace_secs = options.prune_grace.as_secs(),
"objects were unreachable but too recently written to prune"
);
}
(kept, pruned)
} else {
(present.clone(), 0)
};
// ── 2. decide what this run has to produce ──────────────────────────
//
// One pack, no loose objects and nothing to prune means **consolidation**
// has nothing left to do. It does not follow that the repack has nothing
// to do, and assuming it did is the bug this shape exists to make
// impossible: the deltas inside that one pack may be the worst possible
// ones, and this is exactly the state in which nobody would ever find
// out. See the module docs.
let already_consolidated =
folded_packs.len() <= 1 && folded_loose.is_empty() && pruned == 0;
// A repository with no objects at all, and (deliberately) a prune that
// would empty the store. The latter is almost always a repository whose
// refs have not landed yet, and turning it into an unlink is not a
// reclaim, it is data loss.
if keep.is_empty() && !present.is_empty() {
tracing::warn!(
objects = present.len(),
"repack: every object is unreachable, so pruning was skipped rather than \
emptying the repository"
);
}
// What is on disk right now, read off the pack entry headers rather than
// inferred from who pushed. This is the observation `DeltaPolicy::Auto`
// decides on, and it is reported whether or not it decides anything.
let before = delta_fraction_of(&folded_packs, hash_kind, folded_loose.len());
let recompute = match options.delta_policy {
DeltaPolicy::Preserve => false,
DeltaPolicy::Recompute => true,
// `None` — nothing measurable — declines. An empty repository does
// not need a window search over its zero objects.
DeltaPolicy::Auto { min_delta_fraction } => {
before.is_some_and(|f| f < min_delta_fraction)
}
};
// WHICH POLICY FIRED, said once, before anything acts on it.
//
// Until this line the only evidence a policy had declined was the
// *absence* of the "computing deltas" line further down, and an absence
// is what a dropped log target, a wrong `RUST_LOG` and a genuine
// decline all look like. It cost a session: `gc --aggressive` producing
// no smaller pack has three completely different causes — the policy
// never ran, the search ran and lost to the copy, the search ran and
// won by nothing — and only the second and third had an event.
tracing::info!(
target: gunnar_core_target::STORE,
event = "store.repack.delta_policy",
policy = ?options.delta_policy,
order = ?options.delta_order,
delta_fraction = before.unwrap_or(-1.0),
recompute,
"repack: the delta policy decided whether to re-encode"
);
let rewrite = !keep.is_empty() && (!already_consolidated || recompute);
check_interrupt(&interrupt)?;
// Set by the producer below when the search ran and its answer was
// thrown away. Declared here because it is an outcome of step 3 that
// step 5 reports, and threading it out of the `if` would mean making
// `Landed` carry a fact that has nothing to do with the pack.
let mut declined = false;
// ── 3. produce the pack this repack leaves behind ───────────────────
//
// Two producers, one result. Below this line nothing knows or cares
// which one ran, which is the property that stops the next step being
// added to only one of them.
let landed = if rewrite {
// ── a re-encode must beat the copy, or it does not get installed ──
//
// Both producers write a scratch pack and the smaller one wins. That
// is one extra pack generation whenever a re-encode is asked for,
// and it buys a property nothing else does: **`gc` cannot leave a
// repository bigger than a plain `gc` would have**, however badly
// the search does on a particular corpus.
//
// It is not insurance against an imagined failure. MEASURED
// 2026-08-04, `gunnar gc` over the bake-off `large` corpus, 12 packs
// in and 1 out, only the delta policy differing:
//
// copied 9 705 923 B
// recomputed, v2 10 341 181 B 1.065x WORSE
// recomputed, size 30 660 804 B 3.159x worse
//
// The corpus was packed by stock git, and re-deriving deltas that
// git already derived well is not free — it is a loss. On the
// `small` corpus the same comparison runs the other way and
// path-walk beats the copy by 1.326x. Which way it goes is a
// property of the repository, so the only honest rule is to write
// both and look.
//
// The comparison is on the **scratch files**, before either is
// indexed, so the loser costs a `write` and a `unlink` and nothing
// else. The copy pass is the cheap half: 6.0 s against 20.3 s for
// the search over the same two corpora.
let scratch = Scratch::new(&pack_dir)?;
let (digest, generated) = if recompute {
let (recoded_digest, recoded) = self.write_recoded_pack_file(
scratch.path(),
&keep,
hash_kind,
options,
&interrupt,
)?;
check_interrupt(&interrupt)?;
let alternative = Scratch::new(&pack_dir)?;
let (copy_digest, copied) = write_pack_file(
alternative.path(),
handle.clone(),
&keep,
hash_kind,
options,
&interrupt,
)?;
let recoded_bytes = file_len(scratch.path());
let copy_bytes = file_len(alternative.path());
if recoded_bytes <= copy_bytes {
drop(alternative);
(recoded_digest, recoded)
} else {
tracing::info!(
target: gunnar_core_target::STORE,
event = "store.repack.reencode_lost_to_copy",
recoded_bytes,
copy_bytes,
"repack: the recomputed deltas are larger than the ones already in \
the packs, so the existing deltas were copied instead"
);
declined = true;
// The loser's bytes are replaced by the winner's, under the
// same path, so everything downstream reads one scratch.
std::fs::rename(alternative.path(), scratch.path())?;
drop(alternative);
(copy_digest, copied)
}
} else {
write_pack_file(
scratch.path(),
handle.clone(),
&keep,
hash_kind,
options,
&interrupt,
)?
};
check_interrupt(&interrupt)?;
let indexed = {
let file = std::fs::File::open(scratch.path())?;
let mut reader = std::io::BufReader::new(file);
gunnar_pack::index_pack(
&mut reader,
Some(&pack_dir),
&gunnar_pack::IndexOptions::new(hash_kind),
Some(&interrupt),
)
.map_err(|err| {
// The indexer reports a cancellation as whatever its inner
// stage happened to fail with, so the flag is what decides.
if interrupt.load(Ordering::Relaxed) {
Error::Interrupted
} else {
Error::Backend(anyhow::Error::new(err))
}
})?
};
drop(scratch);
if indexed.pack_id != digest {
return Err(Error::backend(format!(
"the pack that landed ({}) is not the pack that was generated ({digest})",
indexed.pack_id
)));
}
// Verify, before anything is removed.
let index = indexed.index_path.clone().ok_or_else(|| {
Error::backend("the repacked index was not written; nothing was removed")
})?;
let data = indexed.data_path.clone().ok_or_else(|| {
Error::backend("the repacked pack was not written; nothing was removed")
})?;
verify_pack_holds(&index, hash_kind, &keep)?;
// ── the rewrite has to have been worth it ───────────────────────
//
// Only on the branch where the rewrite existed **solely** to
// re-encode: one pack in, one pack out, nothing consolidated,
// nothing pruned. There, and only there, "keep what was already
// here" is an available answer, and it is the right one whenever the
// re-encode did not actually make the repository smaller.
//
// It is not hypothetical. A three-object repository measures 0%
// deltified — because three objects have nothing to delta against —
// so `Auto` fires, the search finds nothing, and the pack comes out
// 1388 → 1579 bytes: bigger, with a churned pack id that
// invalidates every reader's mapping, for a repack that reclaimed
// nothing. MEASURED 2026-08-04 by `a_resealed_archive_is_visible_to
// _the_next_reader` failing on exactly that.
//
// A check on the *outcome* rather than a floor on the input, and
// that is the point: any rule of the form "do not bother below N
// objects" is a guess that is wrong on both sides of N. This one
// cannot be, because it compares the two packs that exist.
let old_bytes: u64 = folded_packs
.iter()
.filter(|pack| pack.data != data && pack.index != index)
.map(|pack| pack.bytes())
.sum();
let new_bytes: u64 = [index.as_path(), data.as_path()]
.into_iter()
.filter_map(|p| std::fs::metadata(p).ok())
.map(|m| m.len())
.sum();
//
// `declined` on its own is enough here: it means the search ran and
// the copy won, and a copy of a single pack is that pack. Installing
// it would churn the pack id — invalidating every reader's mapping —
// to write out what is already on disk.
//
// `reproduced_existing` is the case where it did not even get that
// far: the generator produced a pack byte-identical to the one
// already there, so `write_to_directory` handed back the existing
// file rather than writing over itself. Nothing is on disk to
// remove, and nothing changed, so this must not report a rewrite and
// must not unlink the paths — which are the repository's own pack.
let reproduced_existing = folded_packs
.iter()
.any(|pack| pack.data == data || pack.index == index);
if already_consolidated
&& (declined || (old_bytes > 0 && new_bytes >= old_bytes))
&& !reproduced_existing
{
// Reached when nothing needed consolidating, so "keep what is
// already there" is available and is strictly better than
// installing an equal-sized pack under a new id.
tracing::info!(
target: gunnar_core_target::STORE,
event = "store.repack.reencode_declined",
old_bytes,
new_bytes,
objects = generated.objects,
"repack: the re-encoded pack is not smaller than the one it would replace, so the existing pack is kept and nothing was changed"
);
if let Some(keep_file) = &indexed.keep_path {
drop_keep(keep_file);
}
// The new pack goes first, so what is left is the pack the
// repository already had — which is what `single_existing_pack`
// is then asked for, on the same terms as the branch below.
remove_if_present(&index)?;
remove_if_present(&data)?;
sync_dir(&pack_dir)?;
self.objects().refresh();
declined = true;
single_existing_pack(&pack_dir).map(|(index, data)| Landed {
index,
data,
written: None,
})
} else if reproduced_existing {
if let Some(keep_file) = &indexed.keep_path {
drop_keep(keep_file);
}
Some(Landed {
index,
data,
written: None,
})
} else {
Some(Landed {
index,
data,
written: Some(Written {
pack_id: indexed.pack_id,
keep_path: indexed.keep_path.clone(),
generated,
}),
})
}
} else {
single_existing_pack(&pack_dir).map(|(index, data)| Landed {
index,
data,
written: None,
})
};
// Nothing on disk to work with: an empty repository, or a prune that
// would have emptied one, or a pack directory that disagrees with the
// snapshot. Nothing to bitmap, nothing to fold, nothing to retire.
let Some(landed) = landed else {
return Ok(RepackOutcome {
delta_fraction_before: before,
delta_fraction_after: before,
reencode_declined: declined,
..Default::default()
});
};
// ── 4. the tail, which runs for a pack this run wrote and for one it
// did not ───────────────────────────────────────────────────────
//
// Everything from here down used to exist twice, or once and in the
// wrong branch. There is one copy now, and a step added to it cannot be
// added to only half the repositories, because there is no other half.
// 4a. the reachability bitmap. Before the `sync_dir` below, or a power
// cut leaves a durable pack beside a bitmap that is not; after the
// verification above, because a bitmap over a pack that does not hold
// what it claims is a faster wrong answer.
//
// P-020: batch tier — the bitmap build is the same one-at-a-time
// offline job the pack write already is.
let bitmap = match options.write_bitmap {
true => self.bitmap_for(&landed.index, hash_kind),
false => None,
};
check_interrupt(&interrupt)?;
// The directory entry for the new pack has to be durable before the old
// ones are unlinked, or a power cut between the two loses both.
sync_dir(&pack_dir)?;
// 4b. fold into the cold tier, before the `.keep` is retired and before
// the first unlink. The pack is durable and every old pack is still
// there, so an archive that fails to seal costs a retry and nothing
// else. `?` on purpose: a fold that failed and was logged instead would
// be a repository the operator believes has a cold tier and does not.
let folded = match &options.cold_tier {
None => None,
Some(tier) => {
let refs = ref_snapshot(self)?;
// Straight off the index header. `FoldRequest::objects` is what
// the archive records about its own contents, so it has to be
// the pack's real count and not the number of ids some walk
// happened to produce.
let objects = gix_pack::index::File::at(&landed.index, hash_kind)
.map(|index| index.num_objects())
.map_err(|err| Error::Backend(anyhow::Error::new(err)))?;
let request = FoldRequest {
git_dir: self.git_dir(),
pack: &landed.data,
index: &landed.index,
hash_kind,
objects,
refs: &refs,
};
let report = tier.fold(&request).map_err(|err| {
Error::backend(format!(
"the pack could not be folded into the cold tier, so nothing was \
deleted: {err}"
))
})?;
Some(report)
}
};
check_interrupt(&interrupt)?;
// The `.keep` guarded the new pack while the old ones were still the
// ones being read. It goes now, before the deletions, so a crash cannot
// leave a pack that later runs would have to special-case.
if let Some(keep_file) = landed.written.as_ref().and_then(|w| w.keep_path.as_ref()) {
drop_keep(keep_file);
}
// Bytes this run actually put on the disk. A pack it did not write is
// not bytes it wrote, however many of them there are — the CLI subtracts
// this from `bytes_folded` to report what a `gc` reclaimed, and counting
// an untouched pack here would report a reclaim of minus one pack.
let bytes_written: u64 = landed
.written
.as_ref()
.map(|_| [landed.index.as_path(), landed.data.as_path()])
.into_iter()
.flatten()
.chain(bitmap.as_deref())
.filter_map(|p| std::fs::metadata(p).ok())
.map(|m| m.len())
.sum();
// 4c. retire what was replaced.
//
// The filter is what makes this uniform: on the branch that wrote no
// pack, the only pack in `folded_packs` **is** the pack that landed, so
// both lists come out empty and this whole step is a no-op by
// construction rather than by an `if`.
let new_ids: BTreeSet<ObjectId> = keep.iter().copied().collect();
let retire_packs: Vec<&PackFiles> = folded_packs
.iter()
.filter(|pack| pack.data != landed.data && pack.index != landed.index)
.collect();
let retire_loose: Vec<&(ObjectId, PathBuf)> = folded_loose
.iter()
// Only loose objects that are now in the pack. An unreachable loose
// object under `prune_unreachable` is deliberately included, since
// it is the thing being reclaimed; one written *after* the snapshot
// is not in `folded_loose` at all and cannot be touched.
.filter(|(id, _)| new_ids.contains(id) || options.prune_unreachable)
.collect();
let mut bytes_folded = 0u64;
let mut packs_removed = 0usize;
let mut loose_removed = 0usize;
if !retire_packs.is_empty() || !retire_loose.is_empty() {
// The multi-pack index goes FIRST, and it is not an optimisation
// that it does. It names the very `.idx` files about to be
// unlinked, and `gix-odb` prefers it over them, so an index left in
// place for even one unlinked pack answers `contains` with `true`
// for objects whose pack is gone. Removing it first means readers
// fall back to the individual indices, every one of which is still
// there, and then to the new pack. See `crate::midx`.
crate::midx::remove_multi_pack_index(&pack_dir)?;
for pack in retire_packs {
// Nothing is unlinked whose objects were not carried across.
//
// The verification above asserts that the new pack holds
// everything this run decided to keep, but that is the *request*
// checked against itself, and it cannot see a `keep` list that
// was short to begin with. This reads the index about to be
// deleted and asks the opposite question: is every object in it
// somewhere else now? A stale object snapshot, a pack that
// landed after the store's last scan, a generator that silently
// dropped entries: all of them look identical from the `keep`
// side and all of them show up here.
//
// Only when nothing is being pruned. With `prune_unreachable`
// the whole point is that some of these objects are meant to go.
if !options.prune_unreachable {
verify_folded_into(&pack.index, hash_kind, &new_ids)?;
}
bytes_folded += pack.bytes();
for path in pack.all() {
remove_if_present(path)?;
}
packs_removed += 1;
}
for (_, path) in retire_loose {
bytes_folded += std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
remove_if_present(path)?;
loose_removed += 1;
}
prune_empty_loose_dirs(&objects_dir);
sync_dir(&pack_dir)?;
// The pack set just changed under every reader. Say so rather than
// waiting for a miss to notice. The watch in `crate::odb` would, but
// a repack knows now and for free.
self.objects().refresh();
// A repack normally leaves exactly one pack, in which case this
// removes nothing and writes nothing. It fires when one did not
// fold: a pack that appeared after the snapshot, and then the
// consolidated tier and the newcomer are back under one index
// instead of two. `write_multi_pack_index_locked`, not the public
// wrapper: the repack lock this function is holding is the same one,
// and it is not reentrant.
if let Err(err) = self.write_multi_pack_index_locked() {
tracing::warn!(
error = %err,
"the repack landed, but the multi-pack index could not be rewritten"
);
}
}
// Read back off the pack that is on disk now, never predicted from the
// encoder's own report. A re-encode that produced a valid pack of the
// right object count with every delta silently dropped would look
// perfect in every other field and show up only here.
let after = delta_fraction_of(
std::slice::from_ref(&PackFiles {
index: landed.index.clone(),
data: landed.data.clone(),
sidecars: Vec::new(),
}),
hash_kind,
0,
);
let written = landed.written;
let generated = written.as_ref().map(|w| &w.generated);
tracing::info!(
target: gunnar_core_target::STORE,
rewrote = written.is_some(),
objects = generated.map_or(0, |g| g.objects),
copied = generated.map_or(0, |g| g.copied_from_pack),
recompressed = generated.map_or(0, |g| g.recompressed),
deltified = generated.map_or(0, |g| g.deltified),
packs_folded = packs_removed,
loose_folded = loose_removed,
pruned,
delta_fraction_before = before,
delta_fraction_after = after,
reencode_declined = declined,
cold_tier = folded.as_ref().map(|f| f.archive.display().to_string()),
cold_tier_bytes = folded.as_ref().map(|f| f.bytes_out),
"repacked"
);
Ok(RepackOutcome {
folded,
rewrote: written.is_some(),
objects: generated.map_or(0, |g| g.objects),
pack_id: written.as_ref().map(|w| w.pack_id),
packs_folded: packs_removed,
loose_folded: loose_removed,
objects_pruned: pruned,
objects_copied_from_pack: generated.map_or(0, |g| g.copied_from_pack),
objects_recompressed: generated.map_or(0, |g| g.recompressed),
objects_deltified: generated.map_or(0, |g| g.deltified),
bytes_folded,
bytes_written,
bitmap,
delta_fraction_before: before,
delta_fraction_after: after,
reencode_declined: declined,
})
}
}
/// Every reference the cold tier must carry, `HEAD` included.
///
/// `HEAD` is fetched separately because it is not under `refs/` and so is not in
/// `iter`, and an archive without it is an archive a clone cannot pick a default
/// branch from — the failure would be a clone that succeeds with a detached
/// head, which looks like success.
fn ref_snapshot(store: &FsStore) -> Result<Vec<(String, crate::refs::Target)>> {
use crate::refs::RefStore as _;
let refs = store.refs();
let mut out: Vec<(String, crate::refs::Target)> = refs
.iter(None)?
.into_iter()
.map(|(name, target)| (name.to_string(), target))
.collect();
if let Some(head) = refs.head()? {
out.push(("HEAD".to_owned(), head));
}
Ok(out)
}
/// What the generator did, as opposed to what it was asked to do.
struct Generated {
objects: u32,
copied_from_pack: usize,
recompressed: usize,
/// Deltas this run **computed**. Always 0 on the copying producer, which is
/// the point: it is what tells the two apart in the outcome.
deltified: usize,
}
/// The pack this repack is leaving behind, however it got there.
///
/// The whole reason the tail can be one code path: a producer that wrote a new
/// pack and one that found the pack already there hand back the same two paths,
/// and everything downstream reads `written` only for the things that are
/// genuinely different — a `.keep` to retire, bytes this run put on the disk,
/// and the generator's own counters.
struct Landed {
index: PathBuf,
data: PathBuf,
/// `Some` when this run wrote it.
written: Option<Written>,
}
/// What a run that actually wrote a pack has to say about it.
struct Written {
pack_id: ObjectId,
keep_path: Option<PathBuf>,
generated: Generated,
}
/// The one pack already in `pack_dir`, when there is exactly one.
///
/// `None` for none, and `None` for more than one. A caller that decided no
/// rewrite was needed reasoned about a snapshot of the pack set; if the
/// directory disagrees with it right now, the safe answer is to touch nothing
/// rather than to guess which pack was meant.
fn single_existing_pack(pack_dir: &Path) -> Option<(PathBuf, PathBuf)> {
let mut indices: Vec<PathBuf> = std::fs::read_dir(pack_dir)
.ok()?
.filter_map(|entry| entry.ok())
.map(|entry| entry.path())
.filter(|path| path.extension().is_some_and(|ext| ext == "idx"))
.collect();
if indices.len() != 1 {
return None;
}
let index = indices.pop().expect("length checked");
let data = index.with_extension("pack");
data.is_file().then_some((index, data))
}
/// A file's length, or `0` if it cannot be read.
///
/// `0` is the right answer for the one comparison this feeds: a scratch pack
/// that is not there did not win.
fn file_len(path: &Path) -> u64 {
std::fs::metadata(path).map_or(0, |m| m.len())
}
/// What fraction of the objects on disk are stored as deltas.
///
/// **Pack entry headers only**: one mapped read of a few bytes per object, no
/// inflation, no delta application, no object reconstructed. That is what makes
/// it affordable to ask before every repack, and affordable is what makes
/// [`DeltaPolicy::Auto`] an observation rather than a guess about who pushed.
///
/// Loose objects count in the denominator and never in the numerator, because a
/// loose object is not a delta. A repository that is all loose objects therefore
/// measures `0.0` and is re-encoded, which is right: it has no deltas at all.
///
/// `None` only when there is nothing whatever to measure. A pack whose index or
/// data will not open is skipped rather than fatal — this is a heuristic input,
/// and refusing to repack a repository because one pack is unreadable would be
/// refusing to repair the one thing that could fix it.
fn delta_fraction_of(packs: &[PackFiles], hash_kind: gix_hash::Kind, loose: usize) -> Option<f32> {
let mut deltas = 0usize;
let mut total = loose;
for pack in packs {
let (Ok(index), Ok(data)) = (
gix_pack::index::File::at(&pack.index, hash_kind),
gix_pack::data::File::at(&pack.data, hash_kind),
) else {
continue;
};
for entry in index.iter() {
total += 1;
if let Ok(entry) = data.entry(entry.pack_offset) {
if matches!(
entry.header,
gix_pack::data::entry::Header::OfsDelta { .. }
| gix_pack::data::entry::Header::RefDelta { .. }
) {
deltas += 1;
}
}
}
}
(total > 0).then(|| deltas as f32 / total as f32)
}
/// Generate the consolidated pack into `path` and report what it cost.
fn write_pack_file(
path: &Path,
db: gix_odb::store::Handle<Arc<gix_odb::Store>>,
ids: &[ObjectId],
hash_kind: gix_hash::Kind,
options: &RepackOptions,
interrupt: &AtomicBool,
) -> Result<(ObjectId, Generated)> {
use gix_features::parallel::reduce::Finalize as _;
use gix_features::parallel::InOrderIter;
use gix_pack::data::output;
let count_ids: Vec<ObjectId> = ids.to_vec();
let (counts, _) = output::count::objects(
db.clone(),
Box::new(count_ids.into_iter().map(Ok)),
&gix_features::progress::Discard,
interrupt,
output::count::objects::Options {
// ONE THREAD, AND IT IS NOT A PERFORMANCE DECISION. MEASURED on
// oden 2026-08-04, three `gunnar gc` runs over three byte-identical
// copies of the same 19-repository store, default flags, the policy
// declining on all 19 so that every run was a pure copy:
//
// nornir 17 063 575 / 17 550 935 / 18 106 353 spread 6.1%
// knut 3 461 019 / 3 628 435 / 3 647 139 spread 5.4%
// modgunn 1 582 224 / 1 516 439 / 1 558 746 spread 4.3%
//
// Pinned to a single CPU the same runs are BYTE-IDENTICAL, and
// nornir comes out 15 854 870 — 7.6% smaller than the best of the
// three parallel runs and 14.2% smaller than the worst.
//
// The cause is duplicate objects. A pushed store holds one pack per
// push and `index-pack --fix-thin` appends each thin pack's bases,
// so the same object is in several packs; `count::objects` records
// whichever pack the lookup that ran first happened to find, and
// `iter_from_counts` then sorts by `(pack_id, pack_offset)` and can
// copy a delta only when its base precedes it. Which end of each
// delta pair survives as a delta is therefore decided by thread
// scheduling — and gunnar's serve path copies pack entries verbatim
// (`P-001`), so that decision is what goes on the wire, for every
// clone and every fetch, until the next `gc`.
//
// A forge may not roll dice for its own wire format. The count is a
// lookup per object over an mmap'd index and is not where a repack
// spends its time: `iter_from_counts` below keeps the operator's
// full thread limit, which is where the deflate and the delta work
// actually happen.
//
// `two_copy_repacks_of_one_store_produce_the_same_pack` is the
// guard, and it goes red on `thread_limit: options.thread_limit`.
thread_limit: Some(1),
chunk_size: 1000,
// The reachable set is computed by `connectivity`, not here: this
// step is a lookup of where each object already lives, so the
// generator can copy its bytes instead of decoding them.
input_object_expansion: output::count::objects::ObjectExpansion::AsIs,
},
)
.map_err(|err| match err {
// Cancelling is not a fault, and must not be reported as one. The
// caller decides whether to retry, and a `Backend` error would send an
// operator hunting for a broken repository instead.
output::count::objects::Error::Interrupted => Error::Interrupted,
other => Error::Backend(anyhow::Error::new(other)),
})?;
let num_entries =
u32::try_from(counts.len()).map_err(|_| Error::backend("more than 4G objects"))?;
// P-020: batch tier. `iter_from_counts` fans out through
// `gix_features::parallel::in_parallel_if` whenever `counts.len() > 4_000`,
// which nearly every real repository exceeds. On the *serve* path that is
// per-request intra-request fan-out and `P-004` settled against it, so serve
// must pass `thread_limit: Some(1)`. A repack is not a request: it is the
// batch job `P-011` names, one at a time, off the hot path, so the operator's
// `RepackOptions::thread_limit` (`None` = all cores) is the right answer
// here. Declared rather than assumed, because
// `xtask/src/pack_pipeline.rs` cannot tell the two tiers apart by reading
// text and neither can the next reader.
//
// P-020: batch tier
let entries = output::entry::iter_from_counts(
counts,
db,
Box::new(gix_features::progress::Discard),
output::entry::iter_from_counts::Options {
thread_limit: options.thread_limit,
mode: output::entry::iter_from_counts::Mode::PackCopyAndBaseObjects,
// A pack at rest must be self-contained. A thin pack is only ever
// valid in transit, and one written here would be a repository that
// no other git can read.
allow_thin_pack: false,
chunk_size: 1000,
version: gix_pack::data::Version::V2,
..Default::default()
},
);
let mut file = std::fs::File::create(path)?;
let mut writer = std::io::BufWriter::new(&mut file);
let (digest, stats) = {
// P-020: sha1dc trailer accepted. `FromEntriesIter` holds its sink as a
// private `gix_hash::io::Write`, and `gix_hash::io::Write::new`
// hardcodes `gix_hash::hasher`: `sha1_checked`, collision detection on.
// There is no way to inject `gunnar_pack`'s `TrailerHasher` here, so a
// repacked SHA-1 pack's trailer pays the detection `P-008` removed from
// the serve path. Accepted deliberately: a repack is one pass over the
// repository, off the request path, and the alternative is a second byte
// emitter. **This is exactly the seam `P-001` must not adopt on serve**
// There the same call would give back P-008's 4.02×, and
// `crates/gunnar-pack/tests/trailer_hash.rs` is what would notice.
//
// P-020: sha1dc trailer accepted
let mut pack = output::bytes::FromEntriesIter::new(
InOrderIter::from(entries),
&mut writer,
num_entries,
gix_pack::data::Version::V2,
hash_kind,
);
for step in pack.by_ref() {
step.map_err(|err| Error::backend(format!("writing the repacked pack: {err}")))?;
}
let digest = pack
.digest()
.ok_or_else(|| Error::backend("the repacked pack has no checksum"))?;
// `input` is a public field, so the reduce side can be finalised for its
// statistics. Those statistics are the point: they are the only thing
// that distinguishes a pack-copy from a re-deflate.
let stats = pack
.input
.inner
.finalize()
.map_err(|err| Error::backend(format!("finishing the repacked pack: {err}")))?;
(digest, stats)
};
use std::io::Write as _;
writer.flush()?;
drop(writer);
file.sync_all()?;
if stats.missing_objects != 0 {
return Err(Error::backend(format!(
"{} objects went missing while repacking; nothing was removed",
stats.missing_objects
)));
}
Ok((
digest,
Generated {
objects: num_entries,
copied_from_pack: stats.objects_copied_from_pack,
recompressed: stats.decoded_and_recompressed_objects,
// This producer copies deltas and computes none, which is exactly
// what `Mode::PackCopyAndBaseObjects` means. Reporting anything else
// here would make the two producers indistinguishable in the
// outcome, which is the whole reason the field exists.
deltified: 0,
},
))
}
/// Every object whose holding file was written inside `grace`, and which a
/// prune must therefore spare even though no reference reaches it.
///
/// The unit is the **file**, not the object, and that is the same unit stock
/// git prunes on: git's `prune` stats the loose file and its `repack
/// --unpack-unreachable=<when>` stats the pack. An object in a pack that landed
/// a second ago is young whether or not it was itself new to that pack, which
/// is conservative in the only direction that matters here — the cost of
/// sparing an object that could have gone is one more `gc` later, and the cost
/// of dropping one that should have stayed is a destroyed push.
///
/// # Failures are young, deliberately
///
/// A file whose mtime cannot be read — it vanished, the filesystem does not
/// carry one, the clock is behind — makes its objects young rather than
/// prunable. Nothing here returns an error: the one thing this function must
/// never do is let an unreadable stat widen what gets deleted.
fn young_objects(
packs: &[PackFiles],
loose: &[(ObjectId, PathBuf)],
grace: std::time::Duration,
hash_kind: gix_hash::Kind,
) -> BTreeSet<ObjectId> {
let mut young = BTreeSet::new();
if grace.is_zero() {
// `--prune=now`. Asked for explicitly, and the only setting under which
// a live push is at risk.
return young;
}
// A grace longer than the current clock (a machine whose clock is at the
// epoch, or a grace of a century) leaves nothing older than the cutoff, so
// everything is young. Saturating rather than panicking on the subtraction.
let Some(cutoff) = std::time::SystemTime::now().checked_sub(grace) else {
for (id, _) in loose {
young.insert(*id);
}
for pack in packs {
insert_pack_objects(&pack.index, hash_kind, &mut young);
}
return young;
};
for (id, path) in loose {
if written_since(path, cutoff) {
young.insert(*id);
}
}
for pack in packs {
if written_since(&pack.data, cutoff) || written_since(&pack.index, cutoff) {
insert_pack_objects(&pack.index, hash_kind, &mut young);
}
}
young
}
/// Whether `path` was last written at or after `cutoff`. **Unreadable is
/// `true`**, for the reason [`young_objects`] gives.
fn written_since(path: &Path, cutoff: std::time::SystemTime) -> bool {
match std::fs::metadata(path).and_then(|meta| meta.modified()) {
Ok(modified) => modified >= cutoff,
Err(_) => true,
}
}
/// Add every object id in the pack index at `index_path` to `out`.
///
/// An index that will not open contributes nothing: the pack it names is
/// already unreadable, and a pack this repack cannot read is one it cannot fold
/// either, so the failure surfaces at `verify_folded_into` with a message about
/// the actual problem rather than here as a silent widening.
fn insert_pack_objects(index_path: &Path, hash_kind: gix_hash::Kind, out: &mut BTreeSet<ObjectId>) {
if let Ok(index) = gix_pack::index::File::at(index_path, hash_kind) {
for entry in index.iter() {
out.insert(entry.oid);
}
}
}
/// Open the index that landed and assert it holds every object it should.
///
/// This is the step that licenses the deletions, so it reads the file that is
/// now on disk rather than trusting the writer's own report. A writer that
/// believed it had written an object is exactly the failure this catches.
fn verify_pack_holds(index_path: &Path, hash_kind: gix_hash::Kind, ids: &[ObjectId]) -> Result<()> {
let index = gix_pack::index::File::at(index_path, hash_kind)
.map_err(|err| Error::Backend(anyhow::Error::new(err)))?;
for id in ids {
if index.lookup(id).is_none() {
return Err(Error::MissingObject {
id: *id,
referrer: None,
});
}
}
Ok(())
}
/// Every object in the index at `index_path` is in `carried`, or the repack
/// refuses to unlink the pack it belongs to.
///
/// The mirror of [`verify_pack_holds`], and the one that catches the failure
/// that verification cannot: `verify_pack_holds` proves the new pack holds
/// everything the run *asked* for, which is worthless if the ask was short.
fn verify_folded_into(
index_path: &Path,
hash_kind: gix_hash::Kind,
carried: &BTreeSet<ObjectId>,
) -> Result<()> {
let index = gix_pack::index::File::at(index_path, hash_kind)
.map_err(|err| Error::Backend(anyhow::Error::new(err)))?;
for entry in index.iter() {
if !carried.contains(&entry.oid) {
return Err(Error::backend(format!(
"{} holds {} and the repacked pack does not, so nothing was deleted",
index_path.display(),
entry.oid
)));
}
}
Ok(())
}
/// Every complete pack in `pack_dir`, keyed by its index.
///
/// Keyed by the **`.idx`** on purpose. `gix-pack` persists a pack's data file
/// before its index, so a `.pack` with no `.idx` is a push still in flight and
/// must not be folded, let alone deleted.
fn snapshot_packs(pack_dir: &Path) -> Result<Vec<PackFiles>> {
let mut out = Vec::new();
let entries = match std::fs::read_dir(pack_dir) {
Ok(entries) => entries,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(out),
Err(err) => return Err(Error::Io(err)),
};
for entry in entries {
let path = entry?.path();
if path.extension().and_then(|e| e.to_str()) != Some("idx") {
continue;
}
let data = path.with_extension("pack");
if !data.is_file() {
continue;
}
let sidecars = ["keep", "rev", "mtimes", "bitmap", "promisor"]
.iter()
.map(|ext| path.with_extension(ext))
.filter(|p| p.is_file())
.collect();
out.push(PackFiles {
index: path,
data,
sidecars,
});
}
out.sort_by(|a, b| a.index.cmp(&b.index));
Ok(out)
}
/// Every loose object present right now, with the file that holds it.
fn snapshot_loose(
objects_dir: &Path,
hash_kind: gix_hash::Kind,
) -> Result<Vec<(ObjectId, PathBuf)>> {
let store = gix_odb::loose::Store::at(
objects_dir,
gix_odb::loose::Options {
object_hash: hash_kind,
..Default::default()
},
);
let mut out = Vec::new();
for id in store.iter() {
let id = id.map_err(|err| Error::Backend(anyhow::Error::new(err)))?;
let hex = id.to_string();
let path = objects_dir.join(&hex[..2]).join(&hex[2..]);
if path.is_file() {
out.push((id, path));
}
}
out.sort();
Ok(out)
}
/// Remove the two-character loose directories that are now empty.
///
/// Best effort by design: a directory that is not empty, or that another writer
/// is creating right now, is left alone. Nothing depends on it being gone.
fn prune_empty_loose_dirs(objects_dir: &Path) {
let Ok(entries) = std::fs::read_dir(objects_dir) else {
return;
};
for entry in entries.flatten() {
let path = entry.path();
let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
continue;
};
if name.len() != 2 || !name.bytes().all(|b| b.is_ascii_hexdigit()) {
continue;
}
let _ = std::fs::remove_dir(&path);
}
}
fn remove_if_present(path: &Path) -> Result<()> {
match std::fs::remove_file(path) {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(Error::Io(err)),
}
}
/// fsync a directory, so a rename into it survives a power cut.
///
/// # One writer, because there are two modules (`S-030`, LAW 5)
///
/// `midx` had its own copy, and the two had **already diverged**: this one
/// reports an unexpected `open` failure and that one dropped it, so the same
/// fault was a hard error when a repack renamed a pack into `objects/pack` and
/// nothing at all when the multi-pack index was renamed into the same
/// directory. Fixed by construction rather than by keeping two copies in step:
/// there is now one function, and `midx` calls it.
pub(crate) fn sync_dir(dir: &Path) -> Result<()> {
match std::fs::File::open(dir) {
Ok(file) => {
// Some filesystems refuse to fsync a directory handle. That is a
// durability weakening, not a correctness one (the swap is still
// ordered), so it is not worth failing a repack over — but it is
// worth SAYING, because a systemic one means every rename in the
// tree is unordered against a power cut and nothing else would ever
// mention it.
if let Err(err) = file.sync_all() {
tracing::debug!(
target: gunnar_core_target::STORE,
dir = %dir.display(),
%err,
"the directory could not be fsynced, so a rename into it is not durable \
against a power cut"
);
}
Ok(())
}
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(Error::Io(err)),
}
}
fn check_interrupt(flag: &AtomicBool) -> Result<()> {
if flag.load(Ordering::Relaxed) {
return Err(Error::Interrupted);
}
Ok(())
}
/// Remove scratch files a killed run left behind.
fn sweep_stale_scratch(pack_dir: &Path) {
let Ok(entries) = std::fs::read_dir(pack_dir) else {
return;
};
for entry in entries.flatten() {
let path = entry.path();
let is_scratch = path
.file_name()
.and_then(|n| n.to_str())
.is_some_and(|n| n.starts_with(TMP_PREFIX));
if is_scratch {
let _ = std::fs::remove_file(&path);
}
}
}
/// Remove a `.keep` marker, and **say so** if it will not go (`S-030`).
///
/// # One writer, because there are two callers
///
/// A `.keep` tells every other reader of the object store "this pack is being
/// written, do not fold it". Two places retire one: this module, after the new
/// cold tier is durable, and `gunnar-cli`'s receive path, once a pushed pack is
/// indexed. Both used to be a bare `let _ = std::fs::remove_file(…)`, and the
/// failure they swallowed is the same failure — so this is one function, not
/// two matching copies (LAW 5).
///
/// # Why it does not return a `Result`
///
/// A `.keep` that survives is a **stale marker**, not lost data and not a lost
/// repack: the pack it names is complete and correct, and the next `gunnar gc`
/// folds it regardless, because [`PackFiles::all`] does not skip a pack that
/// carries one. Failing a finished repack over it would turn a cosmetic
/// leftover into an operator-visible error. What was wrong before is that
/// nothing was *told* — the audit's own re-check found this exact call site had
/// been added, in the code that closed `S-006`, with the swallow copied along
/// with the shape.
pub fn drop_keep(path: &Path) {
match std::fs::remove_file(path) {
Ok(()) => {}
// Already gone is the outcome this function wants.
Err(err) if err.kind() == std::io::ErrorKind::NotFound => {}
Err(err) => tracing::error!(
target: gunnar_core_target::STORE,
event = "store.keep.not_removed",
keep = %path.display(),
%err,
"the .keep marker could not be removed; the pack it names is complete, but \
this marker is now stale"
),
}
}
/// `gunnar-store` does not depend on `gunnar-core` (the dependency runs the
/// other way), so the one target name it needs is stated here rather than
/// imported. It must stay equal to `gunnar_core::observe::TARGET_STORE`, which
/// `crates/gunnar-cli/tests/observability.rs` pins for the whole tree.
pub(crate) mod gunnar_core_target {
pub const STORE: &str = "gunnar::store";
}
/// The temporary pack, removed however this function is left.
///
/// It lives in `objects/pack` rather than in a system temporary directory so
/// that it is on the same filesystem as its destination, and so that a killed
/// run leaves its leftovers where [`sweep_stale_scratch`] looks for them.
struct Scratch(PathBuf);
impl Scratch {
fn new(pack_dir: &Path) -> Result<Self> {
let unique = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
let path = pack_dir.join(format!("{TMP_PREFIX}{}-{unique}.pack", std::process::id()));
Ok(Scratch(path))
}
fn path(&self) -> &Path {
&self.0
}
}
impl Drop for Scratch {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.0);
}
}