crates/gunnar-delta/examples/repack_compare.rs
//! **How well this packs, next to stock `git`.**
//!
//! ```text
//! cargo run --release --example repack_compare -- /path/to/a/git/repo
//! ```
//!
//! With no argument it uses the repository this crate is checked out in.
//!
//! The comparison is deliberately like-for-like and deliberately unkind to us:
//!
//! * **the same object set.** Every object `git cat-file --batch-all-objects`
//! reports, in one pack, no reachability pruning on either side.
//! * **the same parameters.** `--window=10 --depth=50`, git's defaults and ours.
//! * **git gets to keep its two advantages.** It orders objects by type, path
//! and recency (this crate orders by size alone, which is the cheap half of
//! git's heuristic), and `pack-objects` reuses deltas already sitting in the
//! source pack instead of recomputing them. Neither is corrected for.
//!
//! So the ratio printed here is a floor, not a best case, and a number close to
//! `1.00x` means this encoder is doing the job. It is no claim to beat git. The
//! interesting result is that gitoxide, which today can only copy deltas that
//! already exist, can now compute them at all.
//!
//! # What it measured, 2026-08-03
//!
//! Seven corpora on one machine, every pack verified by `git index-pack` before
//! its size was believed. `vs reusing` is against stock `git pack-objects`;
//! `vs recomputing` is against the same command with `--no-reuse-delta`, which
//! forces git to compute every delta instead of lifting the ones already in its
//! pack. The gap between those two columns is the whole story.
//!
//! | repository | objects | raw | vs reusing | vs recomputing |
//! |---|---|---|---|---|
//! | gunnar (all loose, nothing to reuse) | 3517 | 419 MiB | 0.96x | 0.96x |
//! | nornir | 15091 | 417 MiB | **1.44x** | 0.94x |
//! | garmr | 11072 | 113 MiB | 1.28x | 0.97x |
//! | facett | 10629 | 139 MiB | 1.22x | 0.96x |
//! | dwarves | 4968 | 151 MiB | 1.20x | 0.92x |
//! | znippy | 6098 | 86 MiB | 1.06x | 0.97x |
//! | Njord (large binaries) | 10827 | 1751 MiB | 1.00x | 1.00x |
//!
//! **The delta computation is not the weak half.** Against git computing from
//! scratch this crate is 0.92x to 0.97x on every text corpus — consistently 3%
//! to 8% *smaller*, never once larger. The 0.96x first measured on gunnar's own
//! all-loose repository was not luck, and it was not the whole picture either:
//! that repository simply had no packed deltas for git to reuse, so it was
//! already the `--no-reuse-delta` column wearing the other column's name.
//!
//! **Reuse is the weak half, and it is not this crate's half.** Where git can
//! lift deltas out of an existing pack it wins by 6% to 44%, and the worst case
//! is the densest, most-repacked repository in the set. A repack built on this
//! crate alone, throwing away what is already there, would ship a pack half
//! again as large as `git gc` does.
//!
//! That is an argument for composition, not for a better encoder. gitoxide
//! already has the reuse half — `PackCopyAndBaseObjects` is precisely a delta
//! *copier* — and has no encoder at all. The two fit together: lift every delta
//! that already exists, compute one for every object that has none.
//!
//! **Large binaries are a wash.** Njord is 1.13x against raw for everybody and
//! 1.00x between implementations. Already-compressed bytes do not delta, and no
//! ordering heuristic changes that. If this output ever lands in a cold tier,
//! that tier's own compression is what will decide the number, not this crate.
//!
//! //! `git` is invoked as a subprocess. That is fine here and nowhere else in this
//! tree: the whole point of the example is to compare against the real thing,
//! and it is an example, not the library.
use std::io::Write;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::sync::atomic::AtomicBool;
use gix_object::Kind;
use gunnar_delta::{deltify, Object, Options};
#[path = "../tests/support/mod.rs"]
mod support;
use support::{write_pack, PackEntry};
fn git(dir: &Path, args: &[&str]) -> Vec<u8> {
let out = Command::new("git")
.current_dir(dir)
.args(args)
.output()
.unwrap_or_else(|e| panic!("git {args:?}: {e}"));
assert!(
out.status.success(),
"git {args:?} failed: {}",
String::from_utf8_lossy(&out.stderr)
);
out.stdout
}
fn kind_of(name: &str) -> Kind {
match name {
"blob" => Kind::Blob,
"tree" => Kind::Tree,
"commit" => Kind::Commit,
"tag" => Kind::Tag,
other => panic!("unexpected object kind {other}"),
}
}
/// Every object in `dir` whose raw bytes fit inside `budget`, read through one
/// `cat-file --batch` rather than one process per object. On any repository
/// worth measuring that difference is the whole runtime of this example.
///
/// The budget is not a tuning knob, it is what stops this example taking the
/// machine out. Peak resident here is roughly twice the raw total, once for the
/// objects and once because non-deltified entries are copied into the pack
/// writer, plus both packs. A repository with a 966 MiB blob in it is not
/// hypothetical; there is one on this machine.
fn all_objects(dir: &Path, budget: u64) -> (Vec<(gix_hash::ObjectId, Kind, Vec<u8>)>, bool) {
let listing = git(dir, &["cat-file", "--batch-all-objects", "--batch-check"]);
let listing = String::from_utf8(listing).expect("oid listing is ascii");
let mut wanted = Vec::new();
let mut stdin_text = String::new();
let mut taken_bytes = 0u64;
let mut skipped = 0usize;
for line in listing.lines() {
let mut f = line.split_whitespace();
let (oid, kind) = (f.next().expect("oid"), f.next().expect("kind"));
let size: u64 = f.next().expect("size").parse().expect("size is a number");
// Both this process and `git pack-objects` are handed the identical oid
// list, so a prefix of the repository is still a like-for-like
// comparison. What it is not is a whole-repository result, and the
// caller is told how much was left out.
if taken_bytes + size > budget {
skipped += 1;
continue;
}
taken_bytes += size;
wanted.push((
oid.parse::<gix_hash::ObjectId>().expect("oid"),
kind_of(kind),
));
stdin_text.push_str(oid);
stdin_text.push('\n');
}
if skipped > 0 {
println!(
"budget: {skipped} objects skipped to stay under {:.2} GiB of raw bytes",
budget as f64 / (1024.0 * 1024.0 * 1024.0)
);
}
let mut child = Command::new("git")
.current_dir(dir)
.args(["cat-file", "--batch"])
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.spawn()
.expect("spawn cat-file");
let mut stdin = child.stdin.take().expect("piped stdin");
let writer = std::thread::spawn(move || stdin.write_all(stdin_text.as_bytes()));
let out = child.wait_with_output().expect("cat-file ran");
writer.join().expect("writer thread").expect("feed oids");
assert!(out.status.success(), "cat-file --batch failed");
// `<oid> <kind> <size>\n<content>\n`, repeated.
let mut at = 0usize;
let buf = out.stdout;
let mut objects = Vec::with_capacity(wanted.len());
for (oid, kind) in wanted {
let nl = at
+ buf[at..]
.iter()
.position(|&b| b == b'\n')
.expect("header line");
let header = std::str::from_utf8(&buf[at..nl]).expect("ascii header");
let size: usize = header
.rsplit(' ')
.next()
.expect("size")
.parse()
.expect("size");
let start = nl + 1;
objects.push((oid, kind, buf[start..start + size].to_vec()));
at = start + size + 1;
}
(objects, skipped > 0)
}
fn ours(
objects: &[(gix_hash::ObjectId, Kind, Vec<u8>)],
strict: bool,
) -> (usize, usize, std::time::Duration) {
let borrowed: Vec<Object<'_>> = objects
.iter()
.map(|(id, kind, data)| Object {
id: *id,
kind: *kind,
name_hash: 0,
epoch: 0,
data: data.as_slice(),
})
.collect();
let t = std::time::Instant::now();
let out = deltify(
&borrowed,
&Options::default(),
&mut gix_utils::progress::Discard,
&AtomicBool::default(),
)
.expect("deltify");
let elapsed = t.elapsed();
let deltas = out.iter().filter(|d| d.is_delta()).count();
let entries: Vec<PackEntry> = out
.iter()
.map(|d| match (&d.data, d.base) {
(Some(delta), Some(base)) => PackEntry::OfsDelta {
kind: objects[d.source].1,
base,
delta: delta.clone(),
},
(None, None) => PackEntry::Base {
kind: objects[d.source].1,
data: objects[d.source].2.clone(),
},
_ => unreachable!("a Deltified carries a base and bytes together, or neither"),
})
.collect();
let bytes = write_pack(&entries).bytes;
verify(&bytes, objects.len(), strict);
(bytes.len(), deltas, elapsed)
}
/// Hand the pack to stock `git index-pack --strict` and require it to accept.
///
/// `index-pack` reconstructs **every** object in the pack and hashes the result;
/// an object whose delta rebuilt one byte wrong gets a different object id and
/// the command fails. So "smaller than git's" is only interesting once this has
/// run, because the smallest pack in the world is the one that decodes to
/// nothing.
fn verify(pack: &[u8], expected: usize, strict: bool) {
let dir = std::env::temp_dir().join(format!("gunnar-delta-verify-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("scratch dir");
git(&dir, &["init", "--quiet", "--bare"]);
let mut child = Command::new("git")
.current_dir(&dir)
.args(if strict {
&["index-pack", "--strict", "--stdin"][..]
} else {
// A budgeted run holds a *prefix* of the repository, so a tree in it
// can legitimately reference a blob that was left out. `--strict`
// fails that on connectivity, which says nothing about the deltas.
// Dropping it keeps the part that matters: index-pack still
// reconstructs every object in the pack and re-hashes it, so a delta
// that rebuilt one byte wrong still produces a different object id
// and still fails.
&["index-pack", "--stdin"][..]
})
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.expect("spawn index-pack");
let mut stdin = child.stdin.take().expect("piped stdin");
let owned = pack.to_vec();
let writer = std::thread::spawn(move || stdin.write_all(&owned));
let out = child.wait_with_output().expect("index-pack ran");
writer.join().expect("writer thread").expect("feed pack");
assert!(
out.status.success(),
"git index-pack REJECTED the pack this crate produced, so its size means nothing:\n{}",
String::from_utf8_lossy(&out.stderr)
);
let seen = String::from_utf8(git(
&dir,
&["cat-file", "--batch-all-objects", "--batch-check"],
))
.expect("ascii listing")
.lines()
.count();
assert_eq!(
seen, expected,
"the indexed pack holds {seen} objects, not the {expected} that went in"
);
std::fs::remove_dir_all(&dir).ok();
println!(
"verified: git index-pack{} reconstructed and re-hashed all {seen} objects",
if strict { " --strict" } else { "" }
);
}
/// What `git pack-objects` makes of exactly the same object set.
fn gits(src: &Path, objects: &[(gix_hash::ObjectId, Kind, Vec<u8>)], extra: &[&str]) -> usize {
let mut oids = String::new();
for (oid, _, _) in objects {
oids.push_str(&oid.to_string());
oids.push('\n');
}
let mut child = Command::new("git")
.current_dir(src)
.args(
[
"pack-objects",
"--window=10",
"--depth=50",
"--stdout",
"-q",
]
.iter()
.copied()
.chain(extra.iter().copied()),
)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.spawn()
.expect("spawn pack-objects");
let mut stdin = child.stdin.take().expect("piped stdin");
let writer = std::thread::spawn(move || stdin.write_all(oids.as_bytes()));
let out = child.wait_with_output().expect("pack-objects ran");
writer.join().expect("writer thread").expect("feed oids");
assert!(out.status.success(), "git pack-objects failed");
out.stdout.len()
}
fn main() {
let arg = std::env::args().nth(1);
let repo = match &arg {
Some(p) => PathBuf::from(p),
None => PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.ancestors()
.find(|a| a.join(".git").exists())
.expect("no enclosing git repository; pass one as an argument")
.to_path_buf(),
};
println!("repository: {}", repo.display());
// Second argument: raw-byte budget in GiB. The default keeps peak resident
// near 4 GiB, which is what an example is allowed to cost.
let budget = std::env::args().nth(2).map_or(2.0, |g| {
g.parse::<f64>().expect("budget in GiB is a number")
});
let budget = (budget * 1024.0 * 1024.0 * 1024.0) as u64;
let (objects, truncated) = all_objects(&repo, budget);
let raw: usize = objects.iter().map(|(_, _, c)| c.len()).sum();
let mut by_kind = [0usize; 5];
for (_, k, _) in &objects {
by_kind[match k {
Kind::Commit => 1,
Kind::Tree => 2,
Kind::Blob => 3,
Kind::Tag => 4,
}] += 1;
}
println!(
"objects: {} ({} commit, {} tree, {} blob, {} tag), {:.2} MiB raw",
objects.len(),
by_kind[1],
by_kind[2],
by_kind[3],
by_kind[4],
raw as f64 / (1024.0 * 1024.0)
);
let (our_pack, deltas, elapsed) = ours(&objects, !truncated);
let git_pack = gits(&repo, &objects, &[]);
// The row that separates the two things git is better at. `--no-reuse-delta`
// makes git recompute every delta instead of lifting the ones already in its
// pack, so what is left between that column and ours is *ordering*: git
// sorts by type, path and recency, this crate sorts by size alone.
let git_fresh = gits(&repo, &objects, &["--no-reuse-delta"]);
let mib = |n: usize| n as f64 / (1024.0 * 1024.0);
println!(
"\n{:>18} {:>10} {:>9} {:>8}",
"", "pack MiB", "vs raw", "deltas"
);
println!(
"{:>18} {:>10.2} {:>8.2}x {deltas:>8}",
"gunnar-delta",
mib(our_pack),
raw as f64 / our_pack as f64
);
println!(
"{:>18} {:>10.2} {:>8.2}x {:>8}",
"git, reusing",
mib(git_pack),
raw as f64 / git_pack as f64,
"-"
);
println!(
"{:>18} {:>10.2} {:>8.2}x {:>8}",
"git, recomputing",
mib(git_fresh),
raw as f64 / git_fresh as f64,
"-"
);
println!(
"\ngunnar-delta is {:.2}x git reusing its own deltas, {:.2}x git recomputing them; \
computed in {:.2} s",
our_pack as f64 / git_pack as f64,
our_pack as f64 / git_fresh as f64,
elapsed.as_secs_f64()
);
println!(
"note: git reuses deltas already in the source pack and orders by type/path/recency; \
this crate recomputed every delta from scratch and orders by size."
);
}