crates/gunnar-cli/tests/read_object_copies.rs
//! `Refs.ReadObject` inflates the object **once**.
//!
//! # Why this counts bytes allocated and not bytes returned
//!
//! The RPC used to inflate the object into a buffer and then `to_vec()` a slice
//! of that buffer into the response, so a one-gigabyte blob cost two gigabytes.
//! Both spellings return byte-for-byte the same answer, so **asserting on the
//! response proves nothing at all** — which is the trap this entry names by
//! name. The only thing that changed is how much was allocated to produce it,
//! so that is what is measured.
//!
//! The instrument is a counting `#[global_allocator]`, deliberately scoped to
//! this test binary. It counts *bytes requested*, not peak resident set: peak
//! RSS is the same fact one layer down, and it is at the mercy of the
//! allocator's own reuse, which would make the number depend on what the rest
//! of the binary had just freed.
//!
//! # The three claims, and what is still true of the inflation
//!
//! 1. The object is inflated **once**, not twice.
//! 2. A response no gRPC client could receive is refused **before** the inflate
//! (`MAX_RESPONSE_BYTES`, 4 MiB — tonic's own default receive limit, so this
//! refusal costs nobody anything).
//! 3. An object over the server's ceiling is refused **before** the inflate
//! (`DEFAULT_MAX_OBJECT_BYTES`, 64 MiB = a 512 MiB budget over 8 concurrent
//! control-plane workers). This one really does refuse a request that worked
//! before, which is why it is configurable per deployment.
//!
//! What has NOT changed: a read that is allowed still inflates the whole object
//! to produce a prefix, because `ObjectStore` has no bounded read and for a
//! delta-encoded object one cannot exist honestly (a prefix needs the whole base
//! chain). The cost of that is now **bounded** by (3) rather than unbounded; it
//! is not eliminated.
//!
//! Claims (2) and (3) are both measured on the allocator and not on the error,
//! because an implementation that inflates and only then refuses returns the
//! identical error with the identical text.
use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::Arc;
use gunnar_cli::server::{CoreControlPlane, ServerStores};
use gunnar_grpc::ControlPlane;
use gunnar_proto::RepoPath;
// ── the instrument ──────────────────────────────────────────────────────────
static ALLOCATED: AtomicUsize = AtomicUsize::new(0);
static COUNTING: AtomicBool = AtomicBool::new(false);
struct Counting;
// SAFETY-free: every method forwards to `System` unchanged and only adds an
// atomic tally, so the allocator's own contract is `System`'s.
unsafe impl GlobalAlloc for Counting {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
if COUNTING.load(Ordering::Relaxed) {
ALLOCATED.fetch_add(layout.size(), Ordering::Relaxed);
}
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
if COUNTING.load(Ordering::Relaxed) && new_size > layout.size() {
ALLOCATED.fetch_add(new_size - layout.size(), Ordering::Relaxed);
}
unsafe { System.realloc(ptr, layout, new_size) }
}
}
#[global_allocator]
static ALLOCATOR: Counting = Counting;
/// Bytes allocated while `body` runs.
///
/// Only meaningful while the caller holds [`Fixture`]'s exclusion guard — see
/// [`MEASURING`]. `ALLOCATED` and `COUNTING` are process-global, so a second test
/// running concurrently in this binary allocates *into this window*.
fn bytes_allocated<T>(body: impl FnOnce() -> T) -> (T, usize) {
ALLOCATED.store(0, Ordering::Relaxed);
COUNTING.store(true, Ordering::Relaxed);
let out = body();
COUNTING.store(false, Ordering::Relaxed);
(out, ALLOCATED.load(Ordering::Relaxed))
}
/// **Only one test in this binary may run at a time.**
///
/// `#[global_allocator]` is per *process*, and `cargo test` runs the tests in a
/// binary on parallel threads, so the tally one test starts is fed by every
/// other test's allocations and reset by every other test's `ALLOCATED.store(0)`.
/// It was a coin flip, and it lied in **both** directions:
///
/// ```text
/// reading an 3145728-byte object allocated 6484018 bytes, over the 4718592-byte
/// ceiling ... <- another test's inflation landed inside this window
/// reading an 3145728-byte object allocated only 48772 bytes, so this is not
/// measuring the inflation at all <- another test reset the tally mid-window
/// ```
///
/// The second of those is the dangerous shape: had the floor assertion not been
/// there, "allocated almost nothing" would have read as a *very good* result.
///
/// This was latent from the moment the second measuring test was added — the
/// two of them raced and happened not to lose — so every earlier `SEEN RED`
/// for The original measurement was taken with `--test` filters that ran one
/// test at a time. A third test made it reliable rather than causing it.
///
/// The guard is taken by [`fixture`] rather than by each test, so a test
/// cannot measure without holding it: there is no correct-usage rule to
/// remember, and a fourth test added later gets the exclusion by construction
/// (one implementation). The lock is **recovered** on poison rather than
/// propagating, or one genuinely failing test would turn every other into a
/// misleading panic about a poisoned mutex.
static MEASURING: std::sync::Mutex<()> = std::sync::Mutex::new(());
// ── the fixture ─────────────────────────────────────────────────────────────
/// Three megabytes: far enough above every incidental allocation in the call
/// that one extra copy of it cannot hide in the noise, and **under
/// `MAX_RESPONSE_BYTES`**, so this test still measures the copy rather than
/// running into the refusal the test below is about.
///
/// It was 8 MiB before that ceiling existed. The subject — one materialisation
/// against two — does not depend on the size, only on the separation between
/// one copy and two being wider than the noise, and at 3 MiB it is by three
/// orders of magnitude.
const BLOB_BYTES: usize = 3 * 1024 * 1024;
/// A server with one namespace and one repository, and the handle to write into
/// it. Each test builds its own [`CoreControlPlane`] over `stores`, because the
/// object ceiling is per-plane and that is the thing under test.
struct Fixture {
/// Held for the whole test, so no other test in this binary can allocate
/// into a measurement window. See [`MEASURING`]. Dropped last-but-one;
/// order against `_dir` does not matter.
_measuring: std::sync::MutexGuard<'static, ()>,
_dir: tempfile::TempDir,
stores: Arc<ServerStores>,
path: RepoPath,
}
fn fixture() -> Fixture {
let measuring = MEASURING.lock().unwrap_or_else(|p| p.into_inner());
let dir = tempfile::tempdir().expect("temporary directory");
let stores = Arc::new(ServerStores::open(dir.path()).expect("open the stores"));
let plane = CoreControlPlane::new(stores.clone());
let path = RepoPath::new("ns", "repo");
plane
.create_namespace("ns", "")
.expect("create the namespace");
plane
.create_repo(gunnar_grpc::NewRepo {
path: path.clone(),
hash_kind: gunnar_proto::HashKind::Sha1,
default_branch: String::new(),
visibility: gunnar_proto::Visibility::Private,
description: String::new(),
adopt: false,
})
.expect("create the repository");
Fixture {
_measuring: measuring,
_dir: dir,
stores,
path,
}
}
/// Write an **incompressible** blob of `bytes` and return its id and its bytes.
///
/// Incompressible on purpose: the inflated size is the size that matters here,
/// and a compressible blob could slip under an allocation bound while still
/// costing the full amount in memory.
fn write_blob(f: &Fixture, bytes: usize, seed: u64) -> (gix_hash::ObjectId, Vec<u8>) {
let mut content = Vec::with_capacity(bytes);
let mut state = seed;
while content.len() < bytes {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
content.extend_from_slice(&state.to_le_bytes());
}
content.truncate(bytes);
let repo_ref = gunnar_core::RepoRef::new(&f.path.namespace, &f.path.repo)
.expect("a valid repository reference");
let repo = f
.stores
.catalog()
.get(&repo_ref)
.expect("read the catalogue")
.expect("the repository is there");
let store = f.stores.repo_store(&repo).expect("open the repository");
// One blob, as a one-object pack. `ObjectStore::write(kind, bytes)` is gone
// with the trait; `GitOps` takes packs and nothing else, because an engine
// that stores pack entries verbatim has nowhere to put a loose object. The
// writer is `gunnar_pack`'s, which is what the server itself writes with.
let mut pack = Vec::new();
let mut writer = gunnar_pack::PackWriter::new(
&mut pack,
1,
&gunnar_pack::WriteOptions::new(store.hash_kind()),
)
.expect("start a pack");
writer
.add_object(gix_object::Kind::Blob, &content)
.expect("add the blob");
writer.finish().expect("finish the pack");
gunnar_store::git_storage_trait::GitOps::put_pack(store.serve(), &pack)
.expect("store the blob");
let oid = gix_object::compute_hash(store.hash_kind(), gix_object::Kind::Blob, &content)
.expect("hash the blob");
// 🔴 **Bring the store to rest before anything is measured.**
//
// This file counts *allocations* inside one `read_object` call, and the
// znippy arm's `header`/`has`/`extents` all begin with
// `if self.unindexed_packs() > 0 { self.absorb_pending()? }` — a pack that
// has been written but not yet indexed is absorbed by whoever asks the next
// question. With the fixture writing a pack, that "whoever" was
// `read_object` itself, and the absorb of a 3 MiB blob landed inside the
// measurement: 22 MB charged to a call that had refused the object before
// reading it. The defect being guarded (inflate once, refuse from
// the header) was not what the number was reporting.
//
// One `header` here does the absorb outside the measured region, which is
// the state `ObjectStore::write` used to leave the store in: written *and*
// indexed. It is not a workaround for the property under test; it is the
// fixture finishing its own work before handing over.
let indexed = gunnar_store::git_storage_trait::GitServe::header(store.serve(), oid.as_bytes())
.expect("ask the store for the blob's header")
.expect("the blob the fixture just wrote must be in the store");
assert_eq!(
indexed.1,
content.len() as u64,
"the store disagrees with the fixture about how big the blob is"
);
(oid, content)
}
#[test]
fn reading_an_object_inflates_it_once_and_does_not_copy_it_again() {
let f = fixture();
let plane = CoreControlPlane::new(f.stores.clone());
let path = &f.path;
let (oid, content) = write_blob(&f, BLOB_BYTES, 0x2545_f491_4f6c_dd1d);
let (response, allocated) =
bytes_allocated(|| plane.read_object(path, &oid.to_string(), 0).expect("read"));
// Correctness first: a "no copies" fix that returned the wrong bytes, or an
// empty buffer, would satisfy any allocation bound trivially.
assert_eq!(response.size, BLOB_BYTES as u64);
assert_eq!(
response.data.len(),
BLOB_BYTES,
"the response does not carry the object"
);
assert_eq!(response.data, content, "the response carries wrong bytes");
assert!(!response.truncated);
// The claim. One inflation is unavoidable; a second copy is not. The bound
// sits between one and two copies, so it cannot be satisfied by both.
let ceiling = BLOB_BYTES + BLOB_BYTES / 2;
assert!(
allocated < ceiling,
"reading an {BLOB_BYTES}-byte object allocated {allocated} bytes, over \
the {ceiling}-byte ceiling that sits between one copy and two: the \
object is being materialised more than once"
);
// And the floor, so a stubbed read that allocated nothing cannot pass.
assert!(
allocated >= BLOB_BYTES,
"reading an {BLOB_BYTES}-byte object allocated only {allocated} bytes, \
so this is not measuring the inflation at all"
);
}
/// A response no gRPC client could receive is refused **before** the object is
/// inflated.
///
/// # Asserting on the error would prove nothing
///
/// An implementation that inflated the whole object and then noticed it was too
/// big returns the identical `ResourceExhausted`. So the assertion is on what it
/// cost: the call must allocate far less than the object, which is only possible
/// if `ObjectStore::read` was never reached. That is the entry's restated
/// Verify, measured with the allocator rather than with a wall clock.
#[test]
fn an_undeliverable_response_is_refused_without_inflating_the_object() {
let f = fixture();
let plane = CoreControlPlane::new(f.stores.clone());
let path = &f.path;
// Over `MAX_RESPONSE_BYTES` (4 MiB, tonic's own default receive limit) and
// well UNDER the default object ceiling (64 MiB), so what is measured here
// is the response refusal and not the object one.
const HUGE: usize = 12 * 1024 * 1024;
let (oid, content) = write_blob(&f, HUGE, 0x9e37_79b9_7f4a_7c15);
let (refused, allocated) = bytes_allocated(|| plane.read_object(path, &oid.to_string(), 0));
let err = refused.expect_err("a 12 MiB response must be refused");
assert!(
matches!(err, gunnar_grpc::ControlError::ResourceExhausted(_)),
"the refusal must be RESOURCE_EXHAUSTED, so a client knows to retry \
narrower rather than that it asked something malformed; got {err:?}"
);
assert!(
err.to_string().contains(&HUGE.to_string()),
"the refusal must name the object's size, or a caller cannot choose a \
max_bytes that would work; it said: {err}"
);
// THE claim: the object was never materialised. The bound is a small
// fraction of the object, so an inflate-then-refuse cannot pass it.
assert!(
allocated < HUGE / 8,
"refusing a {HUGE}-byte object allocated {allocated} bytes, which is \
the object itself: it was inflated and only then refused, so the \
memory amplifier this refusal exists to close is still open"
);
// The other side, and it is what stops the ceiling from being a blanket
// refusal: asking for a PREFIX of the same object still works. Without
// this, an implementation that refused every large object regardless of
// `max_bytes` would pass everything above.
let prefix = plane
.read_object(path, &oid.to_string(), 1024)
.expect("a bounded read of the same object must still be answered");
assert_eq!(prefix.data.len(), 1024);
assert_eq!(prefix.size, HUGE as u64);
assert!(prefix.truncated);
assert_eq!(
prefix.data,
content[..1024],
"the prefix is the wrong bytes"
);
}
/// **An object over the server's ceiling is refused without being inflated** —
/// The contract half, the one that really does refuse a request that
/// worked before.
///
/// # Why the error alone proves nothing
///
/// An implementation that inflated the object and *then* noticed it was too big
/// returns the identical `ResourceExhausted` with the identical message. So the
/// claim is measured on the allocator, exactly as the response-ceiling test
/// beside it: refusing must cost a small fraction of the object, which is only
/// possible if `ObjectStore::read` was never reached.
///
/// # Why the ceiling is set small here rather than using the default
///
/// The default is 64 MiB, so testing it against the constant would mean a 64 MiB
/// incompressible fixture in a temp dir on every run. Setting it low instead is
/// cheap AND strictly better evidence: it proves
/// `with_max_object_bytes` is actually *wired to the refusal*, which a test
/// against a hard-coded constant could never distinguish from a coincidence.
#[test]
fn an_object_over_the_ceiling_is_refused_without_inflating_it() {
const CEILING: u64 = 1024 * 1024;
let f = fixture();
let (oid, content) = write_blob(&f, BLOB_BYTES, 0x1234_5678_9abc_def1);
let path = &f.path;
let plane = CoreControlPlane::new(f.stores.clone()).with_max_object_bytes(CEILING);
let (refused, allocated) = bytes_allocated(|| plane.read_object(path, &oid.to_string(), 1024));
let err = refused.expect_err("an object over the ceiling must be refused");
assert!(
matches!(err, gunnar_grpc::ControlError::ResourceExhausted(_)),
"the refusal must be RESOURCE_EXHAUSTED, not an argument error: {err:?}"
);
assert!(
err.to_string().contains(&BLOB_BYTES.to_string())
&& err.to_string().contains(&CEILING.to_string()),
"the refusal must name BOTH the object's size and the ceiling, or an \
operator cannot tell whether to raise the limit or the caller to stop \
asking; it said: {err}"
);
// THE claim. `max_bytes = 1024` was passed above, so an implementation that
// inflated first and refused after would still have allocated the whole
// object — which is the amplifier this exists to close.
assert!(
(allocated as u64) < BLOB_BYTES as u64 / 8,
"refusing a {BLOB_BYTES}-byte object allocated {allocated} bytes, which \
is the object itself: it was inflated and only then refused"
);
// Other side #1: the ceiling is what did it. The SAME object, the same
// request, through a plane with the default ceiling, is still answered.
// Without this, a `read_object` that refused everything would pass above.
let default_plane = CoreControlPlane::new(f.stores.clone());
let ok = default_plane
.read_object(path, &oid.to_string(), 1024)
.expect("under the default 64 MiB ceiling this object is still readable");
assert_eq!(ok.data, content[..1024], "and it is the right bytes");
assert_eq!(ok.size, BLOB_BYTES as u64);
assert!(ok.truncated);
// Other side #2: `0` really disables it, or the escape hatch the doc offers
// an operator would be a lie.
let uncapped = CoreControlPlane::new(f.stores.clone()).with_max_object_bytes(0);
let all = uncapped
.read_object(path, &oid.to_string(), 0)
.expect("a ceiling of 0 is no ceiling");
assert_eq!(all.data, content, "the whole object comes back uncapped");
// Other side #3: an object UNDER the low ceiling is served by the very same
// plane that refused the one above — so the refusal tracks the size rather
// than being a property of that plane.
let (small_oid, small) = write_blob(&f, 4096, 0xfedc_ba98_7654_3210);
let served = plane
.read_object(path, &small_oid.to_string(), 0)
.expect("a 4 KiB object is under the 1 MiB ceiling");
assert_eq!(served.data, small, "the small object comes back whole");
}