plugins/native/znippy-plugin-git/tests/rss_retention_probe.rs
//! **Is the 4.8 GB plateau REACHABLE, or UNREACHABLE-BUT-UNFREED?**
//!
//! # Why this exists before any more eviction work
//!
//! gunnar-on-znippy sits at 4792–4872 MB of anonymous RSS on oden's forge suite,
//! **flat** — the same figure whether the clone it served was 3 257 objects or 352 396.
//! Two rounds of work have treated that as a retention bug and bounded a cache; the
//! second round (`gunnar` `ecc2c842`) reported its own fix as failed, correctly: the
//! bound is `DEFAULT_OPEN_REPOS = 256` and the forge seeds ~20 repositories, so it never
//! fires.
//!
//! **An eviction bound is a tool for a growth curve.** Against a flat plateau it can only
//! help if the plateau exceeds the bound. So the question that has to be answered before
//! any more dropping is which of two completely different defects this is:
//!
//! * **(a) REACHABLE** — something deliberately holds those bytes. The fix is to find the
//! owner and decide whether it should hold them.
//! * **(b) UNREACHABLE BUT UNFREED** — the bytes are `free()`d and the allocator never
//! hands the pages back to the OS. **No amount of dropping helps**; the fix is an
//! allocator or arena change.
//!
//! Every measurement taken so far — RSS while the server runs — is identical under both,
//! which is why three sessions could not tell them apart.
//!
//! # The discriminator
//!
//! For each phase, three readings of anonymous RSS out of `/proc/self/smaps_rollup`
//! (a file read; no shelling, LAW):
//!
//! 1. **live** — the structure built and warm.
//! 2. **dropped** — every owner gone. Anything still resident is now, by definition,
//! unreachable from the program.
//! 3. **trimmed** — after `malloc_trim(0)`, which asks glibc to return free arena pages.
//!
//! | live → dropped | dropped → trimmed | verdict |
//! |---|---|---|
//! | falls | — | the drop works; the plateau's owner is upstream of this structure |
//! | flat | **falls** | **(b)** glibc retention — eviction cannot fix it |
//! | flat | flat | **(a)** something still holds it, or the arena is too fragmented to trim |
//!
//! # The two controls are the point, not decoration
//!
//! glibc routes allocations at or above `M_MMAP_THRESHOLD` (128 KiB by default) straight
//! to `mmap`, and `free` on those `munmap`s immediately — those pages always go back.
//! Below it, allocations come from the arena and `free` returns them to a free list, not
//! to the OS. So *the same number of bytes behaves completely differently depending on
//! the block size it was requested in*, and a probe that does not pin that down cannot
//! attribute what it sees.
//!
//! That matters here specifically: redb's page cache holds **4 KiB pages**, far below the
//! threshold. 64 MiB of cache is ~16 000 small blocks, and freeing them is exactly the
//! shape that stays in the arena. The controls measure that directly on this box rather
//! than quoting it from documentation.
//!
//! # Run
//!
//! ```bash
//! cargo test -p znippy-plugin-git --test rss_retention_probe -- --ignored --nocapture
//! ```
//!
//! `#[ignore]` because it is a measurement, it allocates ~1 GB, and libtest would
//! otherwise run it beside other tests in the same process — where their allocations land
//! in the same arena and the readings stop meaning anything.
use std::path::Path;
use znippy_plugin_git::arms::DEFAULT_REDB_CACHE_BYTES;
// `lookup` reaches the stack through the `ObjectIndex` trait it implements, not an
// inherent method — the trait has to be in scope for the cache-warming reads.
use znippy_plugin_git::index_layout::{IndexEntry, ObjType, ObjectIndex, OneTableFourColumns};
use znippy_plugin_git::read_stack::{ObjectReadStack, RebuildTriggers};
/// Anonymous RSS of THIS process, in KiB, from `/proc/self/smaps_rollup`.
///
/// `Anonymous:` and not `Rss:` deliberately. `Rss` includes file-backed pages — mapped
/// binaries, and any page cache a mapping brings in — which the kernel reclaims on demand
/// and which no allocator change would touch. The plateau under investigation was already
/// reported as anonymous, and this keeps the two from being conflated.
fn anon_rss_kb() -> u64 {
let s = std::fs::read_to_string("/proc/self/smaps_rollup")
.expect("/proc/self/smaps_rollup (Linux 4.14+)");
for line in s.lines() {
if let Some(rest) = line.strip_prefix("Anonymous:") {
let kb = rest.trim().trim_end_matches("kB").trim();
return kb.parse().expect("Anonymous: is a kB count");
}
}
panic!("no `Anonymous:` line in smaps_rollup");
}
fn mb(kb: u64) -> f64 {
kb as f64 / 1024.0
}
/// **The allocator's own books**: `(in_use, from_kernel)` in KiB, via `mallinfo2`.
///
/// This is the reading that settles reachable-vs-unfreed **without dropping anything**,
/// and it is why the probe leads with it. `uordblks + hblkhd` is what the program has
/// asked for and not given back — *live, reachable bytes*. `arena + hblkhd` is what
/// glibc has taken from the kernel. The gap between them is free list: bytes the program
/// no longer owns and the OS has not got back.
///
/// Three sessions measured RSS, which is the SUM of those two and therefore cannot
/// distinguish them. The technique is not new here — `znippy-zoomies`'
/// `osm-katana/src/phase_log.rs` reached for exactly this pair on 2026-08-03 for the same
/// question, with the same reasoning written down ("a number near zero says the memory is
/// genuinely live … a large number says the allocator was the leak"). It is read here
/// rather than imported because `osm-katana` sits DOWNSTREAM of the `znippy-zoomies` root
/// that znippy depends on, so importing it would invert the dependency; see the report's
/// note on where the shared primitive belongs.
fn mallinfo_kb() -> Option<(u64, u64)> {
#[cfg(all(target_os = "linux", target_env = "gnu"))]
{
// SAFETY: `mallinfo2` takes no arguments and returns a plain POD struct.
let mi = unsafe { libc::mallinfo2() };
let in_use = (mi.uordblks as u64 + mi.hblkhd as u64) / 1024;
let from_kernel = (mi.arena as u64 + mi.hblkhd as u64) / 1024;
return Some((in_use, from_kernel));
}
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
None
}
fn mallinfo_line(tag: &str) -> String {
match mallinfo_kb() {
Some((used, from_kernel)) => format!(
"{tag}: in-use {:8.1} MB · from-kernel {:8.1} MB · free-list {:8.1} MB",
mb(used),
mb(from_kernel),
mb(from_kernel.saturating_sub(used))
),
None => format!("{tag}: mallinfo2 unavailable (not glibc)"),
}
}
/// Ask glibc to return free arena pages to the OS. Safe to call at any time; it is a
/// hint, and on a non-glibc libc the symbol would not link — which is itself the honest
/// signal that this whole diagnosis is glibc-specific.
fn malloc_trim() {
unsafe { libc::malloc_trim(0) };
}
/// One phase's three readings, printed as the table the report quotes.
struct Phase {
what: String,
base: u64,
live: u64,
dropped: u64,
trimmed: u64,
}
impl Phase {
fn report(&self) -> Self {
let grew = self.live.saturating_sub(self.base);
let after_drop = self.dropped.saturating_sub(self.base);
let after_trim = self.trimmed.saturating_sub(self.base);
eprintln!(
"\n── {} ──\n grew {:8.1} MB\n after drop {:8.1} MB ({:5.1}% still \
resident)\n after trim {:8.1} MB ({:5.1}% still resident)",
self.what,
mb(grew),
mb(after_drop),
100.0 * after_drop as f64 / grew.max(1) as f64,
mb(after_trim),
100.0 * after_trim as f64 / grew.max(1) as f64,
);
Self { what: self.what.clone(), ..*self }
}
fn held_after_drop_pct(&self) -> f64 {
let grew = self.live.saturating_sub(self.base).max(1);
100.0 * self.dropped.saturating_sub(self.base) as f64 / grew as f64
}
fn held_after_trim_pct(&self) -> f64 {
let grew = self.live.saturating_sub(self.base).max(1);
100.0 * self.trimmed.saturating_sub(self.base) as f64 / grew as f64
}
}
/// **CONTROL A — many SMALL blocks**, the shape redb's 4 KiB page cache has.
/// Expectation, to be confirmed rather than assumed: the drop frees nothing back to the
/// OS and `malloc_trim` is what returns it.
fn control_small_blocks(target_mb: usize) -> Phase {
const BLOCK: usize = 4096;
let n = target_mb * 1024 * 1024 / BLOCK;
let base = anon_rss_kb();
let mut v: Vec<Vec<u8>> = Vec::with_capacity(n);
for i in 0..n {
// Written, not just reserved: an untouched page is not resident, and a probe
// that measured reservations would report a number the kernel never backed.
let mut b = vec![0u8; BLOCK];
b[0] = i as u8;
b[BLOCK - 1] = (i >> 8) as u8;
v.push(b);
}
let live = anon_rss_kb();
drop(v);
let dropped = anon_rss_kb();
malloc_trim();
let trimmed = anon_rss_kb();
Phase { what: format!("CONTROL A — {n} x 4 KiB blocks (redb page shape)"), base, live, dropped, trimmed }
}
/// **CONTROL B — few LARGE blocks**, above glibc's mmap threshold. The counterweight: if
/// this one also failed to return its pages, the probe would be measuring something other
/// than the arena and Control A would prove nothing.
fn control_large_blocks(target_mb: usize) -> Phase {
const BLOCK: usize = 1024 * 1024;
let n = target_mb;
let base = anon_rss_kb();
let mut v: Vec<Vec<u8>> = Vec::with_capacity(n);
for i in 0..n {
let mut b = vec![0u8; BLOCK];
for p in (0..BLOCK).step_by(4096) {
b[p] = i as u8;
}
v.push(b);
}
let live = anon_rss_kb();
drop(v);
let dropped = anon_rss_kb();
malloc_trim();
let trimmed = anon_rss_kb();
Phase { what: format!("CONTROL B — {n} x 1 MiB blocks (above mmap threshold)"), base, live, dropped, trimmed }
}
/// The per-database redb page-cache ceiling this run uses. Defaults to the shipped
/// [`DEFAULT_REDB_CACHE_BYTES`]; override with `PROBE_REDB_CACHE_BYTES` **as a plain byte
/// count** to measure what the ceiling is actually worth.
///
/// Deliberately a *separate* variable from the product's own `ZNIPPY_GIT_REDB_CACHE_BYTES`,
/// and deliberately parsed the same strict way: `arms::redb_cache_bytes()` does
/// `raw.parse::<usize>()` and **`bail!`s on anything that is not a bare integer** — so
/// `=8m` is an error, never 8 MiB. Its own doc says why, in writing: *"a silent fallback
/// would let an operator believe a measurement came from a ceiling that was never
/// applied."* A probe that took `8m` here and reported a number would be that operator.
fn probe_cache_bytes() -> usize {
match std::env::var("PROBE_REDB_CACHE_BYTES") {
Ok(v) if !v.trim().is_empty() => v.trim().parse::<usize>().unwrap_or_else(|_| {
panic!(
"PROBE_REDB_CACHE_BYTES={v:?} is not a bare byte count. Suffixes are NOT \
parsed — write 8388608, not 8m. Failing loudly rather than measuring a \
ceiling that was never applied."
)
}),
_ => DEFAULT_REDB_CACHE_BYTES,
}
}
/// Where the probe writes its redb fixtures.
///
/// **`/tmp` on oden is tmpfs — RAM.** A memory probe that puts a gigabyte of databases in
/// `std::env::temp_dir()` there is charging the box for its own fixtures, and the harness
/// on this machine had already leaked 96 GB into `/tmp` as abandoned store dirs before
/// anyone noticed. Set `PROBE_SCRATCH` to a real disk (md1). If the chosen directory is
/// tmpfs this says so out loud rather than quietly measuring itself — the fixtures land in
/// page cache and not in `Anonymous:`, so they do not corrupt THIS reading, but they do
/// consume the memory the probe exists to reason about.
fn scratch_root() -> std::path::PathBuf {
let root = std::env::var("PROBE_SCRATCH")
.ok()
.filter(|s| !s.trim().is_empty())
.map(std::path::PathBuf::from)
.unwrap_or_else(std::env::temp_dir);
if is_tmpfs(&root) {
eprintln!(
" WARNING: {} is tmpfs (RAM). The redb fixtures for this run are being written \
into memory. Set PROBE_SCRATCH to a real disk.",
root.display()
);
}
root
}
/// Is `p` under a tmpfs mount? Read out of `/proc/mounts` — a file read, no shelling.
/// Longest matching mount point wins, so `/tmp` beats `/`.
fn is_tmpfs(p: &Path) -> bool {
let Ok(mounts) = std::fs::read_to_string("/proc/mounts") else { return false };
let mut best: Option<(usize, bool)> = None;
for line in mounts.lines() {
let mut f = line.split_whitespace();
let (_dev, point, fstype) = (f.next(), f.next(), f.next());
let (Some(point), Some(fstype)) = (point, fstype) else { continue };
if p.starts_with(point) {
let len = point.len();
if best.is_none_or(|(b, _)| len > b) {
best = Some((len, fstype == "tmpfs"));
}
}
}
best.is_some_and(|(_, t)| t)
}
fn synthetic(n: usize, seed: u64) -> Vec<IndexEntry> {
let mut s = seed | 1;
let mut out = Vec::with_capacity(n);
let mut offset = 12u64;
for _ in 0..n {
// xorshift — deterministic, no rand dep, and the oids must be well spread or
// the redb btree degenerates and the page count stops resembling a real store.
s ^= s << 13;
s ^= s >> 7;
s ^= s << 17;
let mut oid = vec![0u8; 20];
oid[..8].copy_from_slice(&s.to_le_bytes());
oid[8..16].copy_from_slice(&s.rotate_left(21).to_le_bytes());
let len = 64 + (s % 4096);
out.push(IndexEntry {
oid,
offset,
len,
obj_type: ObjType::Blob,
uncompressed_size: len * 3,
delta_base: 0,
});
offset += len;
}
out
}
/// **THE REAL PATH — `ObjectReadStack` over redb, at the forge's shape.**
///
/// `repos` stacks, each carrying `rows` entries, each opened with the SAME
/// `DEFAULT_REDB_CACHE_BYTES` the server uses. The cache is deliberately left at its
/// default: shrinking it is a different experiment (one already queued elsewhere), and it
/// answers "is the cache the owner", not "is the plateau reachable" — which is this
/// question and has to be settled first.
fn real_stacks(dir: &Path, repos: usize, rows: usize) -> Phase {
let base = anon_rss_kb();
let mut stacks = Vec::with_capacity(repos);
for r in 0..repos {
let path = dir.join(format!("repo{r}.objects.tail.redb"));
let stack = ObjectReadStack::<OneTableFourColumns>::open(
&path,
RebuildTriggers::manual(),
probe_cache_bytes(),
)
.expect("open a read stack");
let entries = synthetic(rows, 0x9E3779B97F4A7C15 ^ (r as u64));
stack.append(&entries).expect("append");
stack.rebuild().expect("rebuild");
// WARM THE CACHE. Without reads the redb page cache holds almost nothing and the
// probe would measure an empty cache and conclude the cache is small — the exact
// way a memory probe lies.
for e in entries.iter().step_by(7) {
let _ = stack.lookup(&e.oid);
}
stacks.push(stack);
}
let live = anon_rss_kb();
let held_mb = mb(live.saturating_sub(base));
eprintln!(
" [live] {repos} stacks x {rows} rows = {:.1} MB anonymous, {:.1} MB per stack \
(cache cap {} MiB/db)",
held_mb,
held_mb / repos as f64,
probe_cache_bytes() / (1024 * 1024),
);
// THE READING THAT ANSWERS (a) WITHOUT DROPPING ANYTHING. If in-use is close to the
// resident growth here, these bytes are genuinely live and reachable while the
// handles are open — which is the forge's actual state, because its LRU bound (256)
// never fires over ~20 repositories and nothing is ever released.
eprintln!(" {}", mallinfo_line("[live] "));
drop(stacks);
let dropped = anon_rss_kb();
eprintln!(" {}", mallinfo_line("[dropped]"));
malloc_trim();
let trimmed = anon_rss_kb();
eprintln!(" {}", mallinfo_line("[trimmed]"));
Phase {
what: format!("REAL — {repos} x ObjectReadStack over redb ({rows} rows each)"),
base,
live,
dropped,
trimmed,
}
}
#[test]
#[ignore = "measurement: allocates ~1 GB and must not share a process with other tests"]
fn reachable_or_merely_unfreed() {
eprintln!(
"\n=== RSS RETENTION PROBE ===\nallocator: {} (no #[global_allocator] in \
gunnar-server or znippy ⇒ system malloc)\n",
if cfg!(target_env = "gnu") { "glibc malloc" } else { "NOT glibc — read the verdict with care" }
);
// ORDER MATTERS, AND GETTING IT WRONG COST A RUN. On the first pass the small-block
// control ran FIRST, and both controls then held ~99% after drop — the guard below
// fired and refused to attribute anything, correctly. The cause is not exotic: the
// small control frees ~700 MB into the arena, `malloc_trim` returns those pages to
// the kernel but the free CHUNKS remain in glibc's books, and the large control's
// 1 MiB requests are then satisfied out of that free list instead of by `mmap`. A
// control that never reached the mmap path could not measure the mmap path.
//
// Large first, on a fresh arena, is the only order in which the two are independent.
let b = control_large_blocks(700).report();
let a = control_small_blocks(700).report();
let dir = tempfile::Builder::new()
.prefix("znippy-rss-probe-")
.tempdir_in(scratch_root())
.expect("tempdir on the scratch root");
let real = real_stacks(dir.path(), 20, 300_000).report();
eprintln!("\n=== VERDICT ===");
// ── THE CALIBRATION, and it is not the one this probe was first written with ──────
//
// The original guard here asserted the two controls would DISAGREE — small blocks
// held in the arena, large blocks `munmap`ped straight back. **That premise is false
// on this box**: with a fresh arena BOTH controls return ~100% of their bytes on
// `drop` alone, before any trim. Recorded as a wrong prediction rather than quietly
// deleted, because it is load-bearing: it means block size is NOT what separates the
// controls from the real workload, so the real workload's retention needs a different
// explanation (fragmentation across live allocations and per-thread arenas, which
// top-of-heap trimming cannot reach — see the report).
//
// What the controls DO establish, and it is the only thing this probe needs from
// them: **the instrument can see memory come back.** A probe that could not would
// report "still resident" for everything and prove nothing. Both controls freeing
// clean is that calibration.
for c in [&b, &a] {
assert!(
c.held_after_drop_pct() < 10.0,
"{}: a control that allocates and frees a known {:.0} MB left {:.1}% resident. \
This probe cannot detect memory being returned, so NOTHING below can be \
attributed — the reading is about the instrument, not the workload.",
c.what,
mb(c.live.saturating_sub(c.base)),
c.held_after_drop_pct()
);
}
eprintln!(
"instrument OK: both controls returned their bytes on drop ({:.1}% / {:.1}% left), \
so a return IS visible to this probe.",
b.held_after_drop_pct(),
a.held_after_drop_pct()
);
let held_drop = real.held_after_drop_pct();
let held_trim = real.held_after_trim_pct();
// ── (a) — asked of the LIVE reading, which is the forge's actual state ────────────
//
// The forge's LRU bound is DEFAULT_OPEN_REPOS = 256 against ~20 repositories, so it
// never fires and NOTHING IS EVER RELEASED there. The live in-use figure is therefore
// the one that describes production, and the drop/trim pair below describes what
// would happen if eviction were fixed.
eprintln!(
"(a) while LIVE: {:.1} MB resident for {} databases — see the [live] in-use line \
above. If in-use tracks resident, these bytes are REACHABLE and deliberately \
held, and the owner is the per-database redb page cache at its \
{} MiB cap.",
mb(real.live.saturating_sub(real.base)),
20,
probe_cache_bytes() / (1024 * 1024),
);
// ── (b) — asked of the drop/trim pair ────────────────────────────────────────────
if held_drop < 30.0 {
eprintln!(
"(b) NO: dropping the stacks returned {:.1}% on its own, so releasing a store \
does give RSS back and eviction alone would be a complete fix.",
100.0 - held_drop
);
} else if held_trim < held_drop - 20.0 {
eprintln!(
"(b) YES — VERIFIED. {held_drop:.1}% of the bytes survived dropping EVERY handle, \
and malloc_trim(0) then returned them, leaving {held_trim:.1}%. The program owns \
nothing at that point; glibc was sitting on the pages.\n\
\n\
CONSEQUENCE, and it is the reason two eviction fixes changed nothing: eviction \
and trim are BOTH required. Evicting without trimming frees bytes the OS never \
gets back, so RSS stays flat and the fix looks like it failed. Trimming without \
evicting has nothing to trim, because a bound that never fires never frees."
);
} else {
eprintln!(
"(b) NO: {held_drop:.1}% survived the drop AND {held_trim:.1}% survived \
malloc_trim. Something still owns these bytes after every handle is gone. \
Find the owner."
);
}
eprintln!();
}