crates/nordisk-alloc/tests/mimalloc_transient_peak.rs
//! **The OOM cell: a dropped transient must go back to the OS.**
//!
//! `dwarves-server` OOM-killed the 62 GB `dwarf` box eleven times on 2026-08-17/18 —
//! `anon-rss` 49–53 GB at every kill, always equal to systemd's reported `memory peak`
//! for that invocation, and the host itself died at 02:17:31. Not one pipeline run had
//! ever executed, so nothing in the forge accumulated. What the fat process does every
//! 60 s is a monitor sweep whose scan phases build very large **transients** — and
//! those transients were never given back.
//!
//! The cause is one default. `mimalloc = "0.1"` compiles the **v3** C sources
//! (`libmimalloc-sys`'s `build.rs` picks `v3` unless the `v2` cargo feature is set, and
//! nothing in this constellation sets it), and v3 raised `purge_delay` from v2's 10 ms
//! to **1000 ms**. At 1000 ms a freed heap is retained for the life of the process, so
//! RSS stops meaning "memory in use" and becomes the running MAXIMUM of every transient
//! ever built. [`nordisk_alloc::tune_for_long_lived_server`] is the fix for that term.
//!
//! ## Why this is an integration test and how it is RED-when-broken
//!
//! A `#[global_allocator]` may only be declared by a final artifact — and an
//! integration test *is* one, which is what lets this file drive the real allocator a
//! fat binary ships. It is a DIFFERENTIAL cell: the same workload, in the same process,
//! run once with the allocator exactly as it ships and once after the tuning.
//!
//! * **Phase A is the red control**, against the untouched shipped default: N rounds of
//! allocate-and-drop leave the memory resident. It is precisely what every untuned
//! nordisk binary does today, so it pins the bug rather than describing it in a
//! comment. Delete the fix and Phase A still holds while Phase B fails.
//! * **Phase B asserts the fix**: the identical workload's peak is handed back.
//!
//! Sizes are deliberately modest (≈400 MB per round) and every threshold is a fraction
//! of the round's own allocation, so the cell is fast and needs no tuning per host.
//! Only the first test touches the process-global option table, so cargo running the
//! others on parallel threads cannot race it.
//!
//! ## When it cannot measure, it says PROVED NOTHING — it does not guess
//!
//! Phase A's premise is a claim about a 1000 ms deadline on a box that is not itself
//! reclaiming, and neither is this cell's to control. Two conditions abandon the run
//! **loudly** rather than failing (a failure would read as "the fix is broken" when the
//! host was merely busy) — see [`overshot`] and [`control_did_not_hold`], and
//! `the_loud_skips_fire_on_the_conditions_they_name`, which proves both exits are
//! reachable and prints their lines. Phase B's assertions stay hard: a longer window or
//! a quieter box can only help the 10 ms deadline, so nothing about the environment can
//! manufacture a pass there.
// THE allocator under test — the same one every fat nordisk artifact installs.
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use nordisk_alloc::{
self as alloc, AllocTuning, PurgeDelayPlan, DEFAULT_PURGE_DELAY_MS,
MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS,
};
/// One transient's size: 400 000 heap blocks of ~1 KB ≈ 400 MB. ~1 KB `String`s, not
/// one huge `Vec`, because that is the shape the leaking workload actually has (a
/// deep-scan decompresses tens of thousands of source files into `String`s) and because
/// mimalloc routes a single huge block straight to the OS, which would bypass the very
/// retention this cell exists to measure.
const BLOCKS: usize = 400_000;
const BLOCK_BYTES: usize = 1_008;
/// The transient's approximate footprint in KiB.
const TRANSIENT_KB: u64 = (BLOCKS * BLOCK_BYTES / 1024) as u64;
/// How many allocate-and-drop rounds each phase drives. More than one, because the
/// defect is a RATCHET: the question is not "is one peak returned" but "does the floor
/// climb across iterations".
const ROUNDS: usize = 3;
/// Phase A passes only if LESS than a tenth of the transient came back, Phase B only if
/// MORE than two fifths did. The gap between the two bars is what makes the cell a
/// measurement rather than a coin flip, and it is wide on purpose: MEASURED on `dwarf`
/// (16 cores / 62 GB), as-shipped returns **0%** of a ~385 MiB transient and the tuned
/// allocator returns **53%**, so each bar sits a third of the range away from the number
/// it judges — and a repeat run returned 70%, so the bar has room on both sides. A cell
/// that needed per-host tuning would be re-tuned until it was green, which is how a
/// guard stops guarding.
const BARELY_ANY_KB: u64 = TRANSIENT_KB / 10;
const MOST_OF_IT_KB: u64 = TRANSIENT_KB * 2 / 5;
/// The product name this cell tunes under. Deliberately NOT a real binary's name: the
/// per-product env override is derived from it, and a cell that answered to
/// `DWARVES_MIMALLOC_PURGE_DELAY_MS` could be switched off from the outside by an
/// operator's setting on the box it runs on.
const PRODUCT: &str = "nordisk-alloc-oom-cell";
/// **How long a round watches RSS after the drop, and why it is not 1000 ms.**
///
/// It was dwarves' `40 × sleep(25ms)` — a nominal 1000 ms, i.e. EXACTLY mimalloc v3's
/// own `purge_delay`. A count is not a deadline: every `sleep` overshoots a little, so
/// the real window always landed just past 1000 ms (MEASURED here: 1.017–1.039 s), and
/// Phase A was watching the shipped allocator PAST the deadline whose non-expiry is its
/// entire claim. On a quiet box the purge is lazy enough that it still read as retained;
/// on a loaded one it did not, and the red control went red for load.
///
/// 850 ms is the same shape, bounded by the clock instead of by a tick count, leaving
/// ~150 ms of scheduling slack under the deadline. Past that, [`overshot`] refuses to
/// conclude rather than guessing.
const WATCH_TARGET: std::time::Duration = std::time::Duration::from_millis(850);
fn rss_kb() -> u64 {
alloc::self_rss_kb().expect("/proc/self/statm — this cell measures resident set")
}
/// One round's measurement — and how long the round actually got to WATCH.
#[derive(Clone, Copy)]
struct Round {
/// RSS while the transient is live.
peak: u64,
/// RSS after the drop.
after: u64,
/// Wall time the after-drop window really took. Not a diagnostic: Phase A's claim
/// is about a 1000 ms deadline, so a round the scheduler stretched past it is not
/// evidence — see [`overshot`].
watched: std::time::Duration,
}
/// Build a transient the way a scan does, report the peak RSS while it is live, then
/// drop it and report the RSS after the drop.
fn one_round(label: &str) -> Round {
let peak = {
let mut held: Vec<String> = Vec::with_capacity(BLOCKS);
for _ in 0..BLOCKS {
// Written, therefore resident — an untouched allocation would prove nothing.
held.push(String::from_utf8(vec![b'x'; BLOCK_BYTES]).expect("ascii"));
}
let peak = rss_kb();
assert_eq!(
held.len(),
BLOCKS,
"the transient must actually be live at the peak"
);
peak
};
// Give the allocator's delayed-purge deadline room to expire, then poke the
// allocation path so the purge queue is visited (mimalloc purges lazily, on
// allocator activity, not from a timer thread).
//
// ⚠ THIS LOOP IS FINELY BALANCED AND IT IS DWARVES' MEASURED SHAPE. Do not "make
// it more robust" — both ways were tried here on 2026-08-25 and both destroyed the
// differential:
//
// * Poking HARDER (64 blocks a tick instead of one) made the *as-shipped*
// allocator hand back 410 of 410 MB inside 893 ms. mimalloc reclaims under
// allocation PRESSURE regardless of the nominal deadline, so a fat poke erases
// Phase A — the red control passes nothing and the cell proves nothing.
// * Widening the WINDOW past 1000 ms let the shipped 1000 ms `purge_delay` come due
// inside the observation, with the same result.
//
// One ~1 KB block per 25 ms is the minimum activity that services the purge queue
// without becoming pressure, and 40 ticks keeps the window under v3's own deadline.
// What the window CANNOT control is the scheduler: on a loaded host `sleep(25ms)`
// overshoots and the 40 ticks run well past 1000 ms, at which point Phase A is
// watching the shipped allocator PAST its deadline and goes red for load rather
// than for a change in mimalloc. So the elapsed time is recorded and
// [`overshot`] refuses to draw a conclusion from such a round — LOUDLY.
let started = std::time::Instant::now();
let mut after = rss_kb();
while started.elapsed() < WATCH_TARGET {
std::thread::sleep(std::time::Duration::from_millis(25));
drop(String::from_utf8(vec![b'y'; BLOCK_BYTES]).expect("ascii"));
after = rss_kb();
if after + MOST_OF_IT_KB < peak {
break; // already returned — no reason to keep waiting
}
}
let watched = started.elapsed();
eprintln!(
"[{label}] peak {peak} KiB → after-drop {after} KiB (transient ~{TRANSIENT_KB} KiB, \
watched {watched:?})"
);
Round {
peak,
after,
watched,
}
}
/// **A Phase-A round that ran past mimalloc's own deadline proved nothing, and must
/// say so out loud.**
///
/// Returns true when the cell must stop. Phase A's claim is "the shipped allocator has
/// not given the transient back *yet*" — a claim about a 1000 ms deadline. A round the
/// scheduler stretched past that deadline is not evidence about the allocator in either
/// direction, and asserting on it would make the red control report the host's load.
///
/// Phase B needs no equivalent: a longer window can only give the 10 ms deadline MORE
/// room, so an overshoot there cannot manufacture a pass.
fn overshot(rounds: &[&Round]) -> bool {
let deadline = std::time::Duration::from_millis(MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS as u64);
let worst = rounds.iter().map(|r| r.watched).max().unwrap_or_default();
if worst < deadline {
return false;
}
unproven(&format!(
"a round watched for {worst:?}, past mimalloc v3's own \
{MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS} ms purge_delay, so whatever the shipped \
allocator did inside that window says nothing about the deadline"
));
true
}
/// **The red control's premise did not hold: the SHIPPED allocator gave the transient
/// back.**
///
/// Two things produce that, and this cell cannot tell them apart from the inside:
///
/// 1. **The host is reclaiming.** mimalloc hands pages back with a `MADV_FREE`-shaped
/// decommit, and a box under memory pressure has the kernel take them regardless of
/// any `purge_delay`. MEASURED here 2026-08-25 with ~40 concurrent cargo jobs: the
/// untuned allocator's floor fell to 9 364 KiB from a 419 604 KiB peak inside 893 ms.
/// 2. **mimalloc's retention actually changed** — a v4 default, a `v2` feature flip.
/// That is a REAL finding and the constants in `nordisk_alloc` need re-measuring.
///
/// Cause 1 is not a defect and must not be reported as one; cause 2 must not be
/// silently swallowed. So the run says PROVED NOTHING, loudly, and names both — rather
/// than failing (which would blame the fix for the host) or passing (which would let a
/// real mimalloc change through).
fn control_did_not_hold(rounds: &[&Round]) -> bool {
let worst = rounds
.iter()
.map(|r| r.peak.saturating_sub(r.after))
.max()
.unwrap_or(0);
if worst < BARELY_ANY_KB {
return false;
}
unproven(&format!(
"the AS-SHIPPED allocator handed back {worst} KiB of a ~{TRANSIENT_KB} KiB \
transient, so the red control's premise (an untuned mimalloc RETAINS it) did \
not hold on this host. Either this box is reclaiming under memory pressure — \
mimalloc decommits with MADV_FREE and the kernel takes those pages whatever \
purge_delay says — or mimalloc's retention behaviour genuinely changed, which \
is a REAL finding: re-measure nordisk_alloc's option table and the 10 ms value \
before trusting either number"
));
true
}
/// The ONE loud-skip line, so every route out of this cell says the same thing and
/// none of them can be mistaken for a pass that measured something.
fn unproven(why: &str) {
eprintln!(
"SKIPPED and therefore PROVED NOTHING — {why}. UNPROVEN by this run: that an \
untuned mimalloc RETAINS a dropped transient (the red control), and therefore \
also that `tune_for_long_lived_server` is the thing that RETURNS it — the \
differential was abandoned and Phase B was not reached. The OOM this cell \
guards (dwarves-server, 49–53 GB anon-rss, eleven kills, dwarf 2026-08-17/18) \
is UNGUARDED by this run. Re-run on a quiet box before believing the suite."
);
}
#[test]
fn a_dropped_transient_is_retained_as_shipped_and_returned_after_the_tuning() {
// ── Phase A — the allocator EXACTLY as every untuned nordisk binary ships it ──
//
// Nothing is configured yet. This is the code that OOM-killed the box.
let shipped = alloc::current_purge_delay_ms();
assert_eq!(
shipped, MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS,
"this cell's premise is mimalloc v3's 1000 ms purge_delay default; the linked \
mimalloc reports {shipped} ms. If mimalloc changed, re-derive the finding \
before trusting either phase (see nordisk_alloc)."
);
let baseline = rss_kb();
let shipped_rounds: Vec<Round> = (0..ROUNDS)
.map(|round| {
let r = one_round(&format!("as-shipped round {round}"));
assert!(
r.peak > baseline + MOST_OF_IT_KB,
"round {round}: the transient did not become resident (peak {} KiB vs \
baseline {baseline} KiB) — the cell measured nothing",
r.peak
);
r
})
.collect();
// THE DEFECT, the red control: after every round essentially none of the transient
// came back. Measured on `dwarf`: 443 648 KiB peak → 443 712 KiB after the drop.
// Zero. Measured on t14s (quiet): 419 604 KiB → 419 604 KiB. Also zero.
//
// Both exits below are LOUD SKIPS, not failures, and the reason is that a failure
// here would read as "the fix is broken" when what actually happened is that the
// control conditions were absent. See `control_did_not_hold` for the two causes it
// cannot tell apart, and why swallowing either would be worse.
let a = shipped_rounds.iter().collect::<Vec<_>>();
if overshot(&a) || control_did_not_hold(&a) {
return;
}
// ── Phase B — the fix ─────────────────────────────────────────────────────────
let tuning = alloc::tune_for_long_lived_server(PRODUCT);
assert!(
tuning.applied,
"the tuning must apply: {}",
tuning.log_line()
);
assert_eq!(
tuning.purge_delay_before_ms,
MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS
);
assert_eq!(tuning.purge_delay_after_ms, DEFAULT_PURGE_DELAY_MS);
// ONE settle round, measured but not judged. Phase A left this process's heap with
// the purge it scheduled under the shipped 1000 ms delay still pending, and the
// first tuned round frees into that schedule: MEASURED on dwarf 2026-10-30, in 1 of
// 4 runs (and in edda-ci at 10f7d527) the first tuned round gave back nothing
// (428 936 KiB → 428 936 KiB) while every round after it gave back ~55 %. That is the
// differential's own leftover — a server tunes at its start, before any transient
// (`tune_for_long_lived_server`'s contract) — so the judged rounds start after it.
one_round("tuned settle round (Phase A's pending purge, not judged)");
let tuned_rounds: Vec<Round> = (0..ROUNDS)
.map(|round| one_round(&format!("tuned round {round}")))
.collect();
// And the floor does not ratchet: the last round must not sit above the first by
// another whole transient. This is the property the incident was — not one big
// peak, but a floor that climbed with every sweep.
let first_after = rss_kb();
let probe = one_round("tuned ratchet probe");
// THE FIX, asserted: each round's peak is handed back to the OS. Measured on
// `dwarf`, two runs: 444 032 KiB peak → 233 664 / 217 344 / 224 000 KiB, and
// 444 036 KiB peak → 135 300 / 140 420 / 140 548 KiB.
for (round, r) in tuned_rounds.iter().enumerate() {
assert!(
r.peak > r.after,
"tuned round {round}: RSS did not fall at all after the drop ({} KiB → {} KiB)",
r.peak,
r.after
);
let gave_back = r.peak - r.after;
assert!(
gave_back > MOST_OF_IT_KB,
"tuned round {round}: only {gave_back} KiB of a ~{TRANSIENT_KB} KiB \
transient was returned to the OS. purge_delay is \
{DEFAULT_PURGE_DELAY_MS} ms and it is not taking effect — this is the OOM."
);
}
assert!(
probe.after < first_after + MOST_OF_IT_KB,
"the resident floor still ratcheted across rounds ({first_after} KiB → {} KiB, \
transient ~{TRANSIENT_KB} KiB): a long-lived server would still climb to the \
OOM, one sweep at a time.",
probe.after
);
}
/// **GUARD ON THE GUARD — the two loud-skip exits are REACHABLE and they FIRE.**
///
/// A skip route that is never exercised is indistinguishable from one that is wired
/// wrong, and this cell's whole integrity rests on it: without these exits the two
/// conditions below came out as a hard FAILURE that blamed the fix for the host's load.
/// Fed rounds that state each condition, each guard must answer "stop", and rounds that
/// state neither must answer "carry on".
///
/// The `SKIPPED and therefore PROVED NOTHING` lines print here (visible under
/// `--nocapture`) — this test is also the only place they are guaranteed to be seen.
#[test]
fn the_loud_skips_fire_on_the_conditions_they_name() {
let ok = Round {
peak: 419_604,
after: 419_604,
watched: std::time::Duration::from_millis(860),
};
assert!(
!overshot(&[&ok]) && !control_did_not_hold(&[&ok]),
"a round inside the window, with the transient retained, is the measurable case"
);
// The scheduler stretched the window past mimalloc's own deadline.
let slow = Round {
watched: std::time::Duration::from_millis(MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS as u64 + 24),
..ok
};
assert!(
overshot(&[&slow]),
"past 1000 ms must abandon the differential"
);
// The as-shipped allocator handed the transient back — the red control's premise
// did not hold. MEASURED shape: 419 604 KiB peak → 9 364 KiB, on a loaded t14s.
let reclaimed = Round { after: 9_364, ..ok };
assert!(
control_did_not_hold(&[&reclaimed]),
"a control that did not hold must skip loudly, never fail — a failure here \
would read as `the fix is broken` when the host was simply reclaiming"
);
// …and one bad round among good ones is enough: the phase is only as sound as its
// weakest round.
assert!(control_did_not_hold(&[&ok, &reclaimed, &ok]));
}
/// `<PRODUCT>_MIMALLOC_PURGE_DELAY_MS=off` (or the fleet-wide
/// `NORDISK_MIMALLOC_PURGE_DELAY_MS=off`) is a real escape hatch and it is HONEST about
/// it — an operator who turns the fix off gets a boot line saying a dropped transient
/// may stay resident, naming their own product's variable, never silence.
#[test]
fn the_opt_out_is_reported_and_never_silent() {
let off = alloc::purge_delay_plan(Some("off"));
assert_eq!(off, PurgeDelayPlan::LeaveAlone);
let untuned = AllocTuning {
product: "dwarves".into(),
purge_delay_before_ms: MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS,
purge_delay_after_ms: MIMALLOC_V3_DEFAULT_PURGE_DELAY_MS,
arena_purge_mult: 1,
applied: false,
};
let line = untuned.log_line();
assert!(line.contains("may stay resident"), "{line}");
assert!(
line.contains("1000"),
"the line must name the value in force: {line}"
);
assert!(
line.contains("DWARVES_MIMALLOC_PURGE_DELAY_MS"),
"and the variable that turns it back on, for THIS product: {line}"
);
}