crates/gunnar-server/src/budget.rs
//! **The declared budgets**: how many git transfers may be in flight, how many
//! repository handles may be open, and how much memory the two of them may
//! hold — every one of them derived from the box and said out loud at start.
//!
//! # The numbers this replaces were not chosen, they were emergent
//!
//! Three separate ceilings decided gunnar's concurrency before 2026-08-30, and
//! not one of them was a number anybody picked for the machine it was running
//! on:
//!
//! | the ceiling | what actually set it | measured at |
//! |---|---|---|
//! | **~19** concurrent transfers | a point on the CoDel controller's descent: band `[10, 256]`, `limit -= limit/8` every 100 ms window that stands. `256 → 224 → … → 21 → 19 → 17` | the appliance, `already serving its limit of 19`, 268/1000 admitted at 64 clients |
//! | **25** concurrent pushes | `8 GiB budget / 320 MiB flat per-push reservation`, on any box, for any push | the native rig: 1969 of 3000 push-creates refused `MemoryBudget` at 64 clients while RSS never passed 1.6 GB of 8 GiB |
//! | **256** open repositories | `DEFAULT_OPEN_REPOS`, bounded by descriptors only — no memory term at all | the 2 GB appliance: collapse at ~1200 repositories, 256 handles holding the RAM the indexer watchdog then could not find |
//!
//! Two of the three are *arithmetic*, not decisions, and the third counted the
//! wrong resource. This module is the decision, in one place, with the three
//! facts it is derived from named beside it in the log:
//!
//! ```text
//! transfers 64 · open repos 200 · git memory 3.9G (cores 4, fds 1048576, MemAvailable 7.8G)
//! ```
//!
//! # Why it lives here and not in `gunnar-ssh`
//!
//! `gunnar-ssh` may not read the operating system — `crate::governor`'s
//! "OS-less, by law" section is explicit about it, and it is why the memory
//! budget over there is a *reservation the server owns* rather than a
//! measurement of the box. That law is kept. This crate is the one that already
//! reads `/proc` (`crate::limits` for `RLIMIT_NOFILE`, `crate::dashboard::probe`
//! for `/proc/meminfo`), so the derivation happens here and the *result* is
//! handed to `gunnar-ssh` as configuration — which is exactly the shape
//! `SshConfig::memory_budget_bytes` already documented as "the number an
//! operator sets".
//!
//! # Overridable, and the override is honoured exactly
//!
//! `--max-concurrent-transfers N`, or `GUNNAR_MAX_TRANSFERS=N` in the
//! environment (which is how the appliance's kernel command line reaches it:
//! `korp.env.GUNNAR_MAX_TRANSFERS=N`, since Linux hands every unrecognised
//! `key=value` boot parameter to `init` as an environment variable). An
//! operator's number is never raised and never lowered — see
//! [`gunnar_ssh::governor::LimitBand::for_budget`] for the one place that used
//! to do both.
use std::sync::OnceLock;
use crate::limits::{self, Nofile};
/// Concurrent transfers per core.
///
/// **A transfer is not a core's worth of work end to end.** It waits on the
/// client's socket, on the disk, and on the indexer behind it; the packing is
/// the only part that saturates. MEASURED on the rig (2026-08-30, 4 vCPU):
/// a push-create phase at 8 concurrent clients ran the appliance at **48 %**
/// CPU and a parallel-clone phase at 87 %, so at 2 transfers per core the box
/// was already half idle during the phase the gate refuses in.
///
/// Sixteen is where the native rig stopped getting faster per accepted
/// operation, measured 2026-08-30: a 4-core box
/// holds 64 concurrent push-creates at 162 ms each against 69 ms at 8, and 128
/// buys nothing but the extra latency. It is a *ceiling*, not a promise: the
/// governor still sheds inside it, the memory ledger still refuses inside it,
/// and the descriptor term below can be smaller.
pub const TRANSFERS_PER_CORE: usize = 16;
/// Descriptors one in-flight transfer costs on top of the handle cache: the
/// client socket, its channel, and headroom for the store handle it opens if
/// the cache misses.
///
/// Four, against [`limits::FDS_PER_OPEN_REPO`]'s five for a *cached* handle:
/// a transfer's own descriptors are the socket and the pipe pair, and the
/// repository it touches is normally already in the cache and already counted
/// there.
pub const FDS_PER_TRANSFER: u64 = 4;
/// The transfer ceiling never goes below this, whatever the box says.
///
/// A server that admits fewer than eight concurrent transfers is not a forge,
/// it is a queue with a socket. If the machine genuinely cannot carry eight,
/// the honest answer is the WARN row this floor produces and not a gate that
/// refuses everything.
pub const MIN_TRANSFERS: usize = 8;
/// The transfer ceiling never goes above this.
///
/// Half of `gunnar_server::serve::GIT_BLOCKING_THREADS` (512): every in-flight
/// transfer occupies one blocking thread for its whole life, and the pool also
/// carries pack indexing, ref transactions and the arrow log. A ceiling at the
/// pool size would make the pool the real limit again — the exact defect
/// the unbounded listener was about — and `serve.rs`'s
/// `the_blocking_pool_is_larger_than_the_transfer_limit` asserts the ordering
/// rather than leaving it to this paragraph.
pub const MAX_TRANSFERS: usize = crate::serve::GIT_BLOCKING_THREADS / 2;
/// The share of `MemAvailable` the **admission ledger** may give out to git
/// operations in flight, in percent.
///
/// Half. The other half is not spare: gunnar's own resident baseline sits
/// outside this budget by construction (`gunnar_ssh::governor`'s
/// `DEFAULT_MEMORY_BUDGET_BYTES` says so), the handle cache takes
/// [`HANDLE_CACHE_PERCENT`], and the indexer's watchdog holds its own floor at
/// 20 % of `MemTotal` (the memory limit). Two budgets
/// that add to 100 % of the same memory are one budget with extra steps.
pub const TRANSFER_MEMORY_PERCENT: u64 = 50;
/// The share of `MemAvailable` the **open-repository handle cache** may hold,
/// in percent.
///
/// Twenty. This is the term [`limits::open_repo_budget`] did not have:
/// it bounded the cache by descriptors and nothing else, and on the 2 GB
/// appliance 256 cached handles held the RAM that the indexer's watchdog then
/// spent every push trying and failing to reclaim — `scanned_rows_dropped=0` on
/// every store, because the memory was in the *other* 255 handles. The box
/// crawled at 12 s per create, and before the console fix it wedged
/// outright.
pub const HANDLE_CACHE_PERCENT: u64 = 20;
/// **What one indexed object costs in a cached open repository**, in bytes.
///
/// MEASURED on the reference desktop, 2026-08-31,
/// `znippy-plugin-git --example rss_per_open_store`: N identical stores opened
/// one at a time, each read from, all held — the state the handle cache is
/// actually in. Anonymous RSS out of `/proc/self/smaps_rollup`, with glibc's
/// `mallinfo2` in-use figure beside it as the control, and the two agree to
/// within a few percent on every row, so these are **live reachable bytes** and
/// not allocator retention.
///
/// | corpus | objects | MB per additional open store (live, `mallinfo`) | B/object | spread over the per-open steps |
/// |---|---:|---:|---:|---:|
/// | synthetic blobs | 10 000 | 4.3 | 430 | 51 % |
/// | synthetic blobs | 50 000 | 18.7 | 392 | 14 % |
/// | synthetic blobs | 200 000 | 73.4 | 385 | 16 % |
/// | synthetic blobs | 400 000 | 146.2 | 383 | 10 % |
/// | `znippy` pack | 7 314 | 6.8 | **966** | 1.7 % |
/// | `gunnar` pack | 12 849 | 9.2 | **744** | 2.5 % |
///
/// Dead straight over a 40× range of object counts, intercept ≈ 0.5 MB: the
/// cost of a cached handle **is not a constant, it is the repository's object
/// count**. Blobs alone are the floor at 382 B; a real repository, which also
/// folds a commit graph and an ordinal space at open
/// (`znippy_plugin_git::git_ops::Absorber::refold`), measured 744–966 B, so the
/// budget takes 768 — the side that keeps the box alive, as the figure this
/// replaces already said a budget must be.
pub const BYTES_PER_INDEXED_OBJECT: u64 = 768;
/// **The page cache one store's redb may additionally hold**, once reads have
/// warmed it: `znippy_plugin_git::arms::DEFAULT_REDB_CACHE_BYTES`.
///
/// **One database per store, not two.** The "two per repository" sentence in
/// `arms.rs` and in `storage::store_cache`'s docs is stale and it is what
/// the 2026-08-31 reading doubled: `git_ops::open_with_arms` opens exactly
/// one `ObjectReadStack`, and `znippy-plugin-git`'s own `tests/fd_per_store.rs`
/// counts exactly one `objects.tail` descriptor per store.
///
/// It is a ceiling and not a reservation — which is why the table above, taken
/// with only 64 reads per store, does not contain it. It IS reached under real
/// traffic: `znippy-plugin-git`'s `tests/rss_retention_probe.rs`, run on the
/// same box on the same day, measured **61.8 MB per stack** over 20 stacks of
/// 300 000 rows with ~43 000 lookups each. A budget must assume the warm case.
pub const REDB_CACHE_BYTES_PER_OPEN_REPO: u64 = 64 * 1024 * 1024;
/// **The repository size the handle ceiling is derived against**, in objects.
///
/// A stated assumption, not a measurement, and it is stated here precisely
/// because it is the one thing in this module that a box can be wrong about:
/// `derive` has no catalog and cannot know how big the repositories on this
/// machine are. A hundred thousand objects is an ordinary mid-sized repository
/// — `znippy` is 7 314, `gunnar` 12 849, `alice/linux-mirror` is 13 948 244.
///
/// **The by-construction fix this is standing in for** is a byte ledger in
/// [`crate::stores::ServerStores`]'s handle cache: charge each cached handle
/// [`bytes_per_open_repo`] of *its own* object count and evict when the sum
/// passes [`HANDLE_CACHE_PERCENT`] of `MemAvailable`, so that one 13.9-million
/// object repository costs what it costs instead of one slot out of N. That
/// needs an object count on the storage trait, which is a change in a second
/// repository; this constant is what makes the ceiling honest until it lands.
pub const BUDGETED_OBJECTS_PER_REPO: u64 = 100_000;
/// What one **cached open repository** of `objects` objects holds in memory.
///
/// # The figure this replaces, and why it was not merely too small
///
/// It was `8 * 1024 * 1024`, from the native rig on 2026-08-30: 256 cached
/// handles against a 1 114 MB RSS, **4.1 MB per handle**, rounded up to 8 MiB.
/// That measurement was not wrong — it was taken over 15 000 *small*
/// repositories, and 8 MiB is very nearly the right answer for a repository of
/// ~10 000 objects (measured above: 6.8 MB at 7 314, 9.2 MB at 12 849).
///
/// The defect is the **shape**. A constant cannot be the price of a thing whose
/// cost is linear in a number the constant does not mention. Priced at 8 MiB, a
/// 400 000-object repository is under-counted 18×, and `alice/linux-mirror` at
/// 13 948 244 objects is under-counted **1 300×** — one handle for it is
/// ~10.7 GB, which is two thirds of the 16 GB box the product ships on. The
/// derived ceiling on that box was 256 handles; 256 × the real price is not a
/// budget, it is a number with a unit.
pub const fn bytes_per_open_repo(objects: u64) -> u64 {
REDB_CACHE_BYTES_PER_OPEN_REPO + objects * BYTES_PER_INDEXED_OBJECT
}
/// What the handle ceiling is derived against: one cached open repository of
/// [`BUDGETED_OBJECTS_PER_REPO`] objects, with its redb page cache warm.
pub const BYTES_PER_OPEN_REPO: u64 = bytes_per_open_repo(BUDGETED_OBJECTS_PER_REPO);
/// The environment variable that overrides the derived transfer ceiling.
///
/// Reachable from the appliance's kernel command line as
/// `korp.env.GUNNAR_MAX_TRANSFERS=N` — Linux passes unrecognised `key=value`
/// boot parameters to `init` as environment variables, which is the same
/// channel `GUNNAR_NET` and `GUNNAR_DATA_ROOT` already arrive on.
pub const MAX_TRANSFERS_ENV: &str = "GUNNAR_MAX_TRANSFERS";
/// The environment variable that overrides the derived open-handle ceiling.
pub const OPEN_REPOS_ENV: &str = "GUNNAR_MAX_OPEN_REPOS";
/// The environment variable that overrides the derived memory budget, in bytes.
pub const MEMORY_BUDGET_ENV: &str = "GUNNAR_GIT_MEMORY_BYTES";
/// The three facts a budget is derived from.
///
/// Taken as a value rather than read inside the derivation so the whole rule is
/// a pure function of numbers a test can supply — including the boxes this one
/// is not (2 GB, one core, a thousand descriptors).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Machine {
/// CPUs this process may run on (`std::thread::available_parallelism`,
/// which reads the affinity mask, so a `taskset` or a cpuset is honoured).
pub cores: usize,
/// The descriptor soft limit **after** [`limits::nofile`] raised it.
pub nofile_soft: u64,
/// `MemTotal`, bytes. `0` when `/proc/meminfo` could not be read.
pub mem_total: u64,
/// `MemAvailable`, bytes. `0` when `/proc/meminfo` could not be read.
pub mem_available: u64,
}
impl Machine {
/// Read the box. Never fails: a missing `/proc/meminfo` leaves the memory
/// terms at zero, and [`derive`] then simply does not apply them — a budget
/// that refused to exist because one input was unreadable would be worse
/// than one derived from the two that were.
pub fn read(limit: Nofile) -> Machine {
let (mut mem_total, mut mem_available) =
crate::dashboard::probe::memory().unwrap_or((0, 0));
// A cgroup limit is the smaller truth, and gunnar runs in one whenever
// it is a container rather than the appliance. `/proc/meminfo` inside a
// cgroup still reports the HOST's memory: measured on the reference desktop, a server in
// a `MemoryMax=8G` scope read `MemAvailable 432.9G` and would have
// budgeted 216 GiB of a box it may use 8 of. The appliance is PID 1
// with no limit set, so there it changes nothing.
if let Some((cap, used)) = cgroup_memory() {
mem_total = if mem_total == 0 {
cap
} else {
mem_total.min(cap)
};
let free = cap.saturating_sub(used);
mem_available = if mem_available == 0 {
free
} else {
mem_available.min(free)
};
}
Machine {
cores: std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1),
nofile_soft: limit.soft,
mem_total,
mem_available,
}
}
}
/// `(memory.max, memory.current)` from this process's cgroup v2, in bytes.
///
/// `None` when there is no cgroup v2, no limit set (`max`), or the files cannot
/// be read — every one of which means "the box is the limit", which is what the
/// caller then uses. The same `/proc/self/cgroup` walk `serve::cpu_quota` does
/// for `cpu.max`; pure Rust, three file reads, once per process — and once per
/// `Admin.Capacity`, which clamps the RAM it quotes by the same rule
/// ([`crate::capacity::MachineFacts::measure`]).
pub(crate) fn cgroup_memory() -> Option<(u64, u64)> {
let own = std::fs::read_to_string("/proc/self/cgroup").ok()?;
let rel = own
.lines()
.find_map(|line| line.strip_prefix("0::"))?
.trim()
.trim_start_matches('/');
let dir = std::path::Path::new("/sys/fs/cgroup").join(rel);
let max = std::fs::read_to_string(dir.join("memory.max")).ok()?;
let max = max.trim();
if max == "max" {
return None;
}
let max: u64 = max.parse().ok()?;
let current: u64 = std::fs::read_to_string(dir.join("memory.current"))
.ok()?
.trim()
.parse()
.ok()?;
Some((max, current))
}
/// Where a budget's number came from.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Source {
/// Derived from the machine by [`derive`].
Derived,
/// An operator's flag or environment variable, honoured exactly.
Operator,
}
/// The three ceilings, and the box they were derived from.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Budgets {
/// Concurrent git transfers. Becomes
/// `SshConfig::max_concurrent_transfers`, which is the **ceiling of the
/// governor's band**; the floor is half of it
/// (`gunnar_ssh::governor::LimitBand::for_budget`).
pub transfers: usize,
/// Repository handles the cache may keep open.
pub open_repos: usize,
/// Bytes the admission ledger may give out to operations in flight.
pub memory_bytes: u64,
/// How long a transfer may wait for a slot, in milliseconds.
pub wait_ms: u64,
/// **What [`Budgets::open_repos`] handles can hold**, at
/// [`BYTES_PER_OPEN_REPO`] each, against
/// [`Budgets::handle_cache_share`] — the memory the cache is allowed.
///
/// Carried as a value rather than left as arithmetic in a doc comment
/// because it is the number that was wrong: the ceiling said 256 handles on
/// a 16 GB box while one handle really held 137 MB, and nothing anywhere
/// multiplied those two together. Now [`announce`] does, out loud.
pub handle_cache_bytes: u64,
/// `MemAvailable × HANDLE_CACHE_PERCENT / 100`: what the cache may hold.
pub handle_cache_share: u64,
/// **The floor could not be afforded.** `true` when
/// [`limits::MIN_OPEN_REPOS`] handles cost more than
/// [`Budgets::handle_cache_share`] — a box too small for the minimum cache
/// this server insists on having.
///
/// It is not an error and it does not refuse to serve: a server with no
/// handle cache is not a server, so the floor still wins and the process
/// still starts. What changes is that it is now *said*. The 2 GB
/// appliance reached the indexer watchdog's floor with the cache holding the
/// memory it was looking for, and every log line on the way there looked
/// ordinary.
pub over_promised: bool,
/// Where [`Budgets::transfers`] came from.
pub transfers_from: Source,
/// The box.
pub machine: Machine,
}
/// Bytes as `1.5G` / `820M` / `64K`, for one log line that a human reads.
pub fn human(bytes: u64) -> String {
const K: u64 = 1024;
match bytes {
b if b >= K * K * K => format!("{:.1}G", b as f64 / (K * K * K) as f64),
b if b >= K * K => format!("{}M", b / (K * K)),
b if b >= K => format!("{}K", b / K),
b => format!("{b}"),
}
}
/// **The rule.** Pure, so every box can be tested from any box.
///
/// Each ceiling is the *smallest* of its terms, because a budget is a promise
/// about the resource that runs out first:
///
/// ```text
/// open_repos = min( (soft − RESERVED_FDS) / FDS_PER_OPEN_REPO , // unchanged
/// MemAvailable × HANDLE_CACHE_PERCENT / BYTES_PER_OPEN_REPO )
/// clamped to [MIN_OPEN_REPOS, DEFAULT_OPEN_REPOS]
///
/// git_memory = MemAvailable × TRANSFER_MEMORY_PERCENT
///
/// transfers = min( cores × TRANSFERS_PER_CORE ,
/// (soft − RESERVED_FDS − open_repos × FDS_PER_OPEN_REPO) / FDS_PER_TRANSFER )
/// clamped to [MIN_TRANSFERS, MAX_TRANSFERS]
/// ```
///
/// The descriptor term subtracts the handle cache it just sized, because those
/// descriptors are spoken for: that subtraction is the whole of the bug
/// stated the other way round — 256 handles at five descriptors each were over
/// the limit *before* a single transfer had a socket.
///
/// There is deliberately **no** memory term on `transfers`. Memory is metered
/// per operation by the ledger, which now charges what a transfer actually
/// moves (`gunnar_ssh::governor::WIRE_MEMORY_NUMERATOR`) instead of a flat
/// per-class constant; turning the same budget into a second, static
/// concurrency cap is how `8 GiB / 320 MiB = 25` happened.
pub fn derive(machine: &Machine) -> Budgets {
let by_fds_repos =
machine.nofile_soft.saturating_sub(limits::RESERVED_FDS) / limits::FDS_PER_OPEN_REPO;
let by_mem_repos = if machine.mem_available == 0 {
u64::MAX
} else {
machine.mem_available / 100 * HANDLE_CACHE_PERCENT / BYTES_PER_OPEN_REPO
};
let open_repos = usize::try_from(by_fds_repos.min(by_mem_repos))
.unwrap_or(usize::MAX)
.clamp(limits::MIN_OPEN_REPOS, crate::stores::DEFAULT_OPEN_REPOS);
let memory_bytes = if machine.mem_available == 0 {
gunnar_ssh::governor::DEFAULT_MEMORY_BUDGET_BYTES
} else {
machine.mem_available / 100 * TRANSFER_MEMORY_PERCENT
};
let by_cores = machine.cores.saturating_mul(TRANSFERS_PER_CORE);
let spoken_for = limits::RESERVED_FDS + open_repos as u64 * limits::FDS_PER_OPEN_REPO;
let by_fds = usize::try_from(machine.nofile_soft.saturating_sub(spoken_for) / FDS_PER_TRANSFER)
.unwrap_or(usize::MAX);
let transfers = by_cores.min(by_fds).clamp(MIN_TRANSFERS, MAX_TRANSFERS);
// The multiplication nobody was doing. `open_repos` is a count and
// `BYTES_PER_OPEN_REPO` is a price; the product is the promise, and it is
// the promise that has to fit.
let handle_cache_share = machine.mem_available / 100 * HANDLE_CACHE_PERCENT;
let handle_cache_bytes = (open_repos as u64).saturating_mul(BYTES_PER_OPEN_REPO);
Budgets {
transfers,
open_repos,
memory_bytes,
wait_ms: gunnar_ssh::governor::DEFAULT_ADMISSION_WAIT.as_millis() as u64,
transfers_from: Source::Derived,
machine: *machine,
handle_cache_bytes,
handle_cache_share,
// Only when the FLOOR is what over-promises. Above the floor the
// division above already fits by construction, so a `true` here means
// exactly one thing: this box is too small for the minimum cache.
over_promised: machine.mem_available > 0 && handle_cache_bytes > handle_cache_share,
}
}
/// Read `name` as a positive `usize`, or `None`.
fn env_usize(name: &str) -> Option<usize> {
std::env::var(name)
.ok()?
.trim()
.parse()
.ok()
.filter(|n| *n > 0)
}
/// Read `name` as a positive `u64`, or `None`.
fn env_u64(name: &str) -> Option<u64> {
std::env::var(name)
.ok()?
.trim()
.parse()
.ok()
.filter(|n| *n > 0)
}
/// **This process's budgets, derived once and announced once.**
///
/// The first caller does the deriving and writes the INFO row; every later call
/// reads the same answer. It calls [`limits::nofile`], so the descriptor limit
/// is raised before it is budgeted against — the ordering that made the
/// 153-handle answer correct.
pub fn budgets() -> Budgets {
static BUDGETS: OnceLock<Budgets> = OnceLock::new();
*BUDGETS.get_or_init(|| {
let machine = Machine::read(limits::nofile());
let mut budgets = derive(&machine);
if let Some(n) = env_usize(MAX_TRANSFERS_ENV) {
budgets.transfers = n;
budgets.transfers_from = Source::Operator;
}
if let Some(n) = env_usize(OPEN_REPOS_ENV) {
budgets.open_repos = n;
}
if let Some(n) = env_u64(MEMORY_BUDGET_ENV) {
budgets.memory_bytes = n;
}
budgets
})
}
/// One INFO row naming the budgets **and the three facts they came from**.
///
/// The facts are the point. A ceiling in a log that a reader cannot tie to
/// anything about the machine is the same mystery `19` was; with the inputs
/// beside it, "why 64?" is answered in the line that states it.
///
/// **It says what is IN FORCE, not what was derived.** `serve` calls it after
/// an operator's `--max-concurrent-transfers` has been applied, because a row
/// that printed the derivation while the server ran on something else would be
/// a worse lie than no row: the first thing anybody does with this line is
/// compare it to the refusals underneath it.
pub fn announce(b: &Budgets) {
tracing::info!(
target: "gunnar::limits",
event = "limits.budget",
transfers = b.transfers,
transfers_from = ?b.transfers_from,
open_repos = b.open_repos,
git_memory_bytes = b.memory_bytes,
cores = b.machine.cores,
fds = b.machine.nofile_soft,
mem_total = b.machine.mem_total,
mem_available = b.machine.mem_available,
wait_ms = b.wait_ms,
handle_cache_bytes = b.handle_cache_bytes,
handle_cache_share = b.handle_cache_share,
bytes_per_open_repo = BYTES_PER_OPEN_REPO,
"transfers {} · open repos {} (≤ {} of {} at {}/repo) · git memory {} · admission wait \
{} ms (cores {}, fds {}, MemAvailable {})",
b.transfers,
b.open_repos,
human(b.handle_cache_bytes),
human(b.handle_cache_share),
human(BYTES_PER_OPEN_REPO),
human(b.memory_bytes),
b.wait_ms,
b.machine.cores,
b.machine.nofile_soft,
human(b.machine.mem_available),
);
if b.over_promised {
tracing::warn!(
target: "gunnar::limits",
event = "limits.handles.over_promised",
open_repos = b.open_repos,
handle_cache_bytes = b.handle_cache_bytes,
handle_cache_share = b.handle_cache_share,
mem_available = b.machine.mem_available,
"this box cannot afford the handle-cache floor: {} handles is the minimum this \
server keeps, they hold {} at {} each, and the cache's share of {} MemAvailable \
is only {}. Expect the indexer watchdog to pause pushes looking for memory the \
handle cache is holding — that is the 2 GB box's collapse, and it is now said at start \
instead of found at 12 s per create.",
b.open_repos,
human(b.handle_cache_bytes),
human(BYTES_PER_OPEN_REPO),
human(b.machine.mem_available),
human(b.handle_cache_share),
);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn machine(cores: usize, fds: u64, mem_gb: u64) -> Machine {
Machine {
cores,
nofile_soft: fds,
mem_total: mem_gb * 1024 * 1024 * 1024,
mem_available: mem_gb * 1024 * 1024 * 1024,
}
}
/// ★ **The appliance, priced at what a handle was MEASURED to hold.**
///
/// The box the product ships on: 4 cores, 16 GB, a raised descriptor limit.
/// [`Machine`] takes its inputs as a value precisely so a test can supply a
/// box this one is not, and this is the case that was missing: the handle
/// ceiling this module derives, times the memory one handle actually holds,
/// against the share the module says the cache may have.
///
/// **SEEN RED, 2026-08-31, against `BYTES_PER_OPEN_REPO = 8 * 1024 * 1024`:**
///
/// ```text
/// the derived ceiling of 256 open handles, at the 137M one cached
/// repository of 100000 objects was measured to hold, is 34.3G — but the
/// handle cache's share of this 16.0G box is 3.2G. The ceiling over-promises
/// by 10.7x.
/// ```
///
/// 256 is what the descriptor term gives, because the memory term divided a
/// box by a per-handle price that was 18× too small for the repositories on
/// it. Nothing in the log said so: `limits.budget` printed `open repos 256`
/// and every one of those handles looked affordable.
#[test]
fn the_handle_ceiling_may_not_promise_more_memory_than_its_own_share() {
let appliance = machine(4, 1 << 20, 16);
let b = derive(&appliance);
let share = appliance.mem_available / 100 * HANDLE_CACHE_PERCENT;
// **The MEASURED price on the right, `derive`'s own count on the left.**
// Not `BYTES_PER_OPEN_REPO` on both sides: a test that priced the
// ceiling with the same constant the ceiling was derived from is the
// identity value in its exact form — it would pass against ANY value of
// that constant, including the one that was 18× too small. What binds
// here is that `derive`'s divisor must BE the measurement.
let measured = bytes_per_open_repo(BUDGETED_OBJECTS_PER_REPO);
let promised = b.open_repos as u64 * measured;
assert!(
promised <= share,
"the derived ceiling of {} open handles, at the {} one cached repository of {} \
objects was measured to hold, is {} — but the handle cache's share of this {} \
box is {}. The ceiling over-promises by {:.1}x.",
b.open_repos,
human(measured),
BUDGETED_OBJECTS_PER_REPO,
human(promised),
human(appliance.mem_available),
human(share),
promised as f64 / share as f64,
);
}
/// ★ **A handle's price is the repository's object count, and the numbers
/// that says are not small.**
///
/// The second half of the same defect: a *constant* cannot be the price of a
/// linear thing, so the arithmetic is stated here against the two
/// repositories that bracket the range the appliance actually holds. Red
/// against the old constant on both rows — 8 MiB claimed to be the price of
/// a 13.9-million-object repository as readily as of a 7 314-object one.
#[test]
fn a_handle_is_priced_per_object_and_linux_mirror_does_not_fit_the_box() {
// `znippy` itself: 7 314 objects. Measured 6.8 MB of live bytes, which
// is what the cold price below should be in the same neighbourhood of.
let small = bytes_per_open_repo(7_314);
assert!(
small < 128 * 1024 * 1024,
"an ordinary repository must still be cheap: {}",
human(small)
);
// `alice/linux-mirror`: 13 948 244 objects (the count `memguard`'s
// module docs record from the push that panicked the appliance).
let mirror = bytes_per_open_repo(13_948_244);
assert!(
mirror > 10 * 1024 * 1024 * 1024,
"one open handle on linux-mirror is {} — the point of this assertion is that it \
is larger than the 16 GB box, not smaller",
human(mirror)
);
// …and therefore it cannot be held on the 4 GB standby it has never
// replicated to, nor beside 24 other repositories on the 16 GB one.
let standby = machine(4, 1 << 20, 4);
assert!(
mirror > standby.mem_available,
"one open handle on linux-mirror ({}) must not fit a 4 GB box ({})",
human(mirror),
human(standby.mem_available)
);
}
/// The rig the 2026-08-30 run measured on: 4 cores, a raised descriptor limit, 8 GiB.
/// This is the row the log prints and the row the bench table is against.
///
/// **The handle count moved 204 → 11 on 2026-08-31** and that is the fix,
/// not a regression: 204 was `8 GiB × 20 % / 8 MiB`, and 8 MiB was the price
/// of a repository a twentieth the size of the one the ceiling is now
/// derived against ([`BUDGETED_OBJECTS_PER_REPO`]). Eleven handles at 137 MB
/// is the same 1.6 GB of memory 204 handles were *supposed* to be; the
/// difference is that this number is true. A repository touched past the
/// eleventh re-opens, which costs a re-open and never a failure
/// (`crate::stores::DEFAULT_OPEN_REPOS`'s own note).
#[test]
fn the_rig_gets_sixty_four_transfers_and_a_cache_that_fits_its_memory() {
let b = derive(&machine(4, 1 << 20, 8));
assert_eq!(b.transfers, 64, "4 cores x TRANSFERS_PER_CORE");
// 8 GiB x 20 % = 1.6 G, / 137 MB per repository = 11.
assert_eq!(b.open_repos, 11);
assert_eq!(b.memory_bytes, 8 * 1024 * 1024 * 1024 / 100 * 50);
assert!(!b.over_promised, "8 GiB affords the floor");
assert!(b.handle_cache_bytes <= b.handle_cache_share);
}
/// **The 2 GB box's collapse, as arithmetic — and the box that cannot afford even
/// the floor.**
///
/// A 2 GB appliance may not keep 256 repositories open; the descriptor term
/// alone said 256, which is exactly the bug. Priced at 8 MiB it said 51,
/// which was the same bug with a smaller number. Priced at what a handle
/// holds it says **2**, which is under [`limits::MIN_OPEN_REPOS`] — so the
/// floor wins, and eight handles at 137 MB is 1.07 GiB of a box with 429 MB
/// to give the cache.
///
/// That is not made to pass by weakening the assertion. It is
/// [`Budgets::over_promised`], and [`announce`] says it in a WARN at start.
/// The honest statement about the 2 GB box is that it is too small for this
/// server's minimum cache, and the defect was that nothing anywhere said so.
#[test]
fn a_two_gigabyte_box_cannot_afford_even_the_handle_floor_and_now_says_so() {
let small = machine(4, 1 << 20, 2);
let by_fds_only = limits::open_repo_budget(small.nofile_soft);
let b = derive(&small);
assert_eq!(by_fds_only, crate::stores::DEFAULT_OPEN_REPOS);
assert_eq!(
b.open_repos,
limits::MIN_OPEN_REPOS,
"the floor, not the division"
);
assert!(
b.over_promised,
"8 handles x {} is {}, and this box's share is {} — the budget must NOTICE that",
human(BYTES_PER_OPEN_REPO),
human(b.handle_cache_bytes),
human(b.handle_cache_share)
);
// Red for the flag itself: a box that CAN afford the floor must not
// raise it, or the warning would be noise on every appliance.
assert!(!derive(&machine(4, 1 << 20, 16)).over_promised);
}
/// **The 1024-descriptor rig — and what the cheaper store
/// buys there.**
///
/// The descriptor term is the smaller one on that box, so it is the term
/// that decides, and it moved: an open store costs three descriptors
/// instead of five ([`limits::FDS_PER_OPEN_REPO`]), so `(1024 - 256) / 3`
/// is 256 where `(1024 - 256) / 5` was 153. **+67 % concurrently-open
/// repositories at the same `RLIMIT_NOFILE`**, which is the whole point of
/// the change: the divisor was three kernel objects and three page-cache
/// aliases per repository where one would do.
///
/// `transfers` still lands on its floor and that is not a regression - it
/// was on the floor before (3 descriptors left over) and is on it now (0),
/// because [`limits::RESERVED_FDS`], not the leftovers, is what a session
/// draws from. A box that wants more than eight concurrent transfers raises
/// its limit; [`limits::nofile`] does exactly that, and this is the shape of
/// the box that could not.
#[test]
fn a_thousand_descriptors_buys_sixty_seven_percent_more_open_repositories() {
// 256 GiB and not 64: this test is about the DESCRIPTOR term, so the
// memory term must not be the one that binds. At the price a handle was
// measured to hold (137 MB), 64 GiB × 20 % buys 95 handles and the
// memory term would decide — which would make this test quietly stop
// testing descriptors while still passing on the transfers line below.
let b = derive(&machine(4, 1024, 256));
assert_eq!(b.open_repos, limits::open_repo_budget(1024));
assert_eq!(b.open_repos, 256);
assert_eq!(
1024_u64.saturating_sub(limits::RESERVED_FDS) / 5,
153,
"the descriptor-only answer"
);
// The reservation is still whole: 256 handles x 3 + 256 reserved = 1024.
assert!(b.open_repos as u64 * limits::FDS_PER_OPEN_REPO + limits::RESERVED_FDS <= 1024);
assert_eq!(b.transfers, MIN_TRANSFERS);
}
/// A one-core box is still a server, and a 128-core box does not get 2048.
#[test]
fn the_floor_and_the_ceiling_both_bind() {
assert_eq!(
derive(&machine(1, 1 << 20, 64)).transfers,
MIN_TRANSFERS.max(16)
);
assert_eq!(derive(&machine(128, 1 << 20, 512)).transfers, MAX_TRANSFERS);
const { assert!(MAX_TRANSFERS < crate::serve::GIT_BLOCKING_THREADS) };
}
/// An unreadable `/proc/meminfo` costs the memory terms and nothing else.
#[test]
fn no_meminfo_falls_back_rather_than_refusing_to_have_a_budget() {
let blind = Machine {
cores: 4,
nofile_soft: 1 << 20,
mem_total: 0,
mem_available: 0,
};
let b = derive(&blind);
assert_eq!(b.transfers, 64);
assert_eq!(b.open_repos, crate::stores::DEFAULT_OPEN_REPOS);
assert_eq!(
b.memory_bytes,
gunnar_ssh::governor::DEFAULT_MEMORY_BUDGET_BYTES
);
}
#[test]
fn bytes_read_like_bytes() {
assert_eq!(human(0), "0");
assert_eq!(human(2048), "2K");
assert_eq!(human(8 * 1024 * 1024), "8M");
assert_eq!(human(3 * 1024 * 1024 * 1024 + 512 * 1024 * 1024), "3.5G");
}
}