crates/gunnar-ssh/tests/keyed_reserve.rs
//! **The keyed reserve** (D23b): the top of the box belongs to callers with a
//! key, and a stranger can never be holding it.
//!
//! # The hole this closes
//!
//! `tests/anonymous_ceiling.rs` gave the one caller with no key a hard ceiling
//! of [`DEFAULT_MAX_ANONYMOUS_TRANSFERS`] (5), refused instantly and never
//! parked. That bounds how much a stranger can take *in absolute terms*, and on
//! the shipped appliance — 5 against a server-wide 256 — it is very nearly the
//! whole answer: anonymous is under 2 % of the box.
//!
//! It is not the whole answer, and the gap is a number that moves. The ceiling
//! is deliberately **absolute** while the thing it is a share of is
//! **governed**: `Governor`'s CoDel controller sheds the server-wide limit down
//! its band whenever the box is queueing, and the shed was measured
//! reaching a limit of 19. Five out of 19 is a quarter of the machine, held by
//! a caller that cannot be named, billed, revoked or telephoned — and held at
//! precisely the moment the box has least to give. The same arithmetic applies
//! to an operator who raised the ceiling on purpose for a browse replica: the
//! number they raised has no relationship to the number the controller is
//! currently enforcing.
//!
//! Nothing evicts a transfer already in flight (and nothing should — see the
//! module docs of `gunnar_ssh::admission`), so the only way a keyed caller can
//! be *structurally* safe from anonymous load is for the stranger never to
//! **acquire** the contended slot in the first place.
//!
//! # The rule
//!
//! An anonymous read is admitted only while the server is *below*
//! [`CONTENTION_PERCENT`] of its governed limit. At or above it the stranger is
//! refused with [`Refusal::KeyedReserve`] — instantly, never parked, counted on
//! its own — and the top quarter of the box is keyed-only.
//!
//! The threshold is not a new number. It is the *same* one the max-min share
//! already uses, and the symmetry is the argument: `CONTENTION_PERCENT` is
//! defined as the point at which gunnar starts asking keyed identities to share
//! with each other, and that is exactly the point at which it should stop asking
//! them to share with strangers.
//!
//! # Seen red, three times
//!
//! Written before the reserve existed. The first run does not compile —
//! `Refusal::KeyedReserve`, `AdmissionStats::refused_keyed_reserve` and
//! `Admission::keyed_reserve` were not there:
//!
//! ```text
//! error[E0599]: no variant named `KeyedReserve` found for enum `Refusal`
//! error[E0609]: no field `refused_keyed_reserve` on type `AdmissionStats`
//! ```
//!
//! With the types wired in but the rule not applied — the honest red, because
//! an implementation that ignored the number would pass a test that only
//! checked the types existed — three of the seven failed, and each is quoted in
//! the doc comment of its own test.
//!
//! The **third** red is the one worth reading, because it is the reason
//! `Governor::claim_reserving` exists. The first implementation put the rule in
//! `Admission::attempt` where it looked natural: take the claim, read
//! `Governor::occupied`, compare it to the threshold, hand the claim back if it
//! failed. Five of seven went green, and these two did not:
//!
//! ```text
//! ---- a_reserve_refusal_is_never_parked stdout ----
//! refused by the reserve, got AtCapacity { limit: 4 }
//!
//! ---- a_keyed_caller_walks_in_while_strangers_hammer_the_gate stdout ----
//! keyed admission 3 was PARKED for 5.019µs behind anonymous load; the
//! reserve exists so that a caller with a key is admitted on the first ask
//! ```
//!
//! Both are the same defect. A stranger that reads occupancy *after* taking a
//! claim is holding a slot while it decides to give it back, and during that
//! window the server-wide gate is full — so a keyed caller arriving into it was
//! refused by gate 1 (`AtCapacity`), by a slot no stranger was going to keep,
//! with the reserve that existed to protect it never consulted. Checking
//! *before* the claim only moves the race: `N` strangers all read the same low
//! occupancy and all decide they fit. The reserve had to ride inside the same
//! `fetch_update` that enforces the limit itself, which is what
//! `Governor::claim_reserving` does.
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
use gunnar_ssh::admission::{CONTENTION_PERCENT, DEFAULT_MAX_ANONYMOUS_TRANSFERS};
use gunnar_ssh::config::DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL;
use gunnar_ssh::governor::OpClass;
use gunnar_ssh::gunnar_core::ANONYMOUS_PRINCIPAL_ID;
use gunnar_ssh::{Admission, AdmissionGuard, Refusal};
/// A **shed** box: four slots, which is what the CoDel controller leaves of a
/// bigger one when the machine is in trouble.
///
/// Four and not, say, twelve because `LimitBand::for_budget(4)` has `min ==
/// max == 4`: the controller cannot move it, so every number in these tests is
/// the number the test wrote down. The anonymous ceiling is left at its shipped
/// 5 — larger than the whole box, which is the point. On a shed box the
/// absolute ceiling has stopped meaning anything and only the reserve is left.
fn shed_gate() -> Arc<Admission> {
Arc::new(
Admission::new(Some(4), Some(DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL))
.with_anonymous_ceiling(Some(DEFAULT_MAX_ANONYMOUS_TRANSFERS))
// No wait: these tests are about *who* is refused. The fact that the
// reserve's refusal is never parked is tested on its own, in
// `a_reserve_refusal_is_never_parked`, with a real wait.
.with_wait(Duration::ZERO),
)
}
/// How many slots a gate holds back for keyed callers, computed here from the
/// public constant rather than read from the gate, so that the gate's own
/// arithmetic is being *checked* and not merely restated.
fn reserve_of(limit: usize) -> usize {
// The largest occupancy that is still below contention is
// `ceil(limit * P / 100) - 1`; everything above it is reserved.
limit - (limit * CONTENTION_PERCENT).div_ceil(100) + 1
}
/// **The required property**: a stranger never takes the last slots.
///
/// Four slots, one anonymous flood, and afterwards two keyed callers must both
/// walk straight in. Nothing about this test depends on timing: it is a
/// statement about which slots anonymous is *allowed to acquire*.
///
/// # Seen red
///
/// Against the gate with the reserve wired in but switched off:
///
/// ```text
/// ---- a_stranger_never_takes_the_last_slots_a_keyed_caller_needs stdout ----
/// assertion `left == right` failed: anonymous took 4 of the 4 slots on this
/// box, leaving 0; the keyed reserve is 2, so at most 2 may ever be anonymous
/// left: 4
/// right: 2
/// ```
///
/// The whole gap in one line: the anonymous ceiling is 5, the box is 4, so the
/// ceiling never fires and a stranger takes the machine.
#[test]
fn a_stranger_never_takes_the_last_slots_a_keyed_caller_needs() {
let gate = shed_gate();
let limit = gate.governed_limit().expect("a limited gate");
let reserve = reserve_of(limit);
// Take anonymous slots until the door says no. There is no `while true`
// hazard here: either the reserve stops it at `limit - reserve`, or the
// server-wide gate stops it at `limit`.
let mut anonymous: Vec<AdmissionGuard> = Vec::new();
let refusal = loop {
match gate.try_admit(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone) {
Ok(guard) => anonymous.push(guard),
Err(refusal) => break refusal,
}
};
assert_eq!(
anonymous.len(),
limit - reserve,
"anonymous took {} of the {limit} slots on this box, leaving {}; the keyed reserve is \
{reserve}, so at most {} may ever be anonymous",
anonymous.len(),
limit - anonymous.len(),
limit - reserve
);
assert_eq!(
refusal,
Refusal::KeyedReserve {
// The occupancy the gate's own compare-and-swap observed, which is
// the number that made the decision: `limit - reserve` slots are
// taken, and every slot left is reserved.
occupied: limit - reserve,
limit,
reserve,
},
"the refusal must name the RESERVE. `AtCapacity` sends the operator to raise the \
concurrency limit and `AnonymousAtCapacity` sends them to lower the anonymous ceiling; \
neither is what happened, and neither is the knob that changes it"
);
let stats = gate.stats();
assert_eq!(
stats.refused_keyed_reserve, 1,
"the reserve gets its OWN counter: folded into `refused_anonymous` an operator cannot \
tell `strangers are at their cap` (often benign) from `the box is so full that strangers \
are being kept out of it` (which is a capacity problem)"
);
assert_eq!(
stats.refused_anonymous,
0,
"and it is not counted as the absolute ceiling, which did not fire: the ceiling here is \
{:?}, larger than the whole box",
gate.max_anonymous()
);
assert_eq!(stats.refused_at_capacity, 0, "nor as a full server");
assert_eq!(stats.refused(), 1, "it is a refusal, so the total moves");
// The whole point: the reserved slots are really there, and they are really
// keyed-only.
let keyed: Vec<AdmissionGuard> = (0..reserve)
.map(|i| {
gate.try_admit("alice", "team", OpClass::Clone)
.unwrap_or_else(|e| {
panic!("keyed caller {i} must walk into a reserved slot, and was refused: {e}")
})
})
.collect();
assert_eq!(keyed.len(), reserve);
assert_eq!(
gate.stats().in_flight as usize,
limit,
"and between them they fill the box exactly"
);
}
/// **What the reserve guarantees exactly, since it does not evict**: above the
/// threshold, anonymous occupancy only ever goes *down*.
///
/// This is the honest statement of the property, and it is weaker than "a keyed
/// caller is never refused while a stranger holds anything". It has to be
/// weaker: a stranger admitted onto a quiet box is still holding its slot if the
/// box fills a moment later, and the only way to make the stronger claim true is
/// to evict it — which this gate will not do, for the reasons in the module docs
/// of `gunnar_ssh::admission`.
///
/// What is true instead, and what this pins, is a **ratchet**. Once occupancy is
/// at or above [`CONTENTION_PERCENT`] no stranger is admitted again, so every
/// anonymous transfer that finishes hands its slot to the keyed side and cannot
/// take it back. A flood therefore cannot hold a busy box at its anonymous
/// maximum: it decays to zero at the rate transfers complete, and the existing
/// `SshConfig::inactivity_timeout` bounds that rate from above.
///
/// # Seen red
///
/// Against the same file with the reserve forced to `0` in
/// `Admission::attempt` — the wired-but-off arm:
///
/// ```text
/// ---- the_stranger_cannot_retake_a_slot_it_gives_back_on_a_busy_box stdout ----
/// the stranger RETOOK the slot it released on a box at 3 of 4; above the
/// contention threshold anonymous occupancy must only ever fall
/// ```
#[test]
fn the_stranger_cannot_retake_a_slot_it_gives_back_on_a_busy_box() {
let gate = shed_gate();
let limit = gate.governed_limit().expect("a limited gate");
let reserve = reserve_of(limit);
// Anonymous gets in first, on a box that is still quiet — which is exactly
// the case the reserve does NOT cover, and the reason the property below is
// a ratchet rather than an absolute.
let mut anonymous: Vec<AdmissionGuard> = (0..limit - reserve)
.map(|i| {
gate.try_admit(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone)
.unwrap_or_else(|e| panic!("anonymous {i} arrives on a quiet box: {e}"))
})
.collect();
// Now the box fills with real work.
let _keyed: Vec<AdmissionGuard> = (0..reserve)
.map(|i| {
gate.try_admit("alice", "team", OpClass::Clone)
.unwrap_or_else(|e| panic!("keyed {i} takes a reserved slot: {e}"))
})
.collect();
// One stranger finishes. The slot it gives back is above the threshold, so
// it belongs to the keyed side now.
drop(anonymous.pop().expect("a stranger was holding one"));
let occupied = gate.stats().in_flight as usize;
assert_eq!(occupied, limit - 1, "exactly one slot came free");
let refusal = match gate.try_admit(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone) {
Ok(_) => panic!(
"the stranger RETOOK the slot it released on a box at {occupied} of {limit}; above \
the contention threshold anonymous occupancy must only ever fall"
),
Err(refusal) => refusal,
};
assert!(
matches!(refusal, Refusal::KeyedReserve { .. }),
"and it is the reserve that says so, got {refusal:?}"
);
gate.try_admit("ci", "team", OpClass::Clone)
.expect("the freed slot goes to a caller with a key");
}
/// The reserve is **anonymous-only**. A keyed caller is never refused by it,
/// even standing on the very last slot.
///
/// Without this the change would have swapped one unfairness for another: a
/// gate that held a quarter of itself back from *everybody* would refuse work
/// the box can do, which is the exact mistake the max-min share was rewritten
/// to stop making.
#[test]
fn the_reserve_never_refuses_a_keyed_caller() {
let gate = shed_gate();
let limit = gate.governed_limit().expect("a limited gate");
// Four different identities, so the per-principal share is in play too and
// this is not accidentally a test about one principal.
let held: Vec<AdmissionGuard> = ["alice", "ci", "gunnar", "builder"]
.iter()
.take(limit)
.enumerate()
.map(|(i, who)| {
gate.try_admit(who, "team", OpClass::Clone)
.unwrap_or_else(|e| {
panic!("keyed caller {i} ({who}) must be admitted into an empty box: {e}")
})
})
.collect();
assert_eq!(held.len(), limit, "keyed callers get the WHOLE box");
assert_eq!(
gate.stats().refused_keyed_reserve,
0,
"the reserve must never fire on a keyed caller"
);
// And the slot after that is refused by the server-wide gate, with the
// server-wide gate's own reason.
assert_eq!(
gate.try_admit("alice", "team", OpClass::Clone)
.expect_err("the box is full"),
Refusal::AtCapacity { limit },
"a full box refuses a keyed caller as a full box, not as a reserve"
);
}
/// **The floor anonymous keeps.** On a box that is not busy, nothing changed.
///
/// This is the constraint that stops the reserve collapsing into `anonymous
/// gets nothing whenever an operator is logged in`. A public repository that
/// can never be cloned is not a feature, so below [`CONTENTION_PERCENT`] the
/// stranger keeps its entire allowance and the only ceiling it meets is the
/// absolute one.
#[test]
fn an_idle_box_still_gives_the_stranger_its_whole_allowance() {
let gate = Arc::new(
Admission::new(Some(64), Some(DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL))
.with_anonymous_ceiling(Some(DEFAULT_MAX_ANONYMOUS_TRANSFERS))
.with_wait(Duration::ZERO),
);
let held: Vec<AdmissionGuard> = (0..DEFAULT_MAX_ANONYMOUS_TRANSFERS)
.map(|i| {
gate.try_admit(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone)
.unwrap_or_else(|e| {
panic!(
"anonymous read {i} must be admitted on an idle box; the reserve is for a \
BUSY one, and a reserve that binds at rest has taken the public door \
away: {e}"
)
})
})
.collect();
assert_eq!(held.len(), DEFAULT_MAX_ANONYMOUS_TRANSFERS);
// And the refusal after that is still the ceiling's, unchanged: the reserve
// did not steal the older rule's refusal.
assert_eq!(
gate.try_admit(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone)
.expect_err("the ceiling is 5"),
Refusal::AnonymousAtCapacity {
limit: DEFAULT_MAX_ANONYMOUS_TRANSFERS
},
"on an idle box the ceiling is what refuses, and it must keep saying so"
);
let stats = gate.stats();
assert_eq!(stats.refused_anonymous, 1);
assert_eq!(
stats.refused_keyed_reserve, 0,
"the reserve must not fire on a box with 59 free slots"
);
}
/// The reserve's refusal is **final and immediate**, like the ceiling's.
///
/// Same argument, and it is not weakened by being about a busy box rather than
/// a capped caller: parking a stranger holds its TCP socket, its russh session
/// and its key material for the whole bound, and those are the resources a
/// clone fork-bomb spends gunnar's box to acquire. Parking N strangers to spare
/// them a reconnect hands an attacker N held sessions for free.
///
/// It is also pointless on its own terms here. What the parked stranger would
/// be waiting for is the box to fall back below `CONTENTION_PERCENT`, which is
/// a keyed caller finishing — and a stranger has no claim on being first in
/// line for that. The keyed control below is the one that still parks.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_reserve_refusal_is_never_parked() {
let wait = Duration::from_millis(400);
let gate = Arc::new(
Admission::new(Some(4), Some(DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL))
.with_anonymous_ceiling(Some(DEFAULT_MAX_ANONYMOUS_TRANSFERS))
.with_wait(wait),
);
let limit = gate.governed_limit().expect("a limited gate");
// Fill the box with keyed traffic, so the stranger below meets the reserve
// and not the anonymous ceiling.
let _keyed: Vec<AdmissionGuard> = (0..limit)
.map(|i| {
gate.try_admit("alice", "team", OpClass::Clone)
.unwrap_or_else(|e| panic!("keyed {i} fills the box: {e}"))
})
.collect();
let started = Instant::now();
let refusal = gate
.admit_within(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone)
.await
.expect_err("the box is over the contention threshold");
let stranger_took = started.elapsed();
assert!(
matches!(refusal, Refusal::KeyedReserve { .. }),
"refused by the reserve, got {refusal:?}"
);
assert!(
stranger_took < wait / 4,
"a stranger met the reserve and waited {stranger_took:?} of a {wait:?} bound; parking it \
is what holds the socket a flood wants"
);
// The control: a KEYED caller against the same full box does park for its
// whole bound, which is what makes the assertion above a statement about
// anonymity rather than about a broken wait.
let started = Instant::now();
gate.admit_within("ci", "team", OpClass::Clone)
.await
.expect_err("the box is full for everyone");
let keyed_took = started.elapsed();
assert!(
keyed_took >= wait,
"the keyed refusal waited only {keyed_took:?} of its {wait:?} bound, so the comparison \
above proves nothing"
);
}
/// **The property, under real concurrency**: 200 strangers hammering a shed box
/// while a keyed caller walks in and out, and the keyed caller is never once
/// made to wait.
///
/// The gate is given a real, non-zero wait, so "not delayed" is a measurement
/// and not a configuration: `admit_within` reports how long it waited, and every
/// keyed admission here must report `Duration::ZERO` — admitted on the *first*
/// ask, never parked, never woken.
///
/// # Seen red
///
/// Against the gate with the reserve wired in but switched off:
///
/// ```text
/// ---- a_keyed_caller_walks_in_while_strangers_hammer_the_gate stdout ----
/// assertion `left == right` failed: keyed admission 0 was PARKED for
/// 5.845651ms behind anonymous load; the reserve exists so that a caller with a
/// key is admitted on the first ask
/// left: 5.845651ms
/// right: 0ns
/// ```
///
/// Five milliseconds is not a disaster, and that is the point: this is a
/// four-slot box with a two-hundred-client flood on it, the smallest
/// demonstration of the shape there is. The number the appliance would show is
/// whatever a real clone takes to finish, because that is what the operator
/// would have been queued behind.
#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
async fn a_keyed_caller_walks_in_while_strangers_hammer_the_gate() {
const STRANGERS: usize = 200;
const KEYED_ROUNDS: usize = 60;
let gate = Arc::new(
Admission::new(Some(4), Some(DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL))
.with_anonymous_ceiling(Some(DEFAULT_MAX_ANONYMOUS_TRANSFERS))
.with_wait(Duration::from_millis(500)),
);
let limit = gate.governed_limit().expect("a limited gate");
let reserve = reserve_of(limit);
let stop = Arc::new(AtomicBool::new(false));
// Counted by the test itself rather than read from the gate: `peak_in_flight`
// is a lock-free signal that `Admission::release` may leave one high, and a
// gate that lied about its own counter must not be able to make this pass.
let anonymous_now = Arc::new(AtomicUsize::new(0));
let anonymous_peak = Arc::new(AtomicUsize::new(0));
let anonymous_admitted = Arc::new(AtomicU64::new(0));
let mut floods = Vec::with_capacity(STRANGERS);
for _ in 0..STRANGERS {
let gate = Arc::clone(&gate);
let stop = Arc::clone(&stop);
let now = Arc::clone(&anonymous_now);
let peak = Arc::clone(&anonymous_peak);
let admitted = Arc::clone(&anonymous_admitted);
floods.push(tokio::spawn(async move {
while !stop.load(Ordering::Relaxed) {
if let Ok((guard, _)) = gate
.admit_within(ANONYMOUS_PRINCIPAL_ID, "team", OpClass::Clone)
.await
{
admitted.fetch_add(1, Ordering::Relaxed);
let inside = now.fetch_add(1, Ordering::AcqRel) + 1;
peak.fetch_max(inside, Ordering::AcqRel);
// Hold it across an await, which is what a transfer does.
tokio::task::yield_now().await;
now.fetch_sub(1, Ordering::AcqRel);
drop(guard);
}
tokio::task::yield_now().await;
}
}));
}
// Let the flood actually reach the gate before the keyed caller asks;
// otherwise this measures an empty box.
for _ in 0..64 {
tokio::task::yield_now().await;
}
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(
anonymous_admitted.load(Ordering::Relaxed) > 0,
"the flood never reached the gate, so nothing below is under load"
);
for round in 0..KEYED_ROUNDS {
let (guard, waited) = gate
.admit_within("alice", "team", OpClass::Clone)
.await
.unwrap_or_else(|refusal| {
panic!(
"keyed admission {round} was REFUSED while strangers held the box: {refusal}"
)
});
assert_eq!(
waited,
Duration::ZERO,
"keyed admission {round} was PARKED for {waited:?} behind anonymous load; the reserve \
exists so that a caller with a key is admitted on the first ask"
);
tokio::task::yield_now().await;
drop(guard);
}
stop.store(true, Ordering::Relaxed);
for flood in floods {
let _ = flood.await;
}
let peak = anonymous_peak.load(Ordering::Relaxed);
assert!(
peak <= limit - reserve,
"{peak} strangers were in flight at once on a {limit}-slot box that reserves {reserve} \
for keyed callers; the reserve is not being enforced under concurrency"
);
let stats = gate.stats();
assert!(
stats.refused_keyed_reserve > 0,
"the reserve never fired, so the flood never reached the threshold and the keyed \
admissions above prove nothing"
);
}
/// The reserve is **derived, not a fourth magic number**, and it is derived from
/// the constant that already means `this box is contended`.
///
/// Pinned as a decision. A reserve that drifted apart from
/// [`CONTENTION_PERCENT`] would leave the box with two different opinions about
/// when it is busy: one that starts making keyed identities share with each
/// other, and one that stops making them share with strangers. Those two have to
/// be the same moment or the ordering between them is an accident.
#[test]
fn the_reserve_is_the_contention_threshold_and_nothing_else() {
for limit in [4usize, 10, 19, 64, 256] {
let gate = Arc::new(
Admission::new(Some(limit), Some(DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL))
.with_anonymous_ceiling(Some(DEFAULT_MAX_ANONYMOUS_TRANSFERS))
.with_wait(Duration::ZERO),
);
let governed = gate.governed_limit().expect("a limited gate");
assert_eq!(
gate.keyed_reserve(),
reserve_of(governed),
"the gate's reserve at a governed limit of {governed} is not the contention threshold"
);
assert!(
gate.keyed_reserve() >= 1,
"however small the box, at least one slot must be keyed-only, or the guarantee is \
empty at exactly the moment it matters"
);
}
}
/// The reserve's sentence must stay **tellable apart** from the other refusals'.
///
/// `installtest`'s `refusals_in_log` sorts refusals out of the server log by the
/// substrings `"concurrent git transfers"` and `"per-principal limit"`, because
/// the log records a refusal by its `Display` — the same sentence the client is
/// handed. A sixth reason that borrowed either phrase would be counted as the
/// wrong one, and an operator would raise the wrong number.
#[test]
fn the_reserve_does_not_borrow_another_refusals_words() {
let reserve = Refusal::KeyedReserve {
occupied: 2,
limit: 4,
reserve: 2,
}
.client_message();
let ceiling = Refusal::AnonymousAtCapacity { limit: 5 }.client_message();
assert!(
!reserve.contains("concurrent git transfers"),
"installtest would count the reserve as a CAPACITY refusal, and the operator would raise \
the concurrency limit: {reserve}"
);
assert!(
!reserve.contains("per-principal limit"),
"installtest would count the reserve as one identity's fairness throttle: {reserve}"
);
assert_ne!(
reserve, ceiling,
"the two anonymous refusals must not be the same sentence: one says `strangers are at \
their cap` and the other says `this box is too busy for strangers`, and they are fixed \
by different knobs"
);
assert!(
reserve.contains("try again"),
"a transient refusal that does not say `try again` reads as permanent: {reserve}"
);
assert_eq!(
Refusal::KeyedReserve {
occupied: 2,
limit: 4,
reserve: 2,
}
.reason(),
"admission-keyed-reserve",
"the machine-readable reason must name the gate that refused; `5ae4a8f` established that \
a refusal firing on a box with free slots may not report itself as `gate-full`"
);
}