crates/mock-upcloud/src/faults.rs
//! **The fault surface: every measured lie, armable on its own.**
//!
//! Two ways to arm. By NAME, in which case the fault fires every time it can —
//! that is the mode a regression test uses, because a test that only sometimes
//! provokes the defect is a test that only sometimes passes. Or from a SEED, in
//! which case the whole run's weather is derived from that one number: which
//! faults are armed, at what rate, and for how long. A storm prints the seed of
//! every distinct failure, and the seed alone replays it.
//!
//! # Sticky faults
//!
//! [`Fault::OutOfStock`] and [`Fault::RevokedCredential`] are STICKY: once they
//! fire they keep firing until disarmed. That is what was measured — on
//! SCALEWAY, not UpCloud (both are kept as generic cloud-provider faults):
//! Scaleway's `fr-par-1` refused `poweron` with `412 out_of_stock` for days,
//! and a revoked Scaleway key answered 0 servers from the moment it was revoked. A fault that fires
//! once and clears would let a retry loop paper over both, which is precisely
//! the bug that shipped.
use crate::rng::SplitMix64;
use std::collections::BTreeSet;
use std::sync::Mutex;
/// One measured behaviour, armable.
///
/// The variants are the mock's whole vocabulary of wrongness: if a defect cannot
/// be named here it cannot be provoked, and the answer is a new variant, not a
/// flag on a handler.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug)]
pub enum Fault {
/// **Behaviour 2.** `GET /1.3/price` drops the socket instead of answering.
/// Not a status: the connection is closed with no reply at all, which is
/// what made monetize's start-time probe report "the credential in
/// UPCLOUD_TOKEN could not be verified" for what was a transport failure.
/// A mock that answered 503 here would never have shown that.
PriceTransportReset,
/// **Behaviours 3 and 37.** `412 out_of_stock`, at BOTH doors that need
/// capacity: `POST /1.3/server/{uuid}/start` (behaviour 3) and
/// `POST /1.3/server` (behaviour 37). Sticky.
///
/// One variant, not two, because what is out of stock is a PLAN IN A ZONE
/// and both calls ask for one. A mock that could be sold out at the
/// poweron and in stock at the create would let a caller pass by knocking
/// on the luckier door.
///
/// **Verified against Scaleway, not UpCloud; kept as a generic
/// cloud-provider fault** (ledger X17, owner ruling 2026-09-21). The only
/// dated measurement of "412 out_of_stock on poweron, for days" is
/// Scaleway's fr-par-1 from 2026-09-09. UpCloud documents its capacity
/// refusal as `409 STORAGE_RESOURCES_UNAVAILABLE`. Relabelled, not removed:
/// it stays in the seeded weather, because removing it re-maps every seed.
OutOfStock,
/// **Behaviour 5.** The credential is revoked: lists answer 200 with ZERO
/// rows, details answer 403. Never a clean 401 anywhere. Sticky.
///
/// **Verified against Scaleway, not UpCloud; kept as a generic
/// cloud-provider fault** (ledger X17, owner ruling 2026-09-21). The only
/// dated measurement of this shape (2026-08-26) is the Scaleway IAM change
/// ("0 servers, 403 on VPC"). A dead UpCloud token is MEASURED as a clean
/// 401: that is [`Fault::DeadToken`]. Relabelled, not removed: it stays in
/// the seeded weather, because removing it re-maps every seed.
RevokedCredential,
/// **Behaviour 14.** The VNC port reported after a stop/start is the one
/// from before it, and stays wrong until `remote_access_enabled` is toggled
/// no→yes. Default ON: it is not an exceptional condition, it is what the
/// API does.
StaleVncPort,
/// **Behaviour 17.** A re-image interrupted mid-install leaves the CD first
/// in the boot order, so the next boot runs the installer again, forever.
InstallerLoop,
/// **A HYPOTHESIS, not a measurement: withhold `created`.**
///
/// This crate had it the other way round and had it WRONG. The header used
/// to claim the API sends no `created` on these rows, so a young volume and
/// a six-month-old orphan were the same row and only a title could tell
/// them apart. That was the brief, and the brief was mistaken.
///
/// MEASURED against the live account 2026-09-20, two volumes, both
/// endpoints, `created` every time:
///
/// ```text
/// 018a7b17-5698-4aa9-b160-f6a6daea9921
/// GET /1.3/storage/private → "created": "2026-09-20T10:35:18Z"
/// GET /1.3/storage/{uuid} → "created": "2026-09-20T10:35:18Z"
/// the Resize Backup's detail → "created": "2026-09-19T21:54:06Z"
/// ```
///
/// So the field is sent, the mock now sends it, and this fault only takes
/// it away on request. It is the same failure this crate has already made
/// once, in its own words: a mock stricter than the provider is worse than
/// one that is laxer, because a lax mock misses a bug and a strict one
/// INVENTS them — the attach refusal manufactured four hundred failures in
/// one storm, and this manufactured a verdict path the account cannot
/// produce.
///
/// Kept, at rate 0 and never in seeded weather, for the same reason
/// [`Fault::CloneSyncsLikeImport`] is kept: the disagreement it provokes between
/// `classify` and `refine` in `private-gunnar-ops` is real and worth a
/// regression test, and a hypothesis must be asked for BY NAME rather than
/// arrive as weather.
WithholdCreatedField,
/// A 503 on any write, the ordinary provider wobble. The one fault that is
/// not interesting on its own and is here so the retry paths are exercised.
WriteUnavailable,
/// **Behaviour 9, sharpened.** The `Resize Backup` a stop-resize-start
/// leaves behind points its `origin` at a volume that is ALREADY GONE, so a
/// cleanup that resolves the origin gets `STORAGE_NOT_FOUND` rather than a
/// parent.
OrphanResizeBackup,
/// **A HYPOTHESIS, by name: the `Resize Backup` carries NO labels.**
///
/// MEASURED 2026-09-20 on the live account, the backup a twin resize left
/// behind DID carry the origin volume's labels (`site=gunnar.rs`,
/// `repo=private-gunnar-ops`, `role=twin`, `volume=twin`), which is what
/// the mock copies by default. The owner's brief for F39 says the opposite
/// — an object UpCloud titled itself carries NEITHER a product title NOR
/// the estate's labels — and whether the provider copies labels on every
/// backup kind, or only on the one that was measured, is not known.
///
/// So both are reachable, and the sweep is proven against both: the default
/// is the measurement, and this fault is the brief. Rate 0, never in seeded
/// weather, asked for by name — the rule [`Fault::WithholdCreatedField`]
/// already follows, for the same reason: a mock stricter than the provider
/// invents verdict paths, so a hypothesis must announce itself.
///
/// What a sweep has left when both the title and the labels are silent is
/// the backup's `origin`, which names the volume it was taken from. That is
/// the recognition path this fault exists to exercise.
ResizeBackupUnlabelled,
/// **The import that does not finish.** The upload completes, every byte is
/// accounted for — and then the import object goes `failed` with an
/// `error_code`/`error_message` pair and the storage lands in `error`
/// instead of `online`. The caller's poll loop has to tell that apart from
/// `syncing`, and a loop that only looks for `online` waits its whole budget
/// for a volume that is never coming.
ImportFailed,
/// **The sync that outlasts the budget.** The caller polls a direct upload
/// with a 1200 s budget; this makes the `syncing` wait 1300 s. A timeout
/// that has never fired is a timeout nobody has read the handler of.
SyncExceedsBudget,
/// **The OLD pessimistic guess about a clone, kept by name only.** A clone
/// waits in `syncing` for 100–130 s, exactly like an import. This was the
/// mock's DEFAULT until lane T13 found the measurement it said did not
/// exist: gunnar `deploy/upcloud/tests/clone_probe.rs`, 2026-09-20 — 728 ms
/// call, `maintenance` → `online` in 47 s, NO `syncing`. The default is now
/// the measurement; this is the guess, at rate 0 (ledger X3). It replaces
/// `clone-skips-sync`, whose optimistic "guess" is now simply the default,
/// and that name no longer parses — loudly, on purpose.
CloneSyncsLikeImport,
/// **The guest reads the hypervisor's UTC clock as local time.** The
/// hypervisor is right; the GUEST is wrong, because it runs gunnar as PID 1
/// with no `systemd-timedated` and no `/etc/adjtime` to establish what the
/// RTC holds. Measured: the appliance was +7 198 668 ms from the front
/// while the front, on the same hypervisor in the same zone, was +468 ms.
///
/// Armed per-server through [`crate::estate::Estate::create_server`], never
/// applied to the clock itself — a mock that skewed its own clock would
/// model the symptom and let a fix that subtracts two hours somewhere pass.
GuestReadsRtcAsLocalTime,
/// **Inbound UDP replies are dropped.** DEFAULT ON, because it is the
/// provider's normal: NTP and DNS-over-UDP simply never answer, which is
/// why `systemd-timesyncd` is useless up there and why `gunnar-clock` takes
/// signed time from the front over TCP and from nowhere else. A mock that
/// let UDP through would let a fix that "just uses NTP" look correct here
/// and fail there.
UdpInboundDropped,
/// **The guest ignores DHCP option 121 and has no route off its own /22.**
/// It comes up with a good address, answers everything sent TO it, and
/// cannot reach the front at all — so its clock sync and its boot narration
/// die OUTBOUND while every inbound probe says the box is healthy. It reads
/// exactly like a two-hour clock bug and is not one. Measured; fixed in
/// gunnar `35ac0c3`.
///
/// Armed per-server at create, like the RTC interpretation: it is a fact
/// about the image, not a coin flipped per packet.
GuestIgnoresDhcpOption121,
/// **A different machine is answering on the name.** The `name` path's SSH
/// host key stops matching the `front` and `direct` paths'. This is the
/// case the three-path check exists for — a re-image mints a new key every
/// time, so `REMOTE HOST IDENTIFICATION HAS CHANGED` cannot be treated as
/// alarming on its own, and the only thing that separates a new machine
/// from a stolen name is three paths agreeing.
HijackedName,
/// A create that succeeds at the provider and times out at the client: the
/// reply is dropped after the object is committed. The idempotency seam's
/// reason to exist — a plugin that retries on this and does not search by
/// label buys twice.
CommitThenDropReply,
/// **The same, at the clone door** (`POST /1.3/storage/{uuid}/clone`): the clone is
/// committed and its reply is lost. A clone takes no labels at the call, so a caller
/// that retries without searching by the clone's title clones twice, and the first
/// clone carries nothing an orphan sweep can find. Asked for by name only: a draw on
/// every clone would re-map every existing seed's weather.
CloneThenDropReply,
/// **Behaviour 76 (MEASURED on the account 2026-10-17, test2's front-keep):** `POST
/// /1.3/server/{uuid}/start` STARTS the server and its reply never arrives ("transport POST
/// /server/{uuid}/start: operation timed out") — ONCE: the fault disarms itself as it fires,
/// so a caller that asks again is answered (409 SERVER_STATE_ILLEGAL: it is started). A
/// caller that takes the lost reply as the end gives a good start up.
StartThenDropReplyOnce,
/// **Behaviour 77 (MEASURED on gunnar.rs 2026-10-18, the twin's keep boot):** the release
/// clone a caller is about to put in a server's cdrom is GONE (the hourly sweep gave it
/// back) — `POST /1.3/server/{uuid}/storage/attach` of a `cdrom` deletes the storage and
/// answers `404 STORAGE_NOT_FOUND`, ONCE (the fault disarms itself). A caller that stopped
/// the server for it and gives up there leaves the box down.
CdromStorageGoneOnce,
/// **Behaviour 78 (MEASURED on gunnar.rs 2026-10-25, main's keep boot):** the DISK start
/// that brings a box back after its keep medium ran is answered, and the server then
/// reads **`error`** for [`crate::clock::Timings::error_state_ms`] — every write on it
/// (`PUT` the boot order, start, stop, a cdrom eject or attach) answering `409
/// SERVER_STATE_ILLEGAL` meanwhile — before the start completes and it reads `started`.
/// ONCE: the fault disarms itself as it fires. A caller that takes `error` as the end fails
/// a keep boot whose medium RAN, and its put-back meets the 409.
StartErrorStateOnce,
/// **Behaviour 35 — the firewall of a server that READS FINE answers 403.**
///
/// Behaviour 1 is the firewall endpoint answering `403
/// ERROR_AUTHENTICATION_FAILED` for a server that is GONE. This is the same
/// body, byte for byte, for a server that is alive and answers its own
/// `GET /1.3/server/{uuid}` with 200 — the shape of a credential scoped
/// without the firewall permission. The two are indistinguishable AT THE
/// FIREWALL ENDPOINT; only asking the server itself tells them apart, which
/// is the whole point of a caller that reads both before it decides.
///
/// INFERRED from the provider's permission-scoped API tokens, not measured
/// on this estate's account: on 2026-09-20 every 403 seen at this endpoint
/// sat beside a 404 on the server, which is behaviour 1. Rate 0 — a shape
/// that has not been measured is asked for BY NAME and never arrives as
/// weather. Sticky, like every other credential property.
FirewallForbidden,
/// **Behaviour 38 — the detach answers `200` and the volume is STILL
/// ATTACHED.**
///
/// The write that reports its own success and did not happen. Not a
/// refusal, not a timeout, not a slow poll: a clean `200` with the server
/// object in the reply, and the device still on `GET /1.3/server/{uuid}`
/// afterwards — so a caller that trusts the status and does not READ BACK
/// goes on to delete a volume that is attached, or leaves a growth half
/// done believing it finished.
///
/// The mock could not express this at all before: `Estate::detach` had four
/// outcomes, three refusals and a removal, and no path that answered `200`
/// without removing the device. So a sweep or a growth written against this
/// mock could never meet the shape, and every green it printed was a green
/// about a mock that could only tell the truth. This is the whole argument
/// for the fault surface — a defect that cannot be named here cannot be
/// provoked, and the answer is a variant.
///
/// **Not sticky**, because it is a property of ONE WRITE and not of the
/// account or the zone: stickiness here would mean no detach could ever
/// succeed again for the run's lifetime, which hides the read-back-and-
/// retry path this fault exists to exercise (contrast
/// [`Fault::OutOfStock`], where the shortage really does outlive the call).
///
/// REPORTED, not measured on this account: rate 0, asked for by name, the
/// same rule [`Fault::WithholdCreatedField`] and [`Fault::CloneSyncsLikeImport`]
/// already follow.
DetachSaysSuccessButStaysAttached,
/// **Behaviour 36 — `GET /1.3/server/{uuid}` answers 404 for a uuid the
/// LIST carries.** The account contradicting itself: the list says the
/// server exists, its own detail says `SERVER_NOT_FOUND`.
///
/// REPORTED 2026-09-20 on t14s: a `plan` against the live account read the
/// state's server uuid as 404 while `GET /1.3/server` showed gunnar-front,
/// gunnar-appliance, gunnar-twin, holger-front and one more server. Whether the uuid
/// the list carried was the SAME uuid the state named was not captured
/// before that box was wiped — so this is a report and not a measurement,
/// and the more likely reading (a stale state naming a server that was
/// re-laid under a new uuid) needs no fault at all: it is the ordinary
/// state machine with a uuid that was never created. Rate 0, by name only,
/// kept so the caller's "the account contradicts itself" refusal has a path
/// that reaches it.
DetailNotFoundForListedServer,
/// **Behaviour 43 — a DEAD token is a clean `401 AUTHENTICATION_FAILED`,
/// everywhere.** MEASURED 2026-09-14 (private-gunnar-ops ROTATION §2.3:
/// `UPCLOUD_TOKEN_HENTOR` and `MONETIZE_UPCLOUD_TOKEN`, both 401). The
/// "0 rows and 403" shape of [`Fault::RevokedCredential`] is the one whose
/// only dated measurement is Scaleway's (ledger X17); this is UpCloud's.
/// Sticky, like every property of a credential. Rate 0: new faults stay out
/// of the seeded weather so existing seeds keep naming the same runs.
DeadToken,
/// **Behaviour 40 — a READ answers `502`.** REPORTED (gunnar `wait.rs`: "the
/// API occasionally returns 502, so a failed poll is not fatal"). The mock
/// only had 503-on-write; a poll loop that dies on its first 502 was never
/// exercised. Rate 0 (see [`Fault::DeadToken`]).
ReadBadGateway,
/// **Behaviour 60 — the guest's kernel cannot hot-plug PCI.** Decided per
/// server at create, like the RTC reading: it is a property of the image.
/// Such a guest never acks the ACPI `_EJ0`, so a virtio attach or detach on
/// a STARTED server answers **`511 HOTPLUG_FAILED`** — MEASURED 2026-09-14
/// on the live appliance (tunnr 6.12.104, no `HOTPLUG_PCI`). A hot-plug
/// kernel needs `HOTPLUG_PCI(_ACPI/_PCIE/_SHPC)`, `PCIEPORTBUS` and
/// **`PCI_MSI=y`** (without MSI `_OSC` refuses OS control; measured under
/// KVM). Disarmed, the guest can hot-plug and the virtio call succeeds,
/// which is what the Ubuntu template and tunnr ≥ 80c3233 do. Rate 0.
GuestKernelLacksHotplug,
/// **Behaviour 56, the stricter reading — a size grow of ANY attached
/// storage is refused `409 STORAGE_ATTACHED`, "must first be detached".**
/// UpCloud's docs contradict themselves (DATA-SET-GROWTH-DESIGN §2): one
/// page says the server must be `stopped`, the other that the storage must
/// be detached. Both refuse a grow under a RUNNING server, which the mock
/// now always does; this fault is the page that also refuses it under a
/// stopped one. Not measured either way: rate 0, by name.
ResizeRequiresDetach,
/// **Behaviour 69 — the REBOOTING medium** (ledger L113): a medium composed
/// WITHOUT `install_then=poweroff` (every korp-installer < 0.1.4; the
/// medium of the 2026-09-14 loop). Its installer start reads **`started`**
/// — MEASURED 2026-09-14: `cdrom/eject` answered 200 on a STARTED box
/// mid-pass — for the whole pass, 900–1100 s (gunnar plan.rs: every
/// measured pass ≥ 900 s; 1000–1100 s live on 2026-09-08). At its end the
/// guest REBOOTS: with the CD still loaded and first it installs again
/// (behaviour 17), otherwise it boots the disk. Disarmed, the medium is the
/// power-off kind (behaviour 46). A property of the MEDIUM, so asked for by
/// name: rate 0.
InstallerReboots,
/// **Behaviour 70's other half — the stop that outlasts the caller's soft
/// window.** MEASURED on the real test2 2026-10-01 (`estate upgrade`, main):
/// a soft stop with UpCloud's own `timeout` of 110 s, and at +120 s the
/// server still read `maintenance` — UpCloud's own transitional state during
/// a stop (its timeout's forced stop, a storage operation) — so the kit's
/// hard stop met `409 SERVER_STATE_ILLEGAL`. Armed, every stop holds the
/// server in `maintenance` for [`crate::clock::Timings::stop_held_ms`] of
/// WALL time from the request, then `stopped`. How long UpCloud holds it is
/// not measured (≥ 10.6 s past the 120 s window): by name, rate 0.
StopHeldInMaintenance,
/// **Behaviour 71 — UpCloud full, at the start.** MEASURED on the REAL
/// gunnar.rs 2026-10-03 (`estate upgrade`, main): `POST /server/start`
/// answered `409 SERVER_RESOURCES_UNAVAILABLE — There are currently no
/// server resources available. Please try again later.` Armed, the next
/// [`crate::clock::Timings::start_fault_n`] starts answer exactly that, then
/// the fault disarms itself. How long UpCloud stays full is not measured: by
/// name, rate 0.
StartNoCapacity,
/// **Behaviour 71's other face — a start that falls back.** MEASURED the
/// same run, the start before that 409: the keep medium's start was
/// accepted, read `maintenance` at once, never `started`, and `stopped`
/// 2:30 later — no machine ran. Armed, the next
/// [`crate::clock::Timings::start_fault_n`] starts are held in `maintenance`
/// for [`crate::clock::Timings::start_fallback_ms`] of WALL time, then
/// `stopped`; nothing boots. By name, rate 0.
StartFallsBack,
/// **Behaviour 78 (MEASURED on test3-holger-rs r6, 2026-10-20) — the utility
/// network's DHCP is LATE on a server's first boot.** The front's eth1 asked at
/// boot, got no lease, and was never asked again (edda D12 fixed the asking): no
/// box ever reached the relay. Armed, a server created while it is armed — only
/// the hostnames [`crate::estate::Estate::dhcp_late_hosts`] names, when it names
/// any — gets NO answer to its utility DISCOVER/REQUEST for
/// [`crate::clock::Timings::dhcp_late_first_boot_ms`] of WALL time from its first
/// ask on its first boot; after that, and on every later boot, the fabric
/// answers as always. Decided at create, a property of that boot: by name,
/// rate 0.
UtilityDhcpLateFirstBoot,
}
/// **Where a fault's difference can be observed** — and therefore what a green
/// against it is worth.
///
/// `--self-check` proves a fault is EXPRESSIBLE: armed and disarmed, the mock
/// answers differently. That is a statement about the mock's own JSON and
/// nothing else. Six of the twenty faults are not statements about JSON at all
/// — they are things a MACHINE does, and against an arithmetic guest their two
/// runs can differ while no machine has ever been involved. `20 faults: 20
/// expressible` was therefore true and empty, which is the always-yes shape
/// this estate has now found nine times: a check that cannot fail, reporting
/// health.
///
/// So every fault declares where its difference lives, the match is EXHAUSTIVE,
/// and the calibration prints the two populations apart instead of summing
/// them.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Reach {
/// The fault lives entirely in the provider's control plane: a status, a
/// missing field, a dropped socket, a row that is there or is not. An
/// in-process probe against [`crate::Estate`] observes the whole of it, and
/// `Expressible` means what it says.
Api,
/// The fault is something a GUEST does, and no JSON reproduces it. Until a
/// real guest boots, an `Expressible` verdict on one of these is a
/// statement about a number.
Guest,
}
impl Reach {
pub fn label(self) -> &'static str {
match self {
Reach::Api => "api",
Reach::Guest => "GUEST",
}
}
}
impl Fault {
pub const ALL: &'static [Fault] = &[
Fault::PriceTransportReset,
Fault::OutOfStock,
Fault::RevokedCredential,
Fault::StaleVncPort,
Fault::InstallerLoop,
Fault::WithholdCreatedField,
Fault::WriteUnavailable,
Fault::OrphanResizeBackup,
Fault::ResizeBackupUnlabelled,
Fault::CommitThenDropReply,
Fault::CloneThenDropReply,
Fault::StartThenDropReplyOnce,
Fault::CdromStorageGoneOnce,
Fault::StartErrorStateOnce,
Fault::ImportFailed,
Fault::SyncExceedsBudget,
Fault::CloneSyncsLikeImport,
Fault::GuestReadsRtcAsLocalTime,
Fault::UdpInboundDropped,
Fault::GuestIgnoresDhcpOption121,
Fault::HijackedName,
Fault::FirewallForbidden,
Fault::DetailNotFoundForListedServer,
Fault::DetachSaysSuccessButStaysAttached,
Fault::DeadToken,
Fault::ReadBadGateway,
Fault::GuestKernelLacksHotplug,
Fault::ResizeRequiresDetach,
Fault::InstallerReboots,
Fault::StopHeldInMaintenance,
Fault::StartNoCapacity,
Fault::StartFallsBack,
Fault::UtilityDhcpLateFirstBoot,
];
pub fn name(self) -> &'static str {
match self {
Fault::PriceTransportReset => "price-transport-reset",
Fault::OutOfStock => "out-of-stock",
Fault::RevokedCredential => "revoked-credential",
Fault::StaleVncPort => "stale-vnc-port",
Fault::InstallerLoop => "installer-loop",
Fault::WithholdCreatedField => "withhold-created-field",
Fault::WriteUnavailable => "write-unavailable",
Fault::OrphanResizeBackup => "orphan-resize-backup",
Fault::ResizeBackupUnlabelled => "resize-backup-unlabelled",
Fault::CommitThenDropReply => "commit-then-drop-reply",
Fault::CloneThenDropReply => "clone-then-drop-reply",
Fault::StartThenDropReplyOnce => "start-then-drop-reply-once",
Fault::CdromStorageGoneOnce => "cdrom-storage-gone-once",
Fault::StartErrorStateOnce => "start-error-state-once",
Fault::ImportFailed => "import-failed",
Fault::SyncExceedsBudget => "sync-exceeds-budget",
Fault::CloneSyncsLikeImport => "clone-syncs-like-import",
Fault::GuestReadsRtcAsLocalTime => "guest-reads-rtc-as-local-time",
Fault::UdpInboundDropped => "udp-inbound-dropped",
Fault::GuestIgnoresDhcpOption121 => "guest-ignores-dhcp-option-121",
Fault::HijackedName => "hijacked-name",
Fault::FirewallForbidden => "firewall-forbidden",
Fault::DetailNotFoundForListedServer => "detail-404-for-listed-server",
Fault::DetachSaysSuccessButStaysAttached => "detach-says-success",
Fault::DeadToken => "dead-token",
Fault::ReadBadGateway => "read-bad-gateway",
Fault::GuestKernelLacksHotplug => "guest-kernel-lacks-hotplug",
Fault::ResizeRequiresDetach => "resize-requires-detach",
Fault::InstallerReboots => "installer-reboots",
Fault::StopHeldInMaintenance => "stop-held-in-maintenance",
Fault::StartNoCapacity => "start-no-capacity",
Fault::StartFallsBack => "start-falls-back",
Fault::UtilityDhcpLateFirstBoot => "utility-dhcp-late-first-boot",
}
}
/// **The one line `--help` prints for this fault.**
///
/// It lives here, beside the variant, and the match is EXHAUSTIVE: a new
/// variant that forgets its line does not compile. That is the whole
/// mechanism. The help text used to be a hand-written list in the binary
/// and it went stale exactly as hand-written lists do — it advertised a
/// fault called `grant-created-field` that has never existed under that
/// name (the real one is `withhold-created-field`, and it does the
/// OPPOSITE), and it named 9 of the 16 faults there were. A person who
/// read it and typed what it said got `no such fault`.
pub fn summary(self) -> &'static str {
match self {
Fault::PriceTransportReset => "GET /1.3/price drops the socket — a transport failure, not a status",
Fault::OutOfStock => "412 out_of_stock at BOTH doors: the create and the poweron",
Fault::RevokedCredential => "0 rows on lists, 403 on details; never a clean 401",
Fault::StaleVncPort => "the console reported after a stop/start is the one from before it",
Fault::InstallerLoop => "the CD stays first in the boot order, so the installer runs again",
Fault::WithholdCreatedField => "the rows carry no `created` (a HYPOTHESIS; the account does send it)",
Fault::WriteUnavailable => "503 on writes, the ordinary provider wobble",
Fault::OrphanResizeBackup => "the Resize Backup's `origin` names a volume that is already gone",
Fault::ResizeBackupUnlabelled => "the Resize Backup carries NO labels (a HYPOTHESIS; the measurement says it does)",
Fault::CommitThenDropReply => "the object is created and the reply never arrives",
Fault::CloneThenDropReply => "a storage clone is made and the clone call's reply never arrives",
Fault::StartThenDropReplyOnce => "a server start is made and its reply never arrives, once",
Fault::CdromStorageGoneOnce => "the storage a cdrom attach names is gone (swept) and the attach answers 404, once",
Fault::StartErrorStateOnce => "a disk start is answered and the server reads `error` for error_state_ms (every write on it 409 SERVER_STATE_ILLEGAL), then `started`, once",
Fault::ImportFailed => "every byte arrives and the import then goes `failed`, the storage `error`",
Fault::SyncExceedsBudget => "the post-upload `syncing` outlasts the caller's 1200 s budget",
Fault::CloneSyncsLikeImport => "a clone waits in `syncing` like an import (the OLD guess; measured: 47 s, no sync)",
Fault::GuestReadsRtcAsLocalTime => "the guest reads the hypervisor's UTC as local time — two hours, decided at create",
Fault::UdpInboundDropped => "inbound UDP replies never come back, so NTP and DNS-over-UDP are silent",
Fault::GuestIgnoresDhcpOption121 => "the guest takes the address and not the routes: outbound dies, inbound looks healthy",
Fault::HijackedName => "a different machine answers the host key on the NAME path",
Fault::FirewallForbidden => "a LIVE server's firewall_rule answers 403, the same body a deleted one's does",
Fault::DetailNotFoundForListedServer => "GET /1.3/server/{uuid} answers 404 for a uuid the LIST carries",
Fault::DetachSaysSuccessButStaysAttached => "the detach answers 200 and the volume stays attached",
Fault::DeadToken => "a dead token: a clean 401 AUTHENTICATION_FAILED on every call",
Fault::ReadBadGateway => "a read answers 502, the gateway wobble a poll must survive",
Fault::GuestKernelLacksHotplug => "the guest cannot hot-plug: a virtio attach/detach on a running server is 511 HOTPLUG_FAILED",
Fault::ResizeRequiresDetach => "a grow of ANY attached storage is 409 STORAGE_ATTACHED (the stricter docs page)",
Fault::InstallerReboots => "the medium REBOOTS after its install: `started` for the 900-1100 s pass, then the CD again or the disk",
Fault::StopHeldInMaintenance => "a stop holds the server in `maintenance` past the caller's soft window (stop_held_ms, wall time); a stop asked there is 409",
Fault::StartNoCapacity => "the next start_fault_n starts are 409 SERVER_RESOURCES_UNAVAILABLE (UpCloud full), then a start runs",
Fault::StartFallsBack => "the next start_fault_n starts read `maintenance` for start_fallback_ms (wall time), then `stopped`, never `started`: nothing ran",
Fault::UtilityDhcpLateFirstBoot => "the utility DHCP stays silent to a server's first boot for dhcp_late_first_boot_ms (wall time), then answers",
}
}
pub fn parse(s: &str) -> Option<Fault> {
Fault::ALL.iter().copied().find(|f| f.name() == s)
}
/// Sticky faults keep firing once they have fired.
pub fn sticky(self) -> bool {
matches!(self, Fault::OutOfStock | Fault::RevokedCredential | Fault::FirewallForbidden | Fault::DeadToken)
}
/// **Where this fault's difference can be observed.** See [`Reach`].
///
/// The eight `Guest` variants (six when this was written, 2026-09-20; the
/// hot-plug kernel and the rebooting medium since), and the clause of a real
/// machine each one needs — the argument for each is in the variant's own doc comment, and
/// the qemu clause is in [`crate::kvm`]:
///
/// * `stale-vnc-port` — a real VNC listener that really moves across a
/// stop/start, so the port the API answers with refuses a connection.
/// Against a number, "the port changed" is the number changing.
/// * `installer-loop` — the medium really first in the boot order, so the
/// installer really runs a second time. Behaviour 17.
/// * `guest-reads-rtc-as-local-time` — `-rtc base=<now+2h>`; the skew is
/// the GUEST's reading of the clock, not a field the API carries.
/// * `udp-inbound-dropped` — a network where the query leaves and no reply
/// returns. `systemd-timesyncd` inside the guest is what experiences it.
/// * `guest-ignores-dhcp-option-121` — the guest takes the address and not
/// the classless static routes. Only a DHCP exchange with a real client
/// stack in it can be right or wrong about option 121.
/// * `detach-says-success` — whether the volume is STILL THERE is a
/// property of the guest's block devices. The API answering 200 is the
/// premise of the fault, not its evidence.
/// * `guest-kernel-lacks-hotplug` — a kernel that never acks the ACPI
/// eject; the 511 is the provider reporting it.
/// * `installer-reboots` — the medium's pass ending in a reboot that
/// installs again (behaviour 69).
///
/// Every other variant is `Api` and that is not a defect: a 503 on a write
/// is a 503 on a write, and an in-process probe sees all of it.
pub fn reach(self) -> Reach {
match self {
Fault::StaleVncPort
| Fault::InstallerLoop
| Fault::GuestReadsRtcAsLocalTime
| Fault::UdpInboundDropped
| Fault::GuestIgnoresDhcpOption121
| Fault::DetachSaysSuccessButStaysAttached
// A kernel without PCI hot-plug never acks `_EJ0`: the 511 is the
// provider reporting what the GUEST did not do.
| Fault::GuestKernelLacksHotplug
// The medium's pass ends in a REBOOT and the guest installs again:
// behaviour 17 again, done by a machine.
| Fault::InstallerReboots
// Only a real DHCP client that must ask again can be right or wrong here.
| Fault::UtilityDhcpLateFirstBoot => Reach::Guest,
Fault::PriceTransportReset
| Fault::OutOfStock
| Fault::RevokedCredential
| Fault::WithholdCreatedField
| Fault::WriteUnavailable
| Fault::OrphanResizeBackup
| Fault::ResizeBackupUnlabelled
| Fault::CommitThenDropReply
| Fault::CloneThenDropReply
| Fault::StartThenDropReplyOnce
| Fault::CdromStorageGoneOnce
| Fault::StartErrorStateOnce
| Fault::ImportFailed
| Fault::SyncExceedsBudget
| Fault::CloneSyncsLikeImport
| Fault::HijackedName
| Fault::FirewallForbidden
| Fault::DetailNotFoundForListedServer
| Fault::DeadToken
| Fault::ReadBadGateway
| Fault::ResizeRequiresDetach
| Fault::StopHeldInMaintenance
// UpCloud full: the provider answers the start, the guest never runs.
| Fault::StartNoCapacity
| Fault::StartFallsBack => Reach::Api,
}
}
/// The rate, in parts per thousand, this fault fires at in seeded weather.
/// These are not guesses dressed as measurements — they are the rates that
/// make a 100 000-run storm produce every signature a few hundred times,
/// which is the only property a storm's rates need. The MEASURED rate of
/// `out_of_stock` at the provider was, for several days, 1000.
pub fn seeded_rate_per_mille(self) -> u64 {
match self {
Fault::PriceTransportReset => 20,
Fault::OutOfStock => 8,
Fault::RevokedCredential => 2,
// Not a fault so much as the weather: on by default everywhere.
Fault::StaleVncPort => 1000,
Fault::InstallerLoop => 15,
// Never in seeded weather: it is a HYPOTHESIS about the provider,
// contradicted by measurement, kept only so the verdict paths that
// would depend on an absent field stay testable.
Fault::WithholdCreatedField => 0,
Fault::WriteUnavailable => 30,
Fault::OrphanResizeBackup => 250,
// Never in seeded weather: the measurement says the labels ARE copied,
// and this is the brief that says they are not. Asked for by name.
Fault::ResizeBackupUnlabelled => 0,
Fault::CommitThenDropReply => 12,
// By name: a new weather fault would re-map every existing seed.
Fault::CloneThenDropReply => 0,
// By name: a new weather fault would re-map every existing seed.
Fault::StartThenDropReplyOnce => 0,
// By name: a new weather fault would re-map every existing seed.
Fault::CdromStorageGoneOnce => 0,
// By name: a new weather fault would re-map every existing seed.
Fault::StartErrorStateOnce => 0,
Fault::ImportFailed => 10,
Fault::SyncExceedsBudget => 5,
// Never in seeded weather: it is a GUESS about the provider, not a
// fault of it, and a guess must be asked for by name.
Fault::CloneSyncsLikeImport => 0,
// An appliance gets it and a front does not, and which one is being
// created is not something a rate can decide — so it fires whenever
// it is armed, and the caller arms it for the guest that has the
// shape. In seeded weather it is on for a quarter of the runs,
// which is roughly the share of created servers that are
// appliances rather than fronts or twins.
Fault::GuestReadsRtcAsLocalTime => 250,
// Not a fault so much as the weather, like the stale VNC port.
Fault::UdpInboundDropped => 1000,
// Like the RTC misreading: a property of the image a server was
// made from, so roughly the share of created servers that are
// appliances.
Fault::GuestIgnoresDhcpOption121 => 250,
// Rare, and it must be: a name that answers a different key is an
// attack, not weather. It is in seeded weather at all only so the
// three-path check is exercised without being asked for.
Fault::HijackedName => 3,
// Both never in seeded weather. One is INFERRED from the provider's
// permission scopes and one is REPORTED off a box that was wiped
// before the uuids could be compared; neither is a measurement, and
// a shape that has not been measured is asked for by name.
Fault::FirewallForbidden => 0,
Fault::DetailNotFoundForListedServer => 0,
// REPORTED, not measured here. A write that lies about its own
// success is the kind of shape a storm should not hand out as
// weather until one real detach has been caught doing it.
Fault::DetachSaysSuccessButStaysAttached => 0,
// Lane T13's four: all by name. A new weather fault would add a
// draw to every call it is checked on and re-map every existing seed.
Fault::DeadToken => 0,
Fault::ReadBadGateway => 0,
Fault::GuestKernelLacksHotplug => 0,
Fault::ResizeRequiresDetach => 0,
// A property of the MEDIUM a run composed, not weather: by name.
Fault::InstallerReboots => 0,
// MEASURED once, its length not: by name.
Fault::StopHeldInMaintenance => 0,
// MEASURED once on the account; how long it lasts, not: by name.
Fault::StartNoCapacity => 0,
Fault::StartFallsBack => 0,
// MEASURED once (test3-holger-rs r6); a new weather fault would re-map every seed.
Fault::UtilityDhcpLateFirstBoot => 0,
}
}
}
/// The armed set plus the run's generator.
pub struct Faults {
armed: Mutex<BTreeSet<Fault>>,
/// Faults that fire probabilistically rather than always, and the stream
/// that decides. `None` = no seeded weather: armed means always.
weather: Option<Mutex<SplitMix64>>,
seed: u64,
fired: Mutex<BTreeSet<Fault>>,
}
impl Faults {
/// Nothing armed but the weather that is always true: [`Fault::StaleVncPort`]
/// is the provider's normal behaviour, not an exception, and a mock that
/// hid it by default would leave the toggle cure untested.
pub fn none() -> Faults {
let mut armed = BTreeSet::new();
armed.insert(Fault::StaleVncPort);
// Inbound UDP does not come back. That is not an exceptional condition
// up there, it is Tuesday — and a mock that hid it by default would let
// every "just use NTP" fix look correct.
armed.insert(Fault::UdpInboundDropped);
Faults {
armed: Mutex::new(armed),
weather: None,
seed: 0,
fired: Mutex::new(BTreeSet::new()),
}
}
/// Nothing at all, not even the provider's normal. For the one test that
/// asserts the mock can be quiet.
pub fn quiet() -> Faults {
Faults {
armed: Mutex::new(BTreeSet::new()),
weather: None,
seed: 0,
fired: Mutex::new(BTreeSet::new()),
}
}
/// Weather derived from one number. Every fault whose
/// [`Fault::seeded_rate_per_mille`] is non-zero is armed, and fires at that
/// rate off a stream derived from the seed and the fault's own name — so
/// adding a tenth fault does not renumber the other nine's decisions, and a
/// seed printed today still replays tomorrow.
pub fn seeded(seed: u64) -> Faults {
let armed = Fault::ALL
.iter()
.copied()
.filter(|f| f.seeded_rate_per_mille() > 0)
.collect();
Faults {
armed: Mutex::new(armed),
weather: Some(Mutex::new(SplitMix64::new(seed))),
seed,
fired: Mutex::new(BTreeSet::new()),
}
}
pub fn seed(&self) -> u64 {
self.seed
}
pub fn arm(&self, f: Fault) {
self.armed.lock().unwrap().insert(f);
}
pub fn disarm(&self, f: Fault) {
self.armed.lock().unwrap().remove(&f);
self.fired.lock().unwrap().remove(&f);
}
pub fn is_armed(&self, f: Fault) -> bool {
self.armed.lock().unwrap().contains(&f)
}
/// Ask whether `f` fires NOW. Deterministic when armed by name; seeded when
/// the run has weather; sticky faults answer `true` forever once they have
/// answered it once.
pub fn fires(&self, f: Fault) -> bool {
if !self.is_armed(f) {
return false;
}
if f.sticky() && self.fired.lock().unwrap().contains(&f) {
return true;
}
let yes = match &self.weather {
None => true,
Some(m) => {
let mut r = m.lock().unwrap();
let draw = r.below(1000);
draw < f.seeded_rate_per_mille()
}
};
if yes && f.sticky() {
self.fired.lock().unwrap().insert(f);
}
yes
}
/// Everything that has fired at least once, for a run's report line.
pub fn fired(&self) -> Vec<Fault> {
self.fired.lock().unwrap().iter().copied().collect()
}
}
impl Default for Faults {
fn default() -> Self {
Faults::none()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn armed_by_name_fires_every_time() {
let f = Faults::none();
f.arm(Fault::WriteUnavailable);
for _ in 0..100 {
assert!(f.fires(Fault::WriteUnavailable));
}
}
#[test]
fn a_disarmed_fault_never_fires() {
let f = Faults::quiet();
for v in Fault::ALL {
assert!(!f.fires(*v), "{}", v.name());
}
}
/// The default is not "no faults" — the stale VNC port is what the provider
/// does on every stop/start, and hiding it by default is how the toggle cure
/// went untested for a month.
#[test]
fn stale_vnc_is_the_default_weather() {
assert!(Faults::none().fires(Fault::StaleVncPort));
}
#[test]
fn out_of_stock_is_sticky() {
let f = Faults::none();
f.arm(Fault::OutOfStock);
assert!(f.fires(Fault::OutOfStock));
assert!(f.fires(Fault::OutOfStock));
f.disarm(Fault::OutOfStock);
assert!(!f.fires(Fault::OutOfStock));
}
/// A seed names a run. Two `Faults` built from the same seed must make the
/// same decisions in the same order, or a printed seed is decoration.
#[test]
fn a_seed_replays() {
let a = Faults::seeded(4242);
let b = Faults::seeded(4242);
let mut da = vec![];
let mut db = vec![];
for _ in 0..500 {
da.push(a.fires(Fault::WriteUnavailable));
db.push(b.fires(Fault::WriteUnavailable));
}
assert_eq!(da, db);
assert!(da.iter().any(|x| *x), "3 % of 500 should fire at least once");
assert!(da.iter().any(|x| !*x), "3 % of 500 should not fire every time");
}
#[test]
fn every_fault_round_trips_its_name() {
for f in Fault::ALL {
assert_eq!(Fault::parse(f.name()), Some(*f));
}
assert_eq!(Fault::parse("no-such-fault"), None);
}
}