deploy/upcloud/src/plan.rs
//! The re-image itself: the steps, in order, with a bounded wait around every
//! one that can hang — and, since 2026-09-14, **a procedure that cannot leave
//! the appliance boot-looping, and that can be finished after a crash.**
//!
//! Read [`crate`]'s header first — the VIRTIO TRAP is stated there and
//! everything in this file follows from it.
//!
//! # What happened on 2026-09-14, and the four measurements that shape this file
//!
//! The verb attached the installer as cdrom + virtio, set `boot_order=cdrom`,
//! started the box — and the laptop's network died. The verb's only exit from
//! that state was "see the installed appliance's SSH banner, then hard-stop,
//! detach both, boot from disk", and nothing was left running to do it.
//! `korp-installer` ends every install with `reboot(RB_AUTOBOOT)`, the cdrom
//! was still first, so the appliance **looped install → reboot → install for
//! two hours**, rewriting its boot disk every pass. Setting `boot_order=disk`
//! over the API did nothing: a guest reboot does not re-read it. Then, against
//! the live box:
//!
//! 1. `POST /server/{uuid}/storage/detach` of the IDE cdrom while started →
//! `409 IDE_HOTPLUG_UNSUPPORTED`.
//! 2. `POST /server/{uuid}/cdrom/eject` while started → `200`, and the loop
//! ENDED: the next guest reboot booted the disk. **The eject is the
//! never-loop primitive.**
//! 3. `POST …/storage/detach` of the virtio payload while started → `511
//! HOTPLUG_FAILED` (the appliance kernel has no hotplug). The virtio seed
//! comes off only on a stopped box.
//! 4. `boot_order` is writable while started and honoured at the next
//! hypervisor start only.
//!
//! # The rule: nothing that can boot an installer stays on a box this verb leaves
//!
//! * The cdrom is **ejected as soon as the installer has demonstrably booted
//! from it** ([`Plan::eject_after_ms`] after the server reads `started`).
//! From that moment a crash of this tool leaves a box whose next reboot,
//! guest or hypervisor, lands on the installed disk. The pass in flight is
//! unaffected: `korp-installer` reads its payload from the virtio copy and
//! never touches the cdrom again (the trap, used in our favour).
//! * `boot_order=disk` goes out in the same breath, for the hypervisor-start
//! half.
//! * The virtio seed stays until the box is stopped — by the installer's own
//! power-off (`korp.install.then=poweroff`, edda ≥ korp-installer 0.1.4),
//! or by a later `--finish`. A seed that cannot come off yet is REPORTED, by
//! address and uuid, never silently left.
//!
//! # "Demonstrably booted" — why a settle after `started`, and not a signal
//!
//! There is no signal to read: the console shows pixels, not text; the
//! installer brings no network up; the API says `started` and nothing more.
//! What IS measurable is time since `started`: SeaBIOS and the El Torito
//! loader pull `vmlinuz` + `initrd` off the cdrom in the first seconds, and
//! after that the medium is never read again. Every measured pass on this box
//! lasted ≥ 900 s. So at 120 s the pass is in its first fifth, the kernel has
//! long taken over, and the end of the pass — the reboot that would loop — is
//! thirteen minutes away. And the wrong side of the rule is the safe side: an
//! eject that lands before the loader finished makes the installer fail to
//! boot, SeaBIOS falls to the disk, the OLD appliance comes up untouched, and
//! GATE-IDENTITY refuses by name. A wasted run — never a loop, never a wipe.
//!
//! # `--finish`: one loop, for every state a crash can leave
//!
//! Both doors — the full run after its `start`, and `--finish` on a box in
//! any state — end in [`finish_loop`], which reads the server ONCE per round
//! and acts on what it sees: `maintenance` waits; `stopped` detaches every
//! installer device, sets `disk`, starts; `started` sets `disk`, ejects a
//! loaded cdrom, and waits for the pass to end (the box powers off, or the
//! installed appliance answers SSH). A pass that never ends inside its budget
//! is stopped (soft with a timeout, so a hard stop follows) and the loop goes
//! round again. Every round is a step in the [`Ledger`], so the receipt says
//! how far a crashed run got.
//!
//! # It is destructive, and the guards say so
//!
//! The full run **destroys a live appliance**. Three things stand in front of
//! that: [`Plan::confirmed`] (`--confirm-destroy`), the **title check** (the
//! uuid is asked what it IS before anything is stopped), and the **staleness
//! gate** plus the **upload checksum**. `--finish` is not destructive — it
//! uploads nothing and boots what is on the disk — and it refuses BY NAME to
//! finish media carrying another ISO's digest unless `--allow-foreign-media`.
use crate::api::{BootOrder, Client, Device, ServerView, StopKind};
use crate::banner::{self, BannerProbe};
use crate::console::{self, Console};
use crate::front::{self, Front, FrontRefresh, Remote};
use crate::kept;
use crate::machine::{attach_for, Attach};
use crate::token;
use crate::upload::{verify_upload, Iso};
use crate::wait::{wait_for, Budget, Clock, Progress};
use upcloud_api::UpCloudApi;
/// One of the two media the ISO becomes.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Medium {
pub uuid: String,
pub title: String,
/// An existing storage of the same title was reused rather than uploaded.
pub adopted: bool,
}
/// Did GATE-IDENTITY run, and what did it say?
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Gate {
/// No `--expect-version` was given. A skip that is printed and recorded, not
/// a silence.
Skipped,
Passed(String),
}
/// **What the install wait saw, judged AFTER the banner was read.**
///
/// The install wait's budget expiring is not a verdict — this box has come up
/// past it and served perfectly (2026-09-08, twice) — so it is recorded as a
/// MEASUREMENT against the banner that did arrive, never as a warning beside
/// it. The only failure is no banner at the verify, which never reaches this
/// type because there is no [`Outcome`] at all.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum InstallWait {
/// The banner appeared inside the budget.
InTime { budget_ms: u64, took_ms: u64 },
/// The budget ran out, the loop stopped the box and booted it from disk,
/// and the appliance answered anyway. `past_budget_ms` is measured from
/// the moment the budget expired to the moment the banner was READ.
Late {
budget_ms: u64,
past_budget_ms: u64,
timeout: String,
},
}
impl InstallWait {
/// One sentence for the summary and the receipt.
pub fn verdict(&self) -> String {
match self {
InstallWait::InTime { budget_ms, took_ms } => format!(
"the appliance answered after {} s, inside the {} s install budget",
took_ms / 1000,
budget_ms / 1000
),
InstallWait::Late {
budget_ms,
past_budget_ms,
..
} => format!(
"late — the banner was read {} s past the {} s install budget. The install \
worked; the budget was too tight. Pass --install-wait-min above {} min next \
time",
past_budget_ms / 1000,
budget_ms / 1000,
(budget_ms + past_budget_ms).div_ceil(60_000)
),
}
}
pub fn was_late(&self) -> bool {
matches!(self, InstallWait::Late { .. })
}
}
// ── the ledger: the receipt is written at every step ────────────────────────
/// One step the procedure took, stamped with the run's clock.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct StepRecord {
pub at_ms: u64,
pub step: String,
pub detail: String,
}
/// **The receipt-in-progress.** Every step is pushed here and the sink is
/// called with the whole list at once, so a crash — a dead network, a killed
/// terminal — leaves a file that says how far this run got, rather than the
/// previous run's receipt, which is what 2026-09-14 left behind.
pub struct Ledger {
pub steps: Vec<StepRecord>,
sink: Box<dyn FnMut(&[StepRecord])>,
/// How many times the sink ran — the proof that it ran at EVERY step.
pub sink_calls: u32,
}
impl Ledger {
pub fn new(sink: impl FnMut(&[StepRecord]) + 'static) -> Ledger {
Ledger {
steps: Vec::new(),
sink: Box::new(sink),
sink_calls: 0,
}
}
/// A ledger that records and writes nowhere — for tests and dry runs.
pub fn silent() -> Ledger {
Ledger::new(|_| {})
}
pub fn mark(&mut self, at_ms: u64, step: &str, detail: impl Into<String>) {
self.steps.push(StepRecord {
at_ms,
step: step.to_string(),
detail: detail.into(),
});
self.sink_calls += 1;
(self.sink)(&self.steps);
}
pub fn has(&self, step: &str) -> bool {
self.steps.iter().any(|s| s.step == step)
}
}
#[derive(Clone, Debug)]
pub struct Outcome {
/// This ISO's media in the account, if any. `None` on a `--finish` whose
/// seeds were already given back by hand.
pub cdrom: Option<Medium>,
pub virtio: Option<Medium>,
/// Was the cdrom medium ejected by this run?
pub ejected: bool,
/// The device addresses this run took off.
pub detached: Vec<String>,
/// Installer devices still on the box when this run ended — `(address,
/// title)`. A virtio seed on a started box: it comes off at the next stop.
pub left_attached: Vec<(String, String)>,
/// The seed storages given back to the account.
pub deleted: Vec<String>,
pub banner: String,
/// **The SSH host key the re-imaged box presented, `SHA256:…`** — or why
/// no key was read.
///
/// The banner beside it says which BUILD answered; this says which KEY
/// answered, and only the second one can ever make a `known_hosts` line
/// mean something. `known_hosts` keeps no provenance: once written, a line
/// scanned off `:2222` by `ssh-keyscan` and a line taken from an install
/// record are indistinguishable for ever. So the install writes the key
/// down itself, here, at the moment it had the box on the wire.
pub host_key: Result<banner::HostKey, String>,
/// The address the host key was read from — `verify_addr`, recorded beside
/// the key because a fingerprint with no address names nothing.
pub host_key_addr: String,
pub install_wait: InstallWait,
pub front: FrontRefresh,
pub build_reference: Option<String>,
pub gate: Gate,
pub console: Option<Console>,
/// `true` when this run took the `--finish` door.
pub finished: bool,
/// Things that were survivable but that an operator must read.
pub warnings: Vec<String>,
pub took_ms: u64,
}
/// Everything the procedure needs, resolved before a byte moves.
#[derive(Clone, Debug)]
pub struct Plan {
pub server_uuid: String,
/// What the operator says that uuid is. Checked against the API.
pub server_title: String,
/// **The roster machine this is a deploy to** — normally
/// [`crate::machine::MACHINE`]; a field so a test can vary it.
pub machine: String,
pub zone: String,
/// **The UpCloud API base this run speaks to** (`Endpoint::base_for_display`:
/// the account's `https://api.upcloud.com/1.3`, or a mock's loopback base).
/// Part of every upload record's key ([`kept::key`]): mock uuids repeat
/// across mocks, so a record is only ever read back at its own endpoint.
pub api_base: String,
pub iso: Iso,
/// The storage-title stem (`gunnar-installer-open`); with the ISO's digest
/// and a face suffix it is a title, and it is how a device on the server
/// is recognised as an installer medium — ours or another ISO's.
pub stem: String,
pub title_cdrom: String,
pub title_virtio: String,
/// `host:port` — where the banner is read from. A site fact.
pub verify_addr: String,
pub expect_version: Option<String>,
/// `--expect-role`: with `--verify-control`, the role the box must answer
/// (`GATE-ROLE`). `None` = not asked.
pub expect_role: Option<String>,
pub confirmed: bool,
/// Stop after the read-only checks, having changed nothing.
pub dry_run: bool,
/// **`--finish`**: upload nothing; end whatever a previous run left — a
/// loaded cdrom, a `cdrom` boot order, a stopped box, seeds — and verify.
pub finish: bool,
/// `--finish` on media that carry ANOTHER ISO's digest. Off by default: a
/// medium nobody gated is not one this verb boots a box from.
pub allow_foreign_media: bool,
/// How long after `started` the cdrom is ejected. See the module header.
pub eject_after_ms: u64,
/// **`--medium-powers-off`**: the installer was composed with
/// `korp.install.then=poweroff` (korp-installer ≥ 0.1.5), so a pass ENDS in
/// `stopped` and can never loop. Then there is nothing to settle for and
/// nothing to eject mid-pass: the tool waits for `stopped`, detaches both
/// media and boots the disk. The fixed settle was 120 s on its own — twice
/// the owner's 60 s install bar (2026-09-15).
pub medium_powers_off: bool,
/// **One upload, not two** — the medium reads its own CD (korp-installer
/// ≥ 0.1.14 AND a donor kernel with tunnr 939dbe3's CD block), as its
/// `.medium.json` record says ([`crate::medium`]), AND it powers off after
/// INSTALL-OK. The second condition is load-bearing: without it the loop
/// ejects the cdrom mid-pass, and a cd-only installer would lose the very
/// medium it reads its payload from. Decided from the medium, never a flag.
pub cd_only: bool,
pub soft_stop_timeout_s: u64,
pub storage_online: Budget,
pub stopping: Budget,
/// How long one installer pass may take: from the eject to the box
/// powering off or the installed appliance answering SSH.
pub install: Budget,
pub verify: Budget,
/// Refuse to stop a box whose pass never ended inside the budget, instead
/// of stopping it and booting the disk.
pub strict_install_wait: bool,
pub console_password: Option<String>,
/// **The front VM whose services are restarted** once the appliance is
/// proved serving. A site fact; required even when the restart is skipped.
pub front: Front,
pub front_restart: bool,
/// **Upload once per sha256.** `true` (the default) leaves the seed
/// storages on the account so the next run of the SAME bytes adopts them
/// instead of uploading them again.
///
/// MEASURED 2026-09-22/23 over five go-live attempts: the identical ISO
/// went up on every one, at 165 s+ per attempt for the upload plus the
/// storage's `syncing` wait. The adoption gate was already here; what
/// defeated it was this run deleting the media on its way out. Keeping
/// them costs ~1 GiB of storage per face, billed until a run with
/// `--give-back-medium` (or the UpCloud console) gives them back.
///
/// Reuse is never on the title alone — see [`crate::kept`].
pub keep_medium: bool,
/// **The stop before the install is HARD.** `true` (the default) because
/// the disk a polite shutdown would settle is re-laid by korp-installer
/// three steps later. MEASURED 2026-09-22/23: the soft stop cost ~158 s
/// against a 300 s budget and bought a clean unmount of bytes that were
/// overwritten ninety seconds afterwards.
///
/// It governs ONLY that one stop. The mid-pass stops in the install loop
/// stay soft whatever this says, because a half-written boot disk is
/// exactly what is at risk there.
///
/// `--soft-stop-before-install` is the door back, for a run that wants the
/// guest asked first — the appliance's data volumes are separate storages
/// the installer never touches, and a hard stop is a power cut to their
/// xfs (replayed on the next mount, not lost).
pub hard_stop_before_install: bool,
/// **The previous run's receipt, if one is on disk** — its `progress`
/// steps, for the one situation only a receipt can tell apart from a fresh
/// box: `stopped`, no loaded medium, boot order `disk`, and the last run's
/// installer pass ENDED without the box ever being started from its disk.
/// See [`situation`] and [`Situation::Installed`]. Empty when there is no
/// receipt, or it names another server or another ISO.
pub previous: Vec<StepRecord>,
}
/// The two storage titles for an ISO — one per FACE the machine's attach
/// strategy asks for ([`crate::machine::Attach::faces`]).
///
/// The digest is in the name **on purpose**: a storage found under that name
/// IS this ISO, so adopting it is sound, and a device on the server titled
/// with another digest is another ISO's — see [`classify`].
pub fn titles(stem: &str, short_sha: &str, attach: Attach) -> (String, String) {
let [first, second] = attach.faces();
(
format!("{stem}-{short_sha}-{}", first.suffix),
format!("{stem}-{short_sha}-{}", second.suffix),
)
}
/// `<stem>-<sha8>-<face>` → `(sha8, face)`, or `None` for anything that is
/// not an installer-medium title of this stem (the boot disk, a data disk).
pub fn installer_title(stem: &str, title: &str) -> Option<(String, String)> {
let rest = title.strip_prefix(stem)?.strip_prefix('-')?;
let (sha8, face) = rest.split_once('-')?;
if sha8.len() != 8 || !sha8.bytes().all(|b| b.is_ascii_hexdigit()) {
return None;
}
if face != "cdrom" && face != "virtio" {
return None;
}
Some((sha8.to_string(), face.to_string()))
}
// ── what is on the box ──────────────────────────────────────────────────────
/// The installer devices on the server, sorted by what they are.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct InstallerMedia {
/// This ISO's cdrom face, loaded.
pub cdrom: Option<Device>,
/// This ISO's virtio face.
pub virtio: Option<Device>,
/// A cdrom device with nothing in it — ejected. Boots nothing; comes off
/// at the next stop.
pub empty_cdrom: Option<Device>,
/// Installer media carrying ANOTHER digest, and any loaded cdrom that is
/// not ours at all.
pub foreign: Vec<Device>,
}
impl InstallerMedia {
/// Every installer device, in detach order: the loaded cdrom first.
pub fn all(&self) -> Vec<Device> {
let mut v = Vec::new();
v.extend(self.cdrom.iter().cloned());
v.extend(self.foreign.iter().cloned());
v.extend(self.empty_cdrom.iter().cloned());
v.extend(self.virtio.iter().cloned());
v
}
/// Is a cdrom with a medium in it on the box — ours or not?
pub fn loaded_cdrom(&self) -> Option<&Device> {
self.cdrom
.as_ref()
.or_else(|| self.foreign.iter().find(|d| d.is_loaded_cdrom()))
}
pub fn is_empty(&self) -> bool {
self.cdrom.is_none()
&& self.virtio.is_none()
&& self.empty_cdrom.is_none()
&& self.foreign.is_empty()
}
}
/// Sort the server's devices. The boot disk and the data disks are neither
/// named nor touched: only a cdrom, or a disk titled as an installer face,
/// is an installer device.
pub fn classify(devices: &[Device], plan: &Plan) -> InstallerMedia {
let mut m = InstallerMedia::default();
for d in devices {
if d.title == plan.title_cdrom && d.is_loaded_cdrom() {
m.cdrom = Some(d.clone());
} else if d.title == plan.title_virtio {
m.virtio = Some(d.clone());
} else if d.kind == "cdrom" && d.storage.is_empty() {
m.empty_cdrom = Some(d.clone());
} else if d.kind == "cdrom" || installer_title(&plan.stem, &d.title).is_some() {
m.foreign.push(d.clone());
}
}
m
}
/// What the box is in, judged from one read.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Situation {
/// Nothing can boot an installer: no loaded cdrom, boot order `disk`.
/// `leftovers` are seeds and empty devices a full run takes off after
/// its own stop.
Fresh { leftovers: Vec<Device> },
/// A loaded installer cdrom is on the box and/or the boot order is
/// `cdrom`: a run is in flight, or died. Only `--finish` may proceed.
MidRun {
state: String,
boots_cdrom: bool,
loaded_cdrom: Option<Device>,
},
/// **The install is on the disk and the box was never started from it.**
/// The box reads exactly like `Fresh` — `stopped`, no loaded medium, boot
/// order `disk` — and only the previous run's receipt tells them apart:
/// its installer pass ENDED (`pass-ended`) and no `start … from disk`
/// followed. MEASURED 2026-09-23 18:51: the run died at `detach`, 431 s
/// in, with the installation complete; the retry read `Fresh`, adopted the
/// media and went to stop a box that was already stopped
/// (`409 SERVER_STATE_ILLEGAL`). A second install on top of a finished one
/// is the other way that retry could have gone. Neither is right: the
/// run resumes at the start from disk, uploading and attaching nothing.
/// `leftovers` are any devices the dead run had not yet taken off.
Installed {
pass_ended: String,
leftovers: Vec<Device>,
},
}
/// Does a receipt's progress say the installer pass ENDED and the box was
/// never started from its disk afterwards? The `start` step carries two
/// details — `from the installer cdrom` and `from disk, …` — and it is the
/// second that closes the story.
pub fn receipt_says_installed(previous: &[StepRecord]) -> Option<String> {
let last_pass_end = previous.iter().rposition(|s| s.step == "pass-ended")?;
let started_from_disk_after = previous[last_pass_end..]
.iter()
.any(|s| s.step == "start" && s.detail.starts_with("from disk"));
if started_from_disk_after {
return None;
}
Some(previous[last_pass_end].detail.clone())
}
/// **The previous receipt's steps, if it is this box's and this ISO's.** A
/// receipt for another server or another digest is no evidence about this
/// run and reads as no receipt at all — with the reason, so the log says why
/// a resume was not taken.
pub fn previous_steps(
receipt: &serde_json::Value,
server_uuid: &str,
sha256: &str,
) -> Result<Vec<StepRecord>, String> {
let named_server = receipt["server"]["uuid"].as_str().unwrap_or("");
if named_server != server_uuid {
return Err(format!(
"the receipt names server {named_server:?}, this run is {server_uuid:?}"
));
}
let named_sha = receipt["iso"]["sha256"].as_str().unwrap_or("");
if named_sha != sha256 {
return Err(format!(
"the receipt names sha256 {named_sha:?}, this ISO is {sha256:?}"
));
}
let steps = receipt["progress"]
.as_array()
.map(|a| {
a.iter()
.map(|v| StepRecord {
at_ms: v["at_ms"].as_u64().unwrap_or(0),
step: v["step"].as_str().unwrap_or("").to_string(),
detail: v["detail"].as_str().unwrap_or("").to_string(),
})
.collect()
})
.unwrap_or_default();
Ok(steps)
}
pub fn situation(view: &ServerView, media: &InstallerMedia, previous: &[StepRecord]) -> Situation {
match media.loaded_cdrom() {
None if !view.boots_cdrom() => {
if view.state == "stopped" {
if let Some(pass_ended) = receipt_says_installed(previous) {
return Situation::Installed {
pass_ended,
leftovers: media.all(),
};
}
}
Situation::Fresh {
leftovers: media.all(),
}
}
loaded => Situation::MidRun {
state: view.state.clone(),
boots_cdrom: view.boots_cdrom(),
loaded_cdrom: loaded.cloned(),
},
}
}
/// One line an operator reads.
pub fn describe(view: &ServerView, media: &InstallerMedia) -> String {
let dev = |d: &Device| format!("{} {} {:?}", d.address, d.kind, d.title);
let mut parts = vec![format!(
"state {} · boot_order {}",
view.state,
if view.boot_order.is_empty() {
"?"
} else {
&view.boot_order
}
)];
if let Some(d) = &media.cdrom {
parts.push(format!("cdrom LOADED {}", dev(d)));
}
if let Some(d) = &media.empty_cdrom {
parts.push(format!("cdrom EMPTY {}", d.address));
}
if let Some(d) = &media.virtio {
parts.push(format!("virtio seed {}", dev(d)));
}
for d in &media.foreign {
parts.push(format!("FOREIGN {}", dev(d)));
}
if media.is_empty() {
parts.push("no installer media".to_string());
}
parts.join(" · ")
}
/// The refusal for media that carry another ISO's digest.
fn foreign_refusal(plan: &Plan, media: &InstallerMedia) -> Result<(), String> {
if media.foreign.is_empty() || plan.allow_foreign_media {
return Ok(());
}
Err(format!(
"REFUSED: the server carries installer media that are NOT this ISO ({}): {}. This \
ISO's digest is {} and its titles are {:?} / {:?}. Finishing a run on a medium \
nobody gated boots the appliance from something this verb cannot vouch for. Pass \
--allow-foreign-media to eject and detach them anyway (they are never deleted), \
or give --iso the file those titles name.",
plan.iso.short_sha(),
media
.foreign
.iter()
.map(|d| format!("{} {} {:?}", d.address, d.kind, d.title))
.collect::<Vec<_>>()
.join(", "),
plan.iso.short_sha(),
plan.title_cdrom,
plan.title_virtio
))
}
// ── the upload ──────────────────────────────────────────────────────────────
/// **The media this run creates, in order** — the cdrom always; the virtio copy
/// only when the medium cannot read its own CD ([`Plan::cd_only`]). ONE list,
/// read by the dry run's preview and by the real path alike, so the preview
/// cannot promise a second upload the run will not make (T15's TD8U: the dry
/// run printed "…-cdrom and …-virtio" for a cd-only medium).
pub fn media_titles(plan: &Plan) -> Vec<&str> {
if plan.cd_only {
vec![plan.title_cdrom.as_str()]
} else {
vec![plan.title_cdrom.as_str(), plan.title_virtio.as_str()]
}
}
/// The one call that does not go to the API host. The URL is itself the
/// credential, so the API bearer is NOT sent with it — and it must belong to
/// the cloud this run is aimed at (`upcloud-api` refuses one that does not).
pub fn upload_iso<A: UpCloudApi>(
client: &Client<A>,
url: &str,
iso: &Iso,
) -> Result<serde_json::Value, String> {
Client::<A>::must(
&format!("PUT {} (the ISO)", token::redact_url(url)),
client.api.upload_direct(url, &iso.path),
)
}
/// One copy of the medium, **started but not waited for**: adopted if the
/// account already carries it, else created, imported, uploaded and verified.
///
/// # Why this function no longer waits
///
/// It used to end with `wait_for(... storage online ...)`, and because the
/// procedure calls it once per FACE, that wait happened TWICE, one after the
/// other. MEASURED on the live estate 2026-09-20: the `PUT` of the 43 485 184
/// bytes took **5 s** and the storage then sat in state `syncing` for
/// **100–130 s** before it turned `online` — twice, for the same bytes, on
/// every re-image. Roughly four minutes of a bring-up spent waiting for
/// UpCloud to provision two copies of one file, SERIALLY, for no reason except
/// the shape of this function.
///
/// The import object and the storage keep two different clocks, which is why
/// nobody caught it by reading the upload: the import reported `completed` at
/// 09:36:21Z, five seconds after it was created, while the poller still read
/// the STORAGE as `syncing` at 65 s and again at 97 s. The upload was never the
/// cost.
///
/// The two syncs are independent — UpCloud provisions each volume on its own —
/// so they never had to be serial. Both media are STARTED here, and
/// [`wait_media_online`] then waits for all of them in ONE loop. Two × ~110 s
/// becomes ~110 s, and the guarantee is untouched: the digest is still verified
/// against the local ISO, per medium, before anything waits or attaches.
fn start_medium<A: UpCloudApi>(
client: &Client<A>,
plan: &Plan,
title: &str,
existing: &[(String, String, String)],
ledger: &std::collections::BTreeMap<String, kept::Kept>,
log: &mut dyn FnMut(&str),
) -> Result<Medium, String> {
if let Some((uuid, _, state)) = existing.iter().find(|(_, t, _)| t == title) {
if state == "online" {
// The title carries eight characters of the digest. That is a
// label. The PROOF is the full sha256 recorded when this storage's
// upload verified — and a record that disagrees stops the
// adoption rather than warning about it.
match kept::judge(
ledger,
&plan.api_base,
&plan.zone,
title,
uuid,
&plan.iso.sha256,
) {
kept::Verdict::Refused(why) => return Err(why),
kept::Verdict::Proven => log(&format!(
" adopted {uuid} — {title:?} is on the account and its FULL sha256 {} \
was proven against UpCloud's own reply when it was uploaded; nothing is \
uploaded (saving the 165 s+ this took on every attempt before)",
plan.iso.sha256
)),
kept::Verdict::Unrecorded => log(&format!(
" adopted {uuid} — a storage already carries the title {title:?}, and the \
title carries this ISO's digest, so it IS this image (no upload record on \
this box, so the title is all there is to go on)"
)),
}
return Ok(Medium {
uuid: uuid.clone(),
title: title.to_string(),
adopted: true,
});
}
// Not online: it is this ISO by name and it cannot be attached. Until
// 2026-09-23 this returned an error telling the operator to delete it
// in the console — a hand step on every retry after a run that died
// mid-upload. It is given back here instead, by name, and a fresh
// seed is made below. A delete the cloud refuses (a storage still
// `syncing` or `maintenance` — behaviour 59) is the one case that
// stays an error, with the refusal in it.
match client.delete_storage(uuid) {
Ok(()) => log(&format!(
" deleted {uuid} — a storage titled {title:?} was in state {state:?}, not \
online; it cannot be attached, so it is given back and a fresh seed is made"
)),
Err(e) => {
return Err(format!(
"a storage titled {title:?} exists ({uuid}) and is in state {state:?}, not \
online. It is this ISO by name but it cannot be attached, and deleting it \
was refused: {e}. Wait for it to settle, delete it in the UpCloud console, \
or pass a different --title-prefix."
))
}
}
}
log(&format!(
" POST /storage — a {} GiB seed titled {title:?}",
crate::api::IMPORT_SEED_SIZE_GIB
));
let uuid = client.create_storage(title, &plan.zone)?;
log(&format!(" POST /storage/{uuid}/import — direct_upload"));
let url = client.direct_upload_url(&uuid)?;
log(&format!(
" PUT {} — {} bytes",
token::redact_url(&url),
plan.iso.len
));
let reply = upload_iso(client, &url, &plan.iso)?;
verify_upload(&plan.iso, &reply)?;
log(&format!(
" upload verified: sha256 {} matches the local ISO",
plan.iso.sha256
));
// Strictly AFTER the digest was compared: what is written down is the
// proof, not the intention. Best-effort — a ledger that could not be
// written costs the next run an upload, and failing here would cost this
// run the install it has already paid for.
let led = kept::ledger_path(&plan.iso.path);
match kept::record(
&led,
&plan.api_base,
&plan.zone,
title,
&uuid,
&plan.iso.sha256,
) {
Ok(()) => log(&format!(
" recorded in {} — the next run of these bytes adopts {uuid} instead of \
uploading them again",
led.display()
)),
Err(e) => log(&format!(
" ⚠ the upload record could not be written ({e}); this run is unaffected, the \
next one re-uploads"
)),
}
log(&format!(
" {uuid} is provisioning — it is NOT waited for here; see wait_media_online"
));
Ok(Medium {
uuid,
title: title.to_string(),
adopted: false,
})
}
/// **Every medium that is still `syncing`, waited for in ONE loop.**
///
/// This is the whole of the fix described in [`start_medium`]: the waits are
/// concurrent because the SYNCS are concurrent — UpCloud began provisioning
/// each volume the moment its import finished, and polling them one after the
/// other only ever hid the second one's progress behind the first one's.
///
/// It is a single-threaded round-robin and not a thread per medium on purpose.
/// There is no CPU work here to saturate and nothing to split: the cost being
/// removed is *waiting*, and one loop that asks two questions per round removes
/// all of it. A thread pool would add a runtime, a join and two more ways for a
/// re-image to hang, and buy nothing a `for` loop does not already buy.
///
/// An ADOPTED medium is already `online` — the account was read before anything
/// was created and the adoption gate refuses any state but `online` — so it is
/// not polled. Waiting for it would be a poll that cannot fail, and the day one
/// of those reports health is the day it lies.
///
/// The budget is the SAME `storage_online` budget each medium used to get on its
/// own. That is deliberate and it is a tightening: two media that between them
/// used to be allowed 2 × 600 s now share 600 s of wall time. They are
/// provisioned in parallel by the platform, so the honest budget is one of
/// them, and a run where one medium really needs longer than a lone medium ever
/// did is a fault that should be reported rather than absorbed.
fn wait_media_online<A: UpCloudApi, C: Clock>(
client: &Client<A>,
clock: &C,
plan: &Plan,
media: &[Medium],
log: &mut dyn FnMut(&str),
) -> Result<(), String> {
let pending: Vec<Medium> = media.iter().filter(|m| !m.adopted).cloned().collect();
if pending.is_empty() {
log(" nothing to wait for: every face was adopted, and an adopted storage is online by the gate that adopted it");
return Ok(());
}
let names: Vec<&str> = pending.iter().map(|m| m.title.as_str()).collect();
log(&format!(
" {} storage(s) provisioning at once — {}. They sync in parallel at UpCloud, so \
this is ONE wait and not one per medium.",
pending.len(),
names.join(" + ")
));
wait_for(
clock,
&format!("{} to be online", names.join(" and ")),
plan.storage_online,
log,
|| {
let mut still: Vec<String> = Vec::new();
for m in &pending {
let s = client.storage_state(&m.uuid)?;
if s != "online" {
still.push(format!("{} {s:?}", m.title));
}
}
if still.is_empty() {
Ok(Progress::Done)
} else {
Ok(Progress::Waiting(still.join(", ")))
}
},
)?;
Ok(())
}
/// This ISO's media as the account holds them, without uploading — the
/// `--finish` way of knowing which seeds to give back.
fn media_in_account(plan: &Plan, existing: &[(String, String, String)]) -> [Option<Medium>; 2] {
let find = |title: &str| {
existing
.iter()
.find(|(_, t, _)| t == title)
.map(|(uuid, t, _)| Medium {
uuid: uuid.clone(),
title: t.clone(),
adopted: true,
})
};
[find(&plan.title_cdrom), find(&plan.title_virtio)]
}
fn wait_for_state<A: UpCloudApi, C: Clock>(
client: &Client<A>,
clock: &C,
plan: &Plan,
want: &str,
log: &mut dyn FnMut(&str),
) -> Result<(), String> {
wait_for(
clock,
&format!("the server to be {want}"),
plan.stopping,
log,
|| match client.server_state(&plan.server_uuid)? {
s if s == want => Ok(Progress::Done),
s => Ok(Progress::Waiting(format!("state {s:?}"))),
},
)
.map(|_| ())
}
/// The wait right after the installer's start, and what it saw.
///
/// A medium that reboots is read `started` while it runs, so that is what the
/// wait is for. **A medium that powers off may never be read `started` at
/// all.** MEASURED 2026-09-19 on the yvra appliance: `POST /start` answered
/// 200, the server read `maintenance` for 130 s and then `stopped` — the
/// 43 MB installer had booted, written the disk and powered off inside the
/// cloud's own operation, and the API never once said `started`. The old
/// wait wanted `started` and gave up after 300 s with the re-image DONE on
/// the disk and the box left `stopped` with both media on (the shape
/// `finish_loop`'s `stopped` arm ends). So, for a powering-off medium, a
/// `stopped` read AFTER a `maintenance` read is the pass having ended and
/// the wait is over; a `stopped` read before any `maintenance` is the start
/// not yet taken, and the wait goes on. Nothing changes for a medium that
/// reboots: `stopped` after `maintenance` is a start that failed, and the
/// timeout says so.
fn wait_for_installer_start<A: UpCloudApi, C: Clock>(
client: &Client<A>,
clock: &C,
plan: &Plan,
log: &mut dyn FnMut(&str),
) -> Result<String, String> {
if !plan.medium_powers_off {
wait_for_state(client, clock, plan, "started", log)?;
return Ok("the hypervisor reports the installer running".into());
}
let mut seen_maintenance = false;
let mut ended = false;
wait_for(
clock,
"the installer to run (`started`), or its pass to have ended already (`maintenance` then `stopped`)",
plan.stopping,
log,
|| match client.server_state(&plan.server_uuid)?.as_str() {
"started" => Ok(Progress::Done),
"maintenance" => {
seen_maintenance = true;
Ok(Progress::Waiting("state \"maintenance\" (the installer may be running inside it)".into()))
}
"stopped" if seen_maintenance => {
ended = true;
Ok(Progress::Done)
}
s => Ok(Progress::Waiting(format!("state {s:?}"))),
},
)?;
Ok(if ended {
"the pass already ended: read `maintenance` for the whole installer run and `stopped` after it, never `started` — the box is off with the media on, which the loop takes off".into()
} else {
"the hypervisor reports the installer running".into()
})
}
/// A banner, or a control-plane identity (`--verify-control`) — see [`banner::is_gunnar`].
fn is_gunnar(banner: &str) -> bool {
banner::is_gunnar(banner)
}
/// Sleep `ms` in slices, saying so once, so the settle before the eject is
/// visible in the log and interruptible by a fake clock.
fn settle<C: Clock>(clock: &C, ms: u64, what: &str, log: &mut dyn FnMut(&str)) {
log(&format!(" settling {} s — {what}", ms / 1000));
let until = clock.now_ms().saturating_add(ms);
while clock.now_ms() < until {
let left = until - clock.now_ms();
clock.sleep_ms(left.min(5_000));
}
}
// ── the loop ────────────────────────────────────────────────────────────────
/// How [`finish_loop`] ended: the box was just started from its disk (the
/// banner is still to be read), or the installed appliance was already
/// answering.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum LoopEnd {
Booted,
Serving(String),
}
/// Rounds before the loop gives up. Each round is one read of the server and
/// one act on it; a healthy run needs two (eject → power off/reboot → boot).
pub const MAX_ROUNDS: u32 = 8;
/// What the loop tracked, for the [`Outcome`].
#[derive(Default)]
struct LoopTrace {
ejected: bool,
detached: Vec<String>,
/// The install wait ran out at this clock moment, with this message.
budget_expired_at: Option<(u64, String)>,
}
/// **Every state a crash can leave, one round at a time.** See the module
/// header for the rule each arm enforces.
#[allow(clippy::too_many_arguments)]
fn finish_loop<A: UpCloudApi, C: Clock, B: BannerProbe>(
client: &Client<A>,
clock: &C,
probe: &B,
plan: &Plan,
ledger: &mut Ledger,
log: &mut dyn FnMut(&str),
trace: &mut LoopTrace,
warnings: &mut Vec<String>,
) -> Result<LoopEnd, String> {
for round in 1..=MAX_ROUNDS {
let view = client.server_view(&plan.server_uuid)?;
let media = classify(&view.devices, plan);
log(&format!(" round {round}: {}", describe(&view, &media)));
ledger.mark(
clock.now_ms(),
"round",
format!("{round}: {}", describe(&view, &media)),
);
foreign_refusal(plan, &media)?;
match view.state.as_str() {
"maintenance" => {
wait_for(
clock,
"the cloud to finish operating on the server (maintenance)",
plan.stopping,
log,
|| match client.server_state(&plan.server_uuid)? {
s if s == "maintenance" => Ok(Progress::Waiting("maintenance".into())),
_ => Ok(Progress::Done),
},
)?;
continue;
}
"error" => {
return Err(format!(
"REFUSED: the server is in state `error` — {}. Nothing here can act on \
that; look at it in the UpCloud console.",
describe(&view, &media)
))
}
"stopped" => {
// A stopped box is the one state where everything comes off.
if view.boots_cdrom() {
log(" boot order → disk");
client.set_boot_order(&plan.server_uuid, BootOrder::Disk)?;
ledger.mark(clock.now_ms(), "boot-order", "disk (the box is stopped)");
}
for d in media.all() {
log(&format!(
" detaching {} ({} {:?})",
d.address, d.kind, d.title
));
client.detach(&plan.server_uuid, &d.address)?;
trace.detached.push(d.address.clone());
ledger.mark(
clock.now_ms(),
"detach",
format!("{} {} {:?}", d.address, d.kind, d.title),
);
}
log(" starting the appliance from its disk");
client.start_server(&plan.server_uuid)?;
ledger.mark(clock.now_ms(), "start", "from disk, no installer device on");
return Ok(LoopEnd::Booted);
}
"started" => {
// A medium that powers off cannot loop, so the boot order is
// left for the `stopped` branch. MEASURED 2026-09-15 (run 3b):
// the mid-pass PUT put the running installer's server into
// `maintenance` 14 s after the start, and it read `stopped` only
// 146 s later — against an installer that finishes in seconds.
if view.boots_cdrom() && !plan.medium_powers_off {
log(
" boot order → disk (honoured at the next hypervisor start; a guest \
reboot does not re-read it — MEASURED 2026-09-14)",
);
client.set_boot_order(&plan.server_uuid, BootOrder::Disk)?;
ledger.mark(clock.now_ms(), "boot-order", "disk (the box is started)");
}
let serving = match probe.banner(&plan.verify_addr) {
Ok(b) if is_gunnar(&b) => Some(b),
_ => None,
};
// A medium that powers off cannot loop, so the pass in flight is
// left alone and ends in `stopped`, where everything comes off.
if let Some(loaded) = media.loaded_cdrom().filter(|_| !plan.medium_powers_off) {
// THE never-loop primitive. Legal while started; from
// here on no reboot of any kind lands on the installer.
log(&format!(
" ejecting the medium out of {} ({:?}) — the pass in flight reads \
its payload from virtio and is unaffected",
loaded.address, loaded.title
));
match client.eject_cdrom(&plan.server_uuid) {
Ok(()) => {
trace.ejected = true;
ledger.mark(clock.now_ms(), "eject", loaded.address.clone());
}
Err(e) => {
// Unexpected — measured 200 on a started box. Do
// not carry on as if it worked: a loaded cdrom is
// the loop. Stop the box; the next round detaches.
log(&format!(" eject REFUSED: {e} — stopping the box instead"));
ledger.mark(clock.now_ms(), "eject-refused", e.clone());
warnings.push(format!(
"the eject was refused ({e}); the box was stopped mid-pass \
instead, and the cdrom came off while stopped"
));
stop_and_wait(
client,
clock,
plan,
&view.state,
ledger,
log,
"the eject was refused",
)?;
continue;
}
}
}
if let Some(b) = serving {
return Ok(LoopEnd::Serving(b));
}
// The pass in flight ends one of two ways, and both are
// watched: the installer powers off (korp.install.then=
// poweroff) or reboots and, with the cdrom ejected, lands on
// the disk it just installed.
log(&format!(
" waiting up to {} min for the installer pass to end — the box powering \
off, or {} answering SSH",
plan.install.timeout_ms / 60_000,
plan.verify_addr
));
let wait_started = clock.now_ms();
// The last state the wait read: the stop after a timeout is
// judged on it (`ensure_stopped`), not sent blind.
let mut last_read = view.state.clone();
let ended = wait_for(
clock,
"the installer pass to end (power-off, or the installed appliance answering SSH)",
plan.install,
log,
|| {
let state = client.server_state(&plan.server_uuid)?;
last_read = state.clone();
if state == "stopped" {
return Ok(Progress::Done);
}
match probe.banner(&plan.verify_addr) {
Ok(b) if is_gunnar(&b) => Ok(Progress::Done),
Ok(b) => Ok(Progress::Waiting(format!(
"state {state:?}; something answered and it is not gunnar: {b:?}"
))),
Err(e) => Ok(Progress::Waiting(format!("state {state:?}; {e}"))),
}
},
);
match ended {
Ok(ms) => {
ledger.mark(
clock.now_ms(),
"pass-ended",
format!("after {} s", ms / 1000),
);
continue;
}
Err(e) if plan.strict_install_wait => {
return Err(format!(
"{e}\nREFUSED to stop the box (--strict-install-wait): the pass never \
ended inside its budget. The cdrom is {} and the boot order is \
`disk`, so it cannot loop; attach the console and look, then run \
--finish.",
if trace.ejected {
"ejected"
} else {
"not ours to eject"
}
))
}
Err(e) => {
log(&format!(" the pass never ended inside the budget: {e}"));
log(
" stopping the box — soft, and a hard stop follows the timeout. A \
stop mid-pass can leave a half-written boot disk; if the verify \
then finds nothing serving, a full re-image is the remedy (the data \
set is adopted, never re-laid).",
);
trace.budget_expired_at =
Some((wait_started.saturating_add(plan.install.timeout_ms), e));
stop_and_wait(
client,
clock,
plan,
&last_read,
ledger,
log,
"the pass never ended",
)?;
continue;
}
}
}
other => {
return Err(format!(
"the server reads state {other:?}, which this procedure does not know — \
{}",
describe(&view, &media)
))
}
}
}
Err(format!(
"gave up after {MAX_ROUNDS} rounds without a box booted from its disk — the receipt's \
`progress` says what each round saw"
))
}
/// The mid-pass stop: SOFT, whatever `--soft-stop-before-install` says, because
/// a half-written boot disk is exactly what is at risk. `read_state` is the
/// state the caller last read; see [`ensure_stopped`].
fn stop_and_wait<A: UpCloudApi, C: Clock>(
client: &Client<A>,
clock: &C,
plan: &Plan,
read_state: &str,
ledger: &mut Ledger,
log: &mut dyn FnMut(&str),
why: &str,
) -> Result<(), String> {
ensure_stopped(
client,
clock,
plan,
read_state,
StopKind::Soft,
ledger,
log,
why,
)
}
/// **A stop that is conditional on the state that was read.**
///
/// UpCloud answers `409 SERVER_STATE_ILLEGAL` to a stop on a stopped box.
/// Until 2026-09-23 every stop here was unconditional — step 4 stopped the
/// appliance though step 1 had just read its state, and the two mid-pass
/// stops in the loop did the same — so a re-image could never resume a run
/// that died at or after its own stop: the retry read `stopped`, sent the
/// stop, and was refused. The state is now judged first:
///
/// * `stopped` — done; nothing is sent, and the ledger says so;
/// * `maintenance` — the cloud is operating on the box and the outcome is not
/// known yet: wait for it to settle, then judge the state it settled in;
/// * anything else — stop it, as asked, and wait for `stopped`.
#[allow(clippy::too_many_arguments)]
fn ensure_stopped<A: UpCloudApi, C: Clock>(
client: &Client<A>,
clock: &C,
plan: &Plan,
read_state: &str,
kind: StopKind,
ledger: &mut Ledger,
log: &mut dyn FnMut(&str),
why: &str,
) -> Result<(), String> {
let mut state = read_state.to_string();
loop {
match state.as_str() {
"stopped" => {
log(&format!(
" already `stopped` — no stop is sent ({why}); UpCloud answers 409 \
SERVER_STATE_ILLEGAL to a stop on a stopped box"
));
ledger.mark(
clock.now_ms(),
"stop",
format!("not sent: the box read `stopped` — {why}"),
);
return Ok(());
}
"maintenance" => {
log(" the box reads `maintenance` — the cloud is operating on it; waiting for that to end before deciding whether a stop is needed");
let mut settled = String::new();
wait_for(
clock,
"the cloud to finish operating on the server (maintenance)",
plan.stopping,
log,
|| match client.server_state(&plan.server_uuid)? {
s if s == "maintenance" => Ok(Progress::Waiting("maintenance".into())),
s => {
settled = s;
Ok(Progress::Done)
}
},
)?;
log(&format!(" settled in state {settled:?}"));
state = settled;
}
_ => {
match kind {
StopKind::Hard => {
client.stop_server(&plan.server_uuid, StopKind::Hard, 0)?;
ledger.mark(clock.now_ms(), "stop", format!("hard — {why}"));
}
StopKind::Soft => {
client.stop_server(
&plan.server_uuid,
StopKind::Soft,
plan.soft_stop_timeout_s,
)?;
ledger.mark(
clock.now_ms(),
"stop",
format!("soft, hard after {} s — {why}", plan.soft_stop_timeout_s),
);
}
}
return wait_for_state(client, clock, plan, "stopped", log);
}
}
}
}
/// **The media, made and nothing else — the barrier two parallel installs share.**
///
/// # Why this door exists
///
/// MEASURED 2026-09-22/23: main and twin were installed one after the other
/// though they are two servers that share nothing until the election. Running
/// them together halves that wall-clock — but the ONE thing they are not
/// independent about is the medium. Both take the same bytes, both would find
/// the account empty at the same moment, and both would `POST /storage` under
/// the same title. Two storages with one title is a name that no longer
/// identifies an image, and the adoption gate in [`start_medium`] is built on
/// that name meaning something.
///
/// So the medium is the barrier. The caller makes it ONCE, waits for it to be
/// `online`, and only then fans the two installs out; each of them then ADOPTS
/// what is already there, by title and by the full digest [`crate::kept`]
/// recorded here. Serialised because it must be, and nothing else is.
///
/// Nothing about the SERVER is touched or even read: this call is legal against
/// a plan whose `server_uuid` is any of the machines that will use the medium.
pub fn make_media_only<A: UpCloudApi, C: Clock>(
client: &Client<A>,
clock: &C,
plan: &Plan,
ledger: &mut Ledger,
log: &mut dyn FnMut(&str),
) -> Result<Vec<Medium>, String> {
log(&format!(
"the medium only — {} face(s) of {}, sha256 {}",
media_titles(plan).len(),
plan.iso.path.display(),
plan.iso.sha256
));
let existing = client.private_storages()?;
let kept_media = kept::read(&kept::ledger_path(&plan.iso.path));
let mut media = Vec::new();
for title in media_titles(plan) {
let title = title.to_string();
media.push(start_medium(
client,
plan,
&title,
&existing,
&kept_media,
log,
)?);
}
let started = clock.now_ms();
wait_media_online(client, clock, plan, &media, log)?;
ledger.mark(
clock.now_ms(),
"media-only",
format!(
"{:?} online after {} s — the barrier two parallel installs share",
media.iter().map(|m| m.title.as_str()).collect::<Vec<_>>(),
clock.now_ms().saturating_sub(started) / 1000
),
);
log("the media are online. Two installs may now run TOGETHER: each adopts these, and adopting uploads nothing.");
Ok(media)
}
/// The whole procedure.
pub fn reinstall<A: UpCloudApi, C: Clock, B: BannerProbe>(
client: &Client<A>,
clock: &C,
probe: &B,
remote: &mut dyn Remote,
plan: &Plan,
ledger: &mut Ledger,
log: &mut dyn FnMut(&str),
) -> Result<Outcome, String> {
let started = clock.now_ms();
let mut warnings: Vec<String> = Vec::new();
if !plan.finish && !plan.confirmed {
return Err(format!(
"REFUSED: this DESTROYS the live appliance {:?} ({}) — every repository on it is \
gone. Pass --confirm-destroy if that is what you mean.",
plan.server_title, plan.server_uuid
));
}
// ── the machine decides how the medium rides — before the first call ────
let attach = attach_for(&plan.machine)?;
log(&format!(
"0. machine {:?} — {} attaches",
plan.machine,
attach.faces().len()
));
ledger.mark(clock.now_ms(), "machine", plan.machine.clone());
// ── the uuid is never guessed, and the box is read ONCE ─────────────────
log("1. checking the server is the one that was asked for, and what it is in");
let view = client.server_view(&plan.server_uuid)?;
if view.title != plan.server_title {
return Err(format!(
"REFUSED: {} is titled {:?} and the plan says {:?}. A uuid that names a \
different server than the operator believes is how someone else's box gets \
re-imaged.",
plan.server_uuid, view.title, plan.server_title
));
}
let media = classify(&view.devices, plan);
let sit = situation(&view, &media, &plan.previous);
log(&format!(
" {} is {:?} — as declared · {}",
plan.server_uuid,
view.title,
describe(&view, &media)
));
ledger.mark(clock.now_ms(), "situation", describe(&view, &media));
let existing = client.private_storages()?;
match (&sit, plan.finish) {
(
Situation::MidRun {
state,
boots_cdrom,
loaded_cdrom,
},
false,
) => {
return Err(format!(
"REFUSED: a re-image is in flight or died on this box — state {state:?}, \
boot_order {}, {}. Uploading and attaching again on top of it is how two \
installers end up on one box. Run the same command with --finish: it uploads \
nothing, ejects the cdrom, boots the disk and verifies.",
if *boots_cdrom { "cdrom" } else { "disk" },
match loaded_cdrom {
Some(d) => format!("cdrom {} loaded with {:?}", d.address, d.title),
None => "no cdrom loaded".to_string(),
}
))
}
(Situation::Fresh { leftovers }, true) if leftovers.is_empty() => {
log(
" nothing is mid-run: no installer device on the box, boot order disk. \
--finish will verify, give back any seed left in the account, and refresh the \
front.",
);
}
(Situation::Fresh { leftovers }, true) => {
log(&format!(
" nothing can boot an installer here; {} leftover device(s) come off at the \
next stop",
leftovers.len()
));
}
(Situation::MidRun { .. }, true) => {
log(" --finish: ending the run that was left here");
}
(
Situation::Installed {
pass_ended,
leftovers,
},
_,
) => {
log(&format!(
" RESUMING from the previous receipt: its installer pass ended ({pass_ended}) and \
the box was never started from its disk — the install is on the disk. Nothing is \
uploaded or attached; {} leftover device(s) come off and the box is started from \
disk. (The shape of the run that died at `detach` on 2026-09-23 18:51.)",
leftovers.len()
));
ledger.mark(
clock.now_ms(),
"resume",
format!("installed — pass ended {pass_ended}; starting from disk"),
);
}
(Situation::Fresh { .. }, false) => {}
}
let resume = matches!(sit, Situation::Installed { .. });
if plan.finish || resume {
foreign_refusal(plan, &media)?;
}
if plan.dry_run {
log("dry run: everything above is read-only, and this is where the changes would start.");
if plan.finish || resume {
log(&format!(
" would finish from: {}",
describe(&view, &media)
));
} else {
log(&format!(
" would create {}",
media_titles(plan)
.iter()
.map(|t| format!("{t:?}"))
.collect::<Vec<_>>()
.join(" and ")
));
}
return Err("dry run — nothing was changed".to_string());
}
let mut console = None;
let mut trace = LoopTrace::default();
let install_started;
let media_ours: [Option<Medium>; 2];
if plan.finish || resume {
media_ours = media_in_account(plan, &existing);
install_started = clock.now_ms();
log(if resume {
"2. resuming — the loop reads the stopped box, takes any leftover off and starts it from disk"
} else {
"2. finishing — the loop reads the box and acts on what it sees"
});
} else {
// ── the medium, once per FACE. THE TRAP: one medium is not enough ──
let [face_cdrom, face_virtio] = attach.faces();
log(&format!("2. the medium, as a CDROM — {}", face_cdrom.role));
let kept_media = kept::read(&kept::ledger_path(&plan.iso.path));
let cdrom = start_medium(client, plan, &plan.title_cdrom, &existing, &kept_media, log)?;
ledger.mark(
clock.now_ms(),
"media-cdrom",
format!("{} adopted={}", cdrom.uuid, cdrom.adopted),
);
// ── the SECOND upload, only for a medium that cannot read its own CD ─
//
// korp-installer ≥ 0.1.14 on a donor kernel with the CD block reads its
// payload off the cdrom it booted from, so the virtio copy — the second
// upload and the second sync, ~4 min per re-image (mock behaviour 25) —
// is not made. The medium's own record decides it (`Plan::cd_only`);
// any doubt means both, because a cd-only upload of a medium without
// the driver never leaves `maintenance` (T16's negative control).
// The SAME list the dry run previews (`media_titles`): the preview and
// the run cannot disagree about how many media are made.
let virtio = if !media_titles(plan).contains(&plan.title_virtio.as_str()) {
log("3. NO virtio copy — this medium reads its own CD (its .medium.json record) and powers off \
after INSTALL-OK: ONE upload");
ledger.mark(
clock.now_ms(),
"media-virtio",
"not made: the medium reads its own CD",
);
None
} else {
log(&format!(
"3. the SAME medium again, as a virtio DISK — {}",
face_virtio.role
));
let v = start_medium(
client,
plan,
&plan.title_virtio,
&existing,
&kept_media,
log,
)?;
ledger.mark(
clock.now_ms(),
"media-virtio",
format!("{} adopted={}", v.uuid, v.adopted),
);
Some(v)
};
// ── the syncs, TOGETHER ─────────────────────────────────────────────
//
// Every upload is done and verified by here. What is left is UpCloud
// provisioning the volumes, which it does in parallel and which this
// procedure used to watch one at a time. See `start_medium`.
let media: Vec<Medium> = std::iter::once(cdrom.clone())
.chain(virtio.clone())
.collect();
log(if media.len() == 2 {
"3b. both seeds leave `syncing` — ONE wait, not one per medium"
} else {
"3b. the seed leaves `syncing`"
});
let sync_started = clock.now_ms();
wait_media_online(client, clock, plan, &media, log)?;
ledger.mark(
clock.now_ms(),
"media-online",
format!(
"online after {} s of shared waiting",
clock.now_ms().saturating_sub(sync_started) / 1000
),
);
// ── down ────────────────────────────────────────────────────────────
//
// HARD, and the reason is the next three steps: the media are attached,
// the boot order goes to `cdrom`, and korp-installer re-lays the system
// disk from the first byte. There is nothing on that disk for a
// shutdown to settle, because nothing on it survives.
//
// MEASURED 2026-09-22/23: the polite stop cost ~158 s (against a 300 s
// budget) — the guest's own shutdown, then UpCloud's 60 s grace, then
// the poll to `stopped`. It bought a clean unmount of a filesystem that
// was overwritten ninety seconds later.
//
// The one thing it is NOT: a stop before anything else. Mid-pass stops
// stay soft (`stop_and_wait`), because there a half-written boot disk
// is exactly what is at risk; and `--finish` never reaches here.
//
// The caveat an operator should know: the appliance's DATA volumes are
// separate storages the installer does not touch, and a hard stop is a
// power cut to their xfs. xfs replays its journal on the next mount, so
// this is dirty-and-recovered rather than lost — but a run that wants
// the guest asked first has `--soft-stop-before-install`.
//
// And CONDITIONAL on the state step 1 read: a box already `stopped`
// gets no stop (UpCloud refuses it, 409 SERVER_STATE_ILLEGAL — which
// is how a retry after a death at this stop could never get past it),
// one in `maintenance` is waited for and judged again. See
// `ensure_stopped`.
let kind = if plan.hard_stop_before_install {
StopKind::Hard
} else {
StopKind::Soft
};
match kind {
StopKind::Hard => log("4. stopping the appliance (HARD — its disk is re-laid in three steps' time), if it is running"),
StopKind::Soft => log("4. stopping the appliance (soft — --soft-stop-before-install), if it is running"),
}
ensure_stopped(
client,
clock,
plan,
&view.state,
kind,
ledger,
log,
match kind {
StopKind::Hard => "before the install; the disk this would settle is overwritten by the installer (~158 s saved)",
StopKind::Soft => "before the install; the serving appliance goes down here",
},
)?;
// ── leftovers of an earlier run come off while it is stopped ────────
if let Situation::Fresh { leftovers } = &sit {
for d in leftovers {
log(&format!(
" detaching leftover {} ({} {:?})",
d.address, d.kind, d.title
));
client.detach(&plan.server_uuid, &d.address)?;
trace.detached.push(d.address.clone());
ledger.mark(clock.now_ms(), "detach-leftover", d.address.clone());
let ours =
d.storage == cdrom.uuid || virtio.as_ref().is_some_and(|v| d.storage == v.uuid);
if !ours && !d.storage.is_empty() && installer_title(&plan.stem, &d.title).is_some()
{
match client.delete_storage(&d.storage) {
Ok(()) => log(&format!(" gave back the leftover seed {}", d.title)),
Err(e) => warnings.push(format!(
"the leftover seed {} ({}) could not be deleted: {e}",
d.title, d.storage
)),
}
}
}
}
// ── both devices on ─────────────────────────────────────────────────
log(&format!(
"5. attaching {} as {}",
cdrom.uuid,
face_cdrom.kind.as_str()
));
client.attach(&plan.server_uuid, face_cdrom.kind, &cdrom.uuid)?;
ledger.mark(clock.now_ms(), "attach-cdrom", cdrom.uuid.clone());
if let Some(virtio) = &virtio {
log(&format!(
"6. attaching {} as a virtio {}",
virtio.uuid,
face_virtio.kind.as_str()
));
client.attach(&plan.server_uuid, face_virtio.kind, &virtio.uuid)?;
ledger.mark(clock.now_ms(), "attach-virtio", virtio.uuid.clone());
}
// ── boot the installer ──────────────────────────────────────────────
log("7. boot order → cdrom, and start");
client.set_boot_order(&plan.server_uuid, BootOrder::Cdrom)?;
ledger.mark(clock.now_ms(), "boot-order", "cdrom");
client.start_server(&plan.server_uuid)?;
install_started = clock.now_ms();
ledger.mark(clock.now_ms(), "start", "from the installer cdrom");
let seen = wait_for_installer_start(client, clock, plan, log)?;
ledger.mark(clock.now_ms(), "started", seen);
// The console comes up AFTER the start: the VNC port is reallocated on
// every stop/start and the API keeps reporting the old one.
if let Some(pw) = &plan.console_password {
log("8. opening the console (no → yes, then the RFB greeting is verified)");
match console::open(client, clock, probe, &plan.server_uuid, pw, 2_000, 5, log) {
Ok(c) => {
log(&format!(
" console {} — {} — password {}",
c.address(),
c.greeting,
c.password
));
console = Some(c);
}
Err(e) => {
log(&format!(" console UNAVAILABLE: {e}"));
warnings.push(format!("the console never answered: {e}"));
}
}
}
// ── "demonstrably booted": the settle, then the loop ejects ─────────
if plan.medium_powers_off {
log("9. the medium powers off after INSTALL-OK (korp.install.then=poweroff): no settle, no eject — the pass ends in `stopped`");
ledger.mark(clock.now_ms(), "settle", "skipped: the medium powers off");
} else {
log("9. letting the installer boot off the cdrom before the medium is ejected");
settle(
clock,
plan.eject_after_ms,
"the loader reads vmlinuz+initrd off the cdrom in the first seconds; after that the \
installer probes ONLY virtio",
log,
);
ledger.mark(
clock.now_ms(),
"settle",
format!("{} s after started", plan.eject_after_ms / 1000),
);
}
media_ours = [Some(cdrom), virtio];
log("10. ending the pass safely — boot order disk, eject, and watch the pass end");
}
// ── the loop: every state, one round at a time ──────────────────────────
let end = finish_loop(
client,
clock,
probe,
plan,
ledger,
log,
&mut trace,
&mut warnings,
)?;
// ── the API is not evidence ─────────────────────────────────────────────
log(&format!(
"11. reading the SSH banner off {}",
plan.verify_addr
));
let mut seen = match &end {
LoopEnd::Serving(b) => b.clone(),
LoopEnd::Booted => String::new(),
};
wait_for(
clock,
"the re-imaged appliance to answer SSH",
plan.verify,
log,
|| match probe.banner(&plan.verify_addr) {
Ok(b) => {
seen = b.clone();
Ok(Progress::Done)
}
Err(e) => Ok(Progress::Waiting(e)),
},
)
.map_err(|e| {
let install = match &trace.budget_expired_at {
Some((_, timeout)) => {
format!("\nThe install budget ran out too, with the same silence: {timeout}")
}
None => String::new(),
};
format!(
"{e}\nThe API reported success at every step and the appliance is NOT SERVING. \
That is precisely why this check exists: `200 OK` describes the hypervisor, not \
the box.{install}"
)
})?;
ledger.mark(clock.now_ms(), "verify", seen.clone());
// ── the host key, in the same session as the banner ─────────────────────
//
// The box is proved up and answering on `verify_addr` one line above. This
// asks the same address for the one thing the banner cannot carry: the key
// it signs with. An SSH server presents its host key in the key exchange,
// before authentication and before anything else it will ever say, so
// there is nothing to arrange — the session the verify just had already
// reached it.
//
// NEVER fatal. The box being SERVING is the verdict of this procedure and
// it has already been taken; a host key that could not be read is a fact
// recorded in the receipt, in the words of whichever probe could not read
// it, next to the address and the time it was tried. What must never
// happen is a receipt with a blank where a fingerprint goes and nothing
// saying why, because that is unreadable later in exactly the same way
// `known_hosts` is.
let host_key = probe.host_key_fingerprint(&plan.verify_addr);
match &host_key {
Ok(k) => log(&format!(
" host key {} — presented by {}, and this receipt is where that fact lives",
k.fingerprint, plan.verify_addr
)),
Err(why) => {
log(&format!(" host key NOT READ: {why}"));
warnings.push(format!(
"no SSH host key was read off {}: {why}. The receipt names no fingerprint, so \
nothing but a scan can tell you what to trust on that address.",
plan.verify_addr
));
}
}
ledger.mark(
clock.now_ms(),
"host-key",
match &host_key {
Ok(k) => k.fingerprint.clone(),
Err(why) => format!("not read: {why}"),
},
);
let install_wait = match trace.budget_expired_at.take() {
None => InstallWait::InTime {
budget_ms: plan.install.timeout_ms,
took_ms: clock.now_ms().saturating_sub(install_started),
},
Some((deadline_ms, timeout)) => InstallWait::Late {
budget_ms: plan.install.timeout_ms,
past_budget_ms: clock.now_ms().saturating_sub(deadline_ms),
timeout,
},
};
log(&format!(" install: {}", install_wait.verdict()));
banner::identity_gate(&seen, plan.expect_version.as_deref())?;
banner::role_gate(&seen, plan.expect_role.as_deref())?;
let gate =
match &plan.expect_version {
Some(v) => Gate::Passed(v.clone()),
None => {
log(" GATE-IDENTITY: SKIPPED — no --expect-version was given, so this run proves \
the appliance is up and NOT which build it is");
Gate::Skipped
}
};
ledger.mark(
clock.now_ms(),
"gate",
match &gate {
Gate::Passed(v) => format!("passed against {v}"),
Gate::Skipped => "skipped".to_string(),
},
);
// ── the seeds: give back what is off, name what is still on ─────────────
//
// A delete that fails must not hold the boot: the appliance coming up is
// worth more than a gigabyte. UpCloud refuses to delete a storage a server
// still carries, and the virtio seed cannot be detached from a started box
// (511 HOTPLUG_FAILED, measured), so it is NAMED and left for the next stop.
let after = client.server_view(&plan.server_uuid)?;
let mut deleted: Vec<String> = Vec::new();
let mut left_attached: Vec<(String, String)> = Vec::new();
// ── UPLOAD ONCE PER SHA ─────────────────────────────────────────────────
//
// The seeds are KEPT by default. Their titles carry this ISO's digest and
// `kept.rs` carries the full one, so the next run of the same bytes adopts
// them in place of an upload and a `syncing` wait — MEASURED at 165 s+ per
// attempt, paid on every one of five go-live attempts for bytes that never
// changed. `--give-back-medium` is the door to the old behaviour.
if plan.keep_medium {
let titles: Vec<&str> = media_ours
.iter()
.flatten()
.map(|m| m.title.as_str())
.collect();
log(&format!(
"12. KEEPING the seed storage(s) {titles:?} — the title carries this ISO's digest \
and the next run of these bytes adopts them instead of uploading again (~165 s \
saved per retry; ~1 GiB per face billed until `--give-back-medium`)"
));
ledger.mark(
clock.now_ms(),
"seeds",
format!("kept {titles:?} — upload once per sha256"),
);
} else {
for m in media_ours.iter().flatten() {
if let Some(d) = after.devices.iter().find(|d| d.storage == m.uuid) {
left_attached.push((d.address.clone(), m.title.clone()));
warnings.push(format!(
"the seed {} ({}) is still attached at {} — it cannot be detached from a \
started box (511 HOTPLUG_FAILED, measured 2026-09-14) and boots nothing. It \
comes off at the next stop: `--finish` on a stopped box, or the next re-image. \
~1 GiB is billed until then.",
m.title, m.uuid, d.address
));
continue;
}
match client.delete_storage(&m.uuid) {
Ok(()) => {
log(&format!("12. gave back the seed storage {}", m.title));
deleted.push(m.uuid.clone());
}
Err(e) => warnings.push(format!(
"the seed storage {} ({}) could not be deleted and is now an orphan billed to \
the account — delete it by hand: {e}",
m.title, m.uuid
)),
}
}
ledger.mark(
clock.now_ms(),
"seeds",
format!("deleted {:?} left {:?}", deleted, left_attached),
);
}
let after_media = classify(&after.devices, plan);
if let Some(d) = &after_media.empty_cdrom {
warnings.push(format!(
"an empty cdrom device remains at {} (the medium was ejected). It boots nothing; \
it comes off at the next stop.",
d.address
));
}
for d in &after_media.foreign {
warnings.push(format!(
"a foreign installer device remains at {} ({:?}); it was not deleted",
d.address, d.title
));
}
// ── the front, and only now ─────────────────────────────────────────────
let f = &plan.front;
let front_result = if !plan.front_restart {
let manual = front::manual_command(f);
log("13. NOT restarting the front (--no-front-restart)");
log(&format!(
" ⚠ the front still holds gRPC channels to the box that was just replaced. \
Until these run, the site answers `[]` for repositories that exist:\n \
{manual}"
));
FrontRefresh::Skipped { manual }
} else {
log(&format!(
"13. restarting the front's services at {}@{} — a re-image replaces the box \
they dial",
f.user, f.host
));
match front::refresh(remote, clock, f, log) {
Ok(done) => done,
Err(why) => FrontRefresh::Failed {
why,
manual: front::manual_command(f),
},
}
};
ledger.mark(
clock.now_ms(),
"front",
match &front_result {
FrontRefresh::Done { .. } => "restarted and answering".to_string(),
FrontRefresh::Skipped { .. } => "skipped (--no-front-restart)".to_string(),
FrontRefresh::Failed { why, .. } => format!("FAILED: {why}"),
},
);
let [cdrom, virtio] = media_ours;
Ok(Outcome {
cdrom,
virtio,
ejected: trace.ejected,
detached: trace.detached,
left_attached,
deleted,
front: front_result,
build_reference: banner::build_reference(&seen).map(|s| s.to_string()),
banner: seen,
host_key,
host_key_addr: plan.verify_addr.clone(),
install_wait,
gate,
console,
finished: plan.finish || resume,
warnings,
took_ms: clock.now_ms().saturating_sub(started),
})
}
#[cfg(test)]
mod tests {
use super::*;
use crate::wait::FakeClock;
fn dev(address: &str, storage: &str, title: &str, kind: &str) -> Device {
Device {
address: address.into(),
storage: storage.into(),
title: title.into(),
kind: kind.into(),
}
}
fn plan() -> Plan {
let iso = Iso {
path: "/x/gunnar-installer-open.iso".into(),
len: 1,
sha256: "3a413c6060175a23421e96e814eaa18b8f690fff62b0c1985a69313ed660b0a7".into(),
age_s: 0,
};
let (title_cdrom, title_virtio) = titles(
"gunnar-installer-open",
iso.short_sha(),
Attach::CdromAndVirtio,
);
Plan {
server_uuid: "u".into(),
server_title: "gunnar-appliance".into(),
machine: crate::machine::MACHINE.into(),
zone: "se-sto1".into(),
api_base: "https://api.upcloud.com/1.3".into(),
iso,
stem: "gunnar-installer-open".into(),
title_cdrom,
title_virtio,
verify_addr: "203.0.113.55:2222".into(),
expect_version: None,
expect_role: None,
confirmed: true,
dry_run: false,
finish: false,
allow_foreign_media: false,
eject_after_ms: 120_000,
medium_powers_off: false,
cd_only: false,
soft_stop_timeout_s: 60,
storage_online: Budget::new(1, 1),
stopping: Budget::new(1, 1),
install: Budget::new(1, 1),
verify: Budget::new(1, 1),
strict_install_wait: false,
console_password: None,
front: Front::from_spec(
&crate::site::FrontSpec::parse("root@203.0.113.203").unwrap(),
"/dev/null".into(),
),
front_restart: false,
keep_medium: true,
hard_stop_before_install: true,
previous: Vec::new(),
}
}
fn view(state: &str, boot_order: &str, devices: Vec<Device>) -> ServerView {
ServerView {
title: "gunnar-appliance".into(),
state: state.into(),
boot_order: boot_order.into(),
devices,
}
}
const BOOT: (&str, &str, &str, &str) = ("virtio:0", "boot-uuid", "gunnar-boot", "disk");
fn boot() -> Device {
dev(BOOT.0, BOOT.1, BOOT.2, BOOT.3)
}
#[test]
fn the_titles_carry_the_isos_digest_so_adoption_is_sound() {
let (c, v) = titles("gunnar-installer-open", "3f2a1b9c", Attach::CdromAndVirtio);
assert_eq!(c, "gunnar-installer-open-3f2a1b9c-cdrom");
assert_eq!(v, "gunnar-installer-open-3f2a1b9c-virtio");
assert_ne!(c, v, "the two media must never resolve to one storage");
assert_eq!(
installer_title("gunnar-installer-open", &c),
Some(("3f2a1b9c".into(), "cdrom".into()))
);
assert_eq!(
installer_title("gunnar-installer-open", "gunnar-boot"),
None
);
assert_eq!(
installer_title("gunnar-installer-open", "gunnar-installer-open-xyz-cdrom"),
None,
"eight hex digits or it is not a digest"
);
}
/// **The state machine, as a table.** Each row is a state a crash can
/// leave, and the situation it is judged to be.
#[test]
fn a_box_with_nothing_bootable_is_fresh_and_anything_else_is_mid_run() {
let p = plan();
let cd = dev("ide:0:0", "cd", &p.title_cdrom, "cdrom");
let vd = dev("virtio:5", "vd", &p.title_virtio, "disk");
let empty = dev("ide:0:0", "", "", "cdrom");
// The box as a fresh run finds it.
let v = view("started", "disk", vec![boot()]);
assert_eq!(
situation(&v, &classify(&v.devices, &p), &[]),
Situation::Fresh { leftovers: vec![] }
);
// Tonight, 20:47: cdrom ejected, virtio seed on, boot order disk,
// banner up. Nothing can boot an installer; two leftovers.
let v = view("started", "disk", vec![boot(), empty.clone(), vd.clone()]);
match situation(&v, &classify(&v.devices, &p), &[]) {
Situation::Fresh { leftovers } => {
assert_eq!(leftovers, vec![empty.clone(), vd.clone()]);
}
other => panic!("{other:?}"),
}
// The loop, an hour earlier: loaded cdrom, boot order disk (set by
// hand, not honoured by the guest reboot).
let v = view("started", "disk", vec![boot(), cd.clone(), vd.clone()]);
assert!(matches!(
situation(&v, &classify(&v.devices, &p), &[]),
Situation::MidRun {
boots_cdrom: false,
loaded_cdrom: Some(_),
..
}
));
// An orchestrator that died between the attach and the start.
let v = view("stopped", "cdrom", vec![boot(), cd.clone(), vd.clone()]);
assert!(matches!(
situation(&v, &classify(&v.devices, &p), &[]),
Situation::MidRun {
boots_cdrom: true,
..
}
));
// Boot order cdrom with NO medium: still mid-run — a hypervisor start
// would look for a cdrom.
let v = view("stopped", "cdrom", vec![boot()]);
assert!(matches!(
situation(&v, &classify(&v.devices, &p), &[]),
Situation::MidRun {
loaded_cdrom: None,
..
}
));
}
/// A medium with another digest is FOREIGN, ours is ours, and the boot
/// disk is nobody's business.
#[test]
fn media_are_sorted_by_title_and_the_boot_disk_is_never_an_installer_device() {
let p = plan();
let cd = dev("ide:0:0", "cd", &p.title_cdrom, "cdrom");
let other_vd = dev(
"virtio:5",
"vd2",
"gunnar-installer-open-deadbeef-virtio",
"disk",
);
let stray_cd = dev("ide:0:1", "x", "ubuntu-24.04.iso", "cdrom");
let data = dev("virtio:1", "d1", "gunnar-data-1", "disk");
let m = classify(
&[boot(), cd.clone(), other_vd.clone(), stray_cd.clone(), data],
&p,
);
assert_eq!(m.cdrom, Some(cd.clone()));
assert_eq!(m.virtio, None);
assert_eq!(m.foreign, vec![other_vd.clone(), stray_cd.clone()]);
assert_eq!(m.loaded_cdrom(), Some(&cd));
assert_eq!(
m.all(),
vec![cd, other_vd, stray_cd],
"the loaded cdrom comes off first, the boot and data disks never"
);
let err = foreign_refusal(&p, &m).unwrap_err();
assert!(err.contains("REFUSED"), "{err}");
assert!(err.contains("deadbeef"), "names the other digest: {err}");
assert!(err.contains("--allow-foreign-media"), "{err}");
let mut allowed = p.clone();
allowed.allow_foreign_media = true;
foreign_refusal(&allowed, &m).unwrap();
}
/// The ledger calls its sink at EVERY mark, with everything so far.
#[test]
fn the_ledger_writes_at_every_step_not_only_at_the_end() {
use std::cell::RefCell;
use std::rc::Rc;
let seen: Rc<RefCell<Vec<usize>>> = Rc::new(RefCell::new(Vec::new()));
let s = Rc::clone(&seen);
let mut ledger = Ledger::new(move |steps| s.borrow_mut().push(steps.len()));
let clock = FakeClock::new();
ledger.mark(clock.now_ms(), "machine", "upcloud");
clock.sleep_ms(1_000);
ledger.mark(clock.now_ms(), "situation", "state started");
ledger.mark(clock.now_ms(), "eject", "ide:0:0");
assert_eq!(*seen.borrow(), vec![1, 2, 3]);
assert_eq!(ledger.sink_calls, 3);
assert!(ledger.has("eject") && !ledger.has("verify"));
assert_eq!(ledger.steps[1].at_ms, 1_000);
}
/// **The situation table, with the receipt as the third input.** The
/// box of 2026-09-23 18:51 reads exactly like a fresh one; the receipt is
/// what tells them apart, and only for a STOPPED box with nothing bootable.
#[test]
fn a_stopped_box_with_nothing_bootable_is_installed_only_when_the_receipt_says_so() {
let p = plan();
let cd = dev("ide:0:0", "cd", &p.title_cdrom, "cdrom");
let vd = dev("virtio:5", "vd", &p.title_virtio, "disk");
let step = |step: &str, detail: &str| StepRecord {
at_ms: 0,
step: step.into(),
detail: detail.into(),
};
let died_at_detach = vec![
step("start", "from the installer cdrom"),
step("pass-ended", "after 153 s"),
step("boot-order", "disk (the box is stopped)"),
step("detach", "ide:0:0 cdrom"),
];
let ended = {
let mut v = died_at_detach.clone();
v.push(step("start", "from disk, no installer device on"));
v
};
// 18:51: stopped, media off, boot order disk, the receipt above.
let v = view("stopped", "disk", vec![boot()]);
assert_eq!(
situation(&v, &classify(&v.devices, &p), &died_at_detach),
Situation::Installed {
pass_ended: "after 153 s".into(),
leftovers: vec![]
}
);
// The virtio seed still on (died between the two detaches): installed, one leftover.
let v = view("stopped", "disk", vec![boot(), vd.clone()]);
assert_eq!(
situation(&v, &classify(&v.devices, &p), &died_at_detach),
Situation::Installed {
pass_ended: "after 153 s".into(),
leftovers: vec![vd.clone()]
}
);
// No receipt: fresh.
let v = view("stopped", "disk", vec![boot()]);
assert_eq!(
situation(&v, &classify(&v.devices, &p), &[]),
Situation::Fresh { leftovers: vec![] }
);
// A receipt whose story ended (started from disk): fresh.
assert_eq!(
situation(&v, &classify(&v.devices, &p), &ended),
Situation::Fresh { leftovers: vec![] }
);
// The receipt, but the box is STARTED: fresh — something started it, and
// whatever is serving is what the verify reads.
let v = view("started", "disk", vec![boot()]);
assert_eq!(
situation(&v, &classify(&v.devices, &p), &died_at_detach),
Situation::Fresh { leftovers: vec![] }
);
// The receipt, but a loaded cdrom: mid-run, never installed.
let v = view("stopped", "disk", vec![boot(), cd.clone()]);
assert!(matches!(
situation(&v, &classify(&v.devices, &p), &died_at_detach),
Situation::MidRun { .. }
));
// The receipt, but boot order cdrom: mid-run.
let v = view("stopped", "cdrom", vec![boot()]);
assert!(matches!(
situation(&v, &classify(&v.devices, &p), &died_at_detach),
Situation::MidRun { .. }
));
// A pass that ended, then a SECOND installer start with no end: not installed.
let mut twice = died_at_detach.clone();
twice.push(step("start", "from the installer cdrom"));
assert_eq!(
receipt_says_installed(&twice).as_deref(),
Some("after 153 s"),
"the last pass-end still counts: a start from the cdrom is not a start from disk"
);
}
/// **A stop that reads the state first.** A scripted cloud: the states it
/// answers, in order, and the calls it was sent.
struct Script {
states: std::cell::RefCell<std::collections::VecDeque<&'static str>>,
calls: std::cell::RefCell<Vec<String>>,
}
impl Script {
fn answering(states: &[&'static str]) -> Script {
Script {
states: std::cell::RefCell::new(states.iter().copied().collect()),
calls: std::cell::RefCell::new(Vec::new()),
}
}
fn calls(&self) -> Vec<String> {
self.calls.borrow().clone()
}
}
impl upcloud_api::Exchange for Script {
fn describe(&self) -> String {
"scripted".into()
}
fn is_the_account(&self) -> bool {
false
}
fn exchange(&self, call: upcloud_api::Call<'_>) -> Result<upcloud_api::Reply, String> {
let upcloud_api::Call::Api { method, path, body } = call else {
panic!("no upload here")
};
let line = format!(
"{} {path} {}",
method.as_str(),
body.map(|b| b.to_string()).unwrap_or_default()
);
self.calls.borrow_mut().push(line);
if path.ends_with("/stop") {
return Ok(upcloud_api::Reply {
status: 200,
body: serde_json::Value::Null,
text: String::new(),
});
}
let mut q = self.states.borrow_mut();
let s = if q.len() > 1 {
q.pop_front().unwrap()
} else {
*q.front().expect("a state to answer")
};
let body = serde_json::json!({"server": {"state": s}});
Ok(upcloud_api::Reply {
status: 200,
text: body.to_string(),
body,
})
}
}
fn stop_on(
read: &str,
kind: StopKind,
script: &Script,
) -> (Result<(), String>, Ledger, Vec<String>) {
let mut p = plan();
// The unit plan's budgets are zero; a `maintenance` wait needs polls.
p.stopping = Budget::new(60_000, 5_000);
let client = Client::new(upcloud_api::Over(script));
let clock = FakeClock::new();
let mut ledger = Ledger::silent();
let mut lines = Vec::new();
let r = ensure_stopped(
&client,
&clock,
&p,
read,
kind,
&mut ledger,
&mut |l: &str| lines.push(l.to_string()),
"test",
);
(r, ledger, lines)
}
#[test]
fn a_box_read_stopped_is_sent_nothing() {
let s = Script::answering(&["stopped"]);
let (r, ledger, lines) = stop_on("stopped", StopKind::Hard, &s);
r.unwrap();
assert!(s.calls().is_empty(), "not even a read: {:?}", s.calls());
assert!(
ledger
.steps
.iter()
.any(|x| x.step == "stop" && x.detail.contains("not sent")),
"{:?}",
ledger.steps
);
assert!(
lines.iter().any(|l| l.contains("409")),
"the reason is in the log: {lines:?}"
);
}
#[test]
fn a_box_read_in_maintenance_that_settles_stopped_is_sent_nothing() {
let s = Script::answering(&["maintenance", "maintenance", "stopped"]);
let (r, _, _) = stop_on("maintenance", StopKind::Hard, &s);
r.unwrap();
let calls = s.calls();
assert_eq!(
calls.iter().filter(|c| c.contains("/stop")).count(),
0,
"{calls:?}"
);
assert_eq!(
calls.iter().filter(|c| c.starts_with("GET")).count(),
3,
"polled until it settled: {calls:?}"
);
}
#[test]
fn a_box_read_in_maintenance_that_settles_started_is_then_stopped() {
let s = Script::answering(&["maintenance", "started", "stopped"]);
let (r, ledger, _) = stop_on("maintenance", StopKind::Hard, &s);
r.unwrap();
let calls = s.calls();
let stop = calls
.iter()
.position(|c| c.contains("/stop"))
.expect("a stop was sent");
assert!(
calls[..stop].iter().any(|c| c.contains("started")) || stop >= 2,
"after it settled: {calls:?}"
);
assert!(calls[stop].contains("\"hard\""), "{}", calls[stop]);
assert_eq!(calls.iter().filter(|c| c.contains("/stop")).count(), 1);
assert!(ledger
.steps
.iter()
.any(|x| x.step == "stop" && x.detail.starts_with("hard")));
}
#[test]
fn a_box_read_started_is_stopped_and_the_mid_pass_stop_stays_soft() {
let s = Script::answering(&["stopped"]);
let p = plan();
let client = Client::new(upcloud_api::Over(&s));
let clock = FakeClock::new();
let mut ledger = Ledger::silent();
stop_and_wait(
&client,
&clock,
&p,
"started",
&mut ledger,
&mut |_| {},
"the pass never ended",
)
.unwrap();
let calls = s.calls();
assert!(
calls[0].contains("/stop") && calls[0].contains("\"soft\""),
"{calls:?}"
);
assert!(
calls[0].contains(&format!("\"timeout\":\"{}\"", p.soft_stop_timeout_s)),
"{calls:?}"
);
assert!(ledger
.steps
.iter()
.any(|x| x.step == "stop" && x.detail.starts_with("soft")));
}
#[test]
fn the_settle_is_the_declared_length_on_the_clock() {
let clock = FakeClock::new();
let mut lines = Vec::new();
settle(&clock, 120_000, "the loader", &mut |s: &str| {
lines.push(s.to_string())
});
assert_eq!(clock.elapsed_ms(), 120_000);
assert!(lines[0].contains("120 s"), "{lines:?}");
}
}