Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/monetize

crates/mock-upcloud/BEHAVIOURS-LEDGER.md

UpCloud behaviours ledger: everything the estate has learned, reconciled against mock-upcloud

Lane T13, 2026-09-21. The question this file answers is the owner's: "have you collected all behaviours and misunderstandings using UpCloud, so you can mimic timing and behaviour?"

Sources swept. Every memory note that mentions UpCloud (22). The code and docs in gunnar (master), private-gunnar-ops (main), private-holger-ops (main), holger (main), tunnr (main), monetize-impl (main), monetize (main) and edda (agent/r41-warehouse-resolver). This lane's own runs of the real provider UpCloudLtd/upcloud 5.44.1 against the mock (terraform 1.16.2). All provider runs used a fake token, a loopback base URL, and scratch copies of both estates' terraform/.

Column key.

Short repo prefixes: G gunnar · PGO private-gunnar-ops · PHO private-holger-ops · MI monetize-impl · ED edda · T tunnr · H holger · mem: a memory note.

1. API surface and wire shapes

# What UpCloud does Source M/R Timing mock #
L1 Base is https://api.upcloud.com/1.3, with the version in the path. A doubled slash (/1.3//storage) answers 404 with a message that names no path G deploy/upcloud/src/api.rs:22-24,602-605 M – no: the mock collapses empty segments and answers +39
L2 The terraform provider appends /1.3 itself. UPCLOUD_DEBUG_API_BASE_URL must NOT end in /1.3, or the provider calls /1.3/1.3/account T3 run 2026-09-21 (scratch/t3-mock-estate/mock.log) M – always: the mock answers 404 NOT_IMPLEMENTED on /1.3/1.3/…. The contract test pins it contract
L3 A bare GET /storage lists every public template, thousands of rows. /storage/private is the one a cleanup uses G api.rs:62-66 R – always –
L4 The server LIST carries no storage_devices. The storage list does carry labels MI U/api.rs:121,244 R – always 4
L5 Numbers arrive as a number in one reply and a string in the next (size, storage_size, core_number, memory_amount, boot_disk "0"/"1", remote_access_port, credits) G api.rs:11-13; MI U/api.rs:6-8,276 M/R – always (the_inconsistent_stringification_is_reproduced) –
L6 A device's storage_size is an INTEGER in the provider's SDK. A string kills every terraform create T3 2026-09-21 M (provider) – always (fixed on mock-estate-gunnar) tf
L7 Some write bodies want strings. A numeric stop timeout was accepted and silently ignored PGO grow_cloud.rs:73-76,207-208; PGO upcloud.rs:553-555 M – no: the mock ignores timeout either way. Harmless –
L8 Errors are {"error":{"error_code","error_message"}}. A 403 auth failure is problem+json with correlation_id and no error object G api.rs:544-565; PGO upcloud.rs:100-112 M – always 1
L9 The API occasionally answers 502 on a read, so a failed poll is not fatal G wait.rs:102-106 R – no: the mock only has 503-on-write +40
L10 Labels come in two shapes: storage [{key,value}], server {"label":[…]}. Filtered with ?label=k%3Dv. Keys are 2–32 printable ASCII and cannot start with _ MI U/api.rs:9-10; MO monetize-cloud/src/lib.rs:44 R – always for the shapes; no for key validation (the error code is unknown, so it is not modelled) –
L11 created is sent on storage rows, on both the list and the detail PGO upcloud.rs:682-685 M 2026-09-20 – always, plus WithholdCreatedField (a hypothesis, rate 0) 10
L12 Server create answers 202 in the docs and fixtures. The mock answers 201. The provider accepts both MI U/tests.rs:655 R – desc (left at 201: unmeasured) –
L13 PUT /server/{uuid} (boot_order) answers 202 PGO RESUME-2026-09-19:27-32 M – wrong: mock 200 +41
L14 A chunked PUT to the direct-upload URL answers 411 Length Required G http.rs:236-242 R – wrong: mock 400 MOCK_UPCLOUD_CHUNKED_REQUEST +42
L15 The direct-upload URL is the credential. No bearer token is sent to it G api.rs:175-177; G http.rs:67-70 R – always –

2. Credentials and accounts

# What UpCloud does Source M/R Timing mock #
L16 A dead UpCloud token answers 401 AUTHENTICATION_FAILED, a clean 401 PGO ROTATION-2026-09-14.md:126-130 M – no: the mock only has "no header = 401" +43
L17 A live token on an overdrawn account (credits −963.207) still answers 200 PGO ROTATION:130 M – always (nothing checks credits) –
L18 A "revoked credential" answers lists 200 with ZERO rows and 403 on details. Verified against Scaleway, not UpCloud; kept as a generic cloud-provider fault (owner ruling 2026-09-21): the only dated measurement (2026-08-26) is the Scaleway IAM change (mem: scaleway-iam-revoked-2026-08-26: "0 servers, 403 on VPC"). A dead UpCloud token is a clean 401 (behaviour 43, fault DeadToken) PGO upcloud.rs:28-37; MI U/mod.rs:1055-1077 R (for UpCloud) – fault RevokedCredential 5
L19 A token for another account gets 403 SERVER_FORBIDDEN on a server uuid of the first account. A token for the wrong estate authenticates and succeeds everywhere PGO iso.rs:778-790; G deploy/upcloud/src/token.rs:6-32 M – no: the mock has one account –
L20 Tokens start ucat_, are 31 chars, Bearer. They are account-wide (only expires and allowed_ip_ranges). Sub-accounts can be scoped; whether a sub-account token inherits the scope is unmeasured G token.rs:69-70; PGO DATA-SET-GROWTH-FLOW.md:63,345 R – desc 35
L21 A GET /1.3/price transport failure surfaced as "credential could not be verified" PGO remote/net.rs:8-15; MI storm/src/main.rs:428 M – fault PriceTransportReset 2
L22 A firewall-rule read for a DELETED server answers 403 ERROR_AUTHENTICATION_FAILED, not 404. terraform refresh dies "Permission denied (status=403)" PGO estate.rs:925-944 M 2026-09-20 – always 1
L23 The list carries a uuid that the detail answers 404 for PGO estate.rs:1227-1231 R – fault DetailNotFoundForListedServer 36
L24 A live server's firewall answers the same 403 (a permission-scoped credential) inferred R – fault FirewallForbidden 35
L25 Quota exhaustion is 403 *_LIMIT_REACHED (e.g. MAXIOPS_STORAGE_LIMIT_REACHED). The yvra storage_maxiops limit is 10 240 GiB PGO DATA-SET-GROWTH-FLOW.md:212; DESIGN:31 M (limit) / R (code) – no +44
L26 402 INSUFFICIENT_CREDITS exists. UpCloud suspends resources when prepaid credits run out MI U/api.rs:552-560; PGO gunnar_rs_system.md:970 R – no (billing is out of scope) –

3. Server state machine and timing

# What UpCloud does Source M/R Timing mock #
L27 Server states are started · stopped · maintenance · error G api.rs:271; MI U/api.rs:239 R – always (no error) –
L28 Create server: maintenance for 98–105 s lib.rs history M 98–105 s always 6
L29 Delete: front/twin ≈60 s; an appliance with 4 member volumes stays in maintenance >5 min lib.rs history M 60 s / >300 s always 7
L30 terraform destroy reports complete while the 4-member appliance is still maintenance. The provider does not wait for the server to vanish; it deletes the template disk and the data volumes right after the DELETE T13 run: DELETE server → DELETE storage within the same second, servers still maintenance at "Destroy complete" M (provider) – always (reproduced by construction, see §9) 7
L31 Delete requires stopped. A destroy is verified by GET until 404, not by the DELETE status PGO ROTATION:135-137; G gunnar-loadbench/src/upcloud.rs:22-43 R – always 11
L32 Start → started in ~10 s for a disk boot PGO RESUME-2026-09-19:32 M 10 s wrong: start_ms = 20 000 +45
L33 A guest that powers itself off inside the start operation reads started BRIEFLY, then maintenance, then stopped: live re-image 2026-09-21 08:42, start 377 421 ms → started read 381 091 ms → maintenance 381 174 ms → stopped 530 945 ms. The 2026-09-19 reading "maintenance t+10…t+130 s → stopped t+140 s, never started" was polled every 10 s from t+10 s and could not see a window this short; its maintenance/stopped half stands. A waiter that wants started alone misses the window at a 2 s poll and times out on stopped (the 2026-09-19 18:12 run); --medium-powers-off (G plan.rs:676-714) accepts either PGO .reimage/reinstall-receipt.json (2026-09-21); PGO RESUME-2026-09-19:13-25; G plan.rs:668-714 M started at ~3.6 s for < 100 ms; pass 130–154 s always: started 3.6–3.7 s, maintenance to the pass end, stopped 46
L34 A soft stop on a wedged guest hangs for the full timeout, then the stop fails. The tools fall back to hard G main.rs:461; PGO grow_cloud.rs:210-245 R 60/120 s always (Looping/Panicked refuse a soft stop) –
L35 stop_server has no timeout_action → 400. That field belongs to restart mem: upcloud-reimage-install-loop M – no +47
L36 Stop/start on a server already in that state → 409 SERVER_STATE_ILLEGAL MI U/mod.rs:743,751 R – always –
L37 "Stopped" did not mean off: both hentor boxes still answered the next day PGO gunnar_rs_system.md:920-929 M – desc (T12's reap keeps the API from saying stopped over a running VM) timing
L38 The API never says an install finished. A box that answers nothing (xfs duplicate-uuid, no ICMP/22/2222) still reads started G plan.rs:46-49; mem: gunnar-iso-only-as-fixed-as-its-pin M – always (state never reads guest health; Guest::Panicked) 38-ish, see +46
L39 Plan change on a started server → 409 SERVER_STATE_ILLEGAL. Only a stopped server may change plan MI U/mod.rs:73-80 R – always –
L40 A plan change mints a Resize Backup. Unconfirmed: the only measured 44 GB Resize Backup came from the twin's FILESYSTEM resize (POST /storage/{uuid}/resize, L66), not from a plan change PGO live.rs:1377; lib.rs #9 R – always, and lost: the pending mint was dropped when a start arrived during the plan-change maintenance (found by the provider run, §9) 9, +48
L41 Out of stock: 412 out_of_stock on poweron and on create, for days. Verified against Scaleway, not UpCloud; kept as a generic cloud-provider fault (owner ruling 2026-09-21): mem: scaleway-out-of-stock-2026-09-09 records it on Scaleway fr-par-1. UpCloud's documented capacity refusal is 409 STORAGE_RESOURCES_UNAVAILABLE PGO grow_cloud.rs:499-503; MI U/api.rs:552-560 R (for UpCloud) days fault OutOfStock 3, 37
L42 A create that commits and then loses its reply – R – fault CommitThenDropReply –
L43 Two storages are provisioned in parallel by UpCloud; only a serial client made it look slow G lib.rs:79-84 M 2×110 s → ~115 s always (independent transitions) –

4. Boot, firmware, console

# What UpCloud does Source M/R Timing mock #
L44 SeaBIOS only, no UEFI, no firmware switch. A UEFI-only medium gives "noll utdata"; SeaBIOS executes the FAT efiboot.img ED iso_writer.rs:452-460; G installtest/firmware.rs:70-72 M – always (KVM half refuses OVMF) 15
L45 The installed disk must boot under SeaBIOS too: a UEFI-only installed disk never boots. The fix is an MBR plus a BIOS stage. KVM proves it boots under SeaBIOS/i440fx ED KI/src/install.rs:879-885; mem: tunnr-kernel-hotplug M – desc: the virtual guest assumes any boot disk boots; T12's DraupnirGuest boots the real disk under SeaBIOS 15 (T12)
L46 Machine shape: -machine pc (i440fx), 2 virtio NICs (public, utility), virtio-blk boot disk, CPU AMD EPYC 7542 (EPYC-Rome), ACPI, metadata at 169.254.169.254. The bus topology is unmeasured ED BT/target/presets.rs:1050-1071; G probe.rs:99-102 M – desc (T12's guest) –
L47 No UART at ttyS0/0x3f8; ttyS1/ttyS2 exist at 0x2f8/0x3e8. With console=ttyS0, every write gets EIO and PID 1's first print! panics. VGA over VNC is the only console ED KI/src/serial.rs:258-268; G firmware.rs:72-76 M 2026-08-29 – desc: "no serial console" (the KVM half boots -serial none); ttyS1/2 are not modelled 13
L48 No QMP, no serial console API. VNC is the only screen PGO screendump.rs:37-41 R – always 13
L49 The ISO must be attached TWICE: an IDE cdrom for SeaBIOS and a virtio disk for Linux (the kernel has no ATA). SeaBIOS cannot boot the virtio copy G deploy/upcloud/src/lib.rs:8-32 M – always 25
L50 boot_order can be written on a started server but is honoured only at the next HYPERVISOR start; a guest reboot keeps the firmware order G api.rs:273-274; mem: upcloud-reimage-install-loop M – always (implicit: a looping guest stays looping after a PUT). Now tested 17, +49
L51 A re-image killed mid-install loops install→reboot (CD first) for 2+ h, rewriting the disk each pass G plan.rs:10-17 M 2 h+, each pass ≥900 s fault InstallerLoop 17
L52 POST /server/{uuid}/cdrom/eject works on a STARTED server (200); the device stays with an empty storage; the next reboot boots the disk G api.rs:101-108 M eject at 120 s is safe always 16
L53 Ejecting too early (before SeaBIOS has read kernel and initrd) falls back to the old disk G plan.rs:50-58 M <120 s no (the virtual guest has no early window) –
L54 Accepted boot orders: disk, cdrom,disk, disk,cdrom, cdrom. cdrom,disk with no medium falls through to the disk PGO estate.rs:430-431 R – always (disk,cdrom reads back as disk) –
L55 Guests boot +2 h (CEST) or +1 h (CET): no kvmclock (CONFIG_HYPERVISOR_GUEST off) → reads CMOS as local time. The Ubuntu front is within 468 ms G recipe.rs:584-610 M (mechanism inferred) 7 198 668 ms fault GuestReadsRtcAsLocalTime 19
L56 A re-image mints a NEW SSH host key every time (also [host]:2222). A re-created front keeps its public IPv4, so the key changes on the same address mem: gunnar-estate-3vm-2026-09-15; PGO front.rs:18-21 M – always, plus HijackedName 33

5. Console (VNC)

# What UpCloud does Source M/R Timing mock #
L57 Plain RFB, no TLS, at <zone>.vnc.upcloud.com:<port>. The password is in remote_access_password. Banner RFB 003.008 (read 12 bytes, not 40) mem: upcloud-vnc-port-toggle M – always (host and port). The mock never reports a real host: <zone>.vnc.mock.invalid (RFC 2606) with no guest; with a real guest (mock-upcloud-kvm) the guest's own VNC on 127.0.0.1:<port>, which answers RFB 003.008, goes stale on stop/start (14) and is reconciled by the toggle (59) 31
L58 The port is reallocated on every stop/start and the API keeps reporting the stale one (a SYN hangs) mem: upcloud-vnc-port-toggle M – fault StaleVncPort, default ON 14
L59 Cure: PUT remote_access_enabled no → pause → yes, then re-read. The PUT returns before the console moves; without the ~2 s pause the old port comes back G console.rs:17-29,96-99; PGO screendump.rs:896-927 M settle 2 s no: the toggle is instant in the mock +50
L60 The toggle answers 409 while the server is maintenance (retry) mem: upcloud-vnc-port-toggle M – no: a PUT is accepted in any state +51
L61 The toggle rotates the VNC password. The password is 8–32 alphanumerics; VNC-Auth type 2 only PGO screendump.rs:455-460; G console.rs:67-73 R – no (password is echoed back unchanged) –

6. Storage: create, import, clone, attach, delete

# What UpCloud does Source M/R Timing mock #
L62 Direct-upload import: the PUT takes ~5 s for 43 MB; the import says completed while the storage is still syncing 100–130 s more (two clocks). Storage path: maintenance → syncing → maintenance → online G lib.rs:55-77; G tests/clone_probe.rs:19-27 M PUT 1–5 s; import→online 110–114 s always; short one maintenance: the mock goes syncing → online 23, +52
L63 The import object carries read_bytes/written_bytes and real md5sum/sha256sum (possibly upper-case hex). The uploader's PUT reply is the same fields un-enveloped, {"written_bytes": …, "md5sum": …, "sha256sum": …}; only GET /storage/{uuid}/import wraps them in storage_import G upload.rs:5-8; G plan.rs:472-492 (resp.json() read top-level, live done: true 2026-09-21) M – always (real digests; the PUT reply was wrongly enveloped until 2026-09-21) 24
L64 A truncated upload still boots (ISO-9660 header is at the front) and fails ~15 min later G upload.rs:11-16 M ~15 min no (a guest property; out of scope) –
L65 A clone goes maintenance → online in 47 s with NO syncing; the call takes 728 ms; the tier is inherited. It cannot start until its source is online G tests/clone_probe.rs:5-32 (MEASURED 2026-09-20) M 728 ms; 47 s wrong: the mock's default makes a clone sync 100–130 s ("NOT MEASURED"). The measurement exists 26, +53
L66 POST /storage/{uuid}/resize (filesystem) first takes a Resize Backup <n> the size of the volume, hands it back as resize_backup, and leaves it billed. It carries the origin's labels and origin PGO upcloud.rs:20-26,538-545; PGO live.rs:1377-1398 M 2026-09-20 44 GB since 2026-09-19 always 34
L67 terraform filesystem_autoresize plus delete_autoresize_backup = true deletes that backup itself (PUT size → POST resize → DELETE backup) T13 provider run §9 M (provider) – always (the provider does it; the mock answers) contract
L68 A storage only grows: smaller → 400 SIZE_INVALID "The new size must be greater than the old size." The same-size retry gets the same error PGO DATA-SET-GROWTH-DESIGN.md:23; FLOW:271 M – wrong code: the mock says STORAGE_INVALID_SIZE and accepts a same-size PUT 22, +54
L69 A detached 1→2 GiB grow: 200, then online again 37 s later PGO DESIGN:24; mem: gunnar-grow-no-restart M 37 s no: the mock grows instantly, with no maintenance +55
L70 Resize while attached. The docs contradict themselves ("server must be stopped" vs STORAGE_ATTACHED "must first be detached"). Both readings refuse it while the server RUNS. mem: gunnar-grow-no-restart says "attached resize refused" PGO DESIGN:25,282 R (docs) – no: the mock (and so terraform through it) grew four volumes attached to a STARTED appliance, §9 +56
L71 Volume ceiling 4096 GB, floor 10 GB (per the estate's code) PGO grow.rs:919-920 R – no (the codes are unknown) –
L72 A storage not online cannot be attached G plan.rs:527-543 R – always –
L73 Attaching a storage that is already attached → 409 STORAGE_ATTACHED. UpCloud refuses a storage another server holds MI U/mod.rs:651; PHO front.rs:151-154 R – no: the mock attaches the same storage twice +57
L74 address: "virtio" lets UpCloud pick the slot; a named address (virtio:3) is honoured MI U/grow.rs:70-71 R – no: the mock ignores the requested address and takes virtio:<count>, which collides after a middle detach +58
L75 Delete an attached storage → refused (409). Delete a clone's source while the clone is still being made → refused. A second DELETE → 404 G api.rs:518-520; PGO upcloud.rs:286-288 R – attached: always; clone source: no +59
L76 DELETE /storage/{uuid} on a storage in the wrong state → 409 STORAGE_STATE_ILLEGAL MI U/tests.rs:1032 R – no: a storage in maintenance deletes +59
L77 DELETE /server/{uuid}?storages=1&backups=keep takes the OS disk with it MI U/mod.rs:1022 R – always (backups ignored) –
L78 Detaching does not stop billing; a stopped server still bills its storage G api.rs:510-516 M – n/a (billing out of scope) –
L79 A young storage looks like an orphan until created is read PGO upcloud.rs:17-19 R – always 10

7. Hot-plug and growth

# What UpCloud does Source M/R Timing mock #
L80 Detach from a STARTED server: IDE → 409 IDE_HOTPLUG_UNSUPPORTED; virtio → 511 HOTPLUG_FAILED (the guest did not ack the ACPI _EJ0, i.e. the kernel had no PCI hot-plug) mem: upcloud-reimage-install-loop; G api.rs:101-106 M 2026-09-14 – wrong: the mock answers 409 VIRTIO_HOTPLUG_UNSUPPORTED, an invented code, and refuses every virtio detach on a started server, even for a guest that can hot-plug 16, +60
L81 Storage hot-plug is a GUEST property. It needs HOTPLUG_PCI(_ACPI/_PCIE/_SHPC), PCIEPORTBUS and PCI_MSI=y (without MSI, _OSC refuses OS control and pciehp is dead). Proven only under KVM; never measured on UpCloud T tunnr-hardening.config:170-188; mem: tunnr-kernel-hotplug K eject ack ~44 ms no: the mock has no notion of the guest's kernel +60
L82 A virtio ATTACH to a started server answers HOTPLUG_FAILED with the old kernel. One source says 409 SERVER_STATE_ILLEGAL "The server is started." for attach-to-started; HOLGER-PLAN says attach works started or stopped. Hot-attach is unmeasured on UpCloud mem: tunnr-kernel-hotplug; G api.rs:332; H HOLGER-PLAN.md:1726-1732 M (old kernel) / R – lax: the mock always accepts a virtio attach on a started server +60
L83 Resizes are batched in a 10 s window; growth at 80 %/65 %/90 %, at most once per 3600 s mem: gunnar-grow-no-restart; MO grow/batch.rs:4-18 – 10 s n/a: this is the ESTATE's policy, not UpCloud's. The mock must not batch –
L84 Members already resized before an interrupted growth stay bigger and stay billed PGO live.rs:1463-1467 R – always (never shrinks) 22
L85 No live growth has ever run on UpCloud (allow_grow=false) PGO STINK-LIST:138-142 R – – –

8. Network and firewall

# What UpCloud does Source M/R Timing mock #
L86 Two NICs: index 1 public (eth0), index 2 utility (eth1). The guest does not bring up the utility NIC by itself G recipe.rs:1306-1308 M – always (default pair) –
L87 The utility network is NOT one flat /22: hosts sit on different /22s (10.13.0.x, 10.13.2.x, 10.13.7.x, 10.13.8.x) reached via 10.0.0.0/8 via 10.13.4.1, delivered as DHCP option 121. A guest that ignores 121 has no route off its /22 G dhcp.rs:142-165 M 2026-09-20 – always, plus fault GuestIgnoresDhcpOption121 29
L88 Utility and public IPs change on every re-lay mem: gunnar-estate-2026-09-21 M – always, from RFC 5737 TEST-NET only (192.0.2/24, 198.51.100/24, 203.0.113/24). The pool used to be REAL UpCloud addresses, one of them a stranger's host with port 22 open, so a mock step that connected to a handed-out address (verify, ssh to the front, the banner check) reached the internet. TEST-NET is not routed; estate::address_tests holds every address the estate hands out to it 12
L89 The utility net offers option 3 and may offer a /0 via 121; accepting either gives a second default route. The ACK carries option 6 (DNS) G pid1.rs:1308-1362; G dhcp.rs:40-45 M/R – desc (the DhcpOffer carries 121 and no router) 29
L90 No hairpin NAT – M – always 30
L91 IPv6: UpCloud offers it (price 0), and each interface picks ip_address_family. "No IPv6" is this estate's LAW, not a provider limit PHO main.tf:131; PGO main.tf:48; mem: korp-start-initrd-devtmpfs R – wrong: the mock renders IPv4 whatever the body asks, so a declaration that asked for IPv6 would pass here unseen +61
L92 Firewall off, or a server with no rules, is wide open. Rules are evaluated top-down, first match wins; that is why every rule set ends in a catch-all drop PHO terraform_gate.rs:635-649; PGO reimage.rs:1228-1250 R – no: the mock stores rules and evaluates none +62
L93 The catch-all inbound drop also drops the REPLIES to outbound UDP (DNS, NTP); TCP replies pass. dig +tcp answers, +notcp times out G gunnar-clock/src/lib.rs:33-47; mem: gunnar-clocks-front-is-root M 2026-09-14 – fault UdpInboundDropped, default ON, everywhere. It is really a consequence of the rules (L92); a server with firewall off gets its UDP replies. The same holds for the installer link's ANSWER to a box that LAYS another (live 2026-10-14, ONE box → test2.gunnar.rs): Estate::wireguard_pairs, /mock/wireguard/{layer}/{laid} 18, +62
L94 The firewall also filters the UTILITY interface PGO terraform/main.tf:476-489 M 2026-09-13 – no +62
L95 A DROP is silence until the caller's timeout; an admitted port with no listener answers RST PGO live.rs:7963-7975 M 2026-09-20 knock 5 s partly: Reach::Refused exists, a drop does not +62
L96 New servers came up started as bare Ubuntu with TCP/22 open PGO CUTOVER-2026-09-13.md:106-107 M – always (TemplateFirstBoot) 21
L97 First boot of the Ubuntu template: unattended-upgrade-shutdown holds dpkg lock-frontend; sshd may refuse for a few seconds after apt lib.rs #21; mem: gunnar-estate-3vm-2026-09-15 M a few s always / desc 21

9. The terraform provider (UpCloudLtd/upcloud 5.44.1) against the mock

MEASURED by this lane on 2026-09-21. The real provider binary (built with go1.25.3) ran through a full cycle for both estates' declarations: init, apply, plan (no changes), plan with each variable perturbed, apply of growths, and destroy. The mock ran at --speed 20000 --log.

The request contract. These are the only calls 5.44.1 sends. Both estates send the same set. Zero MOCK_UPCLOUD_NOT_IMPLEMENTED.

call gunnar cycle holger cycle notes
GET /1.3/account 12 12 one per provider configure; the credential probe
GET /1.3/plan 12 12 before every server read and write
GET /1.3/zone 12 12 likewise
GET /1.3/storage/template 3 3 template title → uuid, once per server create
POST /1.3/server 3 3 the create carries interfaces, login, template, devices
GET /1.3/server/{uuid} 22 24 refresh, and the provider's own polling
POST /1.3/server/{uuid}/stop 3 3 destroy stops before it deletes
DELETE /1.3/server/{uuid} 3 3 never with ?storages
GET /1.3/server/{uuid}/firewall_rule 6 6
PUT /1.3/server/{uuid}/firewall_rule 6 6 the whole rule set in one call
POST /1.3/storage 5 2
GET /1.3/storage/{uuid} 20 8
DELETE /1.3/storage/{uuid} 8 5 the template disk is deleted separately, right after the server DELETE
growth only: PUT /1.3/server/{uuid}, POST …/start, PUT /1.3/storage/{uuid}, POST /1.3/storage/{uuid}/resize yes – plan change; template grow with autoresize; data volume grow

Provider behaviours, measured through the mock.

# What the provider does M/R mock
L98 tier on upcloud_storage is ForceNew (maxiops → hdd: "must be replaced") M (5.44.1) n/a (provider)
L99 login on upcloud_server is ForceNew (a changed key replaces the front, and its firewall rules follow via server_id) M (5.44.1); PHO adopt.tf:21-28 n/a
L100 In place: plan, template size (both up and down in the plan, so a shrink is only refused at apply by the API), data volume size, hostname, labels. boot_order is ignored by ignore_changes M (5.44.1) n/a
L101 A data volume grow on a RUNNING server is sent straight through: the provider does NOT stop the server for an upcloud_storage size change. Against UpCloud (L70) that apply fails M (5.44.1) the mock accepted it → +56. Re-run after: apply -var appliance_data_disk_gb=20 fails 409 STORAGE_ATTACHED on all four members, while plan change and template grow still apply. The estate cannot grow its data volumes with terraform while the appliance runs
L102 A plan change: stop → PUT server {plan} → start immediately, without waiting for the PUT's maintenance to clear M (5.44.1) the mock let the start overwrite the pending transition and LOST the plan change's Resize Backup → +48
L103 A template grow with filesystem_autoresize: stop → PUT storage → POST resize → DELETE the Resize Backup → start M (5.44.1) always
L104 Destroy: stop → DELETE server → DELETE template storage at once → DELETE data volumes. "Destroy complete" while the servers still read maintenance M (5.44.1) always
L105 ~> 5.44 in both versions.tf will take 5.45 when it ships. Proposed: = 5.44.1 (owner rules; see the report) – contract test asserts the lock
L106 The provider reaches the mock only through the undocumented UPCLOUD_DEBUG_API_BASE_URL. If a release drops it, a "mock" run goes to api.upcloud.com R +63: the mock records which tokens it heard, so a verb can prove the provider spoke to it before apply/destroy
L107 --speed 0 breaks the provider: the mock closes a transition halfway per poll, and the provider's wall-clock budget expires first (SERVER_STATE_ILLEGAL on a resize). Drive it at 20 000 M (T3) desc (clock.rs)
L108 After the grow engine enlarges a data volume, the next plan is an IN-PLACE shrink, not a replace: # upcloud_storage.appliance_data[0] will be updated in-place / ~ size = 20 -> 10, and the apply is refused 400 SIZE_INVALID per volume, exit 1, nothing destroyed. Every later apply then exits 1, whatever else it carries. The fix, lifecycle { ignore_changes = [size] }, gives "No changes". A replace of the data still comes from tier, zone or a lower count; only prevent_destroy stops those. Both estates were checked M (5.44.1 through the mock, 2026-09-21) n/a (provider plus declaration)
L109 A server PUT changes hostname/title in place M (5.44.1) was dropped by the mock, so terraform refused its own apply → 64
L110 POST /server/{uuid}/cdrom/load {"storage_device":{"storage":…}} puts a medium into the existing cdrom device; with CD first, the next hypervisor start boots it. Refused into a full tray (409 CDROM_DEVICE_IN_USE, eject first), with no cdrom device (404 CDROM_DEVICE_NOT_FOUND), for a storage not online (409 STORAGE_STATE_ILLEGAL). The provider's type: cdrom storage requirement is not enforced (L49 attaches normal uploads as cdrom) R (UpCloud 1.3 docs; not measured) always → 65
L111 An import straight after POST /storage is accepted, with no wait for online. gunnar deploy/upcloud/src/plan.rs:550-553 (start_medium) calls create and then import with no wait and no retry (api.rs:350,364-378), and one refusal fails the run. The live re-image private-gunnar-ops/.reimage/reinstall-receipt.json (2026-09-21 08:42) logged media-cdrom at 24 504 ms and media-virtio at 47 439 ms, with done: true. The 2026-09-20 clone probe (tests/clone_probe.rs:160-167) did the same. Unverified: what state the new storage READ at that moment. The 3 s maintenance after a create is the mock's own number, never measured M (acceptance) / R (state) was refused 409 STORAGE_STATE_ILLEGAL by the mock, so a mock re-image stopped at the first import → 66
L112 Which spelling of a number in a request body is VERIFIED against the account (lane T14, 2026-09-21). PUT /storage/{uuid} size: the JSON NUMBER is verified — monetize-impl's grow ran live on 2026-09-14 with {"storage":{"size":<number>}} and a measured 37 s to online (monetize-cloud-impl/src/upcloud/grow.rs:78); UpCloud's SDK declares it int; provider 5.44.1 refuses a string (T7). The STRING was never sent to the account — private-gunnar-ops' "a JSON number is refused" came from 30979d4 ("the shapes are asserted, never sent") and is withdrawn; its grow now sends the number. POST /server/{uuid}/stop timeout: the STRING is verified — sent live by gunnar-upcloud (every re-image since 2026-09-07) and monetize-impl ("60"); the NUMBER is UNVERIFIED (gunnar-upcloud's "accepted and ignored" names no run). PUT /server/{uuid} plan: a name, no number. M (size: number; timeout: string) / R (the other spelling of each) the mock ACCEPTS both spellings of both — it must not pretend to know the unverified half → 68
L113 A REBOOTING medium (composed without install_then=poweroff; every korp-installer < 0.1.4; the 2026-09-14 loop): its installer start reads started for the whole pass, 900–1100 s, and the pass ends in a guest reboot — the CD still loaded and first installs again (17), otherwise the disk boots. cdrom/eject answers 200 on the started box mid-pass G deploy/upcloud/src/plan.rs (every measured pass ≥ 900 s); live 2026-09-08 (1000–1100 s); eject on a started box MEASURED 2026-09-14 (mem: upcloud-reimage-install-loop) M
L114 A server's simple_backup and a backup_rule on a disk it holds cannot coexist: 409 BACKUP_RULE_CONFLICT at server create, server modify, storage modify and storage attach ("The backup rule of a storage conflicts with the simple backup rule of the server"). terraform-provider-upcloud's simple_backup: "you cannot have simple backup set for a server, and individual backup_rules on the storages attached to the server". A shop-laid front carries simple_backup = "0300,dailies" (S6, 5.4); the kit's own fronts carry a storage backup_rule — an adoption that wrote both would fail UpCloud API 1.3 docs (server create/modify, storage modify/attach error tables); terraform-provider-upcloud docs upcloud_server.simple_backup R

10. Timing table, provider vs guest

event UpCloud (outside) guest (inside) mock Timings before after
server create 98–105 s – 98–105 s same
start → started (disk boot) ~10 s – 20 s 10 s
start with an installer that powers off started at ~3.6 s for < 100 ms, then maintenance, stopped 130–154 s after the start (2026-09-21 receipt; 2026-09-19 polled too coarsely to see the started) install 2–5 s (UART), 10 s KVM started 3.6–3.7 s, maintenance, stopped 130–146 s the brief started is real; a waiter must accept stopped after maintenance
re-image wall (ladder) 110–253 s (3 runs); older, heavier runs 1000–2865 s 2–5 s observers only same
storage create – – 3 s same (unmeasured)
import upload 43 MB 1–5 s – 120 ms/MiB same
import sync 100–130 s after "completed" – 100–130 s same, + trailing maintenance
clone 728 ms call, 47 s to online, no sync – 3 s + 100–130 s sync 47 s, no sync
detached grow 37 s back to online – instant 37 s
delete front / 4-member appliance ~60 s / >300 s – 60 s + 65 s/vol same
VNC toggle settle ~2 s – 0 2 s

Misunderstandings: things once believed about UpCloud that were false

The mock must not encode these. Each one names what the mock did about it.

# The belief What is true Mock
X1 The API sends no created, so a young volume looks like an orphan created is on the list and the detail (MEASURED 2026-09-20) corrected (behaviour 10); a hypothesis fault remains
X2 "The boot disk is the smaller one" / "class then largest" is safe No size rule serves both the appliance (50+4×32) and the twin (10+20). It is fixed by declaring korp.data.disks n/a (installer), but see +58 on addresses
X3 A clone syncs like an import, so it only saves the 5 s upload A clone skips syncing: 47 s against 114 s (MEASURED). But it is serial after its source, so two parallel imports win the mock still encoded the belief → fixed, +53
X4 A virtio detach on a running server answers a 409 *_HOTPLUG_UNSUPPORTED It answers 511 HOTPLUG_FAILED when the guest cannot hot-plug. monetize-impl only catches the 409, so a real 511 maps to Unavailable and skips its fallback the mock encoded the invented 409 → fixed, +60
X5 Outbound UDP is blocked The inbound catch-all drops the REPLIES desc → +62
X6 Setting boot_order=disk by API stops an install loop Honoured only at the next hypervisor start; eject is the primitive correct already; now tested (+49)
X7 stop_server takes timeout_action 400; it belongs to restart +47
X8 started means healthy It says nothing about the guest (xfs brick read started) always
X9 stopped means off hentor boxes still answered next day desc
X10 A re-image is done when started is read A powering-off installer never reads started: maintenance → stopped the mock encoded the belief → +46
X11 An IDE cdrom needs CONFIG_ATA A donor kernel with ATA off booted (the ISO rides virtio for Linux) n/a
X12 UpCloud has no UART Only ttyS0 is missing; ttyS1/ttyS2 exist desc
X13 The utility network is one flat /22 Several /22s joined by an option-121 route correct (29)
X14 The utility network is unfiltered The firewall filters it too +62
X15 Detach gives the storage back Detached storage still bills n/a
X16 A present credential is a valid one A dead token started cleanly and failed a day later at the first price read always (2)
X17 412 out_of_stock and "0 rows + 403" are UpCloud behaviours Both dated measurements are Scaleway (fr-par-1 2026-09-09; IAM 2026-08-26). UpCloud's dead token is a clean 401 (MEASURED) relabelled, not removed (owner ruling 2026-09-21): faults 3/37 and 5 are marked "verified against Scaleway, not UpCloud; kept as a generic cloud-provider fault" in lib.rs and faults.rs; they stay armable and in the seeded weather, so no storm seed changes
X18 Resizes are batched in a 10 s window by UpCloud The estate batches; UpCloud resizes each call n/a, the mock must not batch
X19 Resize while attached works (terraform has no stop) Refused while the server runs (docs, both readings) the mock let it through → +56
X20 The ladder's install-time of 110–253 s is the installer It is the provider's wall time; the installer is 2–5 s correct (32)
X21 The mock's 38 behaviours were all measured on UpCloud 3, 5 and 37 are Scaleway; 26's "not measured" was measured; 16's virtio code was invented this ledger
X22 A plan change mints the 44 GB Resize Backup The measured 44 GB came from the twin's FILESYSTEM resize. Plan change → backup is unconfirmed 9 kept as-is, labelled unconfirmed
X23 Reads are consistent with writes A resize answers 200 while the size still reads old; an attach answers 200 while it still reads detached (reported) partly (+55 gives the resize a maintenance window)
X24 terraform destroy "complete" means the servers are gone They are still maintenance for minutes always

Counts

Added by lane T13 (see lib.rs)

# behaviour kind
39 an empty path segment (/1.3//storage) is 404, as UpCloud answers always
40 a read answers 502 fault ReadBadGateway
41 PUT /server/{uuid} answers 202 always
42 a chunked upload answers 411 always
43 a dead token is a clean 401 everywhere fault DeadToken
44 403 MAXIOPS_STORAGE_LIMIT_REACHED on a create/grow past the quota fault StorageLimitReached
45 start → started in 10 s always (timing)
46 a guest that powers off inside the start never reads started always (the guest decides; the clock gives the floor)
47 stop_server with timeout_action → 400 always
48 a start during a plan change's maintenance is refused 409, and the Resize Backup is not lost always
49 a PUT of boot_order does not stop a looping guest; eject does always (test)
50 the VNC toggle settles after 2 s; read too soon, the old port comes back always
51 a PUT to a server in maintenance → 409 always
52 the import's storage goes syncing → maintenance → online always
53 a clone is maintenance → online in 47 s, no sync always (measured); CloneSkipsSync is kept as a no-op alias
54 shrink and same size → 400 SIZE_INVALID always
55 a size grow puts the storage in maintenance for 37 s always
56 a size grow of a storage attached to a non-stopped server → 409 always; fault ResizeRequiresDetach for the stricter reading
57 attach of an already-attached storage → 409 STORAGE_ATTACHED always
58 a requested address is honoured; virtio picks the first free slot; guest names follow the slots (vda = virtio:0) always
59 delete of a storage in maintenance, or of a clone's source mid-clone → 409 STORAGE_STATE_ILLEGAL always
60 virtio hot-plug on a started server is the GUEST's: a hot-plug kernel succeeds; without one → 511 HOTPLUG_FAILED fault GuestKernelLacksHotplug (per server at create)
61 an interface asking IPv6 gets an IPv6 address always
62 the firewall evaluates: off or no rules = open; top-down; drop = silence; utility filtered too; UDP replies are dropped only by a rule set that ends in a drop always (net::firewall_admits)
63 the mock records the tokens it heard (by digest), so a verb can prove the provider spoke to IT before apply/destroy /mock/heard
73 a server's simple_backup beside a backup_rule on a disk it holds is 409 BACKUP_RULE_CONFLICT (L114; S6/S7 lane, 2026-10-11) always