nordisk/monetize

5. The estate lifecycle: lay, pairing, the pull channel, upgrades, grow, the Guardian

From monetize main c9b473a5; paths are under crates/ unless absolute. Chapter 4 has the money side of the same orders.

5.1 The run and its phases

Every private estate is one EstateRun under <data>/monetize/estate-orders/<company>/: run.json (written atomically after every transition and before the next step), the cloud journal (an intent before each create, the id confirmed after) and laid.json (monetize-server/src/estate_order.rs:28-39, 3447, 3554). The driver tick runs every 2 s (cli.rs:991).

ordered ─▶ held ─▶ (waiting_dns) ─▶ laying ─▶ installing ─▶ paired ─▶ captured
   │                                                                     │
   └─▶ released (unpaid)        failed ─▶ owed / refunded          grace ─▶ suspended ─▶ deleting ─▶ deleted
                                                                   terminating ─▶ terminated

(estate_order.rs:151-200; is_term = captured, grace, suspended, deleting; is_gone = released, deleted, refunded, terminated; holds_a_slot = laying, installing, paired.)

5.2 Lay, step by step

1. HELD. A customer domain goes to waiting_dns first (:5810-5855). Admission needs a free slot (max_open = 2 lays at once), first ordered first laid, and room under the spend cap (:5893-5949); otherwise the order queues (chapter 4 §4.5).

2. LAYING (lay(), :5968; lay_now(), :6090):

   • the cloud and the journal are opened;
   • the plan is built by the product's guest, op 91 lifecycle.lay_plan, from the buyer's sizes and the twin plan (lifecycle::plan_of, :6106-6113; the twin is sized as main / 8, at least 1 core and 4096 MiB, monetize-cloud/src/lay.rs:745-747);
   • the price is read again (price::recheck); over the spend cap the run goes back to HELD, never to a refusal (:6119-6145);
   • dress_plan presigns the release media objects (one hour, MEDIA_PRESIGN_SECS) and gives each machine its kind's medium and templates (:6072-6088, 2328);
   • each machine's pairing seed becomes its user_data, from the custody (give_seeds_as, :6170-6184);
   • Cloud::lay(plan, journal) (:6185): every disk, then every server with its disks, labelled monetize_ref/monetize_res, journaled (lay, lay-disk, lay-server, lay-medium per machine, lay-metadata-off, lay-firewall; lay.rs:48-85). A machine boots from a per-machine clone of the seedless release medium attached as its cdrom, or from a template cloned as disk 0 (lay.rs:279-324). Unavailable parks; OverCap goes back to HELD; anything else fails the run. run.laid is set the moment the cloud answers.
   • the DNS records (type A, proxied: false: <company>.<apex> and the product's served prefixes "", www., git.) are written from the front's public IPv4 through the zone's writer (write_estate_dns, :5990-6063). Only the operator's own site may take over existing A/AAAA/CNAME names (ALERT estate-orders.dns-takeover); a sold estate's writer refuses any record it did not make.

3. INSTALLING (install(), :6431): waits until the records answer (settled), then the installer runs (§5.3). On PASS the guardian is told (fleet::estate_installed) and the run moves to PAIRED; the install deadline is install_deadline_mins = 60; a stuck install is explained after 10 min and alerted after 20 (install_why.rs:27-29).

4. PAIRED: the release clones are given back and the metadata turned off (seeds_off, :8068-8089). Then the ready signal, capture (:6698-6806), in three steps: (1) a box called home (fleet::heard_since); (2) the primary says it serves (fleet::serves_since); (3) the sale check over the PUBLIC front, https://<host>, every 30 s. Then CAPTURED with paid_until = now + term, and the ready mail (estate.ready) once.

A run whose capture is not reached within capture_within_secs = 86 400 from the lay fails (:5745-5754).

5.3 Pairing: who holds which key

The shop pairs a sold estate itself (monetize-shop-pairing, registered as [estate_orders] pairing = "shop-pairing").

(monetize-shop-pairing/src/lib.rs:250-433, 773-803, 1093-1099; estate-side/src/pump/key.rs; monetize-front/src/attach.rs.)

The sequence:

1. At payment the material and each machine's seed (seed.<kind>.toml) are minted and sealed in the custody (Installer::mint_for, lib.rs:1220-1227).
2. At the lay each machine gets its seed as user_data.
3. ShopPairing::install (lib.rs:1263-1364): read laid.json, the material and the first-login invite; build the shared, front and box feeds (the Fleet door, fleet_instance, monetize_pubkey, fleet_at); run nordisk_pairing::installer over each box's WireGuard installer link; bundle the material back into the custody; PASS or FAIL.
4. The guardian is told: a role with a named key gets it pinned; otherwise a pin window of 10 min is opened for the role (pin_window_ms, monetize-guardian/src/runner.rs:65).
5. Trust on first use (Guardian::first_call, guardian/src/fleet.rs:77-133): inside an open window, the box_key a box presents is pinned if the call verifies under it, and the window closes. A key already pinned is never replaced by a call; an open window lets a NEW key replace a pinned one once (rule PIN-1).
6. A box laid again (recharge_as, estate_order/sold.rs:533-762): only roles in upgraded_by_shop, only a delivered estate; a box that is merely stopped is started, not relaid; a box that still stands is refused. The plan is relaid with that role's seed only, then forget_box and open_pin(now + 10 min); if that fails: ALERT estate-orders.pin … \site open-pin` opens it. The operator's site open-pin opens every unpinned role's window without end (UNTIL_PINNED); a pinned key is never replaced by it (monetize-server/src/fleet.rs:1650-1685`).

5.4 The pull channel: Fleet, fleet_at, the self-heal re-feed

Monetize never calls into an estate (monetize-proto/proto/monetize/v1/fleet.proto:5; monetize-product/src/pull.rs:3-9). Three calls, every one made by a box and signed with its fleet key under the domain nordisk-fleet-call-v1, fresh within 5 min (monetize-embed/src/fleet.rs:21-30):

The box's loop (estate-side/src/pump/duty.rs:103-224): report → adopt next_report_ms (clamped 1–30 s; the server says 30 s) → pull orders → check each (the signature; for this box; seq strictly higher than the last; not expired; not more than 5 min ahead) → persist last_seq BEFORE it runs → PumpBox::run → report again at once so the acks land. State: <data>/fleet/pulled.json. A forged order is never acked and the sequence does not move.

The core order kinds (monetize-product/src/pull.rs:24-104), the ONE list every box must know:

plus two asks, grow_set and fetch; the guardian's relay kinds (recharge_from_twin, fence_primary, adopt_twin, review_twin, restate_entitlements) and the product's own kinds from its facts.

fleet_at is where a box dials the ONE box's Fleet door when the URL's host is not what it should dial (estate-side/src/pump/config.rs:17-30): the kit feeds it as [estate_orders.sale_feed] fleet_at = "<utility ip>:443", a private address only (private-monetize-ops/xtask/src/sells.rs:2281-2319), and the pairing feeds it to every box with fleet, fleet_instance, fleet_ca and monetize_pubkey (the seam keys, monetize-shop-pairing/src/lib.rs:1810-1816).

The self-heal re-feed (seam_heal, estate_order.rs:7311-7368): a PAIRED estate no box of which has called home 2 min after pairing (SEAM_HEAL_AFTER_MS) gets the seam keys fed again over each box's installer link (Installer::refeed_seam, budget 180 s), with the pin windows opened again, every 10 min (SEAM_HEAL_EVERY_MS) until a box is heard. Nothing is laid or charged. The operator's separate site refeed is the refeed order.

5.5 Upgrades

A sold estate or a site is upgraded by the ONE box through UpdateEstate (estate_order.proto:172-193: UpdateEstate { estate, release, front_first }, UpdateStatus, CancelUpdate, SetLiveUpdate; the operator's alone). The rules (estate_order/update.rs):

• The order is twin, main, front: the front last (ORDER, :47-49; the facts' update_order and front_role, "the role that drives an update and is upgraded LAST").
• begin refuses an estate not delivered, a box not laid, an update already open, no release published, a stale wanted, "up to date", and an estate with neither a keep medium nor a live app (:415-492). It pins the media.
• The front pulls the update (Fleet.Ask{update_take}) and reports each box (Fleet.Ask{update_report}). Per box: the same BASE with live update on → the live app is staged into the other slot; the BASE moved, or live update off → a keep boot from the role's keep medium (Fleet.Ask{upgrade} → sold::upgrade_as), bounded by KEEP_BOOT_WITHIN = 120 min. The front either stages itself or reports needs-keep, and then the ONE box keep-boots it from front-keep over the cloud API.
• custody_decide (:1903-1936): no live app or live off → keep; a trial open → wait; the same app → nothing; r.base != app.record.base → keep ("its BASE moved: a base update"); else live. A BASE change is the slow upgrade because it is a keep boot.
• A failure STOPS the update; the boxes after it are not touched (:605-611, ALERT estate-orders.update). Stall: KEEP_BOOT_WITHIN + 1 h, retold every 6 h; deadline 24 h (:57-64).
• A front with no installer material: the ONE box takes main's and twin's steps itself from the custody (custody_drives, :1851-1862; poll 10 s, confirm 15 min).

Anti-rollback, in full:

5.6 Grow: room past the main's set, and a paid data step

The room rule, verbatim from Pulled::can_absorb (monetize-product/src/pulled.rs:232-258):

for each meter in the caps that has a `room` in the main's report:
    need = target − promised(tenant, meter)
    short if need > room
nothing short              → Absorb::OnHand
short, report.with_iron    → Absorb::WithIron { note }
short otherwise            → Err("<instance>: <meter> needs <need> beyond the <room> the main has room for")

On top of it, can_absorb_queued (monetize-server/src/grow.rs:629-650): when the main reports Servable::Measured with growth not Sealed (queues_room), a refusal with stored demand becomes SOLD as OnHand and queued: "the customer must not know about a cap". The path (grow.rs:1-17; estate_order/sold.rs:7-29):

1. A sale past the room → queue_past_room → one signed grow_room { need_bytes, reason } order to the primary, the newest superseding, TTL 30 days (guardian/src/pulled.rs:456-480). The operator hears it, never the customer (ALERT grow.not_told if it cannot be queued).
2. The box pulls grow_room, keeps the want on its health, acks ok.
3. The buyer pays a ChangePlan step (chapter 4). Once captured, sold::grant signs an EstateGrowth { company, gib, until = paid_until }, keeps it as run.growth, and queues one grow_go_ahead { go_ahead, data_gib } per laid data role with more than one disk (main and twin; the twin's data_gib is main's whole set), TTL to the end of the paid period (sold.rs:154-291). A renewal re-grants.
4. The box verifies the grant under the monetize key, drains and unmounts (the twin first), and asks Fleet.Ask{grow_set} with the gib (embed/src/fleet.rs:60-66; puller.rs:97-108).
5. sold::grow_own (sold.rs:1529-1698): only the caller's own LAID data disks for its role; refuse gib > paid; a disk never shrinks; one growth per estate at a time; journaled in grow-<role>.journal; Change::Grow at the cloud (per member: detach → resize → wait → re-attach → verify). OverCap answers "the shop is preparing this growth; ask again later" and the box asks again.
6. Read back (survey → read_back): laid disk sizes move only up; quote.sizes.main_data_gib is raised to the smallest main data disk. The box remounts and acks in its next report.

Grow is never_shrinks on main_data_gib by the facts (GrowRule::sizes()). There is no auto-grow: a grow is always a paid step.

5.7 The term's actions on the estate, and terminate

From paid_until (chapter 4 §4.9 has the mails): CAPTURED → GRACE at paid_until (the estate still runs); GRACE past the 14 days → power(false): every server stopped, disks kept ("suspended"); SUSPENDED → DELETING at the deletion date (a run whose money was reversed is held for the operator); nothing is deleted while paid_until is in the future; then return_everything → DELETED (estate_order/term.rs:806-897). A buyer who cancelled is stopped at the end of the paid period and started again on payment. The Guardian leaves an estate the term holds alone (World::estate_held).

Terminate (estate_order/terminate.rs): the buyer types the name back; the operator uses site terminate <site> --confirm <name>; an operator site is protected unless terminable = true. No backup is taken, ever (owner, 2026-10-22). The call marks the run terminating; the tick gives everything back (return_everything, estate_order.rs:7824-7965): release clones first, each server destroyed once, wait while servers and disks are on their way out, release_for the references, prove orphans_for is empty, remove the DNS where it was written, forget the custody. Every step is a ledger row plus ALERT estate-orders.terminate on failure; retry backoff 10 s doubling to 5 min. Nothing is refunded.

5.8 The Guardian

The Guardian (monetize-guardian) runs on the ONE box and repairs estates from what their boxes reported; it never calls into an estate. The product supplies the brain (EstateGuardian: diagnose, plan, observe; chapter 3 §3.7); the server supplies the world (World: the cloud, the laid machines, the entitlements, lay_one, restore_one, estate_held, box_held; runner.rs:100-140).

Cases: detected → investigating → approved → running → done | failed, or resolved if the estate recovered; at most one open case per estate (case.rs:6-31). A tick (runner.rs:573-658): an estate the term holds is resolved; the start pass first ("start, don't rebuild": a box only STOPPED at the provider is started, the front first, then the main, the twin once the main is heard; START_WITHIN 10 min, backoff 10 min, handed to the operator after 3 tries, ALERT guardian.start-handed-over); then diagnose → a case opens. advance (:673-816): Detected asks the provider → Investigating; auto-fix approves only if the estate opted in (off by default, principal "guardian (auto-fix)"), else a person presses a remedy with a strong session; Approved → plan once → Running the steps → Done.

Config (runner.rs:58-70): silent_after 90 s, order_ttl 15 min, order_within 6 h, pin_window 10 min, fence_within 15 min, provider_look_every 5 min.

The fence invariant, in full (guardian/src/lib.rs:9; runner.rs:181-206, 876-974):

No lay for a role until its old box is gone.

• A Lay { role } step fails while the role's laid machine still has any resource standing at the provider, or cannot be looked up: "no lay while the old box stands: fence it first".
• A plan that lays a role it promotes (the main, or any role later recharged from the twin) with no Fence of that role before it is refused before any step; at lay time the fence must be CONFIRMED.
• A Fence { role } step: never a box that is only stopped; retire_machine gives back the server, every disk and the clone; forget_box the old key ("a box gone at the provider whose key stays pinned is no fence"); met only when the provider's listing reads it gone within fence_within.
• An operator-laid estate's fence is the owner's signed notice (within 24 h).

The twin is never promoted (monetize-product/src/guardian/steps.rs:137-157): a dead main is rebuilt — fence main → lay main → await heard (30 min) → recharge_from_twin → await recharged (6 h) → restate the entitlements. Auto-fix takes NewServer only when the guard says the main is GONE at the provider; R6: auto-fix never reformats a standing disk (rebuild.rs:1-34).

Orders the Guardian queues are signed with monetize's product key, stay until acked in a report, and supersede older unacked ones of the same kind; 64 acks are kept (store.rs:16-43, 366-407).

5.9 The box watcher outside the cloud

monetize-box-watcher runs as a systemd --user timer on the operator's machine, outside UpCloud (added after UpCloud stopped every server in the account on 2026-11-01). Each run reads each account's token from a KEY=value file (never printed), lists GET /1.3/server, finds the watched ONE boxes by uuid or by their DNS name, and starts a stopped one (POST /1.3/server/<uuid>/start) when apply = true; dry-run by default. It never creates, deletes or stops anything; a box the operator holds with a marker file is reported held and never started. The ONE box starts its own estates once it is up (the Guardian's start pass). Its lines are ALERT box-watcher.* (chapter 7).