nordisk/holger

14 — holger's estate, built exactly like gunnar's

2026-09-27, loki. Owner: "LOOK at gunnar vm1 vm2 vm3 using tunnr and all", "use mock upcloud, create a similar functional test for holger as gunnar has", "tunnr for all holger build things", "study it DEEP". Eight read-only analyses (install kit, vm1 front, PID 1, migration, selling, tunnr/media, functional test, twin runtime), the claims that change the design re-checked by hand. Nothing here is built yet except what §0 says exists. Tags: MEASURED = read in the code today (file:line). DECIDED = the owner or the gunnar agent ruled it. PLAN = proposed here.

0. What exists today (holger side, 2026-09-27)

• holger's own plugin in crates/holger-monetize-guest, signed by Build Thing, registers itself (server/lib/src/self_register.rs), delivers through holger.metered.v1.
• nordisk-pairing 0.5.4 (Skidbladnir branch nordisk-pairing-holger-kind): Kind::Holger, boxside (a product box's side of the installer, shared).
• holger-server paired (server/cli/src/paired.rs): reads the seed, serves the installer's listener, records the elected role, self-registers with its installer certificate.
• The installer medium carries pairing_seed (holger-recipe 165e0e2); mvp-server-musl built in the container.
• holger.estate.v1 + twin_follow + recharge_from; proven with three holgers in one process on one host (see §2 correction 1). Superseded by P3 (2026-09-27): the twin follows over the primary's own mutual-TLS replication door across addresses, the role applies live, and a dead main is recharged from a remote twin (see §4 P3 status).
• private-holger-ops: monetize feed, stripe webhook (LIVE endpoint we_1UKLqX… → https://holger.monetize.ignalina.dk/callback/stripe, created 2026-09-27), install plan, web gets the utility network (declared, not applied).
• The console: one session (passkey), writes through the BFF (holger-ui console-one-session).
• disk_bytes is sold AND enforced (P1 done, 2026-09-27): the metered build refuses an upload past the paid cap at the HTTP and gRPC write doors (metered::Quota, holger-metered-6, 507 / RESOURCE_EXHAUSTED, the numbers and where the bytes are; suspended/retention take no upload; DELETE never refused). The plugin (guest + linked, lockstep) declares it Enforced; Build Thing re-signed it (blake3 2d27d587…, build ref 176d234); prove_holger green again (PAID, fact stored). The slow paths too: every repository backend is wrapped once at wiring (metered::QuotaGuard), so a mirror pull, replication copy, job upload, proxy cache fill and Holger::put hit the same limit; a twin is never policed (not wrapped when configured, recognised by its role record when elected live). Cost: one walk of the repository's places per write — a cached reading is next.

1. How gunnar's estate works — the model (MEASURED)

The one line: private-gunnar-ops cargo xtask estate install --variant gunnar-twin-monetize --where kvm|upcloud [--apply] (xtask/src/install.rs:1-30, stages 505-536): reap → retire (state to .previous/) → lay (terraform, all three VMs) → medium (cargo xtask iso-installer in the gunnar checkout → dwarves buildthing → tunnr korp_installer::isocompose::ForgeComposer, every compile in the pinned container) → media (upload at the provider) → installer + boot (nordisk_pairing::installer::run IN this process over its WireGuard link, mTLS inside; admits, elects the weakest box twin, feeds, streams boot logs, closes when every box reports PAIRED) → dns (Cloudflare, gunnar.rs) → functest → teardown (last).

The boxes (gunnar-server as PID 1, pid1.rs, pairing.rs): korp-start mounts /boot and /data; gunnar brings its OWN network up (netlink + DHCP on eth0 and the utility eth1), clock, reaper, installer listener on the link address only; after the feed its pairing loop raises wg0 to the front's relay (10.44.0.0/24, front .1), installs the fed admin, the twin runs BecomeTwin and follows — everything live, no restart. Control plane :50051 on utility; replication :50052 mutual TLS on utility + wg0 under the PRIMARY's own issuer (the installer CA is bootstrap only).

The front (vm1, its own medium gunnar-front-installer-metered, korp.entry=gunnar-front): relay, clock, twin certification (BecomeTwin → enrol in group twin → TwinCsr → IssueTwinCertificate → InstallTwinCertificate), bare monetize in-process, plugin door :8475 (self-registration policies: GUNNAR and, since 55e7338, HOLGER), web/ACME/cert handover, /callback/<vendor>, the gunnar console (gunnar-ui-server linked in, dist embedded at build), tcp/2222 forward.

Recharge: a fresh main installed from the same medium replays the twin's log (twin::restore_from), driven by the monetize guest (Bootstrap.Status → Mirror.LogHead/ LogFetch on the twin → Bootstrap.Restore). The twin is never promoted.

The functional test (end of every --where kvm install; estate functest alone): push → refs → clone → twin followed (lag 0) → visibility → anonymous clone → plugin registered at :8475 → vm1 restart survives → shop window offer → unsigned webhook refused → signup at the fake Stripe (forged 401, signed 200, account appears) → paid loop by the poller (cap +10 GiB, replays change nothing, data set grows through upcloud.wasm on the mock) → over-cap refused. mock-upcloud-kvm (monetize crates/mock-upcloud/kvm-guest): the same terraform and provider against a local UpCloud API; QEMU/KVM guests; offline mode where guests reach only 10.0.2.100:{18453 clock, 12111 fake Stripe, 8099 API}.

2. Corrections to what was said earlier today (MEASURED)

1. The recharge proof was same-host only. A twin binds EVERY door, gRPC included, to loopback (server/lib/src/twin.rs effective_bind); the three-holger test ran in one process on 127.0.0.1. A real recharge needs a replication listener the twin keeps open.
2. The twin's credential must not be the installer CA or a bearer. Box certificates live 24 h by default (nordisk-pairing door.rs:43), the medium's bootstrap certificate (key on the medium) is signed by the same CA with ClientAuth, and every box cert carries the same SAN — trusting the installer CA trusts anyone holding the medium. gunnar's answer: the primary's own issuer, certify once, pin.
3. The elected role takes effect only at the next start, and may then be refused: the box runs its first boot as primary and may not be "fresh" (twin.rs is_fresh) when the role arrives. The role must apply live.
4. A holger box can never finish an install today: the installer closes on CloseWhen::Paired (nordisk-pairing installer.rs:642-643); holger writes no signal, brings no network (tunnr korp-start.rs:126: "this PID-1 brings no network up"), raises no wg0.
5. "Paired downstream = twin": a DownstreamRecord pins a fingerprint and grants listed repos only (peers.rs:70-82); stinky 02 §4.4 keeps downstreams from the estate. DECIDED (owner 2026-09-27): a paired downstream may ACT as a twin. PLAN: it does so by being certified into group twin (a role, with the estate door), not by widening the pair record.

3. The gap, per layer

4. The plan, ranked (each step with its proof)

P0 — safety, before anything touches the account: copy the real terraform.tfstate from t14s, confirm it holds only front; never apply over it; back up /var/lib/holger, state.redb, Caddy's data.

P1 — disk_bytes (holger only, starts now): Repository::storage_places() (one function shared with flow.rs), allocated bytes per repository, source = where; policed (PER_REPOSITORY_METERS); the plugin bills it (guest + linked in lockstep); prove_holger against a real metered holger reaches PAID + EntitlementPushed + holger stored the fact; then enforcement on the seven write paths (HTTP PUT, gRPC PutArtifact, Holger::put, proxy fill, mirror, replicate, jobs). Prices: owner.

P2 — shared extraction (needs the gunnar agent's word; gunnar stays pinned until it bumps): edda appliance-pid1 (network, DHCP, reaper, SIGTERM); nordisk-pairing boxside loop + wg0 driver + boot-log tap; nordisk-twin (certify state machine, twin credential, replication door); dwarves: kernel/root-key/refusals/plugin-stage/medium-record; ops-kit crate: provider + mock doors, bench clock, HTTPS client, Stripe verdicts, profile, pretest, lay/media/installer run.

P3 — the holger box: live role; replication issuer + mutual :50052 on utility/wg0; certification over the shared crate; twin follow on mTLS only; recharge from a REMOTE twin. Proofs: two holger-server processes on different addresses; no cert / bootstrap cert / other CA refused by name; blake3 manifest equal after recharge.

P3 status (2026-09-27, holger branch holger-twin-mtls, Skidbladnir nordisk-twin, monetize holger-restore-lane) — BUILT, proven in process on separate addresses.

• Shared crate Skidbladnir/crates/installer/nordisk-twin (no product type): the primary's persistent issuer (P-256, key 0600 — holger has no sealed store; gunnar keeps its key sealed, the files are the floor), issue_from_csr (proof of possession, CN = principal or twin-csr-claim, clientAuth, 365 d), the door's server leaf (SAN holger-replication, localhost, 127.0.0.1, bound/advertised IPs), the reader's request/install/load with the four checks (twin-no-key, twin-key-mismatch, twin-not-issued-by via the door's own webpki verifier, twin-repointed; renewal under the same CA taken), a deliberate repoint, TLS-1.3-only ring configs, the door's accept loop naming every refused handshake, the dialler (first recorded host answering a 3 s probe), and the certify state machine over a Boxes trait.
• holger (twin_tls.rs, recharge_lane.rs): replication_door: config key (absent = no door, no issuer); the door at gRPC port + 1 on the CONFIGURED address (a twin keeps it — it is the door a recharge dials), RepositoryService + holger.estate.v1 on a read-only HolgerGrpc whose auth knows two CNs (holger-twin, holger-recharge); AdminService TwinCsr / IssueTwinCertificate / InstallTwinCertificate (+ repoint) / Recharge / RechargeStatus / TwinStatus; codes holger-twin-7..16. The follow dials the door over mTLS (MirrorCredential::Twin, estate pull_over); plain http + bearer is ONLY for an uncertified twin of a LOOPBACK primary (tests), a remote one is refused holger-twin-13. Freshness counts bytes, not directories (a first start's 0-byte audit segment and job redbs no longer block an election — correction 3).
• Live role: TwinPosture holds the role in a lock; BecomeTwin records, switches, starts the follow — writes refused by name from the reply on. Restart-bound, like gunnar's git listener: the loopback bind of the public gRPC/HTTP doors (wire_holger) — a live-elected twin keeps its doors where they were bound (refusing every write) until the next start. The paired box loop (paired.rs HolgerSteps) now does all three twin steps for real (become live, certified = credential installed, PAIRED = a whole cycle over mTLS).
• Recharge of a dead main, ordered like gunnar's restore: monetize decides it (grow.rs::restore_decision, a box that cannot serve — TwinPosture::serving() is the seam holger's GrowStatus.serving folds in; not touched here), re-images, then the plugin's restore lane runs — holger's guest (crates/holger-monetize-guest/src/restore.rs, unary only: the main PULLS over its twin's door) and the linked plugin (monetize products/holger/src/restore.rs, lockstep), or holger-server recharge --main --twin by hand. Four refusals before a byte moves (holger-twin-16): no twin, main not fresh (re-image), empty twin, twin not complete without accept_lost. A main being recharged serves NOTHING (holger-twin-15, both transports and its door; probes and /-/twin answer) until the pull is whole; partial = still refusing, resumable (holger-recharging). The credential is explicit and pinned (the TWIN's own issuer, principal holger-recharge) — never the installer CA or a bearer. After the recharge the lane re-points the twin at the new main (a front/lane step, not automatic in a box): the new main minted its own issuer at its first start (issuers never replicate), the twin installs a leaf from it with repoint and names the new main as its primary. The twin is never promoted.
• Proofs (server/lib/tests/): the_twin_follows_over_mutual_tls_across_addresses.rs (127.0.0.1/.2/.3: certify via RPCs; follow over mTLS only; refused by name — no cert, another CA, the medium's bootstrap cert; re-point refused, renewal taken; write refused live; recharge over the twin's door) and a_dead_main_is_recharged_from_its_remote_twin.rs (every box in its OWN runtime: the main's runtime is shut down = dead; a fresh main refuses read and write by name, the lane refuses no-twin / not-fresh / empty-twin, then recharges — artifacts byte-equal, every estate file blake3-equal, holger.ron = the dead main's config byte for byte, own issuer; the twin then pins the new main's issuer and follows an artifact put there. A second test drives holger's REAL guest through a host door wired to real holgers: ten calls in order, same outcome.)
• A twin's gRPC CONTROL door stays on its configured (utility) address (owner 2026-09-27, "look like gunnar"; NodeRole::effective_control_bind): writes and admin mutations refused by name, pairing/recharge calls answered; only its public HTTP doors go to loopback. Proven: a_restarted_twin_keeps_its_control_door_and_is_recharged_against (certified live, RESTARTED from disk, still on 127.0.0.2, write refused by name, follows over mTLS, a fresh main recharged against it and the twin re-pointed through that door).
• Left: (1) the processes are in-process runtimes on 127.0.0.x, not VMs — the KVM estate run is P6; (2) done — see above; (3) TwinStatus.complete measures the estate cycle, not the artifact mirror's per-pair lag; (4) the replication door runs on 0.0.0.0/utility only as the config says — wg0 binding comes with P2's box loop tunnel address (ron_advertise); (5) the guest needs [products.holger] restoring_from_twin on the front (the allow-list) and a Build Thing re-sign (new lane) — P5/gunnar agent for gunnar-front's holger row; (6) gunnar adopting nordisk-twin is the gunnar agent's.

P4 — holger media through tunnr + dwarves: holger-installer and holger-installer-metered (plugin staged like gunnar's), kernel donor tunnr-kernel, iso-installer a thin door; the red loaded_as_a_plugin roster taken from declared().

P5 — vm1 = gunnar-front for holger: product feed key (console, ESTATE_PRODUCT, twin certify, forwards, RP name); console-holger; console write credential (fed token or vm1's own cert); repo/grpc routing (DNS to vm2 or a feed-declared forward); holger front medium row.

P5 status (2026-09-27, holger branch holger-front-vm1) — SUPERSEDED: vm1 is holger's OWN program, not gunnar-front. Owner, verbatim: "i SAID follow the pattern gunnar has for vm1 vm2 vm3 regarding installer etc. NOT BECOME gunnar!!". holger-estate-front (its own workspace, excluded from the root: the open holger resolves no money) is PID 1 of holger-front-installer-metered, built from holger + the shared crates only: Skidbladnir nordisk-pairing 0.5.7 frontside (vm1's Vm, kind front; branch nordisk-pairing-frontside), edda estate-front (web/ACME/certificate hand-over, callbacks, repository route, forward, relay — moved from gunnar-front as is; branch estate-front), edda appliance-pid1, nordisk-twin certify over holger's OWN AdminService (no BecomeTwin: the installer's election is the box's role record), monetize monetize-front (bare monetize, the plugin door, self-registration; branch estate-front), holger-ui-server (the console). Proven in process: the real installer closes front + two holger boxes on PAIRED; the front certifies a twin between two real holgers over mTLS; :443 serves the console and carries repo. to the primary; holger's primary registers its real plugin and [products.holger] carries the restore lane (restoring_from_twin, the twin on the grpc allow-list) and the console bearer. GUARD: holger-estate-front/tests/no_gunnar_in_the_graph.rs. Medium composed by the Build Thing (dwarves + tunnr/tunnr-kernel, no gunnar checkout on the box): 40 642 560 B. Left: holger-ui declares no Build Thing row for its trunk dist/, so the console BUNDLE is not on the medium (its API and /login are); the front keeps no signed clock of its own (holger's boxes read none).

P6 — holger ops on the mock: holger variant on the ops-kit; mock verb; bench Stripe at 10.0.2.100:12111; web boot_order for a medium; estate install --where kvm; functest H1–H9: front page + console; push; pull (hash); twin holds it and refuses writes; estate parity; plugin registered at :8475; paid order delivered (disk_bytes) through the fake Stripe; recharge a fresh main from the twin; primary restart survives.

P6 status (2026-09-27, CODED, not run): private-holger-ops holger-estate-install — cargo xtask estate install --variant holger-twin-monetize --where kvm|upcloud [--instance test2] in gunnar's stages; holger's own bench ports (mock 8199, clock 18553, fake Stripe 12211/12212); estate functest H1–H13, fill F1–F5, bench estate B1–B16 + P1–P4. The blockers the dry run found are fixed: every VM on a blank system disk with disk,cdrom and holger's media declare install_then (holger estate-fixes); the artifact door is carried by vm1 over the utility network (via = "front", :8443 admitted from web only); paired installs the fed holger_console_token as the console's StaticToken file (0600) and a first boot waits for it.

Bench P1–P4 — needed change (UNRUN until then): bench-scenarios starts its own in-process holger for every arm; only served.rs takes a remote --target, plain http, no CA, no bearer, and it fetches its OWN synthetic corpus. Against an installed estate it needs a remote Target { http_base, grpc, tls_ca_pem, bearer } threaded through served (2 fns + the client builder), dataplane (Gateway::start, 3 arms), grpc_door (Running local-or-remote, tonic ClientTlsConfig + a bearer interceptor, REPO a parameter; 4 arms) and stress (writable_gateway, remote listing/seed; upload_edges edge 3 stays local-only), the corpus seeded through the target first, and the RSS columns reported n/a remotely. And a client half behind a feature without server-lib, so the ops xtask can link the arms without the server.

P7 — the real move: P0 done; lay the new three; recharge the new primary from the old box (its gRPC opened to the new server only); carry passkeys and the certificate; functest on the account; DNS switch on Cloudflare; read-only window; teardown of the old box last.

5. Who decides what

• Owner: holger prices (unit disk_gib_month; SE/FI/DK); OK to run P7 on the account.
• gunnar agent: P2 moves of gunnar code into shared crates (and gunnar's move onto them); the product key in gunnar-front.
• Done by the owner already: disk bytes are billed (2026-09-27); live Stripe endpoint (2026-09-27); vm1 reuses gunnar-front (gunnar agent, 2026-09-27).