nordisk/gunnar

Growing the data set WITHOUT a restart — the runtime path, as built

Written 2026-09-14 against gunnar branch data-set-grow and edda branch data-set-grow, the code half of private-gunnar-ops/DATA-SET-GROWTH-FLOW.md §3 under ruling 1 (the appliance never goes down). data-set-grow-drain.md beside this file is the design note it grew from; DATA-SET-GROWTH-DESIGN.md §5 stays the contract for the engine's steps G1–G6 and their journal. This file says what exists, where, and what the bench has to drive to prove it.

The verbs

CLI: gunnar control grow start|status|resume, gunnar control replication flush.

Bootstrap.GrowStatus is the third credential-free RPC (gunnar_grpc::UNAUTHENTICATED_METHODS), and T6 is why: the actor outside the box that owns the volumes must read waiting-members / failed with nothing to sign with, while the identity table is in RAM — or after a crash in that window, when there is no identity table to sign against at all.

The phases, and what each writes to /serialdat/grow-journal

The journal is the boot path's (korp_installer::grow::Journal, one line per fsynced record, epoch= target_sectors= member= step= at=); the runtime steps ride the same lines and korp-start reads past them to the open epoch.

ABORT closes an epoch (nothing was touched); FAIL keeps it open for the next boot or the next grow start; a prelude-only epoch is joined by the engine whatever target G1 measures, so a runtime growth is one epoch.

How the torture rows are met

• T1 — bounded drain, then abort with nothing changed; no transfer is ever killed (grow::tests::a_drain_that_times_out_aborts_with_nothing_changed).
• T2 — gunnar_core::ControlDb: both redb stores share one handle; suspend copies the tables and closes the file (proven descriptor-free via /proc/self/fd), reads answer from the copy, writes are Error::OffStripe → UNAVAILABLE (D7: refuse); resume verifies RAM == disk.
• T6 — the phase never fails on the wait; Bootstrap.GrowStatus needs no credential; a crash with volumes detached halts the next boot on a missing member, and the actor outside re-attaches — then reboot finishes the epoch from the disk, or GrowResume does.
• T9 — every recoverable failure serves at the old size and names why; only cannot-assemble/mount is fatal, and even then the control plane is up.
• T11 — Mirror.FlushReplication between drain and unmount (step 0 of §3): it proves every push the replicator saw is on the twin as of the flush; it does NOT prove refs outside heads/tags, LFS, the control database, visibility, tombstones, or repositories nobody pushed since the twin existed unless resync ran. The verdict says both, in words.

The go-ahead (D3)

GrowStartRequest.go_ahead is JSON: {v: 1, signer: "operator"|"monetize", target_sectors, nonce, issued_unix_ms, signature}. The structure and the nonce ledger (/serialdat/grow-go-aheads, one line per spent nonce) are checked for every signer; WHO may sign is gunnar_server::grow::GoAheadPolicy. The default OperatorPolicy accepts an unsigned operator go-ahead (the gunnar-admin RPC is the approval) and refuses monetize; a metered build plugs a policy verifying signature under monetize's Ed25519 key through GrowMachine::with_policy (the entitlement facts' monetize_embed::signing pair is the intended verifier). Not built here: monetize's minting side.

Draining{kind: growing}

Presence.Watch does not exist in this tree. GrowMachine::with_announcer is the hook: register the stream's fan-out and it receives Announcement::Draining{kind: "growing"} before the pause and Announcement::Ready after the reopen. Until then both are tracing rows on gunnar::grow (presence.draining, presence.ready).

The bench: a --runtime scenario for gunnar-growbench

What the bench should drive under KVM (software steps first, the hotplug half when the kernel has it); the bench crate is another agent's and is not edited here:

1. boot the appliance image built with gunnar-server --features grow-runtime (needs the sibling edda checkout — see the Cargo.toml note) on 4 member disks; bootstrap; push the manifest set; replication flush if a twin is configured;
2. gunnar control grow start --wait-members-timeout-secs 120 as gunnar-admin; assert the answer is draining and that a git push started now is refused with the growing sentence and exit 75 (and git fetch too);
3. poll gunnar control grow status (NO credential) until waiting-members; assert serving: false, identity_in_ram: true, /proc/mounts on the guest has no /data (a tmpfs is there instead), /sys/block/md_korp_data is gone; the journal reads DRAIN UNMOUNT STOP WAIT; a control-plane WRITE (user add) answers UNAVAILABLE naming the growth; a READ (user list, status) still answers — the identity is in RAM;
4. resize every member: today's kernel (no hotplug) — QMP block_resize on the four attached drives (virtio-blk raises a config change and size updates in place); the hotplug kernel — QMP device_del ×4, resize the backing files, device_add ×4, and assert the guest enumerates them;
5. optionally grow resume, else wait for the 2 s poll; poll status to done; assert total_bytes grew, Admin.Capacity reads the new total with GROWTH_AT_RUNTIME, the journal ends … RESCAN G1 G2:0..3 G3:0..3 G4 G5 G6 REOPEN, the manifest set verifies (design §7's "intact"), a push lands, replication list shows the standby taking it;
6. the crash rows: halt the guest between STOP and the resize (C1b) — assert the next boot HALTS on a missing member until the disks are re-attached, then boots and finishes the epoch from the disk; halt during growing — the boot path resumes at the next G-step (C2–C5 as today);
7. the refusal rows: resize 3 of 4 → status stays waiting-members with PARTIAL in the survey; grow resume --give-up → failed (serving, old size), the journal FAIL REOPEN; an unmount with a descriptor held open → failed{aborted: … EBUSY … fd N → /data/…}, journal DRAIN ABORT, nothing changed.

Native tests cover every phase and every one of these transitions over a scripted world (gunnar-server/src/grow/tests.rs); what only the bench can prove is the real umount/md/xfs behaviour and, with the hotplug kernel, step 4's detach/re-attach.

What needs the hotplug kernel

Only step 4's second form. Everything else — drain, identity in RAM, close, umount, md stop, the guard tmpfs, rescan (a harmless no-op), G1–G6, mount, reopen — runs on today's kernel; a QMP block_resize of an attached virtio disk is enough to exercise the whole runtime path under KVM. On UpCloud the detach/resize/attach is the front-side watchdog's (D5), and the appliance's half is complete.

The registry — closed 2026-09-15

korp-installer 0.1.4 carries grow::runtime (released through dwarves crate-release from edda data-set-grow), gunnar-server's grow-runtime feature resolves it from the registry, and the feature is in all four server variants of .nornir/build-variants.toml. The developer patch config is gone.

Proven under KVM (bench run 6, DATA-SET-GROWTH-DESIGN.md §14)

gunnar-growbench R1–R5: the clean runtime growth (QMP block_resize under the running guest, today's kernel), the partial → give-up row, the power cut in waiting-members, the drain timeout (T1), and the hot-plug form (unplug → resize → re-plug ×4 on the kernel-pci-hotplug kernel, members behind PCIe root ports). 128 → 256 GiB with no reboot on both kernels. What the run found and fixed is in §14.