02 — The twin, growth, and the A→B→C question
Lane 2 of the stinky list, 2026-09-20. Brief: read gunnar's real documentation of how monetize · twin · server · ui cooperate when things grow, generalise it as a collaboration with gunnar, and answer whether holger's twin replication and the upstream→on-prem (A→B→C) sync are one mechanism or two. This lane WRITES; it implements nothing. Patch sketches are in §7.
Every claim is tagged. MEASURED = re-run today (grep / read of the code as it is on disk, or a dated live observation). READ = a document or comment says so; the code was not exercised. DESIGNED = this lane's proposal. Never blurred; where a doc and a measurement disagree, both are shown.
1. Where gunnar's three-part documentation actually lives (READ)
There is no single "how the four cooperate" document. The cooperation is spread over five files in three repositories, and the best single-paragraph statement is a module comment on an operator verb, not a design doc.
| what it documents | file | lines |
| The contract — steps 0–10 of a growth, who does what, the twin as the only way back | /home/rickard/git/private-gunnar-ops/DATA-SET-GROWTH-FLOW.md |
:25-30 ruling 2 (reformat + pull everything back from the twin); :65 F13 gunnar never dials monetize; :66 F14 the standby never promotes itself; :107 caught up = in_flight == 0 ∧ pending == 0 ∧ …; :165-174 step 0 "the twin is proven caught up — before anything … on failure the growth does not start" |
| The engine contract + bench runs (KVM proofs, crash matrix) | /home/rickard/git/private-gunnar-ops/DATA-SET-GROWTH-DESIGN.md |
:206 bench 6 (runtime growth under KVM, both kernels), :245 bench 7 (twin growth TW1), :433 dual heartbeat, :468 money-to-disk chain |
The estate rules — every machine's [grow] block, allow_grow master gate |
/home/rickard/git/private-gunnar-ops/estate.toml |
:62-66 allow_grow = false ("the ONLY gate on it resizing real volumes"); :129-133 main: rule = "per-member-runtime", step_gb = 1, "no reboot"; :147-159 twin: rule = "on-fill", fill_percent = 80, step_gb = 20, mode = "stop-resize-start" |
| Who does what, one paragraph | /home/rickard/git/private-gunnar-ops/xtask/src/grow.rs |
:18-33 "There is ONE growth engine for the stripe … the cloud half is monetize-impl's upcloud/grow.rs … the appliance half is gunnar's Admin.GrowStart / GrowStatus / GrowResume … monetize-server on the front orchestrates the two (flush the twin → signed go-ahead → GrowStart → wait unmounted → cloud → GrowResume → wait done → ledger)" |
| The orchestrator (monetize) — the phase machine | /home/rickard/git/monetize/crates/monetize-server/src/grow.rs |
:1-19 "orchestrated from the front — the OUTSIDE actor … approved → flushing (T11: the twin is caught up, or nothing starts) → starting (Admin.GrowStart, signed by monetize's key) → waiting-unmounted (gunnar drains, we only wait) → cloud → resuming-appliance → recording"; :82-91 twin_needs_gib(target, members) = target × members |
| The batcher — 10 s window, 80 % high water | /home/rickard/git/monetize/crates/monetize-server/src/config.rs |
:463-467 "when it closes the SUM becomes at most ONE cloud resize. Default 10 s — never a resize per purchase"; :755-772 TwinGrowConfig "stop → resize → start. The primary is never restarted.", threshold_permille (800), step_gb (+20), max_gb (4096) |
| The twin lane of the batcher | /home/rickard/git/monetize/crates/monetize-server/src/grow/batch/twin.rs |
:1-10 every Mirror.LogHead a twin sends carries its fill; monetize reads the rows off Admin.Health and queues twin-fill:<ms>; :27-32 "Why the twin may restart and the primary may not — the twin is a follower with no customers" |
| The cloud half (monetize-impl) | /home/rickard/git/monetize-impl/crates/monetize-cloud-impl/src/upcloud/grow.rs |
:1-4 "detach → resize → re-attach, per member, journaled before every call"; :13 HOTPLUG_FAILED; :50-61 the credential must be a sub-account scoped to the four volumes |
| The appliance half (gunnar-server) — drain/unmount design | /home/rickard/git/gunnar/.nornir/data-set-grow-drain.md |
:1-9 (⚠ header says "Design only — no code implements the hot-plug half yet", written 2026-09-14 — stale, see §2.1); :11-26 the four steps; :33-39 Draining{kind: growing} is "what gunnar-ui turns into the customer sentence and what monetize waits on" |
| The appliance half as built | /home/rickard/git/gunnar/.nornir/data-set-grow-runtime.md |
:12-19 verb table incl. Mirror.FlushReplication = the watermark; :35 the customer sentence, 503 Retry-After: 120; :36 umount2("/data") never lazy, EBUSY names the descriptors; :60-64 T11 what the watermark does NOT prove; :145-152 proven under KVM 128→256 GiB no reboot |
| The twin's transport | /home/rickard/git/gunnar/.nornir/twin-transaction-log.md |
:3-8 the rulings; :60-63 Mirror.LogHead (server Read) / Mirror.LogFetch (server Admin); :74-84 the twin's loop (the twin pulls); :123-125 "Never promoted. There is no SetRole(PRIMARY) path on a twin"; :144-162 the KEK ruling; :183-194 gunnar-twin is a pointer variant of the same binary; :278-313 twin mode, Bootstrap.BecomeTwin, Bootstrap.Restore |
| The stink table — the numbers in one place | /home/rickard/git/private-gunnar-ops/STINK-LIST-2026-09-19.md |
:24-40 rules table; :75-86 S1 "a dead twin follower blocks every growth, for ever"; :130-131 S9 "The twin image is un-metered — a promotion would stop enforcing entitlements" |
| The UI words | /home/rickard/git/gunnar-ui/crates/gunnar-ui-web/src/dataset.rs |
:35 GROWTH_WORDS = ["sealed","at_restart","at_runtime","operator_only","unknown"]; :38-46 "Grows at restart" / "Grows while running" / "Grown out of band" |
Code entry points, for the reader who wants the machine rather than the prose: gunnar/crates/gunnar-server/src/grow/mod.rs:506-535 (drain loop, pause_admission at :510), grow/world.rs:159-175 (sync_and_unmount :165, stop_array :166, grow_and_mount :175), gunnar-server/src/twin.rs:1328 start_follower, gunnar-grpc/src/auth.rs:1967-1970 the gated inventory naming Mirror.LogHead/LogFetch/SnapshotFetch.
Neither monetize nor monetize-impl has a .nornir/ (MEASURED). Their design lives in module comments. gunnar-ui/.nornir holds only build-variants.toml.
2. Ground truth, each item verified before use
2.1 "gunnar's growth never restarts" — READ, with a stale header to flag
monetize/config.rs:755"stop → resize → start. The primary is never restarted." andestate.toml:129-131"at RUNTIME (drain, unmount, resize, done — no reboot)" — READ.data-set-grow-runtime.md:145-152says the hot-plug form was proven under KVM 128→256 GiB with no reboot — READ (a bench record, not re-run here).- ⚠
data-set-grow-drain.md:5-7still opens with "Design only — no code implements the hot-plug half yet". That header is from 2026-09-14 and is contradicted by the runtime doc and bygrow/world.rs:159-175. Stale header; the reader who opens the drain note first is told the opposite of what was built. Finding S2-11. - The only blocker was the kernel:
monetize-impl upcloud/grow.rs:13(HOTPLUG_FAILED),gunnar/.nornir/build-variants.toml:534"a member detached from and re-attached to the RUNNING appliance is otherwise invisible, and UpCloud resizes only detached volumes" — READ.
2.2 "UpCloud resizes are batched in a ~10 s window" — READ
monetize/config.rs:463-467: the first request opens the window, the SUM becomes at most ONE cloud resize, default 10 s. Also grow_min_interval_secs 3600 (:472-475), high water 80 % (:480-481), low water 65 %, emergency 90 %.
2.3 "front (shell=true) · main (shell=false, md-raid0) · twin (on-fill 80 % +20 GB, nothing public)" — READ
private-gunnar-ops/estate.toml:135-159: twin shell = false, firewall = ["tcp/50051 main", "tcp/50051 front", "icmp"] ("Nothing public"), one 20 GB hdd volume, [machines.twin.grow] rule = "on-fill" fill_percent = 80 step_gb = 20. Main's rule at :129-133.
2.4 THE SCAR — metered appliance, un-metered twin — READ, and it is in tension with the design
STINK-LIST-2026-09-19.md:130-131"S9 · The twin image is un-metered. A promotion would stop enforcing entitlements — failover means free service for everyone."- But
twin-transaction-log.md:123-125says the twin is never promoted: "There is noSetRole(PRIMARY)path on a twin … restore = replay into a fresh primary", and the memory note of 2026-09-18 corrects the earlier one: "promotion of a twin is not offered by any verb — the real procedure is re-image from the metered medium thengunnar control twin replay". - So the scar is real as a build-drift fact (variant
gunnar-servervsgunnar-server-metered) and hypothetical as a promotion fact. Both statements are true; S9 states the risk as if promotion were a verb. The lesson holger already took is the right one and is stronger than gunnar's own:private-holger-ops/estate.toml:78-96"THE TWIN IS THE SAME BUILD AS THE PRIMARY … a REFUSAL here rather than a note …[estate-twin-drift]…[estate-twin-store]… the emitter writes the twin's config into its own directory where a test asserts it is BYTE-IDENTICAL", andxtask/src/front.rs:266-279step 10 "THE SAME MEDIUM, AND CHECK THE CHECKSUM … NOTHING IN THIS REPOSITORY CAN CHECK WHICH BINARY IS INSTALLED. This step is that check" — MEASURED (read off disk today, private-holger-opsmain78f273c).
2.5 The KEK — READ (gunnar) / MEASURED (holger's warning exists)
twin-transaction-log.md:150-162: the twin HOLDS the KEK ("sealed secrets without their key are a backup of nothing"); a restored box must NOT keep the primary's KEK — reseal_after_restore. private-holger-ops/estate.toml:316-319 "A RE-IMAGE THAT KEEPS THIS VOLUME ON A BOX WHOSE IDENTITY WAS RE-MINTED IS A BRICK … Measured on gunnar's twin, 2026-09-17", and REIMAGE_WARNING at private-holger-ops/xtask/src/front.rs:435 (used :523; terraform/adopt.tf:56; holger/holger-recipe/src/rules.rs:528). The explorer looked for REIMAGE_WARNING in the five gunnar repos and found it only in holger's estate — the warning is holger's, informed by gunnar's 2026-09-17 brick.
2.6 "gunnar's twin is DARK" — MEASURED 2026-09-18, not re-measured today
Memory note gunnar-twin-is-dark: Admin.Health "twin gunnar-twin SILENT — last seen never, applied 0 of 0", replication list "no standby configured", the twin VM answers neither ICMP nor TCP, and the live boot stack carries no grow engine ("no journal: this boot stack does not grow the data set"). This lane did not dial the estate. Treat as last known, two days old. Consequence for holger: the "collaboration with gunnar" this brief asks for is a collaboration with a design and a KVM-proven engine, not with a running twin.
2.7 gunnar's twin PULLS; the primary dials nobody — READ, and it decides §4
twin-transaction-log.md:74-84: the twin's loop is H = primary.LogHead(); if head > h → LogFetch(h+1..), every 2 s (:285). DATA-SET-GROWTH-FLOW.md:65 F13 "gunnar never dials monetize". The primary is the appliance with no outbound; everything that needs data from it comes and gets it. The watermark (Mirror.FlushReplication, runtime.md:12-19) is the primary asking "has the follower caught up to my head" — a read of the follower's applied_seq carried on the follower's own next LogHead call, not a push.
2.8 gunnar's twin has its OWN identity — READ
twin-transaction-log.md:288-290 "Mirror.LogHead / LogFetch admit a member of group twin (gunnar_grpc::TWIN_GROUP) or a principal with the server grant — the twin's machine key reads the log without being an administrator of the box." Bootstrap.BecomeTwin mints twin.key and "answer[s] the public key to enrol on the primary as a machine in group twin" (:302-306). monetize/config.rs:763-764 TwinGrowConfig.principal "The twin's principal on the primary". This is the exact shape the owner's C-KNOWS ruling asks holger for (§4.3): a peer is a machine principal in a named group with a read grant, enrolled once, and every row it produces is keyed by that principal.
3. Holger's side, re-MEASURED today (holger main 542e296, worktree stinky/twin)
| claim from last week | today | evidence |
ReplicationService.Replicate + replicate_into exist, idempotent by content id |
holds | server/lib/src/replication.rs:1-33 (module doc: pure function, "Read-from-source / write-to-dest, never the reverse", sha256 content id), lib.rs:878-905 replicate_into pairs repos BY NAME, grpc/services.rs:1644-1750 the RPC |
| nothing drives them on a schedule | holds | replication-design.md:89 "no new backend, no network client, no scheduler"; :98-104 continuous replication is "deferred … 🧭 §9"; private-holger-ops/xtask/src/front.rs:288-293 step 13 "the replication driver does not exist … NOTHING CALLS THEM ON A TIMER"; estate.toml:362-366 "NOTHING DRIVES THEM ON A SCHEDULE. That driver is OWED" |
no holger-twin build row |
holds, and is correct by design | .nornir/build-variants.toml has 13 variants (embedded … holger-appliance-iso, :51-196), none twin. The estate's twin runs services = ["caddy","holger-server"] (estate.toml:395) — the SAME binary, which is the lesson of §2.4. What is missing is not a build row but a ROLE (S2-3) |
| no twin driver | holds | as above; plus the owed driver is described as push — see S2-2 |
Pool::delta_since / PoolDelta{added,removed,common} is unwired dead code |
holds outside tests | /home/rickard/git/Skidbladnir/src/pool.rs:396 delta_since, :452 PoolDelta. Callers: pool.rs:660,712 and harvest.rs:647 — all inside #[cfg(test)] (harvest.rs:321). Zero callers in holger, gunnar or any other repo under /home/rickard/git |
mirror_into has no callers |
holds | server/lib/src/upstream/mod.rs:177; grep -rn "\.mirror_into(" → 0 hits |
no holger-remote upstream type; a downstream cannot pull from an upstream holger |
holds | upstream/mod.rs constructors: artifactory :89, nexus :107, azure :127, aws :155 — no holger. Agent connectors: agent/connector/{artifactory,nexus,world}. Every impl RepositoryBackendTrait for in the tree (30 hits) — none speaks gRPC to another holger. The tonic client that DOES exist (server/lib/src/object.rs:208-231, RemoteHolger) implements HolgerObject (traits/src/lib.rs:717), not RepositoryBackendTrait, so replicate_pairs cannot be handed it |
| there is no replication-peer concept | holds | grep -rn -i "replication.peer|Role::Peer|replicator" → one prose hit (replication-guide.md:5). The Replicate RPC gates on authorize_admin(to_repo) + authorize_read_repo(from_repo) (services.rs:1650-1660): a peer is an ordinary admin client |
| the audit log has no query surface | holds | feature-complete-gaps.md:324-325 "❌ audit query surface (the log is written; a servable "who pulled X" query belongs in skade)"; audit/mod.rs:212-222 the AuditLog trait has record + flush only; arrow_ipc.rs:299 read_dir exists but is reached only by the CLI's verify path and tests; nothing in holger.proto mentions audit |
Three more facts this lane measured that the brief did not list:
- There is no network replication in holger at all. The RPC replicates between two repos on ONE server (
proto/holger.proto:359-368from_repo/to_repostrings;services.rs:1690-1691get_repoon self). The CLI form is two RON files on one box (replication-design.md:66-73"laptop-testable, no network"). So today neither the twin nor C can be reached by the existing engine; the engine is right, the endpoints are absent. - A replicate run is audited as
AuditAction::Uploadon the destination repo, withfrom_repoin the artifact column (services.rs:1720-1737). Under the C-KNOWS ruling that is the wrong shape: "what did C download" needsDownloadrows keyed by C'sident(S2-6). - The MVP binary already links a gRPC client.
server/lib/src/object.rsispub mod objectunconditionally (lib.rs:70; the only#[cfg]in the file is#[cfg(test)]at:605), and the zero-egress guard deliberately exemptshyper/tonic/h2(holger-tiers.md:290-291;build-space/src/lib.rs:692). So "MVP has no outbound" is precisely "MVP links no outbound HTTP client". A twin or a C that pulls over gRPC is linkable in MVP today; whether it is wanted is the open owner decision (S2-5). The guard does not settle it.
Audit facts that the ruling leans on, MEASURED: audit is on by default (lib.rs:578-596, dir holger-audit); a gRPC fetch writes Download with {who, repo, artifact, source_ip, status, bytes} (services.rs:207); an HTTP GET does the same (exposed/http.rs:1230-1262); ident is the validated subject or "anonymous" (grpc.rs:963-968). The AuditEvent struct (audit/mod.rs:143-170) is exactly the tuple the ruling quotes.
4. The architecture question: one primitive, or two?
4.1 What a twin is, and what C is — stated so the reader can check the reasoning
The twin (gunnar's definition, twin-transaction-log.md:3-8): a machine beside the primary, on a trusted wire (the utility network, estate.toml:171-174), same build, same config, that holds every durable mutation — packs, LFS, control DB writes, sealed secrets and the KEK (:150-153) — so that a dead primary can be re-laid from it. Its correctness property is completeness: "the twin must hold each transaction sent to the master; if it has missed something it must ask the master for the missing part before continuing to append" (:4-6). It is measured by a watermark (caught_up), and a growth on the primary refuses to start until the watermark is proven (DATA-SET-GROWTH-FLOW.md:165-174). It has no customers and no public doors. It never verifies what it copies — a twin that refuses a frame is a restore source with a hole in it.
C (the owner's target, HANDOVER-2026-09-20-holger-promote-attest.md:13-17): a holger inside a protected area that takes only repo B's contents — the set that passed the declared gates — and verifies them (promote-sync-attest-design.md:590-593 "the downstream's engine holds one gate, AttestationGate, at Block"). Its correctness property is selection + verification: it must NOT receive A, must NOT receive the upstream's RBAC map, secrets, audit segments, or tenant data, and must refuse anything whose attestation does not verify. It may be cold, may be airgapped, and the upstream's knowledge of it is bounded by what it served (the ruling: "the upstream knows it SENT, not that C ACCEPTED").
Put side by side:
| twin | C | |
| what travels | the whole data root, incl. secrets + KEK | one repository's promoted artifacts + evidence |
| trust in the link | full (same operator, same build, utility net) | none (boundary; possibly media) |
| gates on the way | none, by definition | AttestationGate=Block, by definition |
| completeness proof | watermark (caught_up) — required before the primary may grow |
none possible; "sent" is the ceiling |
| identity | a machine principal in group twin with a read grant |
its own principal, an ordinary reader of B |
| latency | hot, 2 s poll | cold; whenever it fetches / whenever the courier walks |
| may be promoted | gunnar: never (replay into a fresh box); holger estate: yes, by firewall + DNS + resize (estate.toml:370-372) |
never; it is a different site |
4.2 The verdict — TWO drivers on ONE copy engine; the seam is RepositoryBackendTrait
They are the same engine and different mechanisms. replicate_pairs / promote-sync §5's transfer (promote-sync-attest-design.md:574-593) is a pure function over (source backend, dest backend, gate set, evidence sink) and does not know where either end lives. That function should serve both — for the artifact part of a twin and for all of C. Above it, the two are different objects and must be typed differently:
-
A
Peer{direction}abstraction is wrong for each. Give the twin C's shape and it gets a gate — and a twin with a gate is not a restore source (§4.1). Give C the twin's shape and it gets "everything, ungated, including the KEK" — a boundary breach by construction, and exactly the thingDATA-SET-GROWTH-FLOW.mdand the KEK ruling spend pages preventing on the trusted side. The two differ in what is copied (data root vs. one repo), in whether a gate is allowed (never vs. always), and in what "done" means (watermark vs. sent). One struct with adirectionfield would carry threeOptions that are each mandatory on one side and forbidden on the other. That is two types. -
The seam is the
RepositoryBackendTraitboundary plus the ROLE of the box.- C gets a new backend:
[[upstream]] kind = "holger"— a read-onlyRepositoryBackendTraitover the existing tonic client (object.rs:208-231), holding C's own credential. Then "sync down" istransfer((remote B, local B), AttestationGate=Block), exactly as §5 already says. This is theholger-remotetype thatgrepcannot find. - The twin gets a role, not a backend:
--role twin/<data-root>/twin.json(gunnar's:280-286): stores follower-only, product doors loopback, and a follower thread that pulls from the primary with a machine identity in grouptwin. The artifact part of what it pulls can bereplicate_pairsover the same remote backend as C uses — with an empty gate set — but the twin ALSO needs the evidence store, the audit segments, the properties and the config, whichreplicate_pairscannot move (it moves artifacts by coordinate;replication.rs:6-8). See S2-4.
- C gets a new backend:
-
Both pull. This is not symmetry for its own sake; three facts force it.
- gunnar proved pull (§2.7) and its primary dials nobody (F13). The holger estate's owed driver, "a timer on the PRIMARY that dials this machine's :50051" (
estate.toml:364-365), is the opposite direction; it would make the airgappable appliance the one box in the estate that dials out, and the firewall row"tcp/50051 server"on the twin (estate.toml:401) exists only for that push. Pull needs no new firewall row: the primary's :50051 is alreadyany(estate.toml:259-263). S2-2. - The C-KNOWS ruling only works if C's fetches appear in the upstream's audit as
Downloadrows with C'sident. A push from B into C is audited on B asUploadby B's driver (services.rs:1720-1737) and on C as an upload by whoever pushed; nothing anywhere says "C has X". Pull-by-C makes the ruling true with no new code on the write side; push makes it false. S2-6. - A pushing upstream must hold a credential for C. A pulling C holds a credential for B — a reader credential, revocable per peer, and the natural thing to enrol once. (This is gunnar's
BecomeTwinhandshake, §2.8.)
- gunnar proved pull (§2.7) and its primary dials nobody (F13). The holger estate's owed driver, "a timer on the PRIMARY that dials this machine's :50051" (
4.3 What an airgap does to each — and the standing MVP tension
- The twin is never airgapped from its primary — it is on the same utility net by declaration; an airgapped twin is not a twin. No tension.
- C across an airgap cannot pull a wire; it pulls a bundle.
promote-sync-attest-design.md:640-647already rules: "MVP transport is the carried bundle …holger bundle importover a bundle that crossed the boundary on media … Pull-based sync needsonlineand is therefore afull-tier capability." The C-KNOWS ruling survives this only if the export on the connected side is performed under C's identity — i.e. the station that produces the bundle authenticates to B as principalCand the export is audited asDownloadrows for that principal. Then "sent" honestly means "exported for C, carried by a courier"; and the UI's "as of the last fetch we served" is the export time. If the export runs as an operator's own identity, the audit says the operator has it and C's column is empty. S2-7 names this; it is a two-line rule in the export verb, not a new mechanism. - The open owner decision (unchanged, restated precisely): MVP omits an outbound HTTP client on purpose. A C that pulls over gRPC is linkable in the MVP closure today (§3, third extra fact) — the guard does not forbid it. So the question is not "can the binary" but "is a downstream that dials upstream an MVP posture at all". This lane's recommendation: no for MVP — C's MVP transport is the audited bundle export; the gRPC-pulling
kind = "holger"backend is afull-tier feature (upstream-holger), the same seam, later. The twin's follower is a different feature (role-twin), also gRPC, also pull, and it IS an MVP concern because the estate already declares the twin. Both need the owner's word.
4.4 Can the twin pattern be reused for upstream/downstream holger? — the owner's literal question
Yes for the engine and the identity model, no for the driver:
- reuse:
transfer/replicate_pairs; the remoteRepositoryBackendTrait; "a peer is a machine principal in a named group with a read grant, enrolled once" (twingroup for the twin, adownstreamgroup for C); pull as the only direction; the audit row as the only record. - do not reuse: the watermark (C has none), the data-root completeness (C must not have it), the un-gated copy (C must gate), promotion (C is never promoted), and the fill heartbeat (C's disk is C's problem).
5. Growth for holger
Is rule = "never" on every machine right? Today, yes, and the refusal-by-name is the right way to say it (private-holger-ops/xtask/src/estate.rs:683-691 "holger sells no storage, so there is no batcher, no order and no verb here that resizes a volume. A rule copied from gunnar reads as a disk that looks after itself"). This lane checked the premise: holger has no fill reading at all — grep statvfs|available_space|disk_free|used_bytes over server/lib/src → 0 hits — so even an operator cannot be told the store is at 80 %. A rule with no reading behind it would be a lie in a config file. never is honest.
Does holger need growth for its own store? Yes, and for a reason that has nothing to do with money: holger's store is written by other people's pushes and by proxy caches; it fills whether or not anybody pays. gunnar's argument for runtime growth — "a disk that fills and stops" — applies verbatim. What differs is the driver:
| gunnar | holger | |
| what raises the need | a PAID order raises the entitlement cap (monetize batch.rs:479-481 "It never buys disk"); the batcher grows on the watermark |
a push or a cache fill; nothing is sold |
| who reads the fill | Admin.Health twins[] + Admin.Capacity; monetize on the front |
nobody today |
| who orders the resize | monetize-server's batcher (grow.rs:1-19), cloud half in monetize-impl |
nobody; gb in estate.toml + apply, by hand |
| the appliance half | Admin.GrowStart/GrowResume, drain → unmount → hot-plug (korp-installer grow) |
absent in holger-server; but the appliance is laid by the same korp-installer, so korp_installer::grow::runtime::grow_and_mount is reusable as-is (READ: data-set-grow-runtime.md:38) |
So: growth for holger is driven by fill and an operator, not by monetize — until the feat/metering-doors / monetize-doors worktrees (present on this box: holger-wt-metering, holger-wt-one) make holger sell storage, at which point gunnar's batcher applies unchanged (it is product-agnostic: monetize_product::Product::twins). The order of work, each a precondition for the next, and each of which keeps never until it lands:
- a fill reading on holger's health surface (bytes used / volume size, per data root) — the thing every other step reads; gunnar carries it on
Admin.Healthand on every twin heartbeat; [estate-twin-store]becomes a runtime invariant, not only a declaration: the twin's volume must be ≥ the primary's before the primary grows (gunnar G12,DATA-SET-GROWTH-FLOW.md:349) — and holger's twin, unlike gunnar's, is allowed to be promoted, so its store must not merely hold the set but serve it after a resize;- an operator verb (
cargo xtask grow server --gib N --applyin private-holger-ops, the shape of gunnar'sxtask grow) that does drain → unmount → resize → re-attach → grow by hand, with the hot-plug kernel pin (tunnr kernel-pci-hotplug) required by the installer row; - only then a rule other than
never, and only then a batcher — monetize's, if holger sells; a plain on-fill timer on the front if it does not.
Retention is the other lever and holger already has it (server/lib/src/retention.rs): a proxy cache can be evicted instead of grown. A hosted repo cannot. Growth is for the hosted set.
6. The stinky list
Severity: ★★★ wrong or missing where the design promises; ★★ a design contradiction someone will build on; ★ stale prose.
- S2-1 ★★★ No network replication exists in holger — not to a twin, not to C. The engine is local-only (
proto:359-368,replication-design.md:66-73); everyRepositoryBackendTraitimpl is local or a foreign registry (§3). The estate declares a twin whose only inbound rule is a gRPC port nothing will ever dial (estate.toml:401). MEASURED. - S2-2 ★★ The owed twin driver is specified in the wrong direction.
estate.toml:364-365"a timer on the PRIMARY that dials this machine's :50051". gunnar's proven twin pulls (twin-transaction-log.md:74-84), the primary dials nobody (F13), and pull needs no new firewall row on holger's primary (:259-263). Fix the sentence before somebody builds it. READ. - S2-3 ★★★ The twin has no ROLE. Same binary, same config — correct — but nothing makes the twin refuse writes, bind its product doors to loopback, or run a follower. Today
holger-twin, if it were ever installed, would be a second primary with an empty store and public-shaped doors closed only by a firewall. gunnar'stwin.json/--role twin(:280-286) is the pattern. MEASURED (norole/twininserver/lib/src). - S2-4 ★★★
replicate_pairscannot make a twin. It moves artifacts by coordinate (replication.rs:6-8). A twin built from it holds no evidence store (DSSE/attest blobs), no audit segments, no properties, no RBAC/config, no update-bundle store. Promoted, it serves artifacts with no provenance and a history that starts at promotion — gunnar's un-metered-twin scar in a different coat. The twin driver must ship the data root's side stores too; the audit log is already append-only Arrow IPC segments (arrow_ipc.rs) and the attest store is content-addressed — both shippable by name/digest with no new format. MEASURED + DESIGNED. - S2-5 ★★ "MVP has no outbound" is HTTP-only and the doc does not say so.
object.rs(a tonic client) is compiled into every tier (lib.rs:70); the guard exemptstonic(holger-tiers.md:290-291). The posture question "may a downstream dial upstream" is real and open; the guard is not evidence either way. MEASURED. - S2-6 ★★ A replicate run is audited as
Upload, not as the peer'sDownload(services.rs:1720-1737). Under C-KNOWS the upstream must be able to answer "what did C download" fromDownloadrows keyed by C'sident. Push-shaped replication produces no such row; only a pull by C does. This is the single fact that decides direction. MEASURED. - S2-7 ★★ The airgap export has no identity rule.
holger bundle importexists (secure_update.rs:34-37); nothing says the export on the connected side must run as C's principal, so the "as of the last fetch we served" column would show the operator, not C. Two lines in the export verb. DESIGNED. - S2-8 ★★ No audit query surface (
feature-complete-gaps.md:324-325,AuditLogisrecord+flush). The ruling makes the audit log the source of truth for column C and it cannot be asked a question.read_dir(arrow_ipc.rs:299) is the primitive; aAuditQuery{ident, action, since}RPC over it is the work. MEASURED. - S2-9 ★★ No peer identity model. A peer is any admin (
services.rs:1650-1660). gunnar's answer — a machine principal in a group (twin), enrolled by a bootstrap verb, with a read grant on the log (twin-transaction-log.md:288-306) — maps 1:1 onto holger's RBAC (groups and per-repo ACLs exist). Two groups:twin(reads everything),downstream(reads B). DESIGNED. - S2-10 ★★★
[estate-twin-store]is a declaration, and the twin must grow BEFORE the primary. gunnar G12 (DATA-SET-GROWTH-FLOW.md:349);monetize grow.rs:82-91twin_needs_gib. holger's estate refuses a declared smaller twin (estate.rs:585) and nothing checks the made disk except a manual step (front.rs:280-287). With no fill reading (§5) neither side can be measured. READ + MEASURED. - S2-11 ★ Stale header:
gunnar/.nornir/data-set-grow-drain.md:5-7"Design only — no code implements the hot-plug half yet" contradictsdata-set-grow-runtime.md:145-152andgrow/world.rs:159-175. READ. - S2-12 ★ gunnar S9 vs. the twin design. S9 says promotion would stop enforcing entitlements;
twin-transaction-log.md:123says there is no promotion verb. Both true; S9 should say "a re-image from the twin's medium would". holger's estate DOES promote its twin (estate.toml:370-372), so for holger S9 is live and the same-build refusal is load-bearing. READ. - S2-13 ★★
PoolDeltaandmirror_intoare dead (§3).PoolDeltais the "far side inventory" arithmetic a twin watermark or a C "what is new since the last export" line needs (pool.rs:452-469,new_fraction). Either wire it into the export verb (S2-7) or delete it; a primitive that exists only in its own tests is a promise nobody keeps. MEASURED. - S2-14 ★ Every grow rule is
neverand nothing reports fill (§5). Correct today; write the precondition list (§5 steps 1–4) next to the refusal so the next lane does not copyon-fillfrom gunnar "because the shape matched" — the exact failure the refusal names. MEASURED. - S2-15 ★★ gunnar's twin is dark and its boot stack has no grow engine (§2.6, MEASURED 2026-09-18, not today). Any "collaboration with gunnar" on growth is with a design plus KVM benches. Do not plan a holger runtime growth on the strength of a running gunnar one.
7. Patch sketches (DESIGNED — belongs in the doc, not in code, per the lane's brief)
7.1 The remote backend C uses — and the twin's artifact half
// server/lib/src/upstream/holger.rs (feature `upstream-holger`, full tier for the wire form)
/// Repository `repo` on another holger, read-only, under OUR principal.
pub struct HolgerUpstream { client: object::RemoteHolger, repo: String }
impl RepositoryBackendTrait for HolgerUpstream {
fn is_writable(&self) -> bool { false }
fn list(&self) -> anyhow::Result<Vec<ArtifactId>> { block_on(self.client.list(&self.repo)) }
fn fetch(&self, id: &ArtifactId) -> anyhow::Result<Option<Vec<u8>>> { block_on(self.client.fetch(&self.repo, id)) }
fn put(&self, _: &ArtifactId, _: &[u8]) -> anyhow::Result<()> { anyhow::bail!("remote holger is read-only here") }
}
// RON: (name: "b", kind: Upstream, format: "rust", upstream: Holger(endpoint: "grpc.holger.rs:50051", repo: "rust-b", auth: Mtls(..)))
// sync down = transfer((HolgerUpstream b, local b), gates=[AttestationGate::Block], evidence=None)
Every fetch lands on B as AuditEvent{ident: <C's subject>, action: Download, repo: "rust-b", …} — the ruling, for free.
7.2 The role
// holger-server serve --role twin --primary grpc.holger-server.utility:50051
// or <data-root>/twin.json — same effect at ServerStores::open:
// * every writable backend wrapped in FollowerOnly (put → Error::Follower, except the follower's own apply)
// * product doors bound to 127.0.0.1 regardless of config (the config stays byte-identical to the primary's)
// * spawn twin::follow(primary, cred) every N s:
// 1. replicate_pairs((remote r, local r) for every r) — artifacts
// 2. ship_segments(<primary>/holger-audit → local) — append-only Arrow IPC, by name, verify chain
// 3. ship_blobs(<primary>/attest → local) — content-addressed, by digest
// 4. report {applied_segments, artifacts_present, fill} — the heartbeat gunnar carries on LogHead
// Promotion (holger's estate allows it): remove twin.json, resize, firewall+DNS. Not a deploy.
7.3 The audit query the C column reads
service AuditService { // admin-gated, read-only over holger-audit/*.arrow
rpc Query(AuditQueryRequest) returns (stream AuditRow);
}
message AuditQueryRequest { string ident = 1; string action = 2 /* "download" */; int64 since_nanos = 3; string repo = 4; }
// UI column C: Query{ident: "downstream-<site>", action: "download", repo: "rust-b"} grouped by artifact,
// headed "as of the last fetch we served — sent, not confirmed received".
7.4 The export identity rule (airgap C, MVP)
holger-server bundle export --repo rust-b --as-principal downstream-<site> --out b.bundle
→ refuses unless the caller may impersonate that group member (admin), then records ONE
AuditEvent{ident: downstream-<site>, action: Download, repo, artifact, bytes} per member —
the courier is not the peer; the peer is.
→ optional: --since <snapshot> uses Pool::delta_since / PoolDelta.new_fraction for the
"how much of this bundle is new" line (S2-13).
7.5 The estate sentence to fix now (private-holger-ops/estate.toml:364-365)
- That driver is OWED: a timer on the PRIMARY that dials this machine's :50051 over the utility network.
+ That driver is OWED: a FOLLOWER ON THIS MACHINE that dials the primary's :50051 over the utility network
+ under its own machine principal (group `twin`) — the direction gunnar proved; the primary dials nobody.
and the twin's firewall row "tcp/50051 server" then becomes an outbound fact, not an inbound one.
8. WHAT LANDED — appended 2026-09-21, lane agent/w9-twin (nothing above renumbered or deleted)
This lane WROTE and implemented nothing; this section records what a later lane built on it. The reasoning above is unchanged and is still the source. Pointers only.
| open line above | status 2026-09-21 | where |
| S2-3 ★★★ The twin has no ROLE | CLOSED. role: Primary | Twin(primary: …) on Holger; an absent key is always Primary. A Twin refuses every write and admin RPC by name (holger-twin-1, naming the primary) at authorize_write/authorize_admin and on the HTTP write gate, and wire_holger rewrites every ron_url/ron_http_url onto 127.0.0.1 keeping the port — so the RON file stays byte-identical to the primary's, as §2.4 requires |
server/lib/src/twin.rs; server/lib/src/lib.rs (wire_holger, Holger::role); server/lib/src/grpc.rs; server/lib/src/exposed/http.rs. Tests a_twin_refuses_every_write_by_name, an_absent_role_is_primary_not_twin, a_twin_binds_no_public_door |
S2-4 ★★★ replicate_pairs cannot make a twin |
CLOSED for the stores this section named. replicate_estate sits BESIDE replicate_pairs (the published holger-traits surface is untouched) and moves the attestation store, the audit segments, properties, SBOMs, the sealed PROD store + lineage, update-bundle state, import receipts, the plugin floor and the served config — content-id idempotent and gated exactly as the artifact half, gatling_map_balanced for the read phase |
server/lib/src/replication.rs (ESTATE, EstateReport, replicate_estate). Test a_replicated_twin_holds_the_evidence_not_just_the_bytes |
| S2-2 ★★ The owed twin driver is specified in the wrong direction | HALF CLOSED. The direction is now a TYPE fact in holger: only a Twin yields a FollowPlan, and NodeRole::Primary has no field that could name a follower (a_primary_never_dials_its_twin). The sentence in private-holger-ops/estate.toml:364-365 is still the wrong way round — §7.5's patch is unapplied; the estate file is the owner's |
twin::NodeRole::follow_plan, twin::PrimaryView |
S2-10 ★★★ [estate-twin-store] is a declaration |
CLOSED as a READING. twin::store_verdict states twin.available_bytes ≥ primary.used_by_holger_bytes over two live Capacity readings and reports a shortfall by name, holger-twin-2, on GET /-/twin and in the log. A WARNING, never a refusal. StoreVerdict::Unknown is its own variant and is not a pass |
server/lib/src/twin.rs; plan §4 "?6 — the growth order, now that the fill reading EXISTS" |
| §5's premise: "holger has no fill reading at all" | EXPIRED the day after it was measured. metering::statvfs + capacity::measure + AdminService.Capacity landed 2026-09-20. §5's step list is still the thing to read; its opening sentence is not |
server/lib/src/capacity.rs, server/lib/src/metering.rs |
| S2-1 ★★★ No network replication exists in holger | STILL OPEN, and deliberately so. replicate_estate is local: primary data root → twin data root. The follower's seam is trait PrimaryView and the only implementation shipped is LocalPrimary. The wire form is holger-remote, plan W11 |
twin::PrimaryView / twin::LocalPrimary |
| S2-5, S2-6, S2-7, S2-8, S2-9, S2-11, S2-12, S2-13, S2-14, S2-15 | UNCHANGED. Nothing in this lane touched the C side, the audit query surface, the peer identity model or any growth rule; every machine is still rule = "never" and nothing resizes anything |
— |
⚠ What was NOT built, and must not be read into the above. No follower thread is spawned by start() — follow_once is a function with a test, not a running driver. No xtask grow --apply: it needs the hot-plug kernel pin and it resizes a real volume, so it is the owner's (plan ?6 step 3). No rule other than never.