THE HOLGER PLAN
.nornir/stinky/ holds the ARGUMENTS and the measurements. This holds the QUESTIONS and the ORDER. If they disagree, a stinky section is evidence and this is intent — fix this file.
The rule that protects the questions. A question is never renumbered, never deleted and never quietly dropped because it got stale. Answering one means appending ANSWERED <date>: and leaving everything above it in place, so a year from now the reasoning is still readable. Answered questions stay in §4. Nobody has to answer them in order or all at once — §2 exists precisely so work continues while they wait.
1. THE QUESTION REGISTER
Sorted by what it costs to leave unanswered.
| ? | the question, in one line | blocks | cost of delay |
| ?1 | ANSWERED 2026-09-20: C PULLS. A mirror: RON block plus a mirror engine fetches from upstream on a timer; C stood up on oden and was proven pulling from holger.rs. The upstream still knows what C has, because it SERVED the fetch |
W11, the last MVP feature | closed 2026-09-20 |
| ?46 | ANSWERED 2026-09-20, owner: option (a) — an additive, opaque, SOURCE-SCOPED change_token on ArtifactEntry + RemoteAsset, folded free from the per-chunk BLAKE3s the znippy index already holds and compared only against another token from the same source. Nine of the fifteen znippy backends answer it; the six that list through a typed view cannot name their row and say so |
the other half of ?1 | closed 2026-09-20 — a znippy-backed mirror now settles on the second cycle under the DEFAULT policy |
| ?25 | ANSWERED: no self-update. holger is immutable; a new version is a new image | — | closed 2026-09-20 |
| ?26 | ANSWERED 2026-09-20: REFUSE — enum Floor { InForce, Absent }, and an import against Absent fails with holger-update-38 unless --first-import is passed |
W14 | closed 2026-09-20 |
| ?27 | ANSWERED 2026-09-20: it MAY erase it, and must ANNOUNCE it by path — plus ?26's refusal at the other end, which is the half that cannot be missed | W14 | closed 2026-09-20 |
| ?3 | which metering branch is the base? | W8 | HIGH — two 2 300-line branches drift further apart every week |
| ?10 | ANSWERED 2026-09-20: ONE, and it is holger's own attest.rs DSSE path; modgunn stays scan-only. It is compiled in every row — what is missing is a CALLER (W6/W3) |
W17 | closed 2026-09-20 |
| ?9 | ANSWERED 2026-09-20: a BUG THAT SHIPS — nornir --seal now loads a real key from NORNIR_PROVENANCE_KEY/_CERT and REFUSES without one; --seal-dev is the named opt-in and writes its CA out |
W17 | closed 2026-09-20 |
| ?12 | ANSWERED 2026-09-20: YES, publish all four | — | closed |
| ?19 | ANSWERED 2026-09-20: NAMED on all five server rows (lane/r9 d17a6b8); 433 crates before and after |
W2 | closed 2026-09-20 |
| ?21 | ANSWERED 2026-09-20: the Gate trait is MVP — the 2026-09-19 ruling already named gate orchestration inside the MVP chain |
W3 | closed 2026-09-20 |
| ?23 | ANSWERED 2026-09-20: FINISHED, both. One shared bounded zip reader (holger-zip-entry, cbe35d1) unlocked both: nuget now serves RegistrationsBaseUrl + SearchQueryService off the .nuspec (0e23a0e), composer synthesises p2 off the composer.json in each dist (3a36db9) |
— | closed 2026-09-20 |
| ?24 | ANSWERED 2026-09-20: KEEP — raw/group/proxy/the wasm handler DO implement the seam, the gRPC door enforces it, and the boot refusal is opt-in and by name |
— | closed 2026-09-20 |
| ?28 | a decommission verb — none exists; an entitlement ships with resold hardware | W8 | MEDIUM |
| ?2 | ANSWERED: mimir is ABSENT and already so; modgunn stays LINKED with features = ["scan"] |
the MVP table | closed 2026-09-20 |
| ?16 | does the appliance stay ONE row? | W7 | MEDIUM |
| ?17 | fuel and deadline for a Meter guest | W7 | MEDIUM |
| ?5 | ANSWERED 2026-09-28 by the owner: ReadOnly is the default (keep serving what is verified, accept nothing new, the admin sees red); Refuse is an explicit per-estate choice. on_expired_metadata unstated = ReadOnly; a word that is neither is refused at the door (holger-update-39) |
W13 | CLOSED |
| ?7 | does the holger-remote pull ship in MVP, or only full? |
W11 | LOW — downstream of ?1 |
| ?4 | who owns / after the split? |
— | LOW |
| ?6 | ANSWERED 2026-09-21: steps 1 and 2 of the forced order LANDED; step 3 (xtask grow --apply) is the owner's and needs the hot-plug kernel. §4 "?6 — the growth order, now that the fill reading exists" |
— | closed for steps 1–2 |
| ?11 | MEASURED 2026-09-21 (§4 "W17"): the CLAIM is a rename and is DONE behind korpsig-export; the SIGNATURE is not and cannot be — KORPSIG1 covers 62 fixed bytes, DSSE covers PAE over the whole statement. STILL OPEN, priced at ONE certificate on edda's side |
W17 | LOW |
| ?13 | ANSWERED 2026-09-20: NOT a gap. The boot chain verifies a three-leg verdict; the earlier finding was wrong | - | closed |
| ?15 | ANSWERED 2026-09-20: the paid plugins live in vetra/monetize-impl; the open half is gunnar.rs/nordisk/monetize |
— | closed |
| ?20 | ANSWERED 2026-09-21: YES — holger-ds is a [[bin]] of holger-bench-scenarios (agent/w14-floor), CLI surface unchanged, every shipped closure the same size before and after (§4 "?20") — original text: holger-ds moves to its own binary inside the bench crate — confirm? |
W1 | The bench crate becomes dev-only (?8). ⚠ MEASURED, AND IT NEARLY COST THREE AUTH DOORS: xtask and agent/cli reach the bench crate through [dev-dependencies] ALREADY — the only non-dev edge is server/cli's optional entry behind feature bench, and that edge is not a bench driver at all but the holger-ds CLI verb (?20). But that same pin is the ONLY declaration in the graph of auth-oidc, auth-oauth2 and auth-mtls; auth-passkey was rescued by name in 542e296 and the other three were not. Name them in server/cli in the same commit or three authentication doors vanish silently. Closure: 431 → 421 (bench's reqwest defaults, the actual leaking line) → 414 → 410. Also delete bench_nexus_container.sh: 237 lines of bash, referenced by nothing, superseded, and a LAW violation |
| ?31 | may the competitor numbers be quoted? The honest figure is now 44.5x, not 80x - the 2021 pin was flattering us by ~2x | sales | HIGH |
| ?32 | bump the rivals: Artifactory 6.23.13 → 7.161.26, and a SECOND Nexus arm on 3.96.2 beside 3.72.0? | W0b | MEDIUM |
| ?33 | ANSWERED 2026-09-20: sign NAMED; the three testmatrix edges get a DATED EXEMPTION — they are NOT test-only (E0433 at grpc.rs:195) |
W2 | closed 2026-09-20 |
| ?34 | ANSWERED 2026-09-20: CLOSED by lane R12 — znippy-compress 0.9.14 is published (znippy master 3b60848); Gate B refused twice on stale floors first |
W16 | closed 2026-09-20 |
| ?35 | should the public box run musl again? no declared row gives musl AND front |
— | MEDIUM |
| ?36 | the public build needs GLIBC_2.39 and the box has exactly 2.39 | — | MEDIUM — no headroom |
| ?37 | ANSWERED 2026-09-20: option (a) — nornir dropped from xtask (lane/r9 9b7d3b5); duplicated holger crates 25 → 0, lock packages 1341 → 1010, none added |
— | closed 2026-09-20 |
| ?38 | spend a release cascade on a Method newtype? 49 impls, 174 call sites, published trait |
— | LOW |
| ?39 | ANSWERED 2026-09-21: STALE — every declared row is openssl-free, measured per row with cargo tree -i openssl-sys; closed by W1 + W2 + ?37 landing. Neither option needs taking. §4 "?39" |
the ORIGINAL delivery | closed 2026-09-21 |
| ?47 | may the installer medium carry the admin token? The estate emits a token_file path and NOTHING ever writes it; the declaration door refuses a secret-named param BY DESIGN. Three ways out, all the owner's — §4 "?47" |
an unattended install with a working console | HIGH — a shell-less appliance cannot be given one afterwards |
| ?48 | the emitted RON is the CLOUD box's (/var/lib/holger, artifact door on loopback behind Caddy); the installed appliance's home is /data and it has no Caddy. The estate's plan step and the installer declaration disagree by a path — §4 "?48" |
a medium whose config matches the shape it installs | HIGH |
| ?49 | a gRPC door that FAILS TO BIND prints "listening", prints "Holger is running." and the process stays up (server/lib/src/lib.rs:3341,3346). Fixing it wants a listener-first serve entry in edda's nornir-service — §4 "the installer half" |
every shape | MEDIUM |
| ?40 | ANSWERED 2026-09-21 (§4 "W17"): TWO keys, and the join is a CERTIFICATE rather than a merge — the chains are separated by their SIGNED MESSAGE, not by their key material, so merging seeds changes nothing and issuing a cert joins them either way. Path B's end is BUILT (declared trust root + provenance gate); path A's end is one X.509 over the boot signer, on edda | the whole chain | closed 2026-09-21 |
| ?41 | two nornir defects: an absolute path is ignored, and a nornir.toml with no [storage] SILENTLY moves the warehouse while still printing success |
every lane's measurements | HIGH - any lane can believe it persisted and be wrong |
| ?42 | a true ENOSPC benchmark needs a filesystem the bench may fill and destroy, which needs root | W0b | MEDIUM |
| ?43 | Gitea as a third rival - pulled and ready on oden, but needs its own admin bootstrap and upload path | W0b | LOW |
| ?44 | merge the refuse-to-start-without-GPU default? It stops nornir-server on every box without a GPU until someone adds --cpu |
nornir lane-r8-gpu-provider |
MEDIUM |
| ?45 | should nornir/vector-gpu ship in dwarves-server? The twins/deps lens REFUSES on every run today for want of it |
dwarves | MEDIUM |
| ?29 | ANSWERED with ?25: a plugin is an ARTIFACT, so no bundle change is needed | — | closed 2026-09-20 |
| ?30 | ANSWERED 2026-09-20: YES — a Meter guest owns nothing durable so it ArcSwaps; a PackageHandler owns a serve path and stays restart-only |
W7 | closed 2026-09-20 |
?1 — Does C PULL from B, or receive a carried bundle?
holger's own design document argues BOTH sides (§5 vs §6a.2) and they cannot both ship.
| carried bundle | holger-remote pull |
|
| zero-egress guarantee | intact and true | the guard measures the wrong thing — the tonic client is EXEMPT and already links into every tier |
| does upstream know what C has? | no | yes — AuditEvent{action: Download} grouped by ident |
| what the UI may say | bundled | sent, never accepted |
| prerequisite | an audited export under C's own principal | C needs an identity of its own and a route upstream |
⚠ Either answer needs an AUDIT QUERY SURFACE, which does not exist. W4 is owed whichever way this goes, so it sits in §2 and is not waiting.
?25 and ?29 — ANSWERED 2026-09-20: holger is IMMUTABLE. It does not update itself, and a plugin is an ARTIFACT.
The owner, plainly: "IMMUTABLE! Fattar du vad det innebär? Ta emot en version av sig själv????" He is right and the earlier proposal is withdrawn.
holger never mutates itself. A new version is a NEW IMAGE: build the ISO, boot it. That is what boot_order = "cdrom,disk" on the appliance and the twin is for — a re-image is "attach ISO, reboot". An RPC that took an update into a RUNNING appliance would contradict the design, so it is not being built. UpdateService. ImportBundle is CANCELLED (the W19 rung is struck below).
And the plugin question collapses into something simpler. A plugin is a .wasm file. holger stores files immutably, by digest, with a signature. So the plugin needs NO new way in at all:
- it is published into a repository like any other artifact;
- the RON config names it BY DIGEST, and the digest is the pin — swap the file and it no longer matches;
- it rides the same signature chain as every other artifact;
- and in an airgap it arrives exactly the way all other content arrives.
No new format (the bundle container is already a znippy archive — update-bundle/ src/container.rs:1: "No new binary archive format is invented"), no new store, no new configuration language (config is already RON, server/lib/src/lib.rs:40).
Consequences: the plugin's own anti-rollback question narrows to the ordinary artifact rules, and the two remaining brakes are the signature chain and the capability grant in RON.
?26 — ANSWERED 2026-09-20: REFUSE, unless --first-import is passed by name
RollbackStore::load returns AcceptedState::empty() for a missing directory and check_admissible compares only against the highest accepted sequence — so an empty state makes EVERY sequence New, including an older vulnerable one. The hole cannot be fixed by MOVING the floor (nowhere on an appliance is out of reach of whoever holds the disk); fix the SILENCE: enum Floor { InForce(AcceptedState), Absent }. Recommended: refuse without an explicit first-import flag.
ANSWERED 2026-09-20: refuse. Implemented on lane/r9 9786cb0.
update-bundle/src/rollback.rs now carries the type the question asked for:
pub enum Floor { InForce(AcceptedState), Absent }
RollbackStore::floor(supplier, product) returns it. Floor::check_admissible takes a third argument, first_import: bool:
InForcedelegates to the unchangedAcceptedState::check_admissible;Absentwithfirst_importreturnsAdmissibility::New— this bundle establishes the floor;Absentwithout it REFUSES withBundleError::RollbackFloorAbsent, carrying the new frozen codeholger-update-38.
RollbackStore::load is untouched, so nothing that wanted a plain AcceptedState changed behaviour. holger-server bundle import grows --first-import and prints a warning line whenever the floor is not in force, whichever way it then goes.
Two details that are part of the answer rather than decoration:
- An emptied directory is
Absenttoo. A state directory that exists but holds no.acceptedmarker admits exactly what a missing directory admits, so it has to answer the same way — otherwiserm *.acceptedwould be a quieter erase thanrm -r. Proven byan_emptied_state_directory_reads_as_absent_not_as_a_cleared_floor. --first-importis not a downgrade bypass. With a floor in force it changes nothing: an older sequence is still refused. Proven byfirst_import_does_not_weaken_a_floor_that_is_in_force.
?27 — ANSWERED 2026-09-20: wipe MAY erase it, and both ends now say so
It does today, silently, so wipe-then-import of an OLD paid module is admitted. ⚠ The second brake does not cover this: the module's CMS chain is NOT rollback-aware, so a correctly signed old module passes forever. The floor is the only downgrade stop there is.
ANSWERED 2026-09-20 (lane/r9 9786cb0): yes, wipe may erase it — and it may not do so in SILENCE. Both ends are now closed, on lane/r9.
Keeping the floor through a purge was considered and rejected. The floor is data, it lives in the data tree, and a --purge that quietly preserves state under a name the operator never asked to keep is its own kind of lie — the mirror image of the one being fixed. What the purge owes is a statement, and what the IMPORT owes is a refusal.
- At the purge end:
purge_holger_datawalks each root forholger-update-statedirectories BEFORE erasing (afterwards there is nothing left to name), keeps only the ones that are actually gone, and printsFLOOR GONE : N anti-rollback floor(s) DESTROYEDwith every path, followed by the consequence in plain words: the nextbundle importinto this store will refuse until it is given--first-import. - At the import end: ?26's
Floor::Absentrefusal,holger-update-38.
The second half is the load-bearing one and that is deliberate. An announcement can be missed — scrolled past, run under >/dev/null, executed by a script nobody reads. A refusal cannot. ⚠ And it has to be load-bearing, because the second brake does not cover this at all: the bundle's CMS chain is not rollback-aware, so a correctly signed OLD bundle verifies forever.
?3 — Which metering branch is the base?
monetize-doors (f0ce959) vs feat/metering-doors (f6ff7ec), ~2 300 lines each, written ONE HOUR apart and merged never.
RECOMMENDED, measured: f6ff7ec wins. f0ce959 is fatally unwired — entitlement::install() has ZERO call sites so its door is permanently failed_precondition, its store is a process-global map that forgets every fact on restart, it writes no audit line for a plan change, and its CapVerdict::check() is called by nothing outside its own tests. But f6ff7ec has NO enforcement at all, so the cap, METERED_CAP_EXCEEDED and the 0 = no cap rule must be carried over from the loser. The frozen holger-metered-* numbers disagree, so only one numbering can be frozen.
⚠ Both share a defect to fix first: list(...).unwrap_or(0) stamped REPOSITORY scope is a confident FALSE ZERO on the four backends that never override list, and it turns a broken disk into an empty repo — and f6ff7ec lets a cap be SOLD against that meter.
?9 — ANSWERED 2026-09-20: a BUG THAT SHIPS. --seal now loads a real key and refuses without one
nornir --seal (src/bin/nornir.rs:6003,6031) mints an in-memory CA and DISCARDS the root (seal.rs:58-66), so every provenance signature in the house chains to a CA that no longer exists. from_pkcs8 already exists at seal.rs:33-48; the CLI simply never calls it.
ANSWERED 2026-09-20: a bug in effect, whatever it was in intent. Fixed in the nornir repo, branch lane/r9-seal-key 4e3f18b.
One correction to the finding: ZnippySealer::dev() does NOT discard the root — it RETURNS it, as (Self, Vec<u8>). The CLI discarded it, at both call sites, with let (sealer, _roots) = .... The signature was real and the trust anchor was thrown away one line later.
src/bin/nornir.rs gains provenance_sealer(seal_dev, repo_root), the one place the signing key is resolved, and it has four outcomes:
| configuration | outcome |
NORNIR_PROVENANCE_KEY + NORNIR_PROVENANCE_CERT |
a REAL key via the from_pkcs8 that had no caller. NORNIR_PROVENANCE_ALG (ed25519|p256, default ed25519) and NORNIR_PROVENANCE_CN (default nornir-build-signer) |
| exactly one of the two | REFUSE by name. A half-configured key store is a typo, and falling back quietly is how this bug happened |
neither, no --seal-dev |
REFUSE by name, naming the four variables to set and the flag to pass |
neither, --seal-dev |
the dev CA — and its certificate is WRITTEN OUT, to <repo>/.nornir/build-provenance/dev-ca-<ts>.der, with a warning that says it is the only copy |
The shape is the GPU ruling's: stop, and make the caller name the fallback.
--seal-dev is also added to the viz BUILD button's argv (src/viz/build_tab.rs::attest_seal_argv), which is a DECLARATION rather than a new behaviour — that button has always used the dev signer. The difference is that the CA now survives the process, so the seal it produces can actually be checked. A box with the env pair set uses the real key regardless.
⚠ Every seal produced before this commit is unverifiable and cannot be made verifiable after the fact: the CA that signed them is gone. They have to be re-sealed with a real key.
⚠ NOT COMPILE-VERIFIED, and the reason is not this change. nornir does not build on loki at 27ebcc5 for two independent pre-existing reasons, both in code this commit does not touch:
- registry
znippy-compress 0.9.13needsznippy_common::slotpool, whichznippy-common 0.9.15removed. That is ?34, which is the owner's. It arrives throughembed-holger→holger-server-lib→holger-rust-znippy-repository. - built WITHOUT
embed-holger,src/server_grpc.rs:1508,1551,2417fail with threeE0063:nornir_bench::BenchRunandpb::BenchRungainedidandmachine_facts— lane R6's edda change, which nornir's main has not caught up with.
So nornir cannot be compiled by anyone today, with or without this commit.
?10 — ANSWERED 2026-09-20: ONE gate, and it is holger's own sign/attest.rs DSSE path
modgunn's provenance-verify is ON BY DEFAULT (Cargo.toml:18) while holger pins features = ["scan"] and hides its own attest.rs behind a non-default sign. So in a shipped holger, NEITHER runs.
ANSWERED 2026-09-20: ONE gate, and it is holger's own. Two of the three facts above were re-measured and one of them is wrong.
Wrong: sign is not "non-default" and attest.rs is not hidden. sign was pinned UNCONDITIONALLY on server/cli/Cargo.toml's server-lib line, so attest.rs's DSSE verifier (parse_envelope + verify_dsse at attest.rs:806 and :822, delegating to znippy_common::sign::dsse_verify) is COMPILED INTO ALL FIVE SERVER ROWS and has been all along. As of lane/r9 d17a6b8 the five rows name it, so it is now compiled in BY DECLARATION. The feature-off twin at attest.rs:847-860 returns AttestError::VerificationUnavailable — it refuses rather than degrading — and in a shipped holger that twin is not the one built.
Right, and it is the actual defect: nothing CALLS it. promote/mod.rs:123 still hands modgunn blob_sha256: "", and security.rs:434's promotion_policy passes ProvenancePolicy::Ignore, a fail-open its own doc comment names in full. So the gate exists, is compiled, and is never reached.
Why modgunn's is NOT the one. modgunn/Cargo.toml's provenance-verify opens a znippy-sealed <artifact>.build-provenance.znippy — the blob nornir's seal::dump_beside_artifact writes. holger does not store that object. holger stores DSSE statements, built by attest::statement_bytes and sbom::sbom_statement_bytes, and verified by attest.rs. Pinning provenance-verify would compile a SECOND verifier, for a format holger never receives, with no caller — which is the two-gate answer the question rejects. MEASURED: modgunn/src/service.rs has no server either, so it cannot be reached out-of-process.
The ruling, in one line: attest.rs is holger's provenance gate; modgunn stays scan-only; making it RUN is a CALLER, not a feature flag — W6 and W3, lane R3. Two things R3 must land for this ruling to be true in a running binary: a real blob_sha256 at promote/mod.rs:123, and a promote path that stops handing the gate ProvenancePolicy::Ignore.
⚠ One correction landed outside holger: modgunn/Cargo.toml's provenance-verify doc block still claimed "OFF by default" while default five lines above reads ["scan", "provenance-verify"]. Fixed in modgunn 4b124f5 on branch lane/r9-provenance-comment. That commit is LOCAL ONLY — ssh:// git@codeberg.org/nordisk/modgunn.git answers "Cannot find repository: nordisk/modgunn", so modgunn currently has no reachable remote. Owner action.
?12 — Go-ahead to publish four crates?
nornir-auth-webauthn 0.1.2 (absent from crates.io, and the ONLY blocker to deleting four hand-rolled WebAuthn verifiers) · modgunn 0.0.6 · facett, the last remaining [patch.crates-io] in holger, which alone holds 19 holger crates as fork-riders · and MOVING gunnar-clock to edda as nordisk-clock. The standing permission covers "publish what a build needs"; facett and a crate MOVE are more than that.
?19 — ANSWERED 2026-09-20: NAME it. Done on all five server rows, lane/r9 d17a6b8
MEASURED: sign and znippy-common ship in ALL FIVE server rows and no variant declares either. It rides in on server/cli/Cargo.toml:71, whose own comment says "rides in on every tier". cargo shows the distinction directly: a NAMED feature's chain ends at a (command-line) node, while sign's parent is the bare crate. Naming it on the rows changes no binary. Until answered, RULE 1 is RED by design.
ANSWERED 2026-09-20: named, and RULE 1 is green on sign. lane/r9 d17a6b8. The unconditional features = ["sign"] came off server/cli's server-lib dependency line and became a sign = ["server-lib/sign"] passthrough feature. embedded, mini, full, mvp-server and mvp-server-musl name it in holger-recipe/src/variants.rs; server-local gets it from server/cli's default, which is that row's one writer. .nornir/build-variants.toml was re-emitted by cargo run -p holger-build-thing -- emit.
MEASURED, cargo tree -p holger --edges normal:
| build | crates before | crates after |
default (server-local) |
433 | 433, byte-identical crate list |
embedded,sign |
— | 372, znippy-common present |
mini,appliance,sign |
— | 387, znippy-common present |
full,front,sign |
— | 433, znippy-common present |
mvp-server,appliance,auth-passkey,sign |
— | 394, znippy-common present |
cargo test -p holger-recipe on loki: 15 + 5 + 1 passed, including the_committed_toml_is_the_codes_emission_byte_for_byte.
?21 — ANSWERED 2026-09-20: the Gate trait is MVP
Filed Pro on the "does it run today" test: PolicyGate works and there is no second gate to plug in. But the 2026-09-19 ruling put the WHOLE promote-attest-sync-verify chain, gate orchestration included, into MVP — which may already answer this. The plugin design needs it for the cap, and it is NET-NEGATIVE surface: it deletes env-var policy parsing.
ANSWERED 2026-09-20: MVP. It was already decided and this is the ratification. Three reasons, none of them new:
- The 2026-09-19 ruling put the whole promote-attest-sync-verify chain into MVP, and gate orchestration is named inside that chain. Filing the trait as Pro would put a MVP rung's own orchestration outside MVP.
- It is NET-NEGATIVE surface. W3 deletes the env-var policy parsing it replaces, so "MVP" here does not mean more code to write before MVP ships; it means less code in the shipped binary.
- W7's capability cap needs it. A Meter guest's verdict has to enter the same place a promote policy enters, and there is exactly one such place.
W3 therefore stands in §2 as MVP work, not as a Pro item waiting for a second gate to justify the abstraction. Lane R3 owns the implementation.
?23 — ANSWERED 2026-09-20: FINISH them. Both are finished, on branch agent/zip-backends
The finding stands as written: neither could work, by OMISSION rather than by todo!() — nuget had no RegistrationsBaseUrl or SearchQueryService so dotnet restore of a normal project did not resolve, and composer's packages.json pointed at a metadata-url nothing answered, so every composer install 404ed on p2. The owner's ruling was FINISH, and the 21-backend audit's own estimate held: one shared zip-entry reader unlocked both, because a .nupkg and a Composer dist are both zips and holger could not open one.
| sha | what landed |
cbe35d1 |
holger-zip-entry — a read-only, bounded ZIP member reader under server/repository/zip-entry. Closes the two traps the tar readers close (decompression bomb: the inflate is streamed through Read::take, never Vec::with_capacity(declared); path traversal: is_safe_member_path applied inside members() so a caller cannot forget) plus the third a tar has no analogue for — a central directory that lies: every declared size is a hint, a STORED member is clamped to what exists, and the declared CRC-32 is verified, so a forged member is an Err and not silently wrong metadata. Zip64 handled. flate2 was already in-tree, so no new crate reaches the airgap surface. |
0e23a0e |
znippy-nuget — nuspec.rs (the .nuspec out of each .nupkg, parsed by hand the way helm parses Chart.yaml; a DOCTYPE is skipped whole so no entity bomb; a bare 13.0.1 is emitted as the MINIMUM range NuGet spells [13.0.1, ), which is the single translation that makes a graph resolve) and registration.rs (registration index inlined up to 64 versions and linked past it, page and leaf documents, and the search response with q/skip/take/prerelease; ordering is NuGetVersion, not string). The service index now advertises RegistrationsBaseUrl and SearchQueryService beside PackageBaseAddress. |
3a36db9 |
znippy-composer — p2.rs synthesises p2/{vendor}/{package}.json from the composer.json inside each dist: require, autoload, a normalized version, and a dist block with an absolute URL, the reference and a sha1 shasum (Composer's own verifier uses sha1, not the house sha256). A curated archive's own p2 document still wins, probed with contains(). Non-version references become dev-{ref} and are served from the ~dev document Composer asks for beside every stable one. metadata-url unchanged. server-lib now hands composer its ron_base_url, as it already did nuget. |
The memoization, counted. Both catalogs mean opening every package in the archive, so both are taken ONCE in a OnceLock — the same shape as deb's Release and conda's repodata cache — and both fan out over znippy_zoomies::gatling, never rayon. Measured in the tests, not asserted in prose: twelve registration/search requests over a 4-package nuget archive cost 4 member reads, not 48, and the flat-container arms cost 0; twelve p2 requests over a 4-dist composer archive cost 4, not 48, and packages.json costs 0 because it is built from the file table alone.
Verification. oden has no .NET SDK and no PHP/Composer — checked, not assumed, and not installed. So the documents are verified against the NuGet V3 and Composer v2 specs, and the tests walk the exact request sequence each client walks, following each URL the previous document advertised until the bytes come back: service index → registration → dependency's registration → packageContent → 200 with the .nupkg; packages.json → p2 → dist.url → 200 with the zip, and the document's shasum re-computed over the bytes that were served.
?24 — ANSWERED 2026-09-20: KEEP IT. The premise does not survive re-measurement
ZERO of 17 backends implement coordinate_for_path; the source itself calls the gate "decorative" (exposed/http.rs:1880-1886); and the server REFUSES TO BOOT with HOLGER_QUARANTINE_PROPERTY set (lib.rs:2040-2060). A config field that bricks boot and a gate that judges nothing.
ANSWERED 2026-09-20: KEEP IT, unchanged. No code was deleted, and the reason is that all three parts of the premise are wrong when re-measured at 542e296.
- "Zero of 17 backends implement
coordinate_for_path."server/lib/src/raw.rs:434implements it and:430answersmaps_paths_to_coordinates() == true.server/lib/src/group.rs:70composes the answer over its members andserver/lib/src/proxy.rs:473forwards it.plugin/handler-bundle-core/src/lib.rs:229implements it for a wasm handler, and it is a named function in the plugin ABI (plugin/abi/src/lib.rs:89,178) with a native-vs-wasm agreement test (plugin/host/src/agreement.rs:192). So the seam is implemented, composed and proven across the ABI boundary. - "The gate judges nothing."
server/lib/src/grpc/services.rs:112callsis_quarantinedon the fetch path and refuses. The HTTP door'sserve_quarantined(exposed/http.rs:1899) refuses an artifact fetch when the backend cannot map the path —exposed/http.rs:1914serves only whenmaps_paths_to_coordinates()is true. The word "decorative" in that comment describes what the gate USED TO BE, past tense, in the paragraph explaining the fix; it is not a description of the code below it. - "It bricks boot." The boot check fires only when
HOLGER_QUARANTINE_PROPERTYis set — an operator's explicit opt-in — and it refuses with a named message that lists the repositories that cannot be gated and three ways to resolve it. That is not a brick. It is the house pattern this question was going to be answered with: an absent capability is an ANSWER, the refusal is loud and by name, and the safe posture is what you get unless you ask for something the configuration cannot deliver.
Deleting it would replace a fail-closed gate with a silent one. The register entry is therefore closed as KEEP, and the durable improvement the source itself names — a coordinate_for_path on more backends, so fewer repositories are ungateable — stays available as ordinary work, not as a question.
?28 — A decommission verb
None exists — no holger plugin forget, no entitlement revoke; the only eraser is wipe. An operator reinstalling to RECOVER is served; one reinstalling to RESELL is not: with Keep the entitlement ships with the hardware, and with wipe the floor goes with it. An entitlement that ships with resold hardware is discovered on somebody's invoice.
?2 — ANSWERED 2026-09-20: mimir is ABSENT, modgunn stays LINKED with scan
MEASURED ANSWER, awaiting ratification. mimir: the cost of "not built in" is ZERO because it is ALREADY TRUE — 0 matches of the string in holger's Cargo.toml, .rs, .proto and lock. It is 23 116 lines inside nornir/src/mimir* with no repo and no crate; linking it would mean linking nornir, which holger deliberately has no edge to. modgunn: stays LINKED, scan only — making it optional saves 29 crates but sbom.rs:71 IS modgunn's CycloneDX writer and PolicyGate is the promote gate; and reaching it as a service is impossible because modgunn has no server at all. Which is exactly what server/lib/Cargo.toml:137 already does. Ratify so it stops being re-asked.
ANSWERED 2026-09-20: ratified, and no code changes. Re-measured at 542e296: grep -rn mimir --include=*.toml --include=*.rs --include=*.proto over the holger worktree returns 0, and grep -n mimir Cargo.lock returns 0. mimir is absent, and staying absent costs nothing because nothing has to be undone. modgunn stays a normal, non-optional dependency at server/lib/Cargo.toml:137 — default-features = false, features = ["scan"], which is already the line in the tree. Making it optional would save 29 crates and break two things that are not optional: server/lib/src/sbom.rs:71 re-exports modgunn::sbom::{purl, to_cyclonedx, SbomComponent, SbomInput, SbomVuln} and is holger's ONE CycloneDX serialization seam, and PolicyGate is the promote gate. Reaching modgunn as a service is not an option at all: modgunn/src/service.rs has no server. This question is closed; do not re-open it without a new measurement that contradicts one of those three lines.
?16 — Does the appliance stay ONE row?
Turning the plugin host on there means the OPEN appliance carries the host half. Accept, or declare a second row and knowingly take gunnar's un-metered-twin scar. ⚠ Note the ?18 re-answer weakens the objection: the OPEN allocation guest is itself wasm, so the host stands on its own merits.
ANSWERED 2026-09-20 — ONE ROW, both features on. Full ruling in §4 (ed760dd).
?17 — Fuel and deadline for a Meter guest
evaluate sits on the push path. Proposal: 10e6 fuel / 250 ms, and a slow verdict REFUSES, per gunnar's Err-is-refusal rule.
ANSWERED 2026-09-20 — measured, then capped wide: 100e6 fuel, deadline unchanged. Full ruling in §4 (b5fadab).
?5 — ANSWERED 2026-09-28 by the owner: ReadOnly by default
2026-09-28, the owner's ruling: on_expired_metadata defaults to ReadOnly — keep serving what is already verified, accept nothing new, the admin sees it red. Refuse stays an explicit per-estate choice (Some(Refuse)). TrustPolicy::expired_posture() answers ReadOnly when the policy states none; a posture that is neither word is refused when the policy is read (holger-update-39). The history below is kept.
2026-09-28 (one-signed-time): the default below was a lane's answer to a question the workspace ruling (nornir-workspace.toml, "OPEN RICKARD (2)") still lists as the owner's. It is taken back: on_expired_metadata: Option<ExpiredPosture>, no serde default, and every downstream verb (holger-server bundle import, scrub) calls TrustPolicy::expired_posture() before it touches the store and refuses unset with holger-update-39. The producer's tools (holger-bundle, holger-verify) do not ask — they serve nothing. The text below is kept as the argument for ReadOnly, for the owner to weigh; it is not a ruling.
Written as a config field with NO DEFAULT, awaiting ratification.
ANSWERED 2026-09-20. The field exists now, on lane/r9 2b76459, and it IS defaulted:
// update-bundle/src/policy.rs
pub enum ExpiredPosture { #[default] ReadOnly, Refuse }
// TrustPolicy:
#[serde(default)] pub on_expired_metadata: ExpiredPosture,
Why a default is safe here when it was not for trust_mode. trust_mode decides WHO is trusted, and there is no safe guess to make on an operator's behalf: every possible default either admits a signer he did not choose or refuses one he did. That is why it is Option<TrustMode> with a named refusal (holger-update-34), and it stays that way. on_expired_metadata is a different kind of question. By the time it is read, the release signature has ALREADY verified against a trusted anchor — what is in doubt is only FRESHNESS. Both answers are sound against a forger; they differ only in availability. There is therefore a safe answer to guess, and leaving the field unset would not be caution, it would be an un-migrated policy refusing to boot over a calendar.
ReadOnly is the default because it is the strictly more available of the two safe answers and it still closes the thing that matters: a box on stale metadata accepts no writes, so it can never extend, re-sign or re-date its own staleness. Refuse is one word in the policy file for a deployment where an outage is the correct alarm.
⚠ What is NOT done, stated plainly. The posture has exactly one consumer and it is at SERVE time, and that consumer does not exist yet — it is W13. The IMPORT path refuses an expired bundle under BOTH settings, and that is the setting working, not ignoring it: an import is a WRITE, and ReadOnly forbids writes. verify.rs now says so where the refusal is raised. What the field buys today is that the policy SCHEMA has stopped being the thing that blocks W13, and that an un-migrated policy already carries the posture the ruling chose.
?7 — Does the holger-remote pull ship in MVP, or only in full?
Downstream of ?1. If MVP, the zero-egress guard must be rewritten to measure what it claims.
?4 — Who owns / after the split?
Today the apex is holger-server's front page and the console is mounted on paths beside it. Under the split the console could take /.
?6 — ANSWERED 2026-09-21: steps 1 and 2 have LANDED; step 3 is the owner's
Original text, kept: Every machine is never. holger has NO fill reading anywhere (0 hits for statvfs/available_space in server/lib), so a growth rule today would read nothing. Forced order: fill reading → twin ≥ primary as a runtime invariant → operator xtask grow --apply with the hot-plug kernel pin → only then a rule.
The premise expired the night it was written. See §4 "?6 — the growth order, now that the fill reading exists".
?11 — MEASURED 2026-09-21, STILL OPEN: half a rename. See §4 "W17 — a signature that means something"
Links 4 and 4b never join: gunnar signs SLSA's FIELDS in sshsig, the house verifies SLSA's STATEMENT in DSSE. gunnar's own comment calls the export "a rename and not a redesign". Not doing it may be the right airgap answer — but then say so.
?13 — ANSWERED 2026-09-20: NOT a gap. The earlier finding was WRONG.
The signing lane reported "tunnr signs and verifies NOTHING — sha256/blake3 pins only". That was measured inside tunnr's own repo only, and it missed that the boot chain tunnr ships IS korp-installer's, in edda.
MEASURED in edda/crates/korp-installer:
src/lib.rs:38— "startup re-checks the pin, the signature and znippy's per-chunk hashes". Three independent legs:archive-root-blake3 · signature-over-root · per-chunk-blake3.src/rootkey.rs:50,64—KORP_ROOT_SIGNING_KEYand aRootSigningKeyovered25519_dalek::SigningKey.src/esp.rs:475— the signature travels beside the app;src/rollback.rs:65speaks of "an old, genuinely-signedapp.znippy+app.sig".- The two
plan.require_signed = falselines insrc/raid.rsare TEST FIXTURES, and their own comment says the policy default is closed.
⚠ AND A REAL DEFECT, already found and recorded in src/selftest.rs:14-20, worth carrying because it is the same shape as three other findings today: the signature leg USED TO BE A SEAM THAT ALWAYS SAID YES. "At STARTUP, korp-start used TrustRoot, whose verify_root is Ok(()) unconditionally, and passed it None for the signature. The root blake3 leg and znippy's per-chunk leg were real; the signature leg was a seam that always said yes." And at install time, "run_install never called it, so an unsigned payload installed happily."
The fix adds NO new cryptography — it makes the existing cryptography REPORT, at both moments, as a three-leg verdict. The rule it enforces is worth quoting: "Never verify against a hash carried in the same place as the data." At install the expected root is obtained by hashing the SOURCE stream in RAM and is checked against a re-read of the TARGET DISK — two different places.
?15 — vetra/monetize-plugin: its own private repo, or a crate inside monetize-impl?
?20 — holger-ds moves to its own binary?
The bench feature gates a REAL verb — holger-server holger-ds with --scale/--preset/--data/--target/--eco. Under ?8 it cannot stay in the served binary. Proposal: a [[bin]] inside the bench crate, which declares no variant row and so can never be shipped by the build thing. A release note, not a silent drop.
?29 — payload_kind, or (supplier, product) alone?
PayloadEntry is media-type-typed with a free-form path, so a .wasm validates with NO format change. A defaulted payload_kind {Firmware, Plugin, PluginSignature} would stop a firmware being installed as a module, refused by name rather than by file extension. ⚠ Either way the .p7s must travel as a SECOND payload member, because the host looks for it beside the module.
?30 — ANSWERED 2026-09-20: yes, Meter guests hot-swap; PackageHandlers stay restart-only
A Meter guest can be replaced with no restart (load + verify + instantiate beside the old, ArcSwap the handle, in-flight calls drain on the old) because it holds no bytes, mounts no repository and owns no store. A PackageHandler stays restart-only: it owns a serve path, and a half-swapped handler is a 404 over bytes that exist.
ANSWERED 2026-09-20: confirmed. This is a technical conclusion, not a business one, and it follows from what each guest kind OWNS.
A Meter guest owns nothing durable. It holds no bytes, mounts no repository, opens no store and keeps no file handle; evaluate takes a reading in and hands a verdict back. So the swap is: load the new module, verify its CMS chain over the blake3 digest, instantiate it BESIDE the old one, then ArcSwap the handle. Calls already in flight keep the Arc they loaded and drain on the OLD instance; calls that start after the store see the new one. There is no window in which a call sees neither, and no state to migrate between the two, because there is no state.
A PackageHandler is the opposite case and stays RESTART-ONLY. It owns a serve path. A half-swapped handler answers 404 over bytes that exist on disk, which is worse than being down, because a client treats it as an authoritative absence and caches the conclusion. Restart is the honest failure mode: the port closes and the client retries.
The rule to carry into W7: a guest may hot-swap exactly when it owns no durable resource. Any future guest kind is measured against that one sentence, not against whether hot-swap would be convenient for it.
?31 — May the competitor numbers be quoted as they stand?
The numbers themselves are SOUND and reproduce: on oden, holger 105.33 MB/s / 176 687 ops/s / p999 0.668 ms against Nexus 3.72.0's 9.999 / 14 919 / 72.022 — 10.5x, 11.8x, and 107.8x on the tail. Fairness is generous: FairResources gave Nexus a 191 GB heap and every core, so the tail gap is GC, not bandwidth.
⚠ But the RIVALS are the problem, not the method. Nexus 3.72.0 is from 2024-09-04 and native Cargo landed in 3.73.0 — one release after our pin, so this harness cannot support any "holger beats Nexus on Rust crates" claim. Artifactory 6.23.13 is from 2021-02-21, EOL, and 29 patches behind inside its own dead line; Artifactory OSS serves no cargo, pypi or npm in ANY free edition. So as it stands we beat a 2024 Nexus that cannot serve Rust and a 2021 Artifactory that can serve neither Rust nor Python. Honest framing today: Nexus on maven/pypi, 2024 vintage.
?32 — Bump the rivals?
The source's stated reason for not moving Artifactory — "7.x needs microservices" — is FALSE: 7.x bundles them in one image and a live testcontainers module runs 7.161.26. It was pulled and booted; it exits in under 10 s on a percentage-based free-space guard reading podman's graphroot on oden's 98 %-full /home. 168 GB free is plenty of bytes; the guard counts percent. The blocker is housekeeping, not containerization. Also proposed: a SECOND Nexus arm on 3.96.2 beside 3.72.0 (the only route to a cargo-vs-cargo comparison; needs a EULA POST and minds a 40k/day cap), and Gitea, the only light container covering cargo + maven + pypi at once.
?33 - ANSWERED 2026-09-20: sign NAMED; the three testmatrix edges get a DATED EXEMPTION
RULE 1 is RED on sign: it ships in all five server rows and no variant names it; it rides in on server/cli/Cargo.toml:71. Naming it on the rows changes no binary. RULE 2 is RED on three nornir-testmatrix optional-NORMAL edges (server/cli:106, server/lib:199, agent/cli:75) - the same shape as the bench crate, i.e. the fourth leak waiting to happen. Either move them to dev-dependencies, or grant a dated exemption with a reason.
ANSWERED 2026-09-20, in two halves.
RULE 1 / sign: named. lane/r9 d17a6b8 — see ?19 above for the closure table. 433 crates before, 433 after.
RULE 2 / the three testmatrix edges: they STAY optional-normal, under a dated exemption written into all three manifests. lane/r9 b103d4e. The move to [dev-dependencies] was ATTEMPTED and REVERTED, and the reason is a compiler error rather than a preference: the premise that these edges are test-only is wrong. functional_status is defined in LIBRARY and BINARY code, not under #[cfg(test)], at three places — server/lib/src/grpc.rs:193, called by every gRPC handler and by the RBAC write gate; server/cli/src/main.rs:1095, called by every CLI verb; agent/cli/src/main.rs:194. MEASURED on loki:
cargo check -p holger-server-lib --features testmatrix
error[E0433]: cannot find module or crate `nornir_testmatrix`
--> server/lib/src/grpc.rs:195
A dev-dependency is not on a lib's or a bin's path in a normal build, so the move does not relocate a dependency — it deletes the verb-level emit that nornir test --features testmatrix reads. This is not the bench-crate move; the bench crate's uses really were dev-only.
Why the exemption is safe, and it is not a shrug: RULE 1 already covers the leak RULE 2 fears. testmatrix is in no default set and no variant in holger-recipe/src/variants.rs names it, so if it ever reached a shipped closure RULE 1 would go red on an unnamed feature — the same check that caught sign the same day. The bench crate leaked precisely because its feature WAS in default. The exemption's expiry condition is written in the manifests: re-examine it the moment testmatrix is added to a default set or named by a row.
?34 - ANSWERED 2026-09-20: CLOSED. znippy-compress 0.9.14 is published
MEASURED, and the measurement is why this row existed: znippy-common 0.9.15 REMOVED src/slotpool.rs, which exists in 0.9.13 and 0.9.14. Published znippy-compress 0.9.13 does use znippy_common::slotpool::{Magazine, PoolPlan} and fails with 8 errors. A breaking removal inside a patch bump, which orphaned a published sibling.
CLOSED by lane R12 on 2026-09-20: znippy-compress 0.9.14 is on crates.io (znippy master 3b60848). Gate B refused the upload TWICE before it went — the manifest declared floors (znippy-zoomies 0.1.13, znippy-common 0.9.14) that a resolver could satisfy with a version lacking the API the code calls, i.e. a crate that would have published and not compiled; the floors were raised to 0.1.25 / 0.9.15 and it published, indexed in 46 ms.
Confirmed independently by lane ZNIPPY-APPEND, which hit the orphan from the other side: holger in a lane worktree, resolving znippy from the registry, died with the same 8 slotpool errors until cargo update -p znippy-compress@0.9.13 --precise 0.9.14. So the fix is real, it is published, and W16 is unblocked — see the W16 row for the two things about W16 that are still not what it says.
?35 - Should the public box run musl again?
The binary that WAS live on holger.rs was a static-pie musl build predating the front feature split. No declared row gives musl AND front: mvp-server-musl omits front because the airgap tiers deliberately do, while server-local and full carry front and are glibc. The box now runs server-local, which is what the recipe says the systemd unit stages - so the estate matches its declaration, but the public box moved from musl to glibc. If musl is wanted there, a row must be declared for it.
?36 - GLIBC_2.39 with no headroom
The deployed binary's maximum required symbol version is GLIBC_2.39; the box is Ubuntu 24.04 with glibc 2.39. It fits exactly. Any box one release older cannot run it. Same class of fault as the OpenSSL one, not yet triggered.
?37 - ANSWERED 2026-09-20: option (a). nornir is off xtask's edge, and the 25 are gone
Not one stray pair: 25 holger crates appear twice - traits, errcode, plugin-host, both handlers and all 17 repository backends. One edge causes it: xtask/Cargo.toml [dev-dependencies] nornir = "0.5" pulls registry holger-server-lib 0.6.10, which pulls the other 24. holger's own test graph builds holger's PUBLISHED SELF from crates.io. It is already a dev-dependency so that move is spent. Options: (a) drop nornir from xtask, used by exactly one file (xtask/examples/nornir-bench.rs); or (b) [patch.crates-io] holger-server-lib = { path = "server/lib" }, which is not a foreign fork by the root manifest's own definition since holger produces that crate, and which would collapse all 25 AND make the benchmark measure the local holger instead of a stale published copy. Option (b) touches the publish gate.
ANSWERED 2026-09-20: option (a), done on lane/r9 9b7d3b5. nornir = "0.5" is off xtask's dev-dependency edge. The example now depends on nornir-bench directly — the crate that actually owns BenchResult, api::Bencher and register_bench!, which nornir only re-exports. LAW 5: reuse the leaf, not the warehouse that re-exports it.
MEASURED on Cargo.lock:
| before | after | |
| duplicated crate NAMES | 311 | 160 |
| duplicated holger crates | 25 | 0 |
| unique packages | 1341 | 1010 |
331 packages removed, 0 added.
Option (b) was not taken, for the reason the question already gives: it touches the publish gate, whose step 1 is "remove all [patch.crates-io] overrides".
⚠ And the example did not compile before this either. cargo build --release -p xtask --example nornir-bench failed at 542e296 with ten E0063 — the pre-existing break W21 recorded, nornir-bench 0.1.8 having added BenchResult.source while ten sites still build the struct by literal. Ten errors before the dep swap and the same ten after it, so the swap is not the cause; but ?37 cannot be verified against an example that does not build, so the ten literals are converted to BenchResult::measured in the same commit. lane/r5-bench-wiring makes the same change on the same lines and rebases onto this cleanly. PROVEN on loki: exit 0.
?38 - A Method newtype?
handle_http2_request(method, suburl, body) takes two &str; swapping them compiles silently and every route falls to the 404 catch-all. It cost one lane three round trips. But the method has 49 implementations and 174 call sites, and it lives on holger-traits, which is PUBLISHED and consumed by the registry copies in the same lock - so a newtype is a breaking trait change plus a release cascade across about 21 crates. The comment above the method already records a decision not to churn every implementer. There is no subset fix: the risk is at each call site.
1b. SOLVED 2026-09-20 — the grey button was ONE NUL BYTE
holger-front/assets/front.html carries a literal U+0000 NUL at byte 20659, inside the entry button's open-redirect guard, which was written with raw control bytes: if (/[<NUL>- <DEL>]/.test(start)) return;.
In a browser the HTML tokenizer's script-data state replaces U+0000 with U+FFFD before the JavaScript engine sees the source, so the engine is handed /[\u{FFFD}-\x20\x7f]/ — a character class whose range runs backwards. That is an early SyntaxError, and a SyntaxError discards the ENTIRE <script> element. The page has exactly one script, and it holds both the redirect guard AND the fetch('/-/front') that labels the button and clears disabled. So #msg kept its literal placeholder "reading the server …" and the button stayed grey for ever, with no named refusal.
Proven in a real browser with the house's own CDP harness (gunnar-ui/uitest/src/cdp.rs, through a new read-only --page <url> verb), first try:
#msg reading the server …
#enter off true
scripts 1
page errors uncaught: SyntaxError: Invalid regular expression:
/[<U+FFFD>- ]/: Range out of order in character class
There is no network entry for /-/front at all — the fetch is never reached. So HTTP/3, alt-svc, UDP and timing are all EXCLUDED; the failure is at parse time, before a single byte is requested. curl could never see it because curl does not parse the document.
FIX: /[\x00-\x20\x7f]/, verified in the same browser — the script now runs and reaches its own error path instead of staying silent. holger-front 28 tests green, including a new the_page_carries_no_raw_control_bytes that walks PAGE and fails on any control byte but tab, newline and carriage return.
⚠ THE CLUE THAT WAS EXPLAINED AWAY: file(1) reported the page as data rather than HTML, and grep switched to binary mode and printed nothing. That was read as "UTF-8 em dashes" and dismissed, and it cost hours. A text file that file calls data is a finding, not a curiosity.
?39 - Three ORIGINAL rows cannot be built in the environment the declaration pins
MEASURED on the gate's first real run, in nordisk-cargo-toolchain@sha256:3d5115ba... (rust 1.97.1, network none):
| delivery | result |
| MVP | 5 of 5 cargo rows BUILT - mvp-server 50.7 s, mvp-server-musl 145.9 s, agent 2.9 s, mvp-bundle 85.7 s, mvp-verify 2.5 s |
| ORIGINAL | 3 of 6 - embedded 43.1 s, mini 53.2 s, agent built; full, server-local and agent-online FAILED |
All three failures are openssl-sys 0.9.117: "Could not find openssl via pkg-config". Those are exactly the three rows carrying online. The rows that reach the network cannot be built at all in the declared environment - they build only on a box that happens to have libssl-dev, which is verdict-by-box, the very thing the container rule exists to prevent. The delivery the appliance ships is clean.
Two options, and the second has a cost worth stating: move those rows to rustls, or put libssl-dev into the pinned toolchain image - which means the pinned environment carries OpenSSL headers for EVERY build in the constellation, in a house whose airgap tiers are judged on their dependency closure.
WARNING, already measured: cargo build -p xtask --example nornir-bench fails for the same reason, so the benchmark driver cannot be built in the container either.
?40 - ANSWERED 2026-09-21 in §4 "W17": TWO keys, joined by a CERTIFICATE. The finding below stands as measured; holger's end of the join is now built.
MEASURED by walking it from the korp-installer end rather than from the map:
| what it covers | state | |
| Path A - the appliance a customer boots | blake3 root, KORPSIG1 signature, layout config, per-chunk hashes, fail-closed at every boot | WORKS - and carries no attestation at all |
| Path B - what holger serves | in-toto, DSSE, the VSA, update-bundle | covers no machine, and its signer chains to a discarded CA |
In one sentence: the machine a customer boots carries no attestation, and the attestation a customer can verify covers no machine. Path A never calls nornir --seal, never writes an in-toto Statement and never travels as an update-bundle, so links 4b/5/6/7/8 of the signing map's chain are not on that path at all.
What produces the key KORP_ROOT_SIGNING_KEY verifies against? NOTHING. It is a 32-byte ed25519 seed an operator makes by any means and puts in a file. No minting verb, no key store, no certificate over it.
Joining them costs exactly ONE certificate, and all three reasons are encoding rather than cryptography: znippy_common::sign dispatches on the signer cert's SPKI and already supports ed25519; an ed25519 PKCS#8 v1 private key IS that 32-byte seed behind a fixed 16-byte DER prefix (RFC 8410 7); and the public key is the SAME 32 bytes inside an X.509 SPKI and inside an ssh-ed25519 key. One seed = boot anchor + provenance signer + optionally a gunnar signing key. Three names, one key.
The seam is built and CANNOT be satisfied vacuously - five cases, each with a test: same key -> Bound; different key -> REFUSES to master the medium; no cert -> NotChecked and built as before; cert but no enrolled key -> REFUSES ("a declared join with nothing on one end is not a join"); neither -> NotChecked. The verdict is never Ok(()) and NotChecked is not a pass. Test vectors are EXTERNAL ground truth - a real ssh-keygen key with the fingerprint ssh-keygen itself prints, and a real OpenSSL ed25519 X.509 - so it cannot agree only with itself.
Trust stores if the join is made: artifact anchors 3 -> 1; things a customer enrols 6 -> 4 (the shared artifact key, WebAuthn per credential, UEFI PK/KEK/db, per-machine TPM AKs). Those four answer four different questions and collapsing further would remove a deliberate separation.
Code: edda branch lane/r10-key-lineage, crates/korp-installer/src/keylineage.rs, behind the non-default forge feature so no shipped PID-1 links a byte of it. 425 tests pass, zero clippy warnings, nothing published.
?41 - Two nornir defects that make a measurement lie
MEASURED by lane R4 while persisting its rows:
- A
nornir.tomlwith no[storage]section SILENTLY MOVES THE WAREHOUSE.bench runwrote to~/.nornir/warehousewhilehistory-show- same directory, same config - kept reading<workspace_root>/.nornir/warehouse. The run still printed5 result(s) persisted into bench_runs as 87d15865-..., andhistory-showanswered with ANOTHER lane's rows. Naming storage explicitly fixes it. This is the same shape as the 16-of-46 incident: a success line over a write that went somewhere else. Any lane can believe it persisted and be wrong. nornir bench run holgerresolves the repo to/home/rickard/git/holgerwhatever directory it is invoked from. A lane working in a worktree builds the OTHER checkout's driver, so its new arms are never linked - visible only asrunning 2 of 229 linked armswhere the lane's own binary reports 234. A relativepathworks; an absolute path is ignored, contrary to nornir-config's own doc comment.
?31 - The competitor numbers, re-measured against CURRENT rivals
Both rivals now run at their current versions as well as their pins, through identical code, with every row carrying the tag.
| arm | rival | holger | rival | holger p999 | rival p999 |
| nexus_load | 3.72.0 (2024-09-04) | 102.07 MB/s | 10.580 | 0.720 ms | 76.2 ms |
| nexus_load_current | 3.96.2 (2026-09-18) | 103.68 | 4.220 | 0.689 | 68.8 |
| artifactory_load | 6.23.13 (2021-02-21) | 98.38 | 12.568 | 0.683 | 54.7 |
| artifactory_load_current | 7.161.26 (2026-09-07) | 98.26 | 12.008 | 0.686 | 30.5 |
The modern Artifactory's tail is about twice as good, so our p999 lead falls from 80.2x to 44.5x. 44.5x is the number to quote. The five-year-old pin was flattering us by roughly 2x on the single metric the pitch leans on hardest. The modern Nexus serves 2.5x LESS than its 2024 pin, so the MB/s lead GROWS to 24.6x - true and misleading; keep p999 as the Nexus headline.
FORMAT CEILING, now stated beside the numbers in both READMEs: Artifactory OSS serves maven/gradle/ivy/sbt and generic only - no cargo, no pypi, no npm, in any free edition, 6.x or 7.x. Nexus 3.72.0 cannot serve Cargo at all (it arrived in 3.73.0), so the 3.96.2 arm is the only route to a cargo-versus-cargo comparison.
⚠ Also: Nexus 3.96.2 is Community Edition under a EULA the harness accepts programmatically, with a 40 000/day cap. The 3.72.0 pin is EPL OSS and is the legally cleanest to quote. Someone who is not a benchmark should decide what may be published about a EULA'd product.
⚠ And the near miss worth remembering: 3.96.2 ships anonymous read DISABLED, unlike 3.72.0. The first run on the current tag served 0 of 91 757 GETs, and only an empty-rival-cell guard stopped that being published as ~0 MB/s - that is, as a fabricated holger win.
GPU - CORRECTED 2026-09-20: the premise was WRONG. The GPU path works.
I claimed dwarves-server was running inference on the CPU while an RTX 4090 sat idle, and I stopped the service over it. That was wrong, and it was drawn from a single nvidia-smi sample showing an empty compute-apps list.
MEASURED by lane R8, as rickard AND as the service user dwarves in a stripped environment, identical both times: load_gpu_strict() SUCCEEDS - OK - the cuda execution provider REGISTERED (vendor=cuda, cuda_ready=true). A real CUDA context holding 386-400 MiB of VRAM was observed WHILE it ran. Speed: 3 526.4 chunk/s on GPU against 48.5 on CPU - 72.7x.
The compute-apps list was empty because nothing was being embedded at that moment: the journal for that hour shows 23 members in 4.7 s, every line 0 new vector(s) - N reused. No forward pass, so no context to see. The 3 h 11 min of CPU over 59 min of wall clock was the working-set warm, four preview tables of about 2 M rows cut at a 1 GiB ceiling, SCIP ingest and a 60-second git-fetch loop. A cold boot burns 6 min 28 s of CPU before serving anything.
My source reading was also STALE: select::load_backend has used load_gpu_strict() since commit 92fd339. The fail_silently entry I quoted is reachable only in a single-backend build, which dwarves-server is not.
But the silence was real, and worse than I described. The line reading with ort (ONNX Runtime, NVIDIA CUDA GPU) is a RE-PROBE OF THE HARDWARE reporting what the selector WOULD choose - a forecast printed in the past tense. It would have printed the same words if the EP had failed to register, and vector doctor had the same defect. That is why a yes-or-no question cost an hour.
FIXED on nornir branch lane-r8-gpu-provider: Embedder::backend() is answered by the LOADED object rather than a probe; OrtEmbedder carries ep_verified, true only on the strict path, so it says EP registration VERIFIED or REQUESTED best-effort - NOT VERIFIED; a backend that forgets to override gets a default that claims NOTHING. The embedder is resolved eagerly before the listener opens and one line is always printed. The owner's refuse-to-start ruling is implemented with ONE spelling (--cpu, env twin NORNIR_EMBED_CPU, one reader), and dwarves needs no change because it passes argv through.
⚠ /usr/local/bin/nornir on oden is 0.5.0 and predates $NORNIR_EMBED_BACKEND: it ignored every pin and reported CUDA each time. Anyone diagnosing with that binary gets an answer that cannot be falsified. Reinstall it.
⚠ origin/main of nornir DOES NOT COMPILE: BenchRun gained id and machine_facts in edda today and nornir's initializers were not updated. Three were fixed to unblock the lib; eight remain in the test build (release/gate.rs:974,1136,1403, warehouse/iceberg.rs:5796,6657,6784,6800, warehouse/mod.rs:319). The lane-r6-bench-identity branch owns that seam.
2. WHAT PROCEEDS WITHOUT ANY ANSWER
| # | work | done means |
The benchmark baseline — DONE, stinky/bench tip a5ef6e6, 770 lines, no unsourced number. Authoritative run on oden: 20/20 arms, EXIT=0, thermally traced (Tctl avg 56.0 max 64.1 °C, all-core 4167 MHz — no throttling). The published README claims reproduce within a few percent: they were never wrong, only unreproducible |
✅ | |
| W0b | Make the rivals current (?32): Artifactory to 7.161.26 once oden's /home is under the percentage guard, a second Nexus arm on 3.96.2 for cargo-vs-cargo, and Gitea as the one light container covering cargo+maven+pypi. Plus the stress arms nobody has: concurrent UPLOADS (every arm today is read-heavy), sustained mixed read/write on a many-thousand-artifact repo, and the edges — disk filling, a slow client, a client vanishing mid-upload |
a comparison quotable to a customer, and a write path measured under contention |
LANDED 2026-09-20 (dd0160c, bb82fdf, 5cfc471, 9f1aff4): cargo xtask rows-check builds the declared rows (ORIGINAL 6 of 6, 69.1 s for the feature axis) and cargo xtask gate phase 5 RUNS the declared rescue arms. Gate the bench driver's compile. ⚠ MEASURED: cargo build --release -p xtask --example nornir-bench FAILED at 542e296 with 10× E0063 — nornir-bench 0.1.8 added BenchResult.source and holger built the struct by literal at ten sites. Nobody had built it since, because an --example of a publish = false xtask is built by NOTHING in the normal flow, so the house's performance claims had quietly become unreproducible from HEAD |
one line in the gate, and the numbers stay reproducible | |
LANDED, VERIFIED 2026-09-20 on aa0d654. holger-bench-scenarios is a [dev-dependencies] edge in BOTH shipped binaries — server/cli/Cargo.toml:282 and agent/cli/Cargo.toml:70 — carrying the owner ruling in the manifest comment. MEASURED on this trunk: cargo tree -p holger --edges normal -i openssl-sys answers package ID specification openssl-sys did not match any packages — it is not in the graph at all, not merely unbuilt. The only remaining NORMAL edge is xtask/Cargo.toml:136, and xtask is publish = false and ships to nobody. The acceptance is now a TEST rather than a measurement: no_shipped_binary_normal_depends_on_a_bench_or_test_crate, green. — original text: The bench crate becomes dev-only (?8). MEASURED closure of holger-server: 431 as main ships → 421 after default-features = false on bench's reqwest (:76, the actual leaking line) → 414 after dropping online/auth-oidc/auth-oauth2 from the bench edge → 410 after the dev-dep move. ⚠ auth-passkey must STAY on that edge or served.rs:153 fails to compile. agent/cli:70 is ALREADY a dev-dependency |
dev-dependency everywhere a shipped binary declares it, holger-server links no openssl-sys, and the benches still RUN. Land only after W0 |
|
LANDED, VERIFIED 2026-09-20 on aa0d654. holger-recipe/src/honesty.rs (874 lines) is the pure-Rust, offline, toml-parsing engine the row asked for, behind the default gate feature so the wasm32 cdylib never links toml. It is CALLED — xtask/src/gate.rs:189,248,253,566 — and holger-recipe/tests/feature_honesty.rs is 4 of 4 green in 0.08 s, including the_engine_is_not_vacuous, which is the answer to the old guard that skipped silently. The owed half is now CLOSED too (188f2fe): FORBIDDEN_TLS_BACKENDS is a SEPARATE list, because it is a different question with a different scope — a client is forbidden only where egress is and full links one BY DESIGN, while a native TLS stack is forbidden EVERYWHERE. It is judged FIRST, before the tier split. And the second half of the same hole went with it: judge_artifact ended in a bare Verdict::ok(), so server-local — the row OpenSSL actually got into — was never judged at all; it now returns a verdict that SAYS what it judged, because "accepted" and "no opinion" must not render identically. Two tests, written against the row it happened on. — original text: The feature-honesty rule — written pure-Rust over toml-parsed manifests, because the EXISTING guard shells cargo and SKIPS SILENTLY when cargo is absent, so a cargo-less CI passes it vacuously. RULE 2 finds three more nornir-testmatrix optional-NORMAL edges — the fourth occurrence waiting to happen. And the EGRESS JUDGE is worse than reported: judge_artifact ends in a bare Verdict::ok(), so server-local, the row OpenSSL got into, was NEVER JUDGED, and FORBIDDEN_OUTBOUND_CLIENTS names no TLS backend at all |
two tests, both RED against the real history of all three past leaks | |
| W16 (UNBLOCKED; the dep move waits on one publish) | ⚠ ?34 IS CLOSED and this row was not the one-liner it said it was — for a THIRD reason nobody had written down. (1) The blocker named here is gone: lane R12 published znippy-compress 0.9.14 on 2026-09-20 (znippy master 3b60848), so the slotpool orphan is fixed and registry znippy is takeable. MEASURED by lane ZNIPPY-APPEND, which walked straight into it: holger in a lane worktree failed with 8x E0432/E0433 could not find slotpool in znippy_common until cargo update -p znippy-compress@0.9.13 --precise 0.9.14. (2) There are THREE path edges, not one: server/lib/Cargo.toml:199, plugin/host/Cargo.toml:45 and xtask/Cargo.toml:57 — a row that moves only server/lib collapses nothing, because the other two keep the second znippy-common node alive and with it its own znippy-zoomies -> rotaryengine. (3) The registry version to move to is NOT 0.9.15: HotArchive::append took &mut self there, which is the read-collapse mechanism stinky-12 5.4 measured, so the move must land on 0.9.16 (znippy lane/hotarchive-append 49e171b, dwarves crate-release dry-run plan 0.9.15 -> 0.9.16, Gate B green) or holger takes the defect from the registry instead of from a path. The holger half is written and pushed (lane/znippy-common-registry, 8d26cff + 4b54897); the dep move itself is HELD until 0.9.16 is on crates.io. |
all THREE edges at version = "0.9.16" with no path=, cargo tree -d collapsing znippy-common / znippy-zoomies / rotaryengine to one each, cargo test -p holger-server-lib --lib and cargo check --workspace --all-targets both green |
LANDED 2026-09-20 (03a2bb3): a promotion to a repository nobody serves is REFUSED BY NAME. — The promotion that lies — a promotion to a NON-EXISTENT repo SUCCEEDS and a test asserts it (promote/mod.rs:812-826) |
to_repo resolves to a served repository, or the promotion is REFUSED BY NAME |
|
LANDED, and my own earlier note in this file was WRONG — corrected 2026-09-20. It said AuditService.Query was "still owed". It is not: proto/holger.proto:701 declares service AuditService with both Query(AuditQueryRequest) AND SentTo(AuditSentToRequest) — the second is, in the proto's own words, "What this server SENT to each principal — the A->B->C question", i.e. exactly what ?1 needed. Implemented at server/lib/src/grpc/services.rs:2087 and :2155 over server/lib/src/audit/query.rs, with tests incl. sent_to_reports_what_was_sent_and_not_what_was_refused. The service is deliberately READ-ONLY — no RPC records, deletes, compacts or rotates, because a write reachable from the query plane would launder the blake3 chain. ⚠ The lesson is the one the archaeology lane already wrote down: verify a "this is missing" finding before repeating it |
ADMIN only | |
LANDED 2026-09-20 (71a0837): the .wasm plugin directory is anchored the way the audit dir is. — Anchor ron_plugin_dir — unlike the audit dir it has NO declared home; it is passed to catalog.load_dir(dir) raw, so "does a module survive a reinstall" means "whatever the RON happens to say" |
anchored the way the audit dir is | |
LANDED 2026-09-20 (05d664e): pub trait Gate at server/lib/src/promote/gate.rs:172, and WasmGate joins a loaded meter guest to it (c9109c4). — A Gate trait over a PromotionContext carrying the SET with digest REQUIRED, modelled on gunnar-policy/src/lib.rs:173 plus GUN-007. Runs on PUT before the bytes, not only on promote |
env-var policy deleted, the whole set judged at once, an unknown gate name a boot error. NET-NEGATIVE surface | |
LANDED 2026-09-20 (05d664e): attest.rs has a door, a constructor and a caller; promotion writes evidence. — Wire attest.rs — 1 458 lines with no door, no constructor, no consumer, while promote/mod.rs:123 still calls modgunn with blob_sha256: "" |
promotion writes evidence, one VSA is emitted, the digest seam has a caller |
⚠ AND NOW MEASURED, which raises W12's price: promote_ledger did 606 seals/s on loki's 12 cores and 349 on oden's 32 — twenty more cores made sealing 42 % SLOWER, while the same hardware took pip_wheel_mt32 from 1918 to 4224 MB/s (+120 %). secure_update_prepare has the same shape: 10.84 → 7.69 MB/s while pinning 32 of 32 cores. The seal path is strictly serial and tracks single-core clock. A customer who buys a bigger box gets faster reads and SLOWER WRITES. That is "no job queue, everything inline on the RPC", measured rather than argued.
The archaeology answered W12's precondition, 2026-09-20
holger's history is FOUR MONTHS, not a year: first commit d2f2706, 2026-05-19, 672 commits, no earlier lineage. And the read machinery is not slower - it is faster per core than its own record: get_file did 417 602 ops/s single-thread on a 32-core box in May 2026 and does 571 173 on loki's 12 cores today (1.37x); the serve route costs only 19 % over raw get_file. So nothing was lost in the trait and impl the owner built. Verdicts: a streaming fetch contract NEVER EXISTED (17e161b was already Vec<u8>); the rustls + h2 multiplexing + session resumption design is INTACT and still in the code; the OIDC/JWKS client reuse was DONE and improved in edda. What WAS lost: the Vec->Bytes migration doc, written 81d8e24 and deleted UNEXECUTED by be80f18 in a "prune dated notes" sweep; the "no OpenSSL" promise, lost to six reqwest lines missing default-features = false - one of them in a repository backend, not a bench crate; and the request-path bench, dropped in cd7636d, so since 2026-05-31 nothing has benchmarked the serving path at all. W12 is therefore RESTORE work, not a rewrite, and it is now in progress.
⚠ W12, the data plane, is deliberately NOT here. Whole-Vec<u8> at every hop, sha256 on the request thread, no job queue, HotArchive unused, gatling linked twice and called zero times — but the owner built the original machinery himself and remembers it as terribly fast, with TLS kept up and cached. If that path still exists unwired, restoring beats rewriting, and rewriting first destroys the evidence. W12 waits for the archaeology lane, not for a ruling.
3. BLOCKED, AND BY WHAT
| # | work | waits for |
LANDED 2026-09-21 (agent/w14-floor; §4 "W14 — the anti-rollback floor"): enum Floor { InForce(..), Absent } in plugin/host/src/floor.rs, kept at <data_root>/plugins/floor, a rollback refused by name as holger-plugin-10, five integration tests as a declared arm. — original text: Anti-rollback floor — ⚠ PROMOTED: while injection meant re-imaging this was a blemish; with an injection channel it is a PREREQUISITE. Fix the SILENCE, not the location: enum Floor { InForce, Absent }. The plugin gets its own lane FREE, since the floor is already keyed per (supplier, product) — correct, not merely convenient, because a shared sequence would make a routine plugin update read as a firmware rollback |
?26, ?27 | |
| — | ||
| W7 | Plugin ABI v2 — negotiated AbiRange echoed in every call; a TLV manifest whose high bit means REQUIRED; Capability granted per module from a CONFIG FILE and refused BY NAME at LOAD through two gates, after which the linker defines ONLY granted imports, so an ungranted capability is not merely refused but not linkable; ReadingValue {Counted, Refused{reason}} so an uncounted zero cannot be spelled; WireAuditRecord carrying no time, identity or IP because the host stamps all three; WirePrincipal.ident IS AuditEvent::ident. Every v1 bound intact: fuel 1e9, 256 MiB, 30 s epoch, no WASI, CMS over blake3 before compilation |
?16, ?17, ?30 |
| W7b | Prove the open appliance is clean — three censuses against a BOOTED holger: every loaded module named and the set equal to linked ∪ import-receipted (no anonymous module); the effective grant table holds none of principal.read/digest.read/clock.read/audit.write; and the host functions defined on this boot's Linker equal the granted set, so a module wanting host_audit_write fails INSTANTIATION. A property of THIS BOOT, not this build — arguably stronger than gunnar's symbol-table proof. Needs ServerProfile to report the census |
W7 |
| W8 | The two guests. OPEN holger-plugin-allocation: requires only usage.read + store.list and says so in its manifest; answers capacity() and a selftest() that writes a probe and measures the delta; returns Allow unconditionally with a test asserting the function is total; no cap, no expiry, no money word. CLOSED vetra/monetize-plugin: private, never mirrored, never named in holger's tree or lock; carries monetize-embed compiled to wasm32 and verifies Ed25519 over monetize's canonical form so holger writes no second one; ships as .wasm + .p7s. It talks to NEITHER Stripe NOR UpCloud — those stay behind monetize's factories, in monetize-server, on another box |
W7, ?3, ?15, ?28 |
| W8c | Load the plugin from the artifact store — the guest is published into a repository like any other artifact and the RON config names it BY DIGEST; no import path, no new format, no new store. This replaces everything the cancelled W19 was going to carry | W7 |
| W8b | ⚠ FIRST CODE TASK, UNPROVEN: monetize-embed has never been compiled for wasm32-unknown-unknown. It SHOULD work — ed25519-dalek verify needs no RNG — but if it cannot, the closed guest must carry its own verifier, which breaks "holger writes no second canonical form" and is an ESCALATION, not a quiet workaround |
— (ready now) |
LANDED 2026-09-21 (agent/w9-twin; §4 "the twin gets a role — 2026-09-21"): role: Primary | Twin(primary: …) in the RON config (server/lib/src/twin.rs), an absent key is always Primary, a Twin refuses every write and admin RPC by name (holger-twin-1) on both doors, binds only loopback, and PULLS. — original text: The twin gets a role. Same binary is right, but nothing makes it refuse writes, loopback its doors or follow: installed today it is a second empty primary |
?6 | |
LANDED 2026-09-21 (same lane): replication::replicate_estate BESIDE replicate_pairs (the published holger-traits surface untouched) moves the attestation store, audit segments, properties, SBOMs, the sealed PROD store, update-bundle state, import receipts, the plugin floor and the served config — content-id idempotent, gated, a_replicated_twin_holds_the_evidence_not_just_the_bytes. — original text: Replication that can actually make a twin — replicate_pairs moves artifacts BY COORDINATE only: no evidence store, no audit segments, no properties, no config, no bundle store. Promoted, such a twin serves artifacts with NO provenance — gunnar's un-metered-twin scar in a new coat |
W9 | |
| W11 | holger-remote — 0 grep hits; a downstream holger cannot reach an upstream one at all |
?1, ?7 |
LANDED 2026-09-21 (agent/w15-w13; §4 "W13 — trusted time"): TrustedTime in update-bundle/src/time.rs with HostRtc (named host-rtc, is_trusted() == false) and SignedTime (the front's signed nonce+time, gunnar-clock's shape); verify_member_map_with_time, verify_bundle_file_at, extract_bundle_file_at added BESIDE the published signatures, which delegate to the named RTC; every verdict records its time source; no skew window grew, pinned by a census. — original text: Trusted time — every expiry check trusts the RTC; gunnar-clock has zero gunnar deps and is publish = false only by inheritance, and holger's verify API ALREADY takes now as a parameter |
?5, ?12 | |
LANDED 2026-09-21 (agent/w15-w13 in gunnar-ui + edda; §4 "W15 — the duplicates collapsed"): the WebAuthn pair IS collapsed — holger-ui went first on 2026-09-20 (1e4b98c), gunnar-ui followed, both on nornir-auth-webauthn 0.1.2; 223 lines deleted from gunnar-ui. garmr did NOT move and the reason is measured (p256 0.14 against the crate's 0.13 — that is the "different majors" the row means, and it is garmr vs everyone, not holger-ui vs gunnar-ui). The ed25519 "five call sites" turned out not to be duplicates at all: five different signed messages, one shared policy, closed by a census instead. — original text: Collapse the duplicates — FIVE WebAuthn verifiers (gunnar-ui's and holger-ui's are BYTE-IDENTICAL bodies on different p256 majors), five independent ed25519 sign/verify call sites, and an uncounted sixth crypto copy in garmr/vendor/, which vendors whole edda, facett, skade, znippy, znippy-zoomies incl. a second sign.rs |
?12 | |
LANDED 2026-09-21 (agent/w17-signature; §4 "W17 — a signature that means something"): a declared attestation trust root that cannot be read is now holger-attest-10 and takes the WHOLE declaration down; ProvenanceGate is registered on EVERY row with a declared promotion.provenance: Require | Warn | Off (default Warn, defended in §4), refusing by holger-promote-10 and, with no root declared at all, by holger-attest-11; ?11 MEASURED and still open with its price stated. The third clause is NOT done and was never holger's: dwarves still signs nothing. — original text: Make a signature mean something — the seal chains to a discarded CA, neither provenance gate runs in a shipped holger, and dwarves, the path that actually publishes, signs nothing |
?9, ?10, ?11 |
The injection path, once W19 and W7 exist — eight steps, none of them a re-image
Author the bundle (two approvers plus a distinct release signer) → deliver by ImportBundle, the same verb online and airgapped → chain 1: size cap → control members only → release signature → payloads sized against the now-signed manifest → decompress → sha256+blake3 per member with unreferenced AND missing refused → approvals by CERT FINGERPRINT → two-person plus distinct signer → anti-rollback on the plugin's own (supplier, product) lane → materialize by staging+rename with a receipt and its own audit action → chain 2: blake3 plus detached CMS against the operator's roots BEFORE Module::new, refused by name → grant (installing grants nothing; ungranted is not linkable) → activate (ArcSwap for a meter, restart for a handler) → verify from outside through ServerProfile's census.
⚠ Three independent brakes — the bundle chain, the module's CMS chain, the capability grant — and the middle one has a hole: it is not rollback-aware, so a correctly signed OLD module passes forever. The floor is the only downgrade stop there is, which is why W14 is a prerequisite and not a nicety.
?46 — how does a znippy-backed repository answer "did this change?"
MEASURED 2026-09-20, and the surprise is that the digest is already there and is the wrong SHAPE, not missing.
The raw half of the unchanged gap is closed (774ea3b). This is the other half, and it is the bigger one: 15 of holger's 17 repository backends are znippy-*, and every one of them lists with ..traits::ArtifactEntry::UNKNOWN_PROVENANCE under a comment that says "a znippy file table publishes no mtime and no per-entry digest".
Half of that comment is now false. znippy-common/src/sign.rs:108:
/// Per-artifact digest: a merkle-fold of one file's chunk BLAKE3 hashes, ordered
/// by `chunk_seq` (Law 3). Reuses the existing per-chunk hashes — content bytes
/// are never re-read.
pub fn file_digest(file: &FileMeta) -> [u8; 32]
A per-file digest exists, it is free — folded from the chunk checksums the index already stores (index.rs:557), never re-reading a byte, never extracting an artifact — and a sibling, file_digest_from_parts, is driven straight off the on-disk index so it needs no FileMeta at all. The listing path could have it for the cost of a fold.
And it cannot go in ArtifactEntry.sha256, for two independent reasons:
- it is BLAKE3, and the field's own contract is "Lowercase-hex SHA-256 of the artifact bytes";
- it binds the relative path — deliberately, so two files with identical bytes get distinct provenance digests. It is not a content digest at all.
Writing it there would be the exact lie the field's doc comment forbids, and mirror.rs would then compare it against a Nexus checksum.sha256 and call two identical artifacts changed forever.
⚠ The other half of the old comment is simply TRUE and cannot be fixed here: FileMeta is {relative_path, compressed, uncompressed_size, chunks} — there is no mtime, so published_unix_ms has nowhere to come from either.
So the question. What a mirror actually needs is not a content digest but a CHANGE TOKEN: an opaque value that is stable while the bytes are stable and differs when they differ, compared only token-against-token. file_digest is already exactly that, and for the ?1 case — C pulling from B, both of them holger, both znippy — the two sides compute it the same way.
The options, with what each costs:
| what it is | cost | what it gives up | |
(a) an additive change_token on ArtifactEntry + RemoteAsset, compared only against another change_token |
honest, and it is what the mirror wants | proto field, traits, one arm in ChangeMap::diff, 15 backends filling it |
a third comparator to keep honest; a token from a DIFFERENT producer must never be compared |
| (b) make the znippy backends hash the bytes into a real sha256 | no new concept | extracts and SHA-256s every artifact — O(bytes of the repository) on a listing | the thing raw's listing was explicitly built not to do |
| (c) do nothing | free | every znippy repository stays unproven forever and, under the default UnprovenPolicy::Refetch, re-fetches everything every tick |
the feature |
⚠ Note that (c) is not silent breakage — mirror.rs is already honest about it: unproven is a bucket of its own and is never folded into unchanged. The cost is bandwidth and time, not correctness. That is why this is a question and not a defect.
Not decided here. Option (a) adds a concept to the wire and that is the owner's call.
ANSWERED 2026-09-20, owner: option (a). The answer, what it cost, which nine backends answer the token and why the other six cannot, is in §4 under
### ?46 — ANSWERED. Nothing above this line has been edited.
4. ANSWERED — kept, never deleted
W12 baseline 2026-09-21 — the request path, measured, then restored to stream
Lane agent/w12-dataplane, t14s (16 cores, nvme ext4), release bench cargo run --release -p holger-bench-scenarios --features dataplane --example dataplane_bench (dev-only, plan ?8; knobs in the example's header). The BEFORE column is main 6f8b595's server code driven by the same bench; the AFTER column is the lane. The box was SHARED with two other lanes' builds all night: the throughput columns move with the load average (34 at the second AFTER run, 4 KiB GET fell from 10 651 to 24 ops/s with NO change to that path) — so the resident-set columns are the load-independent finding, and the throughput columns are quoted from the quietest run of each side with the load noted.
| arm | BEFORE (main 6f8b595) | AFTER (lane) | what changed |
| HTTP PUT 1 GiB (raw) | 102 MB/s, server RSS +1025.6 MiB (1.002/byte) | 310 MB/s, RSS +9.6 MiB (0.009/byte) | body streamed into the backend, sha256 gate on the blocking thread |
| HTTP GET 1 GiB (raw) | 285 MB/s, RSS +1025.3 MiB (1.001/byte) | 263 MB/s (load 34), RSS +15.5 MiB (0.015/byte) | reply streamed off fetch_chunks, Content-Length from the store |
| HTTP PUT 4 KiB ×5000 (raw) | 118 ops/s, p999 36.1 ms | 311 ops/s, p999 33.2 ms | same durability chain; less work per request |
| HTTP GET 4 KiB ×5000 (raw) | 6 968 ops/s, p999 2.28 ms | 10 651 ops/s, p999 0.54 ms (load noise: path unchanged in that run) | — |
| gRPC StreamArtifact 32 MiB ×8 (znippy) | 344 MB/s, p999 103 ms, 1.253 RSS/byte | 854 MB/s, p999 42.5 ms, 1.000 RSS/byte | pieces off the backend; znippy still extracts whole, so 1.0 is its floor until znippy-common streams |
| OCI chunked push 256 MiB (64 PATCH) | 182 MB/s, finalize 517 ms, RSS +256 MiB in finalize | 365 MB/s, finalize 1 ms, RSS +0.0 MiB | sha256 per PATCH, finalize = rename |
| blake3 256 MiB, the seal's content address | serial 3 744 MB/s (one core) | 25 173 MB/s on gatling (6.72×) | server_lib::digest, hazmat subtree merge |
sha256 was NOT forked: it is sequential by construction and cargo/OCI/pip name it on the wire; it now streams (per frame at the door, per PATCH in OCI, per buffer in raw) so it costs no second traversal, and it runs on the blocking thread, not the reactor.
What the trait grew (ALL additive default methods on the published RepositoryBackendTrait; no existing signature changed): fetch_chunks, put_from_reader, streams_http2_uploads, handle_http2_upload, streams_http2_reads, plus the free ChunkIter { len, pieces } / chunks_of / READER_PREALLOC_CAP. raw and file-oci override; every znippy backend keeps the defaults and behaves exactly as before.
Guards: a_one_gib_put_never_holds_the_whole_body (its own process, server/lib/tests/, judged on VmHWM < 256 MiB), a_stream_fetch_yields_before_the_last_byte_is_read (grpc::services::stream_tests), the_index_is_parsed_once_per_write_not_per_read (znippy-rust, counts archive opens), blake3_on_gatling_matches_the_reference_at_every_tree_shape (sixteen lengths). Stinky-01 §2.5 was already closed on main by the HotHandle work; the guard now proves it.
Still owed, not this lane: the job queue (§6); znippy extraction as pieces (the 1.0 RSS/byte on the gRPC znippy stream); the write budget on both doors.
?46 — ANSWERED 2026-09-20, owner: option (a) — an additive change_token
The question and its three priced options are in §3 and stay there, unedited. The answer is (a): an additive change_token on ArtifactEntry and RemoteAsset, compared only against another change_token. (b) — hashing the bytes into a real sha256 — was refused for the reason the raw listing was built to avoid: it is O(bytes of the repository) on every listing. (c) was refused because it is the feature.
What landed, and the one thing the options table did not say. A token has to be source-scoped, and "scoped" cannot be a convention somebody remembers — it has to be checked, because the failure it prevents is silent and permanent. Two holgers holding byte-identical artifacts publish DIFFERENT tokens as a matter of course: the fold binds the archive-relative path (exactly as znippy's own provenance digest does, and for the same reason), and two servers lay their archives out their own way. Compare across sources and every coordinate reads changed for ever — the very defect this lane exists to remove, with the sign flipped. So the scope is part of the VALUE: "<scope>|<opaque>", stamped by holger_listing with the endpoint and repository it asked, and traits::change_tokens_comparable REFUSES a comparison whose two scopes differ. A refused comparison is unproven — never changed, never unchanged.
The fold is holger's own, not znippy_common::sign::file_digest. Same inputs — the per-chunk BLAKE3s the lookup sub-index already carries, in chunk_seq order, bound to the relative path — but sign is an optional znippy feature dragging the whole CMS stack, and a source-scoped token is never compared against a value computed by other code, so agreeing with another implementation buys nothing. No znippy-common change was needed and none was made: znippy_common::ArchiveReader (public and re-exported since 0.9.14) already opens the manifest + lookup sub-index + trie once and answers locate(path) -> Vec<ChunkLoc> — with checksum: [u8; 32] per chunk — from memory. lane/hotarchive-append was therefore not disturbed.
Which backends answer it, and which cannot. Nine do: znippy-composer, -docker, -go, -helm, -nuget, -package, -deb, -rpm, -rust — every one whose list() has the row's archive-relative path in hand, which is what the lookup sub-index is keyed by. Six cannot, and the reason is measured rather than assumed: znippy-maven, -npm, -gem, -conda, -python and -build-thing list through a TYPED VIEW (as_maven(), as_npm(), …) or a manifest, and znippy_common::views hands out name/version/size/bytes on a package handle and no path accessor at all. They stay ListingProvenance::Neither and say so out loud. Giving them a token is a znippy question — one accessor on a view — not a holger one.
⚠ published_unix_ms is still None for all fifteen and always will be until FileMeta grows an mtime. Nothing here changed that, and the honest None is what keeps it visible.
?18 — ANSWERED 2026-09-20, then RE-ANSWERED the same day: do the wasm
First answer: a cargo feature plus a const factory, gunnar's proven shape, on the grounds that gunnar ships gunnar-installer-metered beside -open so money is decided at BUILD time, and that a feature's absence is provable in the SYMBOL TABLE while an uninstalled module's absence is only "a file that is not there".
Re-answered when the owner drew the line I had missed: "monetize kör väl på OS — du menar pluginen som kör i gunnar skulle vara en wasm. Där har du faktiskt rätt, gör wasmen." monetize was never the plugin. monetize-server is a PROCESS ON AN OS with Stripe and UpCloud behind its factories; the plugin is the half that runs INSIDE the appliance. That dissolves the objection: the wasm host is not carried FOR MONEY, because the OPEN allocation guest is itself wasm and the host stands on its own merits — format handlers and gates want it too. "No money in the open build" then means the paid .wasm is simply not installed.
⚠ What the reversal genuinely gives up is recovered by W7b: three censuses against a booted appliance, which is a property of THIS BOOT rather than this build.
CI — ANSWERED 2026-09-20: no forgejo, no YAML. The gate is the build-thing, triggered by xtask, in Rust
The problem is real and measured: .forgejo/workflows/security.yml is the only workflow in holger and runs cargo audit alone - its own comment says no cargo build, no cargo test, no cargo deny. Nothing builds or tests holger automatically, which is why 10x E0063 in the benchmark driver went unnoticed for weeks and why three feature leaks had to be found by hand on a deployed binary.
But the fix is NOT a bigger workflow. The owner: "FUCK HELL NO FOREGJ ... codeberg har förbjudit ai! använd BUILD thing triggat av xtask allt i rust." Codeberg, which hosts the nordisk repos, has banned AI, so their CI is not a path to build on. And it fits the law already written: dwarves is the build system and xtask is a thin door onto the build-thing.
So: cargo xtask resolves the declared rows through the recipe and builds them in the container; the three rules are plain Rust tests and run where holger's tests run. dwarves pipelines are allowed but little tested, so nothing depends on them without saying so.
⚠ And holger is an ARTIFACT REPOSITORY. It stores artifacts; it is not a CI system and must not grow one. This item is about holger building and testing ITSELF.
BENCH IDENTITY — DECIDED 2026-09-20: uuid v7 on the run, and the warehouse becomes rebuildable from git
Owner's design: the bencher gives each run a uuid v7, generated by the PRODUCER, so a double upload is REJECTED. Beside it go the machine facts — hostname, cpu, cores, memory, GPU.
Why it is needed, measured: BenchRun has NO id today. The uuid is made server-side by append_bench_run_async, so the on-disk receipt has no identity. That is why gunnar's bench-landed must join receipt to row ON A GUESS ("timestamp truncated to the second, plus the run's arm-name set"), and why the 16-of-46 receipts that never reached a warehouse could not be repaired afterwards.
v7 rather than v4 because it is time-ordered: the id sorts like the date and the rows cluster in time.
The consequence the owner named, and it is bigger than either decision alone: committed receipts PLUS an idempotent id make the warehouse REBUILDABLE FROM GIT. Walk every repo's bench_history.jsonl, replay every line into an empty warehouse, and duplicates are refused — so the operation is correct however many times it is run, in any order, over any overlapping set of repos. Without the id such a sweep would create duplicates and be safe exactly once.
Three properties follow:
- git IS the bench archive — replicated on every machine that cloned, versioned, and readable with no warehouse, no catalog and no server;
- a new machine is filled by cloning and replaying, not by copying a warehouse directory;
- history believed lost is recoverable, because the 16 receipts that never reached a row are still lines in a file.
⚠ So the upload path must be built to run over a WHOLE REPO, not over one run.
BENCH HISTORY — ANSWERED 2026-09-20: it is COMMITTED to git, and the filesystem is the contract
Owner: "så bench skall alltid också lägga i git", and then: "dock bok och bencher samarbetar och läser filer från filsystemet, it's ok for now."
So bench_history.jsonl is TRACKED, not gitignored. It is the backup for when dwarves with nornir/mimir is not available, and the doc mashup reads it from the filesystem. A measurement that exists only in a warehouse nobody can open is a page that prints NOT MEASURED.
The house contradicted itself and is now settled the dwarves way: dwarves tracks bench_history.jsonl and bench_history_mimir_thin.jsonl; gunnar gitignores it (.gitignore:23) with the reason written in benchland.rs — "committing it would make every measurement a merge conflict". That objection is real and is handled rather than ignored: the file is append-only JSONL, so it gets bench_history.jsonl merge=union in .gitattributes and two runs on two branches both survive a merge.
⚠ THE INCIDENT BEHIND THIS is in the law file itself, lines 3625-3660: cargo xtask mashup printed NOT MEASURED over 46 measured runs, from SIX defects. Three matter here:
- TWO WAREHOUSES, ONE REPO.
nornir.tomlwith[storage] local_pathis found by walking up from the checkout;release/docs-nornir.tomlis loaded explicitly three dirs up. Both have the full 41-table schema, so NOTHING ERRORS — a reader simply opens a warehouse nobody wrote to. The law states the general lesson for every repo: "Two configs in one repo = two warehouses, silently. Check yours." holger's has not been checked. - The reader stopped on the newest run rather than the newest READABLE one and reported absence. Skipped runs must be NAMED, not dropped.
- The forge writes the receipt and skips the warehouse, exit 0 — MEASURED, 16 of 46 receipts (35 %) had no warehouse row. Recorded as NOT FIXED: the forge cannot hold nornir's single-writer catalog lock, and inverting that is the owner's architectural call. This is why a guard must check OUTPUT, never an exit code.
?12 — ANSWERED 2026-09-20: YES. Publish all four.
nornir-auth-webauthn 0.1.2 (absent from crates.io, and the only thing blocking the deletion of four hand-rolled WebAuthn verifiers) · modgunn 0.0.6 · facett (the last remaining [patch.crates-io] in holger, which alone holds 19 holger crates on a fork) · and MOVING gunnar-clock to edda as nordisk-clock. Publish through dwarves crate-release — the PRIMARY verb — with --features rehearse compiled in so Gate B runs. Gate B earned its keep today: it caught holger-errcode 0.1.1 being four rows short of its source before upload.
?15 — ANSWERED 2026-09-20: vetra/monetize-impl, not a separate repo
Owner: the open-source variant is at gunnar.rs/nordisk/monetize and the paid plugins live in vetra/monetize-impl. So the closed plugin is a crate inside the existing private repo; no new repository is created. ⚠ MEASURED TODAY: git.gunnar.rs:2222 answers connection refused. Four repos could not be fetched for that reason, monetize among them. The open half is unreachable until the forge is up, which blocks any work that needs to read or push it.
?8 — ANSWERED 2026-09-20: the bench crate is a DEV DEPENDENCY, never in the final binary
"den skall vara dev-beroende, dvs aldrig in i slutbinären!!" Stronger than the feature fix it replaces, because it closes the CLASS rather than the instance: a normal dependency can unify into a shipped graph, a dev-dependency cannot. Consequence to be handled, not dropped: the bench feature exists to put scenarios INTO the served binary, so that capability moves to a bench target or a dev-only binary, named rather than silently deleted (?20).
?14 — ANSWERED 2026-09-20: it IS a plugin, and gunnar has THREE seams
Money is a cargo feature; monetize's open/closed split is a const factory with a fn pointer; the wasm guest is build-thing, whose host is edda's, with NO gunnar dependency on the host side. One of the three reasons for the earlier wrong verdict was a STALE COMMENT: server/lib/Cargo.toml:458-461 still warns about wasmtime 27 while plugin/host requires 48 and deny.toml says the 27 acceptance is gone. Fix that comment ON PURPOSE — it is the line that produced the wrong answer.
MVP — ANSWERED 2026-09-20: size is not the test
"MVP behöver inte vara superslim storleksmässigt, men t.ex. inga egui-grejer — funktionell, men inte science fiction." The tests are: works end to end, no GUI baggage, and the LAST MVP feature is holger in the cloud with repo A→B plus holger on-prem with repo C. Everything else is holger Pro. Under it, 15 repository backends moved Pro → MVP at zero cost (all 17 are already non-optional), and six rows moved out on the "does not run" test.
?1 — ANSWERED 2026-09-20: C PULLS. The carried bundle is withdrawn.
The ladder A→B was declared and proven on holger.rs, and then C, an on-prem downstream, was stood up on oden and proven PULLING from holger.rs (d20c449, merge cf2ee26). What settles the question is not that the pull works but that the objection against it was answered rather than ignored:
- "does upstream know what C has?" — it does, and by the cheapest possible route: it SERVED the fetch.
AuditEvent.identalready grouped it by principal, andaa8c42cadds the tally per client ADDRESS, so a downstream with one identity and many hosts is still legible. No receipt-return design is needed and none was built. - "zero egress" — the pull is C's egress, outbound, to exactly one declared upstream named in C's own
mirror:RON block. That is a protected-area shape, not a true airgap, and the owner's own ruling (AI-land, THE RULING C-KNOWS) says so. ⚠ This changes what MVP's "zero-egress" is allowed to claim, and that sentence has NOT been rewritten anywhere it is printed. - "a mirror is not a proxy" — drawn, not asserted:
/-/flowrenders the whole RON in skade's style with mirror≠proxy edges (98db1e9,e7ccaf1,57d9e1a).
What the answer COSTS, and all of it was paid the same day: a change map so a continuous mirror fetches only what changed (7743d63), a place for the upstream password to live with a literal one refused (4463f21), a timer that survives a failed tick with backoff and health metrics (cacf458, 0e22835), and mTLS so the mirror can present a client certificate per repository or across the board (30a8468, 125803f, 241d3eb, merge afeec1a).
⚠ What it does NOT buy, and this is the one live defect in the answer: unchanged is unreachable against holger.rs's OWN repositories. ArtifactEntry gained published_unix_ms and sha256 (b986449) and the mirror reads them (b537542), but raw publishes a time and no digest, and the znippy-* backends publish neither — the field exists and the backends cannot fill it. A customer's mirror therefore re-fetches everything, every tick. The fix is a digest in the raw backend (one read per blob, memoized the way file-rust does it) or in znippy's file table. It is a real change and it is owed its own item; it is NOT part of this answer.
⚠ ?7 (does the holger-remote pull ship in MVP, or only full?) was downstream of this question and is therefore now UNBLOCKED — and still OPEN. The mechanism exists; which tier declares it is a tiering decision nobody has made.
?16 — ANSWERED 2026-09-20: ONE ROW, and BOTH features on it
agent/meter-rulings ed760dd (+ 9da031f).
The three MVP appliance rows — mvp-server, mvp-server-metered and the static mvp-server-musl the ISO ships — each name wasm-plugins + meter-guests. There is no second appliance row, and holger-appliance-iso stays the single row it already was.
Both or neither. holger links no meter guest of its own: the only guest in the tree is holger-plugin-allocation-wasm, a .wasm an operator installs. So meter-guests without wasm-plugins is the Meter kind's op table and wire types compiled in with nothing that can ever load a guest — inert vocabulary. The pair is named as a pair, and offers() says so where the feature universe is declared.
Why not a second row. Two appliance rows means two appliance ISOs differing in whether metering ENFORCES, which is the trap already on record as "the metered appliance's twin is un-metered". What separates the open appliance from the paid one is which .wasm is installed (?18, re-answered) plus metered's own door — never whether the host half exists.
The cost, accepted and written into the declaration. wasm-plugins pulls wasmtime; wasmtime compiles a module with CRANELIFT, a JIT, inside holger's process. Since tonight that compiler is in the REQUEST PATH — the meter gate runs on PUT (Holger::meter_gates, server/lib/src/promote/meter_gate.rs). That is why ?17 gives the Meter kind its own fuel ceiling.
MEASURED: cargo xtask rows-check 11/11, including the musl row (69.2 s) and the metered row. cargo xtask gate --arms GREEN, and the census guard open_appliance_loads_no_paid_module still passes 6/6 — the OPEN appliance's linker defines exactly host_store_list + host_usage_read. Cargo.lock untouched by this ruling. Every non-appliance row is unchanged and carries no wasmtime.
⚠ It cost one red the gate caught: holger-build::capability_of is held to the declaration, and two declared features with no capability line turned capability_lines_name_real_features red. Lines added in 9da031f; the wasm-plugins one says the part an operator must know — cranelift, in this process, on PUT.
?17 — ANSWERED 2026-09-20: MEASURE first, then cap WIDE. 100 000 000 fuel, deadline unchanged
agent/meter-rulings b5fadab.
The proposal on record was 10e6 fuel / 250 ms with a slow verdict REFUSING. The refusing half stands and was already implemented (Err → undecided → refuse). The NUMBER did not, because the stake moved: the meter gate now runs on PUT, so a ceiling that fires is not a safety bound on a background reading — it is an AVAILABILITY control on every push. An over-tight ceiling is a self-inflicted outage on the one path a repository exists for.
So the ceiling is 100x a measurement. MEASURED on oden against the REAL holger_plugin_allocation_wasm.wasm, asking wasmtime for consumed fuel, over three loads (fuel for ONE call, alloc + plugin_call + result_len in one store):
| op | empty store | 64 blobs / 16 rows | 1024 blobs / 128 rows |
plugin_abi |
6 586 | 6 586 | 6 586 |
plugin_manifest |
6 586 | 6 586 | 6 586 |
meters |
14 110 | 14 110 | 14 110 |
read |
6 151 | 23 852 | 164 574 |
capacity |
6 174 | 6 174 | 6 174 |
selftest |
11 518 | 58 398 | 771 687 |
evaluate |
2 636 | 2 636 | 2 636 |
accept_fact |
2 559 | 2 559 | 2 559 |
describe_fact |
2 703 | 2 703 | 2 703 |
evaluate — the op on the PUT path — is the CHEAPEST of the nine and flat under every load: the open guest's verdict is Allow unconditionally and decodes nothing. Only read and selftest scale, and with the SIZE OF THE ANSWER they decode (~753 fuel per store key for selftest).
CEILING: 100 000 000 = the peak rounded up to a whole million (1e6) x 100. 129x the real measured peak, a tenth of the package handler's 1e9, 38 000x above evaluate. It is a KindSpec field, not a global: PluginLimits::narrowed_to takes the TIGHTER of the kind's ceiling and the operator's config, so a deployment may narrow it and can never widen it. The rounding is deliberate slack, so a compiler bump moves a measurement rather than turning a gate red.
The DEADLINE stays 30 s — NOT 250 ms. The deadline is the backstop for a guest parked inside a host import, and a meter's imports (host_usage_read, host_store_list) walk holger's own disk. Cutting it to 250 ms would turn a slow disk into a refused upload — exactly the availability failure this ruling was careful to avoid on the fuel axis.
⚠ What the ceiling implies, said rather than discovered: at ~753 fuel per key, selftest reaches 100e6 at roughly 130 000 keys in <data_root>/meters. That directory is the meter plane's private store, the open guest holds store.list over it and cannot write a byte, and nothing in holger puts files there. Past six figures it is a NAMED fuel refusal, not a hang and not a wrong number.
WasmMeter now records fuel per call (last + peak, fuel_ceiling() beside them), which is what made the measurement possible at all — nothing in holger had ever asked wasmtime.
Tests: the_open_guests_real_calls_are_measured_and_sit_far_under_the_ceiling, the_kinds_ceiling_binds_and_a_bigger_config_cannot_raise_it, a_meter_guest_that_burns_past_the_ceiling_is_refused_by_name (a hand-written WAT v2 meter with a real TLV manifest — refused in 161 ms naming the fuel code, the export, and the METER ceiling rather than the handler's).
THE STORAGE METERS — ANSWERED 2026-09-20: holger gets a free-space source, statvfs over rustix
agent/meter-rulings 3c18cbb.
storage_total_bytes, storage_servable_bytes and storage_used_bytes answered Refused(NO_FILESYSTEM_SOURCE) because holger could not call statvfs(3). It can now, over the data root.
rustix, not libc. Both were already in Cargo.lock through other crates, so neither adds a name to the tree and the choice is what the call site looks like. rustix::fs::statvfs(path) -> io::Result<StatVfs> is safe Rust — no unsafe, no zeroed MaybeUninit, no errno fishing — and on Linux it goes through rustix's linux_raw backend, so it issues the syscall and pulls in no C library at all, which is what the pure-Rust law actually asks for. libc would have put an unsafe block and a zero-initialised struct in the middle of a path whose entire reason to exist is that a zero must never be mistaken for a measurement. default-features = false, features = ["fs", "std"]. Cargo.lock delta: one line (plus one dev-dep line), no version moved.
The three numbers: total = f_blocks * f_frsize; used = (f_blocks - f_bfree) * f_frsize, so the root reserve is not charged to anybody's data; servable = f_bavail * f_frsize — what an UNPRIVILEGED writer can still place, which is what holger is (f_bfree would promise the 5 % reserve holger cannot reach). So used + servable <= total, which is the inequality the open guest's selftest checks.
The refusal discipline survives, narrowed rather than dropped. No data root ⇒ refused by name (the constant now says "this holger was given no data root"). A root the kernel will not describe ⇒ refused with the path and the errno. A filesystem reporting f_frsize == 0 ⇒ refused rather than multiplied to zero. Never a zero standing in for a failure.
ONE filesystem per box: the door (HolgerGrpc::with_data_root) and the guest seam (HolgerMeterServices::with_data_root) are both handed Holger::data_root() by the serve path, so a guest and the door cannot report two different disks.
PROVED against a tmpfs of KNOWN size: /proc/mounts states size=104857600k for /tmp and statvfs answers total = 107374182400 exactly — the size coming from the kernel somewhere other than the syscall under test. Writing 8 MiB moves used up and servable down by at least that much while total does not move. The test does not skip: a box with no sized tmpfs fails it by name.
And the open guest stopped saying there was nothing to check: capacity() now answers Counted(3) over a real filesystem (total 107 374 182 400, servable 106 607 947 776, used 766 234 624 on oden), and selftest() runs its cross-check — summed keys against the host figure, three inequalities — instead of short-circuiting on a refusal.
Tests: statvfs_reports_the_size_the_kernel_states_for_a_tmpfs, a_path_that_cannot_be_statd_is_a_named_refusal_and_not_a_zero, no_data_root_at_all_is_its_own_named_refusal, a_wired_data_root_counts_all_three_and_says_which_filesystem, the_open_guest::capacity_is_counted_now_and_not_three_joined_refusals, the_open_guest::the_selftest_runs_its_cross_check_instead_of_saying_there_was_nothing_to_check, a_wired_data_root_makes_the_three_storage_meters_measured_through_the_guest_seam.
?3 — ANSWERED 2026-09-20 BY MEASUREMENT: feat/metering-doors IS main
Answered by the branch-prune agent on origin/main 4c4f4ee; recorded here by agent/meter-rulings.
feat/metering-doors is not a branch to choose — it is already main. Its tip f6ff7ec is an ancestor of origin/main and git cherry reports −0 +0. What landed with it: proto/holger-metered.proto, server/lib/src/{metered,metering,meter_services}.rs, promote/meter_gate.rs, the whole plugin/{abi,host} meter path, and the tests metering_doors.rs and the_meter_is_what_answers.rs.
monetize-doors is the rival and it is the one behind: one commit f0ce959 (2 355 lines) main never took, spelling the same two doors under other names — holger-entitlement.proto vs folded into holger-metered.proto; Set/Get vs Get/Set/List; UsageService.Read vs ReadUsage; entitlement.rs/usage.rs vs metered.rs/metering.rs/meter_services.rs. The drift is ONE-SIDED: main is a superset, so there is nothing to merge and nothing to choose between.
The only question left for the owner is whether anything in its open_build_speaks_no_money.rs (239 lines) is worth lifting before the branch goes. Both branches are left untouched. Detail: .nornir/branch-prune-2026-09-20.md.
change tokens: all znippy backends stamp — 2026-09-20, agent/change-tokens-six
?46's answer left six of the fifteen znippy backends stamping NO change_token: znippy-maven, znippy-npm, znippy-gem, znippy-conda, znippy-python list through a typed view and znippy-build-thing through a manifest, and none of them could name the archive row a coordinate came from — so there was no key to ask the lookup sub-index with, and a mirror of any of them re-fetched the whole repository every tick. znippy-common 0.9.16 ships relative_path() on every typed package handle; build-thing's row is the writer's own layout (attestations/{sha}.json). All six now stamp the SAME fold the other nine do (traits::ArchiveTokens, one index open per listing, no artifact byte read); ListingProvenance moves them to ChangeTokenOnly, and the guard that used to insist on at least one Neither now insists that every znippy-* backend answers a token — that is the regression it exists to catch. Neither stays as the honest answer for a backend the table has not met.
Tests, per backend, on two independent seals with one entry rewritten (every row has a token; unchanged rows fold to the same token across the seals; the rewritten one moves):
maven_entries_carry_a_change_token_that_moves_only_when_the_bytes_donpm_entries_carry_a_change_token_that_moves_only_when_the_bytes_dogem_entries_carry_a_change_token_that_moves_only_when_the_bytes_doconda_entries_carry_a_change_token_that_moves_only_when_the_bytes_dopython_entries_carry_a_change_token_that_moves_only_when_the_bytes_dobuild_thing_entries_carry_a_change_token_that_moves_only_when_the_bytes_do
The mirror proof (tests/mirror_change_token.rs) is not table-driven over formats, so the cycle is proven once more over the REAL maven backend behind the REAL maven plugin, maven only: mirror::tests::a_stamped_maven_upstream_settles_and_refetches_only_the_changed_entry — second cycle unchanged 3, to_fetch 0; a rewritten pom changed 1, unchanged 2, one more fetch. The provenance guard every_backend_declares_what_its_listing_can_prove still passes. Two dev-dep edges on holger-server-lib (znippy-plugin-maven, znippy-common), both already in the lock: one dependency line, no new package.
appliance medium 2026-09-20: built, KVM-proven, not attached
What was built, on t14s, through the pinned containers. mvp-server-musl in nordisk-cargo-toolchain@sha256:3d5115ba… (284 s): holger-mvp 58 631 984 B, static-pie, sha256 5f18d10e673b0de11218f4babba20307686f2ffd03532dadda94d27b5078f853, --version = holger v0.6.10 (1cdba57) = main. holger-front is ABSENT from the payload's graph: cargo tree -e normal -p holger --no-default-features --features mvp-server,appliance,auth-passkey,sign,wasm-plugins,meter-guests --target x86_64-unknown-linux-musl = 1 572 lines, 0 of them holger-front. Packed: holger-app.znippy 12 935 130 B, blake3 pin f0e4d33d5d33…. Mastered by cargo xtask installer-iso --kernel tunnr/.kernel-out/tunnr-kernel/bzImage (tunnr main 80c3233, PCI hot-plug merged; kernel sha256 2c279f50…) with --param site=holger-server --param holger_ron=@private-holger-ops/deploy/generated/server/holger.ron --param data_disks=vdb: holger-installer.iso 42 051 584 B, sha256 2cae143b6b5bb183bc090390bce66f5f18106e5936faabb710f04abb1b8dfebf. UNENROLLED (KORP_ROOT_SIGNING_KEY unset → korp.require_signed=0), the same as every medium before it. Recorded, with SHA256SUMS, at /home/rickard/holger-media/1cdba57/ (no earlier medium was recorded anywhere in private-holger-ops; the 2026-09-17 one sits in /home/rickard/compose-clean/holger/.iso-out/, 52 381 696 B, sha256 fc0bbf39…). Nothing was attached. No UpCloud call was made.
KVM proof, the UpCloud shape: SeaBIOS (-machine pc), -serial none, VGA, two virtio disks 40 G + 20 G sparse, korp.data.disks=vdb. Measured, in order:
- The medium must be attached TWICE — once as the firmware's cdrom (SeaBIOS boots the El-Torito legacy stage off it; kernel + initrd load) and once as a read-only virtio disk, because the tunnr kernel has NO ATA/SCSI (
CONFIG_VIRTIO_BLK=y, nothing else) andhwdetectskipssr*: with only the cdrom the installer saysno block device carries the ISO-9660 magic (CD001) (probed: vdb, vda). That is the rule gunnar'sreinstall-applianceverb already states (gunnar/xtask/src/main.rs:320). SeaBIOS cannot El-Torito-boot the virtio copy (Boot failed: Could not read from CDROM (code 0003)), so both are needed. - With both attached,
-serial none: the installer narrates ontty0, installs in ~16 s,INSTALL-OK, reboots; the disk carries GPT p1 ESP / p2 rootfs / p3 korp-data 37.9 GiB FAT;holger.ron(1 983 B) is on P3 verbatim. The declaredvdbwas never touched (the 20 G image stayed at 4 KiB): the declaration elected the boot disk and nothing else,/datais the BOOT disk's P3 (/dev/vda3 mounted at /data (vfat)), and/serialdatis a directory on it. estate.toml's "20 GB maxiops mounted at /var/lib/holger" is not what this installer lays. - Booted from disk,
holgeris PID 1 and with the EMITTEDserver/holger.ronthe appliance PANICS:gRPC server listening on 0.0.0.0:50051, thenError: HTTP/OCI gateway failed to bind 127.0.0.1:8443: Address not available (os error 99)→Kernel panic - not syncing: Attempted to kill init!. The RON binds the artifact door on loopback for Caddy (estate.toml:326,via = "caddy"); the korp shape has no Caddy, no init, and nothing bringsloup (grep of korp-installer/wg-appliance-core: the onlyIFF_UPwriter isaddr.rs, for the declared underlay). Screens:holger-media/kvm-proof/d-20.png. - Same medium, PROOF-ONLY seed (
ron_http_url: None,--param underlay=10.0.2.15/24@eth0because the korp shape has no DHCP client either): installed, rebooted, and a tonic client over slirp hostfwd gotAdminService.health=status=ok version=holger v0.6.10 (1cdba57)andListRepositories= bundles/crates/dev/migtest/prod withsupports_listingpresent. The PAYLOAD is main and it serves; the emitted CONFIG does not fit the shape. Not fixed here — it needs a ruling (a door RON for the appliance withoutvia = "caddy", or the shape growinglo+Caddy), not a lane's guess. - Only
holger.ronrides the medium.holger-server.envandCaddyfilehave nostaged_fileparam on the installer row (holger-recipe/src/packaging.rs:86-214) and no consumer on this shape: there is no/etc, no unit, no Caddy;HOLGER_DATA/HOLGER_CONFIGare replaced bykorp.args=start,--config,/data/holger.ron+korp.env.HOLGER_APPLIANCE=1. Steps 8–9 offront plandescribe holger-front (a package install with a shell), not this shape. And the RON's/var/lib/holger/store/*paths land on the initramfs tmpfs — RAM — because only/datais durable here (datahome.rs).
Token continuity — how the bytes reach /var/lib/holger/admin-token. They do not, on this shape. The chain: [server.bearer] token_file (estate.toml:718) → emitted RON StaticToken(subject: "console", token_file: Some("/var/lib/holger/admin-token")) → nornir-auth-bearer/src/lib.rs:61 → nornir_service::resolve_token (nornir-service/src/lib.rs:673-679: the FILE rung, read + trimmed; empty or absent → TokenSource::Nothing → the method is disabled fail-closed). Nothing mints the file (no writer in holger; provision.rs only verifies OIDC tokens). The medium refuses secret-shaped params by name (nornir-build-thing/src/installer.rs:1063), and the box has no shell. On holger-front — the box that EXISTS — the bytes were placed over SSH by hand (front.rs steps 8 and 16, "ON THE MACHINE"). On a korp appliance they can only SURVIVE, never arrive: korp.reinstall absent = keep (korp-installer/src/cmdline.rs:493-499), and keep leaves P3 byte for byte, does NOT re-mint the identity and does NOT re-write the staged holger.ron (install.rs:636-660: "staged declarations NOT written … korp.reinstall=wipe re-stages them"). So the owner's choice is exactly two:
- keep (the default, an upgrade): payload replaced,
/data(store, identity, a token if one were ever there) survives, the newholger.ronon the medium is IGNORED. No re-mint, so the ⚠ brick does not apply. This is the only choice that can carry a credential across. - wipe: P3 re-laid, identity re-minted, staged
holger.ronapplied, and the token GONE with no way to put it back — every console write RPC refused, silently. Choose it only with a placement path that does not exist yet.
The re-image sequence, for the owner (UpCloud API, terraform show). ⚠ FIRST FINDING: terraform/terraform.tfstate holds ONE server — upcloud_server.front 00bdf2b3-267f-42cc-ae7c-bdf26d198e74 (holger-front, boot_order = disk, has a shell) — and no upcloud_server.server, no server_uuid output. repo.holger.rs is holger-front (the [colocation] block, estate.toml:270-284). There is no appliance to re-image today; the medium is for the day terraform apply creates server + its data volume. When that day comes:
terraform output server_uuid— the appliance's uuid, from the apply, never guessed (outputs.tf:49). Confirmboot_order = cdrom,diskon it.- Upload the ISO twice:
POST /1.3/storage(1 GB, maxiops, titleholger-installer-1cdba57-cd/-virtio), thenPOST /1.3/storage/<uuid>/importwith{"source":"direct_upload"}and the ISO bytes, for each. Verifysha256 = 2cae143b…on the import result. - Attach both to the STOPPED or started server:
POST /1.3/server/<uuid>/storage/attach{"storage_device":{"type":"cdrom","storage":<cd>}}and{"storage_device":{"type":"disk","address":"virtio","storage":<virtio-copy>}}. The virtio copy MUST come after the data volume so it isvdc, notvdb(korp.data.disks=vdbis baked into the medium). POST /1.3/server/<uuid>/restart{"restart_server":{"stop_type":"soft"}}—cdrom,diskboots the medium; the install takes seconds and reboots itself. Watch on VNC only (toggleremote_accessno→yes; the port dies on stop/start). With the medium still attached it re-installs on EVERY boot (keep = a harmless re-upgrade, wipe = a fresh store every time), so do step 5 promptly.- On the STARTED server:
POST /1.3/server/<uuid>/cdrom/eject, then detach the virtio copy (POST …/storage/detach {"storage_device":{"address":"virtio:2"}}; a detach that is refused while running needs a stop/start). Boot falls through to disk. - Prove:
AdminService.healthongrpc.holger.rs:50051saysholger v0.6.10 (1cdba57);RepositoryInfocarriessupports_listing. Every API call answers 200 while the box is a brick — the RPC is the proof. - Rollback = the previous medium (
fc0bbf39…, 2026-09-17) through steps 2–5.
DO NOT attach this medium as it stands: finding 3 means the emitted RON panics the appliance at first boot, and finding 5 means the credential cannot be placed. Both are rulings, not lane work.
holger-front runs main 07a238e since 2026-09-21 00:05 UTC
Variant full (front ON, glibc 2.39), built in the pinned toolchain container on t14s (sha256 040c71aee35fd4c84c09b6a26461af02fa0bb20636ff35302dcd113eaabde51b, 42 459 488 B), swapped by the owner with mv -T; backup /usr/local/bin/holger-server.bak-2026-09-21-75122dc. Verified live: holger-server --version = 07a238e, /-/releases 5 rows, /-/front answers, https://holger.rs/login 200, holger-ops releases 5 rows, console credentialled=true. NOT verified on the wire: RepositoryInfo.supports_listing and AdminService.Capacity — the console's break-glass bearer is RETIRED (an admin passkey exists, by design: auth/middleware.rs:180), grpcurl is not on t14s, and the box has no gRPC client; the proof is the 689 lib + integration tests on this exact main. A browser passkey session on /repositories reads the fields; there is no group repository on holger.rs today, so the visible difference is the empty-listing note only.
There is no appliance. terraform.tfstate holds one server, upcloud_server.front; [colocation] puts repo.holger.rs on it. The installer medium above is for the box that terraform apply has not created; its two rulings stand.
stinky 1.4 — MVP slimmed 2026-09-21
agent/slim-mvp. The 17 repository backends were unconditional path deps of holger-server-lib and pub mod marketplace (~5 000 lines: Azure Marketplace fulfilment, Entra sign-in, lifecycle webhooks, tenant ledger) had no gate, so holger-mvp — the airgap appliance — linked maven, pip, npm, nuget, go, gem, deb, rpm, helm, docker, conda, composer, the znippy package reader and a cloud fulfilment door for an estate that serves raw and rust. Now every format is a fmt-<format> feature (one per backend crate; rust = file + znippy, docker = znippy + the OCI file registry, fmt-raw is the in-crate marker), fmt-all is the umbrella, and marketplace is its own feature.
Who names what (holger-recipe/src/variants.rs, emitted to .nornir/build-variants.toml):
| feature | rows naming it | why |
fmt-all |
embedded, mini, full, server-local (via server/cli default) |
ORIGINAL keeps every format; crate set byte-identical to before (MEASURED below) |
fmt-raw, fmt-rust |
mvp-server, mvp-server-metered, mvp-server-musl |
the MVP delivery serves raw + rust: holger.rs's bundles/crates/dev/migtest/prod are raw + rust; no MVP test forces a third format (the only per-format tests are the ORIGINAL-side census, the maven mirror settle proof, the OCI TLS e2e, the build-history wiring, and the nuget/composer real-client arm — each now gated on its own fmt-*, all in default) |
marketplace |
full; server-local via default |
the doors need online + auth-oidc (marketplace-landing-design.md: "the exchange is server-side in holger-server"); no airgap row may carry it. No RON on the estate configures a marketplace: block today |
fmt-maven … fmt-build-thing (13 more) |
none directly — reached through fmt-all |
declared so a future row can name exactly the formats it serves |
Refusal, by name. Repository::backend_from_config on a format whose arm is #[cfg]ed away answers holger-mvp-1 (repository, format, the feature to rebuild with, and the set this binary DOES serve); Holger::instantiate_backends runs that check over every repository BEFORE its per-repo disable loop, so a RON naming a format the build did not include refuses BOOT — never a DISABLED log line and a quieter estate than the RON says. A marketplace: block in a build without the feature refuses boot with holger-marketplace-7 ("built without the marketplace feature"); the door paths live in an always-compiled marketplace_doors so the read-only gate keeps refusing the prefix on every verb. sign now forwards into repository-znippy-package with ?: naming sign (every server row) no longer drags the znippy format in; the CMS-verified znippy path needs sign + fmt-znippy and refuses by name otherwise.
MEASURED (cargo tree -e normal --prefix none | sort -u, the recipe's own flags; bytes from cargo xtask build --suite MVP --variant mvp-server-musl in nordisk-cargo-toolchain@sha256:3d5115ba…, release, musl):
| row | crates before | crates after | bytes before | bytes after |
mvp-server-musl (holger-mvp) |
463 | 442 (−21) | 58 631 752 | 55 193 648 (−3 438 104 B, −5.9 %) |
full (holger-full) |
422 | 422 (diff empty) | — | — |
server-local (holger-server) |
422 | 422 (diff empty) | — | — |
mini (holger-mikro) |
384 | 384 (diff empty) | — | — |
embedded |
369 | 369 (diff empty) | — | — |
The 21 crates gone from the appliance: the 15 non-MVP backend crates (holger-znippy-{maven,python,npm,nuget,go,gem,deb,rpm,helm,docker,conda,composer,package,build-thing}-repository, holger-file-oci-repository), holger-zip-entry, znippy-plugin-python, md-5, lzip-parallel, liblzma and liblzma-sys — the one C library the appliance was linking, for conda. marketplace removes no crate (its deps were shared); it removes the module, its settings type and its three doors from the binary.
Gates: holger-recipe honesty (RULE 1/2) green, toml_parity byte-for-byte after re-emit, capability_lines_name_real_features green with four new lines, the errcode wiring guard green with holger-mvp-1 wired, the traits listing census unchanged (it reads source files, not link state, so it holds under every feature set), cargo check --workspace --all-targets green, cargo test --workspace --lib 45/45 crates green, cargo xtask rows-check 11 of 11, cargo xtask gate --arms GREEN (11 arms, incl. appliance_gate 6/6 under --no-default-features --features mvp-server,appliance — which needed the bench dev-edge to pin marketplace beside fmt-all, since a dev crate building a GatewaySettings literal must see the same fields as the lib).
⚠ Tension on the record: the MVP entry above ("size is not the test", 2026-09-20) moved 15 backends Pro → MVP "at zero cost (all 17 are already non-optional)". The cost was not zero — 21 crates and a C library — and stinky 1.4 (05-suite-monetize-mvp.md §C.2) rules them Pro. This entry follows the stinky ruling; a row that wants a third format names its fmt-* and says why.
W14 — the anti-rollback floor: LANDED 2026-09-21 on agent/w14-floor
The module chain (blake3 + detached CMS against the operator's roots BEFORE Module::new) answered "is this the module we signed?" and could not answer "is it older than what this box already ran?" — a correctly signed OLD module verified forever. The ruling was to fix the SILENCE, not the location, and that is what shipped:
- The lane. A module now NAMES itself:
ModuleIdentity { supplier, product, version }inplugin/abi— a trailing optional block on the ABI v1WireManifestand optional TLV tagIDENTITY = 0x0012onWireManifest2. It sits inside the manifest, inside the module bytes, inside what the CMS signature covers, so it cannot be edited without breaking the signature. The three shipped guests carry one (nordisk/skidbladnir,nordisk/rust-toolchain,nordisk/allocation; version =CARGO_PKG_VERSION, so a release IS a floor step). - The floor.
plugin/host/src/floor.rs:enum Floor { InForce(FloorRecord), Absent }— never anOption. Per(supplier, product), the highest version ever ACTIVATED on this box, at<floor dir>/<supplier>/<product>.floor(MAJOR.MINOR.PATCHordered numerically), written by staging +rename+ fsync at activation (module loaded, verified, grants checked, name accepted into the catalog), read at load in both catalogs (HandlerCatalog::load_dir_with,MeterCatalog::load_dir).PluginPolicygainedfloor: FloorPolicy { NotKept | Kept { dir } }; the server ALWAYS keeps one —Holger::plugin_policy()at<data_root>/plugins/floor, derived, not configurable, because a setting an operator could leave blank would be theOptionthe ruling forbids. Each loaded module's standing is afloor_notes()line the server logs. - The refusals, all frozen and wired through the errcode guard.
holger-plugin-10PLUGIN_BELOW_FLOOR— below the floor, message names supplier, product, offered version, floor version and the floor file;holger-plugin-11PLUGIN_FLOOR_UNREADABLE— the floor file exists and cannot be read or parsed: a REFUSAL, neverAbsent;holger-plugin-12PLUGIN_IDENTITY_MISSING— a floor is kept and the module declares no orderable identity. In the handler catalog all three areScanError::RolledBack, a BOOT STOP besideUntrustedModule, never a shorter catalog.Floor::Absentloads and logs "first activation sets the floor at X". - The tests —
plugin/host/tests/the_floor_never_lets_a_module_go_back.rs, real WAT modules with real identities, CMS-signed against a real root, loaded through the real catalog:a_first_activation_sets_the_floor_and_says_so,a_lower_version_is_refused_by_name_even_when_correctly_signed,the_floor_survives_a_restart,an_unreadable_floor_is_a_refusal_not_absent,two_products_have_two_floors.#![cfg(all(wasm-plugins, verify-modules))], so under the default closure the file compiles to nothing — which is why it is a declared arm in.nornir/testmatrix-arms.jsonbesidean_unsigned_module_stops_the_boot, andgate --armsruns it. Seven pure unit tests infloor.rsrun under every closure. - What it does NOT do, on purpose. It is still on
/data(stinky-03 §C.4): whoever holds the disk can erase it, and then the next load says "first activation sets the floor" out loud instead of nothing. A witnessed floor outside/data(§E.5) is a separate row. The bundle floor (update-bundle, ?26/?27) is untouched; this is the second half of the same lane at the other end of the injection path.
?20 — ANSWERED 2026-09-21: holger-ds is a [[bin]] of the bench crate
The verb had already left server/cli on 2026-09-20 (feature bench deleted, the driver parked as bench-scenarios/examples/holger_ds.rs). It is now bench-scenarios/src/bin/holger_ds.rs, [[bin]] name = "holger-ds", required-features = ["served"], the same --scale/--preset/--data/--target/--eco surface, run as cargo run -p holger-bench-scenarios --features served --bin holger-ds -- --help (a [[bin]], unlike an example, does not see dev-dependencies, so clap is an optional normal dependency under served — fixed on main dc63e11 after cargo check --all-targets went red on it). The crate declares no variant row, so the build thing can never ship it. Release note: holger-server holger-ds no longer exists; holger-server --help does not list it; the tool is holger-ds from the bench crate. MEASURED, cargo tree --edges normal unique packages, main vs this branch on the same base, twice: on 6f8b595 (before SLIM) holger default 422 → 422, embedded 327 → 327, mini 327 → 327, mvp-server 339 → 339, full 405 → 405, holger-agent-cli 273 → 273; on dc63e11 (after SLIM and the clap fix) 422 → 422, 277 → 277, 277 → 277, 309 → 309, 375 → 375, 273 → 273, and a diff of the full package lists is empty. Nothing entered, nothing left — the move is a door, not a dependency. One more pin, after the merge: the bench's server-lib edge now names marketplace beside auth-passkey — main 633260b put GatewaySettings.marketplace behind a feature and served.rs fills it, so a --features served build of the bench crate (and with it holger-ds) failed with E0560 until the feature was DECLARED on the edge, the same shape as the passkey door in 2026-09-20. Dev-only edge: no shipped closure moved.
W9 + W10 — the twin gets a role — 2026-09-21, agent/w9-twin
MEASURED on main 2fb3602 before the lane: the twin was a second empty primary. grep -rn "role\|twin" server/lib/src found nothing that made it refuse a write, nothing that kept its product doors off the public interface, nothing that followed. Its only protection was a firewall row (private-holger-ops/estate.toml:401) on a port nothing would ever dial (stinky 02 S2-1, S2-3). And replicate_pairs moves artifacts BY COORDINATE (replication.rs:6-8), so a twin built from it held no evidence, no audit, no properties, no SBOMs, no sealed store, no bundle state and no config — promoted, it would have served bytes NOTHING had ever attested, with a history beginning at the promotion. gunnar's un-metered-twin scar (S9) in a different coat.
- The role.
server/lib/src/twin.rs, and one new key onHolger:role: Primary | Twin(primary: "holger-primary.utility:50051", poll_seconds: 60). An absent key isPrimary, always — the other direction of default would turn a lost line into a box that refuses its own operator's writes and names a primary nobody declared.Twincarries its primary as a mandatory field, not anOption, because every refusal NAMES it: "no" is an outage, "this is a twin of X, write there" is an instruction. - The refusal.
holger-twin-1TWIN_REFUSES_WRITES, frozen and wired througherrcode/tests/wiring_guard.rs. Raised at the two choke pointsOperationMode::ReadOnlyalready sits on —HolgerGrpc::authorize_writeandauthorize_admin— and asked first, before the read-only check and before a credential is parsed, because no credential can make a follower a primary. Same gate on the HTTP door, for every non-GET/HEAD verb plus the whole marketplace prefix, so the two transports cannot disagree about whether this box accepts writes.authorize_admin_read(the audit query plane) is deliberately NOT gated: a twin whose history could not be asked about is the un-provable box W10 exists to prevent. - No public door.
NodeRole::effective_bindkeeps the port and moves the host to127.0.0.1;wire_holgerapplies it to everyron_url/ron_http_urlFIRST, before anything is indexed. Done in the wiring engine and not at the bind sites so a door added to the config later is loopbacked without anyone remembering to — and so the RON FILE stays byte-identical to the primary's, which is what[estate-twin-drift]demands. - The direction, as a type. Only a
Twinyields aFollowPlan;NodeRole::Primaryhas no field that could name a follower, so a primary has no address to dial even in principle. gunnar's F13 and stinky 02 §2.7 — and the correction ofestate.toml:364-365, which specified the driver as a timer on the PRIMARY dialling the twin. The follower's seam istrait PrimaryView(endpoint · capacity · data root); the implementation shipped isLocalPrimary, a primary data root this process can read. The wire form isholger-remote, W11, still owed — said out loud here because stinky 02 S2-1 measured that holger has no network replication at all, and a module implying otherwise would be the green-looking dead appliance the serving marker exists to catch. - The estate.
replication::replicate_estatesits BESIDEreplicate_pairs, never inside it:holger-traitsis PUBLISHED, soRepoPair,ReplicationOptions,ReplicationReportandreplicate_pairsare untouched and the CLI verb andReplicationService.Replicatecompile and behave exactly as before.ESTATEis a declared list of nine items, each carrying WHAT a twin loses without it:holger-attestations(evidence),holger-audit(segments),holger-props,holger-sboms,holger-prod-store(the seal + lineage),holger-update-state,holger-import-receipts,plugins(the anti-rollback floor) andholger.ron(the served config). Same two properties as the artifact half: idempotent by content id (the same lowercase-hex sha256) and gated byexecute. Same fan-out shape too —gatling_map_balancedover a 256 MiB window for the read+hash+probe phase, one serial committer (ROOT LAW #0: the house engine, never rayon). An item absent on the primary is reported asabsent_on_primary, by name: "nothing to copy" and "copied nothing" are never the same sentence. - The tests, all of them LIB tests so
cargo test --workspace --libruns them under the default closure:a_twin_refuses_every_write_by_name(every mutating RPC listed BY NAME with the gate it passes — PutArtifact, AttachSbom, SetProperty, PromoteArtifact, Replicate, Migration Discover/Start/Cancel — each refused with the code on both channels and the primary named, with a valid admin token and with none; reads and the audit query still open; a primary refuses none of it),an_absent_role_is_primary_not_twin(the Rust default AND the RON default, plusTwin()with no primary being a parse error),a_twin_binds_no_public_door,a_primary_never_dials_its_twin,a_short_twin_is_named_and_never_refused, anda_replicated_twin_holds_the_evidence_not_just_the_bytes— promote an artifact on A, file evidence about its digest, set a property, attach an SBOM, write an audit segment and a served config; replicate; then read every one of them back through the store that owns it at the path that store computes on B; then replicate again and assert the second run writes nothing and skips every file by content id. - What it does NOT do, on purpose. No follower thread is spawned by
start()—follow_onceis a function, and what drives it on a timer is the same owed driverestate.tomlnames, now pointed the right way. No wire transport. No resize of anything.
W9/W10 AUDITED, and what it took to make the twin real — 2026-09-21, agent/holger-twin (lane T5)
W9 and W10 were verified against the CODE, not the plan text, and three things were true that the plan did not say.
1. The role was UNREACHABLE on the real estate. role: reads only from the RON config, and private-holger-ops' [estate-twin-drift] requires the twin's holger.ron to be byte-identical to the primary's. It is: MEASURED 2026-09-21, deploy/generated/twin/{holger.ron,Caddyfile,holger-server.env} and deploy/generated/server/'s three files do not differ by one byte, and neither carries a role: key. A twin installed from the emitted deploy would have come up as a SECOND PRIMARY — public 0.0.0.0:50051, live write plane, a Caddyfile claiming repo.holger.rs. gunnar's scar, in holger's shape, inside the lane that existed to prevent it.
2. Nothing called the follower. follow_once and replicate_estate had ZERO callers outside their own tests — no CLI verb, no gRPC, no timer. There was no restore path at all: nothing holger shipped could move a twin's estate onto a new box. (backup restore covers only the protected bundle store; replicate --from --to moves artifacts by coordinate and takes RON configs, not data roots.)
3. ESTATE was six stores short, enumerated from the code that builds each path: holger-passkeys (the console's identities — a promoted twin had NO ONE who could log into it), holger-tenants (the tenant ledger and the spent-token digests that make a purchase token single-use), holger-entitlements (gunnar's scar, literally), meters, holger-jobs, jobs. And the largest occupant of a live data root — the ARTIFACT STORE, /var/lib/holger/store — is config-named, so no fixed list can ever cover it, while the report said is_complete().
What landed:
- The role is an INSTALL PARAMETER of the ONE medium (owner ruling below).
HOLGER_ROLE/HOLGER_PRIMARY/HOLGER_TWIN_POLL_SECONDS, read inwire_holgerbefore any door is wired. Absent ⇒ the config decides ⇒Primary. An unreadable value REFUSES; it never reads as primary. holger-installergained--param role=twin --param primary=…, on the samekorp.env.*railunderlayalready uses. Neither isrequired, andholger-recipeasserts both that property and that no twin MEDIUM row exists anywhere in the declaration set.- The six stores, each with the reason it matters, plus
EstateReport::unreplicated— every child of the primary's data root the pass does not cover, named, with the frozenholger-twin-3warning. Dotfiles (a running import's lock and staged znippy) are excluded. holger-server estate --from <data root> --to <data root>— the follow AND the restore, one function, because a restore that ran different code from the follow would be code exercised once, on the worst day. Dry-run by default. It printsholger-twin-3and then the exactreplicateline that moves the artifacts, so a restore can never report success over an artifact-less box.
STILL OWED, and stated plainly rather than implied: there is no second machine (upcloud_server.{server,twin,web} are declared in terraform/main.tf and NOT applied — xtask/tests/terraform_gate.rs is red saying so), there is no wire (PrimaryView's only implementation is a local data root; W11), and nothing runs on a timer. The twin's admin-token is deliberately NOT replicated — a secret — so the twin needs the same bytes written by hand or it refuses to start; front plan step 12 now says so.
?49 — ANSWERED 2026-09-21, then RE-ANSWERED the same day: the role is a BOOTSTRAP fact, and the medium carries nothing
The final shape. Owner rulings, verbatim and in order:
- "gunnar twin must be built EXACTLY like the main, it MUST use the SAME iso !!!" / "DO not build a special gunnar TWIN iso !"
- "the weakest machine must become the twin. PERIOD"
- "IT must not CHANGE role later ! so its a bootstrap thing from xtask"
The first answer below (17e3f65) put the role on the medium as --param role=twin. That was wrong and is reverted. One row with two composes is TWO ISOs, and ruling 1 says one. Ruling 2 also makes a compose-time parameter impossible in principle: which box is weakest is a MEASUREMENT of machines that do not exist yet when the medium is mastered.
There are exactly three places a role could live, and only the third works:
| carrier | verdict |
holger.ron |
Refused. [estate-twin-drift] requires the twin's config to be byte-identical to the primary's; the role there is an exception in the one rule that guards gunnar's scar. |
| the medium's command line | Refused. Two composes = two ISOs (ruling 1), and the election has not happened yet (ruling 2). |
| state the box acquires at bootstrap | This one. <data_root>/holger-role.ron — holger's twin.json. The RON stays byte-identical, the rule stays intact, and there is ONE ISO. |
What landed (92ba574): twin::RoleRecord (role + elected_at + elected_by + the measurement verbatim), twin::elect (written create_new, so two callers racing cannot both win), twin::is_fresh, twin::effective_role (election > environment > config), and the door: AdminService.BecomeTwin, admin-gated, reporting restart_required because doors are bound at wiring time and a running server does not rebind them underneath its clients.
Refusals, by name, none of them a retry: holger-twin-4 already elected — including to the same role, because a caller that thinks it needs a second election has lost track of which box it is talking to; holger-twin-5 not a FRESH box; holger-twin-6 the record and the environment disagree.
The record is excluded from replication, not reported as uncovered: a restore carrying it would bring the dead primary's twin-hood onto the new box, which would come up refusing every write and naming a machine that is gone.
AdminService.Capacity is half the measurement the election needs (servable bytes, used bytes). Cores and RAM are not on any holger door yet — xtask reads them from the cloud API today, and if the election is ever to be made from holger's own reading, that is the gap.
STILL OWED and unchanged by this: no second machine, no wire (PrimaryView's only implementation is a local data root), nothing on a timer, and the data_disks election bug is lane T6's (korp_installer::layout, 81b8b3a) — holger inherits that fix rather than working around it.
?49 (first answer, superseded the same day) — the twin is a ROLE of the primary's medium, never a medium of its own
OWNER RULING, verbatim, given for gunnar and extended to holger in the same breath: "gunnar twin must be built EXACTLY like the main, it MUST use the SAME iso !!!" and "DO not build a special gunnar TWIN iso !"
holger takes the role-parameter shape and there is no reason it cannot: the medium already carries per-site env tokens (underlay → korp.env.HOLGER_UNDERLAY), so role and primary are two more rows on an existing rail. There is no holger-twin build row and there must never be one, asserted in holger-recipe/src/rules.rs.
The property that makes ONE medium safe, pinned by two tests: a medium mastered with no role parameter installs exactly what it always did. Neither parameter is required, so nothing is written, HOLGER_ROLE is unset, role_from_lookup returns None, and the config decides — which is Primary.
This also settles the RON half of ?48: with one medium serving both roles the role CANNOT be baked per image, and it cannot go in holger.ron either without writing an exception into [estate-twin-drift] — the one rule that guards gunnar's scar. The path half of ?48 is not a blocker for the twin: both machines declare services = ["caddy", "holger-server"] and both mount their data volume at /var/lib/holger, so the emitted RON is correct for either role and the only install-time-derived value is the role itself.
?6 — the growth order, now that the fill reading EXISTS — ANSWERED 2026-09-21
The forced order was fill reading → twin ≥ primary as a runtime invariant → operator xtask grow --apply → only then a rule. Steps 1 and 2 are now done.
- The fill reading LANDED 2026-09-20, the night ?6 was written:
metering::statvfsandcapacity::measure/AdminService.Capacity(server/lib/src/capacity.rs) — onestatvfs(3)over the data root plus the archive bytes of every repository that OWNS storage, groups excluded so a view is never counted twice, and a root the kernel will not describe is a namedErrand never a zero. The premise of stinky 02 §5 ("holger has NO fill reading at all") expired within a day of being measured, which is why §5's own step list is the thing to read and not its opening sentence. twin ≥ primaryis now a runtime check —twin::store_verdict(primary, twin)over two liveCapacityreadings. The invariant is stated as the twin's SERVABLEavailable_bytesagainst the primary'sused_by_holger_bytes: what the twin could still write, against what the primary has actually put there. A shortfall is reported by name,holger-twin-2TWIN_STORE_TOO_SMALL, naming both numbers and the difference, on the new read-only doorGET /-/twin(role · primary · accepts_writes · store verdict · warning) and in the log on every cycle.StoreVerdict::Unknownis its own variant and is NOT a pass — an unasked question must not read as a green one, which is the failure modeestate.rs's declared-only check had. It is a WARNING and never a refusal: refusing here would take a restore source offline for a disk that is merely small, the opposite of what a twin is for. The point is that the day of the promotion is never the day somebody discovers the twin cannot hold the data — gunnar G12 (DATA-SET-GROWTH-FLOW.md:349) as something holger can read rather than something its estate file asserts (S2-10).- NOT DONE and not this lane's to do:
cargo xtask grow server --gib N --applyinprivate-holger-ops— drain → unmount → resize → re-attach → grow. It needs the hot-plug kernel pin (tunnr kernel-pci-hotplug, proven under KVM, not in a shipped ISO) and it resizes a real volume, so it is the owner's. - Still
never, on every machine, and correctly so. A rule other thanneveris step 4 and waits on step 3.private-holger-ops/estate.rs:683-691's refusal-by-name stays exactly as it is; what changes is that the refusal can now point at a reading instead of at nothing.
What the next step is: the owner's ruling on step 3, and — independently of growth — the twin follower's DRIVER. follow_once exists and is tested; nothing calls it on a timer, and estate.toml:364-365 still describes that driver in the wrong direction (S2-2). Fixing the sentence before somebody builds it is a two-line change in private-holger-ops and was not made here: this lane owns replication.rs and the RON shape, and the estate file is the owner's to move.
W17 — a signature that means something — 2026-09-21, agent/w17-signature
W17's sentence had three clauses. Two are closed in holger and the third was never holger's. What follows is what was MEASURED on main at bb0302e, what it cost, and the codes a reader will meet.
1. The seal chained to a discarded CA — and so did nothing else on this side
?9 fixed the producer: nornir --seal loads a real key and refuses without one (lane/r9-seal-key 4e3f18b). What ?9 could not fix is the CONSUMER, and that is holger's half. MEASURED:
AttestTrust::from_der_root_files (server/lib/src/attest.rs) read each path in attestation.trust_roots and pushed the bytes. No parse, no emptiness check, no shape check. So:
| what the operator named | what happened before | what happens now |
| a zero-byte file | pushed as a "root"; boot OK; matched nothing, ever | holger-attest-10, boot refused |
| a PEM nobody converted to DER | pushed as a "root"; boot OK | holger-attest-10, and the message gives the openssl line |
| a truncated certificate | pushed as a "root"; boot OK | holger-attest-10 |
| an absent path | AttestError::StoreIo — the code an operator reads as the disk is full |
holger-attest-10 |
| a directory | StoreIo |
holger-attest-10 |
| one bad root beside three good ones | three roots loaded, one dropped, nothing said | the WHOLE declaration refuses |
That last row is the finding. An estate declared n anchors and verified against n − k, the log said the evidence plane was mounted, and nothing anywhere printed the difference. It is the empty trust set's quieter cousin: not "we trust nobody" but "we trust a smaller set than you wrote, arrived at by accident".
The rule already existed one crate over. plugin/host/src/verify.rs has refused this since holger-plugin-8 — "refusing rather than verifying against fewer roots than the operator declared" — and attest.rs's own doc comment cited that mechanism as the one it mirrored. It mirrored the prose, not the code.
The shape of the fix is the plugin host's, deliberately: an operator-declared key/cert set in RON, loaded at boot, refused BY NAME when a declared root cannot be read. Nothing compiled in, no default that verifies, and AttestTrust::new() remains the empty fail-closed set it always was. The check is structural (ASN.1 SEQUENCE tag plus a length that covers the file) and costs no dependency; the parse that decides TRUST is still znippy_common::sign's, at verification time, because a second half-parse here would be a second place for the rule to be wrong.
2. "Neither provenance gate runs in a shipped holger"
?10 already ruled which gate it is: holger's own attest.rs DSSE path, compiled into all five server rows, with modgunn staying scan-only. What ?10 left was "making it RUN is a CALLER, not a feature flag".
MEASURED, and it is worse than a missing caller: there was no gate to call. promote::gate::catalog_from_config registered exactly two gates — osv unconditionally, and attested only when promotion.require_attestation named a predicate type. No build carried a gate named provenance. The proof sits in the module's own unit test: an_unknown_gate_name_is_refused_and_lists_what_exists used the literal string "provenance" as its example of a name no build carries.
So the honest answer to "which shipped row ran no provenance gate" is all of them, and a config that wrote gates: ["provenance"] was a BOOT FAILURE (HOL-GUN-007) on every one:
| row | package | ran a provenance gate before | now |
embedded |
holger |
no — provenance was not a name any build carried |
registered, policy-steered |
mini |
holger |
no | registered |
full |
holger |
no | registered |
mvp-server |
holger |
no | registered |
mvp-server-metered |
holger |
no | registered |
mvp-server-musl |
holger |
no | registered |
ProvenanceGate is registered unconditionally by catalog_from_config, beside OsvGate and not inside the if let that guards AttestedGate. That is the point: a gate that exists only once it has been configured cannot be NAMED by a rung on a build that could have carried it, and HOL-GUN-007 then refuses the boot for the wrong reason. the_gate_is_compiled_into_every_row_that_promotes (holger-recipe/tests/feature_honesty.rs) is the census, in the shape of the other honesty guards: it reads the real variant declaration and the real source, names the six rows, and fails if promote/promote::gate acquires a #[cfg(feature …)] or if the registration moves inside a conditional.
The gate asks one question of every member of a promotion: is there, filed on this server, an accepted provenance statement whose subject digest is THESE bytes? Everything in the store arrived through AttestStore::accept, which verifies the DSSE against the declared roots and the enrolled signer before a byte is written — so "filed" already means "chained to the declared root". The gate enforces the CONSEQUENCE, which is the half that had no caller.
The policy, and the defence of its default. promotion.provenance: Require | Warn | Off, absent ⇒ Warn.
Requirecannot be the default. Every existing estate has filed no evidence and most declare noattestation:block at all. Defaulting toRequiremeans a box that upgraded its binary, and changed not one byte of config, comes up refusing every promotion. Every other absent key inHolgermeans what the file always meant —role:absent isPrimary,mirror:absent is no mirror — and this one must too.Offcannot be the default either, and this is the load-bearing half of the answer to the brief's "an absent key must never read asOff".Offis a sentence an operator WROTE: I have decided not to check. An absent key is an omission. If the two printed the same, an estate that never got round to signing anything would be indistinguishable from one that considered the question and declined — which is the defectpromote/gate.rswas written against, wearing a new coat: "a promotion with no gate looks exactly like one that passed every gate."ProvenanceGate::posture()is the sentence that tells them apart, andan_estate_that_declares_no_root_says_so_and_does_not_verify_everythingasserts the two strings are not equal.Warnis what is left and it is honest. It changes no promotion outcome on any existing estate, and it logs, by frozen code, exactly whatRequirewould have refused — the same sentences, in the same order, from the sameProvenanceGate::findings. Moving toRequireis then one word in the config, and the log has already named which artifacts will stop.
The codes. Two, and they are apart on purpose, because they want different fixes:
| code | when | the fix |
holger-promote-10 |
a root IS declared and these bytes have no provenance filed under it | sign the build and file the statement |
holger-attest-11 |
provenance was demanded and the server declares NO root, so nothing could ever have been verified | declare attestation.trust_roots |
Under Require with no declared root the refusal is reject_all — no subject — because the fault is the server's and blaming an artifact would send the operator to re-sign a build when the fix is one config key. A Require that cannot verify is never a Require that passes: the test asserts that even a fully attested artifact is refused when there is no root for its statement to chain to.
3. "dwarves, the path that actually publishes, signs nothing"
Unchanged, and not holger's to change. It is stated here so the strike in §3 is not read as covering it: the clause survives W17 and belongs with nornir/dwarves.
?11 — ANSWERED 2026-09-21 by measurement: half a rename, and the half that matters is not one. ?11 STAYS OPEN, with its price stated.
gunnar's comment calls exporting its BuildRecord to in-toto/DSSE "a rename and not a redesign". Measured from holger's side, against the real KORPSIG1 layout (edda/crates/korp-installer/src/rollback.rs, read only — this lane touched nothing of gunnar's or korp-installer's):
| the bytes the signature covers | |
KORPSIG1 |
"korp.root-of-trust.v1\0" ‖ version (8 B, big-endian) ‖ blake3 root (32 B) — 62 bytes, fixed |
| DSSE | PAE(payloadType, payload) — "DSSEv1 " ‖ len ‖ type ‖ len ‖ the whole JSON statement |
The CLAIM is a rename. Version n of this image has blake3 root r is an in-toto Statement v1 with subject[0].digest.blake3 = r and the version in the predicate. Total, lossless, about forty lines. It is done, behind the non-default korpsig-export feature (server/lib/src/korpsig.rs), and round-tripped by a_korpsig_record_round_trips_through_an_in_toto_statement.
The SIGNATURE is not. A KORPSIG1 signature is not a valid DSSE signature over ANY statement and cannot be encoded into one — asserted, not argued: the test builds both messages and compares them. Producing a verifiable DSSE for the same claim means signing again, with the appliance root key, which holger does not hold and must not.
So the export exists in holger and is deliberately not shipped: korpsig-export is named by no row, because a claim-shaped object with no verifier on this side of the fence does not belong in a shipped binary. It is compiled and run by a declared arm instead. What holger CAN honestly do with it, once attestation.signer_key is configured, is COUNTERSIGN — this server observed KORPSIG1 version n over root r — chaining to holger's own declared root. The raw signature and public key travel inside the predicate as opaque hex, and the predicate says of itself that the signature covers signedRecord and NOT this statement's PAE, so no reader has to infer it. The observation gets its OWN predicateType (https://nordisk.dev/korp/root-of-trust/v1), never SLSA provenance, so the provenance gate can never read this machine booted as this artifact was built — a_boot_root_observation_is_not_a_slsa_provenance.
What would close ?11, and ?40 already priced it: ONE certificate. The same 32-byte ed25519 seed is a KORPSIG1 boot anchor, a PKCS#8 v1 private key behind a fixed 16-byte DER prefix (RFC 8410 §7), and an X.509 SPKI. Give the appliance signer a certificate and it emits a real DSSE beside its KORPSIG1; one trust store then covers links 3–10. That work is on edda's side of the fence (lane/r10-key-lineage already built the seam), not holger's.
?40 — ANSWERED 2026-09-21: TWO keys, and the join is a CERTIFICATE rather than a merge. holger's end of it is now built.
?40 asked whether the release key and the appliance root key should be ONE key. The answer is that the question is slightly mis-shaped, and the measurement above shows why: the two chains are separated by their SIGNED MESSAGE, not by their key material. Merging the keys without a certificate changes nothing — a KORPSIG1 signature over 62 bytes is still not a DSSE signature over a statement. Issuing a certificate over the seed joins them whether or not it is literally the same seed.
So: keep two keys if the operator wants two, but require ONE CERTIFICATE per signer. Concretely,
- Path B's side is CLOSED by this lane. holger no longer has a signer that chains to a discarded CA; it has an operator-declared trust root, refused by name when it cannot be read, and a promotion gate that depends on it. Path B now covers a machine's boot root too, as an observation, the moment somebody hands holger a
KORPSIG1record. - Path A's side is edda's, and it is one artifact: an X.509 certificate over the boot signer's public key, enrolled in holger's
attestation.trust_roots. ?40's own measurement stands — artifact anchors 3 → 1, customer enrolments 6 → 4 — and nothing in this lane forecloses it.
The sentence ?40 opened with was "the machine a customer boots carries no attestation, and the attestation a customer can verify covers no machine." The second half is answerable today: korpsig-export gives holger a way to say what it observed about a machine, in path B's vocabulary, keyed by the same blake3 root the appliance fails closed on. The first half needs edda to issue that certificate.
Tests, and what each proves
| test | where | proves |
an_unreadable_declared_root_is_a_named_refusal_not_an_empty_trust_set |
server/lib/tests/the_provenance_gate_stands.rs |
empty / PEM / truncated / absent / directory each raise holger-attest-10; a real rcgen DER loads; one bad root refuses the whole declaration, in either order |
a_promotion_whose_evidence_does_not_chain_to_the_declared_root_is_refused_by_name |
same file | holger-promote-10, naming the bytes and the subject; a statement about OTHER bytes still refuses; a two-member set names the offender; through a resolved GateSet the Decision is a policy refusal and not undecided |
an_estate_that_declares_no_root_says_so_and_does_not_verify_everything |
same file | Warn allows AND finds; Off allows and finds nothing; their posture strings differ; Require refuses with holger-attest-11 and no subject; even a fully attested artifact is refused with no root declared |
the_gate_is_compiled_into_every_row_that_promotes |
holger-recipe/tests/feature_honesty.rs |
the six rows that build holger are named; promote and promote::gate carry no #[cfg(feature …)]; ProvenanceGate is registered before the AttestedGate conditional |
five korpsig tests |
server/lib/tests/the_korpsig_export_is_half_a_bridge.rs (arm, korpsig-export) |
the constants match korp-installer's layout by byte vector; the claim round-trips; the signature demonstrably does not cover the statement; the legacy unversioned blob is refused rather than exported with an unsigned version; a boot-root observation is not SLSA provenance |
Both integration files are declared arms in .nornir/testmatrix-arms.json, so cargo xtask gate --arms runs them and silenced_tests_guard holds the file to what is on disk.
What this lane did NOT do
- dwarves still signs nothing. Out of scope and out of repo.
- No enrolment config key.
AttestTrust::enrolled— the SHA-256 fingerprints of gate certificates this server accepts verdicts FROM — has no RON key, soAttestStore::acceptstill refuses every envelope withholger-attest-4on a box configured only through theattestation:block. The trust ROOT is now declarable and honest; the ENROLMENT is the next config key, and it is a one-key change of exactly the shapetrust_rootsalready has. - No door.
attest.rsstill has no RPC: evidence can be filed only in-process. That was ?10's "W6/W3" and is not W17.
⚠ A FLAKE this lane observed and did NOT fix — statvfs_reports_the_size_the_kernel_states_for_a_tmpfs
Recorded here rather than re-run into silence, because §5 already carries the house rule: "a flaky guard teaches a reader to re-run until green, which is how a real red gets waved through."
server/lib/tests/the_storage_meters_measure_a_real_filesystem.rs:145 writes 8 MiB into a probe directory under /run/user/1000 and asserts used grew by exactly 8 388 608 bytes. On a loaded box it failed once with:
wrote 8388608 bytes but `used` went from 241664 to 8626176
— a delta of 8 384 512, short by exactly one 4 KiB page. /run/user/1000 is a shared tmpfs: the other agents on this machine were writing and deleting in it at the same time, so the baseline moved under the measurement. It is the assertion that is wrong, not the statvfs reading: an exact equality over a filesystem this process does not own cannot hold. It passed 6/6 in two full gate --arms runs on this branch and 3/3 on re-run immediately after the failure.
Nothing in W17 touches that file or the statvfs path. The fix belongs to whoever owns the storage meters and is small: give the probe its OWN tmpfs mount, or assert the delta covers the written bytes within a page rather than to the byte, and SAY in the message that the filesystem is shared.
5. LANDED — 2026-09-20
| what | where |
holger's root manifest declared NO version, so nornir_bench::runner::subject_version returned None and EVERY holger bench row recorded so far is stamped 0.1.8 — the HARNESS crate's version, not holger's. [workspace.package] version = "0.6.10" added (what traits, server/lib, server/cli and agent/cli all declare); inert for publishing, since no member says version.workspace = true. ⚠ Which rows are wrong: all of them. Every row in bench_history.jsonl and in the warehouse written before this must be read as "holger at that commit", including the authoritative 20/20 oden run behind W0 and the competitor comparison in ?31. The v0.7.1 git tag (2026-06-15) disagrees with every crate version in the tree; that is a separate question |
lane/r9 9a3b7e2 |
edda/crates/nornir-bench/src/runner.rs:20 was going to be corrected for claiming dwarves bench run consumes these functions — the claim is TRUE and nothing was changed. dwarves has Cmd::Bench/BenchCmd::Run (crates/dwarves/src/main.rs:1001,1193,1719), crates/dwarves/src/bench.rs calls nornir_bench::runner::{find_bench_example, run_bench_example, RunOptions} at :67,208,211,279, and crates/dwarves/Cargo.toml:210,973 pins nornir-bench behind the bench/release features. Verify a "this comment is wrong" finding before editing it |
no change |
the console runs on holger.rs with passkey enabled; /login, /auth/status, the ceremony and the wasm bundle all served; / still the appliance's page, untouched |
live |
| 12 ceremony tests driving a real P-256 key through the PUBLIC door — main had 25 auth tests and NOT ONE signed anything | lane-a/ceremony-proof 74f49c7 |
| the login that silently returned an operator to the stranger's view | same |
front becomes a feature; variants compile IN THE CONTAINER |
holger b8e13a0 |
the appliance built WITH auth-passkey; passkey measured at +399 944 B, +1.2 %, 4 crates — it is NOT the bloat |
holger 542e296 |
the passkey door was an ACCIDENT before this — default never named it; it arrived through the bench edge |
same |
holger-front published (0.1.0) and MOUNTED in the console; holger-errcode 0.1.2 after Gate B caught 0.1.1 being four rows short of its source |
lane-f/* |
the front mount named four paths while the door served six — /vetra.svg and /ignalina.png 404'd on the live page |
fix/front-marks 411d85d |
holger-ui builds in the declared container (mvp-ui, 98.7 s, pinned image digest) |
lane-g/build-door 156e78b |
terraform from 3 destroy to 0 by adopting the live machine as web; the old volume is RELEASED, not destroyed |
private-holger-ops 4af5692, 2d9febb |
| the MUSL wall was LOKI's box, not the house's — re-measured, and the claim replaced rather than deleted | same |
exactly ONE DSSE implementation exists in the house, and attest.rs delegates wholly to it — the "holger reimplements DSSE" claim was wrong |
stinky/signing 8bee189 |
The afternoon and evening of 2026-09-20 — the ladder, the mirror, and the gate that runs
| what | where |
| The A→B promotion ladder declared and proven on holger.rs, and then C, the on-prem downstream, stood up on oden and proven PULLING from it — the last MVP feature, answered rather than argued (?1) | 221f915, d20c449, merge cf2ee26 |
/-/flow draws the WHOLE RON — storage, doors, ron_in/ron_out, the mirror — in skade's style, with mirror≠proxy edges, arrows routed so every one is visible for its whole length, and client addresses no longer drawn as nodes. Mounted at /-/flow and /-/flow.svg, gated exactly like /metrics |
c9d0bbe, 98db1e9, e7ccaf1, bc2e89e, 57d9e1a, merge 4b4d06e |
| The migrate engine: one classifier for the source repo → holger repo plan, a resumable sha256-gated runner, the two missing `nexus | artifactory → holgerarms,MigrationService` with FOUR RPCs because a migration runs for hours, and a console page proven from the CLIENT side of the wire over real HTTP |
The mirror change map — the declaration the engine never had and the caller it never had, a mirror: RON block, the diff MEASURED by COUNT (a shared box cannot be timed), the fetch path that stopped copying every artifact, resilience (a failed tick is not the end of the timer, a stopped mirror says so, named at info in the default floor), and mTLS per repository or across the board |
221f915, 7743d63, 64dcba1, 241d3eb, cacf458, 0e22835, 30a8468, 125803f, merges 7a6b411/afeec1a |
ArtifactEntry gained published_unix_ms and sha256 — and an honest unknown rather than a zero, which is the only reason the gap below is visible at all |
b986449, b537542, merge 066206a |
change_token — the provenance answer a znippy-backed listing CAN give, and the one that makes the mirror settle (?46, owner: option (a)). Additive optional string change_token = 6 on ArtifactEntry, an Option<String> on RemoteAsset, opaque and source-scoped: equal ⇒ unchanged AT THIS SOURCE, different ⇒ changed, different SCOPE or a token on one side only ⇒ unproven — never changed, never unchanged. Folded from the per-chunk BLAKE3s the znippy lookup sub-index already carries, so it costs one index read and no artifact byte (counted: extract_file 0, three tokens). Nine of the fifteen znippy backends stamp it; the six that list through a typed view cannot name their archive row and say so in ListingProvenance. ChangeMap is v2 — one nullable column appended — and a v1 map still loads, so no live mirror re-downloads the world on the upgrade. PROVEN: second cycle over an unchanged znippy repository = unchanged 3, to_fetch 0 under the DEFAULT Refetch, a same-size edit of one artifact = changed 1, unchanged 2, and the same coordinates from a second source = unproven 3. No znippy-common change was needed: ArchiveReader::locate already publishes the chunk checksums |
34a073e, fefb10e, merge ebca94c |
Audit in three selectable modes — Off, Aggregated, Everything — with the Off footgun closed at the CLI door too, the mode PRINTED, the RON key pinned against a config file, and the cost of each mode measured and written down |
4a88e2f, cdeb043, 8ed3111, f7e9a01 |
| sha256 carried from the Nexus and Artifactory connectors, and a mirror's upstream password given a place to live with a literal one REFUSED | a9f0ceb, 4463f21, merge 3425dd6 |
| Per-IP audit tallies — what was sent per client ADDRESS, not only per principal. This is half of ?1's answer: one identity, many hosts, still legible | aa8c42c |
| A vault with multiple holders, and TPM | edda |
| The nornir bench stamp fixed — a row is stamped with holger's version, or with NOTHING | 5f5c364 |
full builds again (the federation upstream was never recompiled after Vec<u8> → Bytes), ORIGINAL records 6 of 6, and cargo xtask rows-check exists: 11 of 11 rows, 69.1 s for the feature axis. The ORIGINAL openssl verdict was re-measured and found STALE |
8262a37, dd0160c, bb82fdf, 13ce90c, merges 501c13c/b13dc03 |
Seven dark test guards lit — six SILENCED security guards declared plus one found by the gate itself; 47 test functions reported ok having run NOTHING, 53 run now. cargo xtask gate phase 5 RUNS the declared rescue arms, and a red arm now says N FAILED instead of "it did not compile" |
a71c58b, 20ea793, 5cfc471, 9f1aff4, merge 6fee924 |
MVP declared a SIBLING of ORIGINAL via independent_suites — and three defects were hiding behind the assumption that it was not |
b8562f3 |
The monetize plugin, both halves, ABI v2, and metering LIVE — WasmGate is the whole join between a loaded meter guest and holger's gate seam; the meter catalog and the module census; the usage source, the audit sink and the clock wired to the guests; the meter stops spelling a zero nobody counted; and the wasmtime comment that was five months stale is gone |
c9109c4, 5f9462a, 6891c9e, e46de72, ca4c8ec, 44ceb7e, merge f6287a2 |
THE PLAN ITSELF reached main. .nornir/HOLGER-PLAN.md and nine investigation reports under .nornir/stinky/ lived only on stinky/index, 46 commits ahead of a main that had never seen them. Merged, cargo check --workspace --all-targets EXIT 0 — nothing under .nornir/ is include_str!'d into a build |
2d14561, 76d8600, 7c7213a, 633190a, c78f47b |
2026-09-20, after the plan reached main — the first half of the unchanged gap is closed
| what | where |
A raw repository can answer unchanged. raw published a time and no digest, so a downstream mirror could never conclude "nothing changed" and re-fetched the whole repository every tick — the one live defect in ?1's answer. The listing still does NOT hash (that was the right instinct and it is kept): what changed is that raw stopped THROWING AWAY a digest it had already paid for. The two doors that hold a whole blob anyway — a PUT and a fetch it SERVED — memoize relpath → ((len, mtime), sha256), and list PEEKS it. One hash per blob per change, never one per listing; the pattern is file-rust's. Raw differs in the one way that matters and it is handled: a raw blob is MUTABLE, so an out-of-band overwrite falls back to an honest None rather than answering with the digest of bytes that are gone, and an unlink forgets the path. Memo bounded at 65 536 entries, growth-stops rather than evicts, so the answer never depends on access order. Three tests, first of them the one that must stay red-able: a blob that appeared behind holger's back is listed with NO digest. 676 lib tests green |
774ea3b, merge 7b38eb9 |
⚠ The other half is NOT done: the znippy-* backends publish neither a time nor a digest. Same defect, different store — it needs a digest in znippy's file table rather than a memo, because those backends do not hold loose bytes. Until it is done, unchanged is reachable for raw and file-rust repositories and unreachable for every znippy-backed format |
OWED |
The egress judge learns what a TLS backend is — and stops having no opinion about the tier the leak actually used. FORBIDDEN_TLS_BACKENDS (openssl, openssl-sys, native-tls, tokio-native-tls, hyper-tls) is judged on EVERY variant including full, before the tier split; the bare Verdict::ok() that silently passed server-local becomes a verdict that names what it judged. This closes W2's owed half |
188f2fe |
The persisted bench door, proven END TO END on oden — not in tests: cargo xtask bench appended run 740cdbfb-e52f-4f28-8363-c5552d9c05a7 (four arms — blake3, checksum_upload, proxy_hop, replication_hop — 4/4), the row landed, the version read back as 0.6.10 from the parquet column, and history-show sees it. That closes two defects for real rather than in a fixture: "a bench run and history-show never meet", and the 0.1.8 stamp that §5's first row is about |
565d83c |
A guard caught my own red, and it is worth writing down which one. 774ea3b taught raw to memoize a digest and did not update ListingProvenance::of, which still declared TimeOnly. every_backend_declares_what_its_listing_can_prove reads each backend's own SOURCE and refuses to let the declaration and the code disagree — it named the file, the declaration and the contradiction. Main was red on -p holger-traits from that merge until the fix, because I ran cargo test -p holger-server-lib --tests and not the workspace. raw is now TimeAndDigestWhenOnHand, the vocabulary file-rust already had |
b4785eb |
A flaky guard made honest. distinct_directories_do_not_multiply_the_upstream_key_fetches failed "got 2" under full-workspace parallelism on this 32-core box and passed 5 of 5 in isolation: jwks_cache() is ONE cache for the whole process and two tests clear it and then assert on what it caused. A flaky guard teaches a reader to re-run until green, which is how a real red gets waved through. The two tests now take one lock for their whole body. 1231 workspace lib tests green, 0 failed |
02c2458 |
the job queue — 2026-09-21, agent/job-queue
stinky 01 §6 ("Job queues: there are none") is struck, with a pointer from the stinky file to here. What it measured on 2026-09-20: promote, replicate, upload and gate all ran INLINE on the RPC; nothing resumed after a restart; a running row was permanent once the process died; and promote_ledger did 606 seals/s on 12 cores against 349 on 32 — the seal path was strictly serial, so a customer who bought a bigger box got faster reads and SLOWER writes.
The seam. server/lib/src/jobs.rs, behind the jobs feature. A Job enum over the four long operations (Promote, Replicate, Upload — the gated finalize half — and Gate), a JobQueue under <data_root>/jobs, and a JobId the RPC returns immediately. It is not a second ledger: the payload is serialized into the SHARED jera::jobs row's detail_json, so the queue needs no table of its own and a reader that only knows jera still sees every job. The worker pool is gatling — gatling_run over N workers on one gatling::background::Job supervisor (ROOT LAW #0), guarded by a census test over this module's own shipped source, the way skade's rayon_free_law.rs does it.
Recovery. At open, every non-terminal holger row is either RESUMED (Replicate/Upload/Gate, all idempotent) or moved to interrupted with a reason (Promote — the seal is the immutability commit and the source bytes may have moved under it). A running row never survives a restart as running. interrupted is holger's own status and is deliberately NOT terminal to jera's pruner, so an interrupted job outlives the rotation until an operator has seen it. Naming it failed would claim it was tried and refused, which is a different fact.
Idempotency. Every job carries a content_id; a second admission for the same coordinate gets the FIRST job's id back. Answered from an in-memory index rebuilt at recovery, not by re-listing the ledger per enqueue — a scan per admission is quadratic in the ledger, which is the shape §7.1 measured.
The RPCs, additive. PromoteArtifactRequest.async_job (4) / PromoteArtifactResponse.job_id (5); ReplicateRequest.async_job (5) / ReplicateResponse.job_id (10); PutArtifactRequest.async_job (4) / PutArtifactResponse.job_id (3). Unset is the ORIGINAL synchronous behaviour, byte for byte, and job_id is then empty — no id is invented for work that ran inline. JobQueue::await_completion keeps the synchronous answer available on the queue's own path. The gates are NOT skipped by the async path: a queued promotion runs them on the worker before the seal, and an async upload is reached only after the ingest gates have allowed. A build with no queue, or built without the feature, refuses async_job BY NAME. JobService.GetJob / ListJobs — READ-ONLY, ADMIN-gated exactly like AuditService, mounted only with the feature — closes §6.4 ("the ledger is written and never shown"). Nothing published changed: holger-traits is untouched by this lane.
The feature. jobs names the three ORIGINAL server rows (embedded, mini, full) and the three MVP server rows; server-local carries it through server/cli's default. It adds NO crate — jera and gatling (rotaryengine) were already unconditional in server-lib — so what a row decides is whether this binary carries a SCHEDULER, which is posture, not size. .nornir/build-variants.toml re-emitted with cargo run -p holger-build-thing -- emit, never hand-edited. cargo xtask rows-check: 11 of 11 cargo rows.
MEASURED, t14s (16 threads, AMD Ryzen AI 7 PRO 360, nvme ext4), release, 5 000 seals × 16 KiB
| arm | seals/s | MB/s | wall | cores busy |
promote_ledger — the inline path, unchanged |
607.97 | 9.50 | 8.22 s | 1.01 / 16 |
promote_queue_admit — what the RPC caller now waits for |
65 870 | 1 029 | 0.076 s | 1 / 16 |
promote_queue_drain — the same burst end to end through the pool |
844.56 | 13.20 | 5.92 s | 12.77 / 16 |
Read it in that order. 1.01 cores → 12.77 cores is the answer to "a bigger box gets SLOWER writes": the seal path now fans out, and end to end it is 1.39× faster than the inline path while writing three durable rows per job against the inline path's two — a pessimistic comparison that still wins. Admission is 15.2 µs and is independent of the artifact's size, because it writes one queued row and hands off; the fetch, the hash and the blob write happen on the worker. So a 1 GiB promote no longer holds the RPC — the caller is released ~108× sooner than the inline path's per-seal cost at 16 KiB, and the gap widens with every byte. Proven structurally as well as numerically: a_replication_returns_a_job_id_before_the_copy_runs admits with NO runner attached and asserts the destination is untouched when the answer comes back.
⚠ The three rows above are a LOCAL measurement (cargo run -p xtask --release --example nornir-bench) and wrote no warehouse row: cargo xtask bench resolves [repo.holger] to /home/rickard/git/holger, so the door measured MAIN, not this worktree, and only the one arm it linked. That run is the independent baseline and it is recorded as such: 150.34 seals/s for promote_ledger on a box that was 16/16 busy with another lane's build (bench_runs 69b7e9bf-e3b3-4dc5-8a5c-99a35f1ba441). The three-arm table was taken on an idle box (host-busy 1/16 for the first two), which is why its promote_ledger reads 607.97 and not 150.34 — same code, different weather. The arms are registered, so the door will land them once the config points at a checkout that has them.
Tests — named for what they prove
a_promotion_returns_a_job_id_before_the_seal_is_written, a_job_survives_a_restart_and_finishes, a_running_job_does_not_survive_a_restart_as_running, two_jobs_for_the_same_coordinate_do_not_both_seal, an_interrupted_job_is_reported_by_name_not_left_running, a_failing_job_carries_its_reason, the_worker_pool_is_gatling_not_rayon, a_replication_returns_a_job_id_before_the_copy_runs, asking_for_a_job_on_a_server_without_a_queue_is_refused_by_name, the_synchronous_shape_is_unchanged_and_carries_no_job_id. All green; cargo test --workspace --lib 45 suites ok.
What this lane did NOT do
The HTTP door has no async shape — only gRPC PutArtifact carries the upload half, and the browser console is out of scope by instruction. There is no retry-with-backoff and no poison count: a failed job is failed with its reason, and re-submitting is the operator's decision. There is no Cancel: cancelling a seal in flight is a separate decision with its own consequences.
W15 — the duplicates collapsed — 2026-09-21, agent/w15-w13
Measured first, and the measurement moved two of the row's claims.
The six copies, with file:line, the major each links, and whether the body differs
| # | copy | file:line | p256 | body differs? |
| 1 | edda nornir-auth-webauthn::pure — the one |
edda/crates/nornir-auth-webauthn/src/pure.rs:267 verify_registration, :359 verify_assertion, :471 verify_es256 |
0.13.2 | it is the reference; strictly a SUPERSET of 2/3/4 (adds public_key_is_loadable at enrolment, an optional require_user_verification, a checked credential-id length, typed CeremonyError) |
| 2 | holger-ui-server | was holger-ui/crates/holger-ui-server/src/auth/mod.rs:1164-1305 |
0.13.2 | GONE before this lane started — collapsed 2026-09-20 in 1e4b98c. The row's count of five was already four |
| 3 | gunnar-ui-server | was gunnar-ui/crates/gunnar-ui-server/src/auth/mod.rs:1290-1431 |
0.13.2 | byte-identical to 2, measured: diff <(sed -n 1164,1305p holger-ui/…/auth/mod.rs) <(sed -n 1290,1431p gunnar-ui/…/auth/mod.rs) was EMPTY, 142 lines. Collapsed in this lane |
| 4 | garmr-cli | garmr/crates/garmr-cli/src/api/passkey.rs:403-560 (parse_registration, verify_signature, cbor_map_get/cbor_int_get) |
0.14.0 | the ANCESTOR — 2 and 3 are ports of it. Semantically the same ceremony; not byte-identical (different helper names, different error strings, ParsedReg vs ParsedRegistration). Did not move |
| 5 | garmr-cli OIDC | garmr/crates/garmr-cli/src/api/oidc.rs:106-142 |
0.14.0 | a DIFFERENT verifier — ES256 over a JWT signing input, not WebAuthn. Counted as a crypto copy by the row; it is not a WebAuthn verifier and has nothing to collapse into nornir-auth-webauthn |
| 6 | korp-installer TPM quote | edda/crates/korp-installer/src/attest.rs:548 verify_ecdsa |
0.13.2 | a DIFFERENT verifier again — P-256 over a TPM quote, and it needs normalize_s() for the high-S form a TPM returns, which no WebAuthn path wants. Nothing to collapse |
"Different majors of the same dependency" is garmr vs everyone, not holger-ui vs gunnar-ui. Both UI servers resolved p256 0.13.2 — identical in source AND in the lock. garmr resolves 0.14.0. That is the thing that makes "identical" false in the lock and true in the source, and it is the reason copy 4 could not move tonight (below).
The garmr/vendor/ note in the row is also corrected: vendor/znippy, vendor/edda, vendor/skade, vendor/facett, vendor/znippy-zoomies are git subtrees, declared as path deps and excluded from the workspace (garmr/Cargo.toml:35,76,82). vendor/znippy/znippy-common/src/sign.rs is a 1 305-line copy of upstream's file, not an independent second implementation. A subtree that drifts is a real hazard, but it is a VENDORING problem and not a sixth hand-written verifier.
What collapsed
gunnar-ui-server → nornir-auth-webauthn 0.1.2, the other half of holger-ui's 1e4b98c. 223 lines out of auth/mod.rs, the whole "Ceremony crypto" block plus the sha2 import. The lock gains nornir-auth-webauthn and nornir-service and nothing moves down. 521 tests green, cargo check --workspace --all-targets clean.
Two behaviours are now stricter and both are at boot or enrolment, never on the login path: PureRelyingParty::new refuses an IP-literal rp_id or an origin that does not belong to the rp_id, by name, in Webauthn::new; and verify_registration refuses a COSE key that is not a loadable P-256 point at ENROLMENT instead of at the first login. The counter rule is unchanged in effect — count != 0 && count <= stored became stored != 0 && count != 0 && count <= stored, and the extra clause can only fire where count <= 0 is already false.
What did NOT move, and why
garmr's WebAuthn ceremony (copy 4): a version cascade. nornir-auth-webauthn 0.1.2 pins p256 = "0.13". garmr-cli pins p256 = "0.14". Linking the crate into garmr today would put two P-256 implementations in one binary — strictly worse than one duplicate, and in an appliance that is a second attack surface for one ceremony. Moving it needs, in order: nornir-auth-webauthn → 0.2.0 on p256 0.14 (a major bump, released through dwarves crate-release); then holger-ui-server, gunnar-ui-server and holger/server/lib (Cargo.toml:358,748) re-pinned to 0.2; then garmr's ceremony deleted. Four crates and one release. That is the cascade, stated, not skipped.
The five ed25519 call sites: they are not duplicates. Measured:
| site | signs | small-order key refused by |
edda/crates/korp-installer/src/trust.rs:127 |
the bare 32-byte blake3 root (legacy) | is_weak() in from_pubkey_bytes:81 |
edda/crates/korp-installer/src/trust.rs:136 |
rollback::signed_record — 62 bytes of domain tag ‖ version ‖ root |
same |
edda/crates/korp-installer/src/layout.rs:1218 |
the layout config body | vk.is_weak() inline at :1214 |
edda/crates/eid-core/src/verify.rs:49 |
the eID assertion signing input | verify_strict |
holger/update-bundle/src/signer.rs:161 |
the detached bundle manifest | verify_strict |
garmr/crates/garmr-audit/src/sign.rs:129 |
an audit checkpoint | verify_strict |
Six sites, six DIFFERENT messages. The message construction is the domain separation: it is the security design and the one thing that must not be shared. What they genuinely share is a single boolean policy — a small-order public key is refused, because such a key verifies a forged signature over ANY message — and every one of them already enforces it, in one of two spellings that close the same hole. An earlier reading of the verify/verify_strict split as a laxity is wrong and is corrected here: korp-installer pairs verify with an explicit is_weak().
So a helper would share vk.verify(..) and nothing else, at the price of a new published edda crate and a release cascade through holger and garmr. Instead: edda/crates/korp-installer/tests/every_ed25519_verifier_refuses_a_weak_key.rs, a census that drives the all-zero key at every ed25519 verifier the crate exposes and requires a NAMED refusal from each, guards layout.rs's inline is_weak, and pins the count of verify sites so a seventh is a test edit somebody has to look at. 3 tests green.
Not pushed: gunnar-ui. git.gunnar.rs:2222 refuses the connection — the forge appliance is down, and oden (the bypass remote) is off. The branch agent/w15-w13 and its merge into main are on disk in /home/rickard/git/gunnar-ui; they need a push when the forge is back.
W13 — trusted time — 2026-09-21, agent/w15-w13
The seam existed and nothing fed it. The grep the plan asked for — SystemTime::now|Utc::now|now() under server/lib, traits, plugin, filtered for expiry words — returns exactly two hits, and both are Instant deadlines, not validity: server/lib/src/proxy.rs:333 (a cache entry) and server/lib/src/jobs.rs:623 (a job timeout). No expiry check in holger's server reaches for the clock, because holger's validity windows all live in update-bundle.
Which checks take a now, and which reached for the clock
| check | takes now? |
before | after |
verify::verify_member_map (bundle expiry :683, maximum_bundle_age_secs :687, certificate windows) |
yes, now: DateTime<Utc> since the crate was written |
— | unchanged, and now records time source: unnamed when a caller hands a bare instant |
verify::verify_control_plane_at |
yes | — | unchanged |
verify::verify_detached_by_trusted_signer_at |
yes | — | unchanged |
verify::verify_control_plane |
no | Utc::now() :495 |
unchanged (it is the _at wrapper; the pattern predates W13) |
verify::verify_detached_by_trusted_signer |
no | Utc::now() :140 |
unchanged, same reason |
api::verify_bundle_file |
no | Utc::now() :31 |
delegates to verify_bundle_file_at(.., &HostRtc) — same instant, now named |
api::extract_bundle_file |
no | Utc::now() :45 |
delegates to extract_bundle_file_at(.., &HostRtc) |
bin/holger_verify.rs:108 |
n/a | Utc::now() |
the CLI's own call site; unchanged this lane, and the _at door is there for it |
The seam, and where it lives
holger/update-bundle/src/time.rs:
trait TrustedTime { source_name() -> &'static str; is_trusted() -> bool;
now() -> Result<DateTime<Utc>, TimeError> }
HostRtc "host-rtc", is_trusted() == false — the default, named
SignedTime "signed-peer", is_trusted() == true — REMOVED 2026-09-28, see below
2026-09-28 — ONE signed-time format (one-signed-time). SignedTime and its holger-time-v1 detached-ed25519 envelope were REMOVED: nothing served that protocol, nothing outside its own tests constructed it, and the 2026-09-20 ruling refused a seventh bespoke envelope. The house's one signed time is edda estate-clock's ESTATE-TIME/1 (SSHSIG, namespace estate-time-v1@nordisk), which holger's front serves on :50123 and estate-clock run serves on an airgapped site's own clock host (ESTATE_CLOCK_TRUST_SYSTEM=1). The verifier's source is holger-server's EstateTime over server_lib::clock; the CLI links it through secure_update → server-lib/clock-follow → estate-clock/peer (edda poll: the peer + SNTP poll with NO HTTP client, so a zero-egress downstream keeps its guard). EstateTime refuses (TimeError::Unverified) when a clock source is CONFIGURED and nothing verified answered, instead of answering from the RTC; with no source configured at all it answers from the RTC and the verdict says UNTRUSTED. The sections below describe the removed type as it was.
In holger, not in edda, and the reason is a cascade and not a cycle. holger depends on edda; edda does not depend on holger; there is no circularity. holger-update-bundle's entire dependency list is nine crates and it is deliberately network-free. edda owns no time crate today — gunnar-clock lives in gunnar, not edda, and is publish = false. An edda home therefore means a NEW published edda crate plus a version cascade through holger's lock, for a trait with two implementations and one consumer. If a second estate ever needs it, the move is a crate extraction, not a redesign.
SignedTime is deliberately gunnar's shape and not a new one (gunnar/crates/gunnar-clock/src/peer.rs): this box picks a fresh nonce, the FRONT signs (nonce, unix_ms), and the answer is taken only if the signature verifies AND the nonce is the one this exchange is waiting for AND the round trip was inside MAX_ROUND_TRIP. No internet, no DNS, no NTP, no TLS — verifying a signature needs no notion of "now", which is exactly the property a certificate-dated transport lacks when the clock IS what is being corrected. It differs from gunnar's in one respect, stated: gunnar signs SSHSIG in namespace gunnar-time-v1@gunnar.rs because its key vocabulary is SSH; update-bundle already links ed25519-dalek and nothing else, so the signature here is a detached ed25519 over the same three-line message under the domain tag holger-time-v1. A time signature can never verify as a bundle blessing and a bundle blessing can never verify as time, whatever key made it.
Additive, because the shape has consumers
verify_member_map, verify_bundle_file and extract_bundle_file keep their exact signatures. verify_member_map_with_time, verify_bundle_file_at and extract_bundle_file_at are SIBLINGS. The old doors delegate with &HostRtc and the name is recorded, so what is accepted is unchanged and what is SAID is not: every report now carries one line before the checks that rest on it — time source: host-rtc (UNTRUSTED — this box has no verified time, so every expiry below was checked against a clock nobody signed), or (verified), or unnamed for a caller that handed a bare instant.
No skew window grew
MAX_ROUND_TRIP is gunnar-clock's five seconds and NONCE_BYTES its thirty-two, both unchanged. holger declares no skew tolerance of its own: the expiry comparison stays now > exp, exact, with no grace on either side. no_skew_window_grew is a census in the shape the repo's other guards use — it pins both constants AND the absence, greping verify.rs for now - grace, now + grace, SKEW, skew_secs and tolerance and failing if one appears.
Tests (87 green in holger-update-bundle)
an_expiry_checked_against_an_untrusted_clock_says_soa_signed_time_that_does_not_verify_is_refused_not_ignored— six refusals: a flipped signature byte, a genuine signature from the wrong key, a replayed reply under a nonce nobody is waiting for, a held reply pastMAX_ROUND_TRIP, a small-order key, and a genuine signature made under gunnar's domain tag. None of them is a fallback — there is deliberately noTimeErrorvariant meaning "so the RTC was used instead".the_published_signature_still_exists_and_delegates— compile-time function pointers at every pre-W13 signature, so a parameter added or retyped stops the file compiling; plus the delegation target is asserted to declare itself untrusted.no_skew_window_grewa_signed_source_advances_without_the_rtca_bare_instant_is_recorded_as_unnamed_not_as_trusted
What is NOT done
ExpiredPosture::{ReadOnly, Refuse} (?5, answered 2026-09-20) still has no SERVE-time consumer. W13 gives that consumer a clock worth reading; wiring it is the next step and is not in this lane. SignedTime has no transport in holger — accept() verifies a reply, and nothing yet opens the TCP exchange gunnar-clock's peer.rs opens. That is deliberate: the verifier is the part that must be right, and a transport with no verifier would have been the wrong half.
4. the audit query seam — 2026-09-21
Appended to §4 ("ANSWERED — kept, never deleted"), never renumbered.
.nornir/stinky/05-suite-monetize-mvp.md §A.3 rules that the one honest product→product edge between holger and nornir/skade is the audit QUERY: a lakehouse reads Arrow, holger writes Arrow, and holger should not grow a query engine. This is what was MEASURED and what was built.
What AuditService.Query can and cannot answer today
MEASURED against proto/holger.proto AuditQueryRequest/AuditSentToRequest and server/lib/src/audit/query.rs AuditQuery::matches, 2026-09-21.
| the operator's question | today | how, and what it cannot do |
| by IDENT | CAN | ident, EXACT string. No prefix, no set, no negation — "everyone except anonymous" is not expressible |
| by ARTIFACT | CAN | artifact_prefix, PREFIX only. acme/ is a namespace, acme/widget@ every version. Not a glob: no suffix match, no contains, no version range |
| by REPOSITORY | CAN | repo, EXACT. One repository per call; "these three" is three calls the caller merges |
| by TIME RANGE | CAN | [since_nanos, until_nanos), half-open so consecutive windows tile. ⚠ in Aggregated mode a row is a FOLD whose ts is the LAST call in it, so a window takes a whole fold or none — the resolution is the flush period, not the call |
| by ACTION | CAN | exact spelling; an unknown one is REFUSED rather than degraded to other |
| by STATUS | CAN | exact code. No ranges — "all 4xx" is not expressible |
| by SOURCE IP | CANNOT | AuditQueryRequest has no such field at all. SentTo can GROUP by address (group_by = "source_ip"), but only over SUCCESSFUL DOWNLOADS and only per artifact — it cannot filter the log, and it cannot answer over uploads, denials or errors. Reads are open by default here, so this is the question that separates one anonymous puller from forty thousand |
| AGGREGATED COUNTS | PARTLY | matched is ONE number for the whole filter. SentTo gives a per-(ident|source_ip) × artifact tally with fetches_served, bytes_sent, last_sent_nanos — downloads-only, by design, so an aggregate named "sent to" can never be widened to count uploads |
| GROUP BY anything else | CANNOT | no group-by on Query. Per-repo, per-action, per-day, per-status counts do not exist on the wire |
SUM / AVG over bytes |
CANNOT | except SentTo.bytes_sent on the downloads-only path |
| TOP-N, ORDER BY | CANNOT | the page is in log order; there is no sort key |
| OR / NOT / IN | CANNOT | every set field ANDs. There is no disjunction anywhere |
| HISTOGRAM / time bucketing | CANNOT | no bucketing primitive; a caller issues one windowed call per bucket |
| DISTINCT values of a column | CANNOT | "which idents have ever pulled" needs a full page-through |
| JOIN with anything | CANNOT | no repository config, no artifact metadata, no SBOM, no build provenance. The audit log is an island |
| ACROSS SERVERS | CANNOT | one box, one directory. A fleet question is N calls the caller merges |
| PAGING | CAN | offset + limit, default 100, clamped at 1000, limit_applied reports what was used, more says a truncation is a truncation |
| COST | ⚠ | every query REPLAYS the whole segment directory: scanned is the size of the log, not the size of the answer. No index, no statistics, no pushdown. This is fine for an appliance's own console and is the wrong shape for analysis |
The shape of the "cannot" column is the finding: it is not a list of missing filters, it is a list of things a QUERY ENGINE does. Adding them to holger means putting a second, weaker engine inside an appliance that has no shell.
The export layout — holger.audit.export.v1
server/lib/src/audit/export.rs. Read-only, and structurally so (the_export_module_never_opens_a_writer holds it against the source text, as query.rs does — this is the read side of a tamper-evident chain and a path that could write, rotate or re-emit would be a way to launder it).
<audit_dir>/
audit-<ts_nanos>-<pid>-<seq>.arrows one Arrow IPC STREAM segment
- Format — Arrow IPC stream. Not the file format, not Parquet. One
RecordBatchper audit event. - Schema — the ten columns of
audit_schema(), none nullable, carried in every stream's own header so a reader never needs one on the side:ts(timestamp ns),ident,action,repo,artifact,source_ip,status(u16),bytes(u64),detail,chain. - Partitioning — by segment, and a segment is a PROCESS LIFETIME: one holger boot opens one and only appends. There is no
dt=/repo=directory scheme, because an append-only log cannot have one without a writer that moves files. - Ordering — write order within a segment (non-decreasing
ts); names sort in creation order across segments. A reader wanting a global order sorts ontsrather than trusting a directory listing. - Immutability — a listed segment changes only by growing at the end while its own process lives. A reader may resume from a row count; it may not assume a closed file.
- Integrity —
chainis the per-record blake3 link digest (64 hex chars) and each stream header carries the segment's seed. A reader verifies it with no holger code. A segment without a chain is reported aschained: false: readable, integrity UNPROVABLE, never implied to be a pass.
The manifest (ExportManifest) names every segment with its rows, bytes and CLOSED time range. It is computed on demand, never stored, so it cannot go stale and cannot be tampered with independently of what it describes. segments_overlapping(since, until) is the whole of the pruning story and deliberately the whole of it: time is the one dimension an append-only log is naturally clustered on, so it is the one that can be pruned honestly. Everything else is a scan, and a reader with a real engine does that scan far better.
The door: AuditService.ListSegments (proto/holger.proto, grpc/services.rs). Admin-gated, behind the same audit_source refusal as Query — a server that records nothing refuses rather than returning an empty list, because an empty list reads as "this box served nothing". A windowed listing's totals describe the WINDOW; a total that silently described the whole log would make a partial pull look complete. An appliance has no shell, so this call is the only way a reader can find the segments at all.
The proof
server/lib/tests/an_exported_audit_segment_answers_a_real_question_through_a_reader.rs. A real holger boots from RON with audit on, two raw repositories serve five real HTTP downloads through the ordinary request path, and then a reader using NO holger reader — arrow_ipc::reader::StreamReader and nothing else — opens the segments by the names the manifest gave it and answers two questions from the CANNOT column: bytes-and-calls per source address, and bytes-and-calls per repository. Both exact (3 × 1024 B from a, 2 × 8192 B from b).
skade is not used, deliberately: it would be a new holger→skade edge existing only for a test, in the exact direction this seam exists to keep empty. The claim is "any engine that reads Arrow can answer these", and plain Arrow demonstrates it more strongly than a sibling crate would. The three Arrow dev-deps are at the versions the library already pins — nothing entered the graph, Cargo.lock is unchanged.
What a lakehouse still CANNOT ask
Honest limits of what shipped, so nobody reads the seam as bigger than it is.
- There is no bulk transfer.
ListSegmentsnames the segments; it does not serve their bytes. A reader needs filesystem access, a mount, a sync, or a transport somebody still has to build. For an airgapped appliance that is arguably correct — the segments leave on the same medium everything else does — but it means "a lakehouse registers holger's audit log" is today "a lakehouse registers a COPY of it". - There is no incremental cursor. A reader can re-read from a row count, but nothing tells it which segments changed since its last manifest. It re-scans the manifest and diffs
rowsitself. - The manifest has no per-column statistics. No min/max on
repo, no bloom filter onident. Pruning is time-only, so a question scoped to one repository still pulls every segment in the window. - A segment is a process lifetime, not a time bucket. A long-lived server writes one enormous segment, and time pruning then prunes nothing. Rotation would fix it and rotation needs a writer touching the chain, which is exactly what this module refuses to be.
- Aggregated mode is still lossy in a window. A fold's
tsis its last call; a reader hits the same limitQuerydoes, and the manifest cannot repair it because the individual timestamps no longer exist. - The chain is verifiable but not verified by the export.
chainedsays a chain is present; it does not say it checks out.audit verifyis the verb that answers that, and a reader that wants the guarantee runs the rule itself. - Nothing joins. Repository config, artifact metadata, SBOMs and build provenance are not in the export, so the lakehouse's own joins are the only ones there are.
- One box. A fleet question is still N appliances, N manifests, N pulls, merged by the reader. Nothing here federates.
The installer half, 2026-09-21 (lane H2) — what a fresh holger actually comes up as
The owner's words were "holger.rs is not operational, they are EMPTY". Lane H1 took the live box and its repositories; this is the INSTALL path. Nothing here is argued — every line was measured against the shipped 2026-09-20 binaries (/home/rickard/holger-media/1cdba57/) and the estate's own emitted config.
There is no panic. There is SILENCE, which is worse. The recorded claim was that the emitted RON panics the appliance at first boot. It does not. Run verbatim against deploy/generated/server/holger.ron:
$ holger-server-full start --config deploy/generated/server/holger.ron
Starting Holger "…/deploy/generated/server/holger.ron"
[ERROR] repository 'docker' DISABLED — backend init failed: creating
/var/lib/holger/store/docker/blobs/sha256: Permission denied (os error 13)
[WARN ] 1/7 repositories disabled (backend init failed); serving the remaining 6
Holger is running. Press Ctrl+C to stop.
HTTP/OCI gateway listening on 127.0.0.1:8443 [http (cleartext)]
gRPC server listening on 0.0.0.0:50051 [h2c (cleartext)] (gRPC-web enabled)
gRPC server error: transport error
Four separate defects are visible in those nine lines, and three of them are the same shape — a door that did not open, reported as a door that did.
1. The admin token nobody writes — FIXED (39dcfad). The estate emits StaticToken(subject: "console", token_file: Some("/var/lib/holger/admin-token")) into all three generated RONs and no code path in any repository ever creates that file; private-holger-ops/xtask/src/front.rs:246 and :363 are printed plan steps asking a human to write 32 bytes ON THE MACHINE, and two of the estate's three machines declare shell = false. The shared ladder's contract is that an empty secret DISABLES the method fail-closed — correct, and reached in total silence. auth::refuse_disabled_static_token now refuses at boot, naming the subject and which of the four absences it is (no source declared / inline empty / env unset / file missing / file blank). 8 tests, all RED before.
2. The gRPC door that failed to bind and said "listening" anyway — OPEN. server/lib/src/lib.rs:3341 prints gRPC server listening on … BEFORE the listener exists, and :3346 swallows the bind error with an eprintln! inside a spawned task, so the process stays up serving nothing on that door. The line above was produced by a real port conflict. Fixing it properly wants a listener-first serve entry in edda's nornir-service (tonic's serve(addr) cannot be bind-checked without a TOCTOU), and edda's checkout is another lane's branch tonight — so it is written down, not bodged.
3. The installer pin was a caret — FIXED (b84c5c5). xtask/Cargo.toml required korp-installer = "0.1.7" — ^0.1.7 — and the lock had resolved 0.1.13. Six patch releases of the crate that writes the partition table, elects the boot disk and stages the per-site declarations, taken with nothing comparing the two numbers, under a manifest comment still claiming 0.1.7 was what edda main carried. holger's shape is NOT gunnar's three-manifests-three- lockfiles: exactly one manifest, one lock entry, and installer_pin.rs asserts that too. The requirement is now =0.1.18 (moved from =0.1.13 on 2026-09-23: edda main and the gunnar estate both carry 0.1.18, and holding holger at 0.1.13 dragged the DWARVES lock backwards through its sibling resolution, so a --locked dwarves build could not resolve and nothing could be released from it) and two tests hold it against the lock. ⚠ The path is what cargo actually takes, and nothing can pin a working tree — what the exact requirement buys is a build-time refusal naming both versions instead of a silently different installer.
4. The boot-disk election is the RIGHT branch. Checked, not assumed: korp_installer::layout::pick_boot_declared is the corrected rule (/data first, boot = the remainder; class-then-LARGEST only where no equal-size group exists; a disk DECLARED /data is never a boot candidate whatever is written on it), and holger's installer row declares the data_disks param that drives it. No change needed.
?47 — may the installer medium carry the admin token?
BLOCKED ON A RULING, and the block is a deliberate refusal in edda. The medium is the only channel a shell-less appliance has, and it already carries per-site declarations, so the obvious fix is a staged_file param landing the secret on P3 as /data/admin-token. It was written and REFUSED by the declaration door itself:
param admin_token: parameter 'admin_token' NAMES A SECRET (token) — secret
values may not travel as build parameters: the cmdline, the declaration and the
build record are all readable
That rule is right about the build record: an applied param value rides the attested descriptor. So there are three ways out and all three are the owner's:
| what it costs | |
(a) let a staged_file param carry a secret, excluded from the descriptor |
a hole in a rule that currently has none, in edda, for every product |
| (b) the appliance MINTS the token at first boot, as korp-installer already mints the wg keypair, and never prints it | the operator must reach the box to read it — over the wg tunnel, which is the channel it already has |
| (c) the token arrives out of band, and the appliance REFUSES to serve until it does | honest, and it is what (1) above now does — but it means no unattended install ever comes up with a working console |
(b) is the recommendation: it is the same mechanism as the identity keypair, one level of the same ceremony, and it leaves nothing secret in the build record. It is edda's code, not holger's.
?48 — the emitted RON is the CLOUD box's, and the appliance's home is /data
private-holger-ops emits ONE holger.ron for server, twin and front, byte-identical, with every path under /var/lib/holger and the artifact door on 127.0.0.1:8443. The installed appliance's durable home is /data — the installer row's own cmd is start --config /data/holger.ron — and there is no Caddy in the korp shape, so a loopback artifact door is a door nobody can reach. ⚠ The estate's own plan step 8 says the medium carries the RON "to /var/lib/holger/holger.ron", which the medium cannot do: a staged_file lands on P3. The declaration and the plan disagree by a path, and neither matches the other repository. Not fixed here: estate.rs had uncommitted changes from another lane while this was measured, and a second writer in a file being edited is how the three-writer drift this plan already records got started.
?39 — ANSWERED 2026-09-21: STALE. Every declared row is openssl-free.
Measured before the priority change, with cargo tree -i openssl-sys --edges normal --target all per row against the declarations' own feature sets: server-local, full and agent-online — the three that failed — each answer package ID specification openssl-sys did not match any packages, and so do mvp-server and xtask (the bench driver the warning named). The finding was true when taken and was closed by three other rows landing: W1 (the bench crate to [dev-dependencies]), W2 (FORBIDDEN_TLS_BACKENDS judged FIRST, before the tier split) and ?37 (nornir off xtask's edge). openssl-sys survives in the lock only through mannequin's cert-helper and through webauthn-rs behind nornir-auth-webauthn's non-default webauthn feature — edda had already defended that one by name. Neither option in the question needs to be taken: no row moves to rustls because none is on OpenSSL, and libssl-dev does not enter the pinned image.
⚠ The consequence of a refusal on a machine where holger is PID 1
Both refusals landed tonight (auth::refuse_disabled_static_token, holger-door-1) end the process. On the cloud box that is a systemd restart loop in the journal — visible, correct, and what a supervisor is for. On the INSTALLED APPLIANCE holger is PID 1, so ending is a kernel panic, and estate.toml gives server and twin shell = false. That is still better than the silence it replaces — a panic is on the console and in /boot/BOOTLOG.TXT, while "authenticates nobody" is nowhere — but it is a real change in failure MODE, and it is the strongest argument for ?47 option (b): if the appliance MINTS its own token at first boot the way it already mints its WireGuard identity, the refusal can never fire on a fresh install at all.
4. holger.rs made operational — 2026-09-21
Appended to §4 ("ANSWERED — kept, never deleted"), never renumbered.
The box was RUNNING and EMPTY: holger.service, holger-ui.service and caddy.service all active, the console served, /health 200 — and /var/lib/holger 4.3 MB total, store 20 KB of test leftovers, five repositories of which four were raw shelves and the fifth (crates) had no data directory at all. Empty of purpose, not broken.
The 401 that was not a contradiction
require_auth_for_reads: false and a 401 on https://holger.rs/api/v1/health read as a contradiction. It is not one: the two names are two different programs. deploy/generated/server/Caddyfile sends holger.rs to 127.0.0.1:3210 — that is holger-ui-server, the console BFF, which gates on the admin bearer and answers log in to use the console — and repo.holger.rs to 127.0.0.1:8443, which is holger's own repository door. MEASURED:
https://holger.rs/api/v1/health → 401 "log in to use the console" (holger-ui)
https://repo.holger.rs/v2/ → 200 {} (holger)
https://repo.holger.rs/api/v1/health → 404 "Unknown repository" (holger's router)
The flag is honoured. Everything a client pulls is on repo.holger.rs, and a probe aimed at the apex measures the console.
What is operational, and the command that proves it
Against the LIVE host, from a workstation, with real clients — not a unit test.
| format | repository | proof | result |
| cargo, pull-through | crates → crates-io |
cargo fetch with sparse+https://repo.holger.rs/crates/index/ replacing crates-io |
12 crates (rand and its graph) downloaded through the box |
| cargo, publish | crates |
cargo publish --registry holger --token <admin-token> |
Published h1-audit-hello v0.1.1 at registry holger |
| cargo, round trip | crates |
a second project depending on it, cargo run |
hi from holger.rs — resolved, downloaded, compiled and run from the box |
| OCI, push | docker |
podman login -u console, podman push repo.holger.rs/docker/alpine:3.20 |
Writing manifest to image destination, exit 0 |
| OCI, anonymous pull | docker |
podman logout, podman pull repo.holger.rs/docker/alpine:3.20 |
pulled; GET /v2/docker/alpine/tags/list → {"name":"alpine","tags":["3.20"]} with no credential |
| raw, hosted | bundles dev prod migtest |
unchanged, already serving | — |
The store on the box went from 20 KB of leftovers to a real crates.znippy + crates.znippy.hot.00000 and an OCI tree under docker/.
What had to be fixed to get there
The doors read Bearer and nothing else (merged to main as agent/door-basic-auth). docker login, podman push, npm publish, mvn deploy, pip and nuget push send Basic base64(user:pass) and have no Bearer mode; cargo publish sends the token VERBATIM with no scheme at all. A correct credential, presented the only way those clients can present it, was refused. credential_of now reads all three spellings into one string.
The /v2/ probe never challenged. A distribution client pings it once and that answer decides whether it will EVER attach a credential. Answered 200 with no WWW-Authenticate, podman filed the registry as wanting nothing and never sent the credential podman login had stored — not even after the upload POST refused it. Fixing the parsing alone changed nothing; this is why both are in the same merge. Anonymous pulls are untouched, and the proof is the logged-out podman pull above.
The estate spec could not express a proxy (private-holger-ops). Repo carried name, type and storage, so every repository was emitted with ron_upstreams: [] and a plain data directory. It grew upstreams, writable_archive and base_url, and storage became optional because an upstream holds no bytes on this box.
What is still empty, and why it is not a to-do
MEASURED against server/lib/src/repository.rs 2026-09-21. Only five repository types come up from config alone: rust (writable archive or data directory), raw, oci, group, and the remotes (rust-remote plus the four federation harbours). Every other format — npm, nuget, maven3, pip, go, gem, deb, rpm, helm, conda, composer, the znippy docker backend, znippy, znippy-build-thing — REQUIRES a pre-existing .znippy archive that no part of the config creates, and each is read-only at the trait level. A declared one is DISABLED at boot with a message about ron_archive_path.
And holger has no generic proxy for npmjs, Maven Central, PyPI or Docker Hub. crates.io is the one public origin in the server (upstream_rust.rs); the four *-remote harbours talk to somebody's Nexus/Artifactory. For every other format the shape is: the offline agent fetches, packs a .znippy, and holger serves it read-only. So an npm repository on this box today would be a name on the roster that nothing can ever pull through, which is why none was declared.
⚠ The cargo proxy does not CACHE. crates.znippy holds the published crate; the 12 crates pulled through the upstream are fetched from crates.io on each resolve. The write-back that build_proxy_for wires is not landing them, and until it does this box is a passthrough, not a mirror — which matters for the airgap posture, because a passthrough is worth nothing when the upstream is gone. Not fixed here: it is a lane of its own.
The owner's questions this leaves
- Should the paid formats be filled by the agent? Making
npm/maven/pypireal on this box means running the offline agent against those registries and installing the resulting archives — a data job, not a config job, and nobody has said this box should hold them. - The console token is the only credential. Every publish above authenticated as
consolewith/var/lib/holger/admin-token. There is one token and it is the admin's; a second principal needs a second method.
The KVM proofs, 2026-09-21 (lane H2) — run on t14s, not argued
Composed by the verb from a real clone (/home/rickard/compose-clean/holger-h2, siblings beside it), payload built through the declared door (cargo xtask build --variant mvp-server-musl --suite MVP, 55 728 552 B, sha256 2113b6bd…), medium holger-installer.iso 64 376 832 B, unenrolled (KORP_ROOT_SIGNING_KEY unset), QEMU/OVMF, -serial capture, no IPv6.
| arm | verdict |
installer_iso_kvm::the_per_site_iso_installs_and_the_disk_carries_the_site |
ok, 74.4 s — INSTALL-OK, the site on the installed cmdline, holger.ron byte-identical on P3 |
installer_iso_token_refusal_kvm (new) |
ok, 28.5 s — the appliance REFUSES, by name, on its own console |
installer_iso_fleet_kvm::served_content_survives_a_power_cycle |
RED first, then ok in 230.4 s — the red is the payload defect below |
The refusal, off the guest's serial (firstboot/serial.log:510) — this is the line that used to be Holger is running.:
INSTALLER: exec argv: ["holger", "start", "--config", "/data/holger.ron"]
Starting Holger "/data/holger.ron"
Error: auth method StaticToken(subject: "console") is DECLARED but its secret
resolves EMPTY, which disables the method fail-closed — it would authenticate
NOBODY while this server reported itself healthy. `token_file:
"/data/admin-token"` does not exist. … REFUSING TO START.
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
And the ISO could not be composed at all before tonight. installer-iso refused before its container started: holger's [build.toolchain] siblings were measured for building HOLGER, but the verb runs a SECOND container build — korp-boot, for x86_64-unknown-uefi, rooted at edda — whose crates escape into funnel-core and skade. One declaration, two escape sets; both are in it now.
Then the fleet arm went red for a different reason, and it is the better finding. pack_fresh_app_znippy hand-spelled --features mvp-server,appliance while the declared mvp-server-musl row carries seven more, so the payload served no formats at all:
Error: [holger-mvp-1] repository 'bundles' is `ron_repo_type: "raw"` but this
server was built without the `fmt-raw` feature; the formats this binary serves
are [] — rebuild with `--features fmt-raw` (or `fmt-all`)
Kernel panic - not syncing: Attempted to kill init!
A second writer of a build argv, inside the proof whose whole job is to catch "boots but does not serve". It now takes the row's own cargo_args() — the same call gate::rows_check makes — appliance-pack's "build it first" advice points at the declaration instead of a feature list, and holger-recipe refuses a payload row declaring no fmt-* feature. After the fix the unattended install comes up SERVING: gateway bound, a blob PUT through it, the machine power cycled, the same bytes GET back off the installed disk.
5. W22 — A PAIRED ON-PREM HOLGER OVER WIREGUARD — 2026-09-22, PROPOSAL ONLY
Status: design, no code. What IS built (branch agent/flow-paired) is the pairing itself, over the ordinary TLS door: server/lib/src/peers.rs, the pair downstream | upstream | list verb, the two records, the pin check, and both sides of the picture. WireGuard is the next layer under it and nothing here is implemented. Anyone picking this up should read crate::peers first: the pairing is the identity, the tunnel is only where the bytes travel.
The shape
A paired downstream holger joins the cloud holger's WireGuard network as a peer, and the mirror then runs over the tunnel instead of the public address. The pairing is unchanged — the certificate pin is what says WHO — and the tunnel is what says WHERE and keeps the artifact door off the public interface entirely.
Enrolment — gunnar's hub + tunnel enrol, in holger's vocabulary
gunnar's model (measured on agent/t19-one-medium-twin, 19133ff), which this follows rather than invents:
- The box mints its own key on first start. x25519, through the shared
wg_appliance_core::ceremony:<data_root>/wg/identity.key,identity.pub, the site-widehub.conf(hub public key, endpoint,AllowedIPs, keepalive) and<data_root>/wg/address. The private key never leaves the box. holger already carries the appliance-to-hub ceremony (server/cli/src/appliance.rs); this is the same ceremony used for a peer. - An operator enrols it, once, from the control side. gunnar's verb is
gunnar control tunnel enrol --at <box> --pool <cidr> --ledger <file> --hub-peers <hub wg conf>: ask the box for its public key, allocate an address from the pool under a.lock, tell the box its address, then append a[Peer]stanza to the hub's configuration. The ledger is one plain line per box (a.b.c.d/p <wg pubkey> <label>), keyed by public key, so repeating the command is idempotent and a taken key or address is refused. holger's verb would beholger-server tunnel enrol, writing the same ledger shape. - The pairing is made over the tunnel address, not the public one. The
pairverbs do not change: the downstream still presents the same client certificate, the upstream still pins it. Onlyron_endpointmoves onto the tunnel (https://10.x.y.z:50051).
Which listener binds wg0
The mirror door only — one exposed endpoint, bound to the box's WireGuard address, carrying the gRPC RepositoryService the mirror uses. Not the public HTTP gateway, which stays where it is for cargo/pip/docker, and not the console. Concretely, a second exposed_endpoints entry whose ron_url is the wg address, with ron_tls still required (mTLS inside the tunnel, belt and braces: the tunnel says which machine, the certificate says which holger). The twin's effective_bind rewrite is the precedent for a door whose address is decided by what the box IS rather than by the file.
What the flow shows
- The mirror edge gains
via wg0in its label:paired · every 5m · via wg0on the downstream, and the upstream'spaired · mirrorarrow readspaired · mirror · via wg0. The word comes from the endpoint's declared interface, never from a guess. - The paired-downstream row gains the tunnel address as a FACT (it is configuration, not a visitor's address — the privacy rule in
crate::visitorsis about clients this box did not enrol). - A paired downstream whose tunnel is down reads
unreachableexactly as it does today: the pin check cannot complete, and that is already a named state.
Open questions for the owner
- Who runs the hub — the cloud holger itself, or a separate box as in gunnar?
- One pool per estate, or one per customer?
- Does the public artifact door stay open at all on a box that has a tunnel?