nordisk/monetize

holger.rs's control plane is cleartext on the public internet, and the estate

says it is not

MEASURED 2026-09-23 by the monetize lane, read-only, while giving monetize's outward doors TLS. Nothing was changed on the box. The fix is the owner's to route — this is the evidence and the recommendation.

What the estate says

# private-holger-ops/estate.toml:330
grpc = { bind = "0.0.0.0:50051", public = "grpc.holger.rs", via = "direct" }
#      ↑ "gRPC on its own port and NOT through Caddy: holger-server terminates
#        its own TLS there (mTLS is one of its authentication methods, and a
#        proxy that terminated TLS would take the client certificate away)."

# estate.toml:308
"tcp/50051 any", # gRPC — holger-server's OWN TLS and its own auth …

# estate.toml:620
server_endpoint = "https://grpc.holger.rs:50051"

Three statements: the port is open to the world, holger-server terminates TLS on it, and mTLS is one of its authentication methods.

What the box does

No TLS. openssl s_client -connect grpc.holger.rs:50051 reports

no peer certificate available
tls_get_more_records:packet length too long

and a rustls client gets received corrupt message of type InvalidContentType — both are a TLS client meeting a plaintext server. A plaintext h2 gRPC call to the same port succeeds and returns real data.

(The earlier reading that this port had a valid certificate came from Verify return code: 0 (ok), which openssl prints even when no certificate was received. That line fooling a reader is exactly how a box comes to be believed encrypted when it is not.)

No mTLS. The emitted appliance config carries one authentication method and it is a static bearer:

# deploy/generated/server/holger.ron:31
auth: Some((methods: [StaticToken(subject: "console", token_file: …)],
            require_auth_for_reads: false)),

So the stated reason for keeping gRPC out of Caddy — a proxy would take the client certificate away — describes a client certificate that no client presents.

Reads are open to anyone who can reach the port, and the port is open to everyone. From an ordinary developer machine, with no credential:

tenants: bundles, crates, dev, docker, migtest, prod
crates  {artifacts: 1, artifact_bytes: 777, archive_bytes: 802, archive_files: 2}
docker  {artifacts: 1, artifact_bytes: 899}

The write credential travels in that cleartext. require_auth_for_reads: false means the bearer is only sent when something is written — and when it is, it is sent in the clear, over the public internet, to a port open to any. Anyone on the path has the console's write credential.

How the console reaches it today, which is the question that decides the fix

holger-ui-server is given HOLGER_UI_HOLGER=https://grpc.holger.rs:50051 (deploy/generated/web/holger-ui.env:25, emitted from estate.toml:620) and it works: https://holger.rs/api/public/releases answers with the six repositories, which can only have come from holger-server over gRPC.

It works because the scheme is ignored:

• holger-ui/crates/holger-ui-server/Cargo.toml:30 — tonic with ["codegen", "transport", "router"] and no TLS feature;
• holger-ui/crates/holger-ui-server/src/grpc.rs:214 — connect_lazy keeps the https:// prefix and never calls tls_config.

A tonic built without TLS dials the port in plaintext whatever the URI says. So the console is not "configured with a different address", and nothing in front terminates TLS: the declaration is decoration, and the one component that reads it cannot honour it.

That also means monetize was the only thing on the estate that ever told the truth about this address — it REFUSED https:// by name rather than quietly dialling plaintext, which is how the discrepancy surfaced at all.

The trap in the obvious fix

Giving holger-server a certificate on 50051 makes the declaration true and takes the console down the same minute: holger-ui-server cannot speak TLS, so every call it makes would fail. Any sequence that starts with the box is a sequence that starts with an outage.

A third finding, one line: the gate reads a stale state

private-holger-ops/terraform/terraform.tfstate is gitignored, dated 2026-09-17, and holds ONE front server while estate.toml has declared three machines since before e73a5ff — so three terraform-gate tests answer questions about an estate that no longer exists, and have been red for days without meaning what they appear to mean. (The true state is on loki.) A gate that reads a stale state is worse than no gate: it fails for the wrong reason, and the day it passes will be a coincidence.

Recommendation, in this order

1. Correct the declaration now, because it is false. estate.toml:620 → server_endpoint = "http://grpc.holger.rs:50051", and the comments at :308 and :327-330 stop claiming TLS and mTLS that are not there. This changes no behaviour — the traffic is already plaintext — and it stops the file telling every future reader that the control plane is encrypted. A declaration that is believed is worse than one that is absent.

2. Close the port. estate.toml:308: tcp/50051 any → the addresses that actually dial it. The console is on the estate's own network; monetize reaches it from the front. A cleartext control plane with open reads should not be on the public internet at all, and this is the change that costs nothing and removes the exposure today.

3. Then make TLS real — and after the ruling above, this step is NOT what it first said. It is no longer "give holger-ui TLS, then a certificate on holger-server's own 50051". It is:

   1. a terminator in front of the appliance's gRPC door (Caddy already terminates for holger.rs and repo.holger.rs on the same box), serving grpc.holger.rs:50051 over TLS and forwarding plaintext h2c to holger-server;
   2. the console keeps dialling plaintext, on the box it shares — it needs no TLS, no ring, and no change beyond the one already made (622e172: a https:// endpoint is refused by name instead of dialled in the clear, and the start line says which);
   3. then flip estate.toml:620 back to https://, because by then it is true. monetize already speaks TLS and needs no further change;
   4. and re-read the mTLS condition above before doing any of it.

4. Rotate the console's bearer afterwards — see the next section. It has been sent in cleartext over a public port for as long as this has been deployed.

What is NOT recommended

Moving gRPC behind Caddy (via = "caddy"). It is tempting — Caddy already terminates TLS for holger.rs and repo.holger.rs — and the estate's stated objection (mTLS) does not currently apply. But it is the largest change of the four, it breaks the day mTLS is configured, and steps 1–2 remove the exposure without it.

Item 3's code half is done, and it cost nothing — but TLS itself needs a ruling

holger-ui 622e172 (branch lane-w1/console-tls, not deployed): the console now refuses a https:// endpoint by name instead of dialling it in the clear, states tls = true|false on its start line, and leaves http:// and bare addresses exactly as they were. The lock is unchanged — no new package, no new edge — and 151 tests pass.

The TLS feature itself is not there, and that is a measurement. Adding tls = ["tonic/tls-ring", "tonic/tls-webpki-roots"] — even OFF by default — puts tonic -> tokio-rustls -> rustls -> ring into Cargo.lock, and holger-ui-server/tests/no_c_crypto.rs walks the LOCK as a graph, feature-blind by design. Measured: with the feature declared and off, the suite was 137 green and this_binarys_graph_carries_no_c_crypto RED. The law is not decoration — the passkey plane is ES256/P-256 over RustCrypto so that the one process that is the way in links no C TLS stack.

So the order we established has a fork in its first step, and it is the owner's:

1. the law bends — widen FORBIDDEN, accept ring in the console;
2. rustls gets a RustCrypto provider — a new, early-stage dependency in a security-critical place;
3. TLS is terminated in front of the appliance — the console keeps dialling plaintext on the box it SHARES with holger-server, and external clients get TLS. This costs no crypto in the console at all, and the measured topology already points at it: all three names are one machine.

RULED 2026-09-23: (3), terminate TLS in front of the appliance. Taken by the coordinating lane rather than parked, and flagged to the owner as theirs. The reasoning, so the question is closed rather than left open:

• The topology decided it. All three names are one machine, so the console's call to holger-server never crosses a wire. Plaintext between two processes on one box is not the exposure; the public port is. A terminator in front closes exactly what is open and nothing else.
• (1) is the worst trade available. no_c_crypto exists so the one process that is the way in — the passkey plane — links no C TLS stack. Widening it to reach a box on the same machine spends a real property to solve a problem that is not there.
• (2) puts an early-stage crypto provider in the security-critical position, to serve a hop that needs no crypto at all.

⚠ THE CONDITION UNDER WHICH THIS RULING IS WRONG. The estate's own stated reason for keeping gRPC out of a proxy was mTLS — a proxy that terminated TLS would take the client certificate away. That objection does not apply today because no client presents a certificate (measured: holger.ron declares one method, a static bearer). The day mTLS is actually configured, (3) is wrong and this must be revisited. Whoever configures it inherits this decision.

What monetize did about it, which is nothing to the box

• Both monetize builds now dial https:// over rustls against the compiled-in Mozilla roots, or a CA named by [products.<id>] tls_ca. Zero new packages.
• guests/holger/tests/against_the_live_holger_rs.rs holds the measurement: it is #[ignore]d, it names this document's finding in its failure message, and it turns green by itself the day holger-server answers TLS on 50051. That is deliberate — a test that goes green on its own is worth more than a note in a file nobody opens.

Item 2, prepared and NOT applied: closing tcp/50051

Prepared 2026-09-23. The apply is the owner's to route. Everything below is evidence or a named gap in the evidence; nothing here is reasoning dressed as a fact.

What the evidence says, with no credential

1. The three public names are ONE machine. holger.rs, repo.holger.rs and grpc.holger.rs all resolve to 95.111.208.165.
2. The three-machine estate is declared, not applied. private-holger-ops/terraform/terraform.tfstate (local copy, 2026-09-17, serial 4) holds exactly upcloud_server.front, upcloud_storage.front_data and upcloud_firewall_rules.front — one server — while estate.toml declares web · server · twin. The two terraform-gate tests that assert three machines are red for this reason and were red before this lane touched anything.
3. So the console and holger-server are on the SAME box. The BFF dials grpc.holger.rs:50051, which is that box's own public address; the kernel delivers it locally. No firewall rule is needed for the console at all, and estate.rs would refuse one anyway — a machine-scoped rule open to itself is "a rule with nothing on the other end of it" (xtask/src/estate.rs:676).
4. admin is currently the whole internet. terraform/estate.auto.tfvars.json:2-3 — admin_ipv4_start = "0.0.0.0", admin_ipv4_end = "255.255.255.255". Changing any to admin would close nothing until those two values are narrowed. This is the part that would look like a fix and not be one.

What could NOT be established, and why

Who has actually dialled 50051 over the retained window. That is AuditService.Query, and it is gated by authorize_admin_read (holger/server/lib/src/grpc/services.rs:2599) — the same gate that refused Admin.Capacity with "[holger-auth-1] valid credentials required to read the audit log". The credential is the static bearer at /var/lib/holger/admin-token, on a machine declared shell = false.

I did not take that credential and did not run that query. The owner (or anyone holding the console's token) completes the evidence with a read-only call — AuditService.Query over the same plaintext port, filtered to the retained window, reading source_ip off each record. Two caveats that must travel with the result, or it will be read as more than it is:

• Reads are unaudited on this deployment (require_auth_for_reads: false, and the flow door is documented as unaudited because it serves aggregates). An audit-log enumeration therefore shows who WROTE, not who dialled. The cargo/pip/docker pulls do not appear at all — they arrive on :443 through Caddy, not here.
• A window that has rolled over shows nothing for a client that has been quiet, and "quiet" is not "gone".

The allow-list as it stands, and what breaks if an entry is missing

estate.toml:323, today:

"tcp/50051 any", # gRPC — the product's public door …

The change that actually closes the door is two lines, not one:

estate.toml:323                       "tcp/50051 any"  →  "tcp/50051 admin"
terraform/estate.auto.tfvars.json:2-3 admin_ipv4_start / admin_ipv4_end
                                      0.0.0.0–255.255.255.255  →  the operator's range

and estate.auto.tfvars.json is emitted, so the narrowed range is written in estate.toml and re-emitted rather than edited in place.

The risk, stated plainly — and then removed

Narrowing admin narrowed tcp/22 on the same two variables (estate.toml:581): getting the range wrong would close the one shell in the estate at the same moment it closed the gRPC door, on a box with no console access of its own.

That coupling is gone (private-holger-ops 3a81909). The gRPC door has its own declared range, product_ipv4, set to exactly what it had before — the whole internet — so nothing changed on the wire. The rule was a LITERAL whole-internet range in main.tf until then, which is why it had never been narrowed: the only ways to close it were editing HCL by hand or borrowing the admin range, and the second would have taken SSH down with it.

So the decision is now one line and it touches nothing else:

# estate.toml, [estate]
product_ipv4 = ["<first>", "<last>"]   # then: cargo xtask estate emit, and apply

SSH stays where it is, on admin_ipv4, as a separate decision to take separately or not at all. The gate tells the two ranges apart and refuses :50051 scoped to the admin one, with the reason in the failure message — before that, every var. reference read as admin and the mistake would have passed silently.

Item 4, specified and NOT performed: rotating the console's bearer

Specified 2026-09-23. Nothing was rotated, nothing was read off the box, and no token appears anywhere in this document.

What rotates

ONE secret in TWO files, and that is the whole difficulty:

They are two files because two service users read them; they are one secret because holger authenticates one subject (private-holger-ops/xtask/src/front.rs steps 8 and 16 place the same 32 bytes in both). Neither travels through git.

What does NOT rotate with it, and must not be confused with it: HOLGER_UI_ADMIN_TOKEN — the break-glass bearer that enrols the FIRST passkey and retires once one exists. It lives in ~/holger_secrets and is appended to /etc/holger-ui.env. Rotating the wrong one takes out the console's login instead of its credential.

Who else holds a copy: nobody, as far as this repository declares — [products.holger] bearer_token is NOT configured on the deployed monetize, so monetize does not hold it today. Any operator who was handed it for a holger CLI call does, and that is not written down anywhere; the rotation is the moment to find out.

What must be updated in the same minute

The console reads its file ONCE, at start (main.rs::load_credential, called from run()), and holds it in BearerAuth for the life of the process. So writing the new bytes into /opt/holger-ui/holger-token does nothing until holger-ui-server restarts.

The appliance re-reads its file on EVERY authenticated request. Settled from the code, read-only, with no call against the box and no token — the open question in the first draft of this section is closed, and the answer is that holger-server does NOT need restarting:

So the sequence is shorter than the cautious one, and this is the version to follow at three in the morning:

1. write the new bytes to /opt/holger-ui/holger-token — inert, nothing reads it until the console restarts;
2. write the same bytes to /var/lib/holger/admin-token — live on the next request;
3. systemctl restart holger-ui-server — immediately after (2), ideally on the same command line;
4. verify: the console's start line says credentialled=true and tls=false, then perform ONE write through the console.

The window is (2) to (3) and nothing else — as long as a service restart, not as long as a deploy.

⚠ AND THE WRITE IN (2) MUST BE ATOMIC. Because the file is read per request, a request landing while it is half-written reads a partial or empty secret and is refused — and an empty one is the fail-closed case the appliance's own refuse_disabled_static_token describes. Write to a temporary file in the SAME directory and rename(2) it over; chmod 600, chown holger:holger on the temporary file before the rename, never after.

What breaks if the two are not simultaneous

The failure is quiet on the half that matters, which is why it needs saying. require_auth_for_reads: false — so with a mismatched bearer:

• every READ keeps working. The catalog, the artifact lists, the front page, the whole console looks healthy.
• every WRITE comes back Unauthenticated — and so does every migration, promotion and replication RPC. The console's own logs show a credential being sent, which is exactly the shape that sends somebody looking at the network instead of at the token. The emitted env file warns about this state in those words; this is how it is entered.
• Admin.Capacity fails too (it is gated like the audit plane), so the day monetize is pointed at holger its servable goes Unmeasured and no invoice can be drawn from a capacity nobody could read.

The zero-window alternative — now much less likely to be needed, since the window is one service restart rather than two, but recorded because it is the right answer if even that is unacceptable: AuthConfig.methods is a LIST. Declare the old and the new as two StaticToken methods, place the RON, restart, cut the console over, then remove the old method and restart again. It costs two deploys instead of one and holds both secrets valid in between — which is also its risk, and why it is written here as the alternative rather than the plan.

What this does not fix

Rotating replaces a secret that has been sent in the clear over a public port with another secret that will be sent in the clear over the same port. It is worth doing only after, or with, the port being closed and TLS terminated in front — otherwise the new bearer joins the old one on the wire the same day. That ordering is the reason this is item 4 and not item 1.