nordisk/Skidbladnir

kubernetes-target — reasoning (what was considered and rejected)

Companion to kubernetes-target-design.md, which states only the chosen shape. This file holds the alternatives, the why-nots, and the sources the analysis rests on.

The frame Rickard set

"kubernetes has hooks that could solve stuff the deployer does, but it's just another way of solving them — both are right I think."

That framing did the analytical work. Once you stop asking "which is better" the table falls out on its own: for each capability there is a Kubernetes spelling, a deployer spelling, and a question of whether our neutral vocabulary has a word for it. Three of the most interesting findings only appear under that frame:

• Kubernetes declines to offer cross-service ordering. Under a "k8s is better" frame that reads as a gap in us to close; under this frame it reads as a capability we have and they refused, which is the correct reading.
• systemd's StartLimitBurst= has no Kubernetes equivalent. CrashLoopBackOff is not tunable. The init-system family is richer in exactly one place, and it is worth knowing which one.
• The deployer's observed worst failure (a node "sat in booting forever", HANDOFF-NEXT-MACHINE.md:45-51) is precisely the bug class Kubernetes bounds with progressDeadlineSeconds — and which Skidbladnir also does not bound. Two designs sharing a hole is stronger evidence than either one having a feature.

Alternatives rejected

1. Build a reconciliation daemon (an operator for Skidbladnir)

Rejected. The README's load-bearing claim is "no agent, no SSH, no shell on the target". A reconcile-forever loop is an agent. What survives from the operator pattern is the level-triggered idempotency contract — re-render and re-apply must be a no-op when the target already matches — which is a testable discipline rather than a process. ensure_system_user, k8s_apply's server-side apply, and the layout symlink swap are already idempotent; the work is to write the contract down and assert it, not to build the loop.

2. Make lifecycle hooks a systemd-only knob

Rejected, and this was the closest call. ExecStartPre=/ExecStartPost=/ ExecStop=/ExecStopPost= map so cleanly that a systemd-shaped hook type is tempting. But the four-slot model turns out to be honoured by seven of eleven backends (systemd ×2, Quadlet, OpenRC, SysV, and — for pre_start/post_stop — runit and s6-rc), and Kubernetes has an exact semantic twin for three of the four slots. A knob at that coverage is not a systemd knob wearing a hat.

The one place we deliberately did not follow systemd: systemd allows an arbitrary number of ExecStartPre= lines with - prefixes for failure-tolerance. HookAction has no failure-tolerance flag, because Kubernetes has none either ("If either a PostStart or PreStop hook fails, it kills the Container") and a hook whose failure is ignored is a hook nobody can reason about. If it is needed later it is additive.

3. Give readiness a host meaning

Rejected. The temptation is to define host readiness as "the health probe passed", which would make the field render everywhere and look tidy. It would be a lie: readiness only means anything when something routes traffic based on it. On a host, nothing does. Declaring the degrade is worth more than a field that is technically populated and operationally meaningless — this is the same principle that keeps renderer_for's Option.

startup, by contrast, does have a host meaning (a bounded poll before start is reported successful, or Type=notify + TimeoutStartSec=), which is why it is in and readiness is in-but-degraded.

4. Model replicas on the host as "run N copies"

Rejected. It would break the constellation law that every service runs as its own dedicated user, it would require a supervisor Skidbladnir does not have, and it would make Lifecycle::health()'s single 127.0.0.1:<port> probe meaningless. replicas stays k8s-scoped with a declared degrade.

5. Emit a Helm chart instead of raw manifests

Rejected. Njord's deployer is Helm and it works, so this was a real option. Against it: (i) it re-introduces a templating layer between ServiceSpec and the apiserver, and the exact-bytes contract tests in k8s.rs exist precisely because we want the mapping to be inspectable; (ii) it puts a Go binary in an airgap bundle, against L9's spirit; (iii) k8s_apply already talks to the apiserver directly with server-side apply and needs neither kubectl nor helm.

What we take from Helm is the hook tier, which Kustomize deliberately lacks ("its lifecycle is left to kubectl/GitOps"). Skidbladnir's RenderedService is Kustomize-shaped (pure manifests) while install::apply is Helm-shaped (an ordered verb list with a release notion) — we already have both halves and had simply never named the hook points between them (B5 in the design).

6. Render after/needs as init-container wait-gates by default

Rejected as the default, kept as opt-in. The semantics do match — "the kubelet repeatedly restarts that init container until it succeeds" is exactly a wait-gate. But it requires the dependency to be resolvable by DNS from inside the pod, which silently converts a declared build-time ordering into a runtime network dependency. The default is therefore to record the degrade and keep the ordering guarantee where it actually holds — in orchestrate::start_waves, which drives the applier.

7. Add the missing knobs per-backend as they come up

Rejected — this is the failure mode L5 exists to prevent, and the reason B8 (RenderedService::degraded) is sequenced first. Without an observable degrade channel, "add it where it fits" produces a spec whose meaning depends on which backend you happened to render, which is exactly the per-init bespoke install scripts the crate set out to delete.

On reading the Njord deployer read-only

Everything cited in the design chapter comes from /home/rickard/talos-njord/, which is the cluster bring-up bundle plus the Helm values — not the application repo (that checkout is on the T9 volume and was not mounted). Nothing was edited, committed, or pushed there (L10). Secret values were deliberately not quoted; only key names from migrate/old-secrets.env, because the shape of the secret set (flat KEY=VALUE env, referenced not embedded) is the load-bearing fact and the values are not.

Two things the bundle taught that no Kubernetes document would have:

• Physical identity needs a stable key. Talos selects the OS disk by WWID because "sd-letters renumber when the boot-USB is pulled". This is the same lesson as B6's numeric uid: a name that the environment can renumber is not an identity. unfold's Talos generator already gets this right.
• Operational knowledge belongs beside the knob. The single most valuable line in njord-new-values.yaml is the comment explaining that nginx defaults client_max_body_size to 1m and uploads therefore 413. A spec field with nowhere to write down why it is set that way loses that knowledge on the next redeploy.

Sources

Kubernetes: container lifecycle hooks · pod lifecycle · init containers · Deployment · StatefulSet · DaemonSet updates · ConfigMap · Secret · ServiceAccount · Operator pattern · Disruptions / PDB

Helm / Kustomize: chart hooks · Kustomize vs Helm (Spacelift) · Kustomize vs Helm (IBM)

Talos: editing machine configuration — apply-config modes

Njord bundle (read-only): /home/rickard/talos-njord/{njord-new-values.yaml, HANDOFF-NEXT-MACHINE.md, worker-join/README-ADD-SLAVE.md, migrate/old-helm-values.yaml, migrate/old-secrets.env}