nordisk/Skidbladnir

kubernetes-target — design (the three-target ServiceSpec)

Class: design · Function axis: f6 (install) · sibling of k8s-unfold.md (which describes the renderer + applier that already ship) and install-design.md (the backend tree). This chapter states the shape we ship next: what ServiceSpec must gain so that Kubernetes is an honest third target family rather than a bolted-on eleventh backend. Alternatives considered and rejected live in kubernetes-target-reasoning.md.

f0 — the frame

Kubernetes and the vetra/Njord deployer solved the same problems. They are not rivals here; they are two completed answers we can read off each other. Skidbladnir already ships a Kubernetes renderer (src/k8s.rs) and applier (src/k8s_apply.rs). The question this chapter answers is not "should we support Kubernetes" — we do — but:

Which capabilities did Kubernetes standardise that our neutral ServiceSpec still cannot express, which ones can it already express under another name, and which ones must we refuse on purpose?

The governing constraint is L5 and the honest-degrade rule: a knob only one backend can honour is a design smell. Every field added below is either renderable on a clear majority of the eleven backends, or it lands with a declared, observable degrade — never a silent drop.

f1 — what the Njord deployer actually does

Read read-only from /home/rickard/talos-njord/ (L10 — nothing was edited). The Njord deploy is two tiers, and it is important that they are separate:

Tier A — the node (Talos machine configuration)

The unit of deployment is a whole machine, described declaratively in one YAML document and pushed over the Talos API.

That last row is the single most instructive thing in the whole bundle: the deployer's failure mode is wait forever with no deadline. Kubernetes has progressDeadlineSeconds for exactly this shape of bug; Skidbladnir has nothing for it either (see §4 row 15).

Tier B — the application (Helm release)

The unit of deployment is a release: one chart plus one values file.

The shape of the whole thing: one declarative document per tier, applied by a tool that reconciles, with secrets referenced rather than embedded, and the hard-won operational facts living as comments beside the knob they fix. That is the same shape as a LakeSpec + ComponentSpec. The correspondence is real.

f2 — the Kubernetes side (established from the docs, not from memory)

Only the facts the table depends on, each with its source.

• Container lifecycle hooks. postStart "is executed immediately after a container is created. It runs concurrently with the container's ENTRYPOINT"; preStop "is called immediately before a container is terminated". Handlers are exec, httpGet, sleep. "PreStop hooks are not executed asynchronously from the signal to stop the Container; the hook must complete its execution before the TERM signal can be sent." "If either a PostStart or PreStop hook fails, it kills the Container." Delivery is at least once. (container-lifecycle-hooks)
• Probes, three of them. startup — "Disables liveness and readiness probes until startup succeeds"; liveness — "If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy"; readiness — the Pod "is not considered ready for serving" and is removed from load balancers. Mechanisms: exec, httpGet, tcpSocket. (pod-lifecycle)
• Termination. Grace period (default 30 s) begins → preStop runs → SIGTERM → grace expires → SIGKILL. The grace period "applies to the total time it takes for both the PreStop hook to execute and for the Container to stop normally". (pod-lifecycle, container-lifecycle-hooks)
• Init containers. "Each init container must complete successfully before the next one starts"; they "always run to completion"; they "can be given access to Secrets that app containers cannot access"; on failure "the kubelet repeatedly restarts that init container until it succeeds". They do not support lifecycle or any probe; sidecars (an init container with restartPolicy: Always) do. (init-containers)
• Deployment rollout. maxSurge / maxUnavailable, Recreate as the alternative strategy, minReadySeconds ("minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available"), progressDeadlineSeconds → ProgressDeadlineExceeded, revisionHistoryLimit, kubectl rollout undo --to-revision=N. (deployment)
• StatefulSet. Ordinal identity, stable DNS (web-0.nginx.default.svc.cluster.local), volumeClaimTemplates giving each Pod its own PVC, a headless Service requirement, podManagementPolicy OrderedReady vs Parallel, RollingUpdate with partition, OnDelete, and the warning that OrderedReady rolling updates can reach "a broken state that requires manual intervention". (statefulset)
• DaemonSet. RollingUpdate / OnDelete; "At most one pod of the DaemonSet will be running on each node during the whole update process." (update-daemon-set)
• ConfigMap. Four consumption modes (command args, env, envFrom, read-only volume); 1 MiB ceiling; volume-projected keys are "eventually updated"; those "consumed as environment variables are not updated automatically and require a pod restart"; immutable for API-server load. (configmap)
• Secret. Same two shapes with the same asymmetry — volume-mounted secrets sync (kubelet period, ~1 min), env vars never update; defaultMode sets file permissions; subPath mounts and immutable: true opt out of updates. (secret)
• ServiceAccount. "provides an identity for processes that run in a Pod"; a default SA per namespace; modern tokens come from the TokenRequest API and are "mounted into Pods using a projected volume" with "bounded lifetimes … automatically invalidated when the Pod they are mounted into is deleted"; automountServiceAccountToken: false opts out; spec.serviceAccountName "can only be set at Pod creation time". (configure-service-account)
• Operator pattern. "software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop" — a controller that "queries the control plane to find out what resources are configured" and makes "reality match the configured resources". (operator)
• Helm hooks. Nine of them (pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, post-delete, pre-rollback, post-rollback, test), ordered by helm.sh/hook-weight "in ascending order", cleaned by hook-delete-policy (before-hook-creation default / hook-succeeded / hook-failed); for a Job or Pod hook "Helm will wait until it successfully runs to completion". (helm charts_hooks)
• Kustomize, by contrast, has no hooks and no release concept — "its lifecycle is left to kubectl/GitOps", it "focuses purely on defining and applying the desired state with no scripting or state management". (Kustomize vs Helm, IBM) This matters to us: Skidbladnir's RenderedService is Kustomize-shaped (pure manifests, no lifecycle) while install::apply is Helm-shaped (an ordered verb list with a release notion). We already have both halves; we just never gave the hook tier a name.
• PodDisruptionBudget. Bounds voluntary disruption only — "Involuntary disruptions cannot be prevented by PDBs; however they do count against the budget" — enforced through the Eviction API, and bypassed entirely by a plain delete. (disruptions)
• Readiness gates. Third-party conditions injected into status, letting an external system hold a Pod un-ready while it stays Running. (pod-lifecycle)

f3 — what ServiceSpec can express today

src/service.rs:185 — the whole neutral vocabulary:

name · description · exec · args · user · group · scope · working_dir · env · after · needs · restart · restart_sec · sockets · health · agent · and the k8s-only quintet image · namespace · replicas · stateful · storage · env_from_secrets.

Renders to 11 backends (renderer_for, all Some — but the Option return is retained deliberately so a future backend can land as an honest None). Output is a RenderedService: files: Vec<RenderedFile> (path + contents + unix mode) plus enable/start/status/stop argv as data. Lifecycle drives the systemd limb over D-Bus and probes health() itself. orchestrate::start_waves Kahn-topo-sorts products into parallel waves. install::{install,update,uninstall,rollback}_product own the generation lifecycle over layout's atomic symlink swap.

Three facts about today that the table below turns on:

1. The neutral spec has no secret axis. SystemdServiceSpec has secret_environment + a validate() that refuses secrets without a mode-0600 EnvironmentFile= (src/systemd.rs:156-170,262) — but to_systemd_spec only ever calls with_env. A secret placed in ServiceSpec::env is rendered inline into the world-readable 0644 unit.
2. There is no hook tier at all, at either level (container or release).
3. Every renderer's limits are documented in a doc comment and observable nowhere. OCI silently drops restart_sec; dinit silently drops group; Windows silently drops working_dir; Kubernetes silently drops exec, user, after, needs, restart, restart_sec, working_dir and scope.

f4 — THE CORRESPONDENCE TABLE

Verdicts: (a) already expressed, spelled differently · (b) a real gap to ADD · (c) genuinely Kubernetes-only, correctly out of scope.

f5 — the (b) rows, as spec changes

Eight additions. Each is stated with its render on every family, so the honest-degrade set is visible before a line is written.

B1 — secret_env on the neutral spec (row 12 — do this first)

/// Sensitive KEY=VALUE entries. NEVER rendered into a world-readable file;
/// every backend writes these only to its mode-0600 private env file.
pub secret_env: Vec<(String, String)>,
/// Secrets carried BY REFERENCE (the Njord `existingSecret` shape): the bundle
/// names them, the target already holds the values.
pub secret_refs: Vec<String>,

• systemd / Quadlet — project to with_secret_env + force with_env_file, so the existing SystemdServiceSpec::validate() invariant finally applies to specs built from a ServiceSpec. This closes a real leak, not a feature gap.
• OpenRC / SysV / runit / s6-rc / dinit / OCI — already write all env to a 0600 file; secret_env merges there and is excluded from any inline render.
• Kubernetes — secret_env emits a Secret manifest + envFrom.secretRef; secret_refs is exactly today's env_from_secrets (rename it, keep the old field as a deprecated alias — L2, additive).
• Windows SCM — the .wxs ServiceConfig env block is not a private file; declared degrade: secrets on the SCM backend must use secret_refs.

B2 — lifecycle hooks (rows 4, 9)

pub enum HookAction { Exec { argv: Vec<String> }, Http { path: String }, Sleep(u32) }
pub struct Hooks {
    pub pre_start:  Vec<HookAction>,   // ordered, run-to-completion, must succeed
    pub post_start: Vec<HookAction>,   // concurrent with the main process
    pub pre_stop:   Vec<HookAction>,   // must complete before the TERM signal
    pub post_stop:  Vec<HookAction>,
}

The four slots are chosen because they are the intersection of what systemd and Kubernetes both name, and the union is renderable widely:

Seven of eleven honour pre_start; the rest degrade, declared. Note the two semantic facts that must be documented on the type, because they are not intuitive and both docs state them explicitly: post_start runs concurrently with the main process, not after it is ready, and pre_stop must complete before SIGTERM is sent and is charged against the shutdown window (B3).

B3 — stop_timeout_sec (row 8)

The most portable knob in this whole document — ten of eleven backends have a native slot, and it fixes an existing conflation (SysV currently derives its --retry TERM/N/KILL from restart_sec, which is a start backoff).

systemd/Quadlet TimeoutStopSec= · OpenRC retry= · SysV --retry TERM/N/KILL · runit/s6-rc timeout-kill · dinit stop-timeout · OCI --stop-timeout · Kubernetes terminationGracePeriodSeconds · Windows SCM ✗ (declared).

B4 — split the health axis (rows 6, 7)

pub startup:   HealthProbe,     // "has it finished coming up"
pub health:    HealthProbe,     // liveness — KEPT, same field, back-compatible
pub readiness: HealthProbe,     // "should traffic be routed here"
pub probe:     ProbeTiming,     // initial_delay / period / timeout / failures

Kubernetes standardised a distinction we conflate, and it is a real distinction: started ≠ alive ≠ ready for traffic. Honest per family:

• Kubernetes — all three, natively, plus ProbeTiming on each.
• systemd family — startup ≈ Type=notify + TimeoutStartSec=, or (without notify support) a pre_start/post_start poll loop from B2; health ≈ WatchdogSec=; readiness has no meaning — there is no traffic router to gate. Declared degrade, not a fake.
• everything else — health remains what Lifecycle::health() probes; startup becomes a bounded poll before start is reported successful; readiness degrades.

readiness is the one field here that is nearly (c). It stays because the Kubernetes target is a first-class family and a Service's endpoint gating is not an exotic corner — but it must be recorded as unexpressible everywhere else.

B5 — config_files (row 11)

pub struct ConfigFile { pub path: String, pub contents: String, pub mode: u32 }
pub config_files: Vec<ConfigFile>,

Skidbladnir can emit files (RenderedFile) but a spec cannot ask for one. Every host backend writes it directly; Kubernetes emits a ConfigMap + a read-only volumeMount (note the 1 MiB ceiling — a ConfigFile above it must be a declared degrade, not a truncation). Universally expressible.

B6 — uid / workload identity (row 14)

pub uid: Option<u32>,   // pin the numeric identity, not just the name

L5 says every service runs as its own dedicated identity, and ensure_system_user allocates one in [100, 1000). Kubernetes' twin is numeric: securityContext.runAsUser / runAsGroup / fsGroup. Pinning the uid in the spec makes one declared identity, two projections — POSIX ownership on a host, runAsUser in a Pod. Alongside it the k8s renderer should emit a ServiceAccount named for the service plus serviceAccountName, because the constellation's law "every service runs as its own dedicated user" has an exact Kubernetes spelling and we currently render the namespace default SA.

B7 — start_timeout_sec + restart burst cap (rows 15, 16)

The Njord bundle's worst failure was an unbounded stall. Kubernetes bounds it with progressDeadlineSeconds; systemd bounds the retry storm with StartLimitIntervalSec=/StartLimitBurst= (which Kubernetes has no equivalent for — CrashLoopBackOff is not configurable). Add both; render start_timeout_sec to TimeoutStartSec= / progressDeadlineSeconds / --wait timeouts, and the burst cap to systemd/Quadlet/OpenRC (respawn_max) with a declared degrade elsewhere.

B8 — RenderedService::degraded (row 28 — cross-cutting, and it unblocks the rest)

/// What THIS backend could not express from the spec. Data, not a doc comment.
pub degraded: Vec<Degrade>,          // { field: &'static str, reason: String }

Today every renderer's limits are prose. A caller cannot discover that OCI dropped restart_sec, that dinit dropped group, or that Kubernetes dropped after, needs, user, exec and working_dir. L4 says no behaviour ships without a way to observe it — and this is behaviour: it is the difference between an install that will work and one that quietly will not.

This is also the mechanism that makes every other (b) row safe to land: a knob may be added the moment its degrade is declarable, and the render-coverage matrix (Backend::all()) gains a second axis — not just "does it render" but "what did it lose".

f6 — the (c) rows, stated plainly

We should not do these. Each is correct-to-refuse for a specific reason, not for lack of effort.

• Rolling update pacing (maxSurge/maxUnavailable) — row 18. These are functions of replica count. On a host there is one process; the pacing knob has no denominator. The host analogue of a safe upgrade already exists and is better suited: layout's atomic symlink swap with the previous generation retained.
• Horizontal replicas — row 19. replicas stays as a k8s-only field. It must not grow a host meaning ("run N copies") — that is a different product (a supervisor), and inventing it would break the one-service-one-uid law.
• PodDisruptionBudget — row 20. A PDB is a negotiation with a scheduler about voluntary disruption. There is no scheduler on an airgapped host, and the docs are explicit that PDBs do not protect against involuntary disruption anyway. No analogue, no value in faking one.
• Readiness gates — row 21. Requires an API-server status subresource that a third party can write. There is no shared status object on a host, and inventing one means inventing an agent — which the charter forbids.
• Operator / CRD / continuous reconciliation — row 22. The README's headline claim is "no agent on the target". A reconcile-forever daemon is precisely an agent. What we should take from the operator pattern is the level-triggered idempotency contract, not the loop: render → apply must be a byte-identical no-op when the target already matches. ensure_system_user is already idempotent, k8s_apply already uses server-side apply, and the layout swap is already atomic — the discipline should be written down and tested, and the daemon should not be built.
• In-place secret rotation — row 13. Kubernetes rotates volume-mounted Secrets because the kubelet is a running agent with a sync loop. A 0600 env file is env-var-shaped: it changes only on restart. Say so; do not simulate a sync loop.
• DaemonSet — row 24. "One per node" presupposes a node set. unfold provisions machines one at a time by design.

And the mirror image — what the deployer/Skidbladnir does that Kubernetes has no answer for

Worth stating as loudly as the gaps:

1. Cross-service dependency ordering (row 10). Kubernetes has no cross-workload ordering primitive; it is a deliberate design position (make services tolerate a missing dependency). Skidbladnir's start_waves Kahn topo sort over after/needs is a capability Kubernetes declines to offer, and the Njord chart works around its absence with helm --wait and subchart order. The Kubernetes renderer must not pretend to have it — see §7.
2. Sealed, content-addressed, verify-on-receipt transport (row 26). Kubernetes assumes a reachable registry. Njord hand-rolls the offline case with a LAN registry and allowInsecureImages: true — which is exactly the hole archive::BundleManifest + fetch::verify_on_receipt fills.
3. Node provisioning (row 3). Kubernetes starts where unfold finishes.
4. A restart burst cap (row 16). systemd's StartLimitBurst= has no Kubernetes equivalent; CrashLoopBackOff is not tunable.

f7 — Kubernetes as the third target: the field map

ServiceSpec → manifests, stated field by field. ● renders today · ○ should render (missing) · ✗ honest degrade, must be recorded via B8.

The four honest None/degrade cases that need naming

1. after/needs on Kubernetes. Two acceptable renders, and one forbidden. Forbidden: dropping them silently, which is what happens today. Acceptable A: record the degrade (B8) and leave ordering to orchestrate::start_waves, which drives the applier, not the manifest — this is the honest one, because it keeps the ordering guarantee where it actually holds. Acceptable B: render a pre_start init-container wait-gate per dependency (initContainers retry until success — the semantics match). Choose A as the default and B as opt-in, because A does not require the dependency to be resolvable by DNS from inside the pod.
2. RestartPolicy::No + no sockets ⇒ this is a Job, not a Deployment. A Deployment cannot express "run once and stop". Render a Job, or degrade loudly. (Row 25.)
3. stateful + storage. A StatefulSet with replicas > 1 sharing one ReadWriteOnce PVC does not work; the documented shape is volumeClaimTemplates (one PVC per ordinal) plus a headless Service (clusterIP: None), which the StatefulSet docs require for its stable network identity. Today k8s.rs emits a standalone PVC and a normal Service. Correct this before anyone runs replicas > 1 stateful.
4. scope: User. There is no per-user scope in a cluster. Degrade, recorded.

Keep renderer_for's Option

All eleven backends resolve to Some today, and the Option return is retained on purpose. It stays. The Job backend (case 2 above), and any future family, must be free to land as an honest None rather than as a renderer that emits something plausible and wrong. None and degraded are the same virtue at two granularities: None = "this backend cannot express this spec at all"; degraded = "it can, but it lost these fields".

f8 — the three targets

Eleven backends, three families. The family — not the backend — is where a degrade decision belongs, because the three differ in all three things that matter at once: who the service is, what supervises it, and how config reaches it.

Why exactly these three, and not two or four:

• T1 vs T2 is a real boundary, not packaging. T1's unit of delivery is a binary at a path plus a user in /etc/passwd; T2's is an image reference. That is why ServiceSpec needs both exec and image — and why the fact that Kubernetes ignores exec while OCI honours it is a bug, not a taste.
• T3 is not "T2 in a cluster". The distinguishing feature is not orchestration, it is the API boundary: identity becomes a token issued by a server, config becomes an object with its own lifecycle, and supervision becomes a controller that outlives the applier. Every single (c) row above is a consequence of that boundary existing — and every (b) row is a capability that turns out to survive the crossing.
• Quadlet is the bridge, and that is why the count is 3 not 2. It has a T2 body (an image) and T1 supervision (systemd), and it is the only backend that honours knobs from both sides — restart_sec (which raw OCI cannot) and image (which systemd proper cannot). A design that only had "host" and "cluster" would have no place to put it.

Ship Backend::family() -> TargetFamily alongside Backend::slug(). It gives the render-coverage matrix its coarse axis, gives degraded its natural grouping, and gives this document's rule a machine-checkable form: a new ServiceSpec field must render on at least one backend in each of the three families, or be declared family-scoped at the point of definition.

Status

Design only — nothing implemented by this chapter. The Kubernetes renderer + applier already ship and are green (k8s-unfold.md: 111 lib + 11 integration by default, 114 + 11 with --features kubernetes). Suggested landing order, cheapest-first and each independently useful:

B8 (degraded — unblocks honest landing of everything else) → B1 (secret_env — closes a real leak) → B3 (stop_timeout_sec — 10 of 11 backends) → B2 (hooks) → B4 (probe split) → B6 (uid + ServiceAccount) → B5 (config_files) → B7 (deadlines) → the four §7 renderer corrections.

Every one is additive (L2): existing specs render byte-identically when the new fields are at their defaults, which is what keeps the exact-bytes contract tests in k8s.rs and service.rs meaningful rather than in the way.