nordisk/Skidbladnir

nornir-install — Installer, Airgap Deployer & Service Orchestrator

Design document for the airgap deploy/installer system (crate nornir-airgap, codename Skíðblaðnir). Replaces: x14-datalabb Ansible playbooks. Targets: holger, nornir-server, mail-server, tunnr, and any future LAKEhouse component. Airgap is the primary use case — every design decision must work with zero network access.

⚠️ CLI MOVED (2026-06-20) — nornir install was repurposed

The verbs in this document originally lived under nornir install …. As of commit aa6745b the nornir install namespace was repurposed for the single-host server lifecycle (add | update | remove), and the airgap bundle/component verbs moved under nornir airgap. Translate while reading:

this doc says	run today
nornir install pack-lake	nornir airgap pack
nornir install from-archive	nornir airgap install (one component)
nornir install verify	nornir airgap fetch (verify-on-receipt)
nornir install upgrade --delta	nornir airgap component upgrade --delta
nornir-install remove <component>	nornir airgap component remove <name>
(by-name install/status)	nornir airgap component add|status <name>

The nornir install verb now means only the nornir-server systemd lifecycle — add (install + create the nornir user + /etc/nornir + unit), update (local binary swap + restart; no registry fetch), remove (tear down). That flow is documented in running-as-a-server.md, not here. This doc remains the design reference for the airgap engine beneath both surfaces. See airgap-install.md for the current airgap guide.

holger self-installs too (2026-06-21)

The systemd-install limb (nornir-airgap::systemd, design: crates/nornir-airgap/.nornir/airgap-installer.md) is now reused by holger: holger-server install {add | update | remove | status | render} drives the same engine — Skíðblaðnir's "every product installs itself the same way" goal, realized for holger first. The engine grew the matching remove_systemd / status_systemd verbs so the whole lifecycle (add · update · delete · status) lives at the spec layer. holger builds its own SystemdServiceSpec (its serve verb is start --config, config is RON) — the engine only applies it.

Problem statement

x14-datalabb is an Ansible-based playbook that installs the full Nordic stack on air-gapped Linux machines. It:

1. Creates Linux users and directory trees (data, program, PID paths).
2. Installs and enables systemd units.
3. Starts services in a specific dependency order.
4. Handles per-machine/per-service PKI (keys + certificates).
5. Pre-stages a compressed archive before connectivity is lost.

Pain points: Ansible is an extra dependency, YAML is not type-safe, ordering is implicit, PKI wiring is bespoke per playbook, and znippy archives need full decompression before any component is reachable.

Airgap model — the primary deployment path

An air-gapped machine has no network access after the initial staging step. Every byte that will ever run on it must arrive before the gap closes. This is not an edge case — it is the x14-datalabb default.

The two phases

PHASE 0: PRE-STAGING  (internet-connected staging host OR secure courier)
──────────────────────────────────────────────────────────────────────────

  nornir install pack-lake --components holger,nornir-server,tunnr \
                           --platform linux-x86_64,windows-x86_64  \
                           --out lake-2026-06-14.znippy

  What this produces (a SINGLE signed znippy archive):
  ┌──────────────────────────────────────────────────────┐
  │  lake-2026-06-14.znippy                              │
  │  ├── installer/nornir-install   ← the installer itself│
  │  ├── manifests/                                      │
  │  │   ├── holger.toml            ← ComponentSpec      │
  │  │   ├── nornir-server.toml                          │
  │  │   └── tunnr.toml                                  │
  │  ├── linux-x86_64/                                   │
  │  │   ├── holger                 ← statically linked  │
  │  │   ├── nornir-server                               │
  │  │   └── tunnr.img              ← UEFI appliance     │
  │  ├── windows-x86_64/                                 │
  │  │   └── x14_sorcerer_1.1.29.msi                    │
  │  └── MANIFEST.blake3            ← whole-archive hash │
  └──────────────────────────────────────────────────────┘

  Transfer: USB stick / encrypted courier / one-time relay drop
  Verify on receive: nornir install verify lake-2026-06-14.znippy


PHASE 1: AIRGAP INSTALL  (target machine, zero network)
──────────────────────────────────────────────────────────────────────────

  # The installer is IN the archive — bootstrap it out first:
  znippy extract lake-2026-06-14.znippy installer/nornir-install \
         | install -m 0755 /dev/stdin /usr/local/bin/nornir-install

  nornir-install from-archive lake-2026-06-14.znippy \
                 --select holger,nornir-server

  ↓ for each selected component (in dependency order):
  ├── stream component binary from archive (NO full decompress)
  ├── blake3-verify the streamed bytes against MANIFEST.blake3
  ├── scaffold filesystem (user, dirs, permissions)
  ├── write binary (atomic: temp → rename)
  ├── generate systemd unit from embedded ComponentSpec
  ├── generate PKI (machine CA if not present, leaf cert)
  └── enable + start (health-check before proceeding)

Airgap constraints that shape every design decision

What "no full decompress" means

znippy archives are Arrow IPC files with an index at the end. The installer uses archive::seek_to(archive, "linux-x86_64/holger") — a single index lookup + ranged read — to stream the binary directly to disk. The rest of the archive (e.g. the Windows MSI) is never touched. This is critical when the USB stick has limited I/O throughput or the target has limited /tmp space.

Secure transfer options

1. USB stick — simplest; verify MANIFEST.blake3 after copy.
2. One-time relay drop — staging host uploads to a short-lived HTTPS endpoint (e.g. presigned S3); target downloads once then the endpoint expires.
3. Encrypted courier — archive encrypted with age (recipient = target's pubkey, provisioned out-of-band); nornir install decrypt-and-verify unwraps before install.

Use case: x14-datalabb air-gapped deployment

┌─────────────────────────────────────────────────────────────────┐
│  OFFLINE MACHINE (air-gap)                                      │
│                                                                 │
│  1. znippy archive arrives (USB / secure relay)                 │
│     ↓ nornir-install from-archive lake.znippy                   │
│  2. Installer streams individual components from the archive    │
│     WITHOUT full decompression (znippy selective read)          │
│     ↓                                                           │
│  3. Per-component:                                              │
│     a. Create Linux user (e.g. _holger, _nornir)                │
│     b. Create directories:                                      │
│        /opt/<name>/bin   (program, non-writable)                │
│        /var/lib/<name>   (data)                                 │
│        /run/<name>       (PID / socket)                         │
│        /etc/<name>       (config, secrets)                      │
│     c. Write binary + config                                    │
│     d. Generate systemd unit from ComponentSpec                 │
│     e. Register PKI: per-machine CA + per-service leaf cert     │
│     f. Enable + start unit                                      │
│  4. Dependency-ordered start: holger → nornir-server → ...      │
│  5. Health-check each service; report failures with context     │
└─────────────────────────────────────────────────────────────────┘

Architecture

nornir-install crate (in nornir workspace)

nornir-install/
  src/
    lib.rs           — public API surface
    spec.rs          — ComponentSpec + DependencyGraph
    installer.rs     — Installer: install / upgrade / remove / status
    filesystem.rs    — user creation, directory layout, permissions
    systemd.rs       — unit file render + journalctl drain
    pki.rs           — PKI: machine CA + per-service leaf cert gen
    archive.rs       — znippy selective read (no full decompress) + pack-lake
    airgap.rs        — airgap-specific: verify MANIFEST.blake3, decrypt (age),
                       bootstrap extraction, idempotency guards
    delta.rs         — DeltaPackager: bsdiff-style patch via znippy
    health.rs        — health-check poller (HTTP /health or TCP)
    redfish.rs       — RedfishClient: bare-metal BMC provisioning
    error.rs         — InstallError enum
    msi.rs           — Windows MSI builder (WiX 4.0.6 via Wine/podman)

ComponentSpec — the unit of installation

pub struct ComponentSpec {
    pub name: &'static str,       // "holger", "nornir-server", "tunnr"
    pub binary: PathBuf,          // path inside znippy archive
    pub user: &'static str,       // "_holger"
    pub data_dir: PathBuf,        // /var/lib/holger
    pub config_dir: PathBuf,      // /etc/holger
    pub pid_dir: PathBuf,         // /run/holger
    pub ports: Vec<u16>,          // [7070]
    pub after: Vec<&'static str>, // ["holger"] — start AFTER these
    pub restart: RestartPolicy,   // Always / OnFailure / Never
    pub health: HealthCheck,      // Http("/health") | Tcp(port) | Pid
    pub pki: PkiSpec,             // None | Leaf { ca_ref, san }
    pub env: Vec<EnvSpec>,        // static + generated (PKI paths)
}

Each product ships its own ComponentSpec::holger(), ComponentSpec::tunnr(), etc.

Phases of nornir install from-archive

Phase 0 — airgap pre-step (znippy archive)

The archive is a znippy .znippy file (Arrow IPC, zero-copy indexed).

lake.znippy
  components/
    holger          ← binary
    nornir-server   ← binary
    tunnr           ← binary (UEFI appliance image)
    ...
  manifests/
    holger.toml
    nornir-server.toml
    ...

nornir install from-archive lake.znippy --select holger,nornir-server streams ONLY the selected components out of the archive — no full decompress. znippy's indexed format means seeking to components/holger is O(1).

Phase 1 — filesystem scaffold

/opt/<name>/bin/<name>    0755  owned by root, group _<name>
/var/lib/<name>/          0750  owned by _<name>
/var/log/<name>/          0750  owned by _<name>
/run/<name>/              0755  owned by _<name>
/etc/<name>/              0700  owned by _<name>

Linux user _<name> is created with --system --no-create-home --shell /sbin/nologin.

Phase 2 — PKI: machine CA + per-service leaf certs

An air-gapped machine needs its own PKI. The installer:

1. Checks if a machine CA exists at /etc/x14-ca/ — creates one if not (ed25519, 10-year validity, CN=<hostname>.x14.local).
2. For each component that declares pki: Leaf { san }: generates a leaf key+cert signed by the machine CA, writes to /etc/holger/holger.key + holger.crt.
3. For inter-service mTLS: each service gets a client cert too.

Option for future: a temp PKI service boots first, services request certs via a gRPC PKI.IssueCert call during their startup, then the PKI service shuts down. This lets certs be re-issued without the installer needing root long-term.

Phase 3 — systemd unit generation

TLS is configured in the RON config, NOT by environment. This unit deliberately sets only HOLGER_DATA. Earlier revisions also set HOLGER_TLS_CERT and HOLGER_TLS_KEY, and holger reads NEITHER — its complete env-read surface is HOLGER_DATA, HOLGER_WAREHOUSE, HOLGER_TOKEN, HOLGER_QUARANTINE_PROPERTY, HOLGER_REQUIRE_NETWORK, HOLGER_PROMOTE_{ENFORCE,REQUIRE_PROPERTY,DENY_LICENSES,ADVISORY_DB} and HOLGER_DEMO_{IMAGE,BEAT_MS}. Transport security comes exclusively from exposed_endpoints[].ron_tls { cert, key, client_ca } in the file passed to --config.

That mattered more than a stale line usually does: holger's own auth guide states that with no auth block a server "accepts anonymous writes over cleartext". An operator who copied this unit, pointed the two variables at real certificates and saw the service come up green had cleartext plus anonymous writes, and two settings that looked like they had done something. A security control that fails silently OPEN — the same shape as the OIDC cache-TTL rename (holger 0e5a635, edda 0c1bffd). Skidbladnir's shipped ComponentSpec was always consistent with the code (product.rs asserts only HOLGER_DATA); it was this DOCUMENT that was fiction.

[Unit]
Description=Holger artifact repository
After=network.target nornir-server.service
Requires=nornir-server.service

[Service]
Type=notify
User=_holger
ExecStart=/opt/holger/bin/holger serve --config /etc/holger/holger.toml
PIDFile=/run/holger/holger.pid
Restart=on-failure
RestartSec=5s
Environment=HOLGER_DATA=/var/lib/holger
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/holger /run/holger /var/log/holger

[Install]
WantedBy=multi-user.target

The unit file is rendered from ComponentSpec — hardening flags (NoNewPrivileges, ProtectSystem, etc.) are always on unless spec.hardening = false.

Phase 4 — dependency-ordered start

The DependencyGraph topologically sorts all ComponentSpec::after edges. Services are started in waves: each wave = all components whose dependencies are already running. After each service starts, the health-check polls up to 30s.

Wave 1: machine-ca-service (if temp PKI mode)
Wave 2: holger (no after-deps)
Wave 3: nornir-server (after: holger)
Wave 4: mail-server (after: nornir-server)
Wave 5: tunnr (after: nornir-server)

If a service fails to pass its health-check, the installer reports:

• Which service failed
• The last N lines from journalctl -u <name> --no-pager -n 50
• The exit code / signal
• Which downstream services are blocked

nornir install CLI verbs

# Staging host (internet-connected)
nornir install pack-lake --components holger,nornir-server,tunnr \
                         --platform linux-x86_64,windows-x86_64  \
                         --out lake-2026-06-14.znippy             # airgap bundle
nornir install pack-delta <old_bin> <new_bin> --out delta.znippy  # upgrade patch
nornir install pack-msi [--spec sorcerer.toml] [--out path]       # Windows MSI
nornir install pack-windows [--spec sorcerer.toml]                # MSI → znippy bundle

# Target machine (airgap — zero network)
nornir-install verify lake.znippy                   # check MANIFEST.blake3
nornir-install decrypt lake.znippy.age --key id_age # unwrap encrypted courier
nornir-install from-archive lake.znippy [--select holger,nornir-server]
nornir-install upgrade --delta delta.znippy         # apply binary patch
nornir-install remove <component>
nornir-install status                               # table: name/version/health/uptime

# Image building
nornir install build-image                          # build WiX podman container

Delta package delivery (znippy)

For upgrades where bandwidth or time matters:

1. nornir install pack-delta v1_binary v2_binary --out delta.znippy — computes bsdiff-style binary delta, wraps in a signed znippy chunk — the patch is zstd-compressed + blake3-signed
2. nornir install upgrade --delta — fetches delta.znippy (or reads from local archive) — verifies signature, applies patch in-memory (zero alloc for the binary delta), writes to a temp file, atomic-rename onto the existing binary
3. Falls back to full binary if patch would be larger than the full binary.

tunnr integration

tunnr is a UEFI appliance — its "binary" in the znippy archive is a disk image. ComponentSpec::tunnr() declares type: Appliance instead of Service. The installer:

1. Writes the image to a block device (or loop-mounted file).
2. Registers it with the tunnr KVM supervisor (a systemd service that manages VMs).
3. Health-check: tunnr status <vm-name> returns BOOT-OK.

For Redfish / bare-metal: ComponentSpec::redfish_target() wraps a RedfishClient that PXE-boots the target, pushes the image, and polls for POST-complete.

Redfish + Terraform stubs

RedfishClient (feature redfish):

• provision(bmc_url, image_url) — POST to Redfish Systems/1/Actions/ComputerSystem.Reset
• wait_post_complete(timeout) — polls Redfish EventService for PostComplete
• eject_virtual_media() — cleanup

Terraform provider stubs (feature terraform):

• A thin terraform-provider-nornir-install binary that wraps the Rust API via the Terraform plugin protocol — resource types: nornir_component, nornir_pki_cert.

Crate structure in nornir workspace

# nornir/Cargo.toml
[workspace.members]
+ "crates/nornir-install"

# crates/nornir-install/Cargo.toml
[package]
name = "nornir-install"
version = "0.1.0"

[dependencies]
znippy-zoomies = "0.1.10"   # selective archive read + GATLING save
bsdiff = "0.2"               # binary delta
zstd = "0.13"
blake3 = "1"
tonic = ...                  # PKI gRPC (optional)
serde = { version = "1", features = ["derive"] }

[features]
default = ["systemd"]
systemd = []                 # systemd unit render + journalctl drain
pki = []                     # machine CA + leaf cert gen
redfish = ["reqwest"]        # BMC provisioning
terraform = []               # provider stubs
delta = ["bsdiff"]           # delta package delivery

Test obligations (LAW 1 — inject-assert)

• Unit file render round-trip: ComponentSpec::holger() → render → parse → assert User=_holger, After=..., all hardening flags present.
• Dependency-ordered start: build a 5-node DependencyGraph, assert topological waves match expected order; assert cycle detection returns Err.
• Delta patch round-trip: pack_delta(a, b) → apply_delta(a, patch) == b.
• Filesystem scaffold: temp dir, scaffold(spec), assert all paths exist with correct permissions (stat mode bits).
• Archive selective read: build a test znippy archive with 3 components, read only one, assert the other two are not touched.
• Health-check poller: mock TCP listener, assert poll_until_healthy returns Ok within 1s; mock that never answers, assert timeout Err.

Prior art: talos-airgap bundler (git/airgap-kubernetes-main)

This Go project implements the exact same two-phase bundle→deploy pattern as nornir-install, but for Talos Linux / Kubernetes clusters (KVM + Scaleway). It was assembled quickly and is the direct inspiration for every major nornir-install architectural decision. Study it.

What it does

Connected machine:                  Airgap machine:
  ./bundler bundle                    tar -xf talos-bundle.tar.zst
    → talos-bundle-v1.12.6.tar.zst     podman load -i images/bundler.tar
                                        ./bundler provision kvm \
                                          --libvirt-addr 192.168.1.10:16509

Five patterns to adopt directly

1. Concurrent download → GATLING
saveImage spawns a goroutine per image into a buffered channel — wall-clock = slowest image.

ch := make(chan result, len(imgs))
for _, img := range imgs { go func() { ch <- result{img, saveImage(img, dir)} }() }

In Rust: znippy GATLING worker pool (one worker per download slot, 32-core encode, save stage also GATLING — LAW 8).

2. Resumable download with atomic rename
dlFile writes to dest.tmp, checks existing .tmp size for Range: bytes=N- resume, renames to dest only on completion. Never leaves a partial file at the real path.
→ Identical to how archive::extract_component must write binaries (temp → atomic rename).

3. Self-bootstrap detection (insideContainer)
The binary detects whether it is running inside its own container by checking /.dockerenv or exe == "/bundler", and switches default paths accordingly. The nornir-install equivalent:

fn inside_archive_bootstrap() -> bool {
    std::env::var("NORNIR_INSTALL_BOOTSTRAP").is_ok()
    || std::env::current_exe().map(|p| p == Path::new("/usr/local/bin/nornir-install")).unwrap_or(false)
}

This lets the same binary be both the pack-side tool and the airgap-side installer.

4. Upgrade bundles carry from+to for rollback
upgradeImageList includes both the current and the target installer image so the cluster can roll back if the new version fails to boot. Talos uses A/B partition swap. → nornir install pack-delta should embed the old binary alongside the patch so upgrade --delta can fall back to the known-good binary without a network fetch.

5. Pure-language provisioner (no CGO, no C libs)
go-libvirt re-implements the libvirt wire protocol in pure Go over TCP:16509 — zero CGO, distroless-compatible. The Rust equivalent is direct TCP using the libvirt RPC protocol (XDRGEN structs) or calling virsh as a subprocess — the latter is acceptable for nornir-install since KVM provisioning is a staging-host-only step, not airgap.

What the Go bundler has that we don't yet

Key file mapping

What this replaces (x14-datalabb Ansible)

The znippy archive replaces the Ansible files/ directory and the Ansible controller's network access — no controller needed at deploy time.

Windows MSI builder (msi-agent)

What msi-agent is

msi-agent (git/msi-agent) is the existing Windows MSI production toolchain for the X14 Sorcerer CDC product: Kafka Connect Standalone + Debezium SQL Server connector + Confluent Schema Registry + Eclipse Temurin JDK 17, wrapped by x14javaservice.exe as a Windows service.

Key facts:

• WiX 4.0.6 (freeware WiX Toolset) builds the .msi
• WiX runs under Wine inside a podman container (wixnotricks.dockerfile) — no Windows host required; builds on Linux
• Perl scripts currently drive: dependency download (JDK/Kafka/Debezium/Confluent SR jars), JAR pruning (strips CloudEventsConverter + unused connectors), UUID-v5 GUID generation for WiX component stability, podman invocation
• Product installs to C:\Program Files\X14\Sorcerer\ as a LocalSystem delayed-auto-start service; env block written to HKLM\SYSTEM\CurrentControlSet\Services\Sorcerer

Integration into nornir-install

The Perl scripts (tools/*.pl) are the shell-equivalent anti-pattern (LAW 7). The nornir-install crate absorbs this as src/msi.rs:

pub struct WindowsComponentSpec {
    pub name: &'static str,          // "Sorcerer"
    pub display_name: &'static str,  // "X14 Sorcerer CDC"
    pub version: &'static str,       // "1.1.29.0"
    pub upgrade_code: Uuid,          // stable per product, never changes
    pub install_dir: &'static str,   // "X14\\Sorcerer"
    pub service: WindowsServiceSpec,
    pub files: Vec<WixFile>,         // source → dest mappings
    pub registry: Vec<RegistryValue>,
}

pub struct WindowsServiceSpec {
    pub account: &'static str,       // "LocalSystem"
    pub start: ServiceStart,         // Auto / Manual / Disabled
    pub delayed: bool,
    pub env_block: Vec<(String, String)>,
}

pub struct WixBuilder {
    pub image: String,               // "wix4-bundlev2" (pre-built container)
}

impl WixBuilder {
    /// Generate Product.wxs + Payload.wxs from spec, run WiX in container → .msi
    pub fn build(&self, spec: &WindowsComponentSpec, out: &Path) -> Result<()>;

    /// Build the WiX container image (wraps `podman build -f wixnotricks.dockerfile`)
    pub fn build_image(&self) -> Result<()>;
}

GUID stability: component GUIDs are generated as UUID v5 (SHA-1, namespace 7f5b3f9e-3a2a-4e96-a5c3-4a2b9f7e6a10, key = lowercase component Id). This is exactly what tools/build.pl does — ported to Rust (uuid crate v5 API).

Dependency staging (replaces tools/download.pl → Rust)

pub struct WindowsPayload {
    pub jdk: JdkSource,    // Temurin v17 or v21, Windows x64
    pub kafka: KafkaSource, // version + scala version
    pub plugins: Vec<PluginSource>, // Debezium, Confluent SR jars
}

pub enum PluginSource {
    Maven { group, artifact, version },
    Direct { url, sha256 },
}

WixBuilder::stage(payload, into_dir):

1. Fetches each source into a local cache dir (idempotent — skip if SHA256 matches)
2. Prunes JAR patterns (kafka-connect-file-*.jar, CloudEventsConverter.class, etc.) using Rust zip manipulation (the zip crate, not shell calls to unzip)
3. Outputs the flat file tree that WiX Product.wxs references

Archive delivery (znippy)

For airgap Windows deployments, nornir install pack-windows --spec sorcerer.toml produces a sorcerer_windows.znippy containing:

• msi/x14_sorcerer_1.1.29.msi — the built installer
• manifest.toml — version + blake3 hash

On the Windows target, nornir install from-archive sorcerer_windows.znippy calls msiexec /i msi\x14_sorcerer.msi /qn (silent install) using a Windows build of nornir-install (cross-compiled x86_64-pc-windows-gnu).

CLI additions

nornir install build-image          # build the WiX container image
nornir install pack-msi [--spec <sorcerer.toml>] [--out <path>]
                                    # stage + WiX-build → .msi
nornir install pack-windows [--spec <sorcerer.toml>]
                                    # pack-msi → znippy bundle

What replaces each Perl script

Test obligations

• guid_v5 matches the Perl SHA-1 UUID v5 output for known inputs (cross-checked).
• inject_guids round-trip: generate wxs with Guid="*" → inject → re-parse → all component GUIDs are deterministic UUID-v5 values, no duplicates.
• prune_jars removes exactly the expected filenames and nothing else.
• prune_zip_entries removes CloudEventsConverter.class entries from a test jar.
• harvest produces a WixFile list with correct Source/Id for a test dir tree.