nordisk/holger

Mini Holger, for Henrik

Build and run holger mini (bin holger-mikro) on your own machine.

Every command below was executed on oden 2026-07-22 and is transcribed from what actually ran. Where something is asserted but was not measured, it says so.

⚠ NAMES CHANGED SINCE THIS TRANSCRIPT (2026-08-01). The commands and --list-variants output below are verbatim from 2026-07-22 and are left as they ran, but two things have been renamed and one claim has since been fixed:

then	now
variant henriks-mini-holger	variant ui-mini (it is a UI tier; the requester is not a build axis)
variant henriks-mikro-holger	variant ui-mikro
bins holger-ui, holger-ui-embedded, holger-ui-mini, holger-ui-mikro	holger-server-ui, -embedded, -mini, -mikro (the bin now says which component it drives)
package holger-ui	package holger-server-ui (renamed 2026-08-01 on Rickard's call; the published holger-ui 0.1.4 stays up, unyanked, 0 reverse deps)

§4's "the TOML declares six" and §5's "mini = ["gui"], so the mini client is the fattest artifact" are both stale: the TOML now declares thirteen variants, and the tier ladder was rebuilt on 2026-07-23 so every heavy dep is optional and the mini/mikro clients really are leaner (measured, and guarded by server/ui/tests/tier_ladder.rs).

0. TL;DR

# one-time: clone holger + the six siblings it path-depends on, side by side
mkdir -p ~/git && cd ~/git
for r in holger edda znippy znippy-zoomies modgunn Skidbladnir draupnir facett; do
  git clone ssh://git@git.gunnar.rs:2222/nordisk/$r.git
done

# build (the flags are NOT optional — see §3)
cd ~/git/holger
cargo build --release -p holger --bin holger-mikro \
      --no-default-features --features mini

# run
./target/release/holger-mikro dev-pair --data /var/tmp/holger-mini

It binds two sockets, both from the config, both cleartext by default:

Override with --grpc / --http. Under start --config, the addresses come from the config's exposed_endpoints[].ron_url / ron_http_url — there is no built-in default there.

1. What mini is, measured

mini is one of three server tiers built from one source tree (server/cli/src/main.rs), declared in build-variants.toml.

Measured on oden, release profile:

Zero-egress was confirmed four independent ways for mini:

1. cargo tree -p holger --no-default-features --features mini contains no reqwest, no ureq.

2. nm -C target/release/holger-mikro | grep -c 'reqwest\|ureq' → 0.

3. objdump -p target/release/holger-mikro | grep NEEDED → only liblzma.so.5, libgcc_s.so.1, libm.so.6, libc.so.6, ld-linux-x86-64.so.2. No libssl, no libcrypto, no libcurl.

4. At runtime. Point it at a rust-remote repo and it refuses, by name:

ERROR holger_server_lib] repository 'crates-io' DISABLED — backend init failed:
repo 'crates-io' is 'rust-remote' (a pull-through network upstream) but this
binary was built WITHOUT the `online` feature (zero-egress airgap tier) — no
HTTP client is linked.

That message is the one that makes the claim trustworthy: the binary fails closed on a network repo rather than quietly degrading.

The only NEEDED shared library beyond glibc is liblzma.so.5 — install xz-libs / liblzma5 on a bare target host.

mini vs embedded vs full, from the feature closure

holger's tier features expand (see server/cli/Cargo.toml, server/lib/Cargo.toml):

embedded = server-lib/embedded = ["auth-bearer", "auth-mtls"]
mini     = server-lib/mini     = ["auth-bearer", "auth-mtls"]
full     = server-lib/full + online + bench
         = ["federation", "auth-bearer", "auth-oidc", "auth-oauth2", "auth-mtls"]
           + reqwest + ureq + holger-bench-scenarios

mini and embedded are the same build. Their dependency closures are byte-identical (diff of the two cargo tree outputs is empty), and grep -rn 'feature = "mini"' server/lib/src/ returns nothing — the mini feature gates no code in the server library at all. The two binaries differ in name and in the declared artifact kind (exe vs service), not in behaviour. See §6.

full has and mini does not: reqwest, ureq, hyper-tls, native-tls, openssl, jsonwebtoken, oauth2, nornir-auth-oidc, nornir-auth-oauth2, webpki-roots, url/idna/icu*, tower-http, holger-bench-scenarios, zip, zopfli.

So, derived from the closure — not from the name:

mini cannot:

• serve rust-remote (crates.io pull-through mirror) — no HTTP client linked
• serve nexus-remote / artifactory-remote / azure-remote / aws-remote — the Skíðblaðnir federation ship is not compiled in
• authenticate with OIDC or OAuth 2 — those provider crates are absent
• run the holger-ds delivery benchmark — bench is off

mini can (all verified against a running holger-mikro):

• serve every local repo type: rust, maven, python, npm, nuget, go, gem, deb, rpm, helm, conda, composer, docker, oci, package — znippy- or file-backed
• serve all eight gRPC services, unconditionally registered: RepositoryService (incl. PutArtifact), ArchiveService, AdminService, PromotionService, SearchService, SbomService, PropertyService, ReplicationService
• serve the HTTP/OCI gateway: sparse index, crate download, /-/search, /-/sbom, /-/properties (read and write)
• authenticate with bearer tokens and mTLS, and enforce RBAC
• install itself as a systemd unit through Skíðblaðnir (install / service)
• run the CLI verbs start dev-pair install service scan retention airgap search sbom properties replicate

⚠ mini is not read-only. build-variants.toml describes it as "the stripped, read-only standalone server". It is not. See §6 — read-only is a property of the config, not of the tier, and §5 shows how to actually get it.

2. What it needs at runtime

Nothing else has to be running. Verified:

The linkage = "fat" in build-variants.toml is correct and verified: mini opens its own state directly and needs no nordisk sibling up. (The build-thing plugin disagrees — see §6, finding 1.)

3. Build it — the exact commands

3.1 Toolchain

Verified with:

rustc 1.97.1 (8bab26f4f 2026-07-14)
cargo 1.97.1 (c980f4866 2026-06-30)

3.2 The sibling checkouts are mandatory

holger does not build from a lone clone. Its manifests carry path = deps that escape the repo, and its root [patch.crates-io] points at siblings by relative path. Clone them side by side under one parent directory:

mkdir -p ~/git && cd ~/git
for r in holger edda znippy znippy-zoomies modgunn Skidbladnir draupnir facett; do
  git clone ssh://git@git.gunnar.rs:2222/nordisk/$r.git
done

All remotes are SSH (ssh://git@git.gunnar.rs:2222/nordisk/...). The forge must know your key — see §7.

3.3 The build line

cd ~/git/holger
cargo build --release -p holger --bin holger-mikro \
      --no-default-features --features mini

Measured on oden: 17.2 s with sibling crates already compiled (the tail — holger-server-lib + holger only — relinks in 9.6 s). A cold first build compiles ~223 crates and takes proportionally longer.

Two rebuilds of that same line produced byte-identical binaries.

2026-08-13: the declared mini atom now spells --features mini,appliance — the WireGuard identity ceremony (edda's wg-appliance-core), compiled in but inert until HOLGER_APPLIANCE=1 is set, so everything measured above still holds for a plain start. The ceremony, the container shape, and how to prove both: .nornir/appliance-for-idiots.md.

3.4 ⚠ Every flag on that line is load-bearing

holger's default = ["mini", "online", "bench"]. So:

# DO NOT DO THIS
cargo build --release -p holger --all-targets

...does produce target/release/holger-mikro — and it is the wrong binary:

The filename is identical. The binary name does not tell you the tier. Only --no-default-features --features mini gives you the zero-egress build. Check what you actually got:

nm -C target/release/holger-mikro | grep -c 'reqwest\|ureq'   # must print 0
objdump -p target/release/holger-mikro | grep NEEDED          # must show no libssl

The mirror-image trap: --all-targets silently skips any [[bin]] whose required-features are not enabled. With default features that is holger-embedded, holger-full and holger-server-ui — cargo builds them without a word and exits 0. That is how five binaries went unbuilt for months. Always build a tier by --bin name with its own feature.

3.5 Alternative: build it through the build-thing engine

The engine materializes the same cargo line from the declaration, so you cannot get the flags wrong:

cd ~/git/workspace_nordisk/release        # any dir; the --config is what matters
nornir build --list-variants --repo holger \
      --config ~/git/workspace_nordisk/release/nornir.toml

Verified output (verbatim, pre-rename — the UI variant/package/bin names in these two lines are the ones of the day; see the §1 rename table):

declared build variants (/home/rickard/git/holger):
  embedded     cargo build -p holger --bin holger-embedded --features embedded --no-default-features --profile release
  mini         cargo build -p holger --bin holger-mikro --features mini --no-default-features --profile release
  full         cargo build -p holger --bin holger-full --features full --no-default-features --profile release
  ui           cargo build -p holger-ui --bin holger-ui --features gui --profile release
  henriks-mini-holger cargo build -p holger-ui --bin holger-ui-mikro --features mini --profile release
  agent        cargo build -p holger-agent-cli --bin holger-agent --profile release

Then build for real:

nornir build --variant mini --repo holger \
      --config ~/git/workspace_nordisk/release/nornir.toml

⚠ --config must point at a nornir.toml, not at build-variants.toml. nornir_config::load_explicit derives the repo root as the config path's three-up ancestor (…/workspace_x/release/nornir.toml → …/). Passing --config <repo>/.nornir/build-variants.toml resolves the root one level too high and cargo then fails with could not find Cargo.toml in /home/rickard/git. Listing still works with the wrong path; building does not.

Wiring check without compiling anything:

dwarves holger-build ~/git/holger

holger build target (holger-tiers) against /home/rickard/git/holger — 3 tier(s), 3 resolved, 0 missing:
  ✓ embedded → bin holger-embedded · cargo build …
  ✓ mini → bin holger-mikro · cargo build -p holger --bin holger-mikro --features mini --no-default-features --profile release
  ✓ full → bin holger-full · cargo build …

3.6 Alternative: the ⚒ Build Thing pane

dwarves-demo carries a ⚒ Build Thing tab (dwarves-demo/src/build_thing.rs) rendering a BuildThingDeck from facett-buildthing: a role-grouped variant menu with a linkage chip, a graph pane, and a result pane.

cd ~/git/dwarves
cargo run --release -p dwarves-demo
# → ⚒ Build Thing tab → role `server` → variant `mini` → Build

Selecting mini arms exactly the cargo line in §3.3 — its own --bin, its own mode feature, --no-default-features. The pane reads holger's .nornir/build-variants.toml (or, on a forge with no holger checkout, the nornir-buildspec-holger wasm plugin), so it cannot drift from the declaration.

The pane is the safer route precisely because it cannot make the §3.4 mistake.

4. Run it — the verified transcript

4.1 Generate a config and start

holger-mikro dev-pair --data /var/tmp/holger-mini \
      --grpc 127.0.0.1:19443 --http 127.0.0.1:19444

Holger dev-pair: /cache (caching crates.io mirror) + /sparring on sparse+http://127.0.0.1:19444/<repo>/index/  (config /var/tmp/holger-mini/dev-pair.ron)
[ERROR holger_server_lib] repository 'crates-io' DISABLED — backend init failed: repo 'crates-io' is 'rust-remote' … built WITHOUT the `online` feature …
gRPC server listening on 127.0.0.1:19443 [h2c (cleartext)]
Running. Press Ctrl+C to stop.
HTTP/OCI gateway listening on 127.0.0.1:19444 [http (cleartext)]

The crates-io DISABLED line is expected and correct on this tier — dev-pair generates a config with a crates.io pull-through mirror, which a zero-egress build cannot serve. The other two repos (cache, sparring) come up fine. If you don't want the error, write your own config with no rust-remote repo (§4.4).

Sockets, from ss -ltnp:

LISTEN 127.0.0.1:19444  users:(("holger-mikro",pid=…,fd=11))
LISTEN 127.0.0.1:19443  users:(("holger-mikro",pid=…,fd=12))

4.2 Prove the HTTP door serves

curl -s http://127.0.0.1:19444/sparring/index/config.json
# {"dl":"http://127.0.0.1:19444/sparring/crates","api":"http://127.0.0.1:19444/sparring"}

curl -s -o /dev/null -w '%{http_code}\n' 'http://127.0.0.1:19444/-/search?name=x'
# 200

4.3 Prove the gRPC door serves

No grpcurl needed — hand-frame a gRPC message (0x00 + 4-byte BE length + body):

printf '\x00\x00\x00\x00\x00' > /tmp/empty.grpc
curl -s --http2-prior-knowledge -X POST \
     -H 'content-type: application/grpc' -H 'te: trailers' \
     --data-binary @/tmp/empty.grpc \
     http://127.0.0.1:19443/holger.v1.AdminService/Health | xxd

grpc-status: 0
00000000: 0000 0000 0d0a 026f 6b12 0530 2e36 2e39  .......ok..0.6.9

status = "ok", version = "0.6.9". And AdminService/ListRepositories:

grpc-status: 0
cache · Rust
sparring · Rust

Two repos live; crates-io correctly absent (disabled, §4.1).

4.4 Full write→read round-trip

Uploading a real artifact via RepositoryService/PutArtifact:

grpc-status: 0
Artifact stored

and reading it straight back with FetchArtifact:

grpc-status: 0
"hello henrik" · application/octet-stream · 12 bytes

/var/tmp/holger-mini/sparring.znippy materialized on disk. The same round-trip through the HTTP door with properties:

holger-mikro properties set --config /var/tmp/holger-mini/dev-pair.ron \
      --repo sparring --name demo --version 1.0.0 --key henrik --value proved
curl -s 'http://127.0.0.1:19444/-/properties?repo=sparring&name=demo&version=1.0.0'
# {"henrik":["proved"]}

Note what this proves: the mini binary accepted a write, over gRPC, with no credential presented (the dev-pair config configures no auth). mini is a full read/write registry unless you configure otherwise. §5.

5. Getting an actually read-only mini

Read-only is derived from the config, not from the tier (server/lib/src/grpc.rs:211 — is_read_only() = the explicit override if set, else no repository is writable). The explicit override (HolgerGrpc::with_read_only_profile) is called by no production code, only a unit test — so in practice the route table is the whole story.

Writable (a rust repo with ron_writable_archive or ron_data_dir):

AdminService/ServerProfile → "dynamic" / "DYNAMIC / WRITABLE"
RepositoryService/PutArtifact → grpc-status 0, "Artifact stored"

Read-only — give every repo a sealed ron_archive_path instead:

(
    exposed_endpoints: [
        ( ron_name: "dev", ron_url: "127.0.0.1:19543", ron_http_url: Some("127.0.0.1:19544") ),
    ],
    storage_endpoints: [
        ( ron_name: "sealed-store", ron_storage_type: "znippy", ron_path: "/var/tmp/holger-mini/" ),
    ],
    repositories: [
        ( ron_name: "sparring", ron_repo_type: "rust",
          ron_archive_path: Some("/var/tmp/holger-mini/sparring.znippy"),
          ron_base_url: Some("http://127.0.0.1:19544"),
          ron_upstreams: [], ron_in: None,
          ron_out: Some(( ron_storage_endpoint: "sealed-store", ron_exposed_endpoint: "dev" )) ),
    ],
)

holger-mikro start --config sealed.ron

Verified, same binary:

AdminService/ServerProfile → "static" / "RIGGED FOR SILENT RUNNING / READ-ONLY"
RepositoryService/PutArtifact → grpc-status 7, "Repository is read-only"

Use absolute paths in the RON. Relative paths resolve against the server's cwd, and a wrong one fails with a bare Error: No such file or directory (os error 2).

6. Pointing at oden's forge

mini does not need the forge — it is a standalone server. The forge is only relevant if you want to drive builds on oden rather than compile locally.

Tokens live on oden at /home/rickard/.dwarves/token and /home/rickard/.nornir/token (mode 0600). Ask Rickard for yours — do not copy values into a file, a script, or a commit.

The CLI must be built with the gRPC client or thin mode refuses:

cd ~/git/dwarves
cargo build --release -p dwarves --bin dwarves --features server,git-source

(Without --features server you get thin mode (--server …) needs the gRPC client: rebuild with --features server; without git-source the key verb does not exist at all.)

Verified against the live server:

dwarves --server http://100.115.167.95:7900 \
        --token "$(cat ~/.dwarves/token)" doctor
# dwarves-server ok · v0.1.4 · 0 pipeline(s) · 0 run(s)

nornir side — same shape, --server works on every subcommand:

nornir build --list-variants --repo holger \
      --server 100.115.167.95:7878 --token "$(cat ~/.nornir/token)" \
      --workspace nordisk

(This invocation was run on oden, where the holger checkout is local, so it does not by itself prove the wire path — the dwarves doctor result above does.)

--token also falls back to $DWARVES_SERVER_TOKEN / $NORNIR_SERVER_TOKEN, then to <home>/.dwarves/token / <home>/.nornir/token — prefer the fallback over putting the value on a command line where it lands in your shell history.

dwarves --server … holger-build currently answers the server found no .nornir/build-variants.toml under ../holger — the live server has no holger checkout at that path. Use the fat form (dwarves holger-build ~/git/holger) against your own clone.

7. Troubleshooting — the failure modes actually hit

7.1 A required-features bin silently didn't build

Symptom: cargo build --all-targets exits 0 and the binary you wanted is not in target/release/. No warning, no error.

Cause: cargo skips any [[bin]] whose required-features are off. In this workspace that is holger-embedded (embedded), holger-full (full), holger-mikro (mini), holger-server-ui (gui), holger-server-ui-mikro (mini).

Fix: build by --bin name with its feature, per §3.3. Never trust --all-targets to cover the tiers.

Related and worse: because mini is in holger's defaults, --all-targets produces a holger-mikro that exists but links reqwest+ureq (§3.4). Always run the nm check.

7.2 SSH deploy key not registered with the forge

Symptom: clones/fetches fail; the forge's poll loop never advances.

Diagnose — dwarves key reads the same ladder gitio::clone_or_fetch authenticates with, so what it prints is what the fetch uses:

dwarves key path      # which rung answered, and whether it's a legacy path
dwarves key show      # the public line to paste into the forge's deploy keys
dwarves key generate  # mint ~/.dwarves/ssh/id_ed25519 (idempotent, never overwrites)
dwarves key probe ssh://git@git.gunnar.rs:2222/nordisk/holger.git

probe does a live SSH ref advertisement and distinguishes the four states. What a rejection looks like:

key rejected   ssh://git@git.gunnar.rs:2222/nordisk/holger.git  ssh publickey auth rejected for git@git.gunnar.rs  (1500 ms)

KEY REJECTED BY REMOTE
1 remote(s) refused this key — it is not registered there (or not on that repo).

probe is serial on purpose — a forge may rate-limit parallel handshakes with a reset that is not an auth failure and would paint the whole fleet red.

dwarves key path on oden currently resolves to ~/.nornir/ssh/id_ed25519 (a nornir-era legacy rung) rather than the zero-config default ~/.dwarves/ssh/id_ed25519. The tool says so; don't be surprised by it.

7.3 The warehouse lock

One process owns a warehouse at a time. oden runs a live dwarves-server systemd unit that holds /home/rickard/.nornir/workspaces/nordisk/builds/warehouse open. A fat client that tries to open it too will block or fail.

Rule: when the server is up, clients go thin — --server http://…:7900 (dwarves) or --server …:7878 (nornir) — and never open the warehouse directly.

None of this affects holger-mikro. holger has its own storage and never touches a nornir/dwarves warehouse (§2).

7.4 dwarves-full <anything> hangs

dwarves-full is the fat server binary, not a CLI. dwarves-full key --help never returns — it parks in a futex with 33 threads. Use the dwarves bin for CLI verbs and dwarves-full / dwarves-server only to run a server.

7.5 no build variants declared

nornir build autodiscovery walks up from cwd and, from ~/git/holger, lands on ~/git/.nornir/build-variants.toml, which does not exist. Pass --config + --repo explicitly (§3.5), or run the raw cargo line.

7.6 could not find Cargo.toml in /home/rickard/git

You passed --config <repo>/.nornir/build-variants.toml. Point --config at a nornir.toml instead — see the warning in §3.5.

7.7 Error: No such file or directory (os error 2) on start

A path inside the RON config did not resolve. RON paths are relative to the server's cwd, not to the config file. Use absolute paths.

8. Things the declarations get wrong

Recorded, not silently fixed. None of these blocks building or running mini.

1. nornir-buildspec-holger calls mini thin and says it reaches skade. shape_of("mini") returns ThinStandalone, commented "the stripped read-only server reaches a remote skade", and the crate's own tests assert it (assert_eq!(e("mini").linkage, Linkage::Thin)), plus assert!(e("full").runtime.contains("embeds skade")). Both are false: no tier links skade, and mini needs nothing running. build-variants.toml says fat and is right. A forge planning holger from the wasm plugin alone would tell you to stand up a skade first. It would be wrong.

2. mini is described as read-only. It is not. The TOML calls it "the stripped, read-only standalone server for airgap/red-zone". All eight gRPC services — including PutArtifact, PromoteArtifact, Replicate, AttachSbom, SetProperty — are registered unconditionally (server/lib/src/lib.rs:1085-1122), the CLI carries the write verbs, a live mini self-reports DYNAMIC / WRITABLE, and a PutArtifact succeeded. See §5.

3. mini and embedded are the same build. Both expand to ["auth-bearer", "auth-mtls"], grep -rn 'feature = "mini"' server/lib/src/ returns nothing, and the two cargo tree closures diff empty. The tiers are declared "mutually exclusive… each links a different dependency closure" — two of the three link the same closure. Only the binary name and the declared kind differ.

4. The wasm plugin knows five variants; the TOML declares six. nornir-buildspec-holger's the_five_declared_builds_are_holgers_real_ones asserts ["embedded", "mini", "full", "ui", "agent"]. The TOML has added henriks-mini-holger. A forge with no holger checkout cannot plan that variant.

5. henriks-mini-holger is not a leaner build. server/ui/Cargo.toml has mini = ["gui"] — it enables gui and removes nothing. Measured: cargo tree -p holger-server-ui --features mini and --features gui produce identical 573-crate closures. That closure contains reqwest, ureq, openssl, native-tls and skade. So the client called "mini" is the fattest artifact in the repo and the only one with a real skade edge — it is emphatically not an airgap companion to the mini server. The TOML admits the size half of this; it does not say the "mini" client is the one thing here that is neither lean nor zero-egress.

6. offers lists features that do not exist on the packages built. offers = [… "sign", "testmatrix", "served", …]. sign is a feature of holger-server-lib, served of holger-bench-scenarios — neither exists on holger or holger-server-ui. A variant naming either passes validation and then fails in cargo. sign, testmatrix and served are named by no variant at all.

7. [build.downstream] holger = ["skade"] has nothing behind it. No holger package depends on skade, optionally or otherwise. The allowlist permits an edge that cannot exist. (The TOML's own header now says this; the entry remains.)

8. The root Cargo.toml comment is stale. It says "The sibling znippy is on branch feat/plugin-read-api"; the checkout on oden is on master.

9. nornir_config::load_explicit mis-derives the repo root for any config not nested exactly two directories below the workspace root (§3.5). It is a silent wrong answer for --list-variants and a confusing cargo error for --variant.

Verified on oden, 2026-07-22, against holger main @ holger 0.6.9.