nordisk/heimdall

The heimdall command line

Heimdall is a defensive sandbox. Milestone 1 runs wasm jobs only.

heimdall run --job wasm --module <guest.wasm> --trust-root <root.der>... --in <dir> --out <dir>
             [--policy <file>] [--pin-out <file>] [--pin-blake3 <hex>]
             [--decoder <decoder.wasm> [--decoder-pin-blake3 <hex>]]
             [--job-id <32 hex>] [--nonce <64 hex>] [--param key=value]...
             [--seccomp generated|none] [--seccomp-sha256 sha256:<hex>]
heimdall verify-result <out>/result.signed --key <key file | key record>
             [--files <dir>] [--job-id <32 hex> --nonce <64 hex>]
heimdall profile probe|gen|check <workload> [--profiles <dir>] [--libc glibc|musl]
             [--binary <file>] [--only <run>]...
heimdall profile probe-exec --workload <name> --run <run> --libc glibc|musl --out <evidence.json>
             [--error-path] [--self-only] [--arg0 <s>] [--flush-secs N] -- <program> [args]...
heimdall profile probe-container --workload <name> --run <run> --out <evidence.json>
             --image <oci layout dir> --digest sha256:<hex> [--error-path]
             [--channel <host dir>] [--env K=V]... [--libc glibc|musl] [--uid N]
             [--wall-secs N] [--memory-mib N] [--pids N] [--cpus F] [--tmp-mib N]
             [--term-grace-secs N] [--flush-secs N] [--log <file>]
             [--shim <heimdall-container-shim>] [--work-dir <dir>]
             [--cgroup-root <dir>] [--cgroup-parent <name>] -- <argv inside the image>...
heimdall profile merge --out <file> [--workload <name>] <evidence.json>...

• --job native prints not in milestone 1 and exits 2.
• --module names the guest; its signature <guest>.wasm.p7s and hash pin <guest>.wasm.blake3 sit beside it and are verified against the --trust-root certificates before load. There is no unsigned mode: without a trust root the job does not start.
• --decoder makes the job a split guest (GATLING-WASM.md): --module is then the verifier and --decoder its decoder module, signed and pinned exactly like the guest (<decoder>.wasm.p7s, .wasm.blake3, and --decoder-pin-blake3 as the caller pin). The decoder is loaded first; any refusal of it says decoder refused: ... and exits 3 before anything runs. The policy key decoders sets how many decoder instances run at once. Verdicts, outputs and verify-result are the same as for the single-instance guest.
• --in, --out and --policy follow RUN.md and POLICY.md.
• --job-id and --nonce let the caller bind the result to its own request; both are random when absent and printed either way.
• --seccomp generated (the default of a release build) runs the guest in a worker process under the checked-in, generated heimdall-wasm-host seccomp profile for this build's arch and libc (unknown call = kill); the parent holds the key, re-checks the worker's outputs and signs. The result says Generated { profile_digest } (sha256: of the profile's .oci.json). A worker killed by the filter gives a signed Refused heimdall-seccomp-1, no outputs, exit 4. --seccomp none (the default of a debug build: the profiles are generated from the release build) runs it in-process with no filter (Protection::None). HEIMDALL_SECCOMP=generated|none sets the default; the flag wins. --seccomp-sha256 demands that digest: another one exits 3 before anything runs. No profile for this arch/libc with generated: exit 3. The worker gets only PATH=/usr/bin:/bin and RUST_BACKTRACE=0 as its environment (the probe's), none of the caller's.
• run prints pinned-key:, seccomp:, job-id:, nonce:, result: and verdict: lines on stdout. --pin-out also writes the pinned key record to a new file.
• verify-result checks the format, the pinned key id and boot id, the Ed25519 signature and the result's well-formedness; with --job-id and --nonce, that the result is for that job; and that the files in --files (default: files/ beside the result file, when it exists) are exactly the listed outputs with matching sha256 and length.

heimdall profile (spec §3)

The syscall profile of a workload is probed, generated and checked from profiles/<workload>/workload.toml (the binary per libc and the probe corpus; format in heimdall_probe::workload). See SECCOMP.md.

• probe runs every corpus run in learning mode (seccomp USER_NOTIF, log and allow) and writes <arch>-<libc>.probe.json: every distinct call with the tests that made it.
• gen reads the probe file and writes <arch>-<libc>.rs (a const table), .json (the same profile plus the review: near-miss calls, W|X mappings left out, bare allows) and .oci.json. A person reviews the diff before it is committed.
• check probes again and exits 1 when a call is outside the checked-in .json, printing new call <call> (kill_process), made by test <run>, or when .rs/.oci.json do not match .json.

Exit codes: 0 done / check passed, 1 check failed, 2 usage error, 3 the workload, binary or a probe run failed (a run that times out or ends with another exit code than its expect_exit fails the probe, naming the run).

Evidence from where a program really runs

• probe-exec runs ONE program on the host (its own stdio, environment, cwd) under the learning filter and merges what it calls into --out.
• probe-container runs ONE program inside Heimdall's own protected container (below), so a workload driver outside its process tree can drive it.
• merge joins evidence files of one arch and libc.

The evidence file of both is the <arch>-<libc>.probe.json shape of probe (heimdall_probe::Observations), merged under a lock (<out>.lock) with what other runs wrote there, rewritten (marked "exit": "running") whenever a new call shows up, at most every --flush-secs (5), and finally with the run's exit (exit 0, killed by signal 9). It goes into merge and, copied to profiles/<workload>/<arch>-<libc>.probe.json, into gen. Both end with the program's exit: its code, or the same signal raised on heimdall (shell status 128 + n). Exit 2 is a usage error, 3 a host error before or around the program (no evidence is claimed for a program that never started).

heimdall profile probe-container (the contract)

The program runs as a normal HeimdallRunner native job: user, mount, pid, ipc, uts and cgroup namespaces, an EMPTY network namespace, the read-only root of the image --digest from the OCI layout --image (manifest, config and every layer verified by sha256 and unpacked fresh at the start), tmpfs /tmp, Landlock, no capabilities, no-new-privs, rlimits and cgroup v2 limits (--memory-mib 1024 with no swap, --pids 256, --cpus 1, --wall-secs 86400). Its environment is the image's plus --env. Its stdio is /dev/null; --log appends its stdout and stderr to a host file. Two deviations, stated here and in the status lines, never signed:

1. seccomp: the LEARNING filter (default SECCOMP_RET_USER_NOTIF, every call recorded and allowed, clone3 answered ENOSYS: the filter and prelude of probe and probe-exec) INSTEAD of the fixed profile or a generated one, which would kill the calls this run is meant to learn. No signed result is made; nothing here is a Protection::Generated claim.
2. --channel <host dir> (optional): bind-mounted read-write, nosuid, nodev, noexec at /run/heimdall-channel, where Landlock allows read, write and making unix sockets. A driver outside connects to the sockets the program makes there (postgres -k /run/heimdall-channel, curl --unix-socket <dir>/x.sock). The network namespace stays empty. Without --channel there is no read-write mount besides /out and /tmp.

--uid N runs the program as uid and gid N (for programs that refuse root, e.g. postgres): before Landlock the job's init enters a nested user namespace where the job's root (your uid on the host) is N; for those maps the init holds CAP_SETFCAP (never in the bounding, inheritable or ambient set) and drops every capability right after them, so the program holds none. --tmp-mib sizes /tmp (counted in --memory-mib); files there may be that large.

Libc: --libc, else read from the program's ELF inside the image (PT_INTERP ld-musl / ld-linux, the glibc ABI note of a static binary; a #! script: its interpreter's); exit 2 when neither tells.

stdout carries one JSON object per line:

{"event":"started","pid":12345,"seccomp":"learning","net":"empty","channel":"/run/heimdall-channel","libc":"glibc"}
{"event":"exited","exit":"exit 0","calls":143,"unknown":0,"out":"/path/x86_64-glibc.probe.json","seccomp":"learning","landlock":"full","term_killed":false,"wall_killed":false,"oom_kills":0}

started comes when the program is about to start under the filter (pid: its host pid; channel: null without one); exited after the final rewrite of --out (calls: distinct calls of this run). SIGTERM, SIGINT and SIGQUIT to heimdall go on to the program (pid 1 of its namespace: it sees only the signals it has a handler for); --term-grace-secs (10) later the job is killed (term_killed). SIGHUP, SIGUSR1 and SIGUSR2 go on as well and stop nothing.

Host setup: the heimdall-container-shim binary (heimdall-container, feature youki; --shim, else $HEIMDALL_CONTAINER_SHIM, else next to heimdall), user namespaces, Landlock, seccomp USER_NOTIF (5.5), and cgroup v2 with the memory, pids and cpu controllers delegated to you below --cgroup-root (/sys/fs/cgroup) / --cgroup-parent (heimdall); the runner refuses to start when one is missing (strict mode, spec §10). --work-dir (default $XDG_RUNTIME_DIR/heimdall, else /tmp/heimdall-<uid>) holds the per-job bundle, removed after the run.

heimdall trace (dynamic-trace jobs)

heimdall trace run --layout <dir> --image <sha256:..> [--tracer <heimdall-trace>] --in <dir> --out <dir>
                   --result <file> [--work <dir>] [--pin-out <file>] [--wall-secs 120] [--memory-mib 1024]
                   [--pids 256] [--max-bytes 4194304] [--env K=V]... [--writable <dir>]... -- <command>...
heimdall trace verify <result> --key <pinned key> --files <out dir> [--max-bytes <n>]
heimdall trace diff <baseline.json> <trace.json> [--max-bytes <n>]

run traces the command in a HeimdallRunner job with no network (heimdall-container, its shim found as for every HeimdallRunner) and prints the outcome as one JSON line ({"outcome":"traced",...} or {"outcome":"unknown","reason":...}); --tracer first adds the static tracer to the image as one more layer. verify checks a trace job's signed result and trace.json. diff prints what the trace does beyond the baseline (JSON). Exit codes: 0 traced / clean diff, 1 Unknown / new behaviour or a trace not complete, 2 usage, 3 a file, the key or the runner could not be read or started. See API.md.

Exit codes