nordisk/heimdall

Rusternetes + OVN: three rows, two hosts

Lane TUNNR-OPEN, 2026-11-07. Stock ovn-kubernetes (release-1.4, by digest) on two-node clusters. 3 measured runs per row, medians over the runs. Row P ran on dwarf, rows B and A on oden (see "Host and load per run"). Every bench pod ran under a seccomp profile learned from the bench itself (see "Seccomp profiles"). Raw JSON per run: results/<row>/all<n>.json, beside this file, with each row's evidence receipt. This page is a copy of the bench report kept in tunnr (a private repo, commit 07ddaa8), where the bench scripts (scripts/ovn-bench/) live; the tables below are printed from the JSON here.

• P = Rusternetes + podman + OVN, B = k3s + OVN (both on Debian 13), A = Rusternetes + OVN + youki on tunnr.
• The same stock ovn-kubernetes manifest on every row. Only its CA ConfigMap differs, because each cluster has its own API server CA.
• Functional = pod-to-pod across nodes and on one node, ClusterIP, NetworkPolicy allow, deny, and deny via ClusterIP.
• Loaded RAM: the mean of MemTotal - MemAvailable per node over the Iggy steady window, sampled every 5 s. Peak: the highest sample in that window. Iggy msgs/s: the messages both consumers polled, each divided by its own run time, summed (the producers send 10 000 msgs/s in all, so about 20 000 means both consumers kept up).
• On row P the Iggy consumers reached Running 15-92 s after they were created (podman starts the init container and then the main one), so they also read a backlog and their msgs/s is above the live rate (24 846 / 22 166 / 53 800 over the 3 runs). Read that cell as "kept up". On rows B and A the consumers ran within 2.3 s.
• Rows B and A ran on oden at the same time on separate cores and bridges, and row P ran on dwarf, a smaller shared host. Compare P with B and A as different hosts, not as one host.

Host and load per run

• dwarf: 12 threads, shared with other lanes; the P VMs are pinned to cores 4-11. Before each boot the host must have load1 <= 6 and no compiler running.
• oden: 32 threads. A 12-core graph consume chain and an 8-core build lane ran there during these runs. The B VMs are pinned to cores 16-23, the A VMs to 24-31. The quiet-host check only records the host state there, so the load columns show what else ran.

Rows

Rows P and B run on the same base image with the same VM shape. They differ only in the Kubernetes layer. On every row the kubelet runs a pod that names no profile under the runtime's default profile (seccompDefault).

Method

• Two VMs per row (control plane + worker), 4 vCPU / 3 GB each, q35 + KVM -cpu host. The cluster link is a multiqueue tap on a host bridge with vhost-net.
• Each row has base disks: installed once, OVN applied once so its images are on the nodes, the bench profiles laid, then a clean power-off and the commit. Each run boots fresh copies of the base disks.
• Nodes ready / OVN ready: from the first VM boot until every Node is Ready with a heartbeat newer than the boot, and until every ovn-kubernetes container is Ready and was started after the boot.
• Idle RAM: MemTotal - MemAvailable per node after 60 s idle, read by a pod on that node.
• Warm pod start: 5 busybox pods (image cached) on the worker, create → Running with an IP. Cold: 5 nginx tags never pulled on that node, from Docker Hub.
• Network: iperf3 TCP pod to pod, 2 × 10 s per direction, across nodes (control plane → worker) and on one node.
• Memory under load: Apache Iggy 0.9.0 (see "Workload" below), a 120 s steady window that starts 30 s after the consumers run.
• Teardown between the bench's pods: 5 s.
• The windows are shorter than in the earlier runs (idle 180 s → 60 s, iperf3 3 × 20 s → 2 × 10 s, Iggy window 180 s → 120 s). Memory settles well within the shorter idle window.
• The image checks of the earlier runs are not part of this round: the ob9 control was dropped (its seed token had expired), and the ob13 image check was skipped.

Seccomp profiles: learned from the bench itself

Every bench container states a Localhost seccomp profile, the same five files on all three rows: busybox, iperf3, nginx-alpine, iggy-server and iggy-bench. A call that is not in a profile kills the process.

• Each profile is learned from the bench itself. One full bench run per row in learning mode (func, idle, warm and cold pod starts, iperf3, the Iggy load), with the bench's own pod specs and commands, across both nodes of the real cluster. In that run the five profile files hand every call to an agent pod on the node, which records it. The run's learning profile allows the container runtime's own calls between its filter install and the program's start (none for crun; write for runc; write sendmsg mmap munmap brk madvise mremap membarrier for youki), and the record of each container names the runtime build that started it.
• The evidence of the three learning runs is merged per program, and the profile is generated from it with the generator's tools only. No profile is hand-written, and none comes from a stand-in workload.
• cargo xtask evidence <row> checks each row before a medium is composed or a run starts. It refuses a profile learned outside the bench, a hand-made profile, a profile learned under another runtime build than the one the row's nodes run, and a kernel without an option a learned call needs. It writes the row's receipt; the measured runs need it, and each run's results directory keeps a copy.
• For the C programs (iperf3, busybox, nginx) no static cross-check applies yet, so the learning run is the only coverage. A code path that the bench never ran is not in the profile.
• On row P the kubelet's Docker-API backend writes each container's /etc/hosts with an exec into the container. That exec runs under the container's profile, so its calls are in the learned profiles too.

• Learning runs: P on dwarf (crun 1.21), B on oden (k3s runc 1.4.2), A on oden (the learning medium ob13-learn). Each passed func 6/6. Every container run carries its runtime record, except the /etc/hosts exec on row P.
• The allowed calls hold the runtimes' own calls named above and the kernel's base set (restart_syscall, rt_sigreturn, exit, exit_group). A few learned calls become fixed rules instead of allows, for example clone3, which every profile answers with ENOSYS so that the libc falls back to clone, whose flags a filter can check.
• The learning profile answers clone3 with ENOSYS too, so the clone fallback is learned. An earlier measured round, on profiles learned while clone3 still went through, killed iperf3 at its first thread (clone) on all three rows; it is not counted.
• Receipts: rusternetes-node e8db242c…, debian-p 3e980547…, debian-b d491eca7… (.nornir/evidence/<row>/).

What was enforced

Per row, from the seccomp audit pods of the counted runs (a privileged pod per node that reads every process's Seccomp mode and filter count; mode 2 = filtered) and the row's evidence receipt. On tunnr a pod's process also carries the filters of the node programs above it (PID 1, Rusternetes, the shim), so it shows 4 filters; on the Debian rows it shows the pod's one.

P: Rusternetes + podman + OVN: what was enforced

Evidence receipt debian-p sha256 3e980547db34f5d6, profile generator 0.4.3 (f093633c1166), kernel Debian 13 genericcloud 6.12.115+deb13.1-cloud-amd64 (the base image's kernel, from the nodes' boot logs).

Exceptions, as the receipt lists them:

• [privileged] OVN ovnkube-controller: privileged, Unconfined — ovn-kubernetes' ovnkube-node DaemonSet: privileged; Kubernetes runs a privileged container unconfined
• [privileged] OVN ovs-daemons: privileged, Unconfined — ovn-kubernetes' ovs-node DaemonSet: privileged; Kubernetes runs a privileged container unconfined
• [runtime-default] OVN nb-ovsdb: the runtime's default profile — stock ovn-kubernetes manifest, no profile; podman's default seccomp profile (the Docker-API backend passes no SecurityOpt for a container without one) — not measured (needs the cluster)
• [runtime-default] OVN sb-ovsdb: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovn-northd: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovn-controller: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovs-metrics-exporter: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovnkube-cluster-manager: the runtime's default profile — as nb-ovsdb
• [not-run] socat: not run — bench.py --fwd (a local registry forward, MODE=local) only; these runs never start it
• [privileged] seccomp audit pods: privileged, Unconfined (evidence only) — the evidence tool itself: a privileged hostPID busybox pod per node that reads every process's /proc//status (Seccomp, Seccomp_filters); started after the idle and the loaded-memory windows

B: k3s + OVN: what was enforced

Evidence receipt debian-b sha256 d491eca7f02de25e, profile generator 0.4.3 (f093633c1166), kernel Debian 13 genericcloud 6.12.115+deb13.1-cloud-amd64 (the base image's kernel, from the nodes' boot logs).

Exceptions, as the receipt lists them:

• [privileged] OVN ovnkube-controller: privileged, Unconfined — ovn-kubernetes' ovnkube-node DaemonSet: privileged; Kubernetes runs a privileged container unconfined
• [privileged] OVN ovs-daemons: privileged, Unconfined — ovn-kubernetes' ovs-node DaemonSet: privileged; Kubernetes runs a privileged container unconfined
• [runtime-default] OVN nb-ovsdb: the runtime's default profile — stock ovn-kubernetes manifest, no profile; RuntimeDefault (containerd's default profile; k3s kubelet-arg seccomp-default=true) — not measured (needs the cluster)
• [runtime-default] OVN sb-ovsdb: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovn-northd: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovn-controller: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovs-metrics-exporter: the runtime's default profile — as nb-ovsdb
• [runtime-default] OVN ovnkube-cluster-manager: the runtime's default profile — as nb-ovsdb
• [not-run] socat: not run — bench.py --fwd (a local registry forward, MODE=local) only; these runs never start it
• [privileged] seccomp audit pods: privileged, Unconfined (evidence only) — the evidence tool itself: a privileged hostPID busybox pod per node that reads every process's /proc//status (Seccomp, Seccomp_filters); started after the idle and the loaded-memory windows

A: Rusternetes + OVN + youki (tunnr): what was enforced

Evidence receipt rusternetes-node sha256 e8db242c4d816f40, profile generator 0.4.3 (f093633c1166), kernel tunnr-kernel-k8s 6.12.104.

Exceptions, as the receipt lists them:

• [hand-written] node floor: HAND-WRITTEN, not measured — the node PID 1's floor (361 calls): the PID 1 and korp-start cannot run under the USER_NOTIF probe (each must be PID 1, one listener per tree); measuring it needs the LOG-record path or a split PID 1
• [approved-edit] Rusternetes: measured, then relaxed + seccomp added by hand — generated from the probe medium's evidence, then edited by the approve script: --relax (calls lose their measured argument conditions) and --add seccomp (Rusternetes installs its children's filters under its own)
• [approved-edit] youki shim (wait phase): measured, then relaxed by hand — generated from the probe medium's evidence, then the approve script --relax
• [approved-edit] CSI node plugin: measured, then relaxed by hand (not used by the bench) — generated from the probe medium's evidence, then the approve script --relax
• [check-only] tunnr/io-uring.json: HAND-MADE, checks only — moby's RuntimeDefault + io_uring_setup/enter/register, written by hand for rusternetes-check.py io-uring and Postgres; no bench pod names it
• [check-only] tunnr/proof-kill-mkdir.json: HAND-MADE, checks only — rusternetes-check.py seccomp's proof profile (mkdir kills); no bench pod names it
• [privileged] OVN ovnkube-controller: privileged, Unconfined — ovn-kubernetes' ovnkube-node DaemonSet: privileged; Kubernetes runs a privileged container unconfined
• [privileged] OVN ovs-daemons: privileged, Unconfined — ovn-kubernetes' ovs-node DaemonSet: privileged; Kubernetes runs a privileged container unconfined
• [runtime-default] OVN nb-ovsdb: RuntimeDefault (seccompDefault) — stock ovn-kubernetes manifest, no profile; seccompDefault puts it under RuntimeDefault (moby's list) — not measured (needs the cluster)
• [runtime-default] OVN sb-ovsdb: RuntimeDefault (seccompDefault) — as nb-ovsdb
• [runtime-default] OVN ovn-northd: RuntimeDefault (seccompDefault) — as nb-ovsdb
• [runtime-default] OVN ovn-controller: RuntimeDefault (seccompDefault) — as nb-ovsdb
• [runtime-default] OVN ovs-metrics-exporter: RuntimeDefault (seccompDefault) — as nb-ovsdb
• [runtime-default] OVN ovnkube-cluster-manager: RuntimeDefault (seccompDefault) — as nb-ovsdb
• [not-run] socat: not run — bench.py --fwd (a local registry forward, MODE=local) only; these runs never start it
• [privileged] seccomp audit pods: privileged, Unconfined (evidence only) — the evidence tool itself: a privileged hostPID busybox pod per node that reads every process's /proc//status (Seccomp, Seccomp_filters); started after the idle and the loaded-memory windows
• [policy] kernel.io_uring_group=2900, kernel.io_uring_disabled=1 (payloads/rusternetes-node src/iouring.rs) — io_uring is compiled in for the measured io_uring_* calls (iggy-server) and off by default; a pod opts in with supplementalGroups [2900] + a profile allowing the calls (docs/io-uring.md); a policy, not a syscall need
• [policy] KUBELET_SECCOMP_DEFAULT=true (payloads/rusternetes-node src/supervise.rs) — a pod that names no profile (the sandbox, the OVN pods above) runs under RuntimeDefault, not unconfined
• [policy] CONFIG_BPF_SYSCALL (the k8s option's typed row beside the generated one) — on cgroup v2 the device controller IS a BPF program (BPF_PROG_TYPE_CGROUP_DEVICE), attached per container by youki/runc: a design need of every pod, stated whether or not a run made the bpf call
• [policy] CONFIG_MULTIUSER (the row's typed row beside the generated one) — lifts the appliance-boot contract's one-user default for this row (pods run as many UIDs; USER_NS and the cgroup controllers depend on MULTIUSER): a decision of the row, stated whether or not a run called setuid

What the rows needed in Rusternetes (each fix test first)

• hostPID / hostIPC (0a978bdb on node/tunnr-open, d649b97b on node/podman-seccomp): the PodSpec fields now use their upstream names. Before this, the default strict decoder refused a pod with hostPID: true ("unknown field spec.hostPID"), so the learning agents and the seccomp audit pods could not start on rows P and A.

• For row P (fork node/podman-cni, then node/podman-seccomp):

  • c3ea074a: KUBELET_DOCKER_HOST_NETNS=1. The Docker-API backend joins the network namespaces the kubelet makes, so CNI is used with a podman service on the node.
  • 50ce0275: every image pull runs on its own task, one per image. Before this, the kubelet's 30 s pod-start timeout cancelled podman's pull of the 1 GB OVN image.
  • 53aa164f: a volumeMount's mountPropagation reaches the bind (Bidirectional = rshared, HostToContainer = rslave). Without it, ovnkube-node never saw the pod network namespaces.
  • 73f605ec: a kubelet that registers again keeps its Node's metadata. Before this, a reboot dropped ovn-kubernetes' node-subnets annotation.
  • 3a5a436f: KUBELET_DOCKER_SECCOMP_BY_PATH=true. podman's Docker-compatible API opens a seccomp option as a file, so the backend names a Localhost profile by its path.

• For the learning run on row A: the youki shim built with the seccomp-listener feature (libcontainer on libseccomp), which can hand a profile's notify listener to the agent. Only the learning medium (ob13-learn) has it; the measured medium ob13 ships the node's own shim.

Known open issue

ovn-kubernetes' pod-annotation PATCH sometimes never returns on Rusternetes. The logical port is then missing and that pod starts late or not at all. It showed once in this round, in row A's first learning run (one warm pod start of 120 s), and in none of the counted runs.

Workload (memory under load)

• Apache Iggy 0.9.0, image docker.io/apache/iggy@sha256:b35fc2841964f2962a5f164d33b3ae3a9de99a66920defd3cd8a1c039470602d. iggy-bench built from apache/iggy at the image's revision (362bd782), static musl.
• One iggy-server pod on the worker (TCP only, HTTP / QUIC / WebSocket off), a ClusterIP Service in front. It runs under its learned profile on every row, in the io_uring group 2900 (only the tunnr kernel gates io_uring by group).
• Two producer pods (one per node): iggy-bench pinned-producer, 1 stream, 1000-byte messages, 100 per batch, rate-limited to 5 MB/s each, 15 000 batches (300 s).
• Two consumer pods (one per node), started 10 s after the producers run, each reading all 30 000 batches of both producers. The control-plane producer and consumer reach the server across nodes over OVN.
• The iggy-bench binary is served inside the cluster by a busybox httpd pod on the control plane; each bench pod's init container fetches it.