bench/floor-cost/README.md · nordisk/heimdall

floor-cost: what a call costs under the floor

Heimdall is a defensive sandbox. This bench measures the per-call cost of what is built of the graph filter (docs/v2/graph-filter.md §12 step 10, without the follower):

the static floor (seccomp BPF, the profile behind the launcher's prelude, default KILL_PROCESS), as a launch installs it;
the floor with three answers (filter::floor_three: never, always, brokered), one program;
static mode with a broker as a container job runs today: the broker filter under the static floor, two programs (stacked below);
the supervisor round trip of a brokered call (openat): answered with CONTINUE (the bare round trip, what the supervisor-held variant of §8 pays per position-dependent call), and answered by heimdall-broker's Broker (it opens the file beneath the job's in/ and injects the descriptor).

The follower (eBPF) is not built, so its cost (t_follow, t_idle) is not measured.

Method

crates/heimdall-broker/examples/floor_cost.rs, run by measure.sh:

Two floors: abcdf, nornir's A { B C D }* F fixture, probed (heimdall profile probe, 3 runs) and its profile generated (heimdall profile gen; the profile is in results/); and heimdall-wasm-host, the wasm host worker's checked-in profile (profiles/heimdall-wasm-host/x86_64-glibc.json).
Per row a fresh child, pinned to CPU 2, installs the programs and then makes only the measured call, again and again: a warm-up of 100,000 calls (2,000 for a brokered call), then 1,000,000 calls (20,000 for a brokered call), each timed alone with clock_gettime(CLOCK_MONOTONIC) (the vDSO, no system call). Every row runs 7 times; the numbers are over all samples, and run_medians_ns in the JSON gives each run's median. The supervisor thread runs on CPU 3.
The calls: each fixture's own calls that are harmless to repeat. abcdf: its five marker calls (bare allows), mprotect(PROT_READ) and openat(O_RDONLY|O_CLOEXEC) (argument rules). The wasm host: gettid, sched_yield (bare), prctl(PR_SET_NAME), mprotect(PROT_READ), openat (argument rules).
The timer reads are in every row, the baseline too: added is a row's median minus the median of the same call with no filter. The clock's step is about 10 ns, so the added cost of a fast call is ±10 ns.
Beside each row: the BPF instructions the floor runs for the call, and whether the kernel answers it from its action cache (Linux 5.11+: a call that every installed filter allows whatever its arguments are is allowed from a bitmap, and the program does not run).

Run it: bench/floor-cost/measure.sh <out dir> (privileged podman, the toolchain image; FLOOR_COST_ARGS, FLOOR_COST_CPU, FLOOR_COST_SUP_CPU).

Results (2026-11-08, oden)

Host: AMD Ryzen Threadripper PRO 3975WX (32 cores, no SMT), kernel 7.0.0-42-generic, clocksource tsc, in a privileged rootless podman container with no seccomp filter of its own. The host was busy: load average 52 to 83 over the run, all 32 CPUs in use by other work, and the bench ran at nice 10. Medians below are from results/2026-11-08-oden/ (run 3); two earlier runs the same day (load 34 to 94) gave the same medians within 10 ns, and for the brokered call within the spread given.

what	call	BPF insns	cache	median, no filter	median, filtered	added	p99, filtered
static floor, abcdf	`getppid` (bare allow)	33	yes	100 ns	120 ns	+20 ns	150 ns
static floor, abcdf	`mprotect` (argument rule)	48	no	211 ns	271 ns	+60 ns	330 ns
static floor, abcdf	`openat` (argument rule)	50	no	1,833 ns	1,953 ns	+120 ns	2,395 ns
static floor, wasm host	`gettid` (bare allow)	37	yes	100 ns	120 ns	+20 ns	140 ns
static floor, wasm host	`prctl` (argument rule)	59	no	210 ns	271 ns	+61 ns	330 ns
static floor, wasm host	`mprotect` (argument rule)	52	no	220 ns	271 ns	+51 ns	340 ns
three answers, abcdf	`mprotect`	50	no	211 ns	261 ns	+50 ns	330 ns
three answers, wasm host	`prctl`	61	no	210 ns	271 ns	+61 ns	331 ns
stacked, wasm host	`prctl`	101	no	210 ns	291 ns	+81 ns	301 ns
three answers, abcdf	`openat`, CONTINUE	52	no	1,833 ns	6,292 ns	+4.5 µs	1.0 ms
three answers, abcdf	`openat`, broker	52	no	1,833 ns	12,043 ns	+10.2 µs	1.5 ms
three answers, wasm host	`openat`, CONTINUE	58	no	1,834 ns	6,542 ns	+4.7 µs	1.1 ms
three answers, wasm host	`openat`, broker	58	no	1,834 ns	12,394 ns	+10.6 µs	1.0 ms
stacked, wasm host	`openat`, CONTINUE	67	no	1,834 ns	6,372 ns	+4.5 µs	122 µs
stacked, wasm host	`openat`, broker	67	no	1,834 ns	9,918 ns	+8.1 µs	70 µs

What the numbers say:

A bare allow costs about 20 ns, the same with one program or two: the kernel answers it from its action cache, so this is the cost of entering seccomp at all, not of the program.
A call with argument rules runs the program: about 50 to 60 ns for 48 to 61 instructions, and about 80 ns for the two stacked programs (90 to 101 instructions). The floor with three answers costs what the static floor costs: it differs only in the brokered calls' rules.
A brokered call costs microseconds: the bare round trip to the supervisor (CONTINUE) adds about 4.5 µs to an openat of 1.8 µs; the broker, which opens the file itself and injects it, adds about 8 to 11 µs. Most per-run medians were 6.2 to 7.4 µs (CONTINUE) and 9.8 to 13.8 µs (broker); runs that met a burst of the host's other work had medians up to 19 µs (CONTINUE) and 107 µs (broker).
The p99 of a brokered call is the host's, not the mechanism's: two threads must be scheduled for each call, and on this host every CPU was busy. In quieter stretches the p99 was 8 to 20 µs; under the load it reached 1 to 2 ms. The p99 of the filtered fast calls stays within 60 to 120 ns of the median.
Not measured: the follower (t_follow, and t_idle on the host's other processes), because it is not built.