nordisk/heimdall

The network observer and the per-job open-file limit (Heimdall 0.4)

Heimdall is a defensive sandbox. This note is the dataflow and the bounds of the two 0.4 additions to the native runner, as the code on main has them. The contract (types, fields, errors) is in ../API.md; this file says how the data moves and what it costs.

1. The open-file limit of one job

nofile(job) = job.policy.limits.nofile          if set
            = runner.nofile                     otherwise (HeimdallRunner::with_nofile, default 1024)

accepted  iff  NOFILE_MIN (16) <= nofile(job) <= NOFILE_MAX (2^20)
          and  nofile(job) <= hard RLIMIT_NOFILE of the runner process

A refused value is RunError::InvalidJob naming limits.nofile, before a bundle exists (HeimdallRunner::nofile_for, heimdall_core::check_nofile). The accepted value is written into the job's OCI config as RLIMIT_NOFILE soft = hard (heimdall_container::spec::set_nofile); the kernel enforces it inside the job, including on fds the broker injects with SECCOMP_IOCTL_NOTIF_ADDFD (the target task installs them under its own limit: EMFILE). heimdall-plain writes the same value from heimdall_plain::spec::build. Learning runs (probe-container) use the same rule.

2. The network observer: dataflow

 job process ──socket()/connect()──▶ seccomp USER_NOTIF
                                          │
                                          ▼
                    broker thread (in the shim)            decision first, answer the job
                    Broker::on_socket / on_connect ─────────────────────────────▶ job resumes
                          │ NetEvent (process read from /proc while the call waits)
                          ▼ EventSink::emit   (stamp: schema, job_id, seq, ts_ms; never blocks)
                    queue A  (cap = DEFAULT_QUEUE = 1024 events + drop markers)
                          │
                    writer thread (shim) ──JSON line──▶ shim stdout (pipe)
                                                            │
                    reader thread (runner) ◀────────────────┘
                    read_shim_stdout: event lines → EventSink::relay, the rest = the response
                          ▼
                    queue B  (cap = 1024 events + drop markers)
                          │
                    observer thread (spawn_observer) ──catch_unwind──▶ NetObserver::on_event

• Only the observer thread calls user code. The broker, the writer and the reader never wait on it: emit and relay take one mutex for a queue operation and return.
• An allowed connect is answered from its own thread (it can take a while); that thread emits the event with the connect's result after answering.
• Without an observer, BrokerRequest.observe is false: no queue A, no writer, no lines.

3. Bounds

Per job and per queue:

events held      <= cap                          (cap = 1024)
markers held     <= events held + 1              (a marker is only pushed after an event)
memory           <= (2·cap + 1) · size_of(NetEvent with its strings)
broker cost      = one /proc read (status, exe, cmdline) + one lock per decision, only with an observer

Drops: when queue A or B holds cap events, the next event is not queued; the tail dropped marker counts it (consecutive drops share one marker, a relayed marker adds its count). So for every job, at the observer:

Σ over delivered events (1)  +  Σ over delivered markers (dropped)  =  events the broker decided

once both queues are drained (tests: seqs_seen_strictly_increase_and_markers_account_for_every_gap, a_stuck_observer_never_holds_the_job_and_learns_how_many_it_lost).

Order: seq strictly increases in delivery order; a marker carries the seq of the last event it stands for, so the next delivered event has a higher one.

The end of a job: the shim drops its sink when the broker returns and joins the writer (which drains queue A). The runner drops its relay sink when the shim exits and waits at most OBSERVER_GRACE (2 s) for the observer thread; a slower observer keeps its thread and still receives the rest, or the marker that counts it.