nordisk/nornir

The job ledger — reasoning

Class: reasoning. This file holds the why, the trade-offs, and every design we rejected for the job ledger. The descriptive shape we actually run is in jobs-design.md (which by rule mentions none of this); the CLI is in jobs-guide.md. If you want "why not an external queue?" — it's here.

Why a job ledger at all

Every nornir op already writes its result to a domain Iceberg table at the end (bench_runs, test_results, architecture_wiring, doc_exports, clone_events, …). What was missing was one place that answers a different question: "what's running right now / did it finish / how long did it take?" — the op lifecycle, orthogonal to the op's output. Without it, a long release or a stuck bench was invisible between "started" and "a row appeared in some table". The ledger exists to make every long-running op (≈1s–minutes) legible in one list, so the CLI and the viz can answer that question uniformly. That single payoff is why the contract exists; everything below keeps it true.

Why every long op is a job (not just some)

The rule is every box-occupying op self-registers — release, fetch, docs, scans, bench, test matrix, index/vector builds, coverage, container runs, and so on. The temptation was to track only the "important" ones. Rejected: the value of the ledger is that it is complete — a Jobs panel with holes ("why isn't the security scan showing?") is worse than no panel, because it teaches you not to trust it. The JOB-ENGINE-SWEEP audit (2026-06-23/24) existed precisely to close the gaps where a server RPC or a fat-CLI path did real work but wrote no ledger row. Wiring is cheap and best-effort (a missing handle never gates the op), so "track it" is the default and "don't" needs a reason.

Rejected: Iceberg rows for the lifecycle (redb instead)

The obvious move is to reuse the warehouse — an op is already going to write an Iceberg row, so why not track its lifecycle there too? We rejected it for the nature of the data:

• A job record is mutable (running → done|failed) and short-lived. Iceberg is append-only, so lifecycle-in-Iceberg means either two rows per job (start + finish) coalesced on read, or a fresh snapshot+manifest per transition. The latter is the classic small-files antipattern — a snapshot per start/finish would bury the catalog in metadata.
• redb overwrites the key in place: one row per job, mutated as it advances, no coalescing dance. So the split is deliberate — Iceberg stays the home for what an op produced; redb is the ledger that an op ran. result_ref links a job back to its Iceberg result row, keeping the two connected without merging them.

Rejected: an external queue / database (single-writer in-process instead)

A "real" job system reaches for Redis, Postgres, or a message queue. We rejected that entirely and chose an in-process, single-writer redb file:

• nornir already runs as either a fat local process or one long-lived server; the server is the natural single owner of a local file. Adding an external broker means a new daemon to deploy, secure, and keep alive — for a workload that is "a few dozen long ops on one box", not a distributed fan-out.
• The single-writer discipline is already proven twice in nornir (registry.redb, the katalog catalog.redb); reusing it means the same lock-tolerant read trick (copy the file aside when the writer holds the lock) and the same operational story, not a new one.
• It mirrors the warehouse's fat-XOR-thin split: a fat client writes the local file directly; a thin client submits each record over gRPC to the server, which owns its jobs.redb. The library never links tonic — the binary supplies the submit closure — so the engine stays a dependency-light leaf crate. An external queue would have forced a network dependency into that leaf.

Rejected: serializing everything (a class per kind instead)

The serial gate could apply to all jobs — one at a time, simple. Rejected: most ops (docs render, fetch, arch generate, container runs) are perfectly happy running alongside each other, and serializing them would cripple throughput for no gain. The real problem is narrower: some ops are box-saturating and only measure true in isolation — a benchmark sharing the machine with another benchmark measures noise, and the server is one long-lived process, so two benches launched together would otherwise both run. So the policy is per-kind (JobClass): Parallel is the default, and only genuinely isolation-demanding kinds are Exclusive (today just bench_run). Adding a heavy kind is one arm in class_of, so the exception stays an exception.

Rejected: an invisible wait (a visible queued state instead)

When an exclusive bench waits behind another, the easy implementation blocks on the semaphore before writing any row — so the waiting job is invisible until it starts. Rejected: a user who submits three benches and sees one row would think two were lost. Instead the waiting job is admitted as a queued row first, then blocks; it flips queued → running only when it acquires the gate. So N benches show as one running + (N−1) queued, draining one at a time — the user sees the queue. On promotion we re-stamp ts_start_micros so elapsed_ms measures real work, not the queue wait (a bench that waited an hour didn't take an hour to run).

Rejected: trusting ops to finalize (a Drop guard instead)

Relying on every code path to call finish/fail is how you get zombie running rows: a panic or an early ? return skips the terminal write, and the job looks stuck forever. So the JobHandle records failed ("dropped without finish/fail") in its Drop impl — the terminal write is guaranteed by the type system, not by discipline. A row is either terminal or its handle is still alive.

Why kind is a string, not an enum

An enum would be tidier and exhaustively matchable. We use a plain string so a new op family can adopt the ledger without a schema change or a recompile of the leaf crate — the kinds are conventional tags readers group/filter on, not a closed set. The cost (a typo'd kind is a silent new family) is bounded because the canonical kinds live as pub consts that call sites reference.

How we got here

The ledger started as the release/bench/docs tracking the server already did, then the JOB-ENGINE-SWEEP (2026-06-23) added the missing CLI reader (nornir jobs) and wrapped the untracked fat-CLI paths (coverage, local test matrix, arch generate, airgap). A second sweep (2026-06-24) wired the server RPC handlers that did real work but showed nothing in the Jobs panel (index snapshot, symbol scan, security scan, docs render/export/book, release gate). Container jobs came next as the generic "run a one-shot container and track it" seam, and the serial scheduler last, when running two server benches at once was found to poison both results. Each step is the same principle: if it saturates the box for a noticeable time, it should be a visible, self-finalizing row in the one ledger.