nordisk/nornir

The lens chain converges — "make current" jobs

Design note for the lens/feed CHAIN, written 2026-09-10 after the chain was measured starving on oden. It refines the chain in src/lens_chain.rs (lenses are jobs, results are tables). Nothing about a fold changes; what changes is what a chain JOB means.

What was measured

oden, 2026-09-10 19:00–19:50Z, nordisk, 20 members, agents pushing every few minutes:

The panes were reading lens tables that were essentially never complete, and the one button the UI has was refused almost every press.

Why — three rules that were each right alone

1. Every republish moves the generation. generation() hashes the snapshot id of every lens-input table; a republish appends to several of them, so the string moves on every republish even when one repo changed one file.
2. A newer generation cancels the pending remainder of the older one. Correct if a job meant "fold at generation G" — but see below, it never did.
3. The re-submitted chain always starts from lens #1. Thirteen fresh submissions in LENSES order, FIFO: change_risk runs at every generation; generic_cost never reaches the head of the queue before the next republish cancels it.

Plus a fourth that halves throughput: the worker defers to a running republish (machine_busy), with a 10-minute ceiling per job — under a continuous republish that is one lens per ten minutes.

The insight

run_job reads the LIVE tables. It does not pin a snapshot; the snapshot_id on the job is only the label written to lens_runs. So cancelling a pending job and re-queuing it at the new generation never bought correctness — the pending job would have folded exactly the same rows. It only reset the queue order.

Therefore a chain job does not mean "fold lens L at generation G". It means "make lens L current". That single change of meaning removes the starvation without touching the republish cadence and without adding a scheduler to jera.

The design

Keys carry no generation

lens:{ws}:{repo}:{lens}[:{params}] and feed:{ws}:{repo}:{feed}. A submit while the same key is pending is Duplicate — a no-op — so eleven republishes in a row leave the queue exactly as it was, draining from wherever it had got to. ChainSpec::generation is a constant for chain jobs: nothing supersedes, nothing is Stale.

The worker folds at the generation current WHEN IT RUNS

run_job reads generation(wh) first, and short-circuits if lens_runs already holds this lens at that generation (the check that was already there, now against the truth rather than the label). The lens_runs.snapshot_id written is the generation the fold started at. run_feed_job does the same with feed_generation.

Re-arm on duplicate-while-RUNNING (jera)

The one hole in the above: a republish that lands while lens L is mid-fold submits L, gets Duplicate (L is live), queues nothing — and L's row is stamped with the generation it started at, which is now old. Without more, L stays stale until the NEXT republish. jera's ChainQueue therefore re-arms: a duplicate submit against a RUNNING key marks the key; when that run finishes the queue pushes one fresh pending copy (new job id, queued row). It costs nothing when the generation did not move (run_job skips in milliseconds) and closes the hole by construction when it did.

The defer ceiling is cumulative, not per job (jera)

await_quiet measures the domain-predicate deferral from the moment the box was last quiet, across jobs, instead of restarting the 10-minute clock for every job. A chain held off for ten minutes by a republish then drains; it goes back to yielding the next time the predicate goes false and true again. The EXCLUSIVE gate (a bench) is unchanged and still never expires.

Repopulate takes its own lock, not the fetch gate (nornir)

The populate WorkspaceGate is held for the whole 20-member fetch (~40 s of every 60 s tick) and the whole republish (~3 min). Repopulate needs neither: it drops derived tables and submits idempotent jobs; it never touches git/, and a concurrent republish's own submit is a Duplicate. It now refuses only a concurrent REPOPULATE (naming its job id) — the "never queue silently behind invisible work" property is kept, and the button lands on the first press.

Orphan chain rows are stopped at boot (nornir)

The queue is in-memory, so a lens/feed/workspace_republish row that is non-terminal when the server boots belongs to a process that is gone. They are closed as stopped with that reason, so the Jera tab and machine_busy stop reading a crash as live work.

What is deliberately NOT done

• No depends_on in jera. Feeds already drain before lenses by priority in the single worker, and a lens's short-circuit is against the warehouse, not against a job. A dependency edge would matter with several workers or cross-workspace jobs; the honest place for it is a second ChainQueue, and nothing needs one today.
• No republish coalescing. The raw tables' freshness is what the Coderoom, Binary and Status panes read; delaying it to help the lenses would trade one pane's truth for another's.
• No change to what a fold computes.

How it is proven

• jera: a duplicate submit against a running key re-arms exactly once; the defer ceiling is cumulative (a predicate that never goes false holds the chain off for max_defer in total, not max_defer × jobs).
• nornir: a chain key carries no generation; submitting the chain again after one job drained leaves the head of the queue at lens #2, not lens #1; run_job stamps the generation current at run time and skips when it already holds it.
• server: Repopulate is accepted while the fetch gate is held and refused only by another repopulate.
• oden: after the refresh, the ledger shows 13 of 13 lenses done at the current generation within one drain, and the Repopulate button is accepted on its first press.