nordisk/facett

The headless harness ecosystem — reasoning

Class: reasoning. This file holds the why, the trade-offs, and every design we rejected. The descriptive shape we actually run is in harness-design.md, which by rule mentions none of this. The base why — why a headless Elm loop exists at all (the f3 payoff: serializable Model + Msg-only mutation = test a GUI with no GUI) — is in elm-reasoning.md and is not repeated here. This file is the why of the ecosystem built on top: the two oracles, the matrix, runtrace, a11y, imgscan, and the tests/ seam.

Why two oracles, and why "it drew something" is not enough

The state oracle (snapshot the Model) is exact and free, but it proves only the logic ran — it cannot prove the painter produced the right pixels. So we added a pixel path. The naive pixel oracle is RenderReport::drew() (vertices > 0), and we deliberately keep it only as a proxy, never the real gate. The reason is a scar: on 2026-06-21 the OSM-3D map smeared green spokes across the entire live wasm frame while every render test was green — because "drew vertices" / "wrote a PNG" / "non-blank" all pass on a garbage frame. A smear draws more vertices than a correct render. So drew() is the cheap always-available proxy in RenderReport; the real pixel oracle is imgscan (below), which reads frame structure. Keeping both — a cheap proxy for the no-GPU path, a real oracle for the wgpu path — is why there are two pixel mechanisms and not one.

Why imgscan analyzes structure (the SCAN-THE-PIXELS law)

A render test that claims it "sees" a frame must run image analysis and assert the result is the expected structure. We rejected the tempting shortcuts:

• Rejected: "non-blank" (any lit pixel). Passes on a smear, a starburst, a single stuck pixel. The green-spoke bug was exactly a non-blank frame.
• Rejected: "wrote a PNG of N bytes". A blank or wrong frame is still N bytes.
• Rejected: a full FFT / an image-diff crate. Heavy dependencies for a test oracle, and a stored golden only catches change, not wrongness (nobody knows the golden itself is right). So imgscan is pure Rust over the raw RGBA slice, and every analyzer returns a number a test bounds tightly: spoke_score (thin radiating runs — the smear signature), high_freq_ratio, coverage, centroid, bbox. The oracle must prove it can go red: imgscan's own tests assert a threshold strictly separates a synthetic clean frame from a synthetic smeared one, with a wide gap — if that separation ever collapses, the oracle is blind and the test fails. An oracle you never proved could fail is not an oracle.

Why the native robot checks geometry, not only pixels

The wgpu pixel scan is blind to one whole bug class: wgpu culls zero-area triangles, but WebGL draws them as lines — the green spokes only appear in the shipped browser, never in the native wgpu render. Scanning the native frame would stay green forever while the deployed demo smears. So the OSM-3D robot also checks the geometry the painter feeds the renderer: it drives the scene across rotations, replicates the paint clip, and asserts the degenerate-triangle cull removes every zero-area sub-triangle before glow — with a sensitivity clause that the raw pre-cull sliver count is > 0 (the bug really is present in the data). We test the thing the target platform will do, not the thing our test platform does.

Why the two-axis law (input coverage × oracle exactness)

A check is assert(invariant(f(input))). It is weak if it feeds only ONE input (Axis A) or if its threshold is a loose multiple / tolerance / drew()-only / literal-true instead of the exact contract bound (Axis B). This is why the matrices are swept (topology × size × canvas × layout × theme = thousands of cells) rather than single-fixture, and why emitters carry the measured value, not a boolean. The canonical failure was facett-map3d's SCREEN_BOUND_K = 4.0 and a half_h*8 extent cap: a green test that tolerated a triangle 8× the viewport — Axis B too loose — and sailed straight past the smear. Both axes strong is the standard; a matrix of weak cells is a green light that means nothing.

Why testmatrix is feature-gated and owned only by facett-core

• Rejected: every leaf crate depends on nornir-testmatrix directly. That scatters the cross-repo edge across 26 crates and drags nornir-testmatrix into release artifacts. Instead facett-core owns the single dep:-optional edge and re-exposes emit/emit_render/emit_non_blank; leaves add a passthrough feature.
• Rejected: emit unconditionally. A shipped facett must not carry a test-harness dependency. So every emit is #[cfg(feature = "testmatrix")] with a no-op twin: feature OFF ⇒ no dependency edge at all (provable with cargo tree), every call site compiled out, even a stray call is a no-op. We pay one passthrough feature + one cfg per call site to buy a release build that is provably harness-free.

Why a dead view is RED, not skipped

emit_render/emit_non_blank gate on geometry > 0 and the pixel proof, where geometry is the component's own domain count (ways+points, nodes, rows). A map with no ways that renders a clean empty pane would otherwise be "green" — passing while showing nothing. We chose to make an empty/dead view a red functional row so the matrix surfaces "this component is drawing nothing" as a failure, not silence.

Why runtrace is in-memory (and not the trace file)

There is already a JSONL trace sink at $FACETT_TRACE, but it is native-only — there is no filesystem in a browser, and the bug we needed to catch was the shipped wasm not running a control. So runtrace is a process-global Mutex<BTreeMap> read back as a JSON object, folded into state_json. The same mechanism then proves execution natively (a headless drive reads state_json.trace.ran) and in wasm (the window.__facett_state() JS hook reads the identical object). BTreeMap (not HashMap) so the snapshot key order is deterministic for tests. The alternative — two different verification paths for native and wasm — could not prove they agree, which is the whole point (a stale wasm bundle missing a tab was the RAGNARÖK bug).

Why a11y attaches nodes to painted elements at all

egui's painter draws pixels but emits no accessibility node, so a custom-painted graph node / chip / map pane is invisible to get_by_label, a screen reader, and a robot driver. We rejected "just unit-test the geometry": the robot suites need to find and drive elements by label the way a user's assistive tech does, and a screen reader is a hard product requirement (FC-4). Keying the node on a stable domain key rather than a loop index (FC-5) is what makes a replayed click land on the same element after the list reorders — the same reasoning as index-vs-id selection in elm-reasoning.md, applied to the accessibility id.

Why the golden PNG is kept — and why the byte-floor was wrong (2026-07-23)

The golden snapshot per crate is retained as an acceptance artifact a human can eyeball. For a long time its automated assertion was only a byte-floor (> N bytes), on the reasoning that a stored golden proves change, not correctness, and that pixel-exact diffing is brittle across GPU drivers. The real gates were said to be the swept matrix + imgscan structural bounds; the golden was "the picture, not the proof".

That reasoning shipped a hole, and the hole cost us. The tests did not merely assert weakly — they overwrote the committed golden every run before asserting. Write-only images cannot fail. Two real changes went in unseen:

• facett-grid's entire toolbar was deleted by the Elm migration 4b92e16 and stayed gone for three weeks — 45 % of the flagship frame. Msg::CycleDensity, Msg::ClearSort and Msg::SetFilter remained green in the headless Msg-driven tests the whole time, because no UI was needed to send a Msg.
• A user-facing tab label was renamed (jera → Jobs) and eight goldens absorbed it silently.

Both are exactly the "proves change, not correctness" case — and detecting change is precisely what was needed. Neither the swept matrix nor imgscan can see a missing toolbar: the frame is still non-blank, still structured, still the right size. Only a comparison against what the picture looked like before can.

So the policy is now:

• The golden is compared, never written by a passing test ([facett_core::golden::assert_golden], feature golden). It is a gate, and the picture as well.
• The match is exact. The driver-brittleness worry did not survive contact: these egui_kittest + wgpu renders are byte-identical across repeated runs on a fixed host, and a tolerance is the knob that lets a real regression through. If a future host genuinely jitters, add the smallest tolerance that host needs and say so at the call site — never globally.
• A missing golden FAILS. A snapshot suite that treats "no golden yet" as success is structurally green over nothing.
• Blessing is the explicit opt-in UPDATE_SNAPSHOTS=1, and the pixels are git-lfs, so every blessing lands in a reviewable commit that has to say why they moved.

The imgscan and swept-matrix gates are unchanged and still the answer to "is this frame right". The golden guard is the answer to "did it move", and a suite needs both. .nornir/'s own advice stands: an oracle you never proved could fail is not an oracle — so the guard is verified red by deleting the toolbar (44.98 %), by changing one glyph in one label (0.40 %), and by deleting the golden (missing → fail, and the golden is not recreated).