nordisk/facett

The headless harness ecosystem — design

Class: design · Function axis: f0 (the headless seam) + f1 render capture · f2 test-matrix bridge · f3 run-ledger · f4 a11y interaction · f5 pixel oracle · f6 the tests/ seam. Strict: this file states only the shape we run. Every why, every alternative, lives in harness-reasoning.md. This builds on the base Elm driver. The drive/snapshot/state_json state seam is documented at elm-design.md f3 (and elm-guide.md f3); this file documents the larger harness ecosystem layered on top: RenderReport, the test-matrix, runtrace, a11y, imgscan, and the tests/ call-chain seam.

f0 — the headless seam (the whole)

The harness is one seam: hand a component data, drive it with no window and no GPU, and read back what it did as data. Every automated consumer — the test-matrix, the robot suites, the golden snapshots — is this seam at scale.

A driven component is observed through two oracles, never "it didn't panic":

Both oracles feed one sink: testmatrix::emit* (f2) appends a functional row — green iff the check's real invariant held, RED otherwise — to the JSON file nornir test reads back into its matrix.

Data flow (input → observable):

  data ──▶ component (impl Elm / impl Facet)
             │
     drive(msgs)│ (no egui, no GPU)          headless_render / render_themed_sized
             ▼                                        │ (offscreen egui, no window)
        state()/state_json  ──── state oracle ──┐     ▼
                                                │  RenderReport{ state, vertices } ── pixel proxy ─┐
        egui_kittest Harness.render()           │                                                 │
             │ (wgpu, offscreen)                │  imgscan::scan(rgba) ── ScanReport ── pixel oracle
             ▼                                  │                                                 │
        RGBA frame ─────────────────────────────┴────────────────────┬────────────────────────────┘
                                                                      ▼
                                                       testmatrix::emit*(component, check, ok, detail)
                                                                      ▼
                                              $NORNIR_TESTMATRIX_OUT (JSON) ──▶ nornir test matrix

Three companion mechanisms make the seam observable beyond a single render: runtrace (f3) is an in-memory ledger of what code ran (native + wasm); a11y (f4) makes custom-painted elements addressable so a robot driver can find and drive them by label; imgscan (f5) is the pixel oracle that reads frame structure, not just "non-blank".

Reuse map (what the harness ecosystem uses): harness depends on egui (offscreen paint), serde_json (the state oracle), and the Elm/Facet traits. testmatrix depends on nornir-testmatrix through an optional dep: edge owned only by facett-core — every leaf crate reaches it via a passthrough feature, so no leaf depends on nornir-testmatrix directly. runtrace, a11y, imgscan are dependency-free over std/egui. The whole ecosystem is used by every facett component crate's tests/ and by nornir test. All five modules are declared in facett-core/src/lib.rs (a11y :9, imgscan :23, harness :26, runtrace :35, testmatrix :38).

f1 — headless render capture & the activity trail

The Facet-side of harness.rs renders a component offscreen and captures what it drew. (The Elm-side drive/snapshot/snapshot_json/state_json state driver is the base API — see elm-design.md f3.)

RenderReport (harness.rs:11) is the render's observable result:

RenderReport::drew() (harness.rs:20) is vertices > 0.

capture (harness.rs:28) is the single render primitive every helper calls: it runs one egui frame into an offscreen egui::Context, tessellates the shapes, sums mesh vertices, reads state_json, and emits structured trace in/out records. The public helpers differ only in context/size/theme:

The activity trail mirrors nornir viz's action_log: a timestamped, kinded, sequenced stderr line per render (harness.rs:168, trail), greppable and — when $FACETT_TRAIL is set — mirrored to that file. Kind (harness.rs:121) is Render / State / Case. log (:100) prints one capped line per render; dump_state (:186) prints the full untruncated state_json; case_summary (:195) prints one per-cell matrix line (component × axis → verts drew state). The stamp is dependency-free (now_stamp :150) and the sequence is a process-global atomic (next_seq :141).

f2 — the test-matrix bridge (testmatrix.rs)

testmatrix.rs is the single seam every facett test uses to feed nornir test one functional row per real check. It wraps nornir_testmatrix::functional_status so a leaf test calls only facett_core::testmatrix::emit(...).

geometry is the component's own domain count (ways+points for a map, nodes for a graph, rows for a table) read from state_json; vertices/ratio is the pixel proof. A blank framebuffer or a zero-geometry view is a RED row, so a dead view is visible in the matrix.

Feature-gated so release strips it. Every item is #[cfg(feature = "testmatrix")] with a no-op #[inline(always)] twin compiled when the feature is off (testmatrix.rs:38,:64,:83). A leaf crate adds a passthrough testmatrix = ["facett-core/testmatrix"] and wraps each call site in the same cfg. Feature OFF → no nornir-testmatrix dependency edge (it is dep: optional), every emit compiled out. Feature ON → each call appends one row to the JSON sink at $NORNIR_TESTMATRIX_OUT (default target/nornir-functional.json), keyed by $NORNIR_TESTMATRIX_REPO / $NORNIR_TESTMATRIX_RUN. functional_status sets aspect = "functional"; status is PASS/FAIL per ok.

f2b — the ONE discovery-driven probe (probe_facet + probe_and_emit)

The emit_* calls above are manual — a leaf test computes ok/geometry and emits. The probe is the generic path that gives ANY dyn Facet a functional cell with one call and no concrete Msg type, so the zero-cell panes backfill through one shared function instead of re-writing render+emit per crate.

Object-safe: the probe drives the Facet seam (title/ui/state_json/ update_json), so it needs neither the pane's Model nor its Msg. A dead update_json (default no-op) registers as not responsive → a RED responds_to_input row, so a silently-inert input surface is visible in the matrix. FacetProbe/ProbeStep are always compiled (in harness.rs); only the emit is feature-gated, so a plain build can still probe-and-assert without pulling nornir-testmatrix. First consumer: facett-brokkrview/tests/robot.rs — one probe_and_emit call closes both its zero-cell gap and its missing json-drive robot scenario. Adoption = build the pane via its demo()/local() ctor, script the messages, call probe_and_emit.

f3 — the run-ledger (runtrace.rs)

runtrace.rs is a process-global, in-memory, wasm-safe ledger of what code ran: a Mutex<BTreeMap<String, u64>> behind a OnceLock (runtrace.rs:22). A component's ui()/render and each interactive control handler calls ran(key) (:31) with a stable key (e.g. "PopGraphTab::ctl:11M"), incrementing its tally.

snapshot() is folded into the demo's state_json["trace"]["ran"], so the same ledger proves execution natively (an egui_kittest headless drive) and in the shipped wasm bundle (read via the window.__facett_state() JS hook). BTreeMap keeps key order deterministic. This is the native↔wasm parity seam: a control that ran natively but not in the shipped wasm surface is RED.

f4 — the a11y interaction seam (a11y.rs)

egui's immediate-mode painter draws pixels but emits no accessibility node, so a robot driver / get_by_label / a screen reader cannot see a custom-painted element. a11y.rs gives the FC-4/FC-5 fix in one call — a painted element becomes a queryable, labelled, value-carrying, stably-addressable AccessKit node.

Semantics (a11y.rs:33) is the payload: typ: WidgetType, label: String (unique per surface, FC-5), value: Option<f64>, selected: Option<bool>, enabled: bool. Constructors: new (:49), button (:53), image (:57), list_item (:61); chainable value (:66) / selected (:71) / enabled (:76). widget_info (:82) lowers it to an egui WidgetInfo.

stable_id(base, key) (a11y.rs:95) derives a stable egui::Id as base.with(key) from a domain key (a node name, a row id, a lon,lat) — never a loop index / timestamp / RNG (FC-5). node(ui, base, key, sense, rect, sem) (a11y.rs:106) is the whole helper: it allocates a hit-testable Response over the painted rect keyed on the stable id, attaches the WidgetInfo, and returns the Response so the caller reads .clicked()/.hovered() like any widget. The painter still draws the visual; node only adds the accessibility node + interaction target on top.

f5 — the pixel oracle (imgscan.rs)

imgscan.rs is the reusable image-analysis oracle (the SCAN-THE-PIXELS law): a render test that claims it "sees" a frame runs real image analysis on the RGBA bytes and asserts the structure, not "non-blank". It is pure Rust over a &[u8] RGBA buffer (4 bytes/pixel, row-major) + width/height — the exact shape egui_kittest's harness.render() produces (image::RgbaImage::as_raw()).

ScanReport (imgscan.rs:71) bundles every analyzer:

scan(px, w, h, bg) (imgscan.rs:372) runs all of them; Rgba (:26) is the background reference. A blank frame reads all-zero; a frame-filling smear reads high coverage + high spoke_score; a clean render sits in a band between. The analyzers degrade gracefully (return the empty read) on bad dimensions, never panic.

f6 — the tests/ seam (call-chain matrix · robot · snapshot)

The seam is consumed by a per-crate family of tests that together make the green matrix mean every link of the input→pixel chain works.

Call-chain matrix (<crate>/tests/callchain_matrix.rs, and the core tests/render_callchain_matrix.rs). Each non-pixel stage of the chain gets an INJECT-real-input → ASSERT-the-real-return case; the painted-pixel stage (no return exposes it) gets an EMIT row. facett-graph/tests/callchain_matrix.rs walks topology × size × canvas × layout: build_scene_returns_correct_counts_and_colours (:154), layout_positions_are_finite_in_rect_and_shaped (:194), edge_endpoints_match_laid_out_node_centres (:267), and the ~2304-cell render_matrix_topo_x_size_x_canvas_x_layout (:316) which renders each cell via harness::render_themed_sized (:338), asserts state_json == the model, and counts drew(). facett-core/tests/render_callchain_matrix.rs covers the raster chain stage-by-stage (stage1..stage10, :75..:416), with a ~2400-case raster_matrix_sweep_emits_one_row_per_case (:274) asserting gatling-vs-sequential byte-identity via an FNV frame_fingerprint.

Robot (<crate>/tests/robot_callchain.rs, facett-demo/tests/robot_deck.rs). Drives the component through egui_kittest's real wgpu frame loop, then asserts on the rendered pixels. facett-graph/tests/robot_callchain.rs: graph_renders_non_blank_and_confined (:40) asserts the paint is non-blank AND confined to the widget rect (no ink in the margin below it); graph_final_picture_png_blob (:102) saves the frame to tests/snapshots/. Pixel legs SKIP gracefully without a wgpu adapter. facett-demo/tests/robot_deck.rs drives every deck tab, runs imgscan::scan on each frame (:33), rotates OSM-3D through 360°×zoom×tilt scanning every pose for the smear (:95), and checks the painter's degenerate-triangle cull so no zero-area sub-triangle reaches WebGL glow (:160).

Snapshot / golden (<crate>/tests/snapshot.rs, *_snapshot.rs). Renders a populated component to a real PNG via the wgpu backend and compares it, pixel for pixel, against the committed golden — facett_core::golden::assert_golden, exact, no tolerance, and a missing golden is a failure rather than a silent pass. facett-demo/tests/snapshot.rs::snapshot_populated_demo_to_png (:18) holds tests/snapshots/facett_demo.png; facett-grid/tests/snapshot.rs holds grid_flagship.png. Until 2026-07-23 these OVERWROTE their golden and asserted only "> 5 KB", which absorbed a deleted facett-grid toolbar — 45 % of the frame — for three weeks. Do not reintroduce a tolerance; that is the knob that lets the next one through.

Two rules the 2026-08-31 standing red bought, both cheap and both skipped once already:

1. A toolchain bump re-blesses every golden, in its own commit. 1a48bfe5 took the fleet to egui 0.35, touched no PNG, and said "facett fully green, all targets". epaint's glyph rasteriser had moved, so grid_flagship went red on every run for a month — 2.29 % of pixels, always the same ones — and nobody read it, because a suite that cannot finish produces no verdict anyone reads. facett-demo's eight goldens were re-blessed on 2026-08-21 by a commit that had to touch them anyway; the grid's was the one nobody happened to walk past.

2. A re-blessed golden says in the commit what visual change it now records, and how that was decided — a golden re-blessed in silence is how a real regression is laundered into the baseline. "Stale" and "regression" are opposite conclusions from the same red, so decide it by measuring the old blob against the new one. Four tests, cheapest first:

   • Was flat ground repainted? The decisive one. Take the pixels where the OLD image was locally flat (all eight neighbours identical) and look at the deltas there. An AA or gamma change can only move pixels that already sat on a gradient — a glyph or a rule edge. A regression cannot be confined to them: a vanished toolbar, a changed heat scale or a wrong colour necessarily repaints flat ground. On grid_flagship 2026-08-31: 2 034 of 14 070 changed pixels were in flat ground and not one exceeded a channel delta of 8 → stale.
   • Did anything move? Best-alignment offset. (0, 0) at mean |Δ| 0.155 against 5.4–9.8 for all eight ±1 px shifts means every glyph is on the same pixel; a layout change shows as a shift.
   • Delta histogram. A long tail of ±1 with the big deltas on glyph edges is a rasteriser (8 922 of 14 070 differ by exactly 1; the 849 that reach ≥ 32 are 61 % in the heading band). A solid region with a large mean is content. Check for solid regions directly: zero 8×8 blocks were ≥ 90 % big-delta.
   • Do the OTHER goldens on this host still pass? If they do, the renderer, the driver and the box are stable and the odd one out is stale.

THE GPU TURN — facett_core::render::gputurn

Two wgpu bring-ups at once do not fail an assertion, they signal: 11 the whole test binary — measured 2/5, 3/5 and 1/8 on three files (2026-08-27) — or wedge it: cargo test -p facett-core --lib sat for 13 minutes with 20 render::gpu::* threads in futex_do_wait, the GPU at 0 % and 632 MiB of VRAM held, in BOTH profiles on 2026-08-31.

The turn lives in facett-core and has two halves:

1. In-process — a Mutex. Thread against thread; the SIGSEGVs and the deadlock.
2. Across processes — an flock on $XDG_RUNTIME_DIR/facett-one-gpu.lock ($FACETT_GPU_LOCK overrides), with a 180 s LOCK_NB deadline so a wedged holder degrades the run loudly instead of hanging every binary behind it. Machine-wide, not per---target-dir, and that is the point: cargo test runs a crate's test binaries one after another, so the collision needs two cargo invocations at once — two lanes on one box — which a build-tree-scoped lock would never have seen. The resource is one RTX 4090; the lock is scoped to the resource. The old comment claiming "separate test binaries do not contend" was measured false when line_gpu SIGSEGV'd beside facett-geomap's device tests.

It was in facett-map/tests/common/one_gpu.rs until 2026-08-31, and a rule enforced in one crate out of four is not enforced, it is documented: facett-core's eight hand-rolled probes and facett-map3d's device tests could not reach a lock in another crate's tests/ directory, and both then produced exactly the failures it exists to prevent. one_gpu.rs is now a re-export — never a second mutex, because two mutexes over one device serialise nothing and the second reads as coverage, which is how six copies coexisted while the crashes went on (tests/one_gpu_turn_is_the_only_one.rs is the guard that keeps it single).

gputurn::probe::open is the ONE WRITER for a headless bring-up (feature wgpu). facett-core had eight copies of the same twenty lines — five headless_device() / compute_device() helpers in render::gpu::* and three library entry points (l1::render_overlay, gpu::offscreen_render, gpu::render_graph_offscreen, which the facett-map, facett-graph3d, facett-helix and facett-graphview pixel tests all reach through). Every copy got the same two things wrong. open fixes both by construction:

• it takes the turn and hands it back to the caller, so the lane covers the whole test body, not just the bring-up. Serialising only the bring-up still left nine label_collide devices alive at once and still deadlocked;
• it names the adapter it got, on stderr, every time, and then applies a policy. It goes through render::gpu::request_best_adapter (which ranks software last) instead of request_adapter(PowerPreference::default()) — that default is LowPower and lets wgpu hand back llvmpipe when the real card cannot give a device. MEASURED 2026-08-31 on oden: with the 4090 saturated by concurrent lanes a suite selected llvmpipe (LLVM 21.1.8, Vulkan, Cpu). It did not error; it rendered different pixels and then passed or failed for reasons unrelated to the code.

libtest CAPTURES stderr, so that receipt surfaces under --nocapture or in the replayed output of a FAILING test — which is when it is needed: a red whose output carries no [<who>] adapter — … line opened no device, and a red that names llvmpipe is not a red about the code. In a captured green run nobody reads stderr at all, which is exactly why the policy below does the work and the print does not. When the adapter is software, OnSoftware::Refuse panics (the five pixel probes: a pass on llvmpipe is as meaningless as a fail) and OnSoftware::Skip returns None (the three Option-returning library entry points, whose every caller already reads None as "no GPU, skip" — and which is also the right production answer on a host that genuinely has no card). $FACETT_ALLOW_SOFTWARE_GPU=1 gives the old behaviour back.

MEASURED after wiring, on this box: cargo test -p facett-core --lib — the run that deadlocked for 13 minutes — completes in 19.86 s, 478 passed, at load 6, and the 78 render::gpu + gputurn tests take 5.31 s serialised. Both arms of both halves are proven red: forcing FACETT_GPU_ADAPTER=software makes a Refuse probe panic naming llvmpipe and a Skip probe skip loudly, and gputurn's own tests fail if the no-guard control does not overlap or if a stranger's flock succeeds while the turn is held.

The lib-test blind spot, MEASURED 2026-09-03. cargo test --workspace --lib ended in signal: 11 inside facett_graph3d, and cargo test -p facett-graph3d --lib -- --test-threads=1 passed 84. The cause was facett-graph3d/src/neon.rs: two #[test]s in a #[cfg(test)] module inside a library source, each building egui_kittest::Harnesses and calling .render() — five device bring-ups, no turn, on parallel libtest threads. Tightened to those two tests: SIGSEGV 17 of 30 runs at --test-threads=2, 0 of 15 at --test-threads=1; after the turn, 30 of 30 green and 10 of 10 for the whole binary at default threads.

They were the ONLY .render() calls in any facett --lib test, which is why one crate's two tests decided a 71-crate suite's verdict. facett-map's guard scans facett-map/tests/ and structurally cannot see another crate's src/ — the same "a rule enforced in one crate is not enforced" shape as 2026-08-31, one directory further in. A sweep of all 395 */src/*.rs found five files that open a device from test code; the other four are feature-gated hand-rolled compute_device() copies (facett-graphview/src/gpu_layout.rs, facett-map3d-flight/src/{cdlod,extrude_gpu,scatter_gpu}.rs, dark in the default --lib run — gpu_morph_matches_cpu_oracle and gpu_layout_matches_cpu_reference appear zero times in its output). All five take the turn now.

Still open:

• facett-geomap's nine .render() files take no turn.
• No guard covers src/ test modules. The five sites above were found by a hand sweep, not by a check that can fail. facett-map/tests/one_gpu_turn_is_the_only_one.rs is the right logic in the wrong scope; the workspace-wide twin belongs in xtask/tests/, next to silenced_tests_guard.rs, which already walks every member.
• facett-graphview --features gpu --lib is its OWN standing red, unrelated to the --lib lane above (the gpu feature is off there). src/gpu.rs has three more unturned bring-ups — probe_usable_gpu, GpuRenderer::new and the process-global cached_renderer() device cache (OnceLock<Mutex<Option<GpuRenderer>>>) — beside the now-turned gpu_layout probe. MEASURED 2026-09-03: signal: 11 once in 10 runs of that lane (1/10, and 9 greens either side of it, so a green run there proves nothing). Not fixed here because render_gpu → cached_renderer → GpuRenderer::new would need the turn taken inside a library entry point that a test may already be holding, and gputurn is deliberately not re-entrant — that is a deadlock, not a fix, and it needs the probe::open-style refactor rather than a hold().

Run-ledger drive (facett-demo/tests/pop_ran_list.rs). Drives the whole demo over the real draw_app path, walks every tab, clicks every control by label, and reads back runtrace from state_json["deck"]["trace"]["ran"]: ran_list_every_tab_and_sverige_controls_run (:51) asserts each control ran and writes the readable list; native_wasm_deck_parity (:144) pins the deck tab set as cfg-independent so native == wasm.

a11y proof (<crate>/tests/a11y.rs). Renders a painted element through egui_kittest, then get_by_label retrieves it from the AccessKit tree. facett-card/tests/a11y.rs::painted_chip_and_avatar_enter_the_accesskit_tree (:39) proves a painter-only enum chip resolves by label and carries its option-set size as numeric_value.