crates/nornir-robotui/src/watch.rs
//! # The **WATCH backend** (feature `watch`)
//!
//! A live replay driver: it mounts the consumer's component in a REAL native
//! `eframe`/`egui` window and replays a [`Scenario`]'s [`Action`]s against the
//! LIVE egui context, one step per pacing delay, moving a visible cursor and
//! injecting synthetic input so a human WATCHES the robot click / type / scroll /
//! drag through it in real time. It is the visible sibling of the headless
//! kittest drive — SAME [`Scenario`], SAME `draw(ui, app, targets)` closure.
//!
//! ## How input is injected
//!
//! egui computes pointer/keyboard interactions in `begin_pass` from the frame's
//! [`egui::RawInput`]. eframe exposes
//! [`App::raw_input_hook`](eframe::App::raw_input_hook), called with the very
//! `RawInput` about to be processed — so this driver PUSHES synthetic
//! [`egui::Event`]s there (a `PointerMoved` to animate the cursor, a
//! `PointerButton` press+release to click, `Text`/`Key` to type, `MouseWheel` to
//! scroll). Those are the same event kinds the headless
//! [`RobotSession::inject_events`](crate::RobotSession::inject_events) path feeds
//! a kittest `Harness`; here they drive a real window.
//!
//! ## How a label resolves to a widget
//!
//! **AccessKit first, [`Targets`] as fallback** — exactly the resolution the
//! headless path uses. Each frame egui produces an AccessKit `TreeUpdate` (this
//! driver forces it on with [`egui::Context::enable_accesskit`] and taps the frame
//! output via an [`egui::Plugin`]); the driver walks that live tree for the node
//! whose accessible name matches the [`Action`]'s label — resolving `.labelled_by`
//! links, roles and values the same way `egui_kittest` does — and takes the node's
//! bounds (egui points) as the click rect. So **any** widget that carries an
//! AccessKit label (all of them — the headless drive already clicks them) is
//! clickable in the VISIBLE window with NO per-app `Targets` recording.
//!
//! If the AccessKit tree has no match (e.g. an app that hasn't drawn the widget
//! yet, or a bespoke non-AccessKit surface), the driver falls back to the
//! [`Targets`] rect the `draw` closure may still record (`t.record("Bump", resp)`)
//! — so the pre-existing rail path keeps working unchanged. Targets addressed only
//! by AccessKit node id (`Action::ClickById`) have no label to resolve and no rect
//! side-channel, so the watch driver dwells on them without a click (they still
//! drive fine headless).
//!
//! ## Two coordinate spaces, and the HiDPI bug they hid
//!
//! AccessKit bounds are **physical pixels**; the synthetic pointer events are
//! **points**. egui scales the AccessKit root by `pixels_per_point`, so every
//! `bounding_box()` comes back pre-multiplied. The driver divides it back out (see
//! `ak_lookup`'s `node_rect`) using the live [`egui::Context::pixels_per_point`].
//!
//! At scale 1 the two spaces are identical, so no test that ran under `Xvfb` — where
//! the scale is always 1 — could ever see the difference. On a 2× desktop the driver
//! aimed at twice each widget's coordinate and every click silently missed, while
//! the AccessKit tree still reported the widget present and clickable. Any new guard
//! on this path must therefore assert at a scale **other than 1**, and must assert
//! the *applied* result (the toggle's state read back from
//! [`Observable::state_json`]), not that a rect was returned.
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use egui::accesskit::TreeUpdate;
use egui_kittest::kittest::{AccessKitNode, State as AkState};
use crate::display::DisplayBackend;
use crate::mode::{Targets, WatchConfig, WatchRenderer};
use crate::scenario::{Action, Assert, FieldValue, Scenario, Step, Target};
use crate::Observable;
/// A shared slot the [`AccessKitTap`] plugin drops the latest frame's AccessKit
/// tree into, for the [`WatchDriver`] to read after each paint.
type AkSlot = Arc<Mutex<Option<TreeUpdate>>>;
/// An [`egui::Plugin`] whose only job is to **capture the AccessKit `TreeUpdate`**
/// egui emits at the end of each frame. egui builds the tree during `end_pass` and
/// hands the whole [`egui::FullOutput`] to each plugin's
/// [`output_hook`](egui::Plugin::output_hook) just before it goes to the backend —
/// the one seam an in-`eframe` app can read the tree from (the platform output is
/// otherwise consumed by eframe's winit integration). We clone it into a shared
/// slot the driver reads next frame to resolve label → rect.
struct AccessKitTap {
slot: AkSlot,
}
impl egui::Plugin for AccessKitTap {
fn debug_name(&self) -> &'static str {
"robotui_accesskit_tap"
}
// egui 0.35 passes the `Context` to `output_hook` as well, so a plugin can read
// context state alongside the frame's output. This tap only needs the AccessKit
// update out of `platform_output`, so the context is unused.
fn output_hook(&mut self, _ctx: &egui::Context, output: &mut egui::FullOutput) {
if let Some(update) = &output.platform_output.accesskit_update {
if let Ok(mut slot) = self.slot.lock() {
*slot = Some(update.clone());
}
}
}
}
/// What a live-tree lookup found for a label: the widget's click [`rect`], its
/// AccessKit `value` (a `TextEdit`'s text / a slider's formatted value) and
/// `toggled` state — enough to resolve a click AND to evaluate the value/checked
/// verdicts in watch mode (which the old `Targets`-only side-channel could not).
struct AkHit {
rect: Option<egui::Rect>,
value: Option<String>,
toggled: Option<bool>,
/// True when the matched node is a plain `Role::Label` (a lower-priority match:
/// the interactive widget it labels is preferred as the click target).
label_role: bool,
}
/// Walk the live AccessKit tree for the node whose **accessible name** matches
/// `label`, mirroring `egui_kittest`'s `query_by_label`: a `Role::Label` node's
/// text is its `value`, every other node's is its `label` (which resolves
/// `.labelled_by` links). Prefers an interactive (non-`Label`) match so
/// `TypeInto("Name")` lands on the `TextEdit`, not on the "Name" caption labelling
/// it. `None` if nothing matches.
///
/// `pixels_per_point` converts the hit's bounds out of AccessKit's **physical
/// pixel** space into the **point** space the synthetic pointer events use — see
/// [`node_rect`].
fn ak_lookup(state: &AkState, label: &str, pixels_per_point: f32) -> Option<AkHit> {
use egui::accesskit::{Role, Toggled};
/// The widget's click rect **in egui points**.
///
/// This is the one line that decided whether a watch drive works on a HiDPI
/// display, so it is worth the paragraph.
///
/// egui builds each widget's AccessKit node with bounds in *points*, but it
/// also sets a transform on the AccessKit **root**:
///
/// ```text
/// // egui-0.35.0/src/context.rs
/// root_node.set_transform(accesskit::Affine::scale(pixels_per_point.into()));
/// ```
///
/// and `accesskit_consumer::Node::bounding_box()` returns the bounds with every
/// ancestor transform already applied (`self.transform().transform_rect_bbox(..)`).
/// So `bounding_box()` hands back **physical pixels**, not points.
///
/// The synthetic [`egui::Event::PointerMoved`] / [`egui::Event::PointerButton`]
/// this driver pushes into [`egui::RawInput`] are in **points**. At
/// `pixels_per_point == 1.0` the two spaces coincide and the bug is invisible —
/// which is exactly why it survived: every previous drive ran under `Xvfb`,
/// where the scale is 1. On a 2× display the driver aimed at twice each
/// widget's coordinate, so every click missed (usually landing off-screen) while
/// the AccessKit tree still reported the widget present. Dividing by
/// `pixels_per_point` inverts the root transform and puts the rect back in the
/// space the events are actually interpreted in.
///
/// The [`Targets`] fallback rects need no such conversion: they come straight
/// from `egui::Response::rect`, which is already in points.
fn node_rect(n: &AccessKitNode<'_>, pixels_per_point: f32) -> Option<egui::Rect> {
// A zero/NaN scale would blow the rect up to infinity; fall back to 1:1.
let ppp = if pixels_per_point.is_finite() && pixels_per_point > 0.0 {
pixels_per_point
} else {
1.0
};
n.bounding_box().map(|r| {
egui::Rect::from_min_max(
egui::pos2(r.x0 as f32 / ppp, r.y0 as f32 / ppp),
egui::pos2(r.x1 as f32 / ppp, r.y1 as f32 / ppp),
)
})
}
fn name_matches(n: &AccessKitNode<'_>, label: &str) -> bool {
let name = if n.role() == Role::Label {
n.value()
} else {
n.label()
};
name.as_deref() == Some(label)
}
fn walk(n: &AccessKitNode<'_>, label: &str, ppp: f32, best: &mut Option<AkHit>) {
if name_matches(n, label) {
let label_role = n.role() == Role::Label;
let take = match best {
None => true,
// An interactive match supersedes a plain-label match.
Some(prev) => prev.label_role && !label_role,
};
if take {
*best = Some(AkHit {
rect: node_rect(n, ppp),
value: n.value(),
toggled: n.toggled().map(|t| matches!(t, Toggled::True)),
label_role,
});
}
}
for child in n.children() {
walk(&child, label, ppp, best);
}
}
let mut best = None;
walk(&state.root(), label, pixels_per_point, &mut best);
best
}
// Fractions of a step's delay at which each sub-phase fires. Cursor travel fills
// the first ~45%; the click/type happens mid-step; the rest is dwell so the
// result is visible before the next step.
const TRAVEL_FRAC: f32 = 0.45;
const PRESS_FRAC: f32 = 0.55;
const RELEASE_FRAC: f32 = 0.66;
const ACT_FRAC: f32 = 0.72;
/// Open a real window and replay `scenario` against `app`/`draw`, paced by `cfg`.
/// Blocks (runs the native event loop) until the scenario finishes and the
/// window closes. Must be called on the main thread (winit's requirement).
pub fn run<A, F>(
app: A,
draw: F,
scenario: &Scenario,
repo: impl Into<String>,
cfg: WatchConfig,
) -> anyhow::Result<()>
where
A: Observable + 'static,
F: FnMut(&mut egui::Ui, &mut A, &mut Targets) + 'static,
{
// ── Wayland or nothing ──────────────────────────────────────────────────────
// REFUSE before `eframe::run_native`, because winit will happily open this window on
// an X server and then the whole drive is measuring the wrong world — or worse, is on
// the operator's real screen when the caller believed it was headless. `DISPLAY` is
// exported on Wayland sessions too (XWayland), so "a display exists" was never the
// question; which KIND it is, is. See [`crate::display`] for why the X backends were
// removed rather than kept as a fallback.
let backend = crate::display::DisplayBackend::detect();
if !backend.is_supported() {
anyhow::bail!(
"watch mode needs a Wayland compositor and found {}. Robot UI is Wayland-only: \
set WAYLAND_DISPLAY (a nested compositor's own socket keeps the window off \
your real screen). Refusing to open a window on an unsupported backend.",
backend.detail()
);
}
let native = eframe::NativeOptions {
viewport: egui::ViewportBuilder::default()
.with_inner_size(cfg.size)
.with_title(cfg.title.clone()),
renderer: resolve_renderer(cfg.renderer),
..Default::default()
};
let slot: AkSlot = Arc::new(Mutex::new(None));
let scale = cfg.scale;
let driver = WatchDriver::new(app, draw, scenario, repo.into(), cfg.clone(), Arc::clone(&slot));
eframe::run_native(
&cfg.title,
native,
Box::new(move |cc| {
// Force the AccessKit tree ON every frame (regardless of whether a
// screen reader is attached) and tap it, so the driver can resolve any
// labelled widget without per-app `Targets` plumbing.
cc.egui_ctx.enable_accesskit();
cc.egui_ctx.add_plugin(AccessKitTap { slot });
// An explicitly requested UI scale, so a HiDPI drive is reproducible on a
// scale-1 display (`ROBOTUI_WATCH_SCALE=2`). egui derives
// `pixels_per_point = zoom_factor * native_pixels_per_point`, so this
// exercises the identical scaling path a 2× monitor would.
if let Some(scale) = scale {
cc.egui_ctx.set_zoom_factor(scale);
}
Ok(Box::new(driver) as Box<dyn eframe::App>)
}),
)
.map_err(|e| anyhow::anyhow!("eframe failed to open the watch window: {e}"))?;
Ok(())
}
/// Map the feature-free [`WatchRenderer`] choice onto an `eframe::Renderer`.
///
/// **There is one backend and no feature decides it.** `eframe` carries `wgpu`
/// unconditionally (LAW #4), so a plain `watch` build already paints on WebGPU and the
/// `watch-wgpu` feature is a no-op alias of `watch`. This doc line said "wgpu only
/// when the `watch-wgpu` feature is compiled (else glow, always available)" until
/// 2026-08-24 — a gate above its own body, contradicted by the `Renderer::Wgpu`
/// two lines below it. This is still the seam that lets korp's Map / facett's 3D paint
/// on the GPU in the visible window instead of falling back to the CPU painter; it
/// just has nothing left to choose.
fn resolve_renderer(_choice: WatchRenderer) -> eframe::Renderer {
// There is one backend. LAW #4 — WebGPU everywhere, no WebGL/glow.
//
// This used to be a three-way match with glow as the always-available fallback, so
// a watch window would quietly paint on OpenGL and every GPU pane degraded to the
// CPU painter with only a stderr note. `WatchRenderer` now has a single variant and
// eframe is pinned to the `wgpu` feature, so there is nothing left to choose.
eframe::Renderer::Wgpu
}
/// The live replay state machine — an [`eframe::App`] that draws the consumer UI
/// each frame and advances one [`Step`] per [`WatchConfig::step_delay`].
struct WatchDriver<A, F> {
app: A,
draw: F,
targets: Targets,
steps: Vec<Step>,
repo: String,
cfg: WatchConfig,
/// Index of the step being replayed (== `steps.len()` when finished).
idx: usize,
/// When the current step began.
phase_started: Instant,
/// Set true once the first frame has drawn (so targets exist to resolve).
started: bool,
/// The visible synthetic cursor position.
pointer: egui::Pos2,
/// Where the cursor started travelling from for this step.
move_from: egui::Pos2,
/// The rect resolved for the current step's target (from the last drawn frame).
target_rect: Option<egui::Rect>,
/// Per-step latches so each synthetic sub-event fires exactly once.
did_press: bool,
did_release: bool,
did_act: bool,
/// Overlay verdicts: per step `Some(true/false)` once evaluated, else `None`.
verdicts: Vec<Option<bool>>,
/// When the last step finished (drives the linger-then-close).
finished_at: Option<Instant>,
/// Shared slot the [`AccessKitTap`] plugin fills with each frame's AccessKit
/// tree (the source for label → rect resolution).
ak_slot: AkSlot,
/// The most recent AccessKit tree, rebuilt from `ak_slot` after every paint —
/// the live tree the driver queries by label (headless-parity resolution).
ak_state: Option<AkState>,
/// The live window's `pixels_per_point`, read off the [`egui::Context`] every
/// frame. AccessKit bounds arrive in physical pixels (egui scales the AccessKit
/// root by this factor); the synthetic pointer events are in points. This is the
/// divisor that reconciles them — see [`ak_lookup`]'s `node_rect`. Under `Xvfb`
/// it is 1.0, which is why the mismatch was invisible there.
ppp: f32,
/// Latch so the end-of-run report + screenshot happen exactly once.
reported: bool,
/// Where a screenshot was asked for (`ROBOTUI_WATCH_SHOT`), if at all.
shot_path: Option<String>,
/// Latch so the viewport screenshot is requested once.
shot_requested: bool,
/// The outcome of the window-framebuffer capture, once the reply has arrived.
shot_result: Option<Result<crate::display::Shot, String>>,
}
impl<A: Observable, F: FnMut(&mut egui::Ui, &mut A, &mut Targets)> WatchDriver<A, F> {
fn new(
app: A,
draw: F,
scenario: &Scenario,
repo: String,
cfg: WatchConfig,
ak_slot: AkSlot,
) -> Self {
let steps = scenario.steps.clone();
let n = steps.len();
WatchDriver {
app,
draw,
targets: Targets::default(),
steps,
repo,
cfg,
idx: 0,
phase_started: Instant::now(),
started: false,
pointer: egui::pos2(24.0, 24.0),
move_from: egui::pos2(24.0, 24.0),
target_rect: None,
did_press: false,
did_release: false,
did_act: false,
verdicts: vec![None; n],
finished_at: None,
ak_slot,
ak_state: None,
// 1.0 until the first frame reports the real scale, so a pre-frame
// lookup is a no-op conversion rather than a divide by zero.
ppp: 1.0,
reported: false,
shot_path: std::env::var("ROBOTUI_WATCH_SHOT")
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty()),
shot_requested: false,
shot_result: None,
}
}
/// Pull the freshest AccessKit tree the [`AccessKitTap`] captured (if any) and
/// rebuild [`ak_state`](Self::ak_state) from it. Called once per paint, after
/// the consumer UI has drawn, so the tree reflects the widgets on screen.
fn refresh_accesskit(&mut self) {
let update = self.ak_slot.lock().ok().and_then(|mut slot| slot.take());
if let Some(update) = update {
self.ak_state = Some(AkState::new(update));
}
}
/// The live-tree lookup for `label`, if a tree has been captured yet. Converted
/// out of AccessKit's physical-pixel space with the frame's
/// [`ppp`](Self::ppp).
fn ak_lookup(&self, label: &str) -> Option<AkHit> {
self.ak_state
.as_ref()
.and_then(|s| ak_lookup(s, label, self.ppp))
}
/// The AccessKit click rect for `label`, if the live tree carries it.
fn accesskit_rect(&self, label: &str) -> Option<egui::Rect> {
self.ak_lookup(label).and_then(|h| h.rect)
}
/// Resolve `label` to a click rect: **AccessKit tree first**, then the
/// [`Targets`] rect side-channel — the back-compat fallback for apps that still
/// record rails or draw bespoke non-AccessKit surfaces.
fn resolve_rect(&self, label: &str) -> Option<egui::Rect> {
self.accesskit_rect(label).or_else(|| self.targets.rect(label))
}
/// Whether the live UI drew anything the robot could see — the AccessKit root
/// has children, OR (fallback) the closure recorded a target this frame.
fn drew_anything(&self) -> bool {
let ak_drew = self
.ak_state
.as_ref()
.map(|s| s.root().children().next().is_some())
.unwrap_or(false);
ak_drew || !self.targets.is_empty()
}
/// The label an action targets, if it addresses a widget by label.
fn action_label(action: &Action) -> Option<&str> {
match action {
Action::ClickLabel(s) | Action::ClickButton(s) => Some(s.as_str()),
Action::TypeInto { label, .. } => Some(label.as_str()),
Action::SetField { target: Target::Label(s), .. } => Some(s.as_str()),
_ => None,
}
}
fn step_delay_secs(&self) -> f32 {
self.cfg.step_delay.as_secs_f32().max(0.001)
}
/// Advance to the next step: latch the verdict for the one just finished, reset
/// per-step state, and point the cursor's travel origin at where it now is.
fn advance(&mut self) {
if self.idx < self.steps.len() {
let ok = self.evaluate_verdict(self.idx);
self.verdicts[self.idx] = ok;
}
self.idx += 1;
self.phase_started = Instant::now();
self.move_from = self.pointer;
self.did_press = false;
self.did_release = false;
self.did_act = false;
self.target_rect = None;
if self.idx >= self.steps.len() && self.finished_at.is_none() {
self.finished_at = Some(Instant::now());
// Proof the live window actually drove the widgets: dump the final
// observable state + per-step verdicts (opt-in, so a real watch run is
// quiet). `ROBOTUI_WATCH_DEBUG=1 robotui watch …`.
if std::env::var("ROBOTUI_WATCH_DEBUG").is_ok() {
eprintln!(
"robotui watch finished: backend={} ppp={} state={} verdicts={:?}",
DisplayBackend::detect().detail(),
self.ppp,
self.app.state_json(),
self.verdicts
);
}
}
}
/// Write the machine-readable end-of-run report to `ROBOTUI_WATCH_REPORT` (a
/// path), and capture a screenshot to `ROBOTUI_WATCH_SHOT` (a path) if asked.
/// Called ONCE, during the post-scenario linger, so the final frame is on screen
/// before it is photographed.
///
/// The report is what makes a watch run *assertable from outside the process*:
/// a harness reads back the APPLIED state (`Observable::state_json`) and the
/// backend it was measured on, rather than trusting an exit code. The driver's
/// own return value proves nothing — it is `Ok(())` whether or not a single
/// click landed.
fn write_report(&mut self) {
let backend = DisplayBackend::detect();
let shot = self.shot_result.as_ref().map(|r| match r {
Ok(s) => serde_json::json!({
"ok": true,
"tool": s.tool,
"backend": s.backend,
"path": s.path.to_string_lossy(),
"bytes": s.bytes,
}),
Err(e) => serde_json::json!({ "ok": false, "error": e }),
});
// The compositor's own view, when a tool for this backend can be used. It
// is reported SEPARATELY from the window capture rather than as a fallback,
// because the two answer different questions ("what did the screen look
// like" vs "what did this window present") and conflating them is how a
// screenshot gets attributed to the wrong world.
let external = self.shot_path.as_ref().map(|p| {
let p = format!("{p}.compositor.png");
match crate::display::screenshot_on(&backend, &p) {
Ok(s) => serde_json::json!({
"ok": true, "tool": s.tool, "backend": s.backend,
"path": s.path.to_string_lossy(), "bytes": s.bytes,
}),
Err(e) => serde_json::json!({ "ok": false, "error": e.to_string() }),
}
});
let report = serde_json::json!({
"backend": backend.tag(),
"backend_detail": backend.detail(),
"pixels_per_point": self.ppp,
"repo": self.repo,
"steps": self.steps.iter().map(|s| s.label.clone()).collect::<Vec<_>>(),
"verdicts": self.verdicts,
// The APPLIED result — what the widgets actually became.
"state": self.app.state_json(),
"screenshot": shot,
"compositor_screenshot": external,
});
if let Ok(path) = std::env::var("ROBOTUI_WATCH_REPORT") {
if !path.trim().is_empty() {
if let Some(parent) = std::path::Path::new(&path).parent() {
let _ = std::fs::create_dir_all(parent);
}
match serde_json::to_string_pretty(&report)
.map_err(|e| e.to_string())
.and_then(|s| std::fs::write(&path, s).map_err(|e| e.to_string()))
{
Ok(()) => eprintln!("robotui watch: report written to {path}"),
Err(e) => eprintln!("robotui watch: could NOT write report to {path}: {e}"),
}
}
}
}
/// Evaluate the step's assertion against the live observable state / targets —
/// for the pass/fail overlay only (visible motion is the must-have; this is the
/// nice-to-have). Unevaluatable asserts return `None` (shown neutral).
fn evaluate_verdict(&self, idx: usize) -> Option<bool> {
let assert = &self.steps[idx].assert;
self.verdict_of(assert)
}
/// The verdict for one assertion — recursive so a chained
/// [`Assert::All`] shows the CONJUNCTION in the overlay (any evaluatable
/// clause false → red; else all-true → green; else neutral), instead of the
/// last clause only.
fn verdict_of(&self, assert: &Assert) -> Option<bool> {
let state = self.app.state_json();
match assert {
Assert::None => None,
Assert::All(clauses) => {
let mut any_known = false;
for c in clauses {
match self.verdict_of(c) {
Some(false) => return Some(false),
Some(true) => any_known = true,
None => {}
}
}
any_known.then_some(true)
}
Assert::Drew => Some(self.drew_anything()),
// Present in the live AccessKit tree, or (fallback) recorded as a target.
Assert::LabelVisible(label) => {
Some(self.ak_lookup(label).is_some() || self.targets.has(label))
}
Assert::State { pointer, expected } => {
Some(state.pointer(pointer).map(|v| v == expected).unwrap_or(false))
}
Assert::StateContains { pointer, needle } => Some(
state
.pointer(pointer)
.map(|v| match v {
serde_json::Value::String(s) => s.contains(needle),
other => other.to_string().contains(needle),
})
.unwrap_or(false),
),
// Widget-value asserts now read the LIVE widget's AccessKit value /
// toggled state from the tree — for label-addressed targets. An
// id-addressed target (or no tree yet) stays neutral (`None`), as
// before, so nothing regresses to a false FAIL.
Assert::FieldValue { target: Target::Label(label), value } => self
.ak_lookup(label)
.and_then(|h| h.value)
.map(|actual| &actual == value),
Assert::FieldChecked { target: Target::Label(label), checked } => self
.ak_lookup(label)
.and_then(|h| h.toggled)
.map(|actual| actual == *checked),
Assert::FieldValue { .. } | Assert::FieldChecked { .. } => None,
}
}
/// The cursor's destination this step: an explicit drag end, else the resolved
/// target rect centre, else stay put.
fn destination(&self) -> Option<egui::Pos2> {
if self.idx >= self.steps.len() {
return None;
}
match &self.steps[self.idx].action {
Action::Drag { to, .. } => Some(*to),
_ => self.target_rect.map(|r| r.center()),
}
}
}
/// Ease-in-out so the cursor accelerates then settles (reads as "aimed", not
/// linear-robotic).
fn ease(t: f32) -> f32 {
let t = t.clamp(0.0, 1.0);
t * t * (3.0 - 2.0 * t)
}
fn lerp(a: egui::Pos2, b: egui::Pos2, t: f32) -> egui::Pos2 {
egui::pos2(a.x + (b.x - a.x) * t, a.y + (b.y - a.y) * t)
}
impl<A: Observable, F: FnMut(&mut egui::Ui, &mut A, &mut Targets)> eframe::App
for WatchDriver<A, F>
{
/// Inject synthetic input for the current step into the frame's `RawInput`,
/// BEFORE egui processes it — the real seam that makes the widgets react.
fn raw_input_hook(&mut self, _ctx: &egui::Context, raw_input: &mut egui::RawInput) {
if !self.started || self.idx >= self.steps.len() {
return;
}
let d = self.step_delay_secs();
let frac = self.phase_started.elapsed().as_secs_f32() / d;
let modifiers = egui::Modifiers::default();
// Always report where the cursor is, so egui hovers/hit-tests there.
raw_input.events.push(egui::Event::PointerMoved(self.pointer));
let action = self.steps[self.idx].action.clone();
match &action {
Action::ClickLabel(_)
| Action::ClickButton(_)
| Action::ClickById(_)
| Action::SetField { value: FieldValue::Bool(_), .. }
| Action::SetField { value: FieldValue::Number(_), .. } => {
if let Some(rect) = self.target_rect {
self.click_at(raw_input, rect.center(), frac, modifiers);
}
}
Action::TypeInto { text, .. } | Action::SetField { value: FieldValue::Text(text), .. } => {
if let Some(rect) = self.target_rect {
// Click to focus, then type.
self.click_at(raw_input, rect.center(), frac, modifiers);
if frac >= ACT_FRAC && !self.did_act {
raw_input.events.push(egui::Event::Text(text.clone()));
self.did_act = true;
}
}
}
Action::Key(key) => {
if frac >= PRESS_FRAC && !self.did_act {
raw_input.events.push(egui::Event::Key {
key: *key,
physical_key: None,
pressed: true,
repeat: false,
modifiers,
});
raw_input.events.push(egui::Event::Key {
key: *key,
physical_key: None,
pressed: false,
repeat: false,
modifiers,
});
self.did_act = true;
}
}
Action::Scroll(delta) => {
if frac >= PRESS_FRAC && !self.did_act {
raw_input.events.push(egui::Event::MouseWheel {
unit: egui::MouseWheelUnit::Point,
delta: *delta,
phase: egui::TouchPhase::Move,
modifiers,
});
self.did_act = true;
}
}
Action::Drag { from, to } => {
if frac >= 0.15 && !self.did_press {
raw_input.events.push(egui::Event::PointerButton {
pos: *from,
button: egui::PointerButton::Primary,
pressed: true,
modifiers,
});
self.did_press = true;
}
if frac >= 0.85 && self.did_press && !self.did_release {
raw_input.events.push(egui::Event::PointerButton {
pos: *to,
button: egui::PointerButton::Primary,
pressed: false,
modifiers,
});
self.did_release = true;
}
}
Action::InjectEvents(events) => {
if frac >= PRESS_FRAC && !self.did_act {
for e in events {
raw_input.events.push(e.clone());
}
self.did_act = true;
}
}
Action::Step | Action::None => {}
}
}
/// Draw one frame: a status bar, the consumer UI, the synthetic cursor + a
/// verdict strip; then animate the cursor and advance the step machine.
/// eframe hands us the central-panel `Ui`; `update` is deprecated in favour
/// of this.
fn ui(&mut self, ui: &mut egui::Ui, _frame: &mut eframe::Frame) {
let ctx = ui.ctx().clone();
// First paint: start the clock so timing begins after something is drawn.
if !self.started {
self.started = true;
self.phase_started = Instant::now();
}
let idx = self.idx;
let total = self.steps.len();
let repo = self.repo.clone();
let step_label = self.steps.get(idx).map(|s| s.label.clone());
// ── status bar ───────────────────────────────────────────────────────
ui.horizontal(|ui| {
ui.strong("robotui watch");
ui.separator();
ui.label(format!("repo: {repo}"));
ui.separator();
if idx < total {
ui.label(format!("step {}/{}", idx + 1, total));
if let Some(l) = &step_label {
ui.separator();
ui.monospace(l);
}
} else {
ui.label(format!("done — {total} steps"));
}
});
ui.separator();
// ── consumer UI (disjoint field borrows) ─────────────────────────────
{
let draw = &mut self.draw;
let app = &mut self.app;
let targets = &mut self.targets;
targets.clear();
draw(ui, app, targets);
}
// ── refresh the live AccessKit tree captured from last frame's output ──
// The scale must be re-read EVERY frame, not once at startup: dragging the
// window to a differently-scaled output changes it mid-run, and on Wayland
// the first frame can report 1.0 before the compositor has told us the
// output's real scale.
self.ppp = ctx.pixels_per_point();
self.refresh_accesskit();
// ── resolve this step's target rect: AccessKit tree first (any labelled
// widget, no app plumbing), then the recorded `Targets` rect fallback ──
self.target_rect = self
.steps
.get(idx)
.and_then(|s| Self::action_label(&s.action))
.and_then(|label| self.resolve_rect(label));
// ── animate the cursor toward its destination ─────────────────────────
let d = self.step_delay_secs();
let elapsed = self.phase_started.elapsed().as_secs_f32();
if let Some(dest) = self.destination() {
let t = ease(elapsed / (d * TRAVEL_FRAC));
self.pointer = lerp(self.move_from, dest, t);
}
// ── cursor + verdict overlay on top ───────────────────────────────────
self.paint_cursor(&ctx);
self.paint_verdicts(&ctx);
// ── advance / finish ──────────────────────────────────────────────────
if idx < total {
if elapsed >= d {
self.advance();
}
ctx.request_repaint_after(Duration::from_millis(16));
} else {
let held = self.finished_at.map(|t| t.elapsed()).unwrap_or_default();
// 1. Ask the WINDOW for its own presented frame, once, a beat after the
// last step so the final state is actually on screen. This is the
// capture that works on every backend — no compositor permission, no
// external tool (KWin refuses `spectacle` inside a nested `--virtual`
// session, and `import` cannot see a Wayland surface at all).
if self.shot_path.is_some()
&& !self.shot_requested
&& held >= Duration::from_millis(250)
{
ctx.send_viewport_cmd(egui::ViewportCommand::Screenshot(
egui::UserData::default(),
));
self.shot_requested = true;
}
// 2. Pick up the reply on whichever later frame it arrives on.
if self.shot_requested && self.shot_result.is_none() {
self.collect_window_screenshot(&ctx);
}
// 3. Report just before closing, so the capture has had time to land.
if held >= self.cfg.linger {
if !self.reported {
self.reported = true;
self.write_report();
}
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
} else {
ctx.request_repaint_after(Duration::from_millis(50));
}
}
}
}
impl<A: Observable, F: FnMut(&mut egui::Ui, &mut A, &mut Targets)> WatchDriver<A, F> {
/// Emit a press then (a beat later) a release at `pos`, latched so each fires
/// once — a synthetic click egui registers on the widget under `pos`.
fn click_at(&mut self, raw_input: &mut egui::RawInput, pos: egui::Pos2, frac: f32, modifiers: egui::Modifiers) {
if frac >= PRESS_FRAC && !self.did_press {
raw_input.events.push(egui::Event::PointerMoved(pos));
raw_input.events.push(egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed: true,
modifiers,
});
self.did_press = true;
}
if frac >= RELEASE_FRAC && self.did_press && !self.did_release {
raw_input.events.push(egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed: false,
modifiers,
});
self.did_release = true;
}
}
/// Look for the `Event::Screenshot` reply to our
/// [`egui::ViewportCommand::Screenshot`] and, if it has arrived, encode the REAL
/// window's presented frame to PNG.
///
/// The reply lands on some later frame (the backend has to finish and read back
/// the render), so this is polled rather than awaited.
fn collect_window_screenshot(&mut self, ctx: &egui::Context) {
let Some(path) = self.shot_path.clone() else {
return;
};
let image = ctx.input(|i| {
i.events.iter().find_map(|e| match e {
egui::Event::Screenshot { image, .. } => Some(image.clone()),
_ => None,
})
});
let Some(image) = image else { return };
let [w, h] = image.size;
// `ColorImage::as_raw` is already tightly-packed RGBA, row-major, top-down —
// exactly what the PNG encoder wants.
self.shot_result = Some(
crate::display::write_rgba_png(&path, w as u32, h as u32, image.as_raw())
.map_err(|e| e.to_string()),
);
match &self.shot_result {
Some(Ok(s)) => eprintln!(
"robotui watch: captured the live window ({}x{}, {} bytes) to {}",
w,
h,
s.bytes,
s.path.display()
),
Some(Err(e)) => eprintln!("robotui watch: window capture FAILED: {e}"),
None => {}
}
}
fn paint_cursor(&self, ctx: &egui::Context) {
let painter = ctx.layer_painter(egui::LayerId::new(
egui::Order::Foreground,
egui::Id::new("robotui_watch_cursor"),
));
let p = self.pointer;
let pressed = self.did_press && !self.did_release;
let fill = if pressed {
egui::Color32::from_rgb(255, 90, 90)
} else {
egui::Color32::from_rgb(90, 170, 255)
};
painter.circle_filled(p, if pressed { 11.0 } else { 8.0 }, fill.gamma_multiply(0.35));
painter.circle_stroke(p, 9.0, egui::Stroke::new(2.0_f32, fill));
painter.circle_filled(p, 2.5, fill);
}
fn paint_verdicts(&self, ctx: &egui::Context) {
let painter = ctx.layer_painter(egui::LayerId::new(
egui::Order::Foreground,
egui::Id::new("robotui_watch_verdicts"),
));
let mut x = 10.0;
let y = ctx.content_rect().bottom() - 16.0;
for (i, v) in self.verdicts.iter().enumerate() {
let (glyph, col) = match v {
Some(true) => ("PASS", egui::Color32::from_rgb(90, 200, 120)),
Some(false) => ("FAIL", egui::Color32::from_rgb(230, 90, 90)),
None if i == self.idx => ("=>", egui::Color32::from_rgb(90, 170, 255)),
None => ("..", egui::Color32::GRAY),
};
painter.text(
egui::pos2(x, y),
egui::Align2::LEFT_CENTER,
glyph,
egui::FontId::monospace(13.0),
col,
);
x += 44.0;
}
}
}
#[cfg(test)]
mod tests {
//! Unit tests for the pure (no-window) parts of the watch backend: the cursor
//! pacing math (`ease`/`lerp`), the label→action projection, the per-step
//! cursor destination, the pass/fail verdict evaluation and the step-advance
//! state machine. None of these open an eframe window — `WatchDriver` is
//! constructed directly and only its pure helpers are exercised.
use super::*;
use crate::scenario::{Action, Assert, FieldValue, Scenario, Step, Target};
use serde_json::json;
/// A minimal `Observable` whose `state_json` is a fixed blob, so verdict
/// evaluation is deterministic.
struct TestApp {
state: serde_json::Value,
}
impl crate::Observable for TestApp {
fn state_json(&self) -> serde_json::Value {
self.state.clone()
}
}
// A plain `fn` (not a closure) so the generic `WatchDriver<A, F>` type is
// nameable in tests — a `fn` item implements `FnMut` and coerces to a pointer.
type DrawFn = fn(&mut egui::Ui, &mut TestApp, &mut Targets);
fn noop_draw(_: &mut egui::Ui, _: &mut TestApp, _: &mut Targets) {}
fn driver(steps: Vec<Step>, state: serde_json::Value) -> WatchDriver<TestApp, DrawFn> {
let scenario = Scenario { suite: "t".into(), steps };
WatchDriver::new(
TestApp { state },
noop_draw as DrawFn,
&scenario,
"testrepo".to_string(),
WatchConfig::default(),
Arc::new(Mutex::new(None)),
)
}
fn a_rect(cx: f32, cy: f32) -> egui::Rect {
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(4.0, 4.0))
}
#[test]
fn ease_is_a_clamped_smoothstep() {
// Endpoints and midpoint of the classic 3t²-2t³ smoothstep.
assert_eq!(ease(0.0), 0.0);
assert_eq!(ease(1.0), 1.0);
assert_eq!(ease(0.5), 0.5);
// Clamped outside [0,1] — never overshoots the cursor destination.
assert_eq!(ease(-3.0), 0.0);
assert_eq!(ease(9.0), 1.0);
// Monotonically non-decreasing across the interval.
let mut prev = ease(0.0);
for i in 1..=20 {
let cur = ease(i as f32 / 20.0);
assert!(cur >= prev, "ease must be monotonic: {cur} < {prev}");
prev = cur;
}
// Ease-in below the midpoint (slower than linear at t=0.25).
assert!(ease(0.25) < 0.25);
// Ease-out above the midpoint (past linear at t=0.75).
assert!(ease(0.75) > 0.75);
}
#[test]
fn lerp_hits_endpoints_and_midpoint() {
let a = egui::pos2(10.0, 20.0);
let b = egui::pos2(30.0, 60.0);
assert_eq!(lerp(a, b, 0.0), a);
assert_eq!(lerp(a, b, 1.0), b);
assert_eq!(lerp(a, b, 0.5), egui::pos2(20.0, 40.0));
}
#[test]
fn action_label_extracts_only_label_addressed_targets() {
type D = WatchDriver<TestApp, DrawFn>;
assert_eq!(D::action_label(&Action::ClickLabel("Bump".into())), Some("Bump"));
assert_eq!(D::action_label(&Action::ClickButton("Save".into())), Some("Save"));
assert_eq!(
D::action_label(&Action::TypeInto { label: "Name".into(), text: "x".into() }),
Some("Name")
);
assert_eq!(
D::action_label(&Action::SetField {
target: Target::Label("Flag".into()),
value: FieldValue::Bool(true),
}),
Some("Flag")
);
// Id-addressed and input-only actions have no rect side-channel → None.
assert_eq!(
D::action_label(&Action::SetField {
target: Target::Id(7),
value: FieldValue::Number(1.0),
}),
None
);
assert_eq!(D::action_label(&Action::ClickById(42)), None);
assert_eq!(D::action_label(&Action::Key(egui::Key::Enter)), None);
assert_eq!(D::action_label(&Action::Scroll(egui::vec2(0.0, -1.0))), None);
assert_eq!(D::action_label(&Action::Step), None);
assert_eq!(D::action_label(&Action::None), None);
}
#[test]
fn destination_prefers_explicit_drag_end_over_target_rect() {
let drag = Step::new("drag").action(Action::Drag {
from: egui::pos2(1.0, 1.0),
to: egui::pos2(99.0, 88.0),
});
let mut d = driver(vec![drag], json!({}));
// Even with a resolved rect, a Drag steers to its explicit end point.
d.target_rect = Some(a_rect(5.0, 5.0));
assert_eq!(d.destination(), Some(egui::pos2(99.0, 88.0)));
}
#[test]
fn destination_uses_target_rect_centre_then_none() {
let click = Step::new("c").action(Action::ClickLabel("Bump".into()));
let mut d = driver(vec![click], json!({}));
// No rect resolved yet → nowhere to travel.
assert_eq!(d.destination(), None);
// Rect resolved → travel to its centre.
d.target_rect = Some(a_rect(50.0, 40.0));
assert_eq!(d.destination(), Some(egui::pos2(50.0, 40.0)));
// Past the last step → None regardless of any stale rect.
d.idx = 1;
assert_eq!(d.destination(), None);
}
#[test]
fn evaluate_verdict_covers_every_assert_kind() {
let steps = vec![
Step::new("none").assert(Assert::None),
Step::new("drew").assert(Assert::Drew),
Step::new("visible").assert(Assert::LabelVisible("Bump".into())),
Step::new("state").assert(Assert::State {
pointer: "/clicks".into(),
expected: json!(3),
}),
Step::new("state-miss").assert(Assert::State {
pointer: "/clicks".into(),
expected: json!(4),
}),
Step::new("contains").assert(Assert::StateContains {
pointer: "/label".into(),
needle: "ell".into(),
}),
Step::new("field").assert(Assert::FieldValue {
target: Target::Label("x".into()),
value: "y".into(),
}),
];
let mut d = driver(steps, json!({ "clicks": 3, "label": "hello" }));
// None is always neutral.
assert_eq!(d.evaluate_verdict(0), None);
// Drew reflects whether the closure registered any targetable widget.
assert_eq!(d.evaluate_verdict(1), Some(false));
d.targets.record_rect("Bump", a_rect(1.0, 1.0));
assert_eq!(d.evaluate_verdict(1), Some(true));
// LabelVisible tracks the targets registry.
assert_eq!(d.evaluate_verdict(2), Some(true));
assert_eq!(
driver(
vec![Step::new("v").assert(Assert::LabelVisible("Ghost".into()))],
json!({})
)
.evaluate_verdict(0),
Some(false)
);
// State: exact JSON-pointer match, and a mismatch is a hard fail.
assert_eq!(d.evaluate_verdict(3), Some(true));
assert_eq!(d.evaluate_verdict(4), Some(false));
// StateContains: substring of the stringified pointer value.
assert_eq!(d.evaluate_verdict(5), Some(true));
// Widget-value asserts have no side-channel in watch mode → neutral.
assert_eq!(d.evaluate_verdict(6), None);
}
/// `Assert::All` — the chained-assert accumulator — in the WATCH overlay.
/// The overlay must show the CONJUNCTION, not the last clause: before the
/// fix a chain collapsed to its final clause here exactly as it did headless.
#[test]
fn evaluate_verdict_of_a_chain_is_the_conjunction_not_the_last_clause() {
let state = json!({ "clicks": 3, "label": "hello" });
let good = || Assert::State { pointer: "/clicks".into(), expected: json!(3) };
let bad = || Assert::State { pointer: "/clicks".into(), expected: json!(4) };
let neutral = || Assert::FieldValue {
target: Target::Id(7),
value: "y".into(),
};
// false-then-true: the exact shape that reported green before the fix.
let mut d = driver(
vec![Step::new("bad then good").assert(bad()).assert(good())],
state.clone(),
);
assert_eq!(d.evaluate_verdict(0), Some(false), "a false clause anywhere is red");
// true-then-false: the last clause is checked too.
let mut d = driver(
vec![Step::new("good then bad").assert(good()).assert(bad())],
state.clone(),
);
assert_eq!(d.evaluate_verdict(0), Some(false));
// all-true is green.
let mut d = driver(
vec![Step::new("good then good").assert(good()).assert(good())],
state.clone(),
);
assert_eq!(d.evaluate_verdict(0), Some(true));
// A clause that cannot be evaluated in watch mode does not drag a known
// TRUE down to neutral, and does not invent a verdict on its own.
let mut d = driver(
vec![Step::new("good then neutral").assert(good()).assert(neutral())],
state.clone(),
);
assert_eq!(d.evaluate_verdict(0), Some(true));
let mut d = driver(
vec![Step::new("neutral only").assert(neutral()).assert(neutral())],
state,
);
assert_eq!(d.evaluate_verdict(0), None, "nothing evaluatable → neutral");
}
#[test]
fn evaluate_verdict_missing_pointer_is_a_fail_not_a_panic() {
let mut d = driver(
vec![Step::new("s").assert(Assert::State {
pointer: "/nope".into(),
expected: json!(1),
})],
json!({ "clicks": 1 }),
);
// Absent pointer resolves to `false`, never unwraps None.
assert_eq!(d.evaluate_verdict(0), Some(false));
// StateContains on an absent pointer is likewise a clean fail.
d.steps[0].assert = Assert::StateContains {
pointer: "/nope".into(),
needle: "x".into(),
};
assert_eq!(d.evaluate_verdict(0), Some(false));
}
#[test]
fn advance_latches_verdict_resets_per_step_state_and_finishes() {
let steps = vec![
Step::new("drew").assert(Assert::Drew),
Step::new("noop").assert(Assert::None),
];
let mut d = driver(steps, json!({}));
// A targetable widget was drawn → step 0's Drew verdict should latch true.
d.targets.record_rect("Bump", a_rect(1.0, 1.0));
// Dirty the per-step latches and move the cursor so we can see the reset.
d.did_press = true;
d.did_release = true;
d.did_act = true;
d.pointer = egui::pos2(123.0, 45.0);
d.target_rect = Some(a_rect(9.0, 9.0));
d.advance();
assert_eq!(d.idx, 1, "advanced to the next step");
assert_eq!(d.verdicts[0], Some(true), "step 0 verdict latched");
assert!(!d.did_press && !d.did_release && !d.did_act, "latches reset");
assert!(d.target_rect.is_none(), "target rect cleared for the new step");
assert_eq!(d.move_from, egui::pos2(123.0, 45.0), "travel origin = current pointer");
assert!(d.finished_at.is_none(), "not finished with one step left");
// Advancing past the last step latches step 1 and marks finished exactly once.
d.advance();
assert_eq!(d.idx, 2);
assert_eq!(d.verdicts[1], None, "Assert::None stays neutral");
let finished = d.finished_at;
assert!(finished.is_some(), "finished_at set when past the last step");
// A further advance must not re-stamp finished_at (idempotent finish).
d.advance();
assert_eq!(d.finished_at, finished, "finish timestamp not overwritten");
}
#[test]
fn step_delay_secs_has_a_nonzero_floor() {
let mut d = driver(vec![Step::new("s")], json!({}));
d.cfg.step_delay = std::time::Duration::ZERO;
// A zero delay would divide-by-zero the cursor animation; the floor guards it.
assert!(d.step_delay_secs() >= 0.001);
d.cfg.step_delay = std::time::Duration::from_millis(500);
assert!((d.step_delay_secs() - 0.5).abs() < 1e-6);
}
// ─── AccessKit resolution (FIX 1): label → rect from the LIVE tree ─────────
/// Build a real AccessKit `TreeUpdate` by running `render` through a raw egui
/// `Context` (accesskit forced on) for one frame — the exact tree eframe emits,
/// captured with no window. This is the same `platform_output.accesskit_update`
/// the `AccessKitTap` plugin taps in a live watch run.
fn tree_of(mut render: impl FnMut(&mut egui::Ui)) -> TreeUpdate {
let ctx = egui::Context::default();
ctx.enable_accesskit();
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(400.0, 300.0),
)),
..Default::default()
};
// `run_ui` hands the closure a root `Ui` over the available rect, which is
// what the old `run` + `CentralPanel::show` pair was building by hand; both
// halves of that pair are deprecated in egui 0.34.
let mut out = ctx.run_ui(input, |ui| render(ui));
// egui 0.36 debug-asserts on a FullOutput dropped with unapplied texture
// deltas; this headless helper uploads nothing, so it clears them (2026-10-03).
out.textures_delta.clear();
out.platform_output
.accesskit_update
.expect("accesskit enabled → a tree is produced")
}
/// A fn-pointer draw so the generic `WatchDriver<A, F>` is nameable in the test.
type BumpDraw = fn(&mut egui::Ui, &mut crate::demo::BumpApp, &mut Targets);
fn bump_noop(_: &mut egui::Ui, _: &mut crate::demo::BumpApp, _: &mut Targets) {}
/// END-TO-END proof of FIX 1: the demo's "Bump" button resolves to a click
/// rect via the LIVE AccessKit tree with **no `Targets` recorded at all** —
/// exactly the plumbing-free drive the consumer apps can now rely on. Also
/// proves the `.labelled_by` "Name" field resolves to its `TextEdit` and that a
/// ghost label falls through to `None`.
#[test]
fn accesskit_resolves_demo_widgets_with_no_targets() {
use crate::demo::{bump_draw, bump_scenario, BumpApp};
// Render the demo to a tree via a THROWAWAY Targets the driver never sees.
let mut app = BumpApp::default();
let tree = tree_of(|ui| {
let mut throwaway = Targets::default();
bump_draw(ui, &mut app, &mut throwaway);
});
// A driver with an EMPTY Targets map — resolution must be pure AccessKit.
let scenario = bump_scenario();
let mut d = WatchDriver::new(
BumpApp::default(),
bump_noop as BumpDraw,
&scenario,
"demo".to_string(),
WatchConfig::default(),
Arc::new(Mutex::new(None)),
);
d.ak_state = Some(AkState::new(tree));
assert!(d.targets.is_empty(), "no rails recorded — this is the plumbing-free path");
// The Bump button resolves to a real click rect via the AccessKit tree.
let rect = d
.accesskit_rect("Bump")
.expect("Bump resolves via the live AccessKit tree, no Targets");
assert!(rect.area() > 0.0, "a real button rect, got {rect:?}");
// resolve_rect (the click path) prefers AccessKit and needs no Targets.
assert_eq!(d.resolve_rect("Bump"), Some(rect));
// The `.labelled_by`-linked field resolves to the TextEdit, not the caption.
assert!(
d.accesskit_rect("Name").is_some(),
"labelled_by 'Name' field resolves via AccessKit"
);
// A widget that isn't there falls through to None (then Targets, also empty).
assert!(d.accesskit_rect("Ghost").is_none());
assert!(d.resolve_rect("Ghost").is_none());
// Drew is satisfied by the AccessKit tree alone (Targets empty).
assert!(d.drew_anything(), "the AccessKit tree alone proves the UI drew");
}
// ─── HiDPI: the AccessKit rect is in PIXELS, the pointer events in POINTS ──
/// Drive a REAL `egui::Context` at `pixels_per_point`, resolving the click
/// target through the driver's **own production path** (`resolve_rect` over a
/// live AccessKit tree, with an EMPTY `Targets` map so nothing but AccessKit can
/// answer), then inject the click at that position into the same context's
/// `RawInput` — exactly the events `raw_input_hook` pushes in a live window.
///
/// Returns the APPLIED result: the app's `Observable::state_json` afterwards,
/// plus the rect the driver aimed at. This asserts what the widget *became*,
/// never that a function was called or that a rect was non-`None` — a driver
/// that resolves a beautiful rect and misses by 2× returns `Some(rect)` all day.
fn click_bump_at_scale(pixels_per_point: f32) -> (serde_json::Value, Option<egui::Rect>) {
use crate::demo::{bump_draw, bump_scenario, BumpApp};
let ctx = egui::Context::default();
ctx.enable_accesskit();
let mut app = BumpApp::default();
// A `RawInput` carrying the display scale the way a real backend reports it.
let input_at = |ppp: f32| {
let mut input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(400.0, 300.0),
)),
..Default::default()
};
input
.viewports
.entry(egui::ViewportId::ROOT)
.or_default()
.native_pixels_per_point = Some(ppp);
input
};
let mut frame = |ctx: &egui::Context, app: &mut BumpApp, input: egui::RawInput| {
let mut out = ctx.run_ui(input, |ui| {
let mut throwaway = Targets::default();
bump_draw(ui, app, &mut throwaway);
});
out.textures_delta.clear(); // egui 0.36: nothing is uploaded headless
out
};
// Frame 1 — draw, and capture the tree the `AccessKitTap` plugin would see.
let out = frame(&ctx, &mut app, input_at(pixels_per_point));
let tree = out
.platform_output
.accesskit_update
.expect("accesskit enabled → a tree is produced");
assert_eq!(
ctx.pixels_per_point(),
pixels_per_point,
"the context really is running at the scale under test"
);
// Resolve through the driver, at the scale the driver would have read.
let scenario = bump_scenario();
let mut d = WatchDriver::new(
BumpApp::default(),
bump_noop as BumpDraw,
&scenario,
"hidpi".to_string(),
WatchConfig::default(),
Arc::new(Mutex::new(None)),
);
d.ak_state = Some(AkState::new(tree));
d.ppp = ctx.pixels_per_point();
assert!(d.targets.is_empty(), "AccessKit is the only rect source here");
let rect = d.resolve_rect("Bump");
// Frames 2 + 3 — press then release where the driver aimed, in POINT space.
if let Some(r) = rect {
let pos = r.center();
let mut press = input_at(pixels_per_point);
press.events.push(egui::Event::PointerMoved(pos));
press.events.push(egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed: true,
modifiers: egui::Modifiers::default(),
});
frame(&ctx, &mut app, press);
let mut release = input_at(pixels_per_point);
release.events.push(egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed: false,
modifiers: egui::Modifiers::default(),
});
frame(&ctx, &mut app, release);
}
(app.state_json(), rect)
}
/// **The guard this whole change exists for.** A watch click must land on the
/// widget at a display scale that is NOT 1 — oden's KDE Wayland desktop is 2×.
///
/// RED against the pre-fix driver: `node_rect` returned
/// `AccessKitNode::bounding_box()` unchanged, which egui has already multiplied
/// by `pixels_per_point` (it sets `Affine::scale(ppp)` on the AccessKit root),
/// so at 2× the driver aimed at twice the widget's coordinates, the click landed
/// on nothing, and `clicks` stayed 0 while the driver cheerfully reported a
/// resolved rect.
#[test]
fn a_watch_click_lands_on_the_widget_at_a_2x_display_scale() {
let (state, rect) = click_bump_at_scale(2.0);
let rect = rect.expect("Bump resolves via the live AccessKit tree at 2×");
// The rect must be inside the 400×300 POINT viewport. Un-divided it would be
// in pixels — up to 800×600 — i.e. mostly off the widget, often off-screen.
assert!(
rect.max.x <= 400.0 && rect.max.y <= 300.0,
"the click rect must be in POINTS (inside the 400x300 viewport), got {rect:?} \
— a rect this large is the raw pixel bounding box"
);
// The APPLIED result: the button actually incremented the counter.
assert_eq!(
state.pointer("/clicks"),
Some(&json!(1)),
"a synthetic click at the resolved rect must actually press Bump at 2× \
scale; state_json says {state}"
);
}
/// Scale 1 must keep working — and this test documents WHY the bug survived so
/// long. At `pixels_per_point == 1.0` the pixel and point spaces are identical,
/// so the buggy and the fixed code are indistinguishable here. Every robot drive
/// on this fleet ran under `Xvfb`, which is always scale 1. This assertion can
/// never fail for the bug it is next to — that is the point of writing it down.
#[test]
fn scale_one_still_works_and_is_exactly_why_xvfb_hid_the_bug() {
let (state, rect) = click_bump_at_scale(1.0);
assert!(rect.is_some(), "Bump resolves at 1×");
assert_eq!(
state.pointer("/clicks"),
Some(&json!(1)),
"the 1× path is unchanged: {state}"
);
}
/// The scale conversion is a real division, not a constant. Resolving the SAME
/// widget at 1× and at 2× must yield the SAME rect in points — if `node_rect`
/// ignored `pixels_per_point`, the 2× rect would be exactly twice as big.
///
/// This is the identity-value trap named explicitly: a test that only ever runs
/// at 1.0 sits on the identity and cannot tell a working transform from a dead
/// one, so this one compares two different scales against each other.
#[test]
fn the_resolved_rect_is_scale_invariant_in_point_space() {
let (_, at_1x) = click_bump_at_scale(1.0);
let (_, at_2x) = click_bump_at_scale(2.0);
let (a, b) = (at_1x.expect("1x rect"), at_2x.expect("2x rect"));
let d = (a.center() - b.center()).length();
assert!(
d < 1.0,
"the same widget must resolve to the same POINT rect at any display \
scale: 1x={a:?} 2x={b:?} (centres {d} points apart — a factor-of-2 gap \
is the un-divided pixel bounding box)"
);
}
/// A degenerate scale must not produce an infinite or NaN rect that would send
/// the cursor to nowhere; the conversion falls back to 1:1.
#[test]
fn a_zero_or_nan_scale_falls_back_to_one_to_one() {
use crate::demo::{bump_draw, BumpApp};
let mut app = BumpApp::default();
let tree = tree_of(|ui| {
let mut t = Targets::default();
bump_draw(ui, &mut app, &mut t);
});
let state = AkState::new(tree);
for bad in [0.0_f32, f32::NAN, -2.0, f32::INFINITY] {
let hit = ak_lookup(&state, "Bump", bad).expect("Bump is in the tree");
let r = hit.rect.expect("a rect");
assert!(
r.is_finite() && r.area() > 0.0,
"scale {bad} must fall back to 1:1, got {r:?}"
);
}
}
/// Back-compat: with NO AccessKit tree captured yet, `resolve_rect` still falls
/// back to a recorded `Targets` rect (the pre-existing rail path is preserved).
#[test]
fn resolve_rect_falls_back_to_targets_when_accesskit_absent() {
let mut d = driver(vec![Step::new("s")], json!({}));
assert!(d.ak_state.is_none(), "no tree captured");
// No AccessKit, no target → nothing.
assert!(d.resolve_rect("Bump").is_none());
// Record a rail rect → the fallback resolves it (unchanged behaviour).
d.targets.record_rect("Bump", a_rect(50.0, 40.0));
assert_eq!(
d.resolve_rect("Bump").map(|r| r.center()),
Some(egui::pos2(50.0, 40.0)),
"Targets rect is used when AccessKit has no match"
);
}
}