nordisk/nornir

EPIC R6 — Richer live-drive for the robot-UI tester

Goal: full widget-level robot control beyond tab/workspace clicking. Set any form field, click any widget by AccessKit id, and inject arbitrary synthetic egui events — and have the SAME verbs work both in-process (egui_kittest harness) and over the live control channel (the running viz the viz.click MCP tool drives). CLI/automation parity with the GUI (LAW).

1. Why

Before R6 the robot could click labels/buttons, type into a linked-label field, key/scroll/drag, and step. That covers navigation but not arbitrary forms: a TextEdit with only hint_text, two buttons with the same text, an icon-only button, a slider, a checkbox you want set (not blindly flipped). R6 adds three escape-hatch verbs at descending levels of abstraction:

2. The API (nornir-robotui)

New public types in crates/nornir-robotui/src/scenario.rs:

pub enum Target { Label(String), Id(u64) }          // Target::label("x") / Target::id(7)
pub enum FieldValue { Text(String), Bool(bool), Number(f64) }

New RobotSession methods (crates/nornir-robotui/src/lib.rs):

fn click_by_id(&mut self, id: u64) -> anyhow::Result<()>;
fn set_field(&mut self, target: &Target, value: &FieldValue) -> anyhow::Result<()>;
fn inject_events(&mut self, events: impl IntoIterator<Item = egui::Event>);
// discover + assert against the AccessKit tree:
fn find_id(&self, target: &Target) -> Option<u64>;       // resolve a Target → node id
fn node_id(&self, label: &str) -> Option<u64>;           // a label's node id
fn field_value(&self, target: &Target) -> Option<String>; // the widget's AccessKit value
fn field_checked(&self, target: &Target) -> Option<bool>; // a toggle's state

• set_field chooses the gesture from the widget's AccessKit role.

  • Text → focus, select-all (Cmd/Ctrl-A) + Backspace, then type_text — a set (replace), not an append.
  • Bool → read toggled(), click only if it differs → idempotent set.
  • Number → emit the AccessKit SetValue action (ActionData::NumericValue) egui honours for sliders / DragValue — the screen-reader path.

• The target is resolved to a node id first (a Copy u64) so no &str borrow of the Target gets tangled into the AccessKit tree's lifetime, and so a widget can be driven across frames (the tree is rebuilt each frame).

Scenario (declarative / TOML-able) mirror

Action grew ClickById(u64), SetField { target, value }, InjectEvents(Vec<egui::Event>). Assert grew FieldValue { target, value } and FieldChecked { target, checked } — these read the value/toggle straight off the rendered widget, independent of state_json. All plain data, so a scenario still round-trips through a .toml/JSON file.

3. Live-channel parity (the running viz)

The in-process verbs have a drive-half on the file control channel (src/viz/control.rs), so the viz.click MCP tool can drive a LIVE viz, not just the in-process harness. VizCommand grew:

{ "set_field": { "name": "funnel.intake", "value": "add dark mode" },
  "click_id":  "funnel.submit_intake" }

Because AccessKit node ids are not stable across viz runs, the live channel addresses widgets by a stable key ("<tab>.<field|button>") instead of a raw id — the durable equivalent of click_by_id. UrdrThreadsApp::apply_command routes set_field/click_id by tab prefix to the owning tab. Today the Funnel tab is wired (set_field_for_drive / click_for_drive run the SAME code paths the GUI buttons do):

The viz.click MCP tool exposes set_field (as "name=value") and click_id strings, so the whole chain is CLI/agent-drivable.

4. Tests (inject-assert LAW — real input, real output)

• crates/nornir-robotui unit tests (self-contained egui app with a button, text field, checkbox, slider): set_field for text/bool/number, click_by_id, inject_events, and the full scenario path incl. a deliberately-wrong FieldValue → a red matrix cell (not a panic).
• tests/robot_viz.rs against the REAL viz: robot_richer_livedrive_sets_field_clicks_by_id_and_injects_events navigates to the 🗂 Funnel tab and (a) sets the paste-in textbox via set_field, asserting state_json["funnel"]["intake"]["text"] AND the widget's AccessKit value; (b) clicks 💡 Submit by AccessKit id, asserting the idea landed in funnel.history; (c) injects a raw Focus + Text event sequence, asserting the text landed in the real TextEdit. A second test runs the same verbs through run_scenario → matrix rows.

Note: in local mode the action-log's durable warehouse sink holds the catalog's exclusive redb lock for the app's lifetime, which blocks the Funnel tab's Store::open. NORNIR_VIZ_NO_DURABLE_ACTIONLOG=1 skips that sink (file/stderr action-log stays live) so another in-process consumer can open the warehouse; the R6 viz test sets it.

5. Idiot guide — drive an arbitrary widget

use nornir_robotui::{RobotSession, Target, FieldValue};

// set any text field (by linked label OR raw node id)
robot.set_field(&Target::label("paste:"), &FieldValue::Text("hello".into()))?;
robot.set_field(&Target::id(node_id),     &FieldValue::Bool(true))?;     // checkbox
robot.set_field(&Target::label("Amount"), &FieldValue::Number(42.0))?;   // slider

// click a widget by its AccessKit node id (discover it first)
let id = robot.node_id("💡 Submit idea").unwrap();
robot.click_by_id(id)?;

// the raw escape hatch: inject any egui events for one frame
robot.inject_events([egui::Event::Text("typed via raw event".into())]);

// assert against the rendered widget (not just state_json)
assert_eq!(robot.field_value(&Target::label("paste:")).as_deref(), Some("hello"));

Drive the LIVE viz the same way over the MCP viz.click tool: viz.click set_field="funnel.intake=hello" click_id="funnel.submit_intake".