4 Commits

Author	SHA1	Message	Date
Michał Pierzchała	3bf3ff130a	fix: Linux input/a11y defects from #1935 (click miss, typed '=', GTK4 text) (#1949) ... * diag: instrument Linux CI to gather evidence for #1935 input/a11y defects Temporary — adds a diagnostic step that dumps raw AT-SPI interfaces/actions for gnome-calculator's digit buttons, tests a raw xdotool click at a button's own rect (bypassing our promotion logic), and isolates the typed '=' character in several configurations. Will be removed once the real fixes land. * diag: harden diagnostic step against bash -e and AT-SPI registration races The prior version crashed 7s in: GH Actions runs steps under bash -e, and an unguarded python3 heredoc threw (iterating a dict instead of a list when the app wasn't found yet), aborting the rest of the script silently under continue-on-error. Guards every fallible command, and replaces the fixed 2s sleep with inspect.py's own poll-until-found loop. * diag: test WINDOW coordtype and static Text.get_text call (round 3) Round 2 proved Component.get_extents(SCREEN) returns (0,0) for every non-toplevel widget (real click miss confirmed on-screen), and Text.get_text() throws — a documented PyGObject binding collision with the deprecated 1-arg Accessible.get_text(). This narrows to the two candidate fixes before writing them: does CoordType.WINDOW give usable relative offsets, and does Atspi.Text.get_text(accessible, ...) (static call) return the real typed text. * fix(linux): resolve click-miss, dropped '=', and GTK4 text exposure defects Three defects surfaced by CI on #1935 (Linux Smoke lane), all confirmed live via instrumented CI runs before being fixed here: 1. Click misses its target: Component.get_extents(Atspi.CoordType.SCREEN) returns (0, 0) as the origin for every non-toplevel widget under this GTK4 build — confirmed by a raw click at the computed rect center landing on the window's own header-bar button instead of the intended digit button. CoordType.WINDOW gives correct, distinct per-widget offsets, so get_rect() now computes screen-absolute rects as that offset plus the enclosing top-level frame's own (correct) screen origin, threaded through traverse_node() alongside the existing window-title tracking. Complementary hardening: role "label" is now excluded from `hittable`, since GTK4 wraps every button's caption in a same-rect "label" child, and the shared cross-platform promotion logic in interaction-targeting.ts would otherwise retarget a click from the button onto that non-interactive label. 2. Typed '=' never arrives: a single isolated synthetic keystroke sent right after a focus change is unreliably delivered — confirmed live, both `xdotool type -- "="` and `xdotool key equal` sent alone produced no character at all, while multi-character bursts always landed in full. typeLinux and sendKey now wait a short settle margin before dispatching to xdotool/ydotool, absorbing the race regardless of which action last changed focus. 3. GTK4 apps expose no editable text: accessible.get_text_iface().get_text() throws "Atspi.Accessible.get_text() takes exactly 1 argument (3 given)" — a documented PyGObject binding collision between Text.get_text and the deprecated 1-argument Accessible.get_text, silently swallowed as "no text" by the broad exception handler. get_text_value() now calls the unbound Atspi.Text.get_text(accessible, ...) form, which correctly returns the real content. The Linux smoke replay is restored to exercise all three fixes together (click a resolved digit button, type a full calculation including the '=' keystroke, wait on the computed result through the tree) instead of staying at the weakened, contract-tier assertions the defects had forced. The coverage manifest promotes click and type from command-contract to live accordingly. * fix(linux): drop unproven keyboard-settle and hittable changes per review Addresses thymikee's review on #1949 (both points correct): P1: the keyboard settle (typeLinux/sendKey) was unjustified. The cited diagnostic evidence for a dropped '=' actually shows the opposite — "100+55=" and "5=5" both computed correctly with zero settle, proving '=' was delivered in every multi-character burst tested. Sending '=' alone to an empty entry showing a blank display is normal calculator semantics (nothing to evaluate), not a lost keystroke. The likelier explanation for the original "100+55" screenshot (run 32487868346) is that its attempt-3 hit the already-fixed mousemove --sync hang, not an independent keyboard-dispatch defect. Reverted; no keyboard-dispatch change was needed. P2: the `role_name != "label"` hittable narrowing was extra surface beyond what the click-miss fix required. The corrected AT-SPI coordinates alone fix the observed miss — the button and its same-rect label child resolve to nearly identical centers, so descendant promotion still lands inside the button either way, and the replay can't distinguish which node it actually targeted. Reverted; only the coordinate fix remains.	2026-08-21 20:02:17 +02:00
Michał Pierzchała	81409f1a7c	refactor: migrate type to the request-bound device runtime (#1935) ... * refactor: migrate type to the request-bound device runtime Wave 5 unit 2 for #1739 (ADR 0019), find's last blocker. `type "text"` and `find <q> type "text"` now reach the device through one admitted, request-bound `typeText` operation instead of the `handleTypeCommand` interactor leaf and its dispatch-table arm. - New `TypeTextRuntimeOperations` contract riding the same `Interactor` seam as focus/screenshot/element-text; the operation returns the interactor's own closed `TypeTextBackendResult`, so Apple route evidence passes through and every other owner types blind, exactly as before. The iOS synthesized-type commit wait (#1676) is Apple-interactor-internal and moves nowhere. - The interaction backend's `typeText` member exists only when the `type` handler admitted and bound a runtime — no caller can fall back to legacy dispatch, so the command keeps exactly one execution path (R41). - `executeBoundTypeText` reproduces the retired leaf byte-for-byte: leading-ref rejection with the same hint, space-joined positionals, the 0-10000 delay bound, and only textEntryRoute surviving from the owner's result. Its parse pins moved from the dispatch-level tests into the daemon runtime test. - Exact-owner facts replace the capability bucket (apple sim+device, android all-but-simulator-row, harmonyos emulator+device, linux device, web device, vega unavailable, providers wherever their interactor is reachable) and `type` leaves HARMONYOS_SUPPORTED_COMMANDS / WEB_INTERACTION_COMMANDS. - The Linux desktop replay types a digit on real hardware; the coverage manifest promotes `type` contract -> live with the two-sided count pins. - Android/webdriver facts helpers extracted (androidTouchFact, interactorCell) to keep inspectFacts under the complexity gate. `find` stays legacy: both of its direct execution legs now share bound runtimes, so the atomic R35 cutover is next. * refactor(type): single-pass daemon routing, shared binder source, owner cell tests Review follow-ups on #1935. - The type handler now calls the bound executor directly: the interaction- runtime hop validated and formatted what executeBoundTypeText validates and formats again, so it is gone — no boundTypeText backend member, no second result rebuild. The ADR 0014 frame expiry moves to the handler. - New contracts/interactor-operation-binding.ts: one local resolver and one fail-closed provider resolver shared by the screenshot, focus, and type binders — three private copies retired, provider error text preserved. - provider-limrun/interaction-operations.ts: the interactor-backed interaction cells move out of the app-log owner (586 -> 563 lines, below its pre-unit size); text interaction is composed by that owner, not defined in it. - Every owner runtime test now pins the focusPoint/typeText fact cells and bound-operation presence for its exact kinds: apple, android (incl. the synthetic-simulator refusal), harmonyos, linux, web, vega (refusal + hint), webdriver (reachability-gated, incl. inactive session), limrun (live + recovery). The webdriver unsupported-capability row documents that interaction gates on interactor reachability, not capture declarations. - The Linux replay assertion is now change-sensitive: type "555" then wait for a 555 node — no calculator button carries that label, so the wait passes only if the keystrokes landed in the display; deleting the type step turns it red. * fix(replay): give the calculator focus before the Linux type assertion The change-sensitive wait exposed what the review predicted: the typed digits never landed, because `focus 100 100` clicks the DESKTOP and takes keyboard focus away from the calculator. The old broad assertion masked exactly this. The retries then wedged on a latent quirk: attempt-1 leaves the pointer at (100,100), and the next attempt's `xdotool mousemove --sync` to the same point waits for a motion event that never comes, so every retry dies at the focus step with a 10s timeout — which is why the lane reported step 7, not the failing wait. New tail: `focus 100 100` (R40 evidence + survival assert), then `click "label=1"` — a resolved press inside the window that restores keyboard focus, proves pointer input lands in the app, and moves the pointer off (100,100) so retries cannot trip the mousemove no-op hang — then `type "55"` and `wait "label=155 || text=155 || value=155"`. No button is labelled 155, so the wait passes only if the typed keystrokes reached the display. * refactor(type): delete the fallow SDK typeText surface, drop dead surface fields Thermo-nuclear review follow-ups (reviewed at 82fb8c2dc; the three-hop relay it names was already deleted in 3d44b6dea — these are the residuals). - typeTextCommand had zero production callers after the direct-executor routing: the daemon was its only consumer, and the released SDK types over the wire (executeCommand('type')), not through the embedded runtime catalog. Deleted: the command, its Options/Result types, the catalog registrations, the AgentDeviceBackend.typeText member, and every fixture/pin that kept the dead surface alive. R41's retirement claim now names typeTextCommand, so a revival fails the gate. One parse/compose owner remains: executeBoundTypeText. - TypeTextInput.options.surface and FocusPointInput.options.surface were dead clones — never set by a projector, never read by a binder. Removed both. - The Linux replay click disambiguates with role=button: the calculator tree carries [text] digit nodes beside the buttons, so a bare label=1 was an AMBIGUOUS_MATCH rejection on attempt-1 — which parked the pointer at (100,100) and made every retry hang in the xdotool no-op move, reporting as the step-7 focus timeout. Live-verified on iPhone 17 Pro at this head: coordinate focus -> bare type -> route synthesized-first-responder -> text observable in the captured tree. * fix(ci): lower the settle.test.ts pin, add pixel evidence to the Linux type wait - settle.test.ts shrank to 2359 lines when its dead typeText fixture member left; the ratchet pin follows it down (the history-backed gate caught the gap on CI while the local affected run had not re-selected the ratchet). - The Linux replay now screenshots the calculator right after the type step. The lane uploads test/screenshots/replays/*.png on pass and fail, so a wait-155 timeout becomes diagnosable from the artifact: display showing 155 means a tree-exposure gap; an empty display means the input never landed. The 20-ref divergence dump cannot show the entry node either way. * fix(replay): assert the Linux type through the computed result Run 32485981780's typed-state artifact settled both open questions with one image: the display shows "55" — the bound typeText keystrokes LAND on Linux CI — while the wait for that value timed out, so the calculator entry does not expose its text to selectors; and the display shows no leading "1", so the resolved click on button 1 missed its target entirely. Both discoveries leave the replay: the click dependency goes (a pre-existing click-coordinate defect is not this unit's evidence chain), and the assertion moves to where the tree can answer — the typed string is now a full calculation ("100+55=") whose `=` creates a history row, and the wait matches the computed 155. No button carries that label and the typed string never contains it, so the wait passes only if the keystrokes executed; deleting the type step turns it red. The typed-state screenshot stays as per-run pixel evidence either way. * fix(linux): guard the no-op --sync mousemove; assert typed digits via value= Two artifact PNGs decided this. Run 32485981780 shows "55" in the entry while the wait for it timed out on a mismatched target; run 32487868346 shows "100+55" — digits and + land, the = keystroke does not, and the retries died in the focus step again because the previous commit removed the pointer-moving click. - moveTo now probes `xdotool getmouselocation --shell` and skips the --sync move when the pointer already sits on the target: a no-op move emits no motion event and hangs until the action timeout, which is what reported every failed replay retry as its first coordinate step. A failed probe never blocks the move. The provider test pins both sequences, including the skip case. - The replay types digits only ("155") and waits on `value=155`: the AT-SPI dumper reads the entry's Text interface into the node's value and the selector engine matches it — the earlier "exposure gap" conclusion came from a pair that never tested the matching value. No button carries 155, so the wait passes only if the keystrokes executed. * fix(replay): keep Linux type at contract tier — GTK4 exposes no entry text Run 32490373693 closed the investigation: the typed-state artifact shows "155" in the calculator entry while the wait for value=155 timed out with no interactive filtering in play and a matcher that does compare node.value. The AT-SPI dumper's get_text_iface() route returns nothing for GTK4 gnome-calculator, so no tree-level assertion on typed text can hold on this lane today. The replay keeps the type step and uploads the typed-entry screenshot every run — live pixel evidence that the migrated typeText path lands keystrokes on real Linux hardware — and closes with the survival assertion. The manifest claim returns to the contract tier with the reason written at the entry, and the count pins follow. The GTK4 exposure defect joins the Linux input-defect chip; fixing PyGObject-vs-GTK4 blind through CI rounds is not a sane loop. The moveTo no-op guard from the previous commit stays: it is why this run finally reported the true failing step on every attempt instead of the step-7 hang.	2026-08-21 17:15:35 +02:00
Michał Pierzchała	46eff36f85	refactor: migrate focus to the request-bound device runtime (#1925) ... * refactor: migrate focus to the request-bound device runtime Wave 5's first unit (#1739, ADR 0019). `focus x y` and `find <q> focus` now reach the device through one admitted, request-bound `focusPoint` operation instead of the `handleFocusCommand` interactor leaf and its dispatch-table arm. - New `FocusRuntimeOperations` contract with local and provider interactor binders, mirroring the screenshot/element-text seam rather than inventing a second way for one operation class to reach its mechanics. - Exact-owner facts replace the capability bucket: apple simulator/device, android emulator/device/unknown, harmonyos emulator/device, linux device, web device, vega none, providers wherever their interactor is reachable. That is the retired bucket's cell table, restated as facts. - `focus` leaves BASE_COMMAND_CAPABILITY_MATRIX and both hand-maintained overlays (HARMONYOS_SUPPORTED_COMMANDS, WEB_INTERACTION_COMMANDS). - R40 is the new parametrized cutover row; `focusPoint` has exactly one owner. - The `x y` positional parse moves to utils and is shared with the still-legacy touch siblings, so a migrated command cannot drift from them. `find` stays legacy: this unit owns its focus leg only, its `type` leg still dispatches, and R35 waits on the Wave 5 `type` unit. * test(focus): cover the owning interactor binders, lower the find ratchet Review follow-ups on #1925. P1: focus-runtime.test.ts bound a fake focusPoint, so deleting the interactor call inside bindLocalFocusInteractor left focus a successful no-op with every test green. Adds packages/contracts/src/focus-runtime.test.ts, which executes both binders and asserts resolver context, positional (x, y) forwarding, the structured missing-provider failure, and that an already-cancelled request never resolves an interactor at all. Two planted mutants confirm it bites: removing `await interactor.focus(input.point.x, input.point.y)` and transposing its two arguments each fail exactly the two forwarding tests, while the daemon-level focus and find suites stay green — which is the gap the reviewer named. Coverage: find.test.ts shrank to 1204 lines when its focus assertion moved off the dispatch mock; the ratchet pin follows it down. * test(focus): add live Linux focus coverage to the desktop replay The Linux `focus` claim rested on the provider scenario at command-contract level. The desktop replay runs on real Linux hardware in the Smoke lane, so it now runs a coordinate focus and re-asserts the session survived it. Coordinate, not selector: the step exists to prove the migrated `focusPoint` path executes on real hardware, so it must not be able to fail on match ambiguity or CI layout drift. Reclassifies focus contract -> live in the Linux coverage manifest and updates the two pinned counts. The manifest gate is two-sided — a live claim must name a command the replay actually invokes — so the claim cannot drift from the file.	2026-08-21 11:34:22 +02:00
Michał Pierzchała	2a9a4ee80c	test: add Linux platform command coverage manifest (#1921) ... * test: add Linux platform command coverage manifest * test: address platform coverage review feedback	2026-08-20 21:48:05 +02:00