mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
main
18 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Michał Pierzchała
|
893ce4b866 |
fix(ci): repair nightly XCTest and conformance lanes (#1989)
* fix(ci): repair nightly XCTest and conformance lanes * fix(ci): harden nightly failure classification * fix(ci): stabilize macOS replay cleanup * fix(ci): close nightly review gaps * fix(ci): classify device claims as infrastructure |
||
|
Michał Pierzchała
|
3bf3ff130a |
fix: Linux input/a11y defects from #1935 (click miss, typed '=', GTK4 text) (#1949)
* diag: instrument Linux CI to gather evidence for #1935 input/a11y defects Temporary — adds a diagnostic step that dumps raw AT-SPI interfaces/actions for gnome-calculator's digit buttons, tests a raw xdotool click at a button's own rect (bypassing our promotion logic), and isolates the typed '=' character in several configurations. Will be removed once the real fixes land. * diag: harden diagnostic step against bash -e and AT-SPI registration races The prior version crashed 7s in: GH Actions runs steps under bash -e, and an unguarded python3 heredoc threw (iterating a dict instead of a list when the app wasn't found yet), aborting the rest of the script silently under continue-on-error. Guards every fallible command, and replaces the fixed 2s sleep with inspect.py's own poll-until-found loop. * diag: test WINDOW coordtype and static Text.get_text call (round 3) Round 2 proved Component.get_extents(SCREEN) returns (0,0) for every non-toplevel widget (real click miss confirmed on-screen), and Text.get_text() throws — a documented PyGObject binding collision with the deprecated 1-arg Accessible.get_text(). This narrows to the two candidate fixes before writing them: does CoordType.WINDOW give usable relative offsets, and does Atspi.Text.get_text(accessible, ...) (static call) return the real typed text. * fix(linux): resolve click-miss, dropped '=', and GTK4 text exposure defects Three defects surfaced by CI on #1935 (Linux Smoke lane), all confirmed live via instrumented CI runs before being fixed here: 1. Click misses its target: Component.get_extents(Atspi.CoordType.SCREEN) returns (0, 0) as the origin for every non-toplevel widget under this GTK4 build — confirmed by a raw click at the computed rect center landing on the window's own header-bar button instead of the intended digit button. CoordType.WINDOW gives correct, distinct per-widget offsets, so get_rect() now computes screen-absolute rects as that offset plus the enclosing top-level frame's own (correct) screen origin, threaded through traverse_node() alongside the existing window-title tracking. Complementary hardening: role "label" is now excluded from `hittable`, since GTK4 wraps every button's caption in a same-rect "label" child, and the shared cross-platform promotion logic in interaction-targeting.ts would otherwise retarget a click from the button onto that non-interactive label. 2. Typed '=' never arrives: a single isolated synthetic keystroke sent right after a focus change is unreliably delivered — confirmed live, both `xdotool type -- "="` and `xdotool key equal` sent alone produced no character at all, while multi-character bursts always landed in full. typeLinux and sendKey now wait a short settle margin before dispatching to xdotool/ydotool, absorbing the race regardless of which action last changed focus. 3. GTK4 apps expose no editable text: accessible.get_text_iface().get_text() throws "Atspi.Accessible.get_text() takes exactly 1 argument (3 given)" — a documented PyGObject binding collision between Text.get_text and the deprecated 1-argument Accessible.get_text, silently swallowed as "no text" by the broad exception handler. get_text_value() now calls the unbound Atspi.Text.get_text(accessible, ...) form, which correctly returns the real content. The Linux smoke replay is restored to exercise all three fixes together (click a resolved digit button, type a full calculation including the '=' keystroke, wait on the computed result through the tree) instead of staying at the weakened, contract-tier assertions the defects had forced. The coverage manifest promotes click and type from command-contract to live accordingly. * fix(linux): drop unproven keyboard-settle and hittable changes per review Addresses thymikee's review on #1949 (both points correct): P1: the keyboard settle (typeLinux/sendKey) was unjustified. The cited diagnostic evidence for a dropped '=' actually shows the opposite — "100+55=" and "5=5" both computed correctly with zero settle, proving '=' was delivered in every multi-character burst tested. Sending '=' alone to an empty entry showing a blank display is normal calculator semantics (nothing to evaluate), not a lost keystroke. The likelier explanation for the original "100+55" screenshot (run 32487868346) is that its attempt-3 hit the already-fixed mousemove --sync hang, not an independent keyboard-dispatch defect. Reverted; no keyboard-dispatch change was needed. P2: the `role_name != "label"` hittable narrowing was extra surface beyond what the click-miss fix required. The corrected AT-SPI coordinates alone fix the observed miss — the button and its same-rect label child resolve to nearly identical centers, so descendant promotion still lands inside the button either way, and the replay can't distinguish which node it actually targeted. Reverted; only the coordinate fix remains. |
||
|
Michał Pierzchała
|
81409f1a7c |
refactor: migrate type to the request-bound device runtime (#1935)
* refactor: migrate type to the request-bound device runtime
Wave 5 unit 2 for #1739 (ADR 0019), find's last blocker. `type "text"` and
`find <q> type "text"` now reach the device through one admitted, request-bound
`typeText` operation instead of the `handleTypeCommand` interactor leaf and its
dispatch-table arm.
- New `TypeTextRuntimeOperations` contract riding the same `Interactor` seam as
focus/screenshot/element-text; the operation returns the interactor's own
closed `TypeTextBackendResult`, so Apple route evidence passes through and
every other owner types blind, exactly as before. The iOS synthesized-type
commit wait (#1676) is Apple-interactor-internal and moves nowhere.
- The interaction backend's `typeText` member exists only when the `type`
handler admitted and bound a runtime — no caller can fall back to legacy
dispatch, so the command keeps exactly one execution path (R41).
- `executeBoundTypeText` reproduces the retired leaf byte-for-byte: leading-ref
rejection with the same hint, space-joined positionals, the 0-10000 delay
bound, and only textEntryRoute surviving from the owner's result. Its parse
pins moved from the dispatch-level tests into the daemon runtime test.
- Exact-owner facts replace the capability bucket (apple sim+device, android
all-but-simulator-row, harmonyos emulator+device, linux device, web device,
vega unavailable, providers wherever their interactor is reachable) and
`type` leaves HARMONYOS_SUPPORTED_COMMANDS / WEB_INTERACTION_COMMANDS.
- The Linux desktop replay types a digit on real hardware; the coverage
manifest promotes `type` contract -> live with the two-sided count pins.
- Android/webdriver facts helpers extracted (androidTouchFact, interactorCell)
to keep inspectFacts under the complexity gate.
`find` stays legacy: both of its direct execution legs now share bound
runtimes, so the atomic R35 cutover is next.
* refactor(type): single-pass daemon routing, shared binder source, owner cell tests
Review follow-ups on #1935.
- The type handler now calls the bound executor directly: the interaction-
runtime hop validated and formatted what executeBoundTypeText validates and
formats again, so it is gone — no boundTypeText backend member, no second
result rebuild. The ADR 0014 frame expiry moves to the handler.
- New contracts/interactor-operation-binding.ts: one local resolver and one
fail-closed provider resolver shared by the screenshot, focus, and type
binders — three private copies retired, provider error text preserved.
- provider-limrun/interaction-operations.ts: the interactor-backed interaction
cells move out of the app-log owner (586 -> 563 lines, below its pre-unit
size); text interaction is composed by that owner, not defined in it.
- Every owner runtime test now pins the focusPoint/typeText fact cells and
bound-operation presence for its exact kinds: apple, android (incl. the
synthetic-simulator refusal), harmonyos, linux, web, vega (refusal + hint),
webdriver (reachability-gated, incl. inactive session), limrun (live +
recovery). The webdriver unsupported-capability row documents that
interaction gates on interactor reachability, not capture declarations.
- The Linux replay assertion is now change-sensitive: type "555" then wait for
a 555 node — no calculator button carries that label, so the wait passes only
if the keystrokes landed in the display; deleting the type step turns it red.
* fix(replay): give the calculator focus before the Linux type assertion
The change-sensitive wait exposed what the review predicted: the typed digits
never landed, because `focus 100 100` clicks the DESKTOP and takes keyboard
focus away from the calculator. The old broad assertion masked exactly this.
The retries then wedged on a latent quirk: attempt-1 leaves the pointer at
(100,100), and the next attempt's `xdotool mousemove --sync` to the same point
waits for a motion event that never comes, so every retry dies at the focus
step with a 10s timeout — which is why the lane reported step 7, not the
failing wait.
New tail: `focus 100 100` (R40 evidence + survival assert), then
`click "label=1"` — a resolved press inside the window that restores keyboard
focus, proves pointer input lands in the app, and moves the pointer off
(100,100) so retries cannot trip the mousemove no-op hang — then `type "55"`
and `wait "label=155 || text=155 || value=155"`. No button is labelled 155, so
the wait passes only if the typed keystrokes reached the display.
* refactor(type): delete the fallow SDK typeText surface, drop dead surface fields
Thermo-nuclear review follow-ups (reviewed at 82fb8c2dc; the three-hop relay
it names was already deleted in
|
||
|
Michał Pierzchała
|
46eff36f85 |
refactor: migrate focus to the request-bound device runtime (#1925)
* refactor: migrate focus to the request-bound device runtime Wave 5's first unit (#1739, ADR 0019). `focus x y` and `find <q> focus` now reach the device through one admitted, request-bound `focusPoint` operation instead of the `handleFocusCommand` interactor leaf and its dispatch-table arm. - New `FocusRuntimeOperations` contract with local and provider interactor binders, mirroring the screenshot/element-text seam rather than inventing a second way for one operation class to reach its mechanics. - Exact-owner facts replace the capability bucket: apple simulator/device, android emulator/device/unknown, harmonyos emulator/device, linux device, web device, vega none, providers wherever their interactor is reachable. That is the retired bucket's cell table, restated as facts. - `focus` leaves BASE_COMMAND_CAPABILITY_MATRIX and both hand-maintained overlays (HARMONYOS_SUPPORTED_COMMANDS, WEB_INTERACTION_COMMANDS). - R40 is the new parametrized cutover row; `focusPoint` has exactly one owner. - The `x y` positional parse moves to utils and is shared with the still-legacy touch siblings, so a migrated command cannot drift from them. `find` stays legacy: this unit owns its focus leg only, its `type` leg still dispatches, and R35 waits on the Wave 5 `type` unit. * test(focus): cover the owning interactor binders, lower the find ratchet Review follow-ups on #1925. P1: focus-runtime.test.ts bound a fake focusPoint, so deleting the interactor call inside bindLocalFocusInteractor left focus a successful no-op with every test green. Adds packages/contracts/src/focus-runtime.test.ts, which executes both binders and asserts resolver context, positional (x, y) forwarding, the structured missing-provider failure, and that an already-cancelled request never resolves an interactor at all. Two planted mutants confirm it bites: removing `await interactor.focus(input.point.x, input.point.y)` and transposing its two arguments each fail exactly the two forwarding tests, while the daemon-level focus and find suites stay green — which is the gap the reviewer named. Coverage: find.test.ts shrank to 1204 lines when its focus assertion moved off the dispatch mock; the ratchet pin follows it down. * test(focus): add live Linux focus coverage to the desktop replay The Linux `focus` claim rested on the provider scenario at command-contract level. The desktop replay runs on real Linux hardware in the Smoke lane, so it now runs a coordinate focus and re-asserts the session survived it. Coordinate, not selector: the step exists to prove the migrated `focusPoint` path executes on real hardware, so it must not be able to fail on match ambiguity or CI layout drift. Reclassifies focus contract -> live in the Linux coverage manifest and updates the two pinned counts. The manifest gate is two-sided — a live claim must name a command the replay actually invokes — so the claim cannot drift from the file. |
||
|
Michał Pierzchała
|
f843dc2df1 |
fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) (#1820)
* fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) `pnpm gate replay-android` failed 4/8 whenever it ran after the full-tier Android E2E (replays-nightly run 32107665052, job 95620294899): 05-app-lifecycle, 06-swipe-gestures and both fixture replays diverged under "A system surface covers the app". The E2E was not the cause. Reproduced on a pixel_7 / API 36 AVD with the same cutout geometry CI's `avdmanager --device pixel_7` produces (status bar 136px, not the 63px of a plain 1080x2400 skin): - `03-scroll-discovery.ad` runs `scroll up 3`. The scroll planner clamps travel to the viewport minus a 5% band, so the touch-down landed at y=120 — inside the 136px status bar — and pulled the notification shade instead of scrolling. On API 36 the app window is edge-to-edge, so the reported viewport starts at y=0 and includes that bar. - The shade then covered every replay until `04`'s `back` closed it. Native readdir order on the runner (03, 05, 06, fixture/02, fixture/01, 04, 01, 02) put four files in that window; the last green run (2026-07-30) had 04 right after 03, so the pull was masked. Fix in the product, not the lane: DEFAULT_EDGE_PADDING_FRACTION 0.05 -> 0.1 in the TS scroll planner and its Swift port. Every real Pixel has a cutout (5.7% of a Pixel 7's height) and an iPhone's Dynamic Island status bar is 6.9%, so any saturated `scroll up` opened the shade / Notification Center for real agents too. Parity vectors updated in both suites plus a Pixel 7 regression vector (1080x2400, amount 3 -> touch-down y=240 > 136). Second contamination the same order exposed once the shade was gone: `fixture/02-selector-routes-covered-diagnosis.ad` is a #1715 reproduction recipe that FAILS BY DESIGN at step 9 (covered-target refusal) and leaves the device in landscape, yet the gate enumerated `test/integration/replays/android` recursively. iOS keeps gate replays in `replays/ios/simulator` and fixture recipes in `replays/ios/fixture`; Android now mirrors that: the six Settings replays move to `replays/android/emulator`, `test:replay:android` points there, and `fixture/` stays E2E-owned (`full:fixture-replays` already runs 01 by path). android.yml and the workflow-evidence fixture follow the path; the replay-compat manifest keeps the historical paths it pins at released tags. Verified live (Pixel 7 geometry, API 36, --retries 0): control run at main head in CI order reproduces exactly CI's 4/8; with the fix, `pnpm gate replay-android` 6/6 in both native and CI order, and `03` leaves Settings on screen (scroll up 3 now touches down at y=240). * test(scroll): drive the TS and Swift scroll-plan parity vectors from one table (#1820 review) The two suites hand-mirrored the same vectors and #1820 had to edit both by hand — the drift class the repo already closes for the tap-point rule via contracts/fixtures/tap-point-policy.json. The scroll vectors (plus both planner constants, pinned behaviourally on a 1000px axis) now live in contracts/fixtures/scroll-gesture.json; scroll-gesture.test.ts and RunnerTests+ScrollGesture.swift iterate it. Verified: vitest 10/10; the four XCTests run on an iOS 26.2 simulator with the unit flag on (Executed 4 tests, 0 failures). Also: test/ci/android-workflow-evidence.json says what it guards. Follow-up for content-safe viewport bounds + discovery order: #1821. |
||
|
Michał Pierzchała
|
e3f3a2488e |
test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) (#1793)
* test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) The nightly Android job has failed on `click id="automation-request-microphone"` since the full tier landed: `settings permission reset microphone` runs `pm revoke`, and revoking a *granted* runtime permission kills the app process, so the round after an accept clicked into an empty launcher surface. Reproduced on the pinned CI image (android-36 google_apis_playstore, Pixel 7): pm revoke leaves pid 4259 alive when the permission is denied and kills it when it is granted, with NexusLauncher resumed afterwards. Fixing that exposed the rest of a scenario that had never executed end to end: the post-revoke readback cold-started on the tabs home instead of Automation lab, the relaunched Automation lab needed its controls revealed, the Form tab does not exist on the Automation root route, that section needs the system IME back, and the IME diagnostic sits above the bottom of the form. The observability scenario then failed the same way (reveal distances tuned for a taller device) plus an event-timeline walk whose page was smaller than the events each page read appends. Validated live against a local Pixel_7_CI emulator (API 36, same profile as the lane): the full tier now runs bootstrap -> inventory -> automation-system -> form-input -> keyboard-ime -> capture-close -> lifecycle-system -> observability-artifacts and stops only in full:fixture-replays. * test(android): repair the drifted fixture replays and pin the catalog canary (#1781 A1) Review follow-up. The nested batch regression now checks a sibling card instead of the notice that owns `dismiss-notice`: resolving a child already proves its parent is present, so the old target could not fail on its own. Confirmed on a Pixel 7 / API 36 emulator that `gesture-lab-card` and `dismiss-notice` are on screen together at the scenario's existing 0.3 reveal (both present at 0.2-0.4; the card is gone by 0.5). Getting a full-tier run to complete then required repairing what the lane had never executed: - `01-navigation-scroll.ad` clicked `label="Catalog, 0 new notifications"`. #1543 made the cart badge conditional, so the live label is `Catalog` — what the iOS twin already used. - The catalog scroll canary lives inside the scrolling content, and Android accessibility snapshots carry on-screen nodes only, so every state except the initial `top` was unobservable: `wait "Catalog scroll: down|bottom|up"` could never pass, whatever the swipe coordinates were. `stickyHeaderIndices` pins that one line, which makes all four states readable at any offset on both platforms rather than tuning the .ad around a canary that scrolls away. - `gesture-lab-android.ad` started its multi-pointer gestures at y=1040, inside the target when the file was last repaired but 90px from its top edge after #1567 moved the card (targets now span y=949-1525). The second pointer landed outside the view, which reads as "the gesture did nothing". Multi-pointer gestures now start at the target centre, and the header comment records the geometry they depend on. Evidence: the lane's own command (`AGENT_DEVICE_ANDROID_E2E_TIER=full` over smoke-android-emulator.test.ts) passes end to end on a Pixel 7 / API 36 AVD with a CI-equivalent fixture APK (cached native + head JS through the same repack the workflow runs): 9/9 scenarios, 153s. |
||
|
Michał Pierzchała
|
338aa2a0d5 |
refactor: route every native selector resolution through the policy interface (#1715)
* refactor: route every native selector resolution through the policy interface #1649 declared the per-caller ambiguity matrix; four native call sites still bypassed it, spreading `selectorResolutionKnobs(row)` into a raw `resolveSelectorChain` instead of naming the row. That left the "one interface" claim aspirational: a caller could restate its contract as engine knobs and nothing would notice. - `is` non-exists, `get text`/`get attrs`, find's read actions, and the covered-selector diagnosis probe now call `resolveSelectorChainWithPolicy` with their existing row. Semantics are byte-identical: the knob-backed branch of that interface forwards to the same engine call the call sites built by hand. - The façade drops `resolveSelectorChain` and `selectorResolutionKnobs`, so no knob-taking resolver is reachable from outside the package and a call site cannot re-acquire the knobs even by accident. `requireUnique`/`disambiguateAmbiguous` are now named in exactly one function, which `resolve-with-policy.ts` and the replay resolver both derive through. - `get` names the two rows it may consume as a type, so pointing it at any other ambiguity contract is a compile error. Tests: selector-read-policy.test.ts pins which row each read command consumes, end to end, on one ambiguous fixture — the only tree the rows disagree on. Each assertion was proven red by re-pointing its caller at a neighbouring row. The knob-consistency check moves into the package beside the now-private helper. Test call sites that used the raw resolver move to `resolveRecordedTarget`, the same knobs and the path that actually replays a recorded chain. Extracting the failure branch drops `resolveSelectorInteractionTarget` below the complexity threshold; its `fallow-ignore` waiver is removed (verified load-bearing before the extraction, unnecessary after). Closes #1630. Structural stages (occlusion, off-screen, promotion, poll budget) stay per-caller pipeline code, tracked in #1656. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HuKzQWn6WQcMYaAZVvJzdD * test: observe which node find's row selected, not just that one existed #1715 review, P2: the find row assertion was only half a pin. `find exists` returns `found: true` for any resolved node, and the `list` call it leaned on goes through listFindMatches — a path that consumes no policy row at all. So repointing findFirstLocatorMatch at `readText` left both assertions green while selection silently moved from the document-order head to the tiebreak winner. Assert through `find get_attrs`, which returns the ref of the node the row actually selected. Both neighbouring rows are now red: `readText` fails '@e3' !== '@e2' (the move the old test missed), `readUnique` fails by refusing the ambiguous screen. `exists` stays as a second, weaker assertion on the same resolution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HuKzQWn6WQcMYaAZVvJzdD * refactor: route is exists through the matrix, collapse the double match pass Follow-up tightening on the same seam. `is exists` reached findSelectorChainMatch directly while the `readAny` row's own doc claimed to serve "`exists` and find's read-only actions" — true of the docs, not of the code, which is the unverifiable-claim shape #1656's review called out. It now names `readAny`, the row it always described. Equivalent by construction: both take the first alternative with any match under requireRect: false, and disclose that alternative's count. That leaves the root façade with no consumer for findSelectorChainMatch, so it goes the way of resolveSelectorChain — dropped from the string-only façade, kept on the published ./ast surface. Its façade-twin type SelectorChainMatch dies with it (fallow caught it). resolveSelectorChainWithPolicy matched twice on the uniqueness path: once via resolveSelectorChain, then again to fill matchedNodes. Hoisting the single list call above the row switch removes that second pass, collapses two duplicated ambiguous literals into one helper, and drops a `?? [resolution.node]` fallback that was unreachable — a resolution implies its alternative matched, so the list is never null there. While hoisting: the resolved arm's matchedNodes can describe a different alternative than resolution.selector, because uniqueness skips an ambiguous alternative to try the next one. Unreachable today (only first-match callers read it, where both come from one list), and left as-is rather than silently changed — but the doc claimed "the alternative it came from", so it now says what is actually true. Tests: is exists gets a caller-level pin on the shared ambiguous fixture — passes with matches: 2 where its fail-closed siblings refuse — proven red by pointing it at readUnique. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HuKzQWn6WQcMYaAZVvJzdD * test: discriminate is exists's row by alternative, guard the façade structurally #1715 review, second regression-validity gap. The `is exists` pin observed only `pass: true` and `matches: 2` on a fixture whose first alternative was merely TIEBREAKABLE — so disambiguation succeeded there and reported the same count first-match would. `readAny`, `readText`, and the pre-migration raw lookup all produced that, and only the readUnique swap I had checked went red. One mutation proven is not the same as the row being pinned. `exists` exposes no node ref, so the row has to be read off WHICH alternative answered. New fixture: alternative one matches two nodes that are genuinely indistinguishable (same depth, same area, both on screen) so the tiebreak declines; alternative two matches exactly one. First-match answers from alternative one; every uniqueness row skips the undecidable alternative and answers from alternative two. Asserting the selector now separates them — readText and readUnique both fail with `id="save-unique"` where `label="Save"` is expected. Restoring the raw lookup stays behaviourally invisible, though: findSelectorChainMatch is equivalent to the readAny row it migrated to, which is precisely why that migration preserved semantics. No fixture assertion can catch that revert, so the guard is structural — the façade's export list must not carry resolveSelectorChain, findSelectorChainMatch, or selectorResolutionKnobs. Follows the packages/maestro index.test.ts absence-assertion precedent. Verified red by re-exporting the lookup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HuKzQWn6WQcMYaAZVvJzdD * fix: cover selector routes in device replays * test: simplify selector replay regression --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
9fc266351f |
fix: stabilize Replay Nightly fixture boundaries (#1610)
* fix: stabilize replay nightly fixture boundaries * test: stabilize exit flush integration coverage * chore: drop the deleted exit-naive fixture from fallow's entry list (#1610 review P3) |
||
|
Michał Pierzchała
|
8a6ddbc11d |
fix(test): repair Android replay fixtures against live device reality (#1538)
* fix(test): repair Android replay fixtures against live device reality Three Android fixture defects from the #1482/#1484 full-tier suite, none of which ever executed in CI (both nightlies since failed on adb infra before the suite ran). All three verified live on a fresh API 36 emulator with a pixel_7-geometry AVD and a Release fixture APK: - 01-navigation-scroll.ad clicked label=Catalog, but the expo-router NativeTabs cart badge leaks '0 new notifications' into the tab's content description even while hidden, and unselected native tabs expose no child text node - exact match can never hit. Target the composed label the device actually exposes (deterministic at fixture start: cart is 0 after --relaunch). The badge does not leak on iOS, so the iOS twin keeps label="Catalog". - checkout-form-android.ad opened by iOS display name 'Agent Device Tester'; Android open resolves packages (the APK label is 'Agentdevicelab'), so APP_NOT_INSTALLED was guaranteed. Use the package id, matching gesture-lab-android.ad. - gesture-lab-android.ad aimed every gesture at y=700, above the gesture card (its targets span y754-1329 on pixel_7 geometry; the home screen gained content above the card since authoring). Re-aim pans inside the exact-two-pointer zone, flings on the image clear of that zone, and pinch/rotate/transform at the card center. Verified: full suite passes 2/2 via the public test command (20 + 32 steps replayed). Refs #1478 * docs(test): pin the Android gesture fixture's validated emulator geometry The re-aimed coordinates are validated on CI's profile (pixel_7 1080x2400 @420); any booted emulator can receive them via test-app:replay:android, so the fixture and README now say which geometry the numbers mean and what a mismatch failure looks like. The checkout twin is selector-driven and unconstrained. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
32b9db2d7a |
test: add Android full emulator coverage (#1484)
* test: add Android full emulator coverage * ci: package Android helpers before nightly coverage * fix: harden Android nightly runtime evidence * fix: expose trace artifacts in MCP schema * style: format Android coverage manifest * fix: address Android coverage review findings * refactor: share live device coverage helpers * fix: restore fixture landmarks in device smokes * refactor: centralize live artifact assertions * fix: normalize fixture canary visibility |
||
|
Michał Pierzchała
|
4e4ecdea0d |
test(ios): expand simulator e2e coverage (#1408)
* test(ios): expand simulator e2e coverage * test(ios): make coverage checks host portable * test(ios): handle deep link confirmation * test(ios): fix deep link prompt selector * ci: stabilize full simulator nightly * test(ios): stabilize permission prompt lifecycle * test(e2e): wait for route-specific landmarks * test(e2e): reset permissions from inactive app * test(e2e): redeliver trusted cold deep links * test(ios): verify orientation native readback * test(ios): stabilize simulator permission coverage * test(ios): wait for tab target after deep link * test(ios): paginate full event timeline * test(ios): simplify event pagination coverage * test(ios): stabilize simulator e2e coverage * test(ios): model simulator recorder lifetime * ci(test-app): cache fixture dependencies * fix(ci): isolate test app cache by node * fix(ios): settle fixture route navigation * test(ci): waive unbenchmarked ios system UI help * fix(ios): tolerate delayed simulator scale lookup * 0.20.1 * test(ci): remove superseded system UI waiver * fix(ios): harden simulator e2e reliability * chore: clarify Apple runner CI steps * fix(ios): wait before fixture home snapshot * fix(ios): require exact catalog navigation * chore(ci): format rebased workflows * fix(ios): retry unobserved fixture navigation * refactor(test): remove iOS e2e workarounds * fix(ci): verify fixture artifact provenance * fix(ci): align fixture artifact fingerprints * fix(ci): use unified Android helper packager * perf(ci): scope fixture build concurrency * chore: format fixture artifact tests * test(ios): update split Apple coverage owner * fix(ios): accept deep-link confirmation alerts |
||
|
Michał Pierzchała
|
287cc18c29 |
fix(gestures): fail pre-removal gesture forms at .ad parse time (#1216) (#1393)
* fix(gestures): fail pre-removal gesture forms at .ad parse time (#1216) #1315 removed the timed forms of `swipe`, `gesture fling`, and `gesture swipe` and `gesture rotate`'s `velocity`, but shipped without the migration guide, the repository sweep, or the parse-time error that issue #1216's own checklist gates a removal on. The sweep finds what that left behind: both `06-swipe-gestures.ad` integration fixtures still carried the 5-argument swipe and would fail at replay, two tests still asserted the removed shapes, and two branches still read the retired positional. Argument arity for every public gesture syntax now lives in one table keyed off the canonical `GESTURE_KINDS`, so a new kind cannot skip it and a form removed from the CLI is removed from `.ad` in the same edit. Both callers read it: the CLI argv parse, and a new `.ad` preflight. A stale script now fails when it is parsed — before the replay executes any device action — naming the line and computing its rewrite, instead of running up to that step and failing as a repairable divergence. The preflight checks arity only: `${VAR}` tokens resolve after planning, and interpolation never splits a token, so the count is decidable while the values are not. Deleting the dead duration read in `readSwipeGeometry` would have left `replay export` emitting no duration, handing Maestro's 400ms default to a gesture the script runs at 100ms, so the export now states `duration: 100`. `.ad` positional gesture parsing is NOT removed. Its only remaining callers are the CLI argv parse and the `.ad` line parse, both the current public syntax rather than a bridge to an older one, so there is nothing to migrate off. ADR 0013 records that and drops the "compatibility" framing that made it read as debt. Both migrated fixtures verified on real devices with the repo's own CLI: iOS simulator 34.9s, Android emulator 45.9s. * fix(gestures): reject removed swipe input at the Node/MCP boundary Review findings on d88c6ed8. P1: `interactionDaemonWriters.swipe` hand-projects five fields, so a JavaScript caller's `durationMs` was dropped before the daemon's `readSwipeInput` could reject it and a default-duration fling ran instead — the exact silent reinterpretation the guide promises does not happen. `gesture` was already safe because its writer runs `readGestureInput` -> `readGesturePayload`, which rejects the removed keys; `swipe` was the one surface with no reader of its own. The rejection now lives in contracts and is shared by the client writer and the daemon handler, so there is one rule and one message. The SDK regression covers all four removed keys and asserts the transport is never reached; reverting the writer call fails it on `swipe durationMs`. P2: the preflight's retired-slot test required a numeric token, so `swipe 197 650 197 300 ${DURATION}` fell back to bare usage text. An unresolved `${VAR}` now counts as the retired slot and is carried into the pan rewrite, while a stray flag or word stays a plain usage error. P2: the removal shipped in 0.20.0, not 0.21 — removal commit |
||
|
Michał Pierzchała
|
aa5b07faa5 |
fix: keep iOS runner hot across app closes (#700)
* fix: keep iOS runner hot across app closes * fix: keep iOS snapshots fast after relaunch * fix: bound compact iOS snapshots on broken AX trees * docs: clarify iOS snapshot backend strategy * test: update Settings replay selectors * fix: stabilize Settings replay selectors * fix: fall back for selector interactions * chore: simplify flat interactive snapshot query |
||
|
Michał Pierzchała
|
59d28e8446 |
refactor: add provider-first device lab tests (#542)
* refactor: add provider-first device lab tests * refactor: tighten device lab provider seams * test: cover provider lab contracts * docs: record device lab harness direction * ci: run device lab integration tests * test: move device lab under integration * test: extract device lab helpers * refactor: centralize apps filter defaults * test: drop lab-covered unit tests * test: fold platform happy paths into device lab * test: reuse device lab helpers * test: move device lab to in-process harness * test: replace session handler cases with device lab * test: harden device lab scenario contracts * docs: define unit test retention policy * test: expand provider device lab coverage * test: harden provider device lab coverage * test: cover manifest install and runner session contracts * chore: remove unused provider cleanup code * test: split android find device lab scenario * test: track provider lab architecture progress * test: clarify provider lab roadmap progress * test: advance provider lab session coverage * test: move menubar click routing to device lab * test: move menubar snapshots to device lab * refactor: centralize screenshot flag plumbing * refactor: colocate screenshot flag metadata * test: cover all public commands in device lab * test: move macos wait success to device lab * test: drop redundant perf and diff units * test: move push payload paths to device lab * test: move network parsing to device lab * test: move log cleanup to device lab * test: move log restart and boot to device lab * test: move ios physical boot to device lab * test: cover perf startup in device lab * test: extract android and ios device lab worlds * test: trim device lab world surface * test: split snapshot capture unit coverage * test: deepen device lab coverage and trim handler units * test: clean up device lab migration scaffolding * test: report device lab public command coverage * refactor: make Apple provider seams semantic * refactor: tighten device inventory and Linux provider seams * refactor: tighten request provider scoping * refactor: add semantic macos host provider * test: broaden device lab find coverage * test: cover workflow flags in device lab * refactor: promote linux input provider seam * test: clarify device lab flag coverage * test: classify snapshot force-full progress * test: enforce device lab progress in ci * test: stabilize device lab ci * test: move packaged metro smoke to integration * test: drop stale provider seam coverage * test: harden provider scope regression coverage * refactor: remove stale platform barrels * refactor: keep linux clipboard and screenshots semantic * refactor: move macos host tools behind provider * fix: honor remote artifact output paths * test: deepen runtime coverage for daemon and runner paths * test: share loopback test helpers * refactor: make daemon runtime importable * fix: honor replay target metadata * chore: tighten final device lab quality gates * test: share device lab setup helpers * test: remove generic apple lab fallback * test: deduplicate device lab helpers * chore: tighten fallow duplication signal * refactor: share apple diagnostic helpers * fix: detect active android ime during fill verification * test: consolidate provider-backed integration suite * ci: fix fallow and iOS smoke setup * chore: consolidate cleanup after ci fixes * test: split vitest unit and integration projects * docs: mention MCP discovery metadata * docs: add agent skills context pointers * fix: close provider recording coverage gaps * fix: restore mcp compatibility smoke * test: cover provider edge regressions * test: consolidate loopback helpers * docs: remove stale provider routing reference * fix: harden final provider review issues * chore: defer mcp cleanup from provider refactor |
||
|
Michał Pierzchała
|
8bcefb754a |
feat: add runtime command boundary (#412)
* feat: add runtime command boundary * refactor: harden runtime command boundary * fix: address runtime boundary review * fix: preserve selector snapshot flags * fix: preserve selector get and screenshot cleanup * fix: harden runtime boundary follow-ups * fix: close runtime parity gaps * test: harden android replay navigation * fix: close screenshot surface edge cases * test: harden packaged runtime API smoke * fix: close runtime review edge cases * test: isolate CLI state dir in unit helpers |
||
|
Michał Pierzchała
|
caf0e834b8 |
feat: add Linux desktop automation support via AT-SPI2 (#356)
* feat: add Linux desktop automation support via AT-SPI2 (Phase 1+2) Add Linux as a first-class platform using AT-SPI2 accessibility framework via node-gtk for accessibility tree snapshots. This mirrors the macOS desktop automation approach using accessibility snapshots. New files: - src/platforms/linux/atspi-bridge.ts: Core AT-SPI2 bridge using node-gtk with lazy loading, recursive tree traversal (max 1500 nodes, depth 12) - src/platforms/linux/role-map.ts: AT-SPI2 role normalization (~100 roles mapped to existing snapshot type conventions) - src/platforms/linux/snapshot.ts: Snapshot entry point with surface, scope, depth, and interactive-only filtering support - src/platforms/linux/devices.ts: Local device discovery for Linux - src/platforms/linux/node-gtk.d.ts: Type declarations for node-gtk Integration: - Extended Platform type with 'linux', backend union with 'linux-atspi' - Wired snapshot into dispatch.ts and snapshot-capture.ts - Added Linux device discovery to dispatch-resolve.ts - Added stub interactor (input actions deferred to Phase 3) - Added 'linux' to CLI --platform flag - node-gtk added as optional dependency (only installs on Linux) https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * refactor: code review cleanup for Linux platform support - Extract SnapshotBackend type alias to replace repeated string union across 5 files (snapshot.ts, snapshot-capture.ts, session-replay-heal.ts, interaction.test.ts) - Remove duplicate scope/interactive/depth filtering from linux/snapshot.ts — let the existing buildSnapshotState pipeline handle it, same as Android - Extract isDesktopBackend() helper in snapshot-capture.ts to consolidate the "skip mobile semantics" pattern for macos-helper and linux-atspi - Collapse 17 repetitive throw statements in Linux interactor stubs into a linuxStub() factory function https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: Linux input synthesis, screenshots, and app lifecycle (Phase 3+4) Add xdotool/ydotool input actions (tap, swipe, scroll, type, fill, right/middle click, long press, double click), screenshot capture via grim/scrot, and app lifecycle management (open, close, back, home). Wire Linux interactors with real implementations and fix device discovery order so Linux doesn't displace Android in auto-selection. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * refactor: consolidate Linux env detection, simplify input actions - Extract linux-env.ts with cached display server + input tool detection so every action avoids repeated `which` lookups - Add moveTo/clickButton/sendKey helpers to eliminate repeated mousemove boilerplate across 5 mouse actions - Make scrollLinux respect amount/pixels options instead of hardcoded scroll count - Have backLinux/homeLinux reuse sendKey instead of duplicating tool detection https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: add Linux CI smoke test with Xvfb and AT-SPI2 Add GitHub Actions workflow that boots a virtual X11 display (Xvfb), installs AT-SPI2 accessibility tooling and xdotool, opens gnome-calculator, takes screenshots, and captures an accessibility snapshot. Screenshots are uploaded as artifacts for visual verification. Also adds 'linux' to replay script metadata platforms and a test:replay:linux script to package.json. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: remove pre-session screenshot from Linux replay test The replay runner requires an active session before any commands can run. Move the screenshot after the open command that creates the session. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: Linux CI — add missing node-gtk build deps and AT-SPI2 env - Add gobject-introspection, libcairo2-dev, build-essential for node-gtk native compilation - Split AT-SPI2 registry start into its own step so it picks up DBUS_SESSION_BUS_ADDRESS from GITHUB_ENV - Set GTK_A11Y=atspi, GTK_MODULES=gail:atk-bridge, NO_AT_BRIDGE=0 to ensure GTK apps expose their accessibility tree on headless CI - Set GSETTINGS_BACKEND=memory to avoid dconf failures - Add node-gtk verification step to catch build failures early https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: explicitly rebuild node-gtk native module in Linux CI pnpm install silently skips failed optional dependency builds and the pnpm cache may not include the native binary. Force a rebuild after install to ensure the node-gtk .node binding is compiled against the system GI/cairo headers. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: use node-pre-gyp directly to build node-gtk from source pnpm rebuild doesn't trigger node-pre-gyp properly for optional deps. Run node-pre-gyp install --fallback-to-build --update-binary directly inside the node-gtk package directory to force compilation when no prebuilt binary exists for the current Node ABI (v127 / Node 22). https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * refactor: replace node-gtk with Python subprocess for AT-SPI2 node-gtk is a native C++ addon that requires compilation against specific Node ABI versions and GObject Introspection headers. This proved unreliable on CI (no prebuilt binaries for Node 22 ABI v127, silent optional dep build failures, pnpm cache staleness). Replace it with a Python helper script (atspi-dump.py) that uses PyGObject — the reference GObject Introspection consumer. python3-gi is trivially installable on any Linux distro with no compilation step. The Node bridge spawns `python3 atspi-dump.py` and parses JSON output. - Remove node-gtk from optionalDependencies - Remove node-gtk.d.ts type stub - Add atspi-dump.py (~200 lines) doing the same tree traversal - Rewrite atspi-bridge.ts to use subprocess instead of in-process GI - Simplify CI workflow: no more native build deps or rebuild steps https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: drop pre-installed packages from Linux CI apt-get python3-gi, gir1.2-atspi-2.0, at-spi2-core, and dbus-x11 are already present on Ubuntu GitHub Actions runners. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: surface support for Linux, unit tests, stronger CI assertions - Allow --surface desktop and --surface frontmost-app on Linux (previously only macOS could use --surface) - Add unit tests for atspi-bridge (9 tests: JSON parsing, role normalization, null coercion, error handling, arg forwarding) - Add unit tests for role-map (3 tests: common roles, case normalization, PascalCase fallback) - Improve .py script path resolution (walk upward instead of hardcoded relative paths) - CI replay test now asserts snapshot contains calculator UI nodes via is-exists https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * docs: add cross-platform snapshot traversal contract Document the shared schema, traversal rules, surface semantics, and normalized role types that all snapshot backends (Swift, Python, Android) must conform to. This serves as the single source of truth when adding or modifying platform backends. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: update lockfile after removing node-gtk optional dependency pnpm-lock.yaml still referenced node-gtk after it was removed from package.json, causing pnpm install --frozen-lockfile to fail in CI. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: address review findings in Linux platform code - atspi-dump.py: use ctx dict for traversal limits instead of globals, fix rect filter (width/height <= 0 should use `or`), add surface validation - input-actions.ts: make sendKey scancodes required to prevent silent no-op on ydotool, fix ydotool longPress/swipe to use click --down/--up - app-lifecycle.ts: use pkill -x (exact match) instead of pkill -f - linux-env.ts: emit diagnostic warning when falling back to xdotool on Wayland https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix(ci): explicitly install all Linux a11y dependencies Ubuntu runners may not have at-spi2-core, python3-gi, gir1.2-atspi-2.0, or dbus-x11 pre-installed. Install them explicitly instead of assuming they exist. Also make the verify step's tree dump non-fatal since no apps are running at that point. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: quote multi-word role value in Linux smoke test selector The selector parser tokenizes on whitespace, so `role=push button` was split into two tokens causing a parse failure. Use single quotes inside the selector: `role='push button'`. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: use valid selector keys in Linux smoke test appName is not a valid selector key. The supported keys are: id, role, text, label, value, visible, hidden, editable, selected, enabled, hittable. Simplified to use label and role only. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: cleanup pass — menubar warning, fix contract doc example - snapshot.ts: emit diagnostic warning when menubar surface is requested on Linux (falls back to desktop silently otherwise) - SNAPSHOT_CONTRACT.md: fix unmapped role example to use a role that isn't actually mapped (was "color chooser" which maps to Dialog) https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: add Python bytecache to gitignore https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: harden Linux platform — capability matrix, CI, error handling P0: Add explicit Linux capability matrix with 3-way platform routing (Apple/Linux/Android) in isCommandSupportedOnDevice. Linux now correctly blocks unsupported commands (clipboard, rotate, scrollIntoView, etc.) at capability level rather than throwing at runtime. Includes tests. P0: Expand Linux CI to run typecheck + unit tests before smoke tests. Add AT-SPI2 registry health probe with fail-fast on missing registry. P1: Harden atspi-dump.py — arg parsing now produces JSON errors on bad int values, and a top-level catch wraps unexpected exceptions in JSON. P1: Add 10s per-action timeout to xdotool/ydotool input commands to prevent indefinite hangs. P1: Tighten smoke test selectors to calculator-specific signals (digit labels) instead of generic role='push button'. P2: Document Linux surface mapping, supported commands, and known limitations in SNAPSHOT_CONTRACT.md. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: apply depth/interactive filtering to Linux snapshots Linux snapshots were bypassing snapshotInteractiveOnly and snapshotDepth filtering that macOS-helper gets via shapeDesktopSurfaceSnapshot. Route Linux through the same function so snapshot -i and --depth flags work. Renamed shapeMacOsSurfaceSnapshot → shapeDesktopSurfaceSnapshot since it's now shared between macOS and Linux desktop backends. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: address review findings — error reporting, Wayland, timeout - app-lifecycle.ts: emit diagnostic on fire-and-forget app launch failure instead of silently swallowing errors - linux-env.ts: make xdotool on Wayland a hard error instead of a broken fallback (xdotool doesn't work on Wayland) - atspi-bridge.ts: increase Python subprocess timeout from 15s to 30s for safety on slow/loaded systems with large a11y trees https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: appName/windowTitle selectors, clipboard, input-action tests Selectors: - Add appname and windowtitle as selector keys for desktop platforms. Both macOS and Linux snapshots already populate these fields — now they're usable in selector expressions (e.g., "label=OK appname=Calc"). Keys are case-insensitive. Clipboard: - Implement readLinuxClipboard/writeLinuxClipboard using xclip/xsel (X11) or wl-copy/wl-paste (Wayland) with descriptive TOOL_MISSING errors. Enable clipboard in Linux capability matrix. 7 unit tests. Input action tests: - Add 18 unit tests covering xdotool and ydotool code paths: press, right/middle click, double click, sendKey, type, scroll, swipe, focus, fill. Tests mock runCmd and verify correct tool + args. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: cache tool detection for screenshot/clipboard, extract get_app_info helper Avoid repeated `which` calls on every screenshot/clipboard operation by caching the resolved tool on first use, matching the input-action pattern. Extract duplicated app_name/pid retrieval in atspi-dump.py into get_app_info. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
497a54deb7 |
test: expand replay coverage and CI reporting with JUnit (#317)
* test: add benchmark .ad scripts for iOS and Android (02-06) Add 10 new replay scripts covering deep navigation, scroll discovery, text input & keyboard, app lifecycle, and swipe gestures for both iOS simulator and Android emulator. Each script combines multiple command types (snapshot, click, wait, is, find, get, scroll, scrollintoview, swipe, back, home, keyboard, screenshot, appstate) to provide comprehensive baseline coverage for version-over-version comparison. https://claude.ai/code/session_01VkHLzVc7NULQFaWHnhuBCm * test: move replay benchmarks out of CI suite * test: trim replay benchmark script comments * test: relaunch replay app opens * fix: stop focusing iOS simulator for screenshots * fix: open Simulator app on cold boot * test: trim iOS cold boot coverage * test: move replay benchmarks into integration suites * ci: align Android emulator with local setup * ci: raise iOS replay daemon timeout * test: add replay junit reports * ci: use supported Android emulator profile * fix: stabilize replay cleanup and Android CI flows * test: trim flaky replay coverage * fix: exclude replay tests from daemon request timeout * test: relax android scroll discovery replay * refactor: rename ios simulator app opener * ci: remove obsolete android replay timeout override * test: remove duplicate android replay opens * test: relax ios swipe replay assertion * ci: move full replay suites to nightly * ci: extract shared replay workflow setup * ci: fail nightly replay jobs on test errors --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
c393ffe612 |
feat: add light test runner for .ad scripts (#266)
* feat: add metadata-aware ad test runner * test: migrate platform integrations to replay suites * fix: stop macos runner during close * fix: harden ad test runner * fix: tighten ad test runner retries and reporting * refactor: tighten replay test suite runtime * test: cover replay suite helpers * docs: trim replay test guidance in skill * refactor: extract replay test cli helpers |