Michał Pierzchała
83d54614d8
* fix: bound iOS capture stalls and make runner recovery session-preserving (#1105) Runner (Swift): - Coalesce duplicate transport sends of one commandId onto the in-flight execution instead of enqueueing them again behind it (capture pileup). - Fail fast with RUNNER_BUSY while watchdog-abandoned main-thread work is draining; escalate to RUNNER_WEDGED past 120s so the daemon recycles. - Carry the capture-plan deadline into the query-sweep and private-AX ladder tiers so chained recovery cannot stack past the watchdog. - Penalize the tree backend after a slow (>5s) or abandoned capture and lead subsequent regular plans with private-AX for that bundle (sticky, 120s), stamped recovered/budget so the deferral stays observable. Daemon (TS): - Per-request runner recycle budget: at most one invalidate+reboot per request, then fail fast with an actionable, session-preserving hint. - RUNNER_WEDGED joins the runner-fatal invalidation reasons. - Interaction commands (click/fill/longpress/press/type/get/is) preserve the daemon on request timeout like snapshot/wait/find: resetting it destroyed every healthy app session the daemon owned. * fix: suppress AX-broken-screen snapshot issues so the runner survives capture XCTest records 'Failed to get matching snapshot: kAXErrorIllegalArgument' issues for every XCUIApplication query on AX-broken screens; after a few of them the test case tears down the moment the in-flight command completes, killing the long-lived runner after every capture of the screen (the restart loop behind #1105). The capture plan already classifies and recovers from AX failures, so this issue class is noise: swallow exactly it in record(_:); everything else still records and still drives XCTEST_RECORDED_FAILURE. * feat: time-slice the XCTest tree capture on a worker thread The tree snapshot XPC is a single blocking call whose duration moves with live content (4s to minutes on Bluesky profile screens); no in-process budget could bound it on the main thread. Run it on a worker bounded to an 8s slice: on timeout the plan penalizes the tree backend, skips the XCTest-backed tiers while the abandoned XPC drains (they would block behind it inside testmanagerd), and recovers through the private AX backend, which does not use testmanagerd. * tune: lower the tree-backend penalty threshold to 3s The Bluesky profile tree grind measures ~4.5s before kAXErrorIllegalArgument, just under the old 5s threshold, so every capture re-paid the doomed grind (9s each). At 3s the second capture onward defers to private AX (2.4s snapshot, 4.9s press on the live repro). * fix: harden the AX-issue suppression per review - Require the kAXError token: 'Failed to get matching snapshot: Timed out while evaluating UI query.' is a genuinely-hung-query signal and must keep recording (and keep driving XCTEST_RECORDED_FAILURE). Sibling AX server codes (kAXErrorCannotComplete, ...) are deliberately included: any AX-server rejection inside a matching-snapshot fetch is the same capture-plan noise. - State honestly that the override is suite-global and why (tap-triggered queries record the same noise; command outcomes stay honest via their own error paths). - Lock-guarded suppressed-issue counter following the file's existing abandoned-work counter pattern, logged with each suppression. - Unit-test the pure classifier (record(_:) itself is not invoked: the must-record variants would record real failures in the test run).
5.9 KiB
ADR 0004: iOS Snapshot Backend Strategy
Status
Accepted. Amended after iOS snapshot capture was simplified to two public modes: regular interactive snapshots and raw diagnostic snapshots.
The current implementation is owned by RunnerTests+SnapshotCapturePlan.swift. Capture plans
declare their XCTest backend chain, and structured snapshot quality verdicts make degraded or
recovered output observable end to end.
Context
Agent Device exposes iOS UI state through snapshots produced by the long-lived XCTest runner. The runner has two durable snapshot needs:
- agent-facing regular context, where the important contract is the effective user-visible UI, fixed controls such as tab bars, and scroll-hidden hints for content outside visible scroll containers;
- rich diagnostics and selector disambiguation, where a raw recursive XCTest snapshot is useful because it preserves hierarchy, static text, wrappers, scroll containers, and ancestry.
These needs should not share one capture strategy blindly. Recursive XCUIElement.snapshot() is
rich, but some real simulator app trees can make XCTest fail with kAXErrorIllegalArgument while
the same app remains visually usable and can be inspected by lower-level simulator accessibility
services. Bluesky is the current known example: Argent's ax-service can describe the screen, but
XCTest recursive snapshots and typed XCUIElementQuery enumeration can degrade to no useful child
nodes.
This is different from presentation filtering. The daemon's snapshot presentation can hide noisy or inaccessible nodes, but it cannot recover nodes that XCTest never returns. More filters, Maestro-specific heuristics, or retries in the daemon would only make this failure slower and less predictable.
Decision
Keep XCTest as the default iOS automation runner and split iOS snapshot capture into explicit strategies:
- Regular visible strategy: use recursive XCTest snapshots, emit the effective user-visible
tree plus visible ancestors and scroll-hidden hints, and fall back through the capture plan when
XCTest returns sparse output. A node inside a scroll container is user-visible only when it
intersects both the app viewport and the nearest visible scroll container. Offscreen descendants
should be visited to set
hiddenContentAbove/hiddenContentBelow, not emitted as normal visible nodes. This strategy must not use an arbitrary node-count cutoff: fixed controls that are later in traversal order, such as bottom tab bars after long lists, are part of the visible UI contract. - Raw diagnostic strategy: use recursive XCTest snapshots for raw snapshots, diagnostics, and cases that need hierarchy. Raw output is allowed to be noisy and large; if the transport cannot carry the response, fail explicitly instead of silently truncating the tree at a hard node count. If XCTest reports a real AX serialization failure, preserve that error instead of pretending the UI is empty.
- Future simulator AX-service strategy: treat Bluesky-class failures as evidence that XCTest is
not a complete semantic snapshot backend. A robust semantic fix should add a host-side simulator
accessibility backend, similar in role to
idbaccessibility commands or Argent'sax-service, and normalize its output into the sameSnapshotNodemodel. That backend can be simulator-only; physical devices can continue using XCTest unless a supported lower-level API exists.
The daemon should make degraded output observable. If an iOS interactive snapshot contains only the application root or another sparse shape, surface a structured quality verdict and warning so agents know the snapshot is degraded output rather than proof that the screen has no controls.
Regression Notes
PR #639 made XCTest AX serialization failures explicit instead of swallowing them as empty snapshots. That was the correct diagnostic change, but it exposed apps whose accessibility trees XCTest cannot serialize.
Later work moved recovery into the regular visible capture plan so healthy apps keep the fast recursive tree path while degraded simulator app classes can still return bounded, honest output when fallback query tiers are the only available source of visible controls.
Issue #1105 showed a second failure shape on the same app class: instead of failing fast with
kAXErrorIllegalArgument, the recursive tree capture can grind for many seconds on
heavy/animating screens before failing, pushing the chained plan past the runner's main-thread
watchdog and burying the main queue under retries. The plan now carries its umbrella deadline
into the query-sweep and private-AX tiers (later ladder rungs stop when the budget is spent),
and a slow or watchdog-abandoned tree capture penalizes the tree backend for that bundle for a
bounded window: subsequent regular plans lead with the private-AX backend
(effectiveSnapshotCapturePlan), stamped as recovered with a budget reason so the deferral
stays observable. The raw diagnostic plan is exempt — it keeps tree-first error propagation.
Consequences
Regular snapshots remain the right tool for agents and Maestro compatibility because they describe what a user can currently perceive and interact with. Raw snapshots remain the right tool when hierarchy matters. Both may still fail loudly on XCTest-broken trees; that failure is useful because retrying the same recursive capture is unlikely to reveal a different tree.
A future AX-service backend is the correct place to regain Bluesky-class semantic coverage. It should be added as a platform backend with its own lifecycle, protocol, normalization, timing metrics, and fallback rules, not as another special case inside the XCTest runner.
When adding new iOS snapshot behavior, maintainers should first decide which strategy owns it. If a change tries to make regular snapshots fast by dropping visible controls behind a node budget, or tries to make raw snapshots safe by silently truncating, it is probably crossing strategy boundaries.