Michał Pierzchała
476979bf2b
Validated against a real-world repro (a production React Native app's login screen, simulator build provided privately by the reporter): a full-screen accessibilityViewIsModal overlay leaves the public snapshot with just Application+Window. Two gaps kept recovery off: - The sparse detector counted the Application label (the app's display name) as content and the full-screen root as hittable, so the app name alone defeated recovery. Application/Window labels and root hittability say nothing about tree health and no longer count. - Interactor-level snapshot warnings were dropped by the daemon capture chain (only the runtime/commands layer kept them); they now thread through CaptureSnapshotResult into BackendSnapshotResult. With both fixes that login screen recovers through the public query tier: 16 nodes with every control addressable (fill @ref + read-back verified), and the output carries the recovery warning. Bluesky-class trees still ladder into the private fallback unchanged.
agent-device iOS Runner
This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.
Intent
- Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
- Allow local builds via
xcodebuildand caching for faster subsequent runs. - Support simulator prebuilds where compatible.
Status
Current internal runner for iOS, tvOS, and macOS desktop automation.
Protocol and maintenance references:
- Protocol overview:
RUNNER_PROTOCOL.md - TypeScript client:
../src/platforms/ios/runner-client.ts - Swift wire models:
AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift
UITest Runner File Map
AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.
RunnerTests.swift: shared state/constants,setUp(), andtestCommand()entry flow.RunnerTests+Models.swift: wire protocol models (Command,Response, snapshot payload models).RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.RunnerTests+Interaction.swift: tap/drag/swipe/type/back/home/rotate/app-switcher helpers.RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.RunnerTests+ScreenRecorder.swift: nestedScreenRecorderimplementation.
Snapshot Strategy
iOS snapshots have two explicit capture modes:
- full/raw snapshots use recursive XCTest snapshots for rich hierarchy and diagnostics;
- compact interactive snapshots use a bounded flat XCTest query path for fast agent-facing refs.
Some simulator apps expose accessibility trees that lower-level AX services can inspect but XCTest
cannot serialize reliably. In those cases compact interactive snapshots may return a sparse root
quickly, while full snapshots preserve the XCTest error. See
../docs/adr/0004-ios-snapshot-backend-strategy.md
for the backend boundary and the rationale for a future simulator AX-service backend.
Protocol Notes
- The daemon posts JSON commands to
POST /commandon the runner's local HTTP listener. - The runner responds with a JSON envelope shaped as
{ ok, data?, error? }. - The protocol is internal to
agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.