mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

Michał Pierzchała c18636315a fix(ios): keyboard-dismiss content settle race (#1542 ) — partial, defect 2 needs a decision (#1559 )

* fix(ios): wait for post-dismiss content settle before the next gesture (#1542)

Dismissing the keyboard can trigger the app's own ScrollView content-offset
correction (e.g. releasing the inset it grew to keep a focused field above
the keyboard). That correction is a separate, unsynchronized animation that
`keyboard.waitForNonExistence` knows nothing about — the keyboard AX element
can disappear well before the app visually settles. The very next command is
frequently a synthesized, AX-free drag (scroll/gesture, kept AX-free so it
still works under #1105-family AX degradation), which has no XCTest
quiescence wait of its own, so it can land mid-animation and net to zero —
the "scroll does nothing" symptom on the Form screen's checkout-form.ad leg.

Add a bounded, AX-free screenshot-stability wait to dismissKeyboard() so the
runner only returns once the screen has actually stopped changing (or a
generous cap elapses). The stopping decision is a pure function
(runnerScreenshotStabilitySettled) covered by unit tests under
AGENT_DEVICE_RUNNER_UNIT_TESTS; the surrounding capture/sleep loop is the
thin, untestable I/O shell around it.

Live-verified on iPhone 17 Pro / iOS 26.2: the scroll now visually lands at
the correct position (confirmed via screen-recording frame correlation)
instead of leaving content at its pre-scroll offset.

Not a full fix for #1542: the checkout-form.ad corpus leg still fails at the
same step, now because the daemon's shared post-gesture snapshot
stabilization (src/daemon/post-gesture-stabilization.ts) can read a
stale-but-internally-consistent AX tree after the AX-free scroll and
mistake "unchanged across polls" for "settled", so the following click's
off-screen guard sees pre-scroll node positions. That is a cross-platform,
cross-command stabilization semantics change and needs a design decision,
not a unilateral fix here — see the PR description.

* ci(ios): execute the screenshot-stability runner tests (#1559 review)

2026-08-03 09:11:05 +02:00

AgentDeviceRunner

fix(ios): keyboard-dismiss content settle race (#1542 ) — partial, defect 2 needs a decision (#1559 )

2026-08-03 09:11:05 +02:00

README.md

refactor: colocate native platform sources under android/, apple/, linux/ (#1273 )

2026-07-15 21:47:38 +02:00

RUNNER_PROTOCOL.md

refactor: colocate native platform sources under android/, apple/, linux/ (#1273 )

2026-07-15 21:47:38 +02:00

README.md

agent-device iOS Runner

This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.

Intent

Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
Allow local builds via xcodebuild and caching for faster subsequent runs.
Support simulator prebuilds where compatible.

Status

Current internal runner for iOS, tvOS, and macOS desktop automation.

Protocol and maintenance references:

Protocol overview: RUNNER_PROTOCOL.md
TypeScript client: ../../src/platforms/apple/core/runner/runner-client.ts
Swift wire models: AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift

UITest Runner File Map

AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.

RunnerTests.swift: shared state/constants, setUp(), and testCommand() entry flow.
RunnerTests+Models.swift: wire protocol models (Command, Response, snapshot payload models).
RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).
RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.
RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.
RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.
RunnerTests+Interaction.swift: tap/drag/swipe/type/home/rotate/app-switcher helpers.
RunnerTests+Navigation.swift: back/navigation-control helpers.
RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.
RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.
RunnerTests+ScreenRecorder.swift: nested ScreenRecorder implementation.

Snapshot Strategy

iOS snapshots have two explicit public capture modes:

full/raw snapshots use recursive XCTest snapshots for rich hierarchy and diagnostics;
interactive snapshots filter the same visible tree down to agent-facing refs.

Some iOS apps expose accessibility trees that lower-level AX services can inspect but XCTest cannot serialize reliably. In those cases interactive snapshots may return a sparse root quickly, while full snapshots preserve the XCTest error. A penalized simulator can recover through private AX; physical devices use a short XCTest probe because no non-XCTest semantic backend is available there. See ../../docs/adr/0004-ios-snapshot-backend-strategy.md for the backend boundary and future simulator AX-service direction.

Protocol Notes

The daemon posts JSON commands to POST /command on the runner's local HTTP listener.
The runner responds with a JSON envelope shaped as { ok, data?, error? }.
The protocol is internal to agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.