mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

History

Michał Pierzchała c78bfd1e62 fix: guide agents past iOS keyboard dismissal and get-text guesses (#1173 )

* fix: guide agents past iOS keyboard dismissal and get-text guesses

Tonight's benchmark leaderboard showed two recurring agent-UX misses:
keyboard dismiss failing after fill (5x, now the #1 failed command) and
get-text guessed as a command name (1x).

- iOS keyboardDismiss now returns a hint explaining the on-screen keyboard
  does not block agent-device interactions, so agents should press the next
  target directly instead of retrying dismiss, and use keyboard enter only
  when submission is actually wanted.
- help manual-qa and help workflow recovery text no longer teach the false
  "dismiss to unblock the target" pattern.
- The command-suggestion curated map now maps get-text/gettext/get_text to
  the real get text command shape; the generic edit-distance fallback did
  not produce any suggestion for these guesses.

* fix: soften keyboard guidance for genuinely covered targets

Review on #1173 flagged the unconditional "does not block" claim as false
for targets visually covered by the keyboard: direct-selector press
resolution is isHittable-gated and falls through to ELEMENT_NOT_FOUND, the
tree path allows non-hittable taps with only a no-visible-effect hint, and
bottom-pinned-button-under-keyboard (#291/#469/#957) is a real case where
dismissal was the remedy.

All three strings (runner hint, help manual-qa, help workflow) now say the
keyboard USUALLY does not block presses and name concrete fallbacks: scroll
the target into view, or keyboard enter when submission is wanted.

2026-07-09 21:03:55 +02:00

AgentDeviceRunner

fix: guide agents past iOS keyboard dismissal and get-text guesses (#1173 )

2026-07-09 21:03:55 +02:00

README.md

fix: harden iOS XCTest recovery paths (#1158 )

2026-07-08 21:11:28 +02:00

RUNNER_PROTOCOL.md

refactor: rename ios-runner -> apple-runner (#981 ) (#996 )

2026-07-01 14:24:48 +02:00

README.md

agent-device iOS Runner

This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.

Intent

Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
Allow local builds via xcodebuild and caching for faster subsequent runs.
Support simulator prebuilds where compatible.

Status

Current internal runner for iOS, tvOS, and macOS desktop automation.

Protocol and maintenance references:

Protocol overview: RUNNER_PROTOCOL.md
TypeScript client: ../src/platforms/apple/core/runner/runner-client.ts
Swift wire models: AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift

UITest Runner File Map

AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.

RunnerTests.swift: shared state/constants, setUp(), and testCommand() entry flow.
RunnerTests+Models.swift: wire protocol models (Command, Response, snapshot payload models).
RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).
RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.
RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.
RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.
RunnerTests+Interaction.swift: tap/drag/swipe/type/home/rotate/app-switcher helpers.
RunnerTests+Navigation.swift: back/navigation-control helpers.
RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.
RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.
RunnerTests+ScreenRecorder.swift: nested ScreenRecorder implementation.

Snapshot Strategy

iOS snapshots have two explicit public capture modes:

full/raw snapshots use recursive XCTest snapshots for rich hierarchy and diagnostics;
interactive snapshots filter the same visible tree down to agent-facing refs.

Some iOS apps expose accessibility trees that lower-level AX services can inspect but XCTest cannot serialize reliably. In those cases interactive snapshots may return a sparse root quickly, while full snapshots preserve the XCTest error. A penalized simulator can recover through private AX; physical devices use a short XCTest probe because no non-XCTest semantic backend is available there. See ../docs/adr/0004-ios-snapshot-backend-strategy.md for the backend boundary and future simulator AX-service direction.

Protocol Notes

The daemon posts JSON commands to POST /command on the runner's local HTTP listener.
The runner responds with a JSON envelope shaped as { ok, data?, error? }.
The protocol is internal to agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.