mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

Michał Pierzchała 3785a17554 fix(ios): unify multi-touch gestures on two-finger synthesis + hinted unsupported errors (#645 )

* fix(ios): unify multi-touch gestures on two-finger synthesis + hinted unsupported errors

Make RunnerSynthesizedGesture the single iOS multi-touch engine and drop the older
incompatible models:

- rotateGesture now drives the two-finger XCTest synthesis path (dx=dy=0, scale=1,
degrees), mirroring the pinch migration in #634. The native XCUIElement.rotate(withVelocity:)
injected a single synthetic rotation that React Native's rotation recognizer did not read
reliably; synthesis fixes it. velocity is ignored on iOS (kept in the wire contract for
compatibility; rotation direction comes from the sign of degrees).
- pinch is now synthesis on iOS and a clear UNSUPPORTED_OPERATION on tvOS/macOS. The macOS
coordinate double-tap+drag heuristic (performCoordinatePinch) is removed: synthesis is
iOS-only, so macOS multi-touch is reported honestly as unsupported rather than approximated.
- RunnerInteractionOutcome.unsupported now carries an actionable hint, mapped to ErrorPayload.hint
(#639). Every unsupported gesture/tvOS path returns a concise message plus a next-step hint
(existing messages kept verbatim).

Net: on iOS, pinch/rotate/transform all flow through one synthesis primitive. swipe/scroll/pan/
fling remain single-finger drags (correct, unchanged).

Coverage: examples/test-app/replays/gesture-lab.ad exercises pinch + rotate against the gesture
lab and asserts "pinch changed yes" / "rotate changed yes".

* fix(ios): fail-fast macOS pinch + align capability/docs with synthesis-only

Follow-through for removing the macOS coordinate pinch path (the runner now returns
UNSUPPORTED_OPERATION for macOS pinch): reject it at admission instead of round-tripping.

- capabilities.ts: pinch now matches rotate-gesture/transform-gesture (Android + iOS
simulator only); macOS dropped. Removes the now-unused isMacOsOrMobileAppleSimulator helper.
- capabilities.test.ts: pinch expected unsupported on macOS and tvOS.
- website/docs/docs/commands.md: pinch listed for Android + iOS simulators only (removed from
the macOS app-session list); documents that iOS rotate ignores the optional velocity arg
(synthesis uses a fixed duration; direction comes from the sign of degrees).

Addresses PR #645 review HIGH #2 and MEDIUM #3.

* fix(ios): surface a hinted unsupported error for synthesis gestures at admission

Removing macOS pinch from the capability matrix makes macOS pinch (and the already-excluded
rotate-gesture/transform-gesture on macOS/tvOS/physical iOS) fail fast in ensureGenericCommandReady
before reaching the runner. That left the runner's macOS-specific hint unreachable on the daemon
path, so callers only saw the generic "<cmd> is not supported on this device".

Add an optional unsupportedHint to the capability matrix and surface it at admission, so the
synthesis-only gestures fail fast (no runner round-trip) AND return an actionable hint pointing to
where they work (Android + iOS simulator). Applied to pinch / rotate-gesture / transform-gesture.

Addresses PR #645 review (P2: route macOS pinch to the hinted failure).

2026-06-01 19:30:36 +02:00

AgentDeviceRunner

fix(ios): unify multi-touch gestures on two-finger synthesis + hinted unsupported errors (#645 )

2026-06-01 19:30:36 +02:00

README.md

feat: add rotate command for iOS and Android (#344 )

2026-04-01 20:34:56 +02:00

RUNNER_PROTOCOL.md

feat: add recording quality flag (#409 )

2026-04-15 13:18:57 +02:00

README.md

agent-device iOS Runner

This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.

Intent

Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
Allow local builds via xcodebuild and caching for faster subsequent runs.
Support simulator prebuilds where compatible.

Status

Current internal runner for iOS, tvOS, and macOS desktop automation.

Protocol and maintenance references:

Protocol overview: RUNNER_PROTOCOL.md
TypeScript client: ../src/platforms/ios/runner-client.ts
Swift wire models: AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift

UITest Runner File Map

AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.

RunnerTests.swift: shared state/constants, setUp(), and testCommand() entry flow.
RunnerTests+Models.swift: wire protocol models (Command, Response, snapshot payload models).
RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).
RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.
RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.
RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.
RunnerTests+Interaction.swift: tap/drag/swipe/type/back/home/rotate/app-switcher helpers.
RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.
RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.
RunnerTests+ScreenRecorder.swift: nested ScreenRecorder implementation.

Protocol Notes

The daemon posts JSON commands to POST /command on the runner's local HTTP listener.
The runner responds with a JSON envelope shaped as { ok, data?, error? }.
The protocol is internal to agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.