Consolidate the repeated `var x = default; catchException({ x = expr }); if let m {
NSLog("..._IGNORED_EXCEPTION=%@", m); return default }; return x` shape used around
exception-prone XCUITest queries into one generic helper (RunnerTests+Exceptions.swift):
func safely<T>(_ tag:, _ fallback:, _ block:) -> T
func safely<T>(_ tag:, _ block: () -> T?) -> T? // nil-default convenience
11 uniform sites adopt it: SystemModal (4), Snapshot (2), TextEntry (2), Interaction (3).
Each drops from ~7-12 lines to 1-3. The NSLog format is preserved byte-for-byte via the tag arg
(tag "MODAL_QUERY" -> "AGENT_DEVICE_RUNNER_MODAL_QUERY_IGNORED_EXCEPTION=%@"), so the
silently-logged-and-continued path keeps its searchable format and now has a single place to add
per-tag exception telemetry.
Left inline by design (not the uniform pattern): the silent `_ = catchException` KVC reads
(elementHasFocus, snapshotHasFocus), the throw-on-AX snapshot path, executeOnMainSafely's
retry-driving catch, and the bespoke pressKeyboardReturn fallback / performElementTap branches
whose result depends on the exception message.
Behavior-preserving: same defaults, same log output, same returned values. catchException is
non-escaping, so the inout capture in safeIsActionableCandidate is preserved.
Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED.
Collapse the gesture-command boilerplate in the executeOnMain switch behind two seams:
- performGesture(app, idleTimeout:) -> (timing, outcome): folds measureGesture + the scroll
idle-timeout/quiescence-skip wrap. Parameterized: touch gestures wrap (idleTimeout: true,
default); synthesis gestures (pinch/rotate/transform) pass idleTimeout: false because
RunnerSynthesizedGesture governs its own timing — a distinction that was previously implicit.
- gestureResponse(message:timing:frame:) over a GestureFrame (none/touch/drag): one factory for
the success DataPayload (message + gesture timing + optional touch/drag visualization frame).
The 13 gesture cases (tap x3, tapSeries x2, longPress, drag, dragSeries, swipe, mouseClick, pinch,
rotateGesture, transformGesture) now use the seams; ~120 lines of repeated quartet (measureGesture +
idle-wrap + unsupported check + 8-field DataPayload) collapse. CommandExecution.swift 949 -> 867.
Behavior-preserving: identical outcomes, timing capture, DataPayload fields/messages, and the same
touch-wrapped vs synthesis-unwrapped distinction. No wire change (the factory emits the already
wire-locked DataPayload internally). mouseClick (throws) and swipe (returns an optional frame) keep
their bespoke measure/wrap but route the success payload through gestureResponse.
Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED (no warnings).
Behavior-preserving relocation: move the text-entry concern out of the (formerly 1805-line)
RunnerTests+Interaction.swift — the repo's most-changed file — into a new
RunnerTests+TextEntry.swift. Interaction.swift drops to 1068 lines (-737).
Moved verbatim (no logic changes):
- value types: TextTypingRepairMode, TextEntryTiming, TextEntryResult, TextEntryTarget
- the focus -> type -> verify -> repair pipeline (typeTextReliably + focus orchestration +
readiness polling + dropped-char/repair heuristics)
- clearTextInput and the text-entry leaf helpers (editableTextValue, isPlaceholderValue,
isGenericTextInputLabel, normalizedElementText, moveCaretToEnd, estimatedDeleteCount,
keyboardBecameVisible, keyboardElementExists)
The whole cluster is text-entry-exclusive (its only callers are within the moved code or the
already-internal command entry points), so every symbol keeps its original visibility — no
access widened. Shared helpers used by gestures/snapshot/get-text (isKeyboardVisible,
visibleKeyboardFrame, textInputAt, textInputCandidatesAt, readableText, ...) stay in
Interaction. The new file is auto-included via the project's file-system-synchronized group.
This is the safe, cosmetic half of the "TextEntry engine" architecture candidate. A deeper
extraction behind a real seam/type is intentionally deferred: it carries the #245 revert risk
and wants stronger text-entry e2e coverage first.
Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED. Pure relocation, no behavior change.
* fix(ios): unify multi-touch gestures on two-finger synthesis + hinted unsupported errors
Make RunnerSynthesizedGesture the single iOS multi-touch engine and drop the older
incompatible models:
- rotateGesture now drives the two-finger XCTest synthesis path (dx=dy=0, scale=1,
degrees), mirroring the pinch migration in #634. The native XCUIElement.rotate(withVelocity:)
injected a single synthetic rotation that React Native's rotation recognizer did not read
reliably; synthesis fixes it. velocity is ignored on iOS (kept in the wire contract for
compatibility; rotation direction comes from the sign of degrees).
- pinch is now synthesis on iOS and a clear UNSUPPORTED_OPERATION on tvOS/macOS. The macOS
coordinate double-tap+drag heuristic (performCoordinatePinch) is removed: synthesis is
iOS-only, so macOS multi-touch is reported honestly as unsupported rather than approximated.
- RunnerInteractionOutcome.unsupported now carries an actionable hint, mapped to ErrorPayload.hint
(#639). Every unsupported gesture/tvOS path returns a concise message plus a next-step hint
(existing messages kept verbatim).
Net: on iOS, pinch/rotate/transform all flow through one synthesis primitive. swipe/scroll/pan/
fling remain single-finger drags (correct, unchanged).
Coverage: examples/test-app/replays/gesture-lab.ad exercises pinch + rotate against the gesture
lab and asserts "pinch changed yes" / "rotate changed yes".
* fix(ios): fail-fast macOS pinch + align capability/docs with synthesis-only
Follow-through for removing the macOS coordinate pinch path (the runner now returns
UNSUPPORTED_OPERATION for macOS pinch): reject it at admission instead of round-tripping.
- capabilities.ts: pinch now matches rotate-gesture/transform-gesture (Android + iOS
simulator only); macOS dropped. Removes the now-unused isMacOsOrMobileAppleSimulator helper.
- capabilities.test.ts: pinch expected unsupported on macOS and tvOS.
- website/docs/docs/commands.md: pinch listed for Android + iOS simulators only (removed from
the macOS app-session list); documents that iOS rotate ignores the optional velocity arg
(synthesis uses a fixed duration; direction comes from the sign of degrees).
Addresses PR #645 review HIGH #2 and MEDIUM #3.
* fix(ios): surface a hinted unsupported error for synthesis gestures at admission
Removing macOS pinch from the capability matrix makes macOS pinch (and the already-excluded
rotate-gesture/transform-gesture on macOS/tvOS/physical iOS) fail fast in ensureGenericCommandReady
before reaching the runner. That left the runner's macOS-specific hint unreachable on the daemon
path, so callers only saw the generic "<cmd> is not supported on this device".
Add an optional unsupportedHint to the capability matrix and surface it at admission, so the
synthesis-only gestures fail fast (no runner round-trip) AND return an actionable hint pointing to
where they work (Android + iOS simulator). Applied to pinch / rotate-gesture / transform-gesture.
Addresses PR #645 review (P2: route macOS pinch to the hinted failure).
* refactor(ios): single CommandTraits table for runner command classification
Replace the three hand-maintained switches in RunnerTests+Lifecycle.swift
(isInteractionCommand / isReadOnlyCommand / isRunnerLifecycleCommand) with one
source of truth: CommandType.traits, an exhaustive switch returning a
CommandTraits struct (interaction / readOnly / lifecycle axes), collocated with
CommandType in RunnerTests+Models.swift.
Pure refactor: every command's classification is reproduced verbatim, and the
three predicates become one-line lookups with unchanged signatures, so call
sites are untouched. The exhaustive switch makes it a compile error to add a
CommandType without classifying it, closing the drift that historically let
tapSeries/dragSeries/keyboardReturn fall out of isInteractionCommand.
readOnly is a 3-state enum (.always/.never/.conditional); .conditional preserves
alert's action-dependent read-only behavior, resolved in isReadOnlyCommand.
Classification feeds ADR-0002 session invalidation (the read-only retry that
nulls currentApp/currentBundleId), so behavior is intentionally unchanged.
Adds the "Runner command traits" term to CONTEXT.md.
* docs(ios): note CommandTraits.readOnly .conditional is alert-only (review follow-up)
* fix(ios): classify tapSeries/dragSeries/keyboardReturn as interaction commands (#643)
* fix(ios): classify tapSeries/dragSeries/keyboardReturn as interaction commands
tapSeries and dragSeries are the series forms of tap/drag (already interaction
commands); keyboardReturn is the sibling of keyboardDismiss (already an
interaction command). All three were missing from the historical
isInteractionCommand switch — a drift the new CommandTraits table (#642) makes
visible. Classifying them as interaction commands gives them the foreground-guard
+ stabilization preflight that their single-shot/sibling forms already get.
Behavior change: these three commands now re-activate a backgrounded target to
foreground and pay the stabilization delays before running. Ships separately from
the CommandTraits refactor (#642) and should land after that bakes.
mouseClick left unchanged: macOS-only and the foreground guard interacts with
bespoke macOS activation, so it needs a macOS smoke check first.
* test: cover iOS runner series commands in perf harness
On iOS the runner lowered `pinch` to performCoordinatePinch — a tap() then a
single-finger press(forDuration:thenDragTo:). React Native reads that as a pan,
so the pinch scale never changes (reported in #629: scale stays 1.00).
Route iOS pinch through the existing two-finger XCTest synthesis path
(transformGesture / RunnerSynthesizedGesture) with zero translation and rotation,
so RN's pinch recognizer fires. macOS keeps the coordinate path.
Validated on iPhone 17 Pro against examples/test-app: the gesture-lab.ad oracle
now passes 1/1 (fling/pan/pinch/rotate); it previously failed at the pinch step.
Refs #629
* perf(ios): early-exit text-entry readiness when the keyboard is visible
The XCUITest text-entry focus/readiness loops keyed their fast-exit on
focusedTextInput(), which is intentionally hardcoded to return nil on iOS (focus
predicates are stale there). As a result stabilizeTextInputBeforeTyping always
burned its full focusTimeout (0.4s) and waitForTextEntryReadiness burned its full
readinessTimeout (2.0s) in the normal case where the software keyboard appears —
~2.4s of dead wait before a single keystroke on every type/fill.
The software keyboard becoming visible is the reliable iOS readiness signal, so
both loops now return as soon as isKeyboardVisible() is true. The warmup-first-char
echo check and post-type verify/repair remain as drop safety nets.
Measured on iPhone 17 sim (Settings search field), median type time:
25 chars: 3342ms -> 1379ms (2.4x)
52 chars: 3969ms -> 2190ms
313 chars: 10.3s -> 8.6s (remainder is genuine per-char XCUITest typing)
Reliability unchanged: 64/65 trials exact (incl. a 50-word lorem ipsum, verified
by read-back + screenshot); the lone miss triggered the existing verify/repair.
* fix(ios): don't clear an already-empty text field (fixes fill mis-navigation)
clearTextInput unconditionally ran moveCaretToEnd (an edge-tap computed from the
element frame) + a 24-key delete burst, even when the field was empty. On a field
that repositions on focus — e.g. the Settings search bar jumping bottom->top and
revealing a 'Suggestions' list — that edge-tap used a stale frame and landed on an
adjacent row (Developer), navigating away instead of clearing. fill (replace) into
the search field went to the Developer pane (0/3 correct).
Skip the clear entirely when the field's value is already empty (placeholder
treated as empty): replacing into an empty field is a no-op, and skipping avoids
the stray edge-tap. fill into the Settings search now types correctly and stays
put: 5/5 exact (read-back + screenshot).
* perf(ios): resolve text fields via typed queries, not full-tree enumeration
textInputAt used app.descendants(.any).allElementsBoundByIndex (snapshots EVERY
element) to find the text input at a point. fill drove this repeatedly: once it has
coordinates, resolveTextEntryElement re-runs textInputAt on every verify/repair poll
iteration whenever the focused-field reference goes stale (e.g. the Settings search
bar repositioning bottom->top), so the full-tree enum dominated fill latency.
Query the text-input element types directly (app.textFields/secureTextFields/
searchFields/textViews) instead. Same matches, but XCUITest resolves typed queries
without snapshotting the whole tree. Measured (iPhone 17 sim, warm runner): fill 25
chars ~14.5s -> ~4.5s (3.2x), 6/6 exact. Same primitive #632 killed for get text.
* fix(ios): address review — focus-change gate + secure-field clear
Review P1 (focus race): the isKeyboardVisible early-exit in stabilizeTextInputBeforeTyping
and waitForTextEntryReadiness fired the instant the keyboard was visible — but when it was
ALREADY up from a previous field (back-to-back fills), that is before first-responder moves
to the newly-tapped field, so app.typeText could target the old field. Gate the fast-path on
a keyboard hidden->visible TRANSITION via a shared keyboardBecameVisible(wasVisibleAtEntry:)
helper; when the keyboard was already up, fall back to the settle/timeout (the prior, correct
behavior) instead of the ~2.4s dead wait the fresh case avoids.
Review P1 (F2): clearTextInput used editableTextValue(...) ?? "" and skipped clearing on
empty — but editableTextValue returns nil for secure (and unknown) fields, so secure fields
were NEVER cleared and replace concatenated stale+new. Distinguish nil (clear) from "" (skip).
Device-validated: fresh fill fast-path preserved + exact; a second fill with the keyboard
already up still types into the correct field and replaces (not concatenates).
Long-running AgentDeviceRunnerUITests sessions can inherit Xcode 26 screenRecording/deleteOnSuccess defaults and leave hidden testmanagerd attachments behind when sessions terminate outside a clean XCTest success path.
Pin an explicit AgentDeviceRunnerUITests.xctestplan with screenshots and keepNever attachment lifetimes, then normalize the generated per-session .xctestrun copy before test-without-building. The .xctestrun normalization protects stale cached artifacts and Xcode versions that ignore some test-plan attachment lifetime keys during build-for-testing.
This only affects XCTest's automatic attachments; user-requested agent-device recordings still use the existing record command paths.
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
* fix: improve large text surface discovery and reads
* chore: trim duplicated skill guidance
* fix: tighten large text read follow-ups
* fix: scope snapshot metadata and log read fallbacks
* fix: use snapshots for raw iOS tree traversal
Fixes#147
* refactor: unify snapshot hittable computation across modes
* refactor: account for cousin occlusion in hittable
* refactor: tighten snapshot occlusion heuristic
* fix(ios): use XCTest runner for screenshots on physical devices
`xcrun devicectl device screenshot` was removed in Xcode 26.x, causing
`agent-device screenshot` to fail on physical iOS devices with exit code 64.
This fix routes physical device screenshots through the AgentDeviceRunner
XCTest runner (already used for snapshot/interaction commands) instead:
- Add `screenshot` command to the Swift runner's `CommandType` enum
- Implement the handler using `XCUIScreen.main.screenshot()`, writing a
timestamped PNG to the app's temp directory (returned as `tmp/<file>`)
- If `appBundleId` is provided, activate the target app before capturing
so the screenshot shows the app under test rather than the runner itself
- In `screenshotIos()`, pass `appBundleId` to the runner command then pull
the file back to the host via `xcrun devicectl device copy from` (same
mechanism used by the recording feature)
- Thread `appBundleId` through `Interactor.screenshot()` → `dispatch.ts`
→ `screenshotIos()` so the active session's app is always captured
- Extract `IOS_RUNNER_CONTAINER_BUNDLE_IDS` to `runner-client.ts` so it
can be shared between `apps.ts` and `record-trace.ts`
Fixes: screenshots failing with "Failed to capture iOS screenshot" on
devices using Xcode 26.x / devicectl 506.6+
* fix: clean up iOS screenshot fallback follow-ups
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Michał Pierzchała