* feat(recording): make iOS export quality configurable
Wire the existing recording-export-quality enum through the record command
down to the Swift export preset. Adds a `--export-quality <medium|high>`
option for iOS recordings that controls the AVAssetExportSession preset used
when a recording is re-encoded.
`medium` stays the default and selects AVAssetExportPresetMediumQuality, which
preserves the fast simulator-friendly export. `high` opts into
AVAssetExportPresetHighestQuality for evidence-grade output. This is separate
from the existing integer `--quality <5-10>` capture flag that scales render
resolution.
Closes#568
* fix(recording): apply export quality to touch-overlay export path
The --export-quality flag was only wired into the resize export path. The
touch-overlay re-encode (finalizeRecordingOverlay -> overlayRecordingTouches ->
recording-overlay.swift) ignored it and always picked AVAssetExportPresetMediumQuality,
so record stop with --export-quality high had no effect when the stop path
re-encodes only to burn in touch overlays.
Thread the recording's exportQuality through finalizeRecordingOverlay and
overlayRecordingTouches, pass it as --export-quality to recording-overlay.swift,
and resolve the preset there via the same exportPresetName() helper used by
recording-resize.swift. Medium stays the default when the arg is absent, so
behavior is unchanged for callers that do not set it.
* feat: align recording quality and size flags
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
* refactor(ios): snapshot capture plans with a structured quality verdict
Implements ADR 0004's explicit-strategies decision as architecture
(candidates 1+2 of the snapshot pipeline review):
- Snapshot backend seam: three adapters (recursive tree, query sweep,
private AX) behind one captureWithBackend dispatch. Each strategy
declares its chain as data (regular: tree→queries→private-ax,
compact: queries→private-ax, raw: tree→private-ax) and one plan
runner walks it under a 20s umbrella budget so chained recovery can
never stack past the 30s main-thread watchdog. Terminal policy is
per-plan: raw rethrows AX failures (diagnostics preserve errors),
interactive fails closed with runnerFatal invalidation.
- Single quality classifier: one sparsePayloadReason predicate (with
reason codes), one collapsed-leaf detector, replacing the three
divergent sparse detectors (Swift structural, daemon count==1, CLI
count<=3) that each patched a different failure shape.
- Structured snapshot quality verdict on the wire (state, backend,
reason, reasonCode, effectiveDepth, collapsedLeafIndexes): the daemon
and CLI render warnings from it instead of re-deriving degradation
from node shapes; budget starvation is no longer blamed on the app's
accessibility. Legacy runner messages and daemon-side detectors stay
behind a verdict-absent gate for mixed-version compat.
- The verdict surfaces in --json (snapshotQuality) for agents; the
generic sparse CLI hint is suppressed when a verdict explains it.
Threading the verdict exposed two more hand-copy field drops
(captureInteractionOutcomeAwareSnapshot, serializeSnapshotResult,
client response mapping) - now carried alongside warnings everywhere.
Verified live: Settings healthy (tree, no warnings), Settings compact
under load (recovered/private-ax/budget), production login (sparse
best-effort with honest warning), collapse fixture (healthy +
collapsedLeafIndexes -> @ref warning), Bluesky Home (recovered/
private-ax, 24 nodes in 2s). Full unit suite 2327 passed, fallow clean,
runner builds.
* fix(ios): correct recovered-snapshot viewport and private-AX scope semantics
Review follow-ups on the capture-plan refactor:
- The query-sweep synthetic root doubles as the daemon's viewport
(find.ts prefers on-screen matches inside nodes[0].rect), but it was
built from candidate bounds, so off-screen controls below the screen
could inflate it and win duplicate-label resolution. The root now
uses the real finite viewport, falling back to candidate bounds only
when viewport capture failed.
- The private-AX backend applied --scope as a per-node text filter,
hiding the matched container's children — diverging from regular
snapshot scope semantics and contradicting the depth-cap hint that
recommends scoped re-runs. Scope now selects the matched subtree:
descendants inherit the match and only the normal option filters
apply to them (in-bundle test covers a non-matching descendant).
Verified live on Bluesky Home: scope homeScreen returns the 52-node
subtree including non-matching descendants; compact root rect equals
the screen (0,0,402,874).
* fix(ios): fail closed on interactive AX failure, stamp fatal verdict, validate parser
Three review findings on the capture-plan terminal path:
- P1: the fail-closed guard required `best == nil`, but the query-sweep
tier always returns a synthetic-root sparse payload that sets `best` —
so an interactive recursive-tree AX serialization failure that no
backend recovered returned a sparse snapshot instead of invalidating
the cached target. Reaching the terminal already means no backend
produced a usable tree, so the sparse `best` must not suppress the
fail-closed path. Extracted the decision into a pure, unit-tested
`resolveSnapshotPlanTerminal` (closes the terminal-ordering testability
gap the architecture review flagged).
- P2: `snapshotAccessibilityUnavailable` returned a payload with no
`snapshotQuality`, leaving one planned sparse result on the
legacy-message path. It now carries a sparse/ax-rejected verdict like
every other planned snapshot, so downstream sparse handling keys off
the verdict.
- P2: `readSnapshotQualityVerdict` cast any string state/backend into the
union, so a malformed object suppressed the legacy node-shape
detectors. State and backend are now validated against their unions
(unknown → verdict-absent → legacy detectors run); an unknown
reasonCode is dropped rather than rejecting the whole verdict, so a
forward-version runner still yields a usable verdict.
Unit-covered: Swift resolveSnapshotPlanTerminal matrix + fatal-verdict
assertion; TS parser accept/reject/forward-compat. Full suite 249 files
/ 2449 tests, fallow, lint, runner build green.
* fix: add iOS private AX snapshot fallback
* fix: add public iOS snapshot query recovery
* fix(ios): make private AX snapshot fallback recover deep React Native trees
Four fixes that turn the #758 private AX fallback from
works-on-one-tree-shape into reliable on Bluesky Home:
- Depth ladder: the AX server rejects bulk snapshot requests outright
(kAXErrorIllegalArgument) once requested depth crosses a
tree-size-dependent limit that moves with live content. Retry at
56/40/24/12 instead of giving up after one attempt at 64.
- Real attribute identifiers: the server silently ignored the raw
keypath strings the bridge passed, so every node came back with a
zero frame (breaking ref taps and the interactive/compact filters,
which is why 'snapshot -i -c' stayed sparse). Map keypaths through
XCElementSnapshot.axAttributesForElementSnapshotKeyPaths (it returns
an NSSet) and drop the mapper's expensive extras (automation type,
window display id, base type) that pushed deep requests past the 30s
main-thread watchdog.
- Viewport from the private root frame when the public windows query
degrades to an infinite viewport, so off-screen drawer content stops
passing the visibility filter.
- Runner source fingerprint now includes .m/.h, so bridge edits stop
reusing stale cached runner builds.
Also hardens the bridge per review: UInt(exactly:) for untrusted
element types, pid_t-sized objc_msgSend for process id matching, and
objCType-checked NSValue frame decoding.
* fix(ios): recover deadline-truncated near-empty compact snapshots
The all-structural sparse detector misses the common large-RN-tree case
where the typed-query sweep resolves one or two stray controls before
its 1s deadline: the payload has 'content', so recovery never fires,
yet 2 nodes is useless in practice. Treat deadline-truncated payloads
with <= 8 nodes as needing recovery, and only replace the original
payload when the recovered tree actually carries more nodes. Completed
sweeps on legitimately minimal screens stay untouched (not truncated).
* chore: fix CI for the AX snapshot fallback branch
- Sync the setup metadata script's fingerprint extension list with the
runtime (.m/.h were added for the ObjC bridge), fixing the cache
metadata parity test.
- Reduce find.ts complexity flagged by fallow: hoist the node fetcher
into createFindNodeFetcher with a recoverSparseInteractiveSnapshot
helper, split match disambiguation and resolution scoring into
narrowMultipleMatches/resolvedTouchScore, extract rectsMatch.
* feat(ios): make accessibility fallbacks and collapsed containers visible in snapshot output
Two transparency gaps from #701's 'no silent fallback' requirement:
- Runner-attached snapshot messages now surface as snapshot warnings
(readAppleSnapshotResult previously dropped them), so every recovery
through the fallback accessibility backend or query tier is announced,
states what it usually means (the app publishes an unhealthy
accessibility tree - fixing the app is the real cure), and points to
screenshot as visual truth.
- A leaf whose label merges many comma-joined segments is flagged as a
collapsed accessible container: the app marks a container accessible,
hiding every descendant from assistive tech and automation alike.
Nothing can be recovered below it (VoiceOver sees the same merged
element), so the warning names the node, estimates the merged label
count, and gives the app-side fix plus the screenshot/coordinate-tap
workaround.
Validated live on the lab stress fixture (adlab://stress?accessible=1):
the 6-node tree now carries '@e5 [Other] merges ~126 labels...'.
* fix(ios): detect sparse trees with labeled roots and surface warnings through the daemon
Validated against a real-world repro (a production React Native app's
login screen, simulator build provided privately by the reporter): a
full-screen accessibilityViewIsModal overlay leaves the public snapshot
with just Application+Window. Two gaps kept recovery off:
- The sparse detector counted the Application label (the app's display
name) as content and the full-screen root as hittable, so the app
name alone defeated recovery. Application/Window labels and root
hittability say nothing about tree health and no longer count.
- Interactor-level snapshot warnings were dropped by the daemon capture
chain (only the runtime/commands layer kept them); they now thread
through CaptureSnapshotResult into BackendSnapshotResult.
With both fixes that login screen recovers through the public query
tier: 16 nodes with every control addressable (fill @ref + read-back
verified), and the output carries the recovery warning. Bluesky-class
trees still ladder into the private fallback unchanged.
Every ok runner response now carries a transport-stamped currentUptimeMs
captured just before the HTTP write. Simulator recording start anchors
gesture overlay timing from the warm snapshot response it already makes,
skipping the standalone uptime request. The standalone uptime path stays
as fallback for older runner builds, and journal-stored responses remain
unstamped so recovered results never pair a stale uptime with a late
receipt time.
Closes#670
* refactor(ios): consolidate series batching onto the sequence runner command
Closes#767
Routes every Apple multi-press variant (plain, double-tap, hold, jitter)
and swipe series through budget-chunked sequence requests, retiring the
daemon-side tapSeries and dragSeries senders:
- Add a doubleTap step kind to the sequence allowlist on both ends,
mirroring the retired tapSeries doubleTapAt branch.
- The single doubleTap interactor sends a one-step sequence and parses
the result, surfacing step failures as errors.
- Swipe series unroll ping-pong daemon-side into per-step endpoints;
the runner's coordinate-drag path ignores durationMs exactly as the
daemon-sent (non-synthesized) dragSeries did.
- Extract runIosSequenceChunks so press and swipe share the chunking,
aggregation, and global step-index rebasing.
- Keep tapSeries/dragSeries runner handlers for wire compatibility with
older daemons, annotated like interactionFrame; remove both from the
preflight-skip allowlist (daemon never sends them) and update ADR
0005 / protocol-optimizations docs.
This also closes the latent watchdog exposure where press --count N
--interval-ms M routed to tapSeries and executed all pauses inside one
30s-watchdog main-thread block with no chunking.
Behavior note: plain tap series now use the synthesized HID tap path on
iOS non-tv (with runner-side tapAt fallback), matching the individual
tap command instead of the retired tapSeries' XCUICoordinate taps.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
* refactor(ios): drop dead series wire surface from the daemon
- Remove chunkRunnerSequenceSteps: superseded by the budget-aware
chunker; no production callers remained.
- Remove tapSeries/dragSeries from the RunnerCommand union along with
their orphaned fields (count, intervalMs, doubleTap, pauseMs,
pattern) and protocol fixtures: this type is the send surface of the
current daemon, which no longer sends either command. The Swift
runner keeps serving both for wire compatibility with older daemons.
- Retarget the ready-mutation preflight test from tapSeries to
sequence.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
* refactor(ios): remove retired series and frame wire commands entirely
Drops the runner-side wire compatibility for tapSeries, dragSeries, and
interactionFrame now that no daemon path sends them (series fuse into
sequence since this branch; interactionFrame was fused into scroll in
#760):
- Swift: delete the three handler cases, performDragSeries, runSeries
(no remaining callers), the CommandType enum cases, journal-retention
and traits entries, and the Command fields (count, intervalMs,
doubleTap, pauseMs, pattern) that existed only for them. The
never-sent synthesized dragSeries branch goes with it.
- TS: drop interactionFrame from the RunnerCommand union and
isReadOnlyRunnerCommand, and its protocol fixture.
- Update stale perf scenario labels referencing the retired commands.
Verified dead before removal: no dynamic command construction anywhere
(runner-command-recovery only echoes in-flight command ids), no
raw-string references in Swift, no docs references. Helpers shared with
live paths (synthesizedDragAt, doubleTapAt, keyboardAvoidingDragPoints,
sleepFor) all retain callers.
Compat: an old daemon paired with a runner built from these sources
gets a CommandType decode rejection; the source-fingerprint check
rebuilds a matching runner on the next session.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
---------
Co-authored-by: Claude <noreply@anthropic.com>
* perf(ios): add lifecycle-safe runner sequence command for hot press series
Adds a narrow 'sequence' runner command that batches an explicit
allowlist of coordinate steps (tap, longPress, drag) into one
lifecycle-tracked request with stop-on-first-failure and small bounded
per-step results. iOS press series with hold/jitter now issue one
sequence request per ~20-step chunk (also budgeted to stay under the
runner's 30s main-thread watchdog) instead of one request per press.
Sequence responses are journaled and retained, so lost-response recovery
returns observed results without replaying the gesture sequence.
Closes#669
* fix: perform every press in direct press series
runDirectPressSeries guarded the awaited interaction itself with ??=,
so presses 2..N were silently skipped once the first result was kept
(affects Android series and doubleTap series; introduced in #512).
The kept-first-result shape is preserved.
* chore: unexport internal sequence chunk budget constant
* perf(ios): make sequence eligible for readiness preflight skip
Rebased onto main with #763 (healthy-mutation preflight skip) and #760
(fused scroll). Per the merge-order note, add 'sequence' to
PREFLIGHT_SKIP_ELIGIBLE_RUNNER_COMMANDS so a successful sequence earns
the next hot-command skip instead of always taking the
conservative_command path. Extend the per-family skip tests and the
allowlist enumeration in ADR 0005 and the protocol-optimizations doc.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
---------
Co-authored-by: Claude <noreply@anthropic.com>
* perf(ios): fuse scroll frame resolution and drag into one runner command
Non-tvOS scroll now sends a single mutating 'scroll' runner command. The
Swift runner resolves the interaction frame and executes the same
non-synthesized drag path, eliminating the separate read-only
interactionFrame request per scroll. The command is lifecycle-journaled
with retained response JSON so lost-response recovery returns the result
without replaying the gesture.
Closes#668
* perf(ios): make fused scroll eligible for readiness preflight skip
#763 landed the healthy-mutation preflight skip with a note that the
fused scroll command should join the allowlist once it exists. Add
'scroll' to PREFLIGHT_SKIP_ELIGIBLE_RUNNER_COMMANDS, drop the
now-resolved code note, extend the per-family skip tests, and update
the allowlist enumeration in ADR 0005 and the protocol-optimizations
doc.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
* test: complete scroll plan parity vector mirror
Address review on the cross-language parity vectors:
- mirror the Swift pixels-plan vector (down, 120px @ 300x600) in the
vitest suite so every vector exists in both languages
- add amount > 1 clamp and tiny-frame (2x2) vectors to both suites;
the tiny frame engages every max(1, ...) floor and the .5 rounding
cases where JS half-up and Swift half-away-from-zero must agree
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
---------
Co-authored-by: Claude <noreply@anthropic.com>
Consolidate the repeated `var x = default; catchException({ x = expr }); if let m {
NSLog("..._IGNORED_EXCEPTION=%@", m); return default }; return x` shape used around
exception-prone XCUITest queries into one generic helper (RunnerTests+Exceptions.swift):
func safely<T>(_ tag:, _ fallback:, _ block:) -> T
func safely<T>(_ tag:, _ block: () -> T?) -> T? // nil-default convenience
11 uniform sites adopt it: SystemModal (4), Snapshot (2), TextEntry (2), Interaction (3).
Each drops from ~7-12 lines to 1-3. The NSLog format is preserved byte-for-byte via the tag arg
(tag "MODAL_QUERY" -> "AGENT_DEVICE_RUNNER_MODAL_QUERY_IGNORED_EXCEPTION=%@"), so the
silently-logged-and-continued path keeps its searchable format and now has a single place to add
per-tag exception telemetry.
Left inline by design (not the uniform pattern): the silent `_ = catchException` KVC reads
(elementHasFocus, snapshotHasFocus), the throw-on-AX snapshot path, executeOnMainSafely's
retry-driving catch, and the bespoke pressKeyboardReturn fallback / performElementTap branches
whose result depends on the exception message.
Behavior-preserving: same defaults, same log output, same returned values. catchException is
non-escaping, so the inout capture in safeIsActionableCandidate is preserved.
Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED.
Collapse the gesture-command boilerplate in the executeOnMain switch behind two seams:
- performGesture(app, idleTimeout:) -> (timing, outcome): folds measureGesture + the scroll
idle-timeout/quiescence-skip wrap. Parameterized: touch gestures wrap (idleTimeout: true,
default); synthesis gestures (pinch/rotate/transform) pass idleTimeout: false because
RunnerSynthesizedGesture governs its own timing — a distinction that was previously implicit.
- gestureResponse(message:timing:frame:) over a GestureFrame (none/touch/drag): one factory for
the success DataPayload (message + gesture timing + optional touch/drag visualization frame).
The 13 gesture cases (tap x3, tapSeries x2, longPress, drag, dragSeries, swipe, mouseClick, pinch,
rotateGesture, transformGesture) now use the seams; ~120 lines of repeated quartet (measureGesture +
idle-wrap + unsupported check + 8-field DataPayload) collapse. CommandExecution.swift 949 -> 867.
Behavior-preserving: identical outcomes, timing capture, DataPayload fields/messages, and the same
touch-wrapped vs synthesis-unwrapped distinction. No wire change (the factory emits the already
wire-locked DataPayload internally). mouseClick (throws) and swipe (returns an optional frame) keep
their bespoke measure/wrap but route the success payload through gestureResponse.
Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED (no warnings).
Behavior-preserving relocation: move the text-entry concern out of the (formerly 1805-line)
RunnerTests+Interaction.swift — the repo's most-changed file — into a new
RunnerTests+TextEntry.swift. Interaction.swift drops to 1068 lines (-737).
Moved verbatim (no logic changes):
- value types: TextTypingRepairMode, TextEntryTiming, TextEntryResult, TextEntryTarget
- the focus -> type -> verify -> repair pipeline (typeTextReliably + focus orchestration +
readiness polling + dropped-char/repair heuristics)
- clearTextInput and the text-entry leaf helpers (editableTextValue, isPlaceholderValue,
isGenericTextInputLabel, normalizedElementText, moveCaretToEnd, estimatedDeleteCount,
keyboardBecameVisible, keyboardElementExists)
The whole cluster is text-entry-exclusive (its only callers are within the moved code or the
already-internal command entry points), so every symbol keeps its original visibility — no
access widened. Shared helpers used by gestures/snapshot/get-text (isKeyboardVisible,
visibleKeyboardFrame, textInputAt, textInputCandidatesAt, readableText, ...) stay in
Interaction. The new file is auto-included via the project's file-system-synchronized group.
This is the safe, cosmetic half of the "TextEntry engine" architecture candidate. A deeper
extraction behind a real seam/type is intentionally deferred: it carries the #245 revert risk
and wants stronger text-entry e2e coverage first.
Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED. Pure relocation, no behavior change.
* fix(ios): unify multi-touch gestures on two-finger synthesis + hinted unsupported errors
Make RunnerSynthesizedGesture the single iOS multi-touch engine and drop the older
incompatible models:
- rotateGesture now drives the two-finger XCTest synthesis path (dx=dy=0, scale=1,
degrees), mirroring the pinch migration in #634. The native XCUIElement.rotate(withVelocity:)
injected a single synthetic rotation that React Native's rotation recognizer did not read
reliably; synthesis fixes it. velocity is ignored on iOS (kept in the wire contract for
compatibility; rotation direction comes from the sign of degrees).
- pinch is now synthesis on iOS and a clear UNSUPPORTED_OPERATION on tvOS/macOS. The macOS
coordinate double-tap+drag heuristic (performCoordinatePinch) is removed: synthesis is
iOS-only, so macOS multi-touch is reported honestly as unsupported rather than approximated.
- RunnerInteractionOutcome.unsupported now carries an actionable hint, mapped to ErrorPayload.hint
(#639). Every unsupported gesture/tvOS path returns a concise message plus a next-step hint
(existing messages kept verbatim).
Net: on iOS, pinch/rotate/transform all flow through one synthesis primitive. swipe/scroll/pan/
fling remain single-finger drags (correct, unchanged).
Coverage: examples/test-app/replays/gesture-lab.ad exercises pinch + rotate against the gesture
lab and asserts "pinch changed yes" / "rotate changed yes".
* fix(ios): fail-fast macOS pinch + align capability/docs with synthesis-only
Follow-through for removing the macOS coordinate pinch path (the runner now returns
UNSUPPORTED_OPERATION for macOS pinch): reject it at admission instead of round-tripping.
- capabilities.ts: pinch now matches rotate-gesture/transform-gesture (Android + iOS
simulator only); macOS dropped. Removes the now-unused isMacOsOrMobileAppleSimulator helper.
- capabilities.test.ts: pinch expected unsupported on macOS and tvOS.
- website/docs/docs/commands.md: pinch listed for Android + iOS simulators only (removed from
the macOS app-session list); documents that iOS rotate ignores the optional velocity arg
(synthesis uses a fixed duration; direction comes from the sign of degrees).
Addresses PR #645 review HIGH #2 and MEDIUM #3.
* fix(ios): surface a hinted unsupported error for synthesis gestures at admission
Removing macOS pinch from the capability matrix makes macOS pinch (and the already-excluded
rotate-gesture/transform-gesture on macOS/tvOS/physical iOS) fail fast in ensureGenericCommandReady
before reaching the runner. That left the runner's macOS-specific hint unreachable on the daemon
path, so callers only saw the generic "<cmd> is not supported on this device".
Add an optional unsupportedHint to the capability matrix and surface it at admission, so the
synthesis-only gestures fail fast (no runner round-trip) AND return an actionable hint pointing to
where they work (Android + iOS simulator). Applied to pinch / rotate-gesture / transform-gesture.
Addresses PR #645 review (P2: route macOS pinch to the hinted failure).
* refactor(ios): single CommandTraits table for runner command classification
Replace the three hand-maintained switches in RunnerTests+Lifecycle.swift
(isInteractionCommand / isReadOnlyCommand / isRunnerLifecycleCommand) with one
source of truth: CommandType.traits, an exhaustive switch returning a
CommandTraits struct (interaction / readOnly / lifecycle axes), collocated with
CommandType in RunnerTests+Models.swift.
Pure refactor: every command's classification is reproduced verbatim, and the
three predicates become one-line lookups with unchanged signatures, so call
sites are untouched. The exhaustive switch makes it a compile error to add a
CommandType without classifying it, closing the drift that historically let
tapSeries/dragSeries/keyboardReturn fall out of isInteractionCommand.
readOnly is a 3-state enum (.always/.never/.conditional); .conditional preserves
alert's action-dependent read-only behavior, resolved in isReadOnlyCommand.
Classification feeds ADR-0002 session invalidation (the read-only retry that
nulls currentApp/currentBundleId), so behavior is intentionally unchanged.
Adds the "Runner command traits" term to CONTEXT.md.
* docs(ios): note CommandTraits.readOnly .conditional is alert-only (review follow-up)
* fix(ios): classify tapSeries/dragSeries/keyboardReturn as interaction commands (#643)
* fix(ios): classify tapSeries/dragSeries/keyboardReturn as interaction commands
tapSeries and dragSeries are the series forms of tap/drag (already interaction
commands); keyboardReturn is the sibling of keyboardDismiss (already an
interaction command). All three were missing from the historical
isInteractionCommand switch — a drift the new CommandTraits table (#642) makes
visible. Classifying them as interaction commands gives them the foreground-guard
+ stabilization preflight that their single-shot/sibling forms already get.
Behavior change: these three commands now re-activate a backgrounded target to
foreground and pay the stabilization delays before running. Ships separately from
the CommandTraits refactor (#642) and should land after that bakes.
mouseClick left unchanged: macOS-only and the foreground guard interacts with
bespoke macOS activation, so it needs a macOS smoke check first.
* test: cover iOS runner series commands in perf harness
On iOS the runner lowered `pinch` to performCoordinatePinch — a tap() then a
single-finger press(forDuration:thenDragTo:). React Native reads that as a pan,
so the pinch scale never changes (reported in #629: scale stays 1.00).
Route iOS pinch through the existing two-finger XCTest synthesis path
(transformGesture / RunnerSynthesizedGesture) with zero translation and rotation,
so RN's pinch recognizer fires. macOS keeps the coordinate path.
Validated on iPhone 17 Pro against examples/test-app: the gesture-lab.ad oracle
now passes 1/1 (fling/pan/pinch/rotate); it previously failed at the pinch step.
Refs #629
* perf(ios): early-exit text-entry readiness when the keyboard is visible
The XCUITest text-entry focus/readiness loops keyed their fast-exit on
focusedTextInput(), which is intentionally hardcoded to return nil on iOS (focus
predicates are stale there). As a result stabilizeTextInputBeforeTyping always
burned its full focusTimeout (0.4s) and waitForTextEntryReadiness burned its full
readinessTimeout (2.0s) in the normal case where the software keyboard appears —
~2.4s of dead wait before a single keystroke on every type/fill.
The software keyboard becoming visible is the reliable iOS readiness signal, so
both loops now return as soon as isKeyboardVisible() is true. The warmup-first-char
echo check and post-type verify/repair remain as drop safety nets.
Measured on iPhone 17 sim (Settings search field), median type time:
25 chars: 3342ms -> 1379ms (2.4x)
52 chars: 3969ms -> 2190ms
313 chars: 10.3s -> 8.6s (remainder is genuine per-char XCUITest typing)
Reliability unchanged: 64/65 trials exact (incl. a 50-word lorem ipsum, verified
by read-back + screenshot); the lone miss triggered the existing verify/repair.
* fix(ios): don't clear an already-empty text field (fixes fill mis-navigation)
clearTextInput unconditionally ran moveCaretToEnd (an edge-tap computed from the
element frame) + a 24-key delete burst, even when the field was empty. On a field
that repositions on focus — e.g. the Settings search bar jumping bottom->top and
revealing a 'Suggestions' list — that edge-tap used a stale frame and landed on an
adjacent row (Developer), navigating away instead of clearing. fill (replace) into
the search field went to the Developer pane (0/3 correct).
Skip the clear entirely when the field's value is already empty (placeholder
treated as empty): replacing into an empty field is a no-op, and skipping avoids
the stray edge-tap. fill into the Settings search now types correctly and stays
put: 5/5 exact (read-back + screenshot).
* perf(ios): resolve text fields via typed queries, not full-tree enumeration
textInputAt used app.descendants(.any).allElementsBoundByIndex (snapshots EVERY
element) to find the text input at a point. fill drove this repeatedly: once it has
coordinates, resolveTextEntryElement re-runs textInputAt on every verify/repair poll
iteration whenever the focused-field reference goes stale (e.g. the Settings search
bar repositioning bottom->top), so the full-tree enum dominated fill latency.
Query the text-input element types directly (app.textFields/secureTextFields/
searchFields/textViews) instead. Same matches, but XCUITest resolves typed queries
without snapshotting the whole tree. Measured (iPhone 17 sim, warm runner): fill 25
chars ~14.5s -> ~4.5s (3.2x), 6/6 exact. Same primitive #632 killed for get text.
* fix(ios): address review — focus-change gate + secure-field clear
Review P1 (focus race): the isKeyboardVisible early-exit in stabilizeTextInputBeforeTyping
and waitForTextEntryReadiness fired the instant the keyboard was visible — but when it was
ALREADY up from a previous field (back-to-back fills), that is before first-responder moves
to the newly-tapped field, so app.typeText could target the old field. Gate the fast-path on
a keyboard hidden->visible TRANSITION via a shared keyboardBecameVisible(wasVisibleAtEntry:)
helper; when the keyboard was already up, fall back to the settle/timeout (the prior, correct
behavior) instead of the ~2.4s dead wait the fresh case avoids.
Review P1 (F2): clearTextInput used editableTextValue(...) ?? "" and skipped clearing on
empty — but editableTextValue returns nil for secure (and unknown) fields, so secure fields
were NEVER cleared and replace concatenated stale+new. Distinguish nil (clear) from "" (skip).
Device-validated: fresh fill fast-path preserved + exact; a second fill with the keyboard
already up still types into the correct field and replaces (not concatenates).
Long-running AgentDeviceRunnerUITests sessions can inherit Xcode 26 screenRecording/deleteOnSuccess defaults and leave hidden testmanagerd attachments behind when sessions terminate outside a clean XCTest success path.
Pin an explicit AgentDeviceRunnerUITests.xctestplan with screenshots and keepNever attachment lifetimes, then normalize the generated per-session .xctestrun copy before test-without-building. The .xctestrun normalization protects stale cached artifacts and Xcode versions that ignore some test-plan attachment lifetime keys during build-for-testing.
This only affects XCTest's automatic attachments; user-requested agent-device recordings still use the existing record command paths.
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Michał Pierzchała