Every declared AgentDeviceRunnerUITests method now belongs to a lane, and
the #if guard is the classification: AGENT_DEVICE_RUNNER_UNIT_TESTS alone
means a pure runner decision (runs on the macOS host on every PR — ci.yml's
existing compile job now executes the bundle it builds), '&& os(iOS)' means
runner/XCTest semantics (simulator lanes only). check:xctest-selection
evaluates the guards per platform, derives each lane's reach, and fails on
a flagged identifier that is undeclared or uncompiled on that lane, on a
declared test no lane reaches (found the two tvOS-only tests, dark since
birth — widened to os(tvOS) || os(macOS)), and on testCommand reaching any
lane. The host and nightly lanes assert executed == derived reach, so a
missing -D flag or a guard that compiles a file out reads red, not as a
smaller green. One duplicate test deleted (sparse-verdict assertions folded
into its twin).
Make iOS regular snapshot eligibility one backend-neutral presentation rule.
Acquire tree nodes conservatively, preserve interactive scroll containers, normalize surviving hierarchy, and keep raw membership plus daemon publication policy unchanged. Part of #1797.
- iOS and macOS unit-enabled runner builds
- 2 focused XCTest cases
- 3 production-path publication tests
- live Settings snapshots: 73 regular nodes and 167 raw nodes, both healthy tree captures
## Summary
Route every iOS capture-plan backend through one SnapshotPresentation boundary while preserving each backend's current output semantics.
SnapshotAcquisition now carries nodes and attempt-level facts, PresentationOptions is the stable policy input, and only the presentation module assembles wire-facing nodes. Part of #1797.
Touches 10 files within the existing iOS snapshot module and its architecture vocabulary; scope did not expand beyond the planned command family.
## Validation
- Unit-enabled iOS runner build and focused presentation XCTest passed.
- Removing the custom-action handoff made the focused test fail with exactly two assertions, proving the routing check is non-vacuous; restoring it returned to 1/1 green.
- macOS runner build passed for the shared Swift path.
- XCTest selection and repository formatting checks passed.
Introduce RawAXNode and PresentedNode so acquisition backends can no longer construct the wire-facing snapshot shape directly. Preserve current output while #1797 moves semantics behind the seam.
Non-vacuity: setting PresentedNode.label to nil made testSnapshotPresentationPreservesCurrentWireShape execute once and fail on the missing label field; restoring the production mapping made the same focused XCTest pass.
* fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1)
`pnpm gate replay-android` failed 4/8 whenever it ran after the full-tier Android E2E
(replays-nightly run 32107665052, job 95620294899): 05-app-lifecycle, 06-swipe-gestures and
both fixture replays diverged under "A system surface covers the app". The E2E was not the
cause. Reproduced on a pixel_7 / API 36 AVD with the same cutout geometry CI's
`avdmanager --device pixel_7` produces (status bar 136px, not the 63px of a plain 1080x2400
skin):
- `03-scroll-discovery.ad` runs `scroll up 3`. The scroll planner clamps travel to the viewport
minus a 5% band, so the touch-down landed at y=120 — inside the 136px status bar — and
pulled the notification shade instead of scrolling. On API 36 the app window is
edge-to-edge, so the reported viewport starts at y=0 and includes that bar.
- The shade then covered every replay until `04`'s `back` closed it. Native readdir order on
the runner (03, 05, 06, fixture/02, fixture/01, 04, 01, 02) put four files in that window;
the last green run (2026-07-30) had 04 right after 03, so the pull was masked.
Fix in the product, not the lane: DEFAULT_EDGE_PADDING_FRACTION 0.05 -> 0.1 in the TS scroll
planner and its Swift port. Every real Pixel has a cutout (5.7% of a Pixel 7's height) and an
iPhone's Dynamic Island status bar is 6.9%, so any saturated `scroll up` opened the shade /
Notification Center for real agents too. Parity vectors updated in both suites plus a Pixel 7
regression vector (1080x2400, amount 3 -> touch-down y=240 > 136).
Second contamination the same order exposed once the shade was gone:
`fixture/02-selector-routes-covered-diagnosis.ad` is a #1715 reproduction recipe that FAILS
BY DESIGN at step 9 (covered-target refusal) and leaves the device in landscape, yet the
gate enumerated `test/integration/replays/android` recursively. iOS keeps gate replays in
`replays/ios/simulator` and fixture recipes in `replays/ios/fixture`; Android now mirrors that:
the six Settings replays move to `replays/android/emulator`, `test:replay:android` points there,
and `fixture/` stays E2E-owned (`full:fixture-replays` already runs 01 by path). android.yml
and the workflow-evidence fixture follow the path; the replay-compat manifest keeps the
historical paths it pins at released tags.
Verified live (Pixel 7 geometry, API 36, --retries 0): control run at main head in CI order
reproduces exactly CI's 4/8; with the fix, `pnpm gate replay-android` 6/6 in both native and
CI order, and `03` leaves Settings on screen (scroll up 3 now touches down at y=240).
* test(scroll): drive the TS and Swift scroll-plan parity vectors from one table (#1820 review)
The two suites hand-mirrored the same vectors and #1820 had to edit both by hand — the drift
class the repo already closes for the tap-point rule via contracts/fixtures/tap-point-policy.json.
The scroll vectors (plus both planner constants, pinned behaviourally on a 1000px axis) now live in
contracts/fixtures/scroll-gesture.json; scroll-gesture.test.ts and RunnerTests+ScrollGesture.swift
iterate it. Verified: vitest 10/10; the four XCTests run on an iOS 26.2 simulator with the unit
flag on (Executed 4 tests, 0 failures).
Also: test/ci/android-workflow-evidence.json says what it guards.
Follow-up for content-safe viewport bounds + discovery order: #1821.
topLeadingNavigationFallbackPoint and isTopNavigationControlFrame only
validated width/height, never the origin. CGRect.infinite has a finite
(if absurd, ~-9e307) origin, so it slipped past the isFinite/>0 guard:
the fallback point collapsed to roughly (-9e307, -9e307), and an
unresolved element frame was classified as sitting in the nav header
band.
Extract isUsableNavigationFrame(_:), which adds the isInfinite check
TapPointPolicy.isAllowed already uses for the same rect, and share it
between both helpers.
Fixes#1812
* fix: reject distinct ambiguous mutation targets
* fix(ios): scope the raw-match rejection to mutating dispatches
`RunnerTests+Interaction.findElement` applied the new fail-closed
classification to `querySelector` as well as press/type, because the read
call site takes the default `allowNonHittableFallback: false`. With one
visible/hittable match and one non-hittable same-selector duplicate the
query started returning AMBIGUOUS_MATCH where it previously selected the
hittable element, and `queryDirectIosSelectorOrFallback` preserves that
error for read callers — so `get`, `is`, and `wait` surfaced an error
instead of their prior answer.
`classifyDirectSelectorCandidates` now takes a `rawMatchPolicy`. Mutations
keep `.rejectDistinctMatches` (the default, so no mutation call site
changes); `queryElement` passes `.preferHittableMatch`, restoring the
prior read rule: prefer the single hittable match, ambiguous only when
hittable matches compete, and never adopt the Maestro coordinate fallback.
The Maestro expected-point path is untouched.
Covers the one-hittable + one-non-hittable read, competing hittable reads,
and the non-hittable-only read. ADR 0011's amendment now states the scope.
* test(ios): execute selector read ambiguity regression
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(ios): expose AX custom actions on merged accessibility elements
Apps that merge a card into one accessibility element for VoiceOver (React
Native's `accessible` prop) publish the card's real affordances as
UIAccessibilityCustomActions rather than as child elements. Our snapshot showed
only the merged node, so an agent looking for a feed card's options control had
nothing to aim at and fell back to coordinate guessing.
`snapshot --actions` now names them:
@e8 [link] "feedItem-by-whiskers.test" actions: ["Reply", "Repost", "Open post options menu"]
Opt-in, because the AX server cannot serve custom actions in a bulk tree
request: adding the attribute makes testmanagerd's reply decoder reject the
nested arrays a custom action serializes into, drop the reply, and time the
request out (~65s vs ~110ms). Only per-element reads answer, at ~100ms each, so
the runner reads at most 12 labelled childless nodes and stops at the
capture-plan deadline. The request pins the private-AX backend, since no other
backend can read the attribute, and reports that as its own `requested-backend`
verdict so a deliberate pin never renders as a degradation warning.
Invocation is not shipped: the actions are readable but not invocable from the
runner. RunnerAXSnapshotBridge.h records the five APIs that were tried.
* fix(ios): disclose a capped custom-action pass, and read on-screen elements first
Two gaps in the first cut.
An element the bounded pass never reached rendered identically to one with no
custom actions, so a capped capture silently taught the reader that later feed
cards have no affordances — the exact mis-inference this feature exists to
prevent. The runner already counted reads against candidates; it now carries
both to the response, and the verdict renders one response-level line when the
pass was incomplete. A complete pass stays silent, and "never asked" stays
distinguishable from "read none" (the key is absent, not (0, 0)).
The obvious remedy for a capped pass would be a scoped re-run, but scope is
applied when the Swift walk builds nodes, long after the read pass, so it does
not redirect the budget at all — measured: reads=12 candidates=18 with and
without --scope. Rather than print a remedy that does nothing, the read pass now
orders candidates on-screen first. That makes the budget land on elements an
agent can act on, and makes the disclosed remedy true: scrolling changes the
on-screen set, so a re-run reads elements the previous pass could not.
Also states plainly, in the flag help and the tool/SDK field description, that
the names are for planning: nothing invokes them, so the affordance is reached
through the element's detail screen, the same control exposed elsewhere, or
coordinates.
* test(snapshot): pin the custom-action coverage pair in the verdict shape assertion
* chore(scripts): classify the --actions flag in the integration progress model
The completeness gate flagged snapshotCustomActions as unclassified, which is
what it is for. It gets its own bucket rather than joining the provider-scenario
table: the values come from the private AX client inside the runner process, and
the fake runner derives its behavior from fixture tables that cannot fabricate
custom actions, so there is no provider-backed scenario to claim. The owning
coverage is named instead — runner XCTest unit, snapshot-lines, snapshot-quality.
* fix(ios): fail closed on unserviceable --actions, bound each read, cap output, and count actions in identity
Four review findings.
1. `--actions` with `--raw`, or on any target that is not an iOS simulator, used
to succeed and return nodes with no actions — a requested capability silently
no-opped, indistinguishable from "this screen has none". Both now fail closed.
The raw pairing is rejected at the shared request seam (INVALID_ARGS) so CLI,
Node client and MCP answer alike before any device work; the platform case is
rejected once the session device is resolved (UNSUPPORTED_OPERATION), naming the
resolved target. `diff --actions` was already rejected as an unsupported flag.
2. The per-element AX read had no timeout, so one wedged element could consume
the whole capture budget. Each read now runs off-thread behind a 1s wait. A
timed-out element counts as unread, never as "read, and it has no actions", so
the existing partial-pass disclosure already covers it.
3. The element budget bounded element count only; one element could still return
an unbounded list of unbounded names. Capped at 8 names of 80 characters, and
clipped elements are counted into the coverage so a truncated list is disclosed
rather than silently presented as complete.
4. Action names were rendered unescaped, and no comparison key read them. Names
now get the same escaping as text previews plus control-character folding, so an
app-authored name cannot split or corrupt a line. `actions` joins the diff
comparable key, the unchanged-comparison projection, and — the sharper bug —
the snapshot presentation key, without which `snapshot` followed by
`snapshot --actions` on a still screen answered "unchanged" and never delivered
the actions that were explicitly requested.
* fix(ios): contain a hung custom-action read instead of accumulating orphans
The 1s read deadline frees the caller, but the underlying AX call is a
synchronous XPC round trip that cannot be cancelled — it keeps running. On a
global concurrent queue that meant repeated `snapshot --actions` against a
wedged element piled up orphaned reads, all using the shared XCAXClient
concurrently. The deadline was containment for the capture, not for the runner.
Since the call cannot be cancelled, contain it instead:
- every read runs on one dedicated serial queue, so a wedged call can never be
joined by a second concurrent user of the shared client;
- a single-flight guard refuses to dispatch at all while an abandoned read is
still outstanding, so a repeated capture adds no work — the dispatch counter
stands still;
- the read pass stops at that point rather than paying a deadline per element
on reads that would all be refused, and reports `blocked` so the capture stays
honest. That gets its own line, because the partial-pass remedy (scroll and
re-run) cannot clear a hang and would send the reader in circles.
Recovery needs no reset: when the hung call finally returns, in-flight drops to
zero and reads resume.
The regression drives a fake AX client that never returns, and asserts the three
things the fix exists for — exactly one in-flight read with no further
dispatches across repeated captures, immediate returns with the skip disclosed
instead of the scroll remedy, and reads working again once the wedge clears.
* ci(ios): execute the custom-action runner regressions instead of only compiling them
The iOS workflow runs a targeted -only-testing list, so a runner test that is
not named there is compiled by the build step and then never executed. All seven
custom-action tests were in that gap — including the containment regression,
which is the only executable proof that a hung AX read cannot accumulate
orphaned in-flight reads.
Red/green against the containment regression, with the fix reverted to its
pre-fix concurrency behavior (global concurrent queue, no single-flight guard,
no blocked exit):
RED in-flight 6 (want 1), dispatches 6 (want 1), each repeat paid the full
1.004s deadline (want <0.2s), blocked=false (want true), and the
in-flight drain never completed — "Exceeded timeout of 5 seconds".
GREEN 7/7 pass, containment regression in 1.02s.
* fix(test): wait for typed text to settle in the hidden-keyboard runner test
testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden read textField.value
in one shot right after executeTypeCommand returned. The simulator commits
synthesized keystrokes after the command responds, so on a loaded CI machine
the read landed mid-word — observed failures reported ("h") and
("hardware-ke"). It failed on 3 of 5 runs of a branch carrying zero Swift
changes and passed on re-run.
Poll the value until it holds the expected text (10s ceiling) and assert on
the last value read, so a real regression still fails with what the field
actually held. Both reads in the test use the same helper; the assertions are
unchanged.
* Revert "fix(test): wait for typed text to settle in the hidden-keyboard runner test"
This reverts commit caecdc6d5b.
* fix(ios): wait for hidden-keyboard synthesized text to commit before responding
The synthesized-first-responder bare-type route returned ok as soon as the
private XCTest event record was posted, while the target app was still
committing characters. On slow CI simulators the trailing characters landed
after the response, so agents (and the smoke test) observed a truncated
field value through the public type path.
After dispatch, poll the tapped element until its value reaches
textBefore + typedText, exit immediately when the app transforms the input
(formatter, mid-text caret, autocomplete), and if progress stalls as a
strict prefix, re-synthesize the missing tail once. Submit-suffixed text
keeps the old immediate return so a repair can never double-submit.
Validated on a booted iPhone 17 Pro simulator: 5/5 passes of
testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden under full-core CPU
load, with the commit wait absorbing up to ~390ms of post-dispatch lag that
the previous code ignored (uniform ~494ms dispatch-only before); the tail
repair never had to fire, consistent with commit lag rather than true drops.
* fix(ios): make the synthesized commit wait observation-only
Addresses the P1 on #1673: a 600 ms quiet prefix cannot distinguish delayed
app-side commit from a genuinely dropped suffix, so re-synthesizing the tail
could post it while the original was still queued and commit the text twice
after the command had already reported ok.
Drop the repair path entirely — the tail builder, the quiet-window constant,
the stall tracking, and the synthesizer dependency the wait only needed in
order to repair. What remains is a bounded observation: poll the tapped
element until the value commits, the app transforms it, the value becomes
unreadable, or the 3s ceiling expires. A dropped suffix still reports ok,
exactly as before this change; only the false truncation from commit lag is
removed.
The submit-key skip stays, for its own reason: the app may clear or rewrite
the field on submit, so textBefore + typedText is not the value to wait for.
Also wire testSynthesizedTextCommitProgressWalksExpectedPrefixOnly into the
iOS smoke lane — that workflow enumerates its tests by hand with
-only-testing, so a new test that is not listed never runs.
* feat: add scale-only screenshot sizing
* fix: refuse retired --max-size inputs on every released surface
Released sizing inputs must fail closed with migration guidance instead of
silently producing native-size artifacts:
- contracts: RETIRED_SCREENSHOT_MAX_SIZE declaration + SCREENSHOT_SCALE_LIMITS
as the single source for the scale bounds and migration messages
- .ad parser: released 'screenshot ... --max-size N' and 'record start ...
--max-size N' lines now refuse at parse time (frozen replay-compat witnesses)
- daemon: screenshot rejects old-client screenshotMaxSize like recording does;
the recording guard now shares the same contract data
- Node client: screenshot/record daemon writers refuse the removed { maxSize }
option before transport
- CLI: --max-size unknown-flag error carries the migration guidance
- config/env: stale screenshotMaxSize config keys and the retired
AGENT_DEVICE_SCREENSHOT_MAX_SIZE env var are refused for sizing commands
(other commands keep working)
Quality: numberField now reuses the canonical readOptionalNumber contract
helper (AppError bounds instead of plain Error); png-resize inlines one-use
wrappers and restores the worker-thread rationale; docs typo fixed.
* test: drop retired maxSize entries from the MCP undocumented-input allowlist
* fix: refuse retired maxSize at the MCP field-projection seam + release-provenance corpus witnesses
- readFieldInput silently dropped undeclared keys before the daemon writers
could refuse them, so an MCP call carrying { maxSize } reached transport and
returned native-size success. New retiredField() combinator declares the
removed key in the field map: the projection seam refuses it with the
canonical migration message and the JSON schema no longer advertises it.
Real-route MCP executor regressions cover screenshot and record.
- replay-compat corpus: derived v0.20.5 witnesses for the released screenshot
and record --max-size forms (SHA-256 pinned, new retired-capture-size
coverage surface) so check:replay-compat proves the shipped syntax refuses
with migration guidance instead of degrading silently.
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
* fix(ios): pin tap-outcome corroboration probes to the baseline's backend
The recorded-failure screens are exactly where the capture plan flips
between XCTest and private-AX (the penalty boundary), so #1605's
same-backend requirement failed closed right where XCTest tap false
negatives actually happen: the baseline was captured via private-AX
under penalty, the probe came back via tree, and a landed tap surfaced
as XCTEST_RECORDED_FAILURE. In the AppControlBench bsky-16 run this
fired four times, each sending the model into a re-observe/retry spiral.
The comparison stays same-backend by design (backends are not comparable
views of a screen); instead the probe is now CAPTURED the way its
baseline was: a new internal preferredBackend option (never CLI-exposed)
threads daemon -> runner, and a private-AX-preferred capture takes the
exact penalized route — privateAX-first plan, 'deferred' verdict, no
degradation warning, no settle budget reset.
Live-verified on the deterministic repro (Bluesky drawer-menu press
under penalty, seeded bench feed): errored with the backend-mismatch
diagnostic before, corroborates as landed after, with no mismatch phase
in the request diagnostics. Daemon tests cover pinned and unpinned
baselines end to end through the dispatch context; the Swift plan gate
is a pure function with an executed in-bundle test (added to the ios.yml
regression list).
* style: oxfmt
* fix: exclude raw baselines from corroboration and prove the pin end to end (review)
Raw baselines could not be pinned: the raw diagnostic plan keeps
tree-first error propagation by contract and is never rerouted by the
penalty or the preferred backend, so preserving 'raw: true' on the probe
recreated exactly the backend-mismatch false failure this PR removes.
Corroboration now declines raw baselines up front (they are diagnostics,
not evidence baselines) with a regression pinning that no probe capture
is dispatched at all.
The wire is now regression-proven at every hop: a dispatch-level test
drives dispatchCommand with the context flag and asserts the emitted
RunnerCommand carries preferredBackend (red if handleSnapshotCommand or
the interactor stops forwarding); the injected-transport test asserts
the interactor's snapshot payload both ways; and a runner unit test
decodes the wire JSON, projects it through the extracted
snapshotOptions(from:), and composes it with the plan rule — pinned
regular plan defers to privateAX-first, RAW plan stays untouched.
Executed on-simulator; added to the ios.yml regression list.
* feat(ios): extend depth-capped private-AX captures via element-rooted requests
The AX server's depth limit is per-request (kAXErrorIllegalArgument above
a tree-size-dependent value), so a capture capped at depth 56 on
Bluesky-class React Native trees returned chains of unlabeled [other]
containers and hid every actionable control below the cap — agents fell
back to screenshot-and-coordinate guessing.
After a capped serialization, the bridge now re-issues the same snapshot
request rooted at each deepest-level childless node's live accessibility
element, splicing the returned children in and chaining further while
capped subtrees remain — bounded by a call limit, the shared node
budget, and the capture-plan deadline. The AX server counts node levels,
not edges: a maxDepth=56 request emits nodes to depth 55, so the
frontier is the deepest observed level, kept only when it sits at the
cap; a tree that ends naturally above the cap yields no frontiers and
costs nothing.
A capture whose frontier extension drained every capped node no longer
reports itself depth-limited — the re-run hint it used to trigger could
not add anything. Explicit --depth requests stay exact captures with no
extension.
Live on the seeded Bluesky bench feed: 8 extension calls (~106ms each)
turn the 117-node all-[other] tree into a 182-node tree carrying post
text, testID links, and every feed control (Reply/Repost/Like/options);
steady-state capture 0.52s -> 1.37s. Observation-only captures paying
the extension needlessly is #1626.
* fix: count missed frontiers and tighten deep-extension shape (review)
The completeness verdict inverted in the failure paths: a frontier whose
live element vanished or whose re-rooted request failed was silently
dropped, so an all-miss extension reported pendingFrontiers=0 and the
capture presented as complete while whole subtrees were missing. Missed
frontiers are now counted, logged, and keep the depth-limited verdict —
the pure decision lives in privateAXDepthLimited with in-bundle tests.
The truncation hint no longer advertises --depth (an explicit --depth
capture disables extension, so following it returned strictly less than
the capture that produced the hint); the honest remedy is a plain
re-run with a fresh extension budget.
Shape: candidates collected only at the cap boundary and not at all
when extension is disabled (exact --depth captures pay zero
bookkeeping); the zero-caller 3-arg overload is gone; the response
shape's keys are shared constants; the unsupported-selector diagnostic
is restored; the exact-depth policy is hoisted to one named local.
* test: execute the deep-extension miss-path contract in CI (review)
The depth-limited regression compiled but never ran (absent from
ios.yml's -only-testing list), and the pure Swift consumer test was
vacuous against the Objective-C producer — a literal missedFrontiers
proves nothing about the increments. RunnerAXSnapshotFrontier and
extendSnapshotFrontiers move to the header as the executed contract
seam, and testDeepExtensionCountsMissedFrontiers drives both real miss
paths with fabricated snapshots: a nil accessibilityElement (explicit
nil property — bare NSObject resolves the key through a UIKit category
and takes the call path instead) counts missed without consuming a
call; a resolving element whose client cannot serve the re-rooted
request consumes a call AND counts missed. Both tests join the
executed ios.yml list.
Red-before verified on-simulator: with both increments stripped the
producer test fails at the counter assertion (0 != 2); restored, both
tests pass.
* fix(ios): corroborate recorded tap outcomes
* fix(ios): preserve corroborated tap target identity
* fix(ios): suppress corroborated tap retries
* fix: require comparable iOS tap evidence
* fix: bound iOS tap corroboration baseline
* test(ios): deterministic injection seam for recorded-tap-failure corroboration (#1605 merge gate)
The field failure cannot be reproduced on this head: the tap false-failures
were a downstream symptom of XCTest-channel saturation, which the #1587
capture fixes removed. The seam records a real XCTIssue AFTER the real
gesture inside the per-command failure-count window, so
xctestRecordedFailureResponse and target invalidation fire byte-for-byte
like the field failure. Armed via a decrementing /tmp flag file (the daemon
regenerates tampered xctestrun templates, so env plumbing cannot reach a
daemon-spawned runner); compiled only under AGENT_DEVICE_RUNNER_UNIT_TESTS.
Live evidence on a daemon-spawned runner (Bluesky, ad-bsky-repro sim):
- landed case: injected failure on a real Search-tab tap -> success with
the corroboration warning, screen verifiably on Search, no redispatch,
runner serving next commands; flag consumed exactly once.
- unchanged case: injected failure on a dead-coordinate tap -> capture
unchanged -> XCTEST_RECORDED_FAILURE preserved with the new honest hint;
runner still usable.
- field-shape race (relaunch -> full snapshot -> immediate press, 5
attempts): no natural recorded failure occurs on this head — the hostile
tree needed for channel saturation is gone, corroborating the causal
story.
* fix: reconcile tap corroboration with current interaction semantics
RunnerSynthesizedGesture.m and RunnerSynthesizedTextEntry.m each independently
reflected into XCSynthesizedEventRecord/XCPointerEventPath. Extract the
common resolution (classes, addPointerEventPath:/setTargetProcessID:/
synthesizeWithError:/processID, the RunnerRequireClass/Selector/
ApplicationSelector checks, the shared objc_msgSend typedefs, and the
"name: reason" exception formatter) into RunnerXCTestEventBridge.h/.m.
Each caller resolves its own extras on top of the shared core: gesture
resolves the 2-arg initWithName:interfaceOrientation: plus touch-path
selectors, text entry resolves the 1-arg initWithName: plus text-input
selectors.
* fix: type into focused iOS inputs without AX
* fix: fill AX-hostile iOS text inputs
* fix: keep scrolling containers from stealing taps
* fix: stop agents after explicit task success
* chore: format benchmark guidance
* fix(ios): preserve fill semantics across fast paths
* test: retire direct selector fill expectations
* test: assert runtime selector fill evidence
* fix(ios): preserve verified and Maestro fill paths
* refactor(ios): isolate synthesized text entry
* fix(client): preserve open diagnostic paths
* fix(ios): expose structured text entry route
* fix(packaging): strip text entry policy tests
* fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598)
The runner already tapped a keyboard's own Hide/Dismiss/Done key when the
AX tree exposed one, but iPhone's default software keyboard has no such
key, so `keyboard dismiss` returned UNSUPPORTED_OPERATION on the common
case and agents proceeded with the keyboard (and any live QuickType
predictive-text bar) still up.
Live-validated on throwaway simulators before choosing a design: hardware
escape key (no effect without a connected hardware keyboard), swipe-down
starting on the keyboard (does not trigger UIKit's interactive dismissal
on Settings/Safari/Contacts), and a private
`performAction:onElement:value:error:` AX call (hung the runner for 90s on
a guessed action name, force-killed by the daemon timeout) were all ruled
out. The dismiss-key tap remains the primary mechanism (iPad, or any app
with an inputAccessoryView Done/Cancel button); a new snapshot-derived
safe-area tap is added as the disclosed last resort, computed to land
outside both the keyboard and every currently-hittable element so it is a
safe no-op even when it fails to dismiss.
The response now discloses which mechanism actually fired
(`mechanism: 'dismissKey' | 'safeAreaTap'`) across the CLI/daemon dispatch
path, the SDK runtime.backend surface, and session-event summaries, so
callers can tell a real dismiss-key press apart from a best-effort tap.
UNSUPPORTED_OPERATION now says both mechanisms were tried.
* fix: satisfy CI formatting and complexity gates
oxfmt on three touched files; buildKeyboardActionSummary split so the
dismiss wording (incl. the safeAreaTap mechanism disclosure) lives in its
own helper below the complexity threshold.
* fix: any-element obstacle rule for the safe-area dismiss tap (#1606 review P1)
A role allowlist cannot prove a point is AX-empty: an unlabeled RN
Pressable surfaces as a hittable Other, and a tappable parent can cover a
point its static-text child does not. Every known element frame now counts
as an obstacle regardless of role or hittability, with only ~window-sized
structural frames exempt (isStructuralRootFrame, 95% coverage) — exempting
those is what keeps the rule satisfiable, and a genuinely tappable
full-screen backdrop staying exempt is the disclosed, accepted behavior of
this fallback. One .any resolution replaces ten typed queries (single tree
snapshot, no per-element isHittable round trips), so the stricter rule is
also cheaper.
* fix: drop the safe-area tap — background-tap dismissal is unsupported (#1606 review P1, round 2)
No geometry or role query can prove a coordinate is side-effect-free: after
the any-element rule, the structural-root exemption still deliberately
removed full-screen actionable elements (RN Pressable backdrops) from the
obstacle set, so the tap could navigate or submit — and report success
because the mutation hid the keyboard. Per review, generic background-tap
dismissal is now explicitly unsupported: the dismiss key is the only
mechanism the runner vouches for, UNSUPPORTED_OPERATION says so and steers
callers to press-the-next-target / keyboard enter, and the mechanism field
narrows to 'dismissKey'. Unrecognized wire mechanisms degrade to the bare
message and are dropped from event details.
The Apple runner ships as source in dist/apple/runner, and packaging strips
#if AGENT_DEVICE_RUNNER_UNIT_TESTS blocks — but tests outside such blocks
shipped whole and compiled on every user's machine. Two files leaked six
tests this way (RunnerTests+LifecycleCacheTests, RunnerTests+SnapshotTraversalIdentityTests).
Wrap the strays and close the class: packaging now fails if any XCTest-shaped
method (func test*) survives stripping, with testCommand in RunnerTests.swift
as the only allowlisted entrypoint.
* perf(ios): stop re-paying known-failing work on penalized private-AX captures
Live-measured on the Bluesky bench feed (139-171 nodes), each private-AX
capture wasted ~1.35s of its ~1.65s runner-side cost re-doing work a prior
capture already proved futile:
- ~1000ms: the viewport read is XCTest main-thread work; under the channel
penalty it reliably burns its full timeout and falls back to the root
frame anyway. Honor the penalty in privateAXSnapshotViewport the same way
capture plans do.
- ~310ms: the depth ladder re-paid the kAXErrorIllegalArgument rejection of
the default depth on every capture. Remember the accepted rung per bundle
(penalty-duration TTL, cleared on target process change); expiry re-probes
the full depth so screens that recover are not capped forever. Explicit
--depth requests bypass the memory in both directions.
Steady-state hostile-screen captures drop 2.65s -> ~0.55s CLI wall, and
press --settle round-trips drop ~5.4s -> ~2s (settle needs two captures).
* fix(snapshot): distinguish penalty-deferred captures from genuine recoveries
A capture whose backend was PRE-selected by the XCTest-channel penalty was
stamped with the same recovered verdict as one that ground through a live
failure. Two costs followed on hostile screens (Bluesky bench: 130 repeats
per 30-task run):
- the daemon repeated the full fell-back warning on every capture, long
after the arming capture had already said it once, and
- settle's one-shot private-AX budget reset fired on every loop even though
the capture paid no grind to give the budget back for.
The deferred plan now stamps reasonCode 'deferred' (a new code; older
daemons drop unknown codes and keep today's behavior). The daemon keeps the
verdict recovered but suppresses the repeated warning, keeps the depth-cap
line, and skips the settle budget reset for deferred captures.
* style: wrap reasonCode union to satisfy oxfmt
* fix(ios): bind accepted-depth memory to the process, pin deferred through the wire parser
Review follow-up (#1587):
- The depth memory was cleared only in refreshCachedTargetIfProcessChanged;
resetTargetAfterExternalRelaunch -> invalidateCachedTarget drops the cached
PID without clearing it, so the next activation had no old PID to compare
and could reuse a stale shallow rung for up to 120s (also A->B->A when A
restarted while inactive). The memory now stores the PID it was learned
under and only matches the same live process; recording without a PID is
refused. Every invalidation path is covered automatically because they all
drop currentAppProcessIdentifier.
- The deferred settle/warning tests constructed typed verdicts directly, so
removing 'deferred' from the accepted reason-code set would silently
restore the repeated warning and budget reset while tests stayed green.
They now parse a raw runner-wire object through readSnapshotQualityVerdict
(red on base: parser strips the code -> warning re-appears, reset fires).
- Extracted shouldReadPrivateAXViewportViaXCTest() and pinned the penalized
viewport skip with an in-bundle regression test.
* fix(ios): wait for post-dismiss content settle before the next gesture (#1542)
Dismissing the keyboard can trigger the app's own ScrollView content-offset
correction (e.g. releasing the inset it grew to keep a focused field above
the keyboard). That correction is a separate, unsynchronized animation that
`keyboard.waitForNonExistence` knows nothing about — the keyboard AX element
can disappear well before the app visually settles. The very next command is
frequently a synthesized, AX-free drag (scroll/gesture, kept AX-free so it
still works under #1105-family AX degradation), which has no XCTest
quiescence wait of its own, so it can land mid-animation and net to zero —
the "scroll does nothing" symptom on the Form screen's checkout-form.ad leg.
Add a bounded, AX-free screenshot-stability wait to dismissKeyboard() so the
runner only returns once the screen has actually stopped changing (or a
generous cap elapses). The stopping decision is a pure function
(runnerScreenshotStabilitySettled) covered by unit tests under
AGENT_DEVICE_RUNNER_UNIT_TESTS; the surrounding capture/sleep loop is the
thin, untestable I/O shell around it.
Live-verified on iPhone 17 Pro / iOS 26.2: the scroll now visually lands at
the correct position (confirmed via screen-recording frame correlation)
instead of leaving content at its pre-scroll offset.
Not a full fix for #1542: the checkout-form.ad corpus leg still fails at the
same step, now because the daemon's shared post-gesture snapshot
stabilization (src/daemon/post-gesture-stabilization.ts) can read a
stale-but-internally-consistent AX tree after the AX-free scroll and
mistake "unchanged across polls" for "settled", so the following click's
off-screen guard sees pre-scroll node positions. That is a cross-platform,
cross-command stabilization semantics change and needs a design decision,
not a unilateral fix here — see the PR description.
* ci(ios): execute the screenshot-stability runner tests (#1559 review)
* chore: remove verified dead code and migration scaffolding
Multi-agent audit of accumulated waste, every finding adversarially
verified against call sites, git history, and the published surface
before removal. Net -710 lines.
- delete src/core/platform-descriptor/ (superseded ADR-0009 migration
scaffold; parity tests now assert an inline table)
- remove test-only seams: registry introspection exports,
CommandFacet.extraDaemonWriters, MaestroEngineOptions.timing
- remove dead flexibility: backend capability allow-list,
screenshot-diff maxRegions, CloudWebDriverSupportLevel 'partial',
clearFirst on the TS+Swift runner wire contract
- remove dead deprecated surface: --session-locked /
--session-lock-conflicts aliases (hard migration error now points at
--session-lock), replay export --format single-value enum,
unused Lease*Payload contract types, runtime-layer rotate duplicate
- collapse pass-throughs/duplication: withRetry adapter,
default-cloud-artifact-provider, connect-profile client-id hashing
(3x sha256 impls -> one helper, byte-identical output), shared
scripts walker, cloneValue -> structuredClone, fill-diagnostics
moved into android/
BREAKING CHANGE: --session-locked and --session-lock-conflicts now fail
with a migration error pointing at --session-lock; replay export
--format is removed (Maestro was the only value); Lease*Payload types
are dropped from the ./contracts subpath.
* chore: satisfy fallow gates tightened by #1363/#1364 after rebase
- drop the consumer-less AndroidFillVerificationNode re-export
- reuse requireSnapshotSession in resolveSnapshotForRef instead of
inlining the same authorized-frame resolution (fallow clone group);
the helper's return type now guarantees the session it already
throws for
* chore: address review — keep cloud-webdriver partial capability metadata
The partial/supported/unsupported levels and their notes are part of the
lease-response capability contract for genuinely limited operations
(Appium page-source snapshots, upload-then-install), not dead
scaffolding. Restore them and the asserting tests unchanged from main.
Also add the missing CHANGELOG entry for the Lease*Payload type removal
from agent-device/contracts.
tvOS has no SpringBoard (it uses PineBoard/HeadBoard), so probing
com.apple.springboard for blocking system modals raises an XCTest failure
("Application com.apple.springboard is not running") that safely() cannot
trap, terminating the whole runner test. Every snapshot and alert
resolution then failed, and the CLI misreported it as the screen
"overwhelming the accessibility capture".
resolveBlockingSystemModal — the single chokepoint every snapshot/alert
path funnels through — assumed a SpringBoard host always exists. Model that
assumption explicitly with `hasSpringBoardSystemModalHost` and bail to
`.absent` when no host exists, so app-owned alert queries still run. The
fourth probe site (shouldRouteToSpringboardBlockingSystemModal) is already
`#if os(iOS)` and needs no change.
Add tvOS regression tests covering the snapshot and alert-resolution paths.
Move the scattered root-level native projects into per-platform folders and drop
the now-redundant platform prefix:
- android-ime-helper/ -> android/ime-helper/
- android-multitouch-helper/ -> android/multitouch-helper/
- android-snapshot-helper/ -> android/snapshot-helper/
- apple-runner/ -> apple/runner/
- macos-helper/ -> apple/macos-helper/
- src/platforms/linux/atspi-dump.py -> linux/atspi-dump.py
Only repo source paths move. Identity surfaces stay frozen so no user's runner
cache is invalidated on upgrade: the derived-cache key hashes source paths
relative to AgentDeviceRunner and excludes packageVersion, and the
~/.agent-device/{apple-runner,macos-helper} namespaces, the
agent-device-android-*-helper artifact/manifest/protocol names, the
AgentDeviceRunner Xcode project, and the `prepare ios-runner` CLI command are
unchanged. Updates build/package scripts, CI, package.json files+scripts,
ignore/attr/fallow configs, runtime path resolvers, and test fixtures.
Also: re-base repo-root-relative refs inside the moved apple/runner for the
added nesting level (gated XCUITest fixture walk + two doc links), and clean the
legacy dist/apple-runner packaged output so the relocated runner can't
double-ship into the wholesale-included dist (with a regression test).
Michał Pierzchała