* fix: guide agents past iOS keyboard dismissal and get-text guesses
Tonight's benchmark leaderboard showed two recurring agent-UX misses:
keyboard dismiss failing after fill (5x, now the #1 failed command) and
get-text guessed as a command name (1x).
- iOS keyboardDismiss now returns a hint explaining the on-screen keyboard
does not block agent-device interactions, so agents should press the next
target directly instead of retrying dismiss, and use keyboard enter only
when submission is actually wanted.
- help manual-qa and help workflow recovery text no longer teach the false
"dismiss to unblock the target" pattern.
- The command-suggestion curated map now maps get-text/gettext/get_text to
the real get text command shape; the generic edit-distance fallback did
not produce any suggestion for these guesses.
* fix: soften keyboard guidance for genuinely covered targets
Review on #1173 flagged the unconditional "does not block" claim as false
for targets visually covered by the keyboard: direct-selector press
resolution is isHittable-gated and falls through to ELEMENT_NOT_FOUND, the
tree path allows non-hittable taps with only a no-visible-effect hint, and
bottom-pinned-button-under-keyboard (#291/#469/#957) is a real case where
dismissal was the remedy.
All three strings (runner hint, help manual-qa, help workflow) now say the
keyboard USUALLY does not block presses and name concrete fallbacks: scroll
the target into view, or keyboard enter when submission is wanted.
* fix: bound iOS capture stalls and make runner recovery session-preserving (#1105)
Runner (Swift):
- Coalesce duplicate transport sends of one commandId onto the in-flight
execution instead of enqueueing them again behind it (capture pileup).
- Fail fast with RUNNER_BUSY while watchdog-abandoned main-thread work is
draining; escalate to RUNNER_WEDGED past 120s so the daemon recycles.
- Carry the capture-plan deadline into the query-sweep and private-AX
ladder tiers so chained recovery cannot stack past the watchdog.
- Penalize the tree backend after a slow (>5s) or abandoned capture and
lead subsequent regular plans with private-AX for that bundle (sticky,
120s), stamped recovered/budget so the deferral stays observable.
Daemon (TS):
- Per-request runner recycle budget: at most one invalidate+reboot per
request, then fail fast with an actionable, session-preserving hint.
- RUNNER_WEDGED joins the runner-fatal invalidation reasons.
- Interaction commands (click/fill/longpress/press/type/get/is) preserve
the daemon on request timeout like snapshot/wait/find: resetting it
destroyed every healthy app session the daemon owned.
* fix: suppress AX-broken-screen snapshot issues so the runner survives capture
XCTest records 'Failed to get matching snapshot: kAXErrorIllegalArgument'
issues for every XCUIApplication query on AX-broken screens; after a few
of them the test case tears down the moment the in-flight command
completes, killing the long-lived runner after every capture of the
screen (the restart loop behind #1105). The capture plan already
classifies and recovers from AX failures, so this issue class is noise:
swallow exactly it in record(_:); everything else still records and
still drives XCTEST_RECORDED_FAILURE.
* feat: time-slice the XCTest tree capture on a worker thread
The tree snapshot XPC is a single blocking call whose duration moves
with live content (4s to minutes on Bluesky profile screens); no
in-process budget could bound it on the main thread. Run it on a worker
bounded to an 8s slice: on timeout the plan penalizes the tree backend,
skips the XCTest-backed tiers while the abandoned XPC drains (they
would block behind it inside testmanagerd), and recovers through the
private AX backend, which does not use testmanagerd.
* tune: lower the tree-backend penalty threshold to 3s
The Bluesky profile tree grind measures ~4.5s before kAXErrorIllegalArgument,
just under the old 5s threshold, so every capture re-paid the doomed grind
(9s each). At 3s the second capture onward defers to private AX (2.4s
snapshot, 4.9s press on the live repro).
* fix: harden the AX-issue suppression per review
- Require the kAXError token: 'Failed to get matching snapshot: Timed out
while evaluating UI query.' is a genuinely-hung-query signal and must
keep recording (and keep driving XCTEST_RECORDED_FAILURE). Sibling AX
server codes (kAXErrorCannotComplete, ...) are deliberately included:
any AX-server rejection inside a matching-snapshot fetch is the same
capture-plan noise.
- State honestly that the override is suite-global and why (tap-triggered
queries record the same noise; command outcomes stay honest via their
own error paths).
- Lock-guarded suppressed-issue counter following the file's existing
abandoned-work counter pattern, logged with each suppression.
- Unit-test the pure classifier (record(_:) itself is not invoked: the
must-record variants would record real failures in the test run).
- Remove the runner .tap text arm: the daemon only ever sends
selectorKey/selectorValue or x/y taps (src/platforms/apple/interactions.ts),
and daemon+runner ship in lockstep. findElement(app:text:) stays for the
findText probe.
- Drop settledAfterMs from the wait stable result: it always equaled waitedMs
and never shipped in a release (v0.18.3 predates #1059).
- Add a loading hint when wait stable settles on a tree with fewer than 5
nodes (#1078): stability on a nearly-empty tree is a weak readiness signal.
Refs #1078
ADR 0011 Layer 2: the offscreen tap-point rule now has exactly one home per
side of the wire, proven equivalent by a shared golden fixture table.
- contracts/fixtures/tap-point-policy.json: 12 cases including this week's
real bug shapes (closed drawer fully left, 0.07pt edge-grazing container)
plus edge-inclusive and empty-frame fail-open semantics.
- Swift: pure TapPointPolicy.isAllowed(elementFrame:windowFrame:) extracted;
the ELEMENT_OFFSCREEN guard (onScreenWindowFrame stays the frame getter)
and hasTappableFrame (Maestro fallback) both delegate the decision to it.
Gated XCTest asserts every table case (runs in the existing
AGENT_DEVICE_RUNNER_UNIT_TESTS surface CI already compiles).
- TS: isTapPointInsideViewport extracted from isNodeVisibleOnScreen (no
duplicated math); vitest parity test asserts the same table.
- Registry: parityTable wired on direct-ios-selector/offscreen and
maestro-non-hittable-fallback/offscreen.
* fix: agent-UX fixes from Bluesky dogfooding
Four fixes found by driving the Bluesky dev build end to end as an agent:
1. wait honors its user-supplied budget in the daemon request envelope.
The wait timeout travels as a positional, so the client never extended
the request timeout: any 'wait ... 180000' died at the 90s default.
resolveDaemonRequestTimeoutMs now parses wait positionals (shared
src/core parser) and extends the envelope to budget + 30s margin.
2. wait/find timeouts no longer reset the daemon. They are repeated
snapshot captures, sharing snapshot's stalled-bridge failure mode, but
sat on the daemon-reset path — one timed-out wait destroyed every
session the daemon owned (observed live: wait timeout -> runner
killed -> SESSION_NOT_FOUND on the next command).
3. Off-screen selector targets are refused instead of silently tapped.
'tap label=Explore' against Bluesky's closed drawer reported success
while tapping (-161, 265) — coordinates that cannot land:
- selector disambiguation now prefers candidates whose CENTER lies in
the root viewport (isNodeVisibleOnScreen; edge-grazing containers
poke fractions of a pixel into the viewport and must not count),
so 'press label=Profile' picks the visible tab over the drawer item;
- selector-resolved interactions get the same off-screen guard refs
already had (reason: offscreen_selector, with a scroll/open hint);
- the runner's direct selector tap refuses matches whose center is
outside the main window frame (app.frame unions transformed
subtrees, so a closed drawer inflates it) with ELEMENT_OFFSCREEN,
which falls back to the tree-based path.
4. The non-hittable 'may have had no visible effect' hint is dropped when
--verify evidence proves the interactive tree changed — the warning
sat directly next to data contradicting it.
Live-verified against Bluesky on the iPhone 17 Pro simulator: offscreen
tap -> offscreen_selector error; ambiguous Profile -> on-screen tab
(355, 817); 100s wait survives the old 90s envelope with the session
intact.
* refactor: dedupe resolved-target tails and off-screen guards, decompose disambiguation
Fallow follow-up on the dogfood fixes: extract the shared ref/selector
resolved-node tail (describeResolvedInteractionNode), the shared
off-screen check-and-throw skeleton (throwIfOffscreenInteractionTarget —
per-caller messages/hints preserved), and the candidate-accumulation
step out of analyzeSelectorMatches. No behavior change; suites and the
fallow gate are green.
Michał Pierzchała