* fix: add iOS private AX snapshot fallback
* fix: add public iOS snapshot query recovery
* fix(ios): make private AX snapshot fallback recover deep React Native trees
Four fixes that turn the #758 private AX fallback from
works-on-one-tree-shape into reliable on Bluesky Home:
- Depth ladder: the AX server rejects bulk snapshot requests outright
(kAXErrorIllegalArgument) once requested depth crosses a
tree-size-dependent limit that moves with live content. Retry at
56/40/24/12 instead of giving up after one attempt at 64.
- Real attribute identifiers: the server silently ignored the raw
keypath strings the bridge passed, so every node came back with a
zero frame (breaking ref taps and the interactive/compact filters,
which is why 'snapshot -i -c' stayed sparse). Map keypaths through
XCElementSnapshot.axAttributesForElementSnapshotKeyPaths (it returns
an NSSet) and drop the mapper's expensive extras (automation type,
window display id, base type) that pushed deep requests past the 30s
main-thread watchdog.
- Viewport from the private root frame when the public windows query
degrades to an infinite viewport, so off-screen drawer content stops
passing the visibility filter.
- Runner source fingerprint now includes .m/.h, so bridge edits stop
reusing stale cached runner builds.
Also hardens the bridge per review: UInt(exactly:) for untrusted
element types, pid_t-sized objc_msgSend for process id matching, and
objCType-checked NSValue frame decoding.
* fix(ios): recover deadline-truncated near-empty compact snapshots
The all-structural sparse detector misses the common large-RN-tree case
where the typed-query sweep resolves one or two stray controls before
its 1s deadline: the payload has 'content', so recovery never fires,
yet 2 nodes is useless in practice. Treat deadline-truncated payloads
with <= 8 nodes as needing recovery, and only replace the original
payload when the recovered tree actually carries more nodes. Completed
sweeps on legitimately minimal screens stay untouched (not truncated).
* chore: fix CI for the AX snapshot fallback branch
- Sync the setup metadata script's fingerprint extension list with the
runtime (.m/.h were added for the ObjC bridge), fixing the cache
metadata parity test.
- Reduce find.ts complexity flagged by fallow: hoist the node fetcher
into createFindNodeFetcher with a recoverSparseInteractiveSnapshot
helper, split match disambiguation and resolution scoring into
narrowMultipleMatches/resolvedTouchScore, extract rectsMatch.
* feat(ios): make accessibility fallbacks and collapsed containers visible in snapshot output
Two transparency gaps from #701's 'no silent fallback' requirement:
- Runner-attached snapshot messages now surface as snapshot warnings
(readAppleSnapshotResult previously dropped them), so every recovery
through the fallback accessibility backend or query tier is announced,
states what it usually means (the app publishes an unhealthy
accessibility tree - fixing the app is the real cure), and points to
screenshot as visual truth.
- A leaf whose label merges many comma-joined segments is flagged as a
collapsed accessible container: the app marks a container accessible,
hiding every descendant from assistive tech and automation alike.
Nothing can be recovered below it (VoiceOver sees the same merged
element), so the warning names the node, estimates the merged label
count, and gives the app-side fix plus the screenshot/coordinate-tap
workaround.
Validated live on the lab stress fixture (adlab://stress?accessible=1):
the 6-node tree now carries '@e5 [Other] merges ~126 labels...'.
* fix(ios): detect sparse trees with labeled roots and surface warnings through the daemon
Validated against a real-world repro (a production React Native app's
login screen, simulator build provided privately by the reporter): a
full-screen accessibilityViewIsModal overlay leaves the public snapshot
with just Application+Window. Two gaps kept recovery off:
- The sparse detector counted the Application label (the app's display
name) as content and the full-screen root as hittable, so the app
name alone defeated recovery. Application/Window labels and root
hittability say nothing about tree health and no longer count.
- Interactor-level snapshot warnings were dropped by the daemon capture
chain (only the runtime/commands layer kept them); they now thread
through CaptureSnapshotResult into BackendSnapshotResult.
With both fixes that login screen recovers through the public query
tier: 16 nodes with every control addressable (fill @ref + read-back
verified), and the output carries the recovery warning. Bluesky-class
trees still ladder into the private fallback unchanged.
* refactor(ios): consolidate series batching onto the sequence runner command
Closes#767
Routes every Apple multi-press variant (plain, double-tap, hold, jitter)
and swipe series through budget-chunked sequence requests, retiring the
daemon-side tapSeries and dragSeries senders:
- Add a doubleTap step kind to the sequence allowlist on both ends,
mirroring the retired tapSeries doubleTapAt branch.
- The single doubleTap interactor sends a one-step sequence and parses
the result, surfacing step failures as errors.
- Swipe series unroll ping-pong daemon-side into per-step endpoints;
the runner's coordinate-drag path ignores durationMs exactly as the
daemon-sent (non-synthesized) dragSeries did.
- Extract runIosSequenceChunks so press and swipe share the chunking,
aggregation, and global step-index rebasing.
- Keep tapSeries/dragSeries runner handlers for wire compatibility with
older daemons, annotated like interactionFrame; remove both from the
preflight-skip allowlist (daemon never sends them) and update ADR
0005 / protocol-optimizations docs.
This also closes the latent watchdog exposure where press --count N
--interval-ms M routed to tapSeries and executed all pauses inside one
30s-watchdog main-thread block with no chunking.
Behavior note: plain tap series now use the synthesized HID tap path on
iOS non-tv (with runner-side tapAt fallback), matching the individual
tap command instead of the retired tapSeries' XCUICoordinate taps.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
* refactor(ios): drop dead series wire surface from the daemon
- Remove chunkRunnerSequenceSteps: superseded by the budget-aware
chunker; no production callers remained.
- Remove tapSeries/dragSeries from the RunnerCommand union along with
their orphaned fields (count, intervalMs, doubleTap, pauseMs,
pattern) and protocol fixtures: this type is the send surface of the
current daemon, which no longer sends either command. The Swift
runner keeps serving both for wire compatibility with older daemons.
- Retarget the ready-mutation preflight test from tapSeries to
sequence.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
* refactor(ios): remove retired series and frame wire commands entirely
Drops the runner-side wire compatibility for tapSeries, dragSeries, and
interactionFrame now that no daemon path sends them (series fuse into
sequence since this branch; interactionFrame was fused into scroll in
#760):
- Swift: delete the three handler cases, performDragSeries, runSeries
(no remaining callers), the CommandType enum cases, journal-retention
and traits entries, and the Command fields (count, intervalMs,
doubleTap, pauseMs, pattern) that existed only for them. The
never-sent synthesized dragSeries branch goes with it.
- TS: drop interactionFrame from the RunnerCommand union and
isReadOnlyRunnerCommand, and its protocol fixture.
- Update stale perf scenario labels referencing the retired commands.
Verified dead before removal: no dynamic command construction anywhere
(runner-command-recovery only echoes in-flight command ids), no
raw-string references in Swift, no docs references. Helpers shared with
live paths (synthesizedDragAt, doubleTapAt, keyboardAvoidingDragPoints,
sleepFor) all retain callers.
Compat: an old daemon paired with a runner built from these sources
gets a CommandType decode rejection; the source-fingerprint check
rebuilds a matching runner on the next session.
https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo
---------
Co-authored-by: Claude <noreply@anthropic.com>
* chore(daemon): takeover notice, dev state-dir pruning, session state-dir command surface
Implements the three follow-ups from #737:
1. Print a one-line stderr notice when the client replaces a running
daemon, stating identity and reason (version mismatch, code-signature
mismatch, or unreachable). Best effort; never fails the command.
2. Add 'pnpm clean:daemon --prune-dev' to remove worktree-scoped state
dirs under ~/.agent-device/dev/ that no live daemon owns (same
pid/start-time liveness check as server-lifecycle) and that have been
idle for 14+ days. Scoped dirs only; one line printed per removal.
3. Fold 'session state-dir' into the regular command surface: the
session contract resolves it locally via the new
client.sessions.stateDir(), the cli.ts pre-dispatch special case is
removed, and the MCP session tool now exposes the state-dir action.
Closes#737https://claude.ai/code/session_013WBrUjQ4WRxRkfVruALKX3
* docs: surface clean:daemon --prune-dev in AGENTS.md
Local agents discover daemon state-dir hygiene through AGENTS.md, not
the website docs, so document the prune flag next to the existing
worktree-scoped state-dir guidance.
https://claude.ai/code/session_013WBrUjQ4WRxRkfVruALKX3
---------
Co-authored-by: Claude <noreply@anthropic.com>
* refactor(ios): single CommandTraits table for runner command classification
Replace the three hand-maintained switches in RunnerTests+Lifecycle.swift
(isInteractionCommand / isReadOnlyCommand / isRunnerLifecycleCommand) with one
source of truth: CommandType.traits, an exhaustive switch returning a
CommandTraits struct (interaction / readOnly / lifecycle axes), collocated with
CommandType in RunnerTests+Models.swift.
Pure refactor: every command's classification is reproduced verbatim, and the
three predicates become one-line lookups with unchanged signatures, so call
sites are untouched. The exhaustive switch makes it a compile error to add a
CommandType without classifying it, closing the drift that historically let
tapSeries/dragSeries/keyboardReturn fall out of isInteractionCommand.
readOnly is a 3-state enum (.always/.never/.conditional); .conditional preserves
alert's action-dependent read-only behavior, resolved in isReadOnlyCommand.
Classification feeds ADR-0002 session invalidation (the read-only retry that
nulls currentApp/currentBundleId), so behavior is intentionally unchanged.
Adds the "Runner command traits" term to CONTEXT.md.
* docs(ios): note CommandTraits.readOnly .conditional is alert-only (review follow-up)
* fix(ios): classify tapSeries/dragSeries/keyboardReturn as interaction commands (#643)
* fix(ios): classify tapSeries/dragSeries/keyboardReturn as interaction commands
tapSeries and dragSeries are the series forms of tap/drag (already interaction
commands); keyboardReturn is the sibling of keyboardDismiss (already an
interaction command). All three were missing from the historical
isInteractionCommand switch — a drift the new CommandTraits table (#642) makes
visible. Classifying them as interaction commands gives them the foreground-guard
+ stabilization preflight that their single-shot/sibling forms already get.
Behavior change: these three commands now re-activate a backgrounded target to
foreground and pay the stabilization delays before running. Ships separately from
the CommandTraits refactor (#642) and should land after that bakes.
mouseClick left unchanged: macOS-only and the foreground guard interacts with
bespoke macOS activation, so it needs a macOS smoke check first.
* test: cover iOS runner series commands in perf harness
* feat: add e2e command perf benchmark harness + nightly CI
Adds scripts/perf, a cheap end-to-end perf benchmark that drives the built
CLI through an ordered Settings tour of ~24 commands for N rounds, on a fully
isolated daemon/state-dir and self-cleaning device, and emits JSON + Markdown
reports. Per-command timing comes from wrapping each batchable command in its
own single-step batch (daemon durationMs) plus wall-clock around the process.
Wires a scheduled + workflow_dispatch CI job (perf-nightly.yml) that reuses the
cached iOS XCUITest runner (setup-apple-replay) and the Android replay host, and
runs the CLI from source via --experimental-strip-types (no dist build).
* refactor(perf): drive the harness CLI via runCmdSync, not spawnSync
Review (P2): repo rule is to spawn processes through src/utils/exec.ts, not
node:child_process directly. Switch the perf harness's invokeCli to runCmdSync
(allowFailure so non-zero exits are recorded as samples) and add a maxBuffer
option to ExecOptions/runCmdSync (snapshot payloads exceed Node's ~1MB default).
* perf(harness): warm the runner after open so the first measured command is clean
The first interaction after open/relaunch pays the one-time iOS XCUITest runner
startup (~10s+ cold) and a per-relaunch first-AX-query settle cost (~4s). That was
landing on the first measured command each round (snapshot -i), inflating it ~10x
vs the next snapshot. Run an untimed warmup snapshot -i after establishSession, after
each round's reset-open, and after every freshRoot relaunch, so no measured command
absorbs runner startup. Noted in the report header.
* refactor(perf): address review + fix Fallow CI
- exec.ts: extract spawnRejectionError + commandCloseFailure helpers, deduping the
error/close handler clones (Fallow duplication ✗ that surfaced once the maxBuffer
change pulled exec.ts into the audit scope).
- .fallowrc: exclude scripts/perf/** (non-shipped benchmark tooling, like examples/
test-app) so its naturally-moderate functions don't trip the complexity gate.
- config.ts: drop unused exports CLI_BIN/DEFAULT_OUT_DIR; add readIntValue so
--n/--rounds/--warmup report the actual flag + reject non-integers clearly.
- harness.ts: extract toSample(); type sampleError param as CliResult.
- scenario.ts: ScenarioStep is now a discriminated union on execMode (removes step.step!/
step.args ?? []).
- comment/legend rewords (platform defaults are local-convenience/CI-overridden;
elements = node count). check:fallow now green; typecheck/lint/unit pass.
* perf(harness): downgrade sample ok when a batch step reports ok:false
Defensive belt-and-suspenders for the Codex review note: stop-only batch already
surfaces a failed step as a top-level failure (caught by invokeCli), but if an
on-error=continue mode ever keeps the batch ok while a step fails, don't silently
count that step as a successful sample — derive ok from the step's own result.ok.
Michał Pierzchała