mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

Michał Pierzchała fa8cce37d1 refactor(ios): snapshot capture plans with a structured quality verdict (#783 )

* refactor(ios): snapshot capture plans with a structured quality verdict

Implements ADR 0004's explicit-strategies decision as architecture
(candidates 1+2 of the snapshot pipeline review):

- Snapshot backend seam: three adapters (recursive tree, query sweep,
  private AX) behind one captureWithBackend dispatch. Each strategy
  declares its chain as data (regular: tree→queries→private-ax,
  compact: queries→private-ax, raw: tree→private-ax) and one plan
  runner walks it under a 20s umbrella budget so chained recovery can
  never stack past the 30s main-thread watchdog. Terminal policy is
  per-plan: raw rethrows AX failures (diagnostics preserve errors),
  interactive fails closed with runnerFatal invalidation.
- Single quality classifier: one sparsePayloadReason predicate (with
  reason codes), one collapsed-leaf detector, replacing the three
  divergent sparse detectors (Swift structural, daemon count==1, CLI
  count<=3) that each patched a different failure shape.
- Structured snapshot quality verdict on the wire (state, backend,
  reason, reasonCode, effectiveDepth, collapsedLeafIndexes): the daemon
  and CLI render warnings from it instead of re-deriving degradation
  from node shapes; budget starvation is no longer blamed on the app's
  accessibility. Legacy runner messages and daemon-side detectors stay
  behind a verdict-absent gate for mixed-version compat.
- The verdict surfaces in --json (snapshotQuality) for agents; the
  generic sparse CLI hint is suppressed when a verdict explains it.

Threading the verdict exposed two more hand-copy field drops
(captureInteractionOutcomeAwareSnapshot, serializeSnapshotResult,
client response mapping) - now carried alongside warnings everywhere.

Verified live: Settings healthy (tree, no warnings), Settings compact
under load (recovered/private-ax/budget), production login (sparse
best-effort with honest warning), collapse fixture (healthy +
collapsedLeafIndexes -> @ref warning), Bluesky Home (recovered/
private-ax, 24 nodes in 2s). Full unit suite 2327 passed, fallow clean,
runner builds.

* fix(ios): correct recovered-snapshot viewport and private-AX scope semantics

Review follow-ups on the capture-plan refactor:

- The query-sweep synthetic root doubles as the daemon's viewport
  (find.ts prefers on-screen matches inside nodes[0].rect), but it was
  built from candidate bounds, so off-screen controls below the screen
  could inflate it and win duplicate-label resolution. The root now
  uses the real finite viewport, falling back to candidate bounds only
  when viewport capture failed.

- The private-AX backend applied --scope as a per-node text filter,
  hiding the matched container's children — diverging from regular
  snapshot scope semantics and contradicting the depth-cap hint that
  recommends scoped re-runs. Scope now selects the matched subtree:
  descendants inherit the match and only the normal option filters
  apply to them (in-bundle test covers a non-matching descendant).

Verified live on Bluesky Home: scope homeScreen returns the 52-node
subtree including non-matching descendants; compact root rect equals
the screen (0,0,402,874).

* fix(ios): fail closed on interactive AX failure, stamp fatal verdict, validate parser

Three review findings on the capture-plan terminal path:

- P1: the fail-closed guard required `best == nil`, but the query-sweep
  tier always returns a synthetic-root sparse payload that sets `best` —
  so an interactive recursive-tree AX serialization failure that no
  backend recovered returned a sparse snapshot instead of invalidating
  the cached target. Reaching the terminal already means no backend
  produced a usable tree, so the sparse `best` must not suppress the
  fail-closed path. Extracted the decision into a pure, unit-tested
  `resolveSnapshotPlanTerminal` (closes the terminal-ordering testability
  gap the architecture review flagged).
- P2: `snapshotAccessibilityUnavailable` returned a payload with no
  `snapshotQuality`, leaving one planned sparse result on the
  legacy-message path. It now carries a sparse/ax-rejected verdict like
  every other planned snapshot, so downstream sparse handling keys off
  the verdict.
- P2: `readSnapshotQualityVerdict` cast any string state/backend into the
  union, so a malformed object suppressed the legacy node-shape
  detectors. State and backend are now validated against their unions
  (unknown → verdict-absent → legacy detectors run); an unknown
  reasonCode is dropped rather than rejecting the whole verdict, so a
  forward-version runner still yields a usable verdict.

Unit-covered: Swift resolveSnapshotPlanTerminal matrix + fatal-verdict
assertion; TS parser accept/reject/forward-compat. Full suite 249 files
/ 2449 tests, fallow, lint, runner build green.

2026-06-12 12:25:40 +02:00

3.9 KiB

Raw Permalink Blame History

Agent Device Domain Context

Terms

Provider-backed integration scenario: device-free integration test that runs the real daemon request path and replaces only external device or host tool execution.
Provider: request-scoped adapter interface for external device, runner, or host tool execution.
Provider transcript: exact record of provider calls used when a test must verify platform command translation.
Scenario transcript: command-level integration flow that describes user-visible behavior through daemon commands.
In-process provider scenario harness: integration runner that invokes the daemon request handler directly without opening an HTTP listener.
HTTP contract test: narrow test that verifies JSON-RPC transport, auth, and response finalization over the daemon HTTP boundary.
Interactor: semantic interface between command dispatch and platform behavior.
Platform module: platform-specific implementation behind the Interactor.
Target: selected automation destination, such as mobile, tv, or desktop.
Modality: broad supported device family, such as mobile, tv, or desktop.
Session: daemon-owned state for a selected target and opened app or surface.
Command surface: catalog of public command identity, interface exposure, adapter policy, and shared command metadata across CLI, Node.js, MCP, and batch entrypoints.
Daemon command registry: daemon-side source of truth for command route ownership and request-policy traits, including admission exemptions, session locking, selector validation, replay-scoped actions, recording invalidation, Android dialog guards, and request provider device resolution.
Runner command traits: the iOS XCTest runner's per-command-type classification across three independent axes — interaction (gates the foreground-guard and stabilization preflight), read-only (gates the session-invalidating retry; the alert command is read-only only for its get action), and runner-lifecycle (skips the app-activation preflight). One source of truth keyed by command type, distinct from the public command surface and daemon command registry.
Coordinate-first resolved element activation: iOS/macOS runner interaction pattern where a selector or text query resolves the semantic XCUIElement, then activation uses the element's resolved center coordinate when a frame is available. This keeps target selection semantic while avoiding XCUIElement.tap() post-action element re-resolution after normal navigation. tvOS remains focus/remote-driven.
Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree, query sweep, private AX) run by one plan runner under a shared wall-clock budget; recovery ordering is declared data, never a per-call-site branch.
Snapshot quality verdict: structured outcome (state, backend, reason code, effective depth, collapsed leaves) computed once by the plan runner and shipped with every planned snapshot payload; the daemon and CLI render it instead of re-deriving degradation from node shapes.
AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot failure such as kAXErrorIllegalArgument marks the cached XCUIApplication target handle suspect. The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the next command reacquire the app through normal activation.

Testing Principles

Provider-backed integration scenarios should exercise the public daemon path whenever practical.
Prefer the in-process provider scenario harness for broad scenarios; keep HTTP contract tests narrow and transport-specific.
Provider seams sit below platform modules so integration tests still cover platform command translation.
Provider transcripts are for exact external command contracts.
Scenario transcripts are for broad, user-rooted workflows that should replace mocked handler unit tests.
Unit tests stay for pure logic, parser matrices, selector matching, capabilities, and important edge cases.

3.9 KiB Raw Permalink Blame History

Agent Device Domain Context

Terms

Testing Principles

3.9 KiB

Raw Permalink Blame History