callstack__agent-device

skills/callstack__agent-device

Fork 0

mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Commit Graph

Author	SHA1	Message	Date
Michał Pierzchała	294654a3a5	fix(android): resolve snapshot scope once and disclose the API 23 occlusion-scan gap (#1832 C1/C2) (#1846 ) * fix(android): resolve snapshot scope once and disclose the API 23 occlusion-scan gap (#1832 C1/C2) - Android resolves --scope inside its projection only, under the shared scope specification (matchesSnapshotScope in @agent-device/contracts/snapshot: first document-order match over label/value/identifier, empty on no match). The daemon post-wire scopeSnapshotNodes pass skips the android backend, so scope has one owner and one no-match semantics instead of BFS+fallback followed by document-order+empty. - Golden table contracts/fixtures/snapshot-scope-policy.json is asserted against the predicate, the Android projection, and the daemon pass; the Swift runner twin (#1797) consumes the same table. - androidSnapshot.occlusionScanUnavailable discloses helper trees without drawing-order (API 23), where the covered-sibling pruner cannot run. Disclosure only; C1 stays open until occlusion moves to the daemon annotator. * fix(android): resolve scope over the presented tree and stop dropping it on interaction captures Adversarial review findings on the first commit: - BLOCKER: captureSnapshotData spread `snapshotScope: undefined` over flags, so an interaction capture (press/click/fill/longpress/hover --scope, --settle observation) reached the Android platform unscoped while buildSnapshotState still saw the scope. The post-wire pass used to rescue it; after skipping android it returned the unscoped tree. One effective scope now feeds both. - Scope resolves over the PRESENTED nodes of the requested projection, not the acquired tree, so an acquired match that membership drops no longer empties the snapshot, and Android matches the domain iOS's pass uses. - Slicing after the walk keeps ancestor context (hittable/collection/chrome) above the scope root, so scoped -i is a subset of unscoped -i; --depth stays scope-relative. - Shared findSnapshotScopeRange/reindexSnapshotNodes so the daemon pass and the Android projection run one implementation; scope slice extracted to ui-hierarchy-scope.ts (mirrors its test). - parseUiHierarchy moved to a test fixture module (it had no production caller left). - Golden rows sharpened (value row no longer matches via label on Android); the Android leg runs raw AND regular. CHANGELOG entry; docs wording corrected for iOS/@ref. * fix(layering): keep the contracts snapshot façade exhaustive over snapshot-scope * fix(android): scope to the first match whose subtree still has presented content Review P1 on #1846: with scope resolved strictly over presented nodes, `snapshot -i --scope panel` answered "no nodes" whenever the matched container was a structural view membership drops — even though the button inside it was exactly what was asked for — and `--depth 0` then hid a node the response prints at depth 0. The scope root is now the first document-order match whose subtree contributes at least one node to the requested projection, and the result is that subtree's presented nodes re-rooted at depth 0. Both failure modes die: a decorative match membership drops no longer empties the snapshot, and a dropped container still scopes to its content. `--depth` under scope filters the depths the response emits, so a node shown at depth 0 survives `--depth 0`. Tests: the golden legs stay raw+regular (bare TextViews cannot survive -i, so an -i leg would measure membership, not scope) with the projection interplay pinned by two dedicated tests on actionable shapes; the 'not re-scoped after the wire' case now runs a real parsed scoped tree instead of fabricated depth-0 siblings.	2026-08-18 18:44:13 +02:00
Michał Pierzchała	f843dc2df1	fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) (#1820 ) * fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) `pnpm gate replay-android` failed 4/8 whenever it ran after the full-tier Android E2E (replays-nightly run 32107665052, job 95620294899): 05-app-lifecycle, 06-swipe-gestures and both fixture replays diverged under "A system surface covers the app". The E2E was not the cause. Reproduced on a pixel_7 / API 36 AVD with the same cutout geometry CI's `avdmanager --device pixel_7` produces (status bar 136px, not the 63px of a plain 1080x2400 skin): - `03-scroll-discovery.ad` runs `scroll up 3`. The scroll planner clamps travel to the viewport minus a 5% band, so the touch-down landed at y=120 — inside the 136px status bar — and pulled the notification shade instead of scrolling. On API 36 the app window is edge-to-edge, so the reported viewport starts at y=0 and includes that bar. - The shade then covered every replay until `04`'s `back` closed it. Native readdir order on the runner (03, 05, 06, fixture/02, fixture/01, 04, 01, 02) put four files in that window; the last green run (2026-07-30) had 04 right after 03, so the pull was masked. Fix in the product, not the lane: DEFAULT_EDGE_PADDING_FRACTION 0.05 -> 0.1 in the TS scroll planner and its Swift port. Every real Pixel has a cutout (5.7% of a Pixel 7's height) and an iPhone's Dynamic Island status bar is 6.9%, so any saturated `scroll up` opened the shade / Notification Center for real agents too. Parity vectors updated in both suites plus a Pixel 7 regression vector (1080x2400, amount 3 -> touch-down y=240 > 136). Second contamination the same order exposed once the shade was gone: `fixture/02-selector-routes-covered-diagnosis.ad` is a #1715 reproduction recipe that FAILS BY DESIGN at step 9 (covered-target refusal) and leaves the device in landscape, yet the gate enumerated `test/integration/replays/android` recursively. iOS keeps gate replays in `replays/ios/simulator` and fixture recipes in `replays/ios/fixture`; Android now mirrors that: the six Settings replays move to `replays/android/emulator`, `test:replay:android` points there, and `fixture/` stays E2E-owned (`full:fixture-replays` already runs 01 by path). android.yml and the workflow-evidence fixture follow the path; the replay-compat manifest keeps the historical paths it pins at released tags. Verified live (Pixel 7 geometry, API 36, --retries 0): control run at main head in CI order reproduces exactly CI's 4/8; with the fix, `pnpm gate replay-android` 6/6 in both native and CI order, and `03` leaves Settings on screen (scroll up 3 now touches down at y=240). * test(scroll): drive the TS and Swift scroll-plan parity vectors from one table (#1820 review) The two suites hand-mirrored the same vectors and #1820 had to edit both by hand — the drift class the repo already closes for the tap-point rule via contracts/fixtures/tap-point-policy.json. The scroll vectors (plus both planner constants, pinned behaviourally on a 1000px axis) now live in contracts/fixtures/scroll-gesture.json; scroll-gesture.test.ts and RunnerTests+ScrollGesture.swift iterate it. Verified: vitest 10/10; the four XCTests run on an iOS 26.2 simulator with the unit flag on (Executed 4 tests, 0 failures). Also: test/ci/android-workflow-evidence.json says what it guards. Follow-up for content-safe viewport bounds + discovery order: #1821.	2026-08-18 14:31:54 +02:00
Michał Pierzchała	415f599c3b	feat: golden tap-point parity table for TS and the iOS runner (ADR 0011) (#1086 ) ADR 0011 Layer 2: the offscreen tap-point rule now has exactly one home per side of the wire, proven equivalent by a shared golden fixture table. - contracts/fixtures/tap-point-policy.json: 12 cases including this week's real bug shapes (closed drawer fully left, 0.07pt edge-grazing container) plus edge-inclusive and empty-frame fail-open semantics. - Swift: pure TapPointPolicy.isAllowed(elementFrame:windowFrame:) extracted; the ELEMENT_OFFSCREEN guard (onScreenWindowFrame stays the frame getter) and hasTappableFrame (Maestro fallback) both delegate the decision to it. Gated XCTest asserts every table case (runs in the existing AGENT_DEVICE_RUNNER_UNIT_TESTS surface CI already compiles). - TS: isTapPointInsideViewport extracted from isNodeVisibleOnScreen (no duplicated math); vitest parity test asserts the same table. - Registry: parityTable wired on direct-ios-selector/offscreen and maestro-non-hittable-fallback/offscreen.	2026-07-04 15:11:55 +02:00

Author

SHA1

Message

Date

Michał Pierzchała

294654a3a5

fix(android): resolve snapshot scope once and disclose the API 23 occlusion-scan gap (#1832 C1/C2) (#1846 )

* fix(android): resolve snapshot scope once and disclose the API 23 occlusion-scan gap (#1832 C1/C2)

- Android resolves --scope inside its projection only, under the shared scope specification
(matchesSnapshotScope in @agent-device/contracts/snapshot: first document-order match over
label/value/identifier, empty on no match). The daemon post-wire scopeSnapshotNodes pass skips
the android backend, so scope has one owner and one no-match semantics instead of BFS+fallback
followed by document-order+empty.
- Golden table contracts/fixtures/snapshot-scope-policy.json is asserted against the predicate,
the Android projection, and the daemon pass; the Swift runner twin (#1797) consumes the same table.
- androidSnapshot.occlusionScanUnavailable discloses helper trees without drawing-order (API 23),
where the covered-sibling pruner cannot run. Disclosure only; C1 stays open until occlusion moves
to the daemon annotator.

* fix(android): resolve scope over the presented tree and stop dropping it on interaction captures

Adversarial review findings on the first commit:

- BLOCKER: captureSnapshotData spread `snapshotScope: undefined` over flags, so an interaction
capture (press/click/fill/longpress/hover --scope, --settle observation) reached the Android
platform unscoped while buildSnapshotState still saw the scope. The post-wire pass used to rescue
it; after skipping android it returned the unscoped tree. One effective scope now feeds both.
- Scope resolves over the PRESENTED nodes of the requested projection, not the acquired tree, so an
acquired match that membership drops no longer empties the snapshot, and Android matches the
domain iOS's pass uses.
- Slicing after the walk keeps ancestor context (hittable/collection/chrome) above the scope root,
so scoped -i is a subset of unscoped -i; --depth stays scope-relative.
- Shared findSnapshotScopeRange/reindexSnapshotNodes so the daemon pass and the Android projection
run one implementation; scope slice extracted to ui-hierarchy-scope.ts (mirrors its test).
- parseUiHierarchy moved to a test fixture module (it had no production caller left).
- Golden rows sharpened (value row no longer matches via label on Android); the Android leg runs
raw AND regular. CHANGELOG entry; docs wording corrected for iOS/@ref.

* fix(layering): keep the contracts snapshot façade exhaustive over snapshot-scope

* fix(android): scope to the first match whose subtree still has presented content

Review P1 on #1846: with scope resolved strictly over presented nodes, `snapshot -i --scope panel`
answered "no nodes" whenever the matched container was a structural view membership drops — even
though the button inside it was exactly what was asked for — and `--depth 0` then hid a node the
response prints at depth 0.

The scope root is now the first document-order match whose subtree contributes at least one node to
the requested projection, and the result is that subtree's presented nodes re-rooted at depth 0.
Both failure modes die: a decorative match membership drops no longer empties the snapshot, and a
dropped container still scopes to its content. `--depth` under scope filters the depths the
response emits, so a node shown at depth 0 survives `--depth 0`.

Tests: the golden legs stay raw+regular (bare TextViews cannot survive -i, so an -i leg would
measure membership, not scope) with the projection interplay pinned by two dedicated tests on
actionable shapes; the 'not re-scoped after the wire' case now runs a real parsed scoped tree
instead of fabricated depth-0 siblings.

2026-08-18 18:44:13 +02:00

Michał Pierzchała

f843dc2df1

fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) (#1820 )

* fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1)

`pnpm gate replay-android` failed 4/8 whenever it ran after the full-tier Android E2E
(replays-nightly run 32107665052, job 95620294899): 05-app-lifecycle, 06-swipe-gestures and
both fixture replays diverged under "A system surface covers the app". The E2E was not the
cause. Reproduced on a pixel_7 / API 36 AVD with the same cutout geometry CI's
`avdmanager --device pixel_7` produces (status bar 136px, not the 63px of a plain 1080x2400
skin):

- `03-scroll-discovery.ad` runs `scroll up 3`. The scroll planner clamps travel to the viewport
minus a 5% band, so the touch-down landed at y=120 — inside the 136px status bar — and
pulled the notification shade instead of scrolling. On API 36 the app window is
edge-to-edge, so the reported viewport starts at y=0 and includes that bar.
- The shade then covered every replay until `04`'s `back` closed it. Native readdir order on
the runner (03, 05, 06, fixture/02, fixture/01, 04, 01, 02) put four files in that window;
the last green run (2026-07-30) had 04 right after 03, so the pull was masked.

Fix in the product, not the lane: DEFAULT_EDGE_PADDING_FRACTION 0.05 -> 0.1 in the TS scroll
planner and its Swift port. Every real Pixel has a cutout (5.7% of a Pixel 7's height) and an
iPhone's Dynamic Island status bar is 6.9%, so any saturated `scroll up` opened the shade /
Notification Center for real agents too. Parity vectors updated in both suites plus a Pixel 7
regression vector (1080x2400, amount 3 -> touch-down y=240 > 136).

Second contamination the same order exposed once the shade was gone:
`fixture/02-selector-routes-covered-diagnosis.ad` is a #1715 reproduction recipe that FAILS
BY DESIGN at step 9 (covered-target refusal) and leaves the device in landscape, yet the
gate enumerated `test/integration/replays/android` recursively. iOS keeps gate replays in
`replays/ios/simulator` and fixture recipes in `replays/ios/fixture`; Android now mirrors that:
the six Settings replays move to `replays/android/emulator`, `test:replay:android` points there,
and `fixture/` stays E2E-owned (`full:fixture-replays` already runs 01 by path). android.yml
and the workflow-evidence fixture follow the path; the replay-compat manifest keeps the
historical paths it pins at released tags.

Verified live (Pixel 7 geometry, API 36, --retries 0): control run at main head in CI order
reproduces exactly CI's 4/8; with the fix, `pnpm gate replay-android` 6/6 in both native and
CI order, and `03` leaves Settings on screen (scroll up 3 now touches down at y=240).

* test(scroll): drive the TS and Swift scroll-plan parity vectors from one table (#1820 review)

The two suites hand-mirrored the same vectors and #1820 had to edit both by hand — the drift
class the repo already closes for the tap-point rule via contracts/fixtures/tap-point-policy.json.
The scroll vectors (plus both planner constants, pinned behaviourally on a 1000px axis) now live in
contracts/fixtures/scroll-gesture.json; scroll-gesture.test.ts and RunnerTests+ScrollGesture.swift
iterate it. Verified: vitest 10/10; the four XCTests run on an iOS 26.2 simulator with the unit
flag on (Executed 4 tests, 0 failures).

Also: test/ci/android-workflow-evidence.json says what it guards.

Follow-up for content-safe viewport bounds + discovery order: #1821.

2026-08-18 14:31:54 +02:00

Michał Pierzchała

415f599c3b

feat: golden tap-point parity table for TS and the iOS runner (ADR 0011) (#1086 )

ADR 0011 Layer 2: the offscreen tap-point rule now has exactly one home per
side of the wire, proven equivalent by a shared golden fixture table.

- contracts/fixtures/tap-point-policy.json: 12 cases including this week's
  real bug shapes (closed drawer fully left, 0.07pt edge-grazing container)
  plus edge-inclusive and empty-frame fail-open semantics.
- Swift: pure TapPointPolicy.isAllowed(elementFrame:windowFrame:) extracted;
  the ELEMENT_OFFSCREEN guard (onScreenWindowFrame stays the frame getter)
  and hasTappableFrame (Maestro fallback) both delegate the decision to it.
  Gated XCTest asserts every table case (runs in the existing
  AGENT_DEVICE_RUNNER_UNIT_TESTS surface CI already compiles).
- TS: isTapPointInsideViewport extracted from isNodeVisibleOnScreen (no
  duplicated math); vitest parity test asserts the same table.
- Registry: parityTable wired on direct-ios-selector/offscreen and
  maestro-non-hittable-fallback/offscreen.

2026-07-04 15:11:55 +02:00

3 Commits