mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

History

Michał Pierzchała f843dc2df1 fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) (#1820 )

* fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1)

`pnpm gate replay-android` failed 4/8 whenever it ran after the full-tier Android E2E
(replays-nightly run 32107665052, job 95620294899): 05-app-lifecycle, 06-swipe-gestures and
both fixture replays diverged under "A system surface covers the app". The E2E was not the
cause. Reproduced on a pixel_7 / API 36 AVD with the same cutout geometry CI's
`avdmanager --device pixel_7` produces (status bar 136px, not the 63px of a plain 1080x2400
skin):

- `03-scroll-discovery.ad` runs `scroll up 3`. The scroll planner clamps travel to the viewport
minus a 5% band, so the touch-down landed at y=120 — inside the 136px status bar — and
pulled the notification shade instead of scrolling. On API 36 the app window is
edge-to-edge, so the reported viewport starts at y=0 and includes that bar.
- The shade then covered every replay until `04`'s `back` closed it. Native readdir order on
the runner (03, 05, 06, fixture/02, fixture/01, 04, 01, 02) put four files in that window;
the last green run (2026-07-30) had 04 right after 03, so the pull was masked.

Fix in the product, not the lane: DEFAULT_EDGE_PADDING_FRACTION 0.05 -> 0.1 in the TS scroll
planner and its Swift port. Every real Pixel has a cutout (5.7% of a Pixel 7's height) and an
iPhone's Dynamic Island status bar is 6.9%, so any saturated `scroll up` opened the shade /
Notification Center for real agents too. Parity vectors updated in both suites plus a Pixel 7
regression vector (1080x2400, amount 3 -> touch-down y=240 > 136).

Second contamination the same order exposed once the shade was gone:
`fixture/02-selector-routes-covered-diagnosis.ad` is a #1715 reproduction recipe that FAILS
BY DESIGN at step 9 (covered-target refusal) and leaves the device in landscape, yet the
gate enumerated `test/integration/replays/android` recursively. iOS keeps gate replays in
`replays/ios/simulator` and fixture recipes in `replays/ios/fixture`; Android now mirrors that:
the six Settings replays move to `replays/android/emulator`, `test:replay:android` points there,
and `fixture/` stays E2E-owned (`full:fixture-replays` already runs 01 by path). android.yml
and the workflow-evidence fixture follow the path; the replay-compat manifest keeps the
historical paths it pins at released tags.

Verified live (Pixel 7 geometry, API 36, --retries 0): control run at main head in CI order
reproduces exactly CI's 4/8; with the fix, `pnpm gate replay-android` 6/6 in both native and
CI order, and `03` leaves Settings on screen (scroll up 3 now touches down at y=240).

* test(scroll): drive the TS and Swift scroll-plan parity vectors from one table (#1820 review)

The two suites hand-mirrored the same vectors and #1820 had to edit both by hand — the drift
class the repo already closes for the tap-point rule via contracts/fixtures/tap-point-policy.json.
The scroll vectors (plus both planner constants, pinned behaviourally on a 1000px axis) now live in
contracts/fixtures/scroll-gesture.json; scroll-gesture.test.ts and RunnerTests+ScrollGesture.swift
iterate it. Verified: vitest 10/10; the four XCTests run on an iOS 26.2 simulator with the unit
flag on (Executed 4 tests, 0 failures).

Also: test/ci/android-workflow-evidence.json says what it guards.

Follow-up for content-safe viewport bounds + discovery order: #1821.

2026-08-18 14:31:54 +02:00

AgentDeviceRunner

fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) (#1820 )

2026-08-18 14:31:54 +02:00

README.md

refactor: colocate native platform sources under android/, apple/, linux/ (#1273 )

2026-07-15 21:47:38 +02:00

RUNNER_PROTOCOL.md

feat(ios): expose AX custom actions on merged accessibility elements (#1665 )

2026-08-07 17:35:48 +02:00

README.md

agent-device iOS Runner

This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.

Intent

Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
Allow local builds via xcodebuild and caching for faster subsequent runs.
Support simulator prebuilds where compatible.

Status

Current internal runner for iOS, tvOS, and macOS desktop automation.

Protocol and maintenance references:

Protocol overview: RUNNER_PROTOCOL.md
TypeScript client: ../../src/platforms/apple/core/runner/runner-client.ts
Swift wire models: AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift

UITest Runner File Map

AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.

RunnerTests.swift: shared state/constants, setUp(), and testCommand() entry flow.
RunnerTests+Models.swift: wire protocol models (Command, Response, snapshot payload models).
RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).
RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.
RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.
RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.
RunnerTests+Interaction.swift: tap/drag/swipe/type/home/rotate/app-switcher helpers.
RunnerTests+Navigation.swift: back/navigation-control helpers.
RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.
RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.
RunnerTests+ScreenRecorder.swift: nested ScreenRecorder implementation.

Snapshot Strategy

iOS snapshots have two explicit public capture modes:

full/raw snapshots use recursive XCTest snapshots for rich hierarchy and diagnostics;
interactive snapshots filter the same visible tree down to agent-facing refs.

Some iOS apps expose accessibility trees that lower-level AX services can inspect but XCTest cannot serialize reliably. In those cases interactive snapshots may return a sparse root quickly, while full snapshots preserve the XCTest error. A penalized simulator can recover through private AX; physical devices use a short XCTest probe because no non-XCTest semantic backend is available there. See ../../docs/adr/0004-ios-snapshot-backend-strategy.md for the backend boundary and future simulator AX-service direction.

Protocol Notes

The daemon posts JSON commands to POST /command on the runner's local HTTP listener.
The runner responds with a JSON envelope shaped as { ok, data?, error? }.
The protocol is internal to agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.