mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
v0.20.1
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Michał Pierzchała
|
9b610fbd1e |
feat(replay): recorded landmark identity for wait, is coverage — read-only step identity (#1349) (#1381)
* refactor(replay): extract shared target-evidence tree helpers into src/replay Move buildIndexMap/buildAncestryChain/filterIdentitySet out of the daemon's session-target-evidence into the shared replay zone so the commands runtime (wait's polling loop, #1349) can consume them without importing the daemon; press-retarget drops its private buildIndexMap duplicate. * feat(replay): recorded landmark identity verification for wait, get-pattern coverage for is (#1349) - New CommandDescriptor trait targetIdentityVerification pins the evidence-carrying command set and routes wait to a post-resolution phase so an annotated wait never enters the generic pre-dispatch verification (an absent landmark is its expected starting condition). - wait <selector> records landmark-mode target-v1 evidence (existence self-check; identity-empty matches record no annotation) and, on replay, keeps polling until a selector match carries the recorded identity; a deadline with only impostor matches fails closed as an identity-mismatch REPLAY_DIVERGENCE, a recorded-unverifiable annotation refuses before polling, and a plain timeout stays an action-failure divergence. - is (except exists) joins the get pattern: evidence at record time, generic pre-dispatch verification, and the post-resolution guard threaded through dispatch; direct-iOS fast paths for wait/is are gated during recording and guarded replays. - Read-only find stays intentionally unannotated (fuzzy-locator resolution has no selector-chain identity token), proven by test. * feat(publication): destination guard requires verified recorded landmark identity (#1349) A qualifying ADR 0016 guard is now a selector wait whose target-v1 annotation is verified; identity-less or unverifiable guards are refused with a recovery hint. Adds the reshuffled-screen false-pass regression: record -> publish -> replay against a same-label/different-ancestry tree diverges as identity-mismatch (matchCount >= 1 proving the selector alone would have false-passed). * refactor(replay): dedupe post-dispatch identity-mismatch shaping, trim evidence-writer complexity Shared buildPostDispatchIdentityMismatchResponse behind the guard and wait-landmark conversions; extracted payload-ceiling helpers from computeTargetEvidence; identity-refusal conversion split out of resolveReplayStepResponse. Docs: ADR 0012 decision 3 amendment (#1349), ADR 0016 guard strengthening, help workflow/save-script text. * refactor(replay): make landmark evidence's record-time verification explicit, trim ADR-restating docs The landmark-mode self-check was provably a tautology (the winner is a member of its own identity set whenever the parent walk is intact), so a membership scan defended only by a comment is replaced with the explicit decision: broken walk fails closed, landmark is verified by construction, action mode keeps decision 3's step-5 self-check. Doc comments that re-argued the ADR amendment now state behavior and point to it. * chore: untrack multitouch-helper build artifacts, ignore its build/dist dirs Generated Android helper output swept into the earlier refactor commit by accident; analogous snapshot-helper/ime-helper build dirs were already ignored. * fix(interaction): wait polls ride out content-unreadable captures (live-validated on Android) Live ADR 0016 validation on a Pixel emulator showed a destination-guard wait replayed immediately after a navigation press deterministically dies: the first poll's capture lands mid-transition and the Android helper's 'insufficient foreground app content' verdict threw out of the polling loop. iOS already yields the same state as a sparse verdict with no matches, so the loop kept polling there — this makes wait semantics platform-consistent. A content-verdict capture failure (isUnreadableCaptureContentError) now counts as a no-match poll for selector and text waits; a wait whose screen never became readable rethrows the last capture verdict at the deadline, so persistent breakage keeps its diagnosis. Other capture failures still throw immediately. * fix(snapshot): narrow unreadable-capture classification to enumerated content verdicts Android stamps androidSnapshotHelperFailureReason on mechanism failures too (helper timeouts, adb failures, missing helper artifact — free-form reason strings), so matching any string made waits poll those to their deadline instead of failing immediately. The predicate now matches only the enumerated content-recovery reasons, and AndroidHelperContentRecoveryDecision derives its reason union from the same list so a new content verdict cannot miss the predicate. Adds the realistic wrapped mechanism-error regression the synthetic test missed. * test(interaction): make the wait mechanism-failure regressions revert-sensitive Assert exactly one capture attempt: the broad any-string classifier would poll the repeated fixture error to the fake-clock deadline and rethrow the same message, passing the message-only assertion. Verified the mechanism test fails against the broadened classifier and passes against the narrowed one. |
||
|
Michał Pierzchała
|
b2b7ddd0d0 |
chore: gitignore .env files (#1322)
The repository is public and the root `.env` was untracked but matched no ignore rule, so it showed up in `git status` and any `git add .` or `git add -A` would have staged it — publishing whatever it holds to the repo and every fork. It has never been committed to any ref, so this is prevention rather than a leak; nothing needs rotating. `.env.*` subsumes the old `examples/test-app/.env.local` entry, which was the only dotenv rule here and covered a single path. The rules match basenames at any depth, so every variant is now covered wherever it appears. `!.env.example` keeps templates trackable; none exist today, but the negation is what makes the broad `.env.*` safe to add. |
||
|
Michał Pierzchała
|
1a1ef7c419 |
feat(android): one persistent automation helper owning snapshot + viewport + canonical injection (#1281)
* feat(android): consolidate touch injection and gesture viewport into the persistent snapshot helper (#1275)
One Android automation helper now owns snapshot capture, gesture viewport
resolution, and canonical one-/two-pointer plan injection. A live persistent
helper session executes gesture/viewport commands over its socket protocol;
without a session the same APK runs one-shot via am instrument. The separate
one-shot multitouch helper APK is deleted (atomic replacement, no fallback).
Touch scheduling/injection is extracted into focused Java classes
(TouchPlan, TouchPlanInjector, PointerEventSchedule, GestureViewportReader)
instead of growing SnapshotInstrumentation. ADR 0013 amended.
* fix(android): stop a structurally-failed helper session before the one-shot viewport retry
A structured ok=false viewport response leaves the session process alive, and
Android permits only one instrumentation owner of UiAutomation - running the
one-shot fallback against a still-live helper contends with it and masks the
original structured failure. Stop the session first; regression pins that the
one-shot retry only executes once the session is gone.
* refactor(android): extract helper touch dispatch into focused classes; split session tests; document helper API v2 (PR #1281 review)
Addresses findings 2 and 3 from PR #1281 review (finding 1, viewport
session-stop ordering, was already fixed in
|
||
|
Michał Pierzchała
|
e58cbcdb5f |
refactor: colocate native platform sources under android/, apple/, linux/ (#1273)
Move the scattered root-level native projects into per-platform folders and drop
the now-redundant platform prefix:
- android-ime-helper/ -> android/ime-helper/
- android-multitouch-helper/ -> android/multitouch-helper/
- android-snapshot-helper/ -> android/snapshot-helper/
- apple-runner/ -> apple/runner/
- macos-helper/ -> apple/macos-helper/
- src/platforms/linux/atspi-dump.py -> linux/atspi-dump.py
Only repo source paths move. Identity surfaces stay frozen so no user's runner
cache is invalidated on upgrade: the derived-cache key hashes source paths
relative to AgentDeviceRunner and excludes packageVersion, and the
~/.agent-device/{apple-runner,macos-helper} namespaces, the
agent-device-android-*-helper artifact/manifest/protocol names, the
AgentDeviceRunner Xcode project, and the `prepare ios-runner` CLI command are
unchanged. Updates build/package scripts, CI, package.json files+scripts,
ignore/attr/fallow configs, runtime path resolvers, and test fixtures.
Also: re-base repo-root-relative refs inside the moved apple/runner for the
added nesting level (gated XCUITest fixture walk + two doc links), and clean the
legacy dist/apple-runner packaged output so the relocated runner can't
double-ship into the wholesale-included dist (with a regression test).
|
||
|
Michał Pierzchała
|
d4146c7f1b |
feat: add Android test IME helper for deterministic text entry (#1198) (#1201)
* feat: add Android test IME helper for deterministic text entry (#1198) Ships a headless InputMethodService (android-ime-helper) as a third Android helper APK, replacing the visible system keyboard during automated sessions. Renders zero accessibility nodes and accepts Unicode/CJK/emoji text over a base64-encoded broadcast channel, fixing both the settle-diff IME-chrome flood and the ASCII-only adb-shell text entry limit in one structural fix. - android-ime-helper/: InputMethodService + build/package scripts on the existing helper-APK toolchain (javac+d8+aapt2+zipalign+apksigner). - src/platforms/android/ime-helper.ts, ime-lifecycle.ts: install/version lifecycle (shared with the other two helpers via the new helper-package-install.ts), activation on session open, and on-device restore-hygiene (previous IME persisted to a device settings key so any daemon/state-dir can recover it; restored on close, daemon teardown, and daemon startup for orphans left by a crashed run). - input-actions.ts: fill/type route through the helper's broadcast channel when active, unicode-safe; unchanged ASCII-shell fallback otherwise. - doctor: new android-test-ime check flags a stuck helper IME with a copy-pasteable `adb shell ime set` remediation command. - Gating: default-on for emulators, opt-in via `open --test-ime` on real devices. - Dead-weight: rewrote the manual ADBKeyBoard workaround doc, dropped the now-provably-live skillgym non-ASCII eval case, updated the ASCII fallback's error message to point at the helper instead of dead-ending. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * fix(#1201 review): permission-gate the IME receiver, fix CI, add opt-out Addresses the independent review's blockers and should-fixes. SECURITY (blocker 1): the text-injection receiver was RECEIVER_EXPORTED with no gate — any co-installed app could inject text into the focused field while the test IME was active. Fixed by requiring the WRITE_SECURE_SETTINGS sender permission on the (in-process, dynamically-registered) receiver: adb shell holds it, third-party apps cannot. The reviewer's suggested exported=false + explicit-component approach was tried first but empirically breaks delivery on API 36 (adb shell cannot reach a non-exported receiver there) — documented in the helper README. Live-verified: a purpose-built rogue APK's broadcasts (implicit and package-scoped, no permission) are silently dropped, field unchanged; adb shell's bare broadcast still injects. Added ime-helper-security.test.ts asserting the permission gate and that no permissionless exported registration returns. CI (blocker 2): (a) added `testIme` to integration-progress-model flag buckets (Integration Tests was red on the unclassified flag). (b) mocked resolveAndroidImeHelperArtifact in session-doctor-android / ime-lifecycle / input-actions-test-ime tests so they no longer depend on android-ime-helper/dist existing on disk (Coverage was red on a fresh checkout); verified by running them with dist removed. Should-fixes: added `--no-test-ime` to opt out on emulators (tri-state gating, parser-tested); PR body's "byte-identical" claim corrected to size/CRC-match. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * docs(#1201): pin the API-36 exported-receiver constraint in a comment The RECEIVER_EXPORTED flag cannot express why it must stay exported. Add a one-line note so a future hardening pass doesn't switch to RECEIVER_NOT_EXPORTED and silently break the CLI (adb shell can't deliver explicit broadcasts to non-exported components on API 36+; WRITE_SECURE_SETTINGS is the actual gate). 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * fix(#1201 review): harden IME restore lifecycle (blockers 1 & 2) Blocker 1 — a failed restore no longer deletes the recovery value. restore now reads back default_input_method after `ime set` and only clears the persisted previous-IME record on a confirmed-successful restore; a failed set keeps the value so a later retry / startup recovery / doctor remediation can still un-strand the user off the helper IME. Blocker 2 — startup orphan-recovery no longer overwrites/races user state. It only restores when the device's CURRENT default IME is still our helper (so a user who legitimately switched away is left alone), and skips any device a live session in this process owns (the fire-and-forget startup vs. concurrent `open` race — activate now marks the device active BEFORE the `ime set`, so any recovery pass that could observe the helper active also observes the flag and skips). Never persists the helper itself as the previous IME. activate also verifies its own switch via read-back. Exported ANDROID_IME_HELPER_SERVICE_COMPONENT so restore compares the active IME without reading the packaged artifact from disk. Tests: failed-restore keeps the value (+ later recovery succeeds), startup no-op when current != helper, startup skips a live-owned device. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * chore(#1201): delete unused ACTION_ENTER path, baseline test-only export seams Rebased onto main (#1202 production-unused-exports gate). Two follow-ups: - Deleted the unused ACTION_ENTER broadcast end-to-end (TS sendAndroidImeHelperEnter + its test, Java handler, README): nothing routes through it — `keyboard enter` uses the keyevent ENTER path — so the new production-exports gate flagged it as dead production code. Removed rather than grandfathered. - Added the three legitimate test-only seams (resetAndroidImeHelperInstallCache, resetAndroidTestImeActivationCacheForTests, setAndroidTestImeActiveForTests) to fallow-baselines/production-unused-exports.json, matching how the sibling helper reset functions (resetAndroidMultiTouchHelperInstallCache, ...) are already grandfathered there. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * fix(#1201): stop daemon-startup adb spawn on non-Android hosts (macOS Smoke) Root cause of the red macOS Smoke shard (proven, not hand-waved): the fire-and-forget restoreOrphanedAndroidTestImeOnDaemonStartup ran `adb devices` at EVERY daemon startup, on every platform. GitHub macOS runners ship the Android SDK, so this cold-started the adb server mid-replay and destabilized the macOS System Settings replay timing — the failed job's cleanup shows "Terminate orphan process: pid (N) (adb)"; main's green runs spawn no adb. Fix: gate the startup orphan scan behind a host-side marker written in the daemon state dir when a session activates the test IME (mirrors the managed-web-browser orphan-cleanup `installed` gate). A host that never uses the Android test IME — the macOS CI runner included — never writes the marker and so never spawns adb at startup. The marker is cleared once nothing is left stuck. Adds SessionStore.resolveStateDir(); tests: startup recovery does not scan adb when no marker exists (+ marker cleared after a clean scan). 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * chore(#1201): suppress fallow class-member false-positive on state-dir accessor CI's Fallow audit flags SessionStore.resolveDaemonStateDir as an unused class member, but it is called via sessionStore.resolveDaemonStateDir() in session-open.ts — fallow's class-member tracer just doesn't resolve a method call sited inside a call argument. Renamed for clarity (avoids the collision with config.ts's free resolveStateDir) and added the localized fallow-ignore-next-line unused-class-member suppression. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * fix(#1201 review): durable persist before switch + device-scoped recovery markers Addresses devin-ai-integration's two P1 restore-safety blockers on 19cbce79d. P1.1 — durably persist the restore target BEFORE the global IME switch. writePersistedPreviousIme now checks the `settings put` exit code AND reads the value back, returning a boolean. activate persists first and, if it cannot be persisted, fails open to the existing input path WITHOUT switching — a rejected `settings put` can no longer strand the user on the helper with no restore target. Regression test added. P1.2 — close the marker crash/offline blind spot. Recovery intent is now recorded per device, BEFORE the switch (ordering: durable record -> marker -> ime set), eliminating the post-switch/pre-marker crash window. Markers are device-scoped and each is retained until that device is actually observed clean: an offline/disconnected-but-stuck device keeps its marker and is recovered on reconnect instead of being cleared because the current `adb devices` scan saw no set-failed. Close-time restore clears only that device's marker (stateDir plumbed through teardown/close). Tests cover the persist-failure, post-switch/pre-marker crash, offline-then-reconnect, live-session-owned, and user-switched-away cases. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
4cd40aa621 |
feat: polish replay test progress reporter (#998)
* feat: polish replay test progress reporter * test: stabilize replay reporter cursor test in CI * refactor: dedupe replay reporter live progress checks * fix: make Expo build cache path configurable |
||
|
Michał Pierzchała
|
c32249503b |
test: cover least-tested parsing modules + verify public exports reach npm; drop dead daemon barrels (#749)
* test: cover least-tested CLI grammar and config parsing modules Add focused unit tests for the modules the coverage report flagged as least covered, exercising real behavior rather than padding metrics: - utils/source-value: env/config value parsing (booleans, enums, enum flags with setValue, int bounds, multiple) — 42% -> 100% lines - commands/cli-grammar/gesture: CLI<->daemon gesture argument translation for every gesture kind incl. error paths — 41% -> 100% - commands/cli-grammar/system: back/rotate/keyboard/clipboard/ react-native readers and writers incl. validation — 57% -> 100% - core/device-rotation: orientation parsing with aliases/errors — 100% - core/dispatch-payload: push payload loading from inline JSON and files, with temp-file I/O and JSON/shape error handling https://claude.ai/code/session_018i1mhcSe6sqM4mKLUKgYXe * chore: gitignore generated coverage report directory https://claude.ai/code/session_018i1mhcSe6sqM4mKLUKgYXe * test: verify public exports reach the npm build; drop dead daemon barrels Strengthen the package-exports test so it verifies the real publish-time invariant: every package.json "exports" subpath maps to a configured rslib build entry that points at an existing source module which actually exposes named exports. This catches a subpath being added to package.json without a matching build entry (which would ship a broken import), and keeps import/types targets in lockstep. Remove four internal re-export barrels under src/daemon that were pure 1:1 forwarders to their src/utils source, and repoint all importers at the real module so it is obvious where the code lives: - daemon/is-predicates.ts -> utils/selector-is-predicates.ts - daemon/selectors-build.ts -> utils/selector-build.ts - daemon/snapshot-diff.ts -> utils/snapshot-diff.ts - daemon/snapshot-processing.ts -> utils/snapshot-processing.ts The curated daemon/selectors.ts facade is kept; only blind forwarders were removed. No behavior change. https://claude.ai/code/session_018i1mhcSe6sqM4mKLUKgYXe --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
45cfad5cc5 |
feat: e2e command perf benchmark harness + nightly CI (#630)
* feat: add e2e command perf benchmark harness + nightly CI Adds scripts/perf, a cheap end-to-end perf benchmark that drives the built CLI through an ordered Settings tour of ~24 commands for N rounds, on a fully isolated daemon/state-dir and self-cleaning device, and emits JSON + Markdown reports. Per-command timing comes from wrapping each batchable command in its own single-step batch (daemon durationMs) plus wall-clock around the process. Wires a scheduled + workflow_dispatch CI job (perf-nightly.yml) that reuses the cached iOS XCUITest runner (setup-apple-replay) and the Android replay host, and runs the CLI from source via --experimental-strip-types (no dist build). * refactor(perf): drive the harness CLI via runCmdSync, not spawnSync Review (P2): repo rule is to spawn processes through src/utils/exec.ts, not node:child_process directly. Switch the perf harness's invokeCli to runCmdSync (allowFailure so non-zero exits are recorded as samples) and add a maxBuffer option to ExecOptions/runCmdSync (snapshot payloads exceed Node's ~1MB default). * perf(harness): warm the runner after open so the first measured command is clean The first interaction after open/relaunch pays the one-time iOS XCUITest runner startup (~10s+ cold) and a per-relaunch first-AX-query settle cost (~4s). That was landing on the first measured command each round (snapshot -i), inflating it ~10x vs the next snapshot. Run an untimed warmup snapshot -i after establishSession, after each round's reset-open, and after every freshRoot relaunch, so no measured command absorbs runner startup. Noted in the report header. * refactor(perf): address review + fix Fallow CI - exec.ts: extract spawnRejectionError + commandCloseFailure helpers, deduping the error/close handler clones (Fallow duplication ✗ that surfaced once the maxBuffer change pulled exec.ts into the audit scope). - .fallowrc: exclude scripts/perf/** (non-shipped benchmark tooling, like examples/ test-app) so its naturally-moderate functions don't trip the complexity gate. - config.ts: drop unused exports CLI_BIN/DEFAULT_OUT_DIR; add readIntValue so --n/--rounds/--warmup report the actual flag + reject non-integers clearly. - harness.ts: extract toSample(); type sampleError param as CliResult. - scenario.ts: ScenarioStep is now a discriminated union on execMode (removes step.step!/ step.args ?? []). - comment/legend rewords (platform defaults are local-convenience/CI-overridden; elements = node count). check:fallow now green; typecheck/lint/unit pass. * perf(harness): downgrade sample ok when a batch step reports ok:false Defensive belt-and-suspenders for the Codex review note: stop-only batch already surfaces a failed step as a top-level failure (caught by invokeCli), but if an on-error=continue mode ever keeps the batch ok while a step fails, don't silently count that step as a successful sample — derive ok from the step's own result.ok. |
||
|
Michał Pierzchała
|
47b981c8ad |
feat: add gesture command coverage (#576)
* feat: add gesture command coverage * fix: align iOS fling provider fixture * feat: group gesture commands * fix: clarify android gesture support * feat: add android multitouch gestures * fix: address gesture review feedback * refactor: simplify gesture plumbing * fix: keep gesture subcommands internal * fix: update iOS provider pan transcript |
||
|
Michał Pierzchała
|
d77a9211ab |
feat: add Android snapshot helper (#454)
* feat: add android snapshot helper * fix: harden android snapshot helper packaging * fix: harden android xml attribute parsing * fix: capture android helper window roots * fix: validate android helper install args * fix: harden android helper manifest and traversal * refactor: drop unused helper timeout parameter * feat: bundle android snapshot helper in npm package * feat: enable bundled android snapshot helper by default * refactor: simplify android snapshot helper resolution * fix: harden android snapshot helper artifacts * refactor: split android snapshot helper modules * fix: use type-only snapshot helper imports |
||
|
Michał Pierzchała
|
7c5b7670c8 | feat: add skillgym tests (#453) | ||
|
Michał Pierzchała
|
f07e82e308 |
chore: add fallow quality gate (#449)
* chore: add fallow quality gate * fix: refresh pnpm lockfile |
||
|
Michał Pierzchała
|
caf0e834b8 |
feat: add Linux desktop automation support via AT-SPI2 (#356)
* feat: add Linux desktop automation support via AT-SPI2 (Phase 1+2) Add Linux as a first-class platform using AT-SPI2 accessibility framework via node-gtk for accessibility tree snapshots. This mirrors the macOS desktop automation approach using accessibility snapshots. New files: - src/platforms/linux/atspi-bridge.ts: Core AT-SPI2 bridge using node-gtk with lazy loading, recursive tree traversal (max 1500 nodes, depth 12) - src/platforms/linux/role-map.ts: AT-SPI2 role normalization (~100 roles mapped to existing snapshot type conventions) - src/platforms/linux/snapshot.ts: Snapshot entry point with surface, scope, depth, and interactive-only filtering support - src/platforms/linux/devices.ts: Local device discovery for Linux - src/platforms/linux/node-gtk.d.ts: Type declarations for node-gtk Integration: - Extended Platform type with 'linux', backend union with 'linux-atspi' - Wired snapshot into dispatch.ts and snapshot-capture.ts - Added Linux device discovery to dispatch-resolve.ts - Added stub interactor (input actions deferred to Phase 3) - Added 'linux' to CLI --platform flag - node-gtk added as optional dependency (only installs on Linux) https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * refactor: code review cleanup for Linux platform support - Extract SnapshotBackend type alias to replace repeated string union across 5 files (snapshot.ts, snapshot-capture.ts, session-replay-heal.ts, interaction.test.ts) - Remove duplicate scope/interactive/depth filtering from linux/snapshot.ts — let the existing buildSnapshotState pipeline handle it, same as Android - Extract isDesktopBackend() helper in snapshot-capture.ts to consolidate the "skip mobile semantics" pattern for macos-helper and linux-atspi - Collapse 17 repetitive throw statements in Linux interactor stubs into a linuxStub() factory function https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: Linux input synthesis, screenshots, and app lifecycle (Phase 3+4) Add xdotool/ydotool input actions (tap, swipe, scroll, type, fill, right/middle click, long press, double click), screenshot capture via grim/scrot, and app lifecycle management (open, close, back, home). Wire Linux interactors with real implementations and fix device discovery order so Linux doesn't displace Android in auto-selection. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * refactor: consolidate Linux env detection, simplify input actions - Extract linux-env.ts with cached display server + input tool detection so every action avoids repeated `which` lookups - Add moveTo/clickButton/sendKey helpers to eliminate repeated mousemove boilerplate across 5 mouse actions - Make scrollLinux respect amount/pixels options instead of hardcoded scroll count - Have backLinux/homeLinux reuse sendKey instead of duplicating tool detection https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: add Linux CI smoke test with Xvfb and AT-SPI2 Add GitHub Actions workflow that boots a virtual X11 display (Xvfb), installs AT-SPI2 accessibility tooling and xdotool, opens gnome-calculator, takes screenshots, and captures an accessibility snapshot. Screenshots are uploaded as artifacts for visual verification. Also adds 'linux' to replay script metadata platforms and a test:replay:linux script to package.json. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: remove pre-session screenshot from Linux replay test The replay runner requires an active session before any commands can run. Move the screenshot after the open command that creates the session. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: Linux CI — add missing node-gtk build deps and AT-SPI2 env - Add gobject-introspection, libcairo2-dev, build-essential for node-gtk native compilation - Split AT-SPI2 registry start into its own step so it picks up DBUS_SESSION_BUS_ADDRESS from GITHUB_ENV - Set GTK_A11Y=atspi, GTK_MODULES=gail:atk-bridge, NO_AT_BRIDGE=0 to ensure GTK apps expose their accessibility tree on headless CI - Set GSETTINGS_BACKEND=memory to avoid dconf failures - Add node-gtk verification step to catch build failures early https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: explicitly rebuild node-gtk native module in Linux CI pnpm install silently skips failed optional dependency builds and the pnpm cache may not include the native binary. Force a rebuild after install to ensure the node-gtk .node binding is compiled against the system GI/cairo headers. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: use node-pre-gyp directly to build node-gtk from source pnpm rebuild doesn't trigger node-pre-gyp properly for optional deps. Run node-pre-gyp install --fallback-to-build --update-binary directly inside the node-gtk package directory to force compilation when no prebuilt binary exists for the current Node ABI (v127 / Node 22). https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * refactor: replace node-gtk with Python subprocess for AT-SPI2 node-gtk is a native C++ addon that requires compilation against specific Node ABI versions and GObject Introspection headers. This proved unreliable on CI (no prebuilt binaries for Node 22 ABI v127, silent optional dep build failures, pnpm cache staleness). Replace it with a Python helper script (atspi-dump.py) that uses PyGObject — the reference GObject Introspection consumer. python3-gi is trivially installable on any Linux distro with no compilation step. The Node bridge spawns `python3 atspi-dump.py` and parses JSON output. - Remove node-gtk from optionalDependencies - Remove node-gtk.d.ts type stub - Add atspi-dump.py (~200 lines) doing the same tree traversal - Rewrite atspi-bridge.ts to use subprocess instead of in-process GI - Simplify CI workflow: no more native build deps or rebuild steps https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: drop pre-installed packages from Linux CI apt-get python3-gi, gir1.2-atspi-2.0, at-spi2-core, and dbus-x11 are already present on Ubuntu GitHub Actions runners. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: surface support for Linux, unit tests, stronger CI assertions - Allow --surface desktop and --surface frontmost-app on Linux (previously only macOS could use --surface) - Add unit tests for atspi-bridge (9 tests: JSON parsing, role normalization, null coercion, error handling, arg forwarding) - Add unit tests for role-map (3 tests: common roles, case normalization, PascalCase fallback) - Improve .py script path resolution (walk upward instead of hardcoded relative paths) - CI replay test now asserts snapshot contains calculator UI nodes via is-exists https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * docs: add cross-platform snapshot traversal contract Document the shared schema, traversal rules, surface semantics, and normalized role types that all snapshot backends (Swift, Python, Android) must conform to. This serves as the single source of truth when adding or modifying platform backends. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: update lockfile after removing node-gtk optional dependency pnpm-lock.yaml still referenced node-gtk after it was removed from package.json, causing pnpm install --frozen-lockfile to fail in CI. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: address review findings in Linux platform code - atspi-dump.py: use ctx dict for traversal limits instead of globals, fix rect filter (width/height <= 0 should use `or`), add surface validation - input-actions.ts: make sendKey scancodes required to prevent silent no-op on ydotool, fix ydotool longPress/swipe to use click --down/--up - app-lifecycle.ts: use pkill -x (exact match) instead of pkill -f - linux-env.ts: emit diagnostic warning when falling back to xdotool on Wayland https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix(ci): explicitly install all Linux a11y dependencies Ubuntu runners may not have at-spi2-core, python3-gi, gir1.2-atspi-2.0, or dbus-x11 pre-installed. Install them explicitly instead of assuming they exist. Also make the verify step's tree dump non-fatal since no apps are running at that point. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: quote multi-word role value in Linux smoke test selector The selector parser tokenizes on whitespace, so `role=push button` was split into two tokens causing a parse failure. Use single quotes inside the selector: `role='push button'`. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: use valid selector keys in Linux smoke test appName is not a valid selector key. The supported keys are: id, role, text, label, value, visible, hidden, editable, selected, enabled, hittable. Simplified to use label and role only. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: cleanup pass — menubar warning, fix contract doc example - snapshot.ts: emit diagnostic warning when menubar surface is requested on Linux (falls back to desktop silently otherwise) - SNAPSHOT_CONTRACT.md: fix unmapped role example to use a role that isn't actually mapped (was "color chooser" which maps to Dialog) https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: add Python bytecache to gitignore https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: harden Linux platform — capability matrix, CI, error handling P0: Add explicit Linux capability matrix with 3-way platform routing (Apple/Linux/Android) in isCommandSupportedOnDevice. Linux now correctly blocks unsupported commands (clipboard, rotate, scrollIntoView, etc.) at capability level rather than throwing at runtime. Includes tests. P0: Expand Linux CI to run typecheck + unit tests before smoke tests. Add AT-SPI2 registry health probe with fail-fast on missing registry. P1: Harden atspi-dump.py — arg parsing now produces JSON errors on bad int values, and a top-level catch wraps unexpected exceptions in JSON. P1: Add 10s per-action timeout to xdotool/ydotool input commands to prevent indefinite hangs. P1: Tighten smoke test selectors to calculator-specific signals (digit labels) instead of generic role='push button'. P2: Document Linux surface mapping, supported commands, and known limitations in SNAPSHOT_CONTRACT.md. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: apply depth/interactive filtering to Linux snapshots Linux snapshots were bypassing snapshotInteractiveOnly and snapshotDepth filtering that macOS-helper gets via shapeDesktopSurfaceSnapshot. Route Linux through the same function so snapshot -i and --depth flags work. Renamed shapeMacOsSurfaceSnapshot → shapeDesktopSurfaceSnapshot since it's now shared between macOS and Linux desktop backends. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * fix: address review findings — error reporting, Wayland, timeout - app-lifecycle.ts: emit diagnostic on fire-and-forget app launch failure instead of silently swallowing errors - linux-env.ts: make xdotool on Wayland a hard error instead of a broken fallback (xdotool doesn't work on Wayland) - atspi-bridge.ts: increase Python subprocess timeout from 15s to 30s for safety on slow/loaded systems with large a11y trees https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * feat: appName/windowTitle selectors, clipboard, input-action tests Selectors: - Add appname and windowtitle as selector keys for desktop platforms. Both macOS and Linux snapshots already populate these fields — now they're usable in selector expressions (e.g., "label=OK appname=Calc"). Keys are case-insensitive. Clipboard: - Implement readLinuxClipboard/writeLinuxClipboard using xclip/xsel (X11) or wl-copy/wl-paste (Wayland) with descriptive TOOL_MISSING errors. Enable clipboard in Linux capability matrix. 7 unit tests. Input action tests: - Add 18 unit tests covering xdotool and ydotool code paths: press, right/middle click, double click, sendKey, type, scroll, swipe, focus, fill. Tests mock runCmd and verify correct tool + args. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT * chore: cache tool detection for screenshot/clipboard, extract get_app_info helper Avoid repeated `which` calls on every screenshot/clipboard operation by caching the resolved tool on first use, matching the input-action pattern. Extract duplicated app_name/pid retrieval in atspi-dump.py into get_app_info. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
d1d2f3852a |
refactor: migrate leaf modules from test-only DI to vitest mocking (#289)
* refactor: migrate leaf modules from test-only DI to vitest mocking Drop optional DI parameters from 6 production modules that existed solely as test seams. Replace with vi.mock() in new vitest test files. Production files simplified: - recording-telemetry.ts: remove writeTelemetry? - find.ts: remove dispatch?, use dispatchCommand directly - install-source.ts: remove entire deps? object (7 params) - session-runtime-command.ts: remove clearRuntimeHints? - android-system-dialog.ts: remove 4 DI params, simplify internals - record-trace-recording.ts: remove writeRecordingTelemetry from deps Closes #281 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore DI seams broken by premature removal, fix test script - request-router.ts: remove 4 dead DI params (snapshotAndroidUi, reopenAndroidApp, readAndroidAppState, execCommand) from RequestRouterDeps since recoverAndroidBlockingSystemDialog no longer accepts them - session-runtime-command.ts: restore clearRuntimeHints? param that handleSessionCommands threads through — removing it broke callers that override the seam - session.ts: restore clearRuntimeHints pass-through to runtime branch - session.test.ts: restore the end-to-end runtime clear DI test - package.json: include vitest in the default `test` script Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rename *.vitest.test.ts → *.vitest.ts to avoid node:test collision Node's default test discovery matches *.test.ts, which picked up vitest files and crashed. Rename to *.vitest.ts so the two runners have disjoint file sets without needing find-based exclusions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use vi.mocked().mock.calls instead of module-level mutable state Replace manual clearCalls array with direct assertion on the mock, consistent with the pattern used in request-router-android-modal.vitest.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
3ddd3a1ef5 |
Complete issue #39 boot diagnostics, telemetry, and boot command (#46)
* Complete issue #39 phase diagnostics telemetry and boot command * Address review findings for boot diagnostics and command gating * Use agent-device boot preflight in iOS CI workflow * Run iOS boot preflight via source CLI in CI |
||
|
Michał Pierzchała
|
fb53d72b63 | feat: AX snapshot goes brrrr | ||
|
Michał Pierzchała
|
4da4745a6c | initial commit |