Michał Pierzchała
1a1ef7c419
* feat(android): consolidate touch injection and gesture viewport into the persistent snapshot helper (#1275)
One Android automation helper now owns snapshot capture, gesture viewport
resolution, and canonical one-/two-pointer plan injection. A live persistent
helper session executes gesture/viewport commands over its socket protocol;
without a session the same APK runs one-shot via am instrument. The separate
one-shot multitouch helper APK is deleted (atomic replacement, no fallback).
Touch scheduling/injection is extracted into focused Java classes
(TouchPlan, TouchPlanInjector, PointerEventSchedule, GestureViewportReader)
instead of growing SnapshotInstrumentation. ADR 0013 amended.
* fix(android): stop a structurally-failed helper session before the one-shot viewport retry
A structured ok=false viewport response leaves the session process alive, and
Android permits only one instrumentation owner of UiAutomation - running the
one-shot fallback against a still-live helper contends with it and masks the
original structured failure. Stop the session first; regression pins that the
one-shot retry only executes once the session is gone.
* refactor(android): extract helper touch dispatch into focused classes; split session tests; document helper API v2 (PR #1281 review)
Addresses findings 2 and 3 from PR #1281 review (finding 1, viewport
session-stop ordering, was already fixed in 5961b9247).
- Extract SnapshotInstrumentation.java's one-shot/session touch dispatch
into TouchCommandHandler.java (viewport/gesture population, UiAutomation-
parameterized) and SessionResponseWriter.java (session response encoding),
with shared PROTOCOL/HELPER_API_VERSION/OUTPUT_FORMAT constants moved to
a tiny HelperProtocol.java. SnapshotInstrumentation.java shrinks from 908
to 803 lines; wire format (header keys/values, error shapes) is unchanged.
- Split touch-helper.test.ts (~720 lines) into touch-helper.test.ts
(normalize/parse/one-shot gesture+viewport+result envelope) and
touch-helper-session.test.ts (persistent-session transport + fake-session
harness), moving shared device/plan/install-probe fixtures used by both
files into touch-helper.fixtures.ts.
- Update android/snapshot-helper/README.md to document helper API v2: the
one-shot viewport/gesture modes, the android-touch-plan-v1 payload shape,
and the persistent session's socket command/response contract.
* fix(android): invalidate helper session after APK replacement; recycle viewport windows; align ADR 0002 (PR #1281 re-review)
- prepareAndroidTouchHelper now mirrors the snapshot path: when
ensureAndroidSnapshotHelper replaces the APK (install.installed), the
persistent session started against the previous binary is stopped before
any touch command, so gestures run one-shot against the fresh install
instead of a dead/stale session socket. Regression: 'an APK replacement
stops the stale session and the gesture runs one-shot' drives a live fake
session through an outdated-install probe (new outdatedVersionAdb fixture)
and asserts the session socket receives no gesture, the one-shot
instrumentation path executes, and the session is gone.
- GestureViewportReader.read no longer leaks AccessibilityWindowInfo: a
single pass copies the active/focused and first-application bounds into
locals, every window is recycled in a finally, and the existing precedence
(active/focused app bounds, root-in-active-window, fallback app bounds,
IllegalStateException) is applied afterwards, unchanged.
- ADR 0002's touch-synthesis paragraph is amended (2026-07, issue #1275) to
the shared-helper model, consistent with ADR 0013: a live persistent
helper session executes touch commands directly, one-shot otherwise; the
old stop-before-gesture requirement is kept as historical context.
* fix(android): resolve touch helper artifact from the ADB provider like snapshots do (PR #1281 re-review)
- prepareAndroidTouchHelper now uses the same artifact precedence as
snapshot capture: the scoped adbProvider's snapshotHelperArtifact when
present, otherwise the bundled resolver (whose strict unavailable error
is preserved). The provider artifact drives both the install decision
and the instrumentationRunner used for one-shot commands, so an
ADB-backed provider that supplies a helper artifact but no native touch
override runs snapshots and gestures against the same single helper
(issue #1275). Regression: 'a provider-supplied snapshotHelperArtifact
overrides the bundled artifact for touch' pins the provider packageName
on the install probe, the provider apkPath on the install call, the
provider instrumentationRunner on the am instrument args, and that the
bundled resolver is never invoked.
- ADR 0002 now states explicitly that one-shot retry applies only to
idempotent reads (viewport) after the failed session is stopped;
non-idempotent gesture failures surface directly.
- Helper README session transport corrected: a persistent process serving
one short-lived socket connection per request (the server closes each
accepted connection), not a single long-lived connection.
* fix(android): guard touch session reuse on helper identity, stop mismatched sessions (PR #1281 re-review)
Persistent helper sessions are keyed by device, so touch reuse must also
prove the live session runs the helper binary the command selected. The
session record now stores its helper identity (packageName, runner,
helperVersion, helperVersionCode — the same values that feed the snapshot
session identity), and runAndroidSnapshotHelperSessionTouchCommand takes
the requesting helper identity: on mismatch (packageName/runner always;
version/versionCode when both sides define them) it stops the session and
returns undefined, so the touch command runs one-shot against the selected
artifact — gestures never start sessions; the next snapshot restarts one
with the right artifact. Matching identity reuses the session as before.
Snapshot capture identity and behavior are unchanged.
Regression: 'a provider artifact that mismatches the live session helper
stops it and runs one-shot' — a live fake session from the bundled fixture
artifact, then a gesture through an ADB provider supplying an
already-current artifact with a distinct packageName/runner (no install):
the old session socket receives zero gesture commands, the session is
stopped, the one-shot am instrument args end with the provider runner, and
helperTransport is 'instrumentation'.
* fix(android): include artifact sha in helper session identity; evict stale install memo entries (PR #1281 re-review)
Same-version binary replacement changes only the APK sha, so identity
guards keyed on package/runner/version/versionCode could not detect a
crossover between two artifacts that differ only in bytes:
- The artifact sha256 now joins the helper identity end-to-end:
AndroidSnapshotHelperCaptureOptions gains helperSha256 (snapshot.ts
passes artifact.manifest.sha256 alongside version/versionCode), the
session record stores it, createSessionIdentity includes it (making
snapshot session reuse sha-aware, consistent with the install path's
existing sha check), and the touch identity guard compares it via the
same both-defined rule.
- ensureAndroidSnapshotHelper's install memo now evicts every other
cached decision for the same device+package when it records an
install/current decision, so installing B invalidates A's stale
'current' memo and a later command selecting A re-inspects the device
instead of skipping the sha check.
Regressions: 'a same-version artifact with a different sha stops the live
session and runs one-shot' (touch-helper-session.test.ts — B owns the live
session, a gesture selecting same-version different-sha A sends zero
commands to B's socket, stops it, and completes one-shot) and 'installing
a same-version different-sha helper evicts the stale install memo'
(snapshot-helper.test.ts — A:current cached, B installed, selecting A
re-inspects and reinstalls instead of serving the stale memo). Both
verified to fail without their fix.
Android Test IME Helper
A minimal headless InputMethodService used as a deterministic, Unicode-safe text-entry backend
for local Android sessions. onEvaluateInputViewShown() returns false and
onCreateInputView() returns null, so activating this IME contributes zero accessibility nodes
to the UI tree -- no per-key [group]+[text] chrome, no clipboard/translate/voice-dictation
buttons -- unlike the real system keyboard (Gboard et al.), which pulls its visible keyboard into
the UIAutomator tree the moment a field is focused.
Text is injected through a dynamically-registered BroadcastReceiver carrying base64-encoded
UTF-8 Intent extras, the same payloadBase64 convention the snapshot helper's gesture mode uses for the
same reason: adb shell re-tokenizes raw spaces, and base64 sidesteps that entirely. This makes
Unicode/CJK/emoji round-trip exactly, unlike adb shell input text, which is ASCII-only.
The helper is a service, not an instrumentation, so it cannot use the snapshot helper's
am instrument invocation shape. It is installed and its version
verified the same way (bundled npm-packaged dist + version-keyed manifest.json), but invoked with
adb shell ime enable/ime set (lifecycle) and adb shell am broadcast (text entry) instead.
Build
VERSION="$(node -p 'require("./package.json").version')"
sh ./scripts/build-android-ime-helper.sh "$VERSION" .tmp/android-ime-helper
Run
PACKAGE="com.callstack.agentdevice.imehelper"
SERVICE="$PACKAGE/.TestInputMethodService"
VERSION="$(node -p 'require("./package.json").version')"
adb install -r -t ".tmp/android-ime-helper/agent-device-android-ime-helper-$VERSION.apk"
# Record the current default IME before switching, so it can be restored exactly.
PREVIOUS_IME="$(adb shell settings get secure default_input_method)"
adb shell ime enable "$SERVICE"
adb shell ime set "$SERVICE"
# Focus a text field, then inject Unicode-safe text via base64 extras. adb shell holds
# WRITE_SECURE_SETTINGS, which the receiver requires; a third-party app cannot.
TEXT_B64="$(printf '%s' '你好世界' | base64)"
adb shell am broadcast -p "$PACKAGE" \
-a com.callstack.agentdevice.imehelper.ACTION_INPUT_TEXT_B64 \
--es text "$TEXT_B64"
# Clear the focused field.
adb shell am broadcast -p "$PACKAGE" -a com.callstack.agentdevice.imehelper.ACTION_CLEAR_TEXT
# Restore the previous IME (critical -- do not skip on a real device).
adb shell ime set "$PREVIOUS_IME"
Broadcast actions
The receiver is registered in the running IME process (so getCurrentInputConnection() is live)
but requires the android.permission.WRITE_SECURE_SETTINGS sender permission. adb shell
(uid 2000) holds that signature|privileged permission, but a co-installed third-party app cannot
be granted it, so it cannot deliver broadcasts to the receiver — the text-injection surface is
closed to other apps while the IME is active. (An earlier design used a manifest receiver with
android:exported="false" + explicit-component targeting, but on API 36 adb shell cannot deliver
to a non-exported component, which broke the CLI path; the permission gate is what actually works.)
Every extra is still treated as untrusted input: bounded length, defensive base64 decoding, and the
whole handler is wrapped so a malformed broadcast can never crash the IME process and strand a
field without an active input method.
ACTION_INPUT_TEXT_B64(--es text <base64 utf-8>) -- commit decoded text at the cursor.ACTION_INPUT_TEXT(--es text <string>) -- commit text directly (subject toadb shell's own tokenization; prefer the base64 variant for anything with spaces or non-ASCII).ACTION_CLEAR_TEXT-- select-all and commit an empty string.
An optional --es protocol android-ime-helper-v1 extra is a defensive sanity check (not a
security boundary): if present and it doesn't match, the broadcast is dropped and logged.
Restore hygiene
Switching the active IME is machine-global, not session-scoped. A crashed daemon that switched the
IME and never restored it leaves the device with an invisible keyboard -- on a real device, this
means the user cannot type anywhere until they manually reset their IME in Settings. The TS-side
lifecycle (src/platforms/android/ime-lifecycle.ts) persists the previously active IME to disk
before switching, restores it on session close/teardown, and best-effort restores any orphaned
switch left behind by a previous crashed daemon on startup.