mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

History

Nicolas Bataille 4b8bcaca60 feat(interaction): accept fill <target> "" as the clear-field primitive (#2066 )

* feat(interaction): accept fill <target> "" as the clear-field primitive

Emptying an input was not expressible: `fill` refused the empty string
("Expected text to be a non-empty string"), `type` only appends, and `keyboard`
has no delete verb. Clearing a field before typing is a routine QA step, so the
only route was the app's own clear button or N locale-dependent keyboard delete
presses read out of a snapshot.

`fill <target> ""` now means "replace with nothing". Both platforms already own
the clear half of replace, so this is the validation and reporting that stood in
front of it, not a new interaction:

- `stringField` takes an opt-in `allowEmpty`, used only by `fill`'s `text`.
  `requiredField` still refuses a MISSING text, so `fill @e57` stays an error
  rather than silently erasing the field — `readFillTargetFromPositionals` now
  reports `undefined` for "no text argument" instead of collapsing it to `''`.
  `type` keeps refusing an empty text: appending nothing is not a clear.
- The Apple runner's empty-text early return skipped the clear while reporting
  "typed". For a replacement it now runs `clearTextInput` and verifies the field
  came back empty (secure fields stay unverifiable, as elsewhere).
- Android already clears before typing and skips an empty shell/IME write, but
  its verifier read a cleared field's absent `text` attribute as a mismatch
  against `''`. An empty expectation now accepts null or "".

Whitespace-only text keeps its established per-shape rules; only `''` is new.

Closes #2063

* fix(interaction): fail the empty-fill clear closed on every backend

Addresses the P1 review on #2066, then closes the same fail-open class
on the backends the PR did not reach:

- Android: an empty expectation no longer matches when the verification
  scan observed NO input node at all — actual is null both for a cleared
  field and for a wrong point/lost focus, and three empty samples of
  nothing were a stable success for a clear that never touched a field.
- Apple runner: when the empty-replacement path cannot resolve a clear
  target (including the synthesized first-responder route, whose target
  carries no element), it returns the typed TEXT_INPUT_NOT_FOCUSED
  failure instead of falling through to the vacuous-typing
  verified-success return. Regression runs in the ios.yml XCTest lane.
- webdriver: fill is tap + sendKeys and owns no clear mechanism, so an
  empty fill refuses as UNSUPPORTED_OPERATION before touching the
  device, instead of reporting a clear it cannot perform.
- linux + web coordinate fill: typing zero characters over the
  select-all selection left the old value intact; the empty fill now
  deletes the selection.
- recording: an empty --record-as literal matches inside every string;
  it now parameterizes only the fill's own text field instead of
  rewriting every empty field and empty evidence label in the entry.
  (The session-wide echo registry already excluded empty literals.)
- help: the text-entry topic taught agents that fill "" is not a
  clear-field command; it now states the new contract.

Each new test was observed red against the pre-fix code.

* fix(android): read hint-showing from the helper so a cleared field verifies

Live Pixel 9 emulator, adb-shell channel: clearing the Settings search
field succeeded on the device but reported 'Android fill verification
failed', because a cleared EditText dumps its HINT as text — getText()
returns the hint for an empty field on modern Android, so 'Search
settings' read back as a residual value. This is the same
placeholder-as-value trap the Apple runner already handles with
treatingPlaceholderAsEmpty.

The helper now emits hint-showing (isShowingHintText, API 26+), the
hierarchy parser carries it, and fill verification matches against the
field's VALUE — hint-only text is an empty value, for empty and
non-empty expectations alike. A field whose real value equals its hint
string keeps failing the clear check: only the authoritative flag, never
the text, says it is a hint. Raw uiautomator dumps carry no such fact
and keep the fail-closed behavior.

Live evidence, both admission channels, after this fix: test-ime and
adb-shell clears both report Filled 0 chars with the field back on its
placeholder; the pre-fix adb-shell run failed closed (never a false
success).

* fix(interaction): close the adversarial-review findings on the empty-fill clear

- android adb-shell: the delete burst is sized from the value being
  REMOVED (pre-mutation read; the attempt's cap when unreadable), not
  from the empty incoming text, which sent the 12/24-delete minimums and
  could never empty a field longer than 36 characters.
- android: the unconfirmed soft-success no longer applies to an empty
  expectation — nothing app-formats the empty value, so residue after a
  clear is a failed clear, and the soft-success also skipped the second,
  bigger delete burst.
- android masked fields: an empty expectation accepts an observed masked
  node with no dump text (a masked field WITH content dumps its bullet
  run), so clearing a password field no longer fails after the clear
  worked — matching iOS, where a secure-field clear succeeds unverified.
- find: 'find <q> fill ""' now reaches the fill leaf as the clear
  request on both the CLI reader and the daemon positional parse; a
  MISSING value keeps its refusal at each producer, so the typed
  value: string contract is unchanged.
- maestro export: a recorded clear exports as tapOn + eraseText instead
  of a vacuous inputText: "" (with the 50-character-default warning).
- the missing-text refusals teach the clear form: (use "" to clear
  the field).

Full unit suite green (1061 files); each behavioral fix carries a test
observed red against the prior code.

* refactor(interaction,android): extract the fill parse and shell-attempt branches

The review commits pushed parseFillTarget and fillAndroid over the
complexity gate (13 cyclomatic each). Each fill target shape parses in
its own function sharing one missing-text response, and the adb-shell
attempt (clear sizing + clear + type + verify) moves out of the fill
loop. Behavior-preserving; the existing tests cover every branch.

* refactor(interaction,android): one owner per empty-fill fact

Design pass after review: the missing-vs-empty rule and the observed-
value rule each had several owners; now each has one.

- parseFillTarget decodes ONCE through readFillTargetFromPositionals —
  which already owns shape detection and documents the undefined-vs-''
  contract on DecodedFillTarget — and keeps only what the wire owns:
  versioned-ref admission, the selector whitespace rule, and the daemon
  responses. This deletes the point branch's duplicated slicing, the
  hasFillText guard, and the three per-shape parse functions.
- observedAndroidValue() is the single statement of Android's value
  rule (absent attribute and hint-only text are the empty value); the
  text branch, the match rule, and the masked branch all consume it.
  The masked branch thereby gains the hint-showing collapse it was
  missing, and isAcceptableAndroidFillMatch narrows to plain strings.
- The empty-text-is-clear contract is stated once, on Interactor.fill
  in contracts, instead of implied per backend.

Behavior-preserving except the masked+hint gain; the existing tests
cover every branch (494 Android, 15 fill-target).

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>

2026-08-27 10:59:30 +02:00

src

feat(interaction): accept fill <target> "" as the clear-field primitive (#2066 )

2026-08-27 10:59:30 +02:00

AndroidManifest.xml

…

debug.keystore

…

README.md

fix(android): publish covered state from exact order evidence (#1981 )

2026-08-24 18:09:05 +02:00

README.md

Android Snapshot Helper

Small instrumentation APK used to capture Android accessibility snapshots without relying on uiautomator dump's fixed idle wait behavior. The helper enables Android's interactive-window retrieval flag and serializes every accessible window root returned by UiAutomation.getWindows() so keyboards and system overlays can appear in the same snapshot. If interactive window roots are unavailable, it falls back to the active-window root.

The helper is intentionally provider-neutral. Local adb, cloud ADB tunnels, and remote device providers can all install and run the same APK as long as they can execute ADB-style operations. Released helper APKs use the committed debug.keystore; do not rotate it casually, because Android requires a stable signing certificate for adb install -r upgrades.

Build

VERSION="$(node -p 'require("./package.json").version')"
AGENT_DEVICE_ANDROID_HELPER=snapshot sh ./scripts/build-android-helper.sh "$VERSION" .tmp/android-snapshot-helper

The build uses Android SDK command-line tools directly. It expects ANDROID_HOME or ANDROID_SDK_ROOT to point at an SDK with platforms/android-36 and matching build tools. pnpm prepack builds the npm-bundled helper into android/snapshot-helper/dist; npm users get that APK in the package and the first helper-backed snapshot installs it automatically when missing or outdated.

Run

VERSION="$(node -p 'require("./package.json").version')"
adb install -r -t ".tmp/android-snapshot-helper/agent-device-android-snapshot-helper-$VERSION.apk"
adb shell am instrument -w \
  -e waitForIdleTimeoutMs 500 \
  -e waitForIdleQuietMs 100 \
  -e timeoutMs 8000 \
  -e maxDepth 128 \
  -e maxNodes 5000 \
  com.callstack.agentdevice.snapshothelper/.SnapshotInstrumentation

maxDepth also caps recursive traversal depth inside the helper. The -t install flag is required because the helper is a test-only instrumentation APK. Devices or providers that block test-package installs must allow this package before helper capture can run.

waitForIdleTimeoutMs defaults to 500, which is a maximum wait, not a fixed sleep. Direct helper invocations can pass 0 when immediate capture during ongoing animation is preferred. Root acquisition has a separate 500 ms stabilization bound that is used only when no root is available or an active/focused window is temporarily missing its root; complete captures pay no additional wait.

One-Shot Modes

Passing -e mode snapshot|viewport|gesture selects what a single instrumentation run does; snapshot is the default and matches the Run section above.

# Read the interactive-window viewport without capturing a snapshot.
adb shell am instrument -w -e mode viewport \
  com.callstack.agentdevice.snapshothelper/.SnapshotInstrumentation

# Inject a planned touch gesture, described by a base64 JSON payload.
adb shell am instrument -w -e mode gesture -e payloadBase64 "$PAYLOAD" \
  com.callstack.agentdevice.snapshothelper/.SnapshotInstrumentation

The gesture payload is a base64-encoded JSON object using protocol android-touch-plan-v1:

kind: swipe (one pointer) or transform (two pointers, e.g. pinch/rotate)
durationMs: integer, 0-120000
pointers: array of { pointerId, samples }, pointerIds ordered from 0, one pointer for swipe and exactly two for transform
each pointer's samples is { offsetMs, x, y }[] with at least two entries; offsetMs values must be strictly increasing (equal offsets are only allowed when durationMs is 0), the first sample's offsetMs must be 0, and the last must equal durationMs. For transform gestures, both pointers must share the same offsetMs sequence.

Persistent Session

Passing -e sessionPort <port> keeps the instrumentation alive after startup and serves repeated commands over a local TCP server on 127.0.0.1:<port>, instead of exiting after one snapshot. This avoids paying UiAutomation connect/teardown cost per call. Each command uses one short-lived connection: the client connects, sends a single command line, reads the response, and the server closes that connection; the process stays alive for the next connection:

snapshot <requestId> — capture and return an XML snapshot, same semantics as the default mode
viewport <requestId> — return interactive-window viewport bounds
gesture <requestId> <payloadBase64> — inject a planned touch gesture (same payload as the one-shot gesture mode)
quit <requestId> — acknowledge and stop the session

viewport and gesture responses are headers-only (no body): kind, injectedEvents, elapsedMs for gesture, and x, y, width, height for viewport. snapshot responses carry the XML body after the header block, as described below. The response protocol literal is always android-snapshot-helper-v1, regardless of session or one-shot transport.

Output Contract

The APK emits instrumentation status records using agentDeviceProtocol=android-snapshot-helper-v1.

The XML node attributes intentionally mirror acquisition facts decoded by the host, including visible-to-user, drawing-order, bounds, text/description/id, interaction booleans, and window metadata on window roots. The helper emits drawing-order on Android API 24+ and omits it on API 23, where the platform API is unavailable. The host keeps that fact as private capture evidence; the daemon uses it to annotate covered actions without adding it to normalized snapshot nodes.

Each XML chunk is sent with:

outputFormat=uiautomator-xml
chunkIndex
chunkCount
payloadBase64

The final instrumentation result for the default snapshot mode includes:

ok=true
helperApiVersion=2
waitForIdleTimeoutMs
waitForIdleQuietMs
timeoutMs
maxDepth
maxNodes
rootPresent
captureMode (interactive-windows or active-window)
windowCount
nodeCount
truncated
elapsedMs

viewport and gesture one-shot results carry agentDeviceProtocol/helperApiVersion/ outputFormat plus the mode-specific fields described under "One-Shot Modes" above, instead of the snapshot-mode fields listed here.

Failures return ok=false, errorType, and message in the final result.

The release manifest is a stable provider contract for the current helper protocol. Providers should resolve the APK from apkUrl, verify sha256, install using installArgs, and run instrumentationRunner. installArgs must start with install; extra arguments are limited to the allowlisted adb install flags -r, -t, -d, and -g, and the consumer appends the APK path.