mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
Michał Pierzchała
81409f1a7c
* refactor: migrate type to the request-bound device runtime
Wave 5 unit 2 for #1739 (ADR 0019), find's last blocker. `type "text"` and
`find <q> type "text"` now reach the device through one admitted, request-bound
`typeText` operation instead of the `handleTypeCommand` interactor leaf and its
dispatch-table arm.
- New `TypeTextRuntimeOperations` contract riding the same `Interactor` seam as
focus/screenshot/element-text; the operation returns the interactor's own
closed `TypeTextBackendResult`, so Apple route evidence passes through and
every other owner types blind, exactly as before. The iOS synthesized-type
commit wait (#1676) is Apple-interactor-internal and moves nowhere.
- The interaction backend's `typeText` member exists only when the `type`
handler admitted and bound a runtime — no caller can fall back to legacy
dispatch, so the command keeps exactly one execution path (R41).
- `executeBoundTypeText` reproduces the retired leaf byte-for-byte: leading-ref
rejection with the same hint, space-joined positionals, the 0-10000 delay
bound, and only textEntryRoute surviving from the owner's result. Its parse
pins moved from the dispatch-level tests into the daemon runtime test.
- Exact-owner facts replace the capability bucket (apple sim+device, android
all-but-simulator-row, harmonyos emulator+device, linux device, web device,
vega unavailable, providers wherever their interactor is reachable) and
`type` leaves HARMONYOS_SUPPORTED_COMMANDS / WEB_INTERACTION_COMMANDS.
- The Linux desktop replay types a digit on real hardware; the coverage
manifest promotes `type` contract -> live with the two-sided count pins.
- Android/webdriver facts helpers extracted (androidTouchFact, interactorCell)
to keep inspectFacts under the complexity gate.
`find` stays legacy: both of its direct execution legs now share bound
runtimes, so the atomic R35 cutover is next.
* refactor(type): single-pass daemon routing, shared binder source, owner cell tests
Review follow-ups on #1935.
- The type handler now calls the bound executor directly: the interaction-
runtime hop validated and formatted what executeBoundTypeText validates and
formats again, so it is gone — no boundTypeText backend member, no second
result rebuild. The ADR 0014 frame expiry moves to the handler.
- New contracts/interactor-operation-binding.ts: one local resolver and one
fail-closed provider resolver shared by the screenshot, focus, and type
binders — three private copies retired, provider error text preserved.
- provider-limrun/interaction-operations.ts: the interactor-backed interaction
cells move out of the app-log owner (586 -> 563 lines, below its pre-unit
size); text interaction is composed by that owner, not defined in it.
- Every owner runtime test now pins the focusPoint/typeText fact cells and
bound-operation presence for its exact kinds: apple, android (incl. the
synthetic-simulator refusal), harmonyos, linux, web, vega (refusal + hint),
webdriver (reachability-gated, incl. inactive session), limrun (live +
recovery). The webdriver unsupported-capability row documents that
interaction gates on interactor reachability, not capture declarations.
- The Linux replay assertion is now change-sensitive: type "555" then wait for
a 555 node — no calculator button carries that label, so the wait passes only
if the keystrokes landed in the display; deleting the type step turns it red.
* fix(replay): give the calculator focus before the Linux type assertion
The change-sensitive wait exposed what the review predicted: the typed digits
never landed, because `focus 100 100` clicks the DESKTOP and takes keyboard
focus away from the calculator. The old broad assertion masked exactly this.
The retries then wedged on a latent quirk: attempt-1 leaves the pointer at
(100,100), and the next attempt's `xdotool mousemove --sync` to the same point
waits for a motion event that never comes, so every retry dies at the focus
step with a 10s timeout — which is why the lane reported step 7, not the
failing wait.
New tail: `focus 100 100` (R40 evidence + survival assert), then
`click "label=1"` — a resolved press inside the window that restores keyboard
focus, proves pointer input lands in the app, and moves the pointer off
(100,100) so retries cannot trip the mousemove no-op hang — then `type "55"`
and `wait "label=155 || text=155 || value=155"`. No button is labelled 155, so
the wait passes only if the typed keystrokes reached the display.
* refactor(type): delete the fallow SDK typeText surface, drop dead surface fields
Thermo-nuclear review follow-ups (reviewed at 82fb8c2dc; the three-hop relay
it names was already deleted in 3d44b6dea — these are the residuals).
- typeTextCommand had zero production callers after the direct-executor
routing: the daemon was its only consumer, and the released SDK types over
the wire (executeCommand('type')), not through the embedded runtime catalog.
Deleted: the command, its Options/Result types, the catalog registrations,
the AgentDeviceBackend.typeText member, and every fixture/pin that kept the
dead surface alive. R41's retirement claim now names typeTextCommand, so a
revival fails the gate. One parse/compose owner remains: executeBoundTypeText.
- TypeTextInput.options.surface and FocusPointInput.options.surface were dead
clones — never set by a projector, never read by a binder. Removed both.
- The Linux replay click disambiguates with role=button: the calculator tree
carries [text] digit nodes beside the buttons, so a bare label=1 was an
AMBIGUOUS_MATCH rejection on attempt-1 — which parked the pointer at
(100,100) and made every retry hang in the xdotool no-op move, reporting as
the step-7 focus timeout.
Live-verified on iPhone 17 Pro at this head: coordinate focus -> bare type ->
route synthesized-first-responder -> text observable in the captured tree.
* fix(ci): lower the settle.test.ts pin, add pixel evidence to the Linux type wait
- settle.test.ts shrank to 2359 lines when its dead typeText fixture member
left; the ratchet pin follows it down (the history-backed gate caught the
gap on CI while the local affected run had not re-selected the ratchet).
- The Linux replay now screenshots the calculator right after the type step.
The lane uploads test/screenshots/replays/*.png on pass and fail, so a
wait-155 timeout becomes diagnosable from the artifact: display showing 155
means a tree-exposure gap; an empty display means the input never landed.
The 20-ref divergence dump cannot show the entry node either way.
* fix(replay): assert the Linux type through the computed result
Run 32485981780's typed-state artifact settled both open questions with one
image: the display shows "55" — the bound typeText keystrokes LAND on Linux
CI — while the wait for that value timed out, so the calculator entry does not
expose its text to selectors; and the display shows no leading "1", so the
resolved click on button 1 missed its target entirely.
Both discoveries leave the replay: the click dependency goes (a pre-existing
click-coordinate defect is not this unit's evidence chain), and the assertion
moves to where the tree can answer — the typed string is now a full
calculation ("100+55=") whose `=` creates a history row, and the wait matches
the computed 155. No button carries that label and the typed string never
contains it, so the wait passes only if the keystrokes executed; deleting the
type step turns it red. The typed-state screenshot stays as per-run pixel
evidence either way.
* fix(linux): guard the no-op --sync mousemove; assert typed digits via value=
Two artifact PNGs decided this. Run 32485981780 shows "55" in the entry while
the wait for it timed out on a mismatched target; run 32487868346 shows
"100+55" — digits and + land, the = keystroke does not, and the retries died
in the focus step again because the previous commit removed the pointer-moving
click.
- moveTo now probes `xdotool getmouselocation --shell` and skips the --sync
move when the pointer already sits on the target: a no-op move emits no
motion event and hangs until the action timeout, which is what reported
every failed replay retry as its first coordinate step. A failed probe never
blocks the move. The provider test pins both sequences, including the
skip case.
- The replay types digits only ("155") and waits on `value=155`: the AT-SPI
dumper reads the entry's Text interface into the node's value and the
selector engine matches it — the earlier "exposure gap" conclusion came
from a pair that never tested the matching value. No button carries 155, so
the wait passes only if the keystrokes executed.
* fix(replay): keep Linux type at contract tier — GTK4 exposes no entry text
Run 32490373693 closed the investigation: the typed-state artifact shows
"155" in the calculator entry while the wait for value=155 timed out with no
interactive filtering in play and a matcher that does compare node.value. The
AT-SPI dumper's get_text_iface() route returns nothing for GTK4
gnome-calculator, so no tree-level assertion on typed text can hold on this
lane today.
The replay keeps the type step and uploads the typed-entry screenshot every
run — live pixel evidence that the migrated typeText path lands keystrokes on
real Linux hardware — and closes with the survival assertion. The manifest
claim returns to the contract tier with the reason written at the entry, and
the count pins follow. The GTK4 exposure defect joins the Linux input-defect
chip; fixing PyGObject-vs-GTK4 blind through CI rounds is not a sane loop.
The moveTo no-op guard from the previous commit stays: it is why this run
finally reported the true failing step on every attempt instead of the
step-7 hang.
53 lines
1.8 KiB
TypeScript
53 lines
1.8 KiB
TypeScript
import type { AgentDeviceBackend } from '../../../src/backend.ts';
|
|
import type { SnapshotState } from '@agent-device/kernel/snapshot';
|
|
import { createLocalArtifactAdapter } from '../../../src/io.ts';
|
|
import {
|
|
createAgentDevice,
|
|
createMemorySessionStore,
|
|
localCommandPolicy,
|
|
} from '../../../src/runtime.ts';
|
|
|
|
type ContractBackendOverrides = Partial<
|
|
Pick<
|
|
AgentDeviceBackend,
|
|
| 'captureSnapshot'
|
|
| 'tap'
|
|
| 'tapTarget'
|
|
| 'fill'
|
|
| 'fillTarget'
|
|
| 'performGesture'
|
|
| 'resolveGestureViewport'
|
|
>
|
|
> & {
|
|
platform?: AgentDeviceBackend['platform'];
|
|
};
|
|
|
|
/**
|
|
* The plain runtime harness for contract scenarios on the paths that never
|
|
* touch the runner: runtime-selector, runtime-ref, native-ref (backend
|
|
* tapTarget/fillTarget present) and coordinate. Path forcing is natural:
|
|
* selector/ref targets pick the runtime path, a `tapTarget`/`fillTarget`
|
|
* backend picks the native-ref fast path, x/y picks the coordinate path.
|
|
*/
|
|
export function createContractDevice(
|
|
snapshot: SnapshotState,
|
|
overrides: ContractBackendOverrides = {},
|
|
): ReturnType<typeof createAgentDevice> {
|
|
return createAgentDevice({
|
|
backend: {
|
|
platform: overrides.platform ?? 'ios',
|
|
captureSnapshot: async (...args) =>
|
|
overrides.captureSnapshot ? await overrides.captureSnapshot(...args) : { snapshot },
|
|
tap: async (...args) => await overrides.tap?.(...args),
|
|
tapTarget: overrides.tapTarget,
|
|
fill: async (...args) => await overrides.fill?.(...args),
|
|
fillTarget: overrides.fillTarget,
|
|
performGesture: overrides.performGesture,
|
|
resolveGestureViewport: overrides.resolveGestureViewport,
|
|
} satisfies AgentDeviceBackend,
|
|
artifacts: createLocalArtifactAdapter(),
|
|
sessions: createMemorySessionStore([{ name: 'default', snapshot }]),
|
|
policy: localCommandPolicy(),
|
|
});
|
|
}
|