mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
feat/android-atomic-clear
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hubert Gancarczyk
|
c2f3d0fd24 |
fix(tool-server): fit clear to the text/key exclusivity below it
The base now rejects a keyboard call carrying both `text` and `key`, so
the legal shapes are `{}`, `{text}`, `{key}`, `{clear}`, `{clear,text}`
and `{clear,key}`. `clear` is unaffected — it is the one parameter that
still combines — but a lot of this branch was written against the
three-part `{ clear, text, key }` call and no longer describes what runs.
Behaviour:
- Vega's clear refusal ended every message with "send the same call
without `clear`". For `{ clear: true }` alone — the empty-the-field
call the parameter exists for — and for `{ clear: true, text: "" }`,
which the injection guard no-ops, re-sending does nothing at all, so
the field is left neither emptied nor typed into. It now discriminates
the three shapes the way the TV twin does, and says the key press is
what to re-send on `{ clear, key }`, which Vega does accept.
- The TV refusal had the same hole one arm over: `{ clear: true, text: "" }`
named `text`, so a `!== undefined` check sent the caller to
`{ text: "" }`, which types nothing. It now falls in with a clear-only
request, which is the shape it actually is.
Budget: the worst case was `{ clear, text, key }` at 26s + 15s + 15s.
It is now `{ clear, text }` at 26s + 15s. Still past the MCP adapter's
30s per-request fetch timeout, so `longRunning` stays — with the
arithmetic corrected in all four places that carried the old sum.
Comments: three backends justified their resolve-before-inject hoist as
"so an unknown key name fails after the text has already been typed".
A key can no longer follow text, so the surviving reason is `clear`,
which empties the field before either. Android's tie-break for a request
"bad in BOTH halves" is gone with it — that needs `text` and `key`
together. Chromium's key block said it ran after the text; it runs after
the clear.
Tests: the three "orders clear → text → key" backend tests become
"orders clear → key", the other combination a caller can still send and
the one nothing else pins — pressing Enter before the delete run would
submit the field's OLD contents. The chromium read-back test moves from
"reads the field back BEFORE the named key" to what the gate actually
does now: a key-only call never reaches the split check, so a submit
handler that empties and blurs the field cannot be reported as a split.
`interaction-messages` keeps the three-part shape on `startedMsg` only,
which is what an event log really shows for a request about to 400.
The TV ordering test's justification was wrong on its own terms:
`blueprints/android-tv-control.ts` is `typeTv`'s callee, not a second
caller, and its API takes a bare string. `typeTv` has exactly two call
sites, both below the guard, so no caller can produce the shape — the
test now says so and stands as an internal-order pin.
|
||
|
Hubert Gancarczyk
|
83440b83f4 |
test(tool-server): pin that clear survives a run-sequence step
`run-sequence`'s own copy lists `clear` in the keyboard step's arg shape and spells out the clear → text → key order, but nothing pinned that the parameter reaches the tool. A step whose `clear` was dropped still reports `completed` while APPENDING to the field it was meant to replace. Verified live on Chromium too: the step replaces `old value` with `new@example.com`, and the same step without `clear` appends. |
||
|
Mateusz Ścianek
|
b8afbed183 |
feat(tv): Apple TV (tvOS) and Android TV support (#374)
Adds end-to-end support for driving **Apple TV (tvOS)** and **Android TV (leanback)** targets through argent, using a focus-driven interaction model (TVs have no touchscreen — interaction is moving a focus highlight with the remote). This PR was reworked after review to **reuse the existing cross-platform tool surface and the `tv-remote` tool from the Vega/Fire TV work (#366)** rather than ship a separate family of `tv-*` tools. ## What this delivers TV targets are driven by the **same cross-platform tools as every other platform**, routed through the existing `dispatchByPlatform` fork mechanism — there are no dedicated `tv-describe`/`tv-navigate`/`tv-type`/`tv-set-focus` tools: - **`describe`** — on a `runtimeKind: "tv"` target, returns the focus-driven view (focused + focusable elements) instead of a tap tree. On Android TV it auto-falls-back to the full `uiautomator` tree when the RN focus engine exposes nothing. - **`tv-remote`** — the cross-platform remote/D-pad tool from #366, now extended with `ios` (Apple TV HID daemon) and `android` (Android TV adb keyevents) branches alongside the existing `vega` branch. Single button, a `repeat`, or a whole path (`["up","right","select"]`) in one call. - **`keyboard`** — types into the focused field on a TV target (named keys are rejected there — they're navigation, which belongs to `tv-remote`). - **`button`** stays hardware-only (phones/tablets); it is not a TV tool. **Apple TV** runs two native daemons (in-sim AX service + host-side HID daemon), shipped via the `argent-private` submodule. **Android TV** reuses the same tool surface, adb-backed (`input keyevent`, `uiautomator dump`, `input text`). ## Remote vocabulary parity `tv-remote` exposes the full 16-button vocabulary (`up`/`down`/`left`/`right`/`select`/`back`/`home`/`menu`/`playPause`/`rewind`/`fastForward`/`next`/`previous`/`volumeUp`/`volumeDown`/`mute`): | | Apple TV | Android TV | Vega | |---|---|---|---| | D-pad / select / back / menu / home / playPause | ✅ | ✅ | ✅ | | media-transport + volume/mute | ❌ rejected | ✅ | ✅ | Media-transport/volume keys **genuinely work on Android TV** (real keycodes — verified live: `volumeUp` moved `STREAM_MUSIC`, `mute` toggled per `dumpsys audio`). On the **Apple TV simulator** they are **rejected with a clear error**: on-device testing confirmed the tvOS sim's HID stack silently drops Consumer-Control events, so returning success would be a lie. ## Key design points - **tvOS UDIDs are UUID-shaped and indistinguishable from iOS by shape.** Handled with `runtimeKind` detection (tvOS runtime string for Apple TV; `pm list features` leanback/television for Android TV — *not* `ro.build.characteristics`, which lies on TV emulators). - Converted launch-app / restart-app / screenshot / screenshot-diff / run-sequence from **eager** to **lazy** service resolution, so a tvOS target never spins up (and hangs on) the iOS-only simulator-server / native-devtools blueprints. - Native injection selects the platform-matched (TVOSSIMULATOR) dylib slice; daemons recycle across sim reboots and app relaunches so injection and focus survive. - SKILL.md footprint minimized per review: the two TV skills were collapsed into a single lean **`argent-tv-interact`** (~40 lines). ## Verification - `npm run build`, ESLint, Prettier, and the full tool-server suite (**1565 tests**) all pass; CI green (incl. the Apple TV / Android TV / Vega e2e jobs). - Core flows verified live against real apps on both an Apple TV 4K simulator and a Google ATV emulator. ## Dependency / merge order Pins the `argent-private` submodule to the tip of its `feat/tv-support` branch (software-mansion/argent-private#20). **Merge that PR to `main` first, then bump the submodule pointer here to the merged SHA before merging this PR.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Hubert Gancarczyk
|
55587745a7 |
feat(tool-server): implement await-ui-element tool (#396)
## Summary
Adds a `await-ui-element` tool that blocks until a UI condition is
satisfied or a timeout elapses.
This is one of the most impactful missing tools. Without it, agents have
to poll manually with `screenshot → describe → check` loops — slow,
token-heavy, and unreliable (fixed `sleep` delays either over-wait or
fire before the screen settles). `await-ui-element` moves the poll loop
server-side: one call in, a definitive verdict out.
Works on **iOS, Android, and Chromium (CDP)**, polling the same
accessibility / DOM tree as `describe`.
## Conditions
- `visible <selector>` — wait until an element appears on screen
- `hidden <selector>` — wait until an element disappears
- `exists <selector>` — wait until an element is in the tree
- `text <selector> <expectedText>` — wait until the matched element
contains the text
**selector** is `{ text?, identifier?, role? }` — every field provided
must match (case-insensitive substring). Polls internally; default
`timeoutMs` **5000**, default `pollIntervalMs` **400**. Returns `{
success: boolean, elapsed: number }`, with a `note` explaining what was
seen on timeout.
```json
{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }
```
Also usable as a step inside `run-sequence`, so a tap → await-ui-element
→ tap chain runs in a single call.
## Notable details
- **Multi-match aware** — a substring selector can hit several nodes, so
conditions evaluate the whole match set: `visible` holds if any match is
on-screen, `hidden` only if none is (a zero-area container can't flip
the verdict).
- **`hidden` guard** — a selector that matches nothing counts as
already-hidden; the `note` flags when the selector never matched, so a
typo doesn't read as a silent success.
- **Deadline-clamped poll** — a large `pollIntervalMs` can't overshoot
`timeoutMs`.
- **Abort-aware** — both the `time` sleep and the poll loop stop
promptly on cancellation.
## Changes
- `tools/await-ui-element/index.ts` — new tool (factory, mirrors
`describe`'s service resolution)
- `utils/setup-registry.ts` — registers the tool
- `tools/run-sequence/index.ts` — allows `await-ui-element` as a step +
arg-doc + example
- `test/await-ui-element.test.ts` — new test suite
- `skills/rules/argent.md` + `argent-device-interact/SKILL.md` —
document `await-ui-element`, replace the "don't poll `screenshot`"
guidance with it
## Test plan
- [x] `time` sleeps and succeeds without touching a device
- [x] `visible` succeeds once the element appears; times out with a
diagnostic note when it doesn't
- [x] `exists` / `hidden` / `text` happy paths; `text` timeout note
reports last-seen text
- [x] `hidden` instant success **and** the "selector never matched" flag
- [x] poll sleep clamped to the deadline (large `pollIntervalMs` can't
overshoot `timeoutMs`)
- [x] prompt cancellation aborts both the poll loop and the `time` sleep
- [x] multi-match evaluation (`findAll` / `evaluateMatches`)
- [x] schema validation (required fields per condition, `durationMs`
cap)
|
||
|
Hubert Gancarczyk
|
0edefebe06 |
fix(telemetry): orchestator sub-tools invocations was missing ai_client and platform attributes (#400)
## Summary Fixes telemetry attribution for tools dispatched by orchestrators. `run-sequence`, `flow-execute`, and `flow-add-step` run their steps via `registry.invokeTool()` directly, which minted a fresh, unregistered invocation id per nested call — so the `ai_client` / `platform` the HTTP layer captured for the outer request never reached the nested gestures, and they were recorded as **anonymous** in telemetry. Remove `opt_out` event that is a leftover. ## Changes - Add an optional `recordChildInvocation` recorder to `InvokeToolOptions` / `ToolContext`, bound by the HTTP layer to the outer request's attribution. - New `invokeSubTool` helper: registers each sub-tool under a minted id inheriting the request's AI client, with the platform re-derived from the sub-tool's **own** device arg (`udid` / `device_id` / `avdName`); forwards the recorder so propagation survives arbitrary nesting (e.g. `flow-execute → run-sequence → gesture-tap`). - Route the three orchestrators' sub-dispatch through - Refactor platform inference in the HTTP layer into a shared `platformFromArgs` helper (also hardened to not throw on a malformed device arg). - Remove opt_out event leftover Why re-derive platform per child: an orchestrator like `flow-execute` has no platform of its own and a single flow can target several devices, so the child's device arg is the only correct platform source; the parent's platform is the fallback. |