mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
flow-visual-settle
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mateusz Ścianek
|
b8afbed183 |
feat(tv): Apple TV (tvOS) and Android TV support (#374)
Adds end-to-end support for driving **Apple TV (tvOS)** and **Android TV (leanback)** targets through argent, using a focus-driven interaction model (TVs have no touchscreen — interaction is moving a focus highlight with the remote). This PR was reworked after review to **reuse the existing cross-platform tool surface and the `tv-remote` tool from the Vega/Fire TV work (#366)** rather than ship a separate family of `tv-*` tools. ## What this delivers TV targets are driven by the **same cross-platform tools as every other platform**, routed through the existing `dispatchByPlatform` fork mechanism — there are no dedicated `tv-describe`/`tv-navigate`/`tv-type`/`tv-set-focus` tools: - **`describe`** — on a `runtimeKind: "tv"` target, returns the focus-driven view (focused + focusable elements) instead of a tap tree. On Android TV it auto-falls-back to the full `uiautomator` tree when the RN focus engine exposes nothing. - **`tv-remote`** — the cross-platform remote/D-pad tool from #366, now extended with `ios` (Apple TV HID daemon) and `android` (Android TV adb keyevents) branches alongside the existing `vega` branch. Single button, a `repeat`, or a whole path (`["up","right","select"]`) in one call. - **`keyboard`** — types into the focused field on a TV target (named keys are rejected there — they're navigation, which belongs to `tv-remote`). - **`button`** stays hardware-only (phones/tablets); it is not a TV tool. **Apple TV** runs two native daemons (in-sim AX service + host-side HID daemon), shipped via the `argent-private` submodule. **Android TV** reuses the same tool surface, adb-backed (`input keyevent`, `uiautomator dump`, `input text`). ## Remote vocabulary parity `tv-remote` exposes the full 16-button vocabulary (`up`/`down`/`left`/`right`/`select`/`back`/`home`/`menu`/`playPause`/`rewind`/`fastForward`/`next`/`previous`/`volumeUp`/`volumeDown`/`mute`): | | Apple TV | Android TV | Vega | |---|---|---|---| | D-pad / select / back / menu / home / playPause | ✅ | ✅ | ✅ | | media-transport + volume/mute | ❌ rejected | ✅ | ✅ | Media-transport/volume keys **genuinely work on Android TV** (real keycodes — verified live: `volumeUp` moved `STREAM_MUSIC`, `mute` toggled per `dumpsys audio`). On the **Apple TV simulator** they are **rejected with a clear error**: on-device testing confirmed the tvOS sim's HID stack silently drops Consumer-Control events, so returning success would be a lie. ## Key design points - **tvOS UDIDs are UUID-shaped and indistinguishable from iOS by shape.** Handled with `runtimeKind` detection (tvOS runtime string for Apple TV; `pm list features` leanback/television for Android TV — *not* `ro.build.characteristics`, which lies on TV emulators). - Converted launch-app / restart-app / screenshot / screenshot-diff / run-sequence from **eager** to **lazy** service resolution, so a tvOS target never spins up (and hangs on) the iOS-only simulator-server / native-devtools blueprints. - Native injection selects the platform-matched (TVOSSIMULATOR) dylib slice; daemons recycle across sim reboots and app relaunches so injection and focus survive. - SKILL.md footprint minimized per review: the two TV skills were collapsed into a single lean **`argent-tv-interact`** (~40 lines). ## Verification - `npm run build`, ESLint, Prettier, and the full tool-server suite (**1565 tests**) all pass; CI green (incl. the Apple TV / Android TV / Vega e2e jobs). - Core flows verified live against real apps on both an Apple TV 4K simulator and a Google ATV emulator. ## Dependency / merge order Pins the `argent-private` submodule to the tip of its `feat/tv-support` branch (software-mansion/argent-private#20). **Merge that PR to `main` first, then bump the submodule pointer here to the merged SHA before merging this PR.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Hubert Gancarczyk
|
55587745a7 |
feat(tool-server): implement await-ui-element tool (#396)
## Summary
Adds a `await-ui-element` tool that blocks until a UI condition is
satisfied or a timeout elapses.
This is one of the most impactful missing tools. Without it, agents have
to poll manually with `screenshot → describe → check` loops — slow,
token-heavy, and unreliable (fixed `sleep` delays either over-wait or
fire before the screen settles). `await-ui-element` moves the poll loop
server-side: one call in, a definitive verdict out.
Works on **iOS, Android, and Chromium (CDP)**, polling the same
accessibility / DOM tree as `describe`.
## Conditions
- `visible <selector>` — wait until an element appears on screen
- `hidden <selector>` — wait until an element disappears
- `exists <selector>` — wait until an element is in the tree
- `text <selector> <expectedText>` — wait until the matched element
contains the text
**selector** is `{ text?, identifier?, role? }` — every field provided
must match (case-insensitive substring). Polls internally; default
`timeoutMs` **5000**, default `pollIntervalMs` **400**. Returns `{
success: boolean, elapsed: number }`, with a `note` explaining what was
seen on timeout.
```json
{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }
```
Also usable as a step inside `run-sequence`, so a tap → await-ui-element
→ tap chain runs in a single call.
## Notable details
- **Multi-match aware** — a substring selector can hit several nodes, so
conditions evaluate the whole match set: `visible` holds if any match is
on-screen, `hidden` only if none is (a zero-area container can't flip
the verdict).
- **`hidden` guard** — a selector that matches nothing counts as
already-hidden; the `note` flags when the selector never matched, so a
typo doesn't read as a silent success.
- **Deadline-clamped poll** — a large `pollIntervalMs` can't overshoot
`timeoutMs`.
- **Abort-aware** — both the `time` sleep and the poll loop stop
promptly on cancellation.
## Changes
- `tools/await-ui-element/index.ts` — new tool (factory, mirrors
`describe`'s service resolution)
- `utils/setup-registry.ts` — registers the tool
- `tools/run-sequence/index.ts` — allows `await-ui-element` as a step +
arg-doc + example
- `test/await-ui-element.test.ts` — new test suite
- `skills/rules/argent.md` + `argent-device-interact/SKILL.md` —
document `await-ui-element`, replace the "don't poll `screenshot`"
guidance with it
## Test plan
- [x] `time` sleeps and succeeds without touching a device
- [x] `visible` succeeds once the element appears; times out with a
diagnostic note when it doesn't
- [x] `exists` / `hidden` / `text` happy paths; `text` timeout note
reports last-seen text
- [x] `hidden` instant success **and** the "selector never matched" flag
- [x] poll sleep clamped to the deadline (large `pollIntervalMs` can't
overshoot `timeoutMs`)
- [x] prompt cancellation aborts both the poll loop and the `time` sleep
- [x] multi-match evaluation (`findAll` / `evaluateMatches`)
- [x] schema validation (required fields per condition, `durationMs`
cap)
|
||
|
Hubert Gancarczyk
|
0edefebe06 |
fix(telemetry): orchestator sub-tools invocations was missing ai_client and platform attributes (#400)
## Summary Fixes telemetry attribution for tools dispatched by orchestrators. `run-sequence`, `flow-execute`, and `flow-add-step` run their steps via `registry.invokeTool()` directly, which minted a fresh, unregistered invocation id per nested call — so the `ai_client` / `platform` the HTTP layer captured for the outer request never reached the nested gestures, and they were recorded as **anonymous** in telemetry. Remove `opt_out` event that is a leftover. ## Changes - Add an optional `recordChildInvocation` recorder to `InvokeToolOptions` / `ToolContext`, bound by the HTTP layer to the outer request's attribution. - New `invokeSubTool` helper: registers each sub-tool under a minted id inheriting the request's AI client, with the platform re-derived from the sub-tool's **own** device arg (`udid` / `device_id` / `avdName`); forwards the recorder so propagation survives arbitrary nesting (e.g. `flow-execute → run-sequence → gesture-tap`). - Route the three orchestrators' sub-dispatch through - Refactor platform inference in the HTTP layer into a shared `platformFromArgs` helper (also hardened to not throw on a malformed device arg). - Remove opt_out event leftover Why re-derive platform per child: an orchestrator like `flow-execute` has no platform of its own and a single flow can target several devices, so the child's device arg is the only correct platform source; the parent's platform is the fallback. |