2 Commits

Author	SHA1	Message	Date
Piotr Krawiec	ab35c5653e	fix(describe): descend into Vega WebView (Chromium sub-app) (#474) (#475) ... ## What & why Fixes #474. On Vega, `describe` collapsed a `WebView` into a single opaque node — only the page `<title>` surfaced, none of the DOM. Root cause: a `WebView` embeds a nested `<app toolkitName="Chromium">` whose text nodes carry **no `role`** and keep their string in a `<text>` **element** child. The native RN UIToolkit instead tags text nodes `role="text"` with **inline** text. `isMeaningful` only counted inline `node.text`, so every WebView text node looked empty → the whole web DOM (grid tiles, headings, labels) was judged unmeaningful and flattened away. Notably the underlying automation toolkit *does* publish the full Chromium a11y tree (confirmed by querying the same backend through the Appium kepler driver) — this was purely a parser-side traversal gap. ## The fix (`source-parser.ts`) 1. **`ownText(node)`** — a node's text is now its inline text **plus** the text of its direct `<text>` element children. Both `isMeaningful` and `labelOf` use this one notion (they were already asymmetric: `labelOf` read `<text>` children, `isMeaningful` didn't). 2. **Sub-span collapse in `convert`** — the Chromium a11y tree aggregates a container's accessible name **and** repeats it as child spans (a tile `"10Tile 10"` nests `"10"` + `"Tile 10"`). Children are dropped only when they are **all plain text spans whose in-order labels exactly reconstruct the parent's aggregated label** (Chromium joins inline spans with no separator). Exact reconstruction — not substring containment — means a distinct `"Play"` under `"Playlist"` (or `"Tile 1"` under `"Tile 10"`) is never swallowed, and a real list whose container name joins rows with spaces (`"Home Settings About"`) never reconstructs from its rows, so separately-navigable rows keep their own frames. ## Verification - New regression tests in `vega-source-parser.test.ts` (real-device fixture `vega-webview-grid.xml`): all 12 grid tiles surface; frames normalize against the window; focused/selected state preserved; exact sub-spans collapse; **distinct substring child kept** (`Play`/`Playlist`); **distinct list rows kept with their own frames** (`Home`/`Settings`/`About`); labelless structural container keeps all children; structural/interactive same-text child kept. - Full `tool-server` suite green (one unrelated pre-existing failure: `await-ui-element.test.ts`, fails on `main` too). - **End-to-end on a running VVD**: a 12-tile WebView grid now renders every tile with pixel-accurate frames and live `Selected`/`focused` state, where `describe` previously returned one opaque node. ### Before / after (live device) ``` # before view "Argent WebView Grid" [clickable,focused] (0,0,1,1) # after view "Argent WebView Grid" [clickable,focused] (0,0,1,1) header view "Argent WebView Grid" view "Selected:" view "Tile 3" view "| Activations:" view "1" view "1Tile 1" ... view "12Tile 12" (each with its own frame) ```	2026-07-07 10:05:42 +02:00
Piotr Krawiec	0bed3624a0	feat(vega): Amazon Vega (Fire TV) platform support (#366) ... ## Summary Adds **Vega (Amazon Fire TV)** as a third Argent platform alongside iOS and Android. Vega apps are React Native 0.72 + Hermes running on a QEMU **Virtual Device (VVD)**, driven by a **D-pad tv-remote rather than touch**. This PR extends Argent's platform abstraction so the existing tool surface gates correctly on `vega`, and adds the Vega-specific tools needed to actually drive a Fire TV app end to end. ### Platform abstraction - `Platform` gains `vega`; `DeviceKind` gains `vvd` (the QEMU VVD); `ToolCapability` gains a `vega { vvd, device }` block; `ToolDependency` gains `vega`. - `assertSupported()` selects the `vega` capability matrix for Vega devices; `dispatchByPlatform()` gains an optional `vega` branch, and a Vega dispatch with no branch throws `NotImplementedOnPlatformError` (501). - New `vega-cli` util (modeled on `adb.ts`) resolves `vega`/`kepler` from `PATH` or `~/vega/bin` with the same hardened exec/quoting; `check-deps` resolves and hints the `vega` dependency. - The `simulator-server` blueprint is never wired for `vega` (Vega uses `adb` + the `vega`/`kepler` CLI, not the proprietary binary). ### How Vega is wired - **Input / screenshot / describe** go over `adb` (input via `inputd-cli`, screenshots host-side via the Android emulator console, element tree via the on-device automation toolkit). - **App lifecycle** goes over the `vega`/`kepler` CLI. - v1 is **Virtual-Device-only**: every Vega capability is declared `vvd`. Physical Fire TV is out of scope and untested — the tool-server does not connect to or detect hardware, so `resolveDevice` classifies every `amazon-` serial as a VVD by shape. A physical device is therefore not classified as `device` and not rejected at the capability gate; supporting and gating real hardware is deferred to a version where it can be tested. - Vega runs **React Native 0.72**, which predates the new React Native DevTools. The CDP/Hermes debugger surface (`debugger-*`, React/native profilers) depends on integration points the new DevTools provides, so **debugging and profiling are out of scope for this PR** — Vega profiling/crash analysis is handled by the `amazon-devices-buildertools-mcp` server instead. ### Also includes - `argent-vega` skill documenting the full Vega workflow (target discovery, the describe → compute-path → `tv-remote` navigation loop, text injection). - Vega VVD end-to-end CI workflow. ## Feature support by platform Legend: ✅ supported · ❌ not supported · ⚠️ degraded / conditional · — not applicable | Feature (tool) | iOS | Android | Chromium | **Vega (VVD)** | Vega notes | |---|:---:|:---:|:---:|:---:|---| | **Device discovery** (`list-devices`) | ✅ | ✅ | ✅ | **✅** | Tagged `platform:"vega"`, `kind:"vvd"`; pass `serial` as `udid` | | **Boot / start device** | ✅ | ✅ | — | ✅ | | | **Launch app** (`launch-app`) | ✅ | ✅ | ✅ | **✅** | `bundleId` = interactive-component app id from `manifest.toml` | | **Restart app** (`restart-app`) | ✅ | ✅ | ✅ | **✅** | Terminate + relaunch | | **Reinstall app** (`reinstall-app`) | ✅ | ✅ | ❌ | **✅** | Uninstall + install a `.vpkg` | | **Screenshot** (`screenshot`) | ✅ | ✅ | ✅ | **✅** | Host-side via emulator console (needs `adb`); rotation ignored (fixed landscape) | | **Describe / element tree** (`describe`) | ✅ | ✅ | ✅ | **✅** | On-device automation toolkit; surfaces `[focused]`/`[selected]` + normalized frames | | **Touch gestures** (`gesture-tap`/`swipe`/`scroll`/`drag`/`custom`) | ✅ | ✅ | ✅ | **❌** | Vega is tv-remote-driven; gestures error — use `tv-remote` | | **Pinch / rotate gesture** (`gesture-pinch`/`gesture-rotate`) | ✅ | ❌ | ⚠️ | **❌** | N/A on a TV | | **Remote / D-pad** (`tv-remote`) | — | — | — | **✅** | Vega-only tool; single key, repeat, or a whole path in one call | | **Hardware buttons** (`button`) | ✅ | ✅ | — | **❌** | Remote keys live in `remote` instead | | **Text input** (`keyboard`) | ✅ | ✅ | ✅ | **✅** | Whole-string injection via `inputd-cli send_text` | | **Rotate orientation** (`rotate`) | ✅ | ✅ | — | **❌** | TV framebuffer is fixed landscape | | **Open URL / deep link** (`open-url`) | ✅ | ✅ | ✅ | **❌** | Not wired on Vega | | **JS debugger** (`debugger-*`) | ✅ | ✅ | ✅ | **❌** | RN 0.72 predates the new RN DevTools — out of scope for this PR | | **Native devtools / AX** (`native-describe-screen`, etc.) | ✅ | ❌ | — | **❌** | iOS-only | | **Native profiler** (`native-profiler-*`) | ✅ | ✅ | — | **❌** | Use `amazon-devices-buildertools-mcp` (Perfetto) instead | | **React profiler** (`react-profiler-*`) | ✅ | ✅ | — | **❌** | Out of scope; RN 0.72 / new RN DevTools not supported | | **Screenshot diff** (`screenshot-diff`) | ✅ | ✅ | ✅ | **✅** | Host-side image comparison; platform-agnostic | | **Flows record/replay** (`flow-*`) | ✅ | ✅ | ✅ | **✅** | Records any tool-call sequence | ## Test plan - [x] `list-devices` surfaces connected Vega devices (`platform:"vega"`, `kind:"vvd"`) - [x] `launch-app`, `restart-app`, `reinstall-app`, `list-installed-apps` work on Vega - [x] `tv-remote` drives focus (single key, `repeat`, and multi-key path) and `keyboard` injects whole strings - [x] `screenshot` returns a frame via the host-side VVD path - [x] `describe` returns the on-screen element tree with `[focused]` state and normalized coordinates - [x] Vega VVD e2e CI workflow passes - [ ] Test multiple platforms being driven at the same time - [x] Android + Vega - [ ] iOS + Vega - [ ] chrome + Vega --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 20:31:45 +02:00