skills/software-mansion__argent
1
0
Fork 0
mirror of https://github.com/software-mansion/argent.git synced 2026-09-14 19:27:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
641 Commits 225 Branches 38 Tags
flow-visual-settle
Commit Graph

3 Commits

Author SHA1 Message Date
Piotr Krawiec ab35c5653e fix(describe): descend into Vega WebView (Chromium sub-app) (#474) (#475) 
## What & why

Fixes #474. On Vega, `describe` collapsed a `WebView` into a single
opaque node — only the page `<title>` surfaced, none of the DOM.

Root cause: a `WebView` embeds a nested `<app toolkitName="Chromium">`
whose text nodes carry **no `role`** and keep their string in a `<text>`
**element** child. The native RN UIToolkit instead tags text nodes
`role="text"` with **inline** text. `isMeaningful` only counted inline
`node.text`, so every WebView text node looked empty → the whole web DOM
(grid tiles, headings, labels) was judged unmeaningful and flattened
away.

Notably the underlying automation toolkit *does* publish the full
Chromium a11y tree (confirmed by querying the same backend through the
Appium kepler driver) — this was purely a parser-side traversal gap.

## The fix (`source-parser.ts`)

1. **`ownText(node)`** — a node's text is now its inline text **plus**
the text of its direct `<text>` element children. Both `isMeaningful`
and `labelOf` use this one notion (they were already asymmetric:
`labelOf` read `<text>` children, `isMeaningful` didn't).
2. **Sub-span collapse in `convert`** — the Chromium a11y tree
aggregates a container's accessible name **and** repeats it as child
spans (a tile `"10Tile 10"` nests `"10"` + `"Tile 10"`). Children are
dropped only when they are **all plain text spans whose in-order labels
exactly reconstruct the parent's aggregated label** (Chromium joins
inline spans with no separator). Exact reconstruction — not substring
containment — means a distinct `"Play"` under `"Playlist"` (or `"Tile
1"` under `"Tile 10"`) is never swallowed, and a real list whose
container name joins rows with spaces (`"Home Settings About"`) never
reconstructs from its rows, so separately-navigable rows keep their own
frames.

## Verification

- New regression tests in `vega-source-parser.test.ts` (real-device
fixture `vega-webview-grid.xml`): all 12 grid tiles surface; frames
normalize against the window; focused/selected state preserved; exact
sub-spans collapse; **distinct substring child kept**
(`Play`/`Playlist`); **distinct list rows kept with their own frames**
(`Home`/`Settings`/`About`); labelless structural container keeps all
children; structural/interactive same-text child kept.
- Full `tool-server` suite green (one unrelated pre-existing failure:
`await-ui-element.test.ts`, fails on `main` too).
- **End-to-end on a running VVD**: a 12-tile WebView grid now renders
every tile with pixel-accurate frames and live `Selected`/`focused`
state, where `describe` previously returned one opaque node.

### Before / after (live device)

```
# before
view "Argent WebView Grid" [clickable,focused]  (0,0,1,1)

# after
view "Argent WebView Grid" [clickable,focused]  (0,0,1,1)
  header
    view "Argent WebView Grid"
    view "Selected:"  view "Tile 3"  view "|  Activations:"  view "1"
  view "1Tile 1" ... view "12Tile 12"   (each with its own frame)
```
 2026-07-07 10:05:42 +02:00
Ignacy Łątka 05e619e55d ci: device E2E smoke tests for every hosted-CI platform (#371) 
## What

`wayland-e2e.yml` boots an Android AVD on Linux under headless Weston
and asserts the full pipeline works without a display. It guards exactly
one cell of the support matrix. This adds the same
boot→screenshot→gesture-tap smoke test for every other (target × host)
cell that can run on GitHub-hosted runners.

| target | host | job | status |
|---|---|---|---|
| Android emulator | Linux (KVM) | `wayland-e2e.yml` (existing) | ✅ |
| iOS simulator | macOS | `ios-sim-macos` | ✅ |
| Chromium / Electron | Linux (Xvfb) | `chromium-linux` | ✅ |
| Chromium / Electron | macOS | `chromium-macos` | ✅ |

## How

- **`scripts/e2e/drive-device.sh`** — shared driver. Boots one device
through the tool-server, then asserts `booted:true` → screenshot has
real (non-blank) pixels → `gesture-tap` round-trips. Every tool takes
the device id as `udid` and screenshots come back as the same
`data.image.hostPath` envelope on all platforms, so the body is uniform;
each job just supplies the cell-specific boot JSON + device id.
- **`packages/tool-server/test/fixtures/electron-smoke-app/`** — minimal
self-contained Electron app the Chromium jobs point `electronAppPath`
at. It vendors its own electron via its own lockfile (kept out of the
repo root lockfile); CI `npm ci`s it so `boot-electron` resolves
`./node_modules/.bin/electron`.
- **`.github/workflows/e2e-device-smoke.yml`** — three jobs,
`workflow_dispatch` + path-filtered `pull_request`. iOS downloads the
darwin native binaries (injection must succeed for `bootIos` to report
`booted:true`); Chromium-Linux runs under Xvfb with `--no-sandbox`. The
tool-server is started with `TS_NODE_TRANSPILE_ONLY=1` so cold ts-node
startup doesn't flakily exceed the readiness poll on loaded macOS
runners (types are still enforced by the typecheck job).

## Known gaps (supported on real machines, not runnable on hosted
runners)

- **Android emulator on macOS** — the arm64 emulator needs HVF
(`-enable-hvf`), but hosted macOS runners are themselves VMs with no
nested virtualization. Confirmed by running it: qemu dies with `HVF
error: HV_UNSUPPORTED` ~18s into boot, regardless of GPU mode or RAM.
The Android boot/screenshot/tap path is already regression-guarded on
Linux via KVM (`wayland-e2e.yml`), so macOS adds no runnable coverage.
- **Physical devices** — real iOS (CoreDevice HID) and real Android
(adb-USB) have no attached hardware on hosted runners.

Both would need self-hosted runners (a Mac exposing HVF / wired-up
devices). Documented in the workflow header.

## Verification

- All checks green: the 3 device jobs + ESLint + Prettier + Unit tests.
- Locally (before CI): ran `drive-device.sh` end-to-end against a real
tool-server → boot `booted:true`, screenshot **142 KB** (floor 20 KB),
tap `tapped:true`, exit 0; and confirmed via the Argent stack that a tap
at (0.5, 0.5) lands on the fixture's centered button and increments its
counter.
 2026-06-26 17:44:17 +02:00
Piotr Krawiec 0bed3624a0 feat(vega): Amazon Vega (Fire TV) platform support (#366) 
## Summary

Adds **Vega (Amazon Fire TV)** as a third Argent platform alongside iOS
and Android. Vega apps are React Native 0.72 + Hermes running on a QEMU
**Virtual Device (VVD)**, driven by a **D-pad tv-remote rather than
touch**. This PR extends Argent's platform abstraction so the existing
tool surface gates correctly on `vega`, and adds the Vega-specific tools
needed to actually drive a Fire TV app end to end.

### Platform abstraction

- `Platform` gains `vega`; `DeviceKind` gains `vvd` (the QEMU VVD);
`ToolCapability` gains a `vega { vvd, device }` block; `ToolDependency`
gains `vega`.
- `assertSupported()` selects the `vega` capability matrix for Vega
devices; `dispatchByPlatform()` gains an optional `vega` branch, and a
Vega dispatch with no branch throws `NotImplementedOnPlatformError`
(501).
- New `vega-cli` util (modeled on `adb.ts`) resolves `vega`/`kepler`
from `PATH` or `~/vega/bin` with the same hardened exec/quoting;
`check-deps` resolves and hints the `vega` dependency.
- The `simulator-server` blueprint is never wired for `vega` (Vega uses
`adb` + the `vega`/`kepler` CLI, not the proprietary binary).

### How Vega is wired

- **Input / screenshot / describe** go over `adb` (input via
`inputd-cli`, screenshots host-side via the Android emulator console,
element tree via the on-device automation toolkit).
- **App lifecycle** goes over the `vega`/`kepler` CLI.
- v1 is **Virtual-Device-only**: every Vega capability is declared
`vvd`. Physical Fire TV is out of scope and untested — the tool-server
does not connect to or detect hardware, so `resolveDevice` classifies
every `amazon-` serial as a VVD by shape. A physical device is therefore
not classified as `device` and not rejected at the capability gate;
supporting and gating real hardware is deferred to a version where it
can be tested.
- Vega runs **React Native 0.72**, which predates the new React Native
DevTools. The CDP/Hermes debugger surface (`debugger-*`, React/native
profilers) depends on integration points the new DevTools provides, so
**debugging and profiling are out of scope for this PR** — Vega
profiling/crash analysis is handled by the
`amazon-devices-buildertools-mcp` server instead.

### Also includes

- `argent-vega` skill documenting the full Vega workflow (target
discovery, the describe → compute-path → `tv-remote` navigation loop,
text injection).
- Vega VVD end-to-end CI workflow.

## Feature support by platform

Legend: ✅ supported · ❌ not supported · ⚠️ degraded / conditional · —
not applicable

| Feature (tool) | iOS | Android | Chromium | **Vega (VVD)** | Vega
notes |
|---|:---:|:---:|:---:|:---:|---|
| **Device discovery** (`list-devices`) | ✅ | ✅ | ✅ | **✅** | Tagged
`platform:"vega"`, `kind:"vvd"`; pass `serial` as `udid` |
| **Boot / start device** | ✅ | ✅ | — | ✅ |  |
| **Launch app** (`launch-app`) | ✅ | ✅ | ✅ | **✅** | `bundleId` =
interactive-component app id from `manifest.toml` |
| **Restart app** (`restart-app`) | ✅ | ✅ | ✅ | **✅** | Terminate +
relaunch |
| **Reinstall app** (`reinstall-app`) | ✅ | ✅ | ❌ | **✅** | Uninstall +
install a `.vpkg` |
| **Screenshot** (`screenshot`) | ✅ | ✅ | ✅ | **✅** | Host-side via
emulator console (needs `adb`); rotation ignored (fixed landscape) |
| **Describe / element tree** (`describe`) | ✅ | ✅ | ✅ | **✅** |
On-device automation toolkit; surfaces `[focused]`/`[selected]` +
normalized frames |
| **Touch gestures** (`gesture-tap`/`swipe`/`scroll`/`drag`/`custom`) |
✅ | ✅ | ✅ | **❌** | Vega is tv-remote-driven; gestures error — use
`tv-remote` |
| **Pinch / rotate gesture** (`gesture-pinch`/`gesture-rotate`) | ✅ | ❌
| ⚠️ | **❌** | N/A on a TV |
| **Remote / D-pad** (`tv-remote`) | — | — | — | **✅** | Vega-only tool;
single key, repeat, or a whole path in one call |
| **Hardware buttons** (`button`) | ✅ | ✅ | — | **❌** | Remote keys live
in `remote` instead |
| **Text input** (`keyboard`) | ✅ | ✅ | ✅ | **✅** | Whole-string
injection via `inputd-cli send_text` |
| **Rotate orientation** (`rotate`) | ✅ | ✅ | — | **❌** | TV framebuffer
is fixed landscape |
| **Open URL / deep link** (`open-url`) | ✅ | ✅ | ✅ | **❌** | Not wired
on Vega |
| **JS debugger** (`debugger-*`) | ✅ | ✅ | ✅ | **❌** | RN 0.72 predates
the new RN DevTools — out of scope for this PR |
| **Native devtools / AX** (`native-describe-screen`, etc.) | ✅ | ❌ | —
| **❌** | iOS-only |
| **Native profiler** (`native-profiler-*`) | ✅ | ✅ | — | **❌** | Use
`amazon-devices-buildertools-mcp` (Perfetto) instead |
| **React profiler** (`react-profiler-*`) | ✅ | ✅ | — | **❌** | Out of
scope; RN 0.72 / new RN DevTools not supported |
| **Screenshot diff** (`screenshot-diff`) | ✅ | ✅ | ✅ | **✅** |
Host-side image comparison; platform-agnostic |
| **Flows record/replay** (`flow-*`) | ✅ | ✅ | ✅ | **✅** | Records any
tool-call sequence |

## Test plan

- [x] `list-devices` surfaces connected Vega devices (`platform:"vega"`,
`kind:"vvd"`)
- [x] `launch-app`, `restart-app`, `reinstall-app`,
`list-installed-apps` work on Vega
- [x] `tv-remote` drives focus (single key, `repeat`, and multi-key
path) and `keyboard` injects whole strings
- [x] `screenshot` returns a frame via the host-side VVD path
- [x] `describe` returns the on-screen element tree with `[focused]`
state and normalized coordinates
- [x] Vega VVD e2e CI workflow passes
- [ ] Test multiple platforms being driven at the same time
    - [x] Android + Vega
    - [ ] iOS + Vega
    - [ ] chrome + Vega

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-22 20:31:45 +02:00