skills/software-mansion__argent
1
0
Fork 0
mirror of https://github.com/software-mansion/argent.git synced 2026-09-14 19:27:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
flow-visual-settle
software-mansion__argent/packages/tool-server/test/fixtures/vega-page-source.xml
T
Piotr Krawiec 0bed3624a0 feat(vega): Amazon Vega (Fire TV) platform support (#366) 
## Summary

Adds **Vega (Amazon Fire TV)** as a third Argent platform alongside iOS
and Android. Vega apps are React Native 0.72 + Hermes running on a QEMU
**Virtual Device (VVD)**, driven by a **D-pad tv-remote rather than
touch**. This PR extends Argent's platform abstraction so the existing
tool surface gates correctly on `vega`, and adds the Vega-specific tools
needed to actually drive a Fire TV app end to end.

### Platform abstraction

- `Platform` gains `vega`; `DeviceKind` gains `vvd` (the QEMU VVD);
`ToolCapability` gains a `vega { vvd, device }` block; `ToolDependency`
gains `vega`.
- `assertSupported()` selects the `vega` capability matrix for Vega
devices; `dispatchByPlatform()` gains an optional `vega` branch, and a
Vega dispatch with no branch throws `NotImplementedOnPlatformError`
(501).
- New `vega-cli` util (modeled on `adb.ts`) resolves `vega`/`kepler`
from `PATH` or `~/vega/bin` with the same hardened exec/quoting;
`check-deps` resolves and hints the `vega` dependency.
- The `simulator-server` blueprint is never wired for `vega` (Vega uses
`adb` + the `vega`/`kepler` CLI, not the proprietary binary).

### How Vega is wired

- **Input / screenshot / describe** go over `adb` (input via
`inputd-cli`, screenshots host-side via the Android emulator console,
element tree via the on-device automation toolkit).
- **App lifecycle** goes over the `vega`/`kepler` CLI.
- v1 is **Virtual-Device-only**: every Vega capability is declared
`vvd`. Physical Fire TV is out of scope and untested — the tool-server
does not connect to or detect hardware, so `resolveDevice` classifies
every `amazon-` serial as a VVD by shape. A physical device is therefore
not classified as `device` and not rejected at the capability gate;
supporting and gating real hardware is deferred to a version where it
can be tested.
- Vega runs **React Native 0.72**, which predates the new React Native
DevTools. The CDP/Hermes debugger surface (`debugger-*`, React/native
profilers) depends on integration points the new DevTools provides, so
**debugging and profiling are out of scope for this PR** — Vega
profiling/crash analysis is handled by the
`amazon-devices-buildertools-mcp` server instead.

### Also includes

- `argent-vega` skill documenting the full Vega workflow (target
discovery, the describe → compute-path → `tv-remote` navigation loop,
text injection).
- Vega VVD end-to-end CI workflow.

## Feature support by platform

Legend: ✅ supported · ❌ not supported · ⚠️ degraded / conditional · —
not applicable

| Feature (tool) | iOS | Android | Chromium | **Vega (VVD)** | Vega
notes |
|---|:---:|:---:|:---:|:---:|---|
| **Device discovery** (`list-devices`) | ✅ | ✅ | ✅ | **✅** | Tagged
`platform:"vega"`, `kind:"vvd"`; pass `serial` as `udid` |
| **Boot / start device** | ✅ | ✅ | — | ✅ |  |
| **Launch app** (`launch-app`) | ✅ | ✅ | ✅ | **✅** | `bundleId` =
interactive-component app id from `manifest.toml` |
| **Restart app** (`restart-app`) | ✅ | ✅ | ✅ | **✅** | Terminate +
relaunch |
| **Reinstall app** (`reinstall-app`) | ✅ | ✅ | ❌ | **✅** | Uninstall +
install a `.vpkg` |
| **Screenshot** (`screenshot`) | ✅ | ✅ | ✅ | **✅** | Host-side via
emulator console (needs `adb`); rotation ignored (fixed landscape) |
| **Describe / element tree** (`describe`) | ✅ | ✅ | ✅ | **✅** |
On-device automation toolkit; surfaces `[focused]`/`[selected]` +
normalized frames |
| **Touch gestures** (`gesture-tap`/`swipe`/`scroll`/`drag`/`custom`) |
✅ | ✅ | ✅ | **❌** | Vega is tv-remote-driven; gestures error — use
`tv-remote` |
| **Pinch / rotate gesture** (`gesture-pinch`/`gesture-rotate`) | ✅ | ❌
| ⚠️ | **❌** | N/A on a TV |
| **Remote / D-pad** (`tv-remote`) | — | — | — | **✅** | Vega-only tool;
single key, repeat, or a whole path in one call |
| **Hardware buttons** (`button`) | ✅ | ✅ | — | **❌** | Remote keys live
in `remote` instead |
| **Text input** (`keyboard`) | ✅ | ✅ | ✅ | **✅** | Whole-string
injection via `inputd-cli send_text` |
| **Rotate orientation** (`rotate`) | ✅ | ✅ | — | **❌** | TV framebuffer
is fixed landscape |
| **Open URL / deep link** (`open-url`) | ✅ | ✅ | ✅ | **❌** | Not wired
on Vega |
| **JS debugger** (`debugger-*`) | ✅ | ✅ | ✅ | **❌** | RN 0.72 predates
the new RN DevTools — out of scope for this PR |
| **Native devtools / AX** (`native-describe-screen`, etc.) | ✅ | ❌ | —
| **❌** | iOS-only |
| **Native profiler** (`native-profiler-*`) | ✅ | ✅ | — | **❌** | Use
`amazon-devices-buildertools-mcp` (Perfetto) instead |
| **React profiler** (`react-profiler-*`) | ✅ | ✅ | — | **❌** | Out of
scope; RN 0.72 / new RN DevTools not supported |
| **Screenshot diff** (`screenshot-diff`) | ✅ | ✅ | ✅ | **✅** |
Host-side image comparison; platform-agnostic |
| **Flows record/replay** (`flow-*`) | ✅ | ✅ | ✅ | **✅** | Records any
tool-call sequence |

## Test plan

- [x] `list-devices` surfaces connected Vega devices (`platform:"vega"`,
`kind:"vvd"`)
- [x] `launch-app`, `restart-app`, `reinstall-app`,
`list-installed-apps` work on Vega
- [x] `tv-remote` drives focus (single key, `repeat`, and multi-key
path) and `keyboard` injects whole strings
- [x] `screenshot` returns a frame via the host-side VVD path
- [x] `describe` returns the on-screen element tree with `[focused]`
state and normalized coordinates
- [x] Vega VVD e2e CI workflow passes
- [ ] Test multiple platforms being driven at the same time
    - [x] Android + Vega
    - [ ] iOS + Vega
    - [ ] chrome + Vega

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 20:31:45 +02:00

4 lines
15 KiB
XML
Raw Permalink Blame History

<?xml version="1.0"?>
<root id="1"><app id="163208757249" appName="com.giolaq.multitv.vega"><traits><publisherRoot uccVersionMajor="0" uccVersionMinor="7" uccProtocolVersionMajor="0" toolkitVersion="1.0" parentElementId="0" uccProtocolVersionMinor="8" keplerAppId="9128" toolkitName="UIToolkit" applicationName="com.giolaq.multitv.vega"/></traits><window x="0" y="0" width="1920" height="1080" id="163208757250" index="0" alpha="100" zOrder="0" test_id="1"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100" zOrder="0"/><window id="3" index="0"/></traits><child x="0" y="0" width="1920" height="1080" id="163208757251" test_id="2"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="0" y="0" width="1920" height="1080" id="163208757252" test_id="336"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="0" y="0" width="1920" height="1080" id="163208757253" test_id="334"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="0" y="0" width="1920" height="1080" id="163208757254" test_id="328"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="0" y="0" width="1920" height="1080" id="163208757255" test_id="326"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="0" y="0" width="1920" height="120" id="163208757256" test_id="50"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="67" y="23" width="177" height="74" id="163208757257" selectable="true" selected="false" focusable="true" focused="false" test_id="14" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="19" hint=""/><focusable setFocus="20" focused="false"/></traits><child x="67" y="23" width="165" height="74" id="163208757258" test_id="12"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="102" y="42" width="95" height="36" id="163208757259" role="text" test_id="10"><text>Search</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child><child x="244" y="23" width="165" height="74" id="163208757260" selectable="true" selected="true" focusable="true" focused="false" test_id="22" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="true" selectFunction="21" hint=""/><focusable setFocus="22" focused="false"/></traits><child x="241" y="22" width="159" height="76" id="163208757261" test_id="20"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="277" y="41" width="89" height="38" id="163208757262" role="text" test_id="18"><text>Home</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child><child x="409" y="23" width="205" height="74" id="163208757263" selectable="true" selected="false" focusable="true" focused="false" test_id="30" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="23" hint=""/><focusable setFocus="24" focused="false"/></traits><child x="409" y="23" width="193" height="74" id="163208757264" test_id="28"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="444" y="42" width="124" height="36" id="163208757265" role="text" test_id="26"><text>Libraries</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child><child x="614" y="23" width="227" height="74" id="163208757266" selectable="true" selected="false" focusable="true" focused="false" test_id="38" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="25" hint=""/><focusable setFocus="26" focused="false"/></traits><child x="614" y="23" width="215" height="74" id="163208757267" test_id="36"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="649" y="42" width="146" height="36" id="163208757268" role="text" test_id="34"><text>Favourites</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child><child x="841" y="23" width="199" height="74" id="163208757269" selectable="true" selected="false" focusable="true" focused="false" test_id="46" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="27" hint=""/><focusable setFocus="28" focused="false"/></traits><child x="841" y="23" width="187" height="74" id="163208757270" test_id="44"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="876" y="42" width="117" height="36" id="163208757271" role="text" test_id="42"><text>Settings</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child><child x="0" y="120" width="1920" height="960" id="163208757272" test_id="322"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="0" y="120" width="1920" height="922" id="163208757273" test_id="320"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="0" y="120" width="1920" height="1960" id="163208757274" test_id="318"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="0" y="176" width="1920" height="48" id="163208757275" role="text" test_id="54"><text>Kontynuuj odtwarzanie</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="67" y="240" width="1000" height="420" id="163208757276" test_id="98"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="67" y="240" width="240" height="350" id="163208757277" selectable="true" selected="true" focusable="true" focused="false" test_id="62" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="true" selectFunction="29" hint=""/><focusable setFocus="30" focused="false"/></traits><child x="59" y="226" width="236" height="378" id="163208757278" test_id="58"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="65" y="232" width="225" height="365" id="163208757279" role="image" test_id="56"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child><child x="317" y="240" width="240" height="350" id="163208757280" test_id="74"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="317" y="240" width="240" height="350" id="163208757281" selectable="true" selected="false" focusable="true" focused="false" test_id="72" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="31" hint=""/><focusable setFocus="32" focused="false"/></traits><child x="317" y="240" width="220" height="350" id="163208757282" test_id="68"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="322" y="245" width="210" height="340" id="163208757283" role="image" test_id="66"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child></child><child x="567" y="240" width="240" height="350" id="163208757284" test_id="84"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="567" y="240" width="240" height="350" id="163208757285" selectable="true" selected="false" focusable="true" focused="false" test_id="82" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="33" hint=""/><focusable setFocus="34" focused="false"/></traits><child x="567" y="240" width="220" height="350" id="163208757286" test_id="78"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="572" y="245" width="210" height="340" id="163208757287" role="image" test_id="76"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child></child><child x="817" y="240" width="240" height="350" id="163208757288" test_id="94"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="817" y="240" width="240" height="350" id="163208757289" selectable="true" selected="false" focusable="true" focused="false" test_id="92" role="button"><traits><actions><actions/></actions></traits><traits><compoundComponent mergeDescendants="true"/><visibility alpha="100"/><selectable selected="false" selectFunction="35" hint=""/><focusable setFocus="36" focused="false"/></traits><child x="817" y="240" width="220" height="350" id="163208757290" test_id="88"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="822" y="245" width="210" height="340" id="163208757291" role="image" test_id="86"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child></child></child><child x="0" y="660" width="1920" height="380" id="163208757292" test_id="144"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="90" y="680" width="1740" height="300" id="163208757293" test_id="142"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits><child x="90" y="680" width="533" height="300" id="163208757294" role="image" test_id="106"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="663" y="680" width="1167" height="48" id="163208757295" role="text" test_id="110"><text>Jungle Book</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="663" y="740" width="1167" height="26" id="163208757296" role="text" test_id="114"><text>1942 &#xB7; NR &#xB7; 1h 45m &#xB7; &#x2605; 6.3</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="663" y="782" width="122" height="40" id="163208757297" test_id="120"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="680" y="791" width="89" height="22" id="163208757298" role="text" test_id="118"><text>Adventure</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="797" y="782" width="99" height="40" id="163208757299" test_id="126"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="814" y="791" width="66" height="22" id="163208757300" role="text" test_id="124"><text>Fantasy</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="908" y="782" width="91" height="40" id="163208757301" test_id="132"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="925" y="791" width="57" height="22" id="163208757302" role="text" test_id="130"><text>Family</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="663" y="838" width="1167" height="102" id="163208757303" role="text" test_id="138"><text>Mowgli, lost in the jungle when a toddler, raised by wolves, years later happens upon his human village and reconnects with its inhabitants, including his widowed mother. Continuing to maintain a relationship with the jungle, adventures follow.</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child><child x="0" y="1056" width="1920" height="48" id="163208757304" role="text" test_id="150"><text>Do obejrzenia</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child></child><child x="0" y="1080" width="0" height="0" id="163208757305" test_id="332"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child></child></child></window></app><app id="4294967297" appName="com.amazon.keplerlauncherapp"><traits><publisherRoot uccVersionMajor="0" uccVersionMinor="7" uccProtocolVersionMajor="0" toolkitVersion="1.0" parentElementId="0" uccProtocolVersionMinor="8" keplerAppId="1" toolkitName="UIToolkit" applicationName="com.amazon.keplerlauncherapp"/></traits><window x="0" y="0" width="1920" height="1080" id="4294967298" index="2147483647" alpha="100" zOrder="2147483647" test_id="1"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100" zOrder="2147483647"/><window id="3" index="2147483647"/></traits><child x="0" y="0" width="1920" height="1080" id="4294967299" role="image" test_id="2"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="1486" y="60" width="314" height="108" id="4294967300" clickable="true" focusable="true" focused="false" test_id="8" role="button"><traits><compoundComponent mergeDescendants="true"/><voiceAssistant entityType="AMAZON.Thing"/><visibility alpha="100"/><focusable setFocus="21" focused="false"/><tappable handler="22"/></traits><child x="1526" y="80" width="245" height="68" id="4294967301" role="text" test_id="6"><text>Register this device
(optional)</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child><child x="1800" y="88" width="50" height="52" id="4294967302" clickable="true" focusable="true" focused="false" test_id="14" role="button"><traits><compoundComponent mergeDescendants="true"/><voiceAssistant entityType="AMAZON.Thing"/><visibility alpha="100"/><focusable setFocus="23" focused="false"/><tappable handler="24"/></traits><child x="1815" y="88" width="36" height="52" id="4294967303" role="text" test_id="12"><text>&#x24D8;</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></child><child x="856" y="317" width="208" height="208" id="4294967304" role="image" test_id="22"><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="531" y="625" width="858" height="78" id="4294967305" role="text" test_id="26"><text>Kepler Virtual Device is ready</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child><child x="442" y="723" width="1036" height="40" id="4294967306" role="text" test_id="30"><text>You can now launch your app and start testing with the virtual device</text><traits><compoundComponent mergeDescendants="false"/><visibility alpha="100"/></traits></child></window></app></root>
Reference in New Issue View Git Blame Copy Permalink