Files
Mateusz Ścianek b8afbed183 feat(tv): Apple TV (tvOS) and Android TV support (#374)
Adds end-to-end support for driving **Apple TV (tvOS)** and **Android TV
(leanback)** targets through argent, using a focus-driven interaction
model (TVs have no touchscreen — interaction is moving a focus highlight
with the remote).

This PR was reworked after review to **reuse the existing cross-platform
tool surface and the `tv-remote` tool from the Vega/Fire TV work
(#366)** rather than ship a separate family of `tv-*` tools.

## What this delivers

TV targets are driven by the **same cross-platform tools as every other
platform**, routed through the existing `dispatchByPlatform` fork
mechanism — there are no dedicated
`tv-describe`/`tv-navigate`/`tv-type`/`tv-set-focus` tools:

- **`describe`** — on a `runtimeKind: "tv"` target, returns the
focus-driven view (focused + focusable elements) instead of a tap tree.
On Android TV it auto-falls-back to the full `uiautomator` tree when the
RN focus engine exposes nothing.
- **`tv-remote`** — the cross-platform remote/D-pad tool from #366, now
extended with `ios` (Apple TV HID daemon) and `android` (Android TV adb
keyevents) branches alongside the existing `vega` branch. Single button,
a `repeat`, or a whole path (`["up","right","select"]`) in one call.
- **`keyboard`** — types into the focused field on a TV target (named
keys are rejected there — they're navigation, which belongs to
`tv-remote`).
- **`button`** stays hardware-only (phones/tablets); it is not a TV
tool.

**Apple TV** runs two native daemons (in-sim AX service + host-side HID
daemon), shipped via the `argent-private` submodule. **Android TV**
reuses the same tool surface, adb-backed (`input keyevent`, `uiautomator
dump`, `input text`).

## Remote vocabulary parity

`tv-remote` exposes the full 16-button vocabulary
(`up`/`down`/`left`/`right`/`select`/`back`/`home`/`menu`/`playPause`/`rewind`/`fastForward`/`next`/`previous`/`volumeUp`/`volumeDown`/`mute`):

| | Apple TV | Android TV | Vega |
|---|---|---|---|
| D-pad / select / back / menu / home / playPause | ✅ | ✅ | ✅ |
| media-transport + volume/mute | ❌ rejected | ✅ | ✅ |

Media-transport/volume keys **genuinely work on Android TV** (real
keycodes — verified live: `volumeUp` moved `STREAM_MUSIC`, `mute`
toggled per `dumpsys audio`). On the **Apple TV simulator** they are
**rejected with a clear error**: on-device testing confirmed the tvOS
sim's HID stack silently drops Consumer-Control events, so returning
success would be a lie.

## Key design points
- **tvOS UDIDs are UUID-shaped and indistinguishable from iOS by
shape.** Handled with `runtimeKind` detection (tvOS runtime string for
Apple TV; `pm list features` leanback/television for Android TV — *not*
`ro.build.characteristics`, which lies on TV emulators).
- Converted launch-app / restart-app / screenshot / screenshot-diff /
run-sequence from **eager** to **lazy** service resolution, so a tvOS
target never spins up (and hangs on) the iOS-only simulator-server /
native-devtools blueprints.
- Native injection selects the platform-matched (TVOSSIMULATOR) dylib
slice; daemons recycle across sim reboots and app relaunches so
injection and focus survive.
- SKILL.md footprint minimized per review: the two TV skills were
collapsed into a single lean **`argent-tv-interact`** (~40 lines).

## Verification
- `npm run build`, ESLint, Prettier, and the full tool-server suite
(**1565 tests**) all pass; CI green (incl. the Apple TV / Android TV /
Vega e2e jobs).
- Core flows verified live against real apps on both an Apple TV 4K
simulator and a Google ATV emulator.

## Dependency / merge order
Pins the `argent-private` submodule to the tip of its `feat/tv-support`
branch (software-mansion/argent-private#20). **Merge that PR to `main`
first, then bump the submodule pointer here to the merged SHA before
merging this PR.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 14:59:37 +02:00

171 lines
6.8 KiB
TypeScript

import { describe, it, expect, vi, beforeEach } from "vitest";
// Stub the device round-trip; keep REMOTE_BUTTONS real so the schema enum is genuine.
const injectVegaButtons = vi.fn();
vi.mock("../src/utils/vega-input", async (importActual) => {
const actual = await importActual<typeof import("../src/utils/vega-input")>();
return { ...actual, injectVegaButtons: (...a: unknown[]) => injectVegaButtons(...a) };
});
import type { ToolContext } from "@argent/registry";
import { createTvRemoteTool } from "../src/tools/tv-remote";
import { UnsupportedOperationError } from "../src/utils/capability";
import type { TvControlApi } from "../src/blueprints/tv-control-types";
// resolveDevice is pure (shape-based), so real udids drive the platform branch.
const VEGA_UDID = "amazon-test01";
const TVOS_UDID = "DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD";
const ANDROID_TV_SERIAL = "emulator-5556";
// A registry whose resolveService hands back a navigate spy for the focus-driven
// TV backends (Apple TV / Android TV). Vega never resolves a service — it injects
// over adb directly.
function makeRegistry() {
const navigate = vi.fn().mockResolvedValue(undefined);
const api = { navigate } as unknown as TvControlApi;
const registry = { resolveService: vi.fn(async () => api) } as never;
return { registry, navigate };
}
beforeEach(() => {
injectVegaButtons.mockReset();
injectVegaButtons.mockResolvedValue(undefined);
});
describe("tv-remote execute — Vega", () => {
it("flattens `repeat` over a single button", async () => {
const { registry } = makeRegistry();
const res = await createTvRemoteTool(registry).execute!(
{},
{ udid: VEGA_UDID, button: "down", repeat: 3 }
);
expect(injectVegaButtons).toHaveBeenCalledWith(["down", "down", "down"]);
expect(res).toEqual({ pressed: ["down", "down", "down"], count: 3 });
});
it("flattens `repeat` over a button path and reports the right count", async () => {
const { registry } = makeRegistry();
const res = await createTvRemoteTool(registry).execute!(
{},
{ udid: VEGA_UDID, button: ["up", "down"], repeat: 2 }
);
expect(injectVegaButtons).toHaveBeenCalledWith(["up", "down", "up", "down"]);
expect(res).toEqual({ pressed: ["up", "down", "up", "down"], count: 4 });
});
it("defaults to a single press when no repeat is given", async () => {
const { registry } = makeRegistry();
const res = await createTvRemoteTool(registry).execute!(
{},
{ udid: VEGA_UDID, button: "select" }
);
expect(injectVegaButtons).toHaveBeenCalledWith(["select"]);
expect(res.count).toBe(1);
});
it("supports Vega-only media/volume buttons", async () => {
const { registry } = makeRegistry();
const res = await createTvRemoteTool(registry).execute!(
{},
{ udid: VEGA_UDID, button: "volumeUp" }
);
expect(injectVegaButtons).toHaveBeenCalledWith(["volumeUp"]);
expect(res.count).toBe(1);
});
});
describe("tv-remote execute — Apple TV / Android TV (focus-driven)", () => {
it("routes a D-pad path on Apple TV through the tv-control navigate", async () => {
const { registry, navigate } = makeRegistry();
const res = await createTvRemoteTool(registry).execute!(
{},
{ udid: TVOS_UDID, button: ["right", "select"] }
);
expect(navigate.mock.calls.map((c) => c[0])).toEqual(["right", "select"]);
expect(injectVegaButtons).not.toHaveBeenCalled();
expect(res).toEqual({ pressed: ["right", "select"], count: 2 });
});
it("passes each remote button straight through to navigate on Android TV", async () => {
const { registry, navigate } = makeRegistry();
await createTvRemoteTool(registry).execute!(
{},
{ udid: ANDROID_TV_SERIAL, button: ["back", "menu"] }
);
expect(navigate.mock.calls.map((c) => c[0])).toEqual(["back", "menu"]);
});
it("supports media-transport / volume buttons on Android TV (real adb keyevents)", async () => {
const { registry, navigate } = makeRegistry();
const res = await createTvRemoteTool(registry).execute!(
{},
{ udid: ANDROID_TV_SERIAL, button: ["fastForward", "volumeUp", "mute"] }
);
expect(navigate.mock.calls.map((c) => c[0])).toEqual(["fastForward", "volumeUp", "mute"]);
expect(res).toEqual({ pressed: ["fastForward", "volumeUp", "mute"], count: 3 });
});
it("rejects inert media / volume buttons on Apple TV without pressing anything", async () => {
const { registry, navigate } = makeRegistry();
await expect(
createTvRemoteTool(registry).execute!({}, { udid: TVOS_UDID, button: ["right", "volumeUp"] })
).rejects.toBeInstanceOf(UnsupportedOperationError);
// Whole path rejected up front — no partial execution.
expect(navigate).not.toHaveBeenCalled();
});
it("stops pressing mid-path when the invocation signal is already aborted", async () => {
const { registry, navigate } = makeRegistry();
const controller = new AbortController();
controller.abort();
await expect(
createTvRemoteTool(registry).execute!(
{},
{ udid: ANDROID_TV_SERIAL, button: ["up", "down", "select"] },
{ artifacts: {}, signal: controller.signal } as unknown as ToolContext
)
).rejects.toThrow();
// Aborted before the first press — nothing fired at the held device.
expect(navigate).not.toHaveBeenCalled();
});
it("aborts partway through a long path once the client disconnects", async () => {
const { registry, navigate } = makeRegistry();
const controller = new AbortController();
// Abort after the 2nd press lands; the loop must not fire the remaining ones.
navigate.mockImplementation(async () => {
if (navigate.mock.calls.length >= 2) controller.abort();
});
await expect(
createTvRemoteTool(registry).execute!(
{},
{ udid: ANDROID_TV_SERIAL, button: ["up", "up", "up", "up", "up"] },
{ artifacts: {}, signal: controller.signal } as unknown as ToolContext
)
).rejects.toThrow();
// Pressed exactly the two before the abort, then bailed — not all five.
expect(navigate).toHaveBeenCalledTimes(2);
});
});
describe("tv-remote button schema", () => {
const schema = createTvRemoteTool({} as never).zodSchema!;
it("coerces a JSON-array string back to an array", () => {
expect(schema.parse({ udid: "x", button: '["up","down"]' }).button).toEqual(["up", "down"]);
});
it("coerces a comma-separated string back to an array", () => {
expect(schema.parse({ udid: "x", button: "up, down" }).button).toEqual(["up", "down"]);
});
it("keeps a single button as a scalar", () => {
expect(schema.parse({ udid: "x", button: "select" }).button).toBe("select");
});
it("rejects an unknown button and an empty path", () => {
expect(() => schema.parse({ udid: "x", button: "nope" })).toThrow();
expect(() => schema.parse({ udid: "x", button: [] })).toThrow();
});
});