skills/software-mansion__argent
1
0
Fork 0
mirror of https://github.com/software-mansion/argent.git synced 2026-09-14 19:27:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
641 Commits 225 Branches 38 Tags
flow-visual-settle
Commit Graph

5 Commits

Author SHA1 Message Date
Mateusz Ścianek b8afbed183 feat(tv): Apple TV (tvOS) and Android TV support (#374) 
Adds end-to-end support for driving **Apple TV (tvOS)** and **Android TV
(leanback)** targets through argent, using a focus-driven interaction
model (TVs have no touchscreen — interaction is moving a focus highlight
with the remote).

This PR was reworked after review to **reuse the existing cross-platform
tool surface and the `tv-remote` tool from the Vega/Fire TV work
(#366)** rather than ship a separate family of `tv-*` tools.

## What this delivers

TV targets are driven by the **same cross-platform tools as every other
platform**, routed through the existing `dispatchByPlatform` fork
mechanism — there are no dedicated
`tv-describe`/`tv-navigate`/`tv-type`/`tv-set-focus` tools:

- **`describe`** — on a `runtimeKind: "tv"` target, returns the
focus-driven view (focused + focusable elements) instead of a tap tree.
On Android TV it auto-falls-back to the full `uiautomator` tree when the
RN focus engine exposes nothing.
- **`tv-remote`** — the cross-platform remote/D-pad tool from #366, now
extended with `ios` (Apple TV HID daemon) and `android` (Android TV adb
keyevents) branches alongside the existing `vega` branch. Single button,
a `repeat`, or a whole path (`["up","right","select"]`) in one call.
- **`keyboard`** — types into the focused field on a TV target (named
keys are rejected there — they're navigation, which belongs to
`tv-remote`).
- **`button`** stays hardware-only (phones/tablets); it is not a TV
tool.

**Apple TV** runs two native daemons (in-sim AX service + host-side HID
daemon), shipped via the `argent-private` submodule. **Android TV**
reuses the same tool surface, adb-backed (`input keyevent`, `uiautomator
dump`, `input text`).

## Remote vocabulary parity

`tv-remote` exposes the full 16-button vocabulary
(`up`/`down`/`left`/`right`/`select`/`back`/`home`/`menu`/`playPause`/`rewind`/`fastForward`/`next`/`previous`/`volumeUp`/`volumeDown`/`mute`):

| | Apple TV | Android TV | Vega |
|---|---|---|---|
| D-pad / select / back / menu / home / playPause | ✅ | ✅ | ✅ |
| media-transport + volume/mute | ❌ rejected | ✅ | ✅ |

Media-transport/volume keys **genuinely work on Android TV** (real
keycodes — verified live: `volumeUp` moved `STREAM_MUSIC`, `mute`
toggled per `dumpsys audio`). On the **Apple TV simulator** they are
**rejected with a clear error**: on-device testing confirmed the tvOS
sim's HID stack silently drops Consumer-Control events, so returning
success would be a lie.

## Key design points
- **tvOS UDIDs are UUID-shaped and indistinguishable from iOS by
shape.** Handled with `runtimeKind` detection (tvOS runtime string for
Apple TV; `pm list features` leanback/television for Android TV — *not*
`ro.build.characteristics`, which lies on TV emulators).
- Converted launch-app / restart-app / screenshot / screenshot-diff /
run-sequence from **eager** to **lazy** service resolution, so a tvOS
target never spins up (and hangs on) the iOS-only simulator-server /
native-devtools blueprints.
- Native injection selects the platform-matched (TVOSSIMULATOR) dylib
slice; daemons recycle across sim reboots and app relaunches so
injection and focus survive.
- SKILL.md footprint minimized per review: the two TV skills were
collapsed into a single lean **`argent-tv-interact`** (~40 lines).

## Verification
- `npm run build`, ESLint, Prettier, and the full tool-server suite
(**1565 tests**) all pass; CI green (incl. the Apple TV / Android TV /
Vega e2e jobs).
- Core flows verified live against real apps on both an Apple TV 4K
simulator and a Google ATV emulator.

## Dependency / merge order
Pins the `argent-private` submodule to the tip of its `feat/tv-support`
branch (software-mansion/argent-private#20). **Merge that PR to `main`
first, then bump the submodule pointer here to the merged SHA before
merging this PR.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-01 14:59:37 +02:00
filip131311 8718d82905 feat(android): drive connected physical Android devices (#342) 
## What

Adds support for driving a **physically connected Android phone** (USB
or wireless adb), not just emulators. Once a phone with USB debugging is
attached, it shows up in `list-devices` and every existing interaction
tool works against it — no new tools, no per-tool special-casing.

## How it works

A physical phone is driven by a different simulator-server controller
than an emulator: it deploys the Android Studio **screen-sharing agent**
to the device and streams **H264 over adb** (`android_device`
subcommand), whereas an emulator streams decoded RGB over the emulator
gRPC bridge (`android` subcommand). Both controllers expose the **same**
HTTP/WebSocket/stdin API, so the only thing that has to change on the
argent side is *which subcommand the device's service blueprint spawns*.
Everything downstream — `screenshot`, the `gesture-*` family, `button`,
`keyboard` (simulator-server) and `describe`, `launch-app`,
`restart-app`, `reinstall-app`, `open-url` (adb) — is unchanged.

- **Classification** (`utils/device-info.ts`): a non-`emulator-*`
Android serial (a USB hardware serial, or an `ip:port` from wireless
debugging) resolves to kind `device` instead of `emulator`, purely from
serial shape — the same signal radon uses to detect connected phones,
and with no extra adb round-trip on the hot path. New
`isAndroidEmulatorSerial` helper.
- **Routing** (`blueprints/simulator-server.ts`): spawn `android_device
--id <serial>` for kind `device`, `android` for an emulator, `ios` for a
simulator. The capability matrix already declared `android.device: true`
across the tools, so nothing else gates physical devices out.
- **Surfacing** (`tools/devices/list-devices.ts`): Android entries now
carry a `kind` (`"emulator"` | `"device"`); the tool description
documents physical-device detection. Physical phones don't need
`boot-device` — just connect them.
- **describe/android**: resolves the real device kind instead of
hard-coding `emulator` (the adb path is identical either way).
- **Packaging**: `download-simulator-server.sh` fetches and
`bundle-tools.cjs` ships the screen-sharing agent (`.jar` + per-ABI
`.so`) next to each platform's simulator-server binary, where the
`android_device` controller resolves it at runtime.

## Dependencies (draft)

This is a **draft** because it assumes the matching simulator-server
changes land and publish:

- **software-mansion/radon#128** — Linux/cross-platform OpenH264 H264
*decoder* for the `android_device` controller (macOS keeps
VideoToolbox). Gives the argent-built binary a decoder on every host.
- **software-mansion/radon#127** — enables `android-device` in `argent`
builds and publishes the `screen-sharing-agent.tar.gz` release asset
that the download script consumes.

Until those release artifacts are available, the download step degrades
gracefully (warns, skips) and physical-device support is simply
unavailable.

## Testing

Verified end-to-end against a physical **Galaxy S24** (`arm64-v8a`,
Android API 36) by running the tool-server against a locally built
argent simulator-server (radon#128) with the agent resources staged:

| tool | path exercised | result |
|------|----------------|--------|
| `list-devices` | adb | phone listed as `kind: "device"`, `isEmulator:
false` |
| `screenshot` | screen-sharing agent → H264 → VideoToolbox decode | ✅
correct, correctly-oriented PNG |
| `button` (home) | simulator-server stdin `pressKey` | ✅ |
| `gesture-swipe` | simulator-server WebSocket touch | ✅ |
| `describe` | adb (android-devtools) | ✅ |

Full tool-server unit suite: **1157 passed**. Added coverage for the new
classification (`device-info.test.ts`) and the `android_device`
subcommand routing (`simulator-server-blueprint.test.ts`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 10:23:01 +02:00
filip131311 c01f41cd5f feat(remote): input-side file boundary — flows, screenshot-diff, profiler reports, workspace tools (#319) 
## Why

PR #278 fixed the **output** half of the remote file boundary: files a
tool *produces* travel to the client as `ArtifactHandle`s. But seven
tools still break when the tool-server runs on a different machine than
the agent, in one (or both) of two ways:

- **Output-side leftovers** — `native-profiler-analyze` and
`screenshot-diff` still returned raw host paths (and embedded them in
report/summary prose), so the agent can't open them remotely.
- **Input-side gap** — nothing moved a file the *agent* owns onto the
tool-server. Any tool taking a client-local path as a parameter
(`screenshot-diff` baselines, the three flow tools' `project_root`,
`react-profiler-component-source`, `gather-workspace-data`) silently
read/wrote the wrong machine's disk. Flows were fully broken in remote
mode: the YAML landed on the server's disk under the agent's path.

## What

### Output side (apply the existing pattern)
- **native-profiler-analyze** — `reportFile` is now an `ArtifactHandle`
(mirrors `react-profiler-analyze`'s `fileArtifact()`); the inline report
now says *"use the Read tool on the `reportFile` path in this result"*
instead of embedding the server path.
- **screenshot-diff** — `diffPath` / `contextDiffPath` are now
`ArtifactHandle`s; the MCP adapter renders the context-diff image from
materialized bytes (legacy string-path results from older servers still
render co-located). The summary references the result fields instead of
embedding host paths.

### Input side (new, symmetric mechanism)
A declarative **file-input boundary**, mirroring the artifact gate in
the opposite direction:

- A tool declares `fileInputs: [{ target, path: "${param}…", kind }]` on
its `ToolDefinition`; the declaration ships to the client via `GET
/tools`.
- Before sending a call, the client (`@argent/tools-client`, shared by
CLI and MCP) interpolates each template from the args and replaces the
target arg with a wrapper: `{ __argentFileInput, path, size, mtimeMs,
content? }`. File bytes are base64-inlined **only when routed to an
external tool-server** (`argent link` / `ARGENT_TOOLS_URL`) — unlinked
local calls never pay for encoding.
- The server resolves wrappers **before zod validation**, so tools
always execute against a plain local path:
- path matches on the server's own filesystem (existence for dirs,
size+mtime for files) → used **in place**, zero copies — the exact dual
of the artifact materializer's co-located gate;
- otherwise `kind: "file"` content is materialized into a temp file
(with size-integrity check, 32 MB cap);
- `kind: "directory"` (trees that can't ride in a call) → 422 with
actionable remote-mode guidance;
- `kind: "probe"` → never fails; the tool learns host-presence via
`ctx.fileInputs` and adapts.
- The reverse channel: a tool whose output belongs in the **agent's
project** returns a `__argentClientFile` directive (path + content); the
client writes it and rewrites the directive to the path. Writes are
constrained to `**/.argent/flows/<safe-name>.yaml` so a tool-server
can't direct writes anywhere else on the client machine.

### Tool migrations
| Tool | Boundary use |
|---|---|
| `flow-start-recording` | `probe` on `project_root`: present → host
persistence exactly as before; absent (remote) → recording kept in
server memory, YAML returned as a client-write directive |
| `flow-add-step` / `flow-add-echo` / `flow-finish-recording` | follow
the session's persist mode; every mutation returns `savedTo` (path
locally, directive remotely) so the agent-side file is always current |
| `flow-execute` / `flow-read-prerequisite` | `file` input derived from
`${project_root}/.argent/flows/${name}.yaml` into an internal
`flow_file` param — read in place locally, uploaded when remote |
| `screenshot-diff` | `baselinePath`/`currentPath` as `file` inputs;
`outputDir` now **optional** (temp default) and probe-ignored when
absent on the host |
| `react-profiler-component-source`, `gather-workspace-data` |
`directory` gate — unchanged locally; clear error remotely instead of a
silent empty AST index / all-nulls snapshot |

### Version skew
Both halves degrade to today's behavior: an old client sends plain
strings (the resolver passes them through, tools take their legacy
paths), and an old server doesn't advertise `fileInputs` (the new client
never wraps). Wrappers on undeclared params fail the tool's own schema
validation, so nothing can be smuggled through free-form args.

## Tests

**Unit** (all green: registry 68, tool-server 992, tools-client 107,
argent-mcp 66, argent-cli 117):
- `tool-server/test/file-inputs.test.ts` — in-place gate, stat-mismatch
→ upload, truncated-upload rejection, directory gate, probe semantics,
undeclared-target smuggling.
- `argent-tools-client/test/file-inputs.test.ts` — wrapping (stat-only
vs inlined content), multi-param template derivation, explicit-override
respect, directive writes incl. path-validation refusals (relative,
traversal, outside `.argent/flows`, bad charset/extension).
- `tool-server/test/flows/flow-remote-recording.test.ts` — remote
recording stays in memory + returns directives, nothing written on the
host, replay/read from a boundary-resolved `flow_file`.
- Updated existing screenshot-diff / native-profiler tests to the
artifact-handle contract.

**Local / co-located (unlink scenario)** — isolated tool-server + CLI on
an iPhone 17 Pro simulator: full flow
record→finish→read-prerequisite→execute loop (YAML written via the same
path as before, in-place gate hit, no byte copies); screenshot-diff with
and without `outputDir`; wire-level checks of the unlinked client shape
(wrapper without content → in-place read) and the legacy plain-string
shape; missing-file → 422 with a precise message.

**Remote (Lima Linux VM as the agent machine, tool-server on the host
Mac, `--host` bind + bearer auth)** — via both the CLI and the real MCP
stdio server:
- flow recording from the VM: YAML lands **in the VM's project**,
`savedTo` rewritten to the VM path, *no* agent-path directories created
on the host; replay from the VM uploads the YAML and drives the host
simulator.
- `screenshot` materializes into the VM temp cache; `screenshot-diff`
round-trips both directions (VM baseline uploaded, live capture on host,
diff artifacts downloaded back; MCP returns `image,text` content with
the context diff inlined).
- `gather-workspace-data` / `react-profiler-component-source` with
VM-only paths → the new actionable 422.

## Notes
- `native-profiler-analyze` was not exercised live (xctrace export is
unusable on this machine's iOS 26.4 simulators — known, unrelated); its
change is the same one-line `fileArtifact` pattern
`react-profiler-analyze` already ships, covered by unit tests.
- `react-profiler-analyze`'s own `project_root` (AST resolve stage) is
left as-is deliberately: it degrades gracefully (report still renders)
and gating it would break currently-working remote analysis. Tracked as
follow-up alongside nested invocations (`run-sequence`/`flow-execute`
steps bypass the HTTP boundary by design).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 17:14:04 +02:00
Ignacy Łątka 8c080c9189 fix(screenshot-diff): make live capture work on Android + surface real capture errors (#307) 
## Problem

`screenshot-diff` with `baselinePath` + `captureCurrent: true` — the
tool's own
documented *"common visual-regression flow"* — fails reproducibly on
Android with:

> Screenshot failed: server response missing url or path. The
simulator-server may be misconfigured. Try restarting it.

That message is misleading (the server is healthy), and the flow is 100%
broken on Android.

## Root cause (verified against a live emulator)

`captureLiveInput` requests a **full-resolution** capture (`scale:
1.0`). The Android
simulator-server can't stream some framebuffers at full res and returns
**HTTP 200**
with an `error` field. Confirmed directly against the running
simulator-server:

| request | response |
|---|---|
| `{"scale":0.5}` | `{"path":…,"url":…}` ✅ |
| `{}` (full res) | `{"error":"wrong data size, expected 7853760 got
17627328"}` ❌ |
| `{"scale":1.0}` | `{"error":"wrong data size, expected 7853760 got
17627328"}` ❌ |

(`7853760 = 1080×2424×3`; the emulator returns ~2.24× that.)

Two compounding bugs:
1. **`httpScreenshot`** only reads `error` on a non-2xx status. On the
`200 + {error}` path it falls through to the generic *"missing url or
path … restart it"*, hiding the real cause.
2. **`captureLiveInput`** hard-codes `scale: 1.0` with no fallback, so
the whole diff fails on Android.

## Fix (TS-only)

- **`httpScreenshot`**: when `url`/`path` are missing, surface
`resBody.error` if present (so *"wrong data size …"* reaches the caller)
before the generic hint.
- **`captureLiveInput`**: attempt full-res, and on any failure **retry
at the server's default scale**, which captures reliably. #285's
same-aspect normalization already makes a scaled capture diff-compatible
with a baseline saved at any scale, and full-res is preserved wherever
it works (iOS). If the fallback also fails, its error propagates.

## Tests

`screenshot-diff-tool.test.ts` — full-res→default-scale fallback
(asserts the `1.0` call then a default-scale retry), and error
propagation when both fail.
`simulator-client.test.ts` (new) — `httpScreenshot` surfaces the server
`error` on a `200+{error}` response, still uses the generic hint when
there's genuinely no info, and returns `url`/`path` on success.

All 21 screenshot-diff/simulator-client tests pass; `tsc` clean on the
changed files.

## Scope note

The underlying inability to capture a full-resolution Android frame is a
**simulator-server (native)** issue — the *"wrong data size"*
framebuffer mismatch. This PR makes the tool **degrade gracefully**
(works at the default scale) and **diagnose clearly** instead of failing
outright; a proper native fix for full-res Android capture is a separate
follow-up.
 2026-06-09 10:55:21 +02:00
Hubert Gancarczyk de0dbbda66 feat(tools): implement screenshot-diff tool (#229) 
## Summary

Adds a new `screenshot-diff` tool for comparing mobile screenshots
during UI QA and visual regression checks. The tool can compare two
saved PNGs, or compare one saved PNG against one live full-resolution
device capture, then returns a compact text summary and generated diff
artifacts.

## Context

Mobile UI validation often needs more than a single screenshot: small
layout, color, typography,
copy, clipping, and regression changes are easy to miss manually. This
feature gives agents a
deterministic before/after comparison path while keeping
`screenshot-diff` as supporting evidence.

## What Changed

- Registered a new `screenshot-diff` tool in the tool server.
- Supports saved PNG comparisons and one-sided live capture via
`captureBaseline` or
`captureCurrent`.
- Adds PNG diffing with fixed internal defaults, changed-region
detection, status-bar-band
ignoring, full-size diff output, and downscaled context diff output.
- Adds optional OCR/text analysis for moved, appeared, disappeared,
changed, and restyled text.
- Updates MCP content rendering to attach the downscaled context diff
image plus the text summary.

1. Capture a stable full-resolution baseline.
2. Perform an interaction or apply a UI change.
3. Compare the current screen against the baseline.
4. Review a summarized diff plus generated artifacts.
 2026-06-01 15:04:25 +02:00