2 Commits

Author	SHA1	Message	Date
Jakub Piasecki	551c781fb0	feat(flow): support running end-to-end flows without agent (#429) ... # Flow E2E This PR turns flows from replayable tool-call recordings into a declarative, portable e2e test system that runs headlessly via `argent flow run <name>` — no LLM in the loop, non-zero exit on failure, suitable for CI. Agents can still replay flows via `flow-execute` and get the full output from each step after it finishes. ## Flow format Flows are YAML files in `.argent/flows/`, now built from **directives** interpreted by the runner rather than raw tool calls: - `launch` — terminate + relaunch the app under test (per-platform app id map supported); its presence marks a flow as **e2e** (controls its own start state) vs. a **fragment** (runs against current state, declares an `executionPrerequisite`, composable via `run:` with cycle/depth guards). - `tap`, `type` (focus handshake + optional submit), `scroll-to` (momentum-free, increments by half the container, `within:` for nested scrollers), `await`/`assert` (`visible`/`exists`/`hidden`/`text` conditions), `wait`, `snapshot`, `echo`, and raw `tool:` as escape hatch — still fully supported, so previously recorded flows keep working unchanged. - **Selectors** are `{ text, identifier, role }`; a bare string is a loose selector resolving identifier-first, then visible text. Any directive failure hard-stops the flow; later steps report `skip`. ## Execution engine - Flows store no device id — the runner binds a device at replay (explicit `--device`/`--platform` or the single booted one). - Selector resolution runs against **platform-unified UI trees**: iOS native view hierarchy (full testID coverage, gated on native-devtools connecting), Android full accessibility hierarchy (full resource-id coverage), Chromium CDP DOM, and the Vega automation toolkit. A shared flatten/text-hoisting pass lets `text` assertions read what an identified container visibly shows. Vega is remote-driven, so touch directives fail with guidance to use `tv-remote`/`keyboard`. - Steps wait for the tree to **settle** (two consecutive identical reads) before acting; `launch` additionally gates on platform tree-source readiness instead of silently degrading. ## Visual snapshots `snapshot` diffs a full-resolution screenshot against baselines in `.argent/flows/__baselines__/<flow>/`, keyed by platform + resolution, with configurable `maxMismatch` and `--update-baselines`. The status bar is pinned for the whole run (iOS `simctl status_bar`, Android demo mode) so clocks/battery don't drive diffs. Baseline/current/diff images are returned as artifacts on failure (and the written baseline on first capture/update); a clean pass carries none, keeping reports and agent context lean. ## Recording The record-live workflow (`flow-start-recording` → `flow-add-step` → `flow-finish-recording`) now emits directives automatically: coordinate taps are rewritten to portable selector `tap:` steps (selector captured from the pre-tap tree), a leading `restart-app` becomes `launch:`, `flow-execute` of a sibling fragment becomes `run:`, and device ids are stripped. The skill instructs the agent to polish the saved YAML afterwards (fold tap+keyboard into `type:`, swipes-to-reach into `scroll-to:`, awaits into sugar form). ## Also included - `argent flow run` / `argent flow list` CLI with per-step report rendering, artifact materialization, and `--json` output. - New structured failure codes for flow validation, device resolution, composition, and devtools readiness. - Extensive unit test coverage (tree adapters, selector matching, scrolling, composition, type focus, snapshots, status bar).	2026-07-13 17:15:01 +02:00
filip131311	c01f41cd5f	feat(remote): input-side file boundary — flows, screenshot-diff, profiler reports, workspace tools (#319) ... ## Why PR #278 fixed the **output** half of the remote file boundary: files a tool *produces* travel to the client as `ArtifactHandle`s. But seven tools still break when the tool-server runs on a different machine than the agent, in one (or both) of two ways: - **Output-side leftovers** — `native-profiler-analyze` and `screenshot-diff` still returned raw host paths (and embedded them in report/summary prose), so the agent can't open them remotely. - **Input-side gap** — nothing moved a file the *agent* owns onto the tool-server. Any tool taking a client-local path as a parameter (`screenshot-diff` baselines, the three flow tools' `project_root`, `react-profiler-component-source`, `gather-workspace-data`) silently read/wrote the wrong machine's disk. Flows were fully broken in remote mode: the YAML landed on the server's disk under the agent's path. ## What ### Output side (apply the existing pattern) - **native-profiler-analyze** — `reportFile` is now an `ArtifactHandle` (mirrors `react-profiler-analyze`'s `fileArtifact()`); the inline report now says *"use the Read tool on the `reportFile` path in this result"* instead of embedding the server path. - **screenshot-diff** — `diffPath` / `contextDiffPath` are now `ArtifactHandle`s; the MCP adapter renders the context-diff image from materialized bytes (legacy string-path results from older servers still render co-located). The summary references the result fields instead of embedding host paths. ### Input side (new, symmetric mechanism) A declarative **file-input boundary**, mirroring the artifact gate in the opposite direction: - A tool declares `fileInputs: [{ target, path: "${param}…", kind }]` on its `ToolDefinition`; the declaration ships to the client via `GET /tools`. - Before sending a call, the client (`@argent/tools-client`, shared by CLI and MCP) interpolates each template from the args and replaces the target arg with a wrapper: `{ __argentFileInput, path, size, mtimeMs, content? }`. File bytes are base64-inlined **only when routed to an external tool-server** (`argent link` / `ARGENT_TOOLS_URL`) — unlinked local calls never pay for encoding. - The server resolves wrappers **before zod validation**, so tools always execute against a plain local path: - path matches on the server's own filesystem (existence for dirs, size+mtime for files) → used **in place**, zero copies — the exact dual of the artifact materializer's co-located gate; - otherwise `kind: "file"` content is materialized into a temp file (with size-integrity check, 32 MB cap); - `kind: "directory"` (trees that can't ride in a call) → 422 with actionable remote-mode guidance; - `kind: "probe"` → never fails; the tool learns host-presence via `ctx.fileInputs` and adapts. - The reverse channel: a tool whose output belongs in the **agent's project** returns a `__argentClientFile` directive (path + content); the client writes it and rewrites the directive to the path. Writes are constrained to `**/.argent/flows/<safe-name>.yaml` so a tool-server can't direct writes anywhere else on the client machine. ### Tool migrations | Tool | Boundary use | |---|---| | `flow-start-recording` | `probe` on `project_root`: present → host persistence exactly as before; absent (remote) → recording kept in server memory, YAML returned as a client-write directive | | `flow-add-step` / `flow-add-echo` / `flow-finish-recording` | follow the session's persist mode; every mutation returns `savedTo` (path locally, directive remotely) so the agent-side file is always current | | `flow-execute` / `flow-read-prerequisite` | `file` input derived from `${project_root}/.argent/flows/${name}.yaml` into an internal `flow_file` param — read in place locally, uploaded when remote | | `screenshot-diff` | `baselinePath`/`currentPath` as `file` inputs; `outputDir` now **optional** (temp default) and probe-ignored when absent on the host | | `react-profiler-component-source`, `gather-workspace-data` | `directory` gate — unchanged locally; clear error remotely instead of a silent empty AST index / all-nulls snapshot | ### Version skew Both halves degrade to today's behavior: an old client sends plain strings (the resolver passes them through, tools take their legacy paths), and an old server doesn't advertise `fileInputs` (the new client never wraps). Wrappers on undeclared params fail the tool's own schema validation, so nothing can be smuggled through free-form args. ## Tests **Unit** (all green: registry 68, tool-server 992, tools-client 107, argent-mcp 66, argent-cli 117): - `tool-server/test/file-inputs.test.ts` — in-place gate, stat-mismatch → upload, truncated-upload rejection, directory gate, probe semantics, undeclared-target smuggling. - `argent-tools-client/test/file-inputs.test.ts` — wrapping (stat-only vs inlined content), multi-param template derivation, explicit-override respect, directive writes incl. path-validation refusals (relative, traversal, outside `.argent/flows`, bad charset/extension). - `tool-server/test/flows/flow-remote-recording.test.ts` — remote recording stays in memory + returns directives, nothing written on the host, replay/read from a boundary-resolved `flow_file`. - Updated existing screenshot-diff / native-profiler tests to the artifact-handle contract. **Local / co-located (unlink scenario)** — isolated tool-server + CLI on an iPhone 17 Pro simulator: full flow record→finish→read-prerequisite→execute loop (YAML written via the same path as before, in-place gate hit, no byte copies); screenshot-diff with and without `outputDir`; wire-level checks of the unlinked client shape (wrapper without content → in-place read) and the legacy plain-string shape; missing-file → 422 with a precise message. **Remote (Lima Linux VM as the agent machine, tool-server on the host Mac, `--host` bind + bearer auth)** — via both the CLI and the real MCP stdio server: - flow recording from the VM: YAML lands **in the VM's project**, `savedTo` rewritten to the VM path, *no* agent-path directories created on the host; replay from the VM uploads the YAML and drives the host simulator. - `screenshot` materializes into the VM temp cache; `screenshot-diff` round-trips both directions (VM baseline uploaded, live capture on host, diff artifacts downloaded back; MCP returns `image,text` content with the context diff inlined). - `gather-workspace-data` / `react-profiler-component-source` with VM-only paths → the new actionable 422. ## Notes - `native-profiler-analyze` was not exercised live (xctrace export is unusable on this machine's iOS 26.4 simulators — known, unrelated); its change is the same one-line `fileArtifact` pattern `react-profiler-analyze` already ships, covered by unit tests. - `react-profiler-analyze`'s own `project_root` (AST resolve stage) is left as-is deliberately: it degrades gracefully (report still renders) and gating it would break currently-working remote analysis. Tracked as follow-up alongside nested invocations (`run-sequence`/`flow-execute` steps bypass the HTTP boundary by design). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 17:14:04 +02:00