mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
feat/flow-script-drop-logs
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hubert Gancarczyk
|
fdd1774ca4 |
fix(flows): stop reporting what a script prints, on both sides
A `script:` step used to bring its stdout and stderr back with it. The recorder returned `log`/`logTruncated`, the step report carried `scriptLog`/`scriptLogTruncated`, and the CLI and the MCP content builder each rendered them. Both surfaces land in an agent's context, and nothing in the flow file declares what a script prints, so nothing can decide in advance whether that text is safe to forward. The channel goes rather than being gated behind a parameter the agent would set itself. A script step now answers with its verdict and its `reason` only. Deleting the capture removes what existed to bound it: the 64 KiB per-step cap, the 256 KiB run-wide budget, the V8 frame collapser, the partial-marker guard and the streaming half of the secret scrub. Both pipes are still drained, and stderr is still scanned for V8's heap-exhaustion banner and for a watchdog that never armed. |
||
|
Hubert Gancarczyk
|
9c4bc9ef5b |
feat(flows): run an external script as a flow step (#865)
Stacked on #864, which adds the executor. This branch adds the `script:` step itself: the YAML directive, the runner integration, the CLI and MCP rendering, and the authoring reference. ```yaml - script: { path: ../../scripts/seed-order.mjs } - script: { path: ../../scripts/seed-order.mjs, timeout: 60000 } ``` A `script:` step runs an external `.mjs` file from the project, in a fresh Node process. What the script does is its own business — seeding an order, creating a test account, cleaning up what a run left behind are the reasons flows have wanted one. It drives no device, so a flow of nothing but scripts runs with nothing booted. ## What it adds - **The directive.** The value is always a map; `path` is required and obeys the same name rules a `run:` target does, spelled for `.mjs`. It resolves against the directory of the flow file containing the step, so a fragment reaches the same script whichever flow composed it. A mis-cased path is refused rather than run, because macOS and Windows would open it and Linux CI would not. - **The report.** The script's stdout and stderr come back on the step and print under the step line, on a pass as well as a failure — it is the only record of what the step did to the backend. Both renderers (CLI, MCP) carry it, live and buffered. - **The fail/error split.** A `fail` is the script's own answer; an `error` is the host's. That is what lets CI tell a regression from the machine it ran on. - **Two boundaries.** An uploaded flow carrying a `script:` step is refused before anything runs — its `.mjs` never left the client. A flow whose script step sits beside a `run:` step still resolves a device. ## Review follow-ups The last three commits answer a review of the three above them. - A launch behind a leading `script:` was not read as the flow's leading launch, so a Chromium flow that seeded a backend first hoisted no boot, bound to whatever browser happened to be up, and passed a launch it never performed. The same lead-in walked a fragment past the `executionPrerequisite` refusal its `echo:`-led twin is given. Both readers now share one exhaustively-switched predicate. - Coverage: the whole failure-kind table, the mis-cased arm CI never ran, the two unreached `scriptFileProblem` arms, symlink resolution for a script path, and the CLI live-progress closure that no test had executed. - Docs: the reference claimed the log was secret-redacted (it is not), never mentioned the environment allowlist, the working directory, the time-limit ceiling or the log caps, stated a fail/error rule that was not the code's, and was missing from its own table of contents. The skills stated a closed list of legal unrecorded insertions that excluded the only way this step can ever be written. ## Verification `npm run build`, `npx eslint . --max-warnings 0`, `npx prettier --check .`, `npm run knip`, `npm run typecheck:scripts` and `npm run test:scripts` are green, as are the three `tsconfig.test.json` typechecks. Suites: tool-server 357 files / 4552 passed, `@argent/cli` 469, `@argent/mcp` 83, `@argent/registry` 90. Reproduced end to end against a real tool server and a real Electron app, not only in tests: a flow of `script:` then `launch:` now boots its own Chromium instance and reports it, where before it attached to a stray browser and reported a green launch for an app path that did not exist. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for local `.mjs` scripts as flow steps, including configurable timeouts, validation, cancellation, and device-free execution. - Script output is displayed in CLI runs and included in MCP flow results, with indentation and truncation notices. - Script failures now provide clearer diagnostics and verdicts. - Flow authoring tools recognize and guide creation of script steps. - **Documentation** - Added guidance for script syntax, execution limits, output handling, classification, and remote-run restrictions. - Updated flow creation and QA guidance to include script steps. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com> |
||
|
Ignacy Łątka
|
e895dd0a5d |
fix(flow): give every flow in a directory run a verdict on stdout (#739)
## Every flow run says how it ended, on stdout
`argent flow run` wrote some outcomes only to stderr. A CI job that
archives stdout — the
common case — got a log with a hole in it.
**A directory run**, three flows, one with a `run:` the parser rejects:
```
$ argent flow run <dir> > out.txt 2> err.txt ; echo $?
1
── out.txt ── ← the log that gets archived
[1/3] a-ok.yaml
PASS — 0 passed, 0 failed, 0 errored, 0 skipped (no test steps)
[2/3] b-broken.yaml
[3/3] c-ok.yaml
PASS — 0 passed, 0 failed, 0 errored, 0 skipped (no test steps)
FAIL — 3 flows: 2 passed, 1 failed, 0 skipped
```
`[2/3]` has a header and no outcome. The tally says one flow failed and
stdout names none;
the reader's inference that `b-broken` was the one is available only
because it is the sole
entry missing a line.
**A single-flow run** had the same gap, and worse — with live step lines
the last thing the
log holds is a `✓`, so a run that died reads as one that passed and got
truncated:
```
$ argent flow run checkout.yaml > out.txt 2> err.txt ; echo $?
1
── out.txt ──
Flow "checkout"
✓ 1 tap
✓ 2 assert ← ends here
```
## The fix
An indented verdict on stdout for every branch that had none, on both
runners:
```
✗ not run (invalid flow) ← rejected by validation; a batch continues
✗ not run (no device resolved) ← nothing matched the run, or several did and none was singled out
✗ not run (rejected) ← rejected for a reason neither of the above names
✗ did not finish (run error) ← in a batch, this flow is the one that stopped it
✗ did not finish (no run report)
```
stderr is untouched, so anything reading that stream sees exactly what
it did before, and the
detail is not duplicated onto the merged stream. Both runners print the
verdict directly under
the header and the stderr detail after it, so a `2>&1` log of a batch
reads like a `2>&1` log of
a single run. No verdict reaches `--json` — every one is behind
`!args.json` — so neither runner
ever mixes prose into that stream. What each emits is unchanged: a batch
still prints its one
aggregate document on every path, and a single run still prints the
report alone, which means
nothing at all when the run produced no report.
### What a batch's `--json` carries
The aggregate is the mode a CI job consumes for a directory, so the
classification the verdict is
chosen from travels with it: a failed entry carries `error_code` and
`error_kind` beside `error`,
under the names `--json-stream` already publishes. Without them an
invalid flow file and an
unresolvable device are the same `{"status": "fail", "error":
"<prose>"}` entry with the same key
set, told apart only by a message the server assembles from device lists
and remedies.
### Picking the wording from the failure code, not the kind
`error_kind: "validation"` marks any rejection the server scopes to one
call; it says nothing
about the flow file. Device resolution rejects that way too
(`FLOW_DEVICE_RESOLUTION`), so
keying the wording off the kind made a batch run with nothing booted
report every one of a
directory of good YAML files as `not run (invalid flow)` — sending the
reader to the wrong
file. `rejectionVerdict` switches on `error_code` instead: only a code
whose subject is the
file (`FLOW_FILE_INVALID`, `FLOW_ENTRY_UNRECOGNIZED`,
`FLOW_E2E_HAS_PREREQUISITE`) licenses
"invalid flow"; anything unrecognized stays neutral and leaves the
reason to the stderr line
beside it.
`FLOW_NAME_INVALID` is deliberately absent. `argent flow run` only ever
sends `flow_path`, and
both of its paths screen the stem against `SAFE_FLOW_NAME` first — which
*is*
`FLOW_NAME_PATTERN`, the pattern `assertSafeFlowName` applies on the
server — so the code
cannot come back from a flow run. `FLOW_PROJECT_ROOT_INVALID` is out for
the same reason:
`project_root` is `process.cwd()`, always absolute and never
`..`-bearing.
Which rejections let the batch continue is unchanged — still `errorKind
=== "validation"`.
### Also on the single-flow path
`!report || !("steps" in report)` took a primitive wire value straight
into `in`, which throws
on one — `TypeError: Cannot use 'in' operator to search for 'steps' in
a-string`, surfacing
through the CLI's top-level catch as a raw stack trace. The batch path
guards this deliberately
("must classify as no report, not as an infra throw"); this one now does
too.
### Docs
The help text said a directory run prints "only failing steps plus a
final summary" and named
an invalid file as the only rejection it survives. It now describes what
it prints and what it
continues past.
`packages/docs/docs/reference/flow-yaml.mdx` still described the batch
as printing "only the
failing steps and a summary" and named an invalid file as the only
rejection it survives. It now
describes what a directory run prints, tables the six outcome lines with
their meanings, states
what the `--json` aggregate holds down to `error_code`, notes that a
single-flow `--json` failure
writes nothing at all to stdout, and adds `--json-stream` to the option
table — the mode that
carries the failure as a record.
Two claims that survived into the first pass of that page were wrong and
are corrected here: a
flow that fails its *steps* also leaves the batch running (only a
transport failure, a rejection
of another kind, or a reply that is not a report stops it), and the
aggregate entry for such a
flow carries a report and no `error`. The `flow run --help` blurb
carried the first of those two
as well.
`argent-create-flow` carried the same stale sentence when this branch
opened, and an earlier
commit here fixed it too. #730 has since split that skill into
`references/` and dropped the
standalone-runner paragraph from the tree, so there is nothing left to
correct there and the
merge resolution takes the hunk out.
## Checks
- E2E through the built CLI (`packages/argent/dist/cli.js`) for every
transcript above, plus
the `--json` shapes piped through `python3 -m json.tool` to prove one
document.
- `packages/argent-cli`: 512 passed across 24 files (`flow.test.ts`
148).
- Mutation matrix over the new behaviour, 24 mutations applied and
reverted individually:
deleting each of the five verdicts, dropping either `!args.json` guard,
weakening
`isFlowReport` in six ways (each conjunct dropped in turn,
`every`→`some`, and reverts to
both earlier readings), deleting each `rejectionVerdict` arm, swapping
two wordings,
dropping either `liveSteps === 0` header guard, dropping `stopped =
true`, restricting the
batch verdict to the first flow, and emitting a verdict on the success
path. Every one
reddens a test; no gaps.
- Reverting only the source hunks fails exactly the tests added here; no
pre-existing test
changes state.
- E2E against a real tool-server for the review follow-ups: a directory
of one unparseable
YAML plus one flow needing a device, run with the verdict and the stderr
detail merged
(`2>&1`) to show the order, and again under `--json` —
`FLOW_FILE_INVALID` / `validation`
arrive in the aggregate entry and stdout parses as one document. A
`--platform vega` run with
no VVD gives the real `not run (no device resolved)` line.
- Mutations for the follow-ups, each applied and reverted: dropping the
failure signal from the
aggregate, relaxing the batch guard to `!args.json ||
rejectedThisFlowOnly`, dropping
`&& !args.jsonStream` from the single-flow guards, restoring the old
detail-before-verdict
order, and deleting the batch verdict. Every one reddens a test.
- prettier, eslint (`--max-warnings 0`), `tsc --build`, `tsc -p
tsconfig.test.json`, knip and
`npx docusaurus build` in `packages/docs` all clean.
Two findings from reviewing this branch are out of scope and filed
instead: #1025 (`--json`
prints the usage text on stdout for a flag error, where `--json-stream`
keeps stdout
machine-readable) and #1026 (a directory run's artifact export sits
outside the loop's `try`, so
an export throw loses the batch summary, the skip entries and the
aggregate).
Directory runs arrived in #576, so the batch half is a defect in a new
feature rather than a
regression; the single-flow half predates it.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added clear outcome lines for flow runs, including validation
rejections, execution errors, missing reports, and skipped flows.
* Batch runs continue after flow-specific validation rejections and stop
when other errors occur.
* Added machine-readable failure classifications to JSON output,
including batch aggregates.
* Added `--json-stream` for one JSON record per line during single-flow
runs.
* JSON modes now emit only machine-readable output, with aggregate
details for directory runs.
* **Documentation**
* Expanded flow command documentation with batch behavior, outcome
statuses, and JSON output details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
filip131311
|
26405e68c7 |
refactor(cli): one declarative option parser for every hand-written subcommand (#953)
Follows #952 (merged); rebased onto main. ## Why `argent-cli` had **seven** copies of the same argv loop — `server start`, `link`/`unlink`, `flow run`, `lens`, `config`, `enable`/`disable`, and (from #952) `telemetry` — each hand-rolling `--x value` / `--x=value` / `-x` / unknown-flag / missing-value handling with slightly different messages and gaps (e.g. `lens` silently ignored unknown flags). Adding an option meant another `if (tok === "--x") … else if (tok.startsWith("--x="))` pair. ## What `packages/argent-cli/src/command-args.ts` (introduced in #952) now covers the whole subcommand surface, and every command declares its options as a spec: ```ts const START_OPTIONS = { help: { kind: "boolean", alias: "h" }, port: { kind: "value", alias: "p" }, … } as const satisfies OptionSpecs; ``` Parser features: `--name value`, `--name=value`, single-letter aliases, boolean switches, `--` end-of-options, `choices` validation, "don't swallow a following flag as a value", and the guard against a stray `true`/`false` word after a switch (previously duplicated in `flow` and `lens`). Value *semantics* (port ranges, host wildcards, flag-name charset, link targets) stay in each command — the parser only shapes argv. `flag-parser.ts` is untouched: it is the schema-driven `argent run <tool>` payload builder (JSON Schema → tool args, `--args`/stdin hatches), a different job. ## Behavior changes (bad-input paths only) | Command | Before | After | |---|---|---| | `lens --bogus` / `lens extra` | silently ignored | `lens: Unknown flag: --bogus`, exit 2 | | `lens --terminal x` | `--terminal expects "iterm" or "terminal", got "x"` | `--terminal must be "iterm" or "terminal", got "x"` | | `flow run --platfrom=ios` | `unknown flag --platfrom=ios` | `Unknown flag: --platfrom=ios` | | `unlink foo` | `Unknown flag: foo` | `Unexpected argument "foo"` | | `config … --scope x` | `--scope must be "global" or "project", got "x"` | unchanged wording | | `enable/disable`, `server start`, `link` | — | unchanged wording (tests pinned) | Happy paths are byte-identical; `server`/`link` still throw `StartFlagError`, `flow` still throws `FlagParseException` (wrapped from `UsageError`), so callers and tests keep their contracts. ## Tests `argent-cli` 489/489 (26 files): rewritten `command-args.test.ts` (aliases, `-` positional, empty-token value, true/false guard, choices message), pinned-string updates only where the table above says the wording changed (`flow.test.ts` ×4, `parse-link-flags.test.ts` ×1). `tsc --build`, `typecheck:tests`, eslint, prettier, knip green. Smoke-tested every command's error path through the built `packages/argent/dist/cli.js` (see the "Behavior changes" table — all exit 2 with usage where the command prints one). ## Docs No docs update needed: the pages document flags and their meaning, not error strings; no flag was added, removed or renamed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BkkAihA8Kp26hHS4uhkNGu |
||
|
schicaelmott
|
dd5d823833 |
feat(flow): stream run progress as NDJSON (#855)
## Summary - add `argent flow run --json-stream` - stream flow progress as JSON lines - finish with a JSON result or error ## Why `--json` prints only after a flow finishes. `--json-stream` lets scripts and CI see live progress. ## Why this is a separate PR This is useful without the custom-tools feature in #856. ## Validation - `npm test -w @argent/cli -- flow.test.ts` — 106 tests passed - `npm run typecheck:tests -w @argent/cli` - `npm run build -w @argent/cli` --------- Co-authored-by: Filip131311 <contact@filipkaminski.com> |
||
|
Jakub Piasecki
|
6d907cc86b |
feat(flow): boot a Chromium instance per launch step (#585)
A `launch:` on Chromium now starts its own Electron instance, so nested e2e flows and mid-flow relaunches work as they do on iOS/Android. Previously the runner booted one app per run and rejected any second chromium launch. A chromium device id _is_ its CDP port, so a launch that starts an app from scratch has to boot one and move the run onto it: `state.device` became reassignable (steps already read it per call), and `state.owned` tracks what the runner booted, torn down in reverse at run end. An instance the runner did not boot — pinned via `--device`, or auto-detected — is attached to, never killed. Relaunching the same app kills its instance and waits for the process to actually exit, or the replacement loses a race against the app's single-instance lock. The runner needs a device before step 1, so `resolveRunDevice` follows a leading `run:` chain to find the launch the run _begins_ with: a fragment composing a chromium e2e flow now boots instead of failing with "No booted chromium device found". A launch declaring no app id for the run's platform stays an error — the run's platform is fixed start to finish. Also fixes a pre-existing teardown leak: the spawned handle is the npm `electron` node wrapper, not Electron, so signalling it left the app and its helpers running (reparented to launchd, still in the dock). Teardown now sweeps the child's process group. **Behavior change:** a fragment leading with `run:` into a chromium e2e flow now boots a fresh instance where it previously attached to a running one; pass `--device` to keep attaching. ## Tests Cover the boot/attach/settle three-way, teardown ordering and the group sweep, device rebinding after a nested launch, same-app retire ordering, cancellation mid-boot, both "no chromium app declared" guards, and the leading-`run:` chain (multi-hop, cyclic, no-launch). The guard, exit-wait and sweep tests are mutation-checked. |
||
|
Jakub Piasecki
|
22033e1418 |
feat(flow): allow to run every flow in a directory from flow run (#576)
`argent flow run` now accepts a directory and runs every flow in it sequentially: ``` argent flow run .argent/flows --recursive ``` Discovery uses the same acceptance rules as `flow list` (lowercase `.yaml`, safe stem, readable file), sorted for a deterministic run order. `-r`/`--recursive` walks subdirectories too, skipping dot-directories and `node_modules` and never following directory symlinks. A directory named `foo.yaml` still batches (stat wins over extension). Batch output is failures-only: each flow prints a `[i/n] <path>` header, its failing/errored steps — numbered exactly as a single-mode rerun would number them — and a per-flow summary; the run ends with a `PASS/FAIL — N flows: X passed, Y failed, Z skipped` verdict. `--json` prints one aggregate object with per-flow reports. Exit code reflects the batch (0 pass, 1 any failure, 2 discovery/config errors). `--output` exports keep the single-run `<output>/<flow>/` layout, keyed by subdirectory in recursive runs so same-stem flows in different directories can't clobber each other's snapshots. **Failure semantics:** a flow failing its steps — or one the tool-server rejects as invalid (bad YAML, unparseable step) — fails alone and the batch continues. An infra error (transport death, unclassified failure) stops the batch and counts the remaining flows skipped. To make that distinction possible, tool-error responses now carry the server's failure signal (`error_code`/`error_kind`) on the wire — on the streaming terminal error line and on JSON error bodies — and the tools client surfaces server-answered errors as a typed `ToolInvocationError`. Against a pre-signal server this degrades to stopping on any throw. Not breaking: single-file runs behave exactly as before. |
||
|
Jakub Piasecki
|
a644d52d39 |
feat(flows): run CLI flows from explicit paths (#568)
`argent flow run` now takes an explicit YAML path instead of a saved-flow name: ``` argent flow run .argent/flows/checkout.yaml --platform ios ``` The path may point anywhere on the local filesystem; the filename stem names the run's report/artifacts, so it must be `[A-Za-z0-9_-]+` with a lowercase `.yaml` extension. A bare name fails with the exact replacement command, and `argent flow list` prints paths instead of names. Because sibling `run:` files and `__baselines__` resolve beside the YAML, the CLI now requires the auto-started local tool server and refuses `ARGENT_TOOLS_URL` / `argent link` routing. The `flow-execute` tool stays remotely callable via `name` + `project_root`, and gains a `flow_path` parameter (exactly one of `name`/`flow_path`) that is only honored when the file-input boundary verified it as a real host path — uploads and boundary bypasses are rejected, so it can't execute arbitrary server YAML. Also: the flow-name pattern is now a single shared export from `@argent/registry`, and the recorder composes a `flow_path` call into a `run:` step when it targets a sibling of the recording. |
||
|
Jakub Piasecki
|
551c781fb0 |
feat(flow): support running end-to-end flows without agent (#429)
# Flow E2E
This PR turns flows from replayable tool-call recordings into a
declarative, portable e2e test system that runs headlessly via `argent
flow run <name>` — no LLM in the loop, non-zero exit on failure,
suitable for CI.
Agents can still replay flows via `flow-execute` and get the full output
from each step after it finishes.
## Flow format
Flows are YAML files in `.argent/flows/`, now built from **directives**
interpreted by the runner rather than raw tool calls:
- `launch` — terminate + relaunch the app under test (per-platform app
id map supported); its presence marks a flow as **e2e** (controls its
own start state) vs. a **fragment** (runs against current state,
declares an `executionPrerequisite`, composable via `run:` with
cycle/depth guards).
- `tap`, `type` (focus handshake + optional submit), `scroll-to`
(momentum-free, increments by half the container, `within:` for nested
scrollers), `await`/`assert` (`visible`/`exists`/`hidden`/`text`
conditions), `wait`, `snapshot`, `echo`, and raw `tool:` as escape hatch
— still fully supported, so previously recorded flows keep working
unchanged.
- **Selectors** are `{ text, identifier, role }`; a bare string is a
loose selector resolving identifier-first, then visible text. Any
directive failure hard-stops the flow; later steps report `skip`.
## Execution engine
- Flows store no device id — the runner binds a device at replay
(explicit `--device`/`--platform` or the single booted one).
- Selector resolution runs against **platform-unified UI trees**: iOS
native view hierarchy (full testID coverage, gated on native-devtools
connecting), Android full accessibility hierarchy (full resource-id
coverage), Chromium CDP DOM, and the Vega automation toolkit. A shared
flatten/text-hoisting pass lets `text` assertions read what an
identified container visibly shows. Vega is remote-driven, so touch
directives fail with guidance to use `tv-remote`/`keyboard`.
- Steps wait for the tree to **settle** (two consecutive identical
reads) before acting; `launch` additionally gates on platform
tree-source readiness instead of silently degrading.
## Visual snapshots
`snapshot` diffs a full-resolution screenshot against baselines in
`.argent/flows/__baselines__/<flow>/`, keyed by platform + resolution,
with configurable `maxMismatch` and `--update-baselines`. The status bar
is pinned for the whole run (iOS `simctl status_bar`, Android demo mode)
so clocks/battery don't drive diffs. Baseline/current/diff images are
returned as artifacts on failure (and the written baseline on first
capture/update); a clean pass carries none, keeping reports and agent
context lean.
## Recording
The record-live workflow (`flow-start-recording` → `flow-add-step` →
`flow-finish-recording`) now emits directives automatically: coordinate
taps are rewritten to portable selector `tap:` steps (selector captured
from the pre-tap tree), a leading `restart-app` becomes `launch:`,
`flow-execute` of a sibling fragment becomes `run:`, and device ids are
stripped. The skill instructs the agent to polish the saved YAML
afterwards (fold tap+keyboard into `type:`, swipes-to-reach into
`scroll-to:`, awaits into sugar form).
## Also included
- `argent flow run` / `argent flow list` CLI with per-step report
rendering, artifact materialization, and `--json` output.
- New structured failure codes for flow validation, device resolution,
composition, and devtools readiness.
- Extensive unit test coverage (tree adapters, selector matching,
scrolling, composition, type focus, snapshots, status bar).
|