Files
software-mansion__argent/packages/tool-server/test/flows/script/flow-script-step-parse.test.ts
Hubert Gancarczyk 9c4bc9ef5b feat(flows): run an external script as a flow step (#865)
Stacked on #864, which adds the executor. This branch adds the `script:`
step
itself: the YAML directive, the runner integration, the CLI and MCP
rendering,
and the authoring reference.

```yaml
- script: { path: ../../scripts/seed-order.mjs }
- script: { path: ../../scripts/seed-order.mjs, timeout: 60000 }
```

A `script:` step runs an external `.mjs` file from the project, in a
fresh Node
process. What the script does is its own business — seeding an order,
creating
a test account, cleaning up what a run left behind are the reasons flows
have
wanted one. It drives no device, so a flow of nothing but scripts runs
with
nothing booted.

## What it adds

- **The directive.** The value is always a map; `path` is required and
obeys
the same name rules a `run:` target does, spelled for `.mjs`. It
resolves
against the directory of the flow file containing the step, so a
fragment
reaches the same script whichever flow composed it. A mis-cased path is
refused rather than run, because macOS and Windows would open it and
Linux CI
  would not.
- **The report.** The script's stdout and stderr come back on the step
and
print under the step line, on a pass as well as a failure — it is the
only
record of what the step did to the backend. Both renderers (CLI, MCP)
carry
  it, live and buffered.
- **The fail/error split.** A `fail` is the script's own answer; an
`error` is
the host's. That is what lets CI tell a regression from the machine it
ran
  on.
- **Two boundaries.** An uploaded flow carrying a `script:` step is
refused
before anything runs — its `.mjs` never left the client. A flow whose
script
  step sits beside a `run:` step still resolves a device.

## Review follow-ups

The last three commits answer a review of the three above them.

- A launch behind a leading `script:` was not read as the flow's leading
launch, so a Chromium flow that seeded a backend first hoisted no boot,
bound
  to whatever browser happened to be up, and passed a launch it never
  performed. The same lead-in walked a fragment past the
`executionPrerequisite` refusal its `echo:`-led twin is given. Both
readers
  now share one exhaustively-switched predicate.
- Coverage: the whole failure-kind table, the mis-cased arm CI never
ran, the
two unreached `scriptFileProblem` arms, symlink resolution for a script
path,
  and the CLI live-progress closure that no test had executed.
- Docs: the reference claimed the log was secret-redacted (it is not),
never
mentioned the environment allowlist, the working directory, the
time-limit
ceiling or the log caps, stated a fail/error rule that was not the
code's,
and was missing from its own table of contents. The skills stated a
closed
list of legal unrecorded insertions that excluded the only way this step
can
  ever be written.

## Verification

`npm run build`, `npx eslint . --max-warnings 0`, `npx prettier --check
.`,
`npm run knip`, `npm run typecheck:scripts` and `npm run test:scripts`
are
green, as are the three `tsconfig.test.json` typechecks. Suites:
tool-server
357 files / 4552 passed, `@argent/cli` 469, `@argent/mcp` 83,
`@argent/registry` 90.

Reproduced end to end against a real tool server and a real Electron
app, not
only in tests: a flow of `script:` then `launch:` now boots its own
Chromium
instance and reports it, where before it attached to a stray browser and
reported a green launch for an app path that did not exist.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added support for local `.mjs` scripts as flow steps, including
configurable timeouts, validation, cancellation, and device-free
execution.
- Script output is displayed in CLI runs and included in MCP flow
results, with indentation and truncation notices.
  - Script failures now provide clearer diagnostics and verdicts.
  - Flow authoring tools recognize and guide creation of script steps.

- **Documentation**
- Added guidance for script syntax, execution limits, output handling,
classification, and remote-run restrictions.
  - Updated flow creation and QA guidance to include script steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com>
2026-09-07 12:08:13 +02:00

237 lines
8.6 KiB
TypeScript

import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import { MIN_SCRIPT_TIMEOUT_MS } from "@argent/configuration-core";
import type { Registry } from "@argent/registry";
import { parseFlow, serializeFlow, type FlowStep } from "../../../src/tools/flows/flow-utils";
import { flowStartRecordingTool } from "../../../src/tools/flows/flow-start-recording";
import { createFlowAddStepTool } from "../../../src/tools/flows/flow-add-step";
import { flowFinishRecordingTool } from "../../../src/tools/flows/flow-finish-recording";
import { summarizeStep } from "../../../src/tools/flows/flow-step-definitions";
const parse = (yaml: string): FlowStep[] => parseFlow(yaml).steps;
const step = (body: string): FlowStep[] => parse(`steps:\n - script: ${body}\n`);
describe("script step syntax", () => {
it("parses the canonical map form", () => {
expect(step("{ path: scripts/seed.mjs }")).toEqual([
{ kind: "script", path: "scripts/seed.mjs" },
]);
});
it("parses a time limit beside the path", () => {
expect(step("{ path: scripts/seed.mjs, timeout: 45000 }")).toEqual([
{ kind: "script", path: "scripts/seed.mjs", timeout: 45000 },
]);
});
it("round-trips through serialization, minimal spelling intact", () => {
const yaml = serializeFlow({
executionPrerequisite: "",
steps: [{ kind: "script", path: "scripts/seed.mjs" }],
});
expect(yaml).toContain("path: scripts/seed.mjs");
expect(yaml).not.toContain("timeout");
expect(parseFlow(yaml).steps).toEqual([{ kind: "script", path: "scripts/seed.mjs" }]);
});
it("round-trips a time limit", () => {
const steps: FlowStep[] = [{ kind: "script", path: "../shared/seed.mjs", timeout: 5000 }];
expect(parseFlow(serializeFlow({ executionPrerequisite: "", steps })).steps).toEqual(steps);
});
it("summarizes as its own kind, with the limit that changes what runs", () => {
expect(summarizeStep({ kind: "script", path: "scripts/seed.mjs" }, 1)).toBe(
"1. script: scripts/seed.mjs"
);
expect(summarizeStep({ kind: "script", path: "scripts/seed.mjs", timeout: 5000 }, 2)).toBe(
"2. script: scripts/seed.mjs (timeout 5000ms)"
);
});
});
describe("script step rejections", () => {
it("refuses a bare path — the value is always a map", () => {
expect(() => step("scripts/seed.mjs")).toThrow(/takes a map, not a bare path/);
});
it("refuses an option written beside the directive key", () => {
expect(() => parse("steps:\n - script: { path: seed.mjs }\n timeout: 30000\n")).toThrow(
/step options go inside the `script:` value, not beside it/
);
});
it("refuses an unknown key inside the map", () => {
expect(() => step("{ path: seed.mjs, retries: 3 }")).toThrow(
/script has unknown key `retries`.*allowed keys: path, timeout/s
);
});
it("refuses `env`, whose release has not landed", () => {
expect(() => step("{ path: seed.mjs, env: { TOKEN: abc } }")).toThrow(
/script has unknown key `env`/
);
});
it("refuses a missing, empty or non-string path", () => {
for (const body of [
"{ timeout: 1000 }",
"{ path: }",
'{ path: "" }',
"{ path: 42 }",
"{ path: true }",
]) {
expect(() => step(body), body).toThrow(/needs a `path`/);
}
});
it("refuses a non-positive or non-finite timeout", () => {
for (const body of [
"{ path: seed.mjs, timeout: 0 }",
"{ path: seed.mjs, timeout: -1 }",
"{ path: seed.mjs, timeout: .inf }",
"{ path: seed.mjs, timeout: .nan }",
"{ path: seed.mjs, timeout: fast }",
]) {
expect(() => step(body), body).toThrow(/script.timeout needs a positive number/);
}
});
it("admits a fractional time limit above the floor, unlike idle.stableFor", () => {
expect(step("{ path: seed.mjs, timeout: 1500.5 }")).toEqual([
{ kind: "script", path: "seed.mjs", timeout: 1500.5 },
]);
});
it("refuses a time limit the step spends on its own startup", () => {
for (const body of [
"{ path: seed.mjs, timeout: 0.5 }",
"{ path: seed.mjs, timeout: 1 }",
"{ path: seed.mjs, timeout: 30 }",
"{ path: seed.mjs, timeout: 99.9 }",
]) {
expect(() => step(body), body).toThrow(
/script.timeout is in milliseconds and needs at least 100/
);
}
});
it("admits the floor itself and the sub-second limits above it", () => {
expect(step("{ path: seed.mjs, timeout: 100 }")).toEqual([
{ kind: "script", path: "seed.mjs", timeout: 100 },
]);
expect(step("{ path: seed.mjs, timeout: 800 }")).toEqual([
{ kind: "script", path: "seed.mjs", timeout: 800 },
]);
});
// The literals above read 100 because the shared bound does. Parse and the
// executor floor the same value from the same constant, so a change to it
// moves both at once — pinned here so a second literal cannot reappear and
// let the two drift apart while every fixed-number case still passes.
it("takes its floor from the bound the executor clamps to", () => {
const floor = MIN_SCRIPT_TIMEOUT_MS;
expect(step(`{ path: seed.mjs, timeout: ${floor} }`)).toEqual([
{ kind: "script", path: "seed.mjs", timeout: floor },
]);
expect(() => step(`{ path: seed.mjs, timeout: ${floor - 1} }`)).toThrow(
new RegExp(`script.timeout is in milliseconds and needs at least ${floor}\\b`)
);
});
it("refuses a body that is not a map", () => {
for (const body of ["", "[seed.mjs]", "42"]) {
expect(() => step(body), body).toThrow(/script needs \{ path, timeout\? \}|takes a map/);
}
});
});
describe("script path rules, shared with a run: target", () => {
it("refuses a backslash", () => {
expect(() => step("{ path: scripts\\\\seed.mjs }")).toThrow(/uses forward slashes/);
});
it("refuses an absolute path", () => {
expect(() => step("{ path: /abs/seed.mjs }")).toThrow(/must be relative to the flow file/);
});
it("refuses a Windows drive-relative prefix", () => {
// `C:foo` resolves against that drive's own current directory.
expect(() => step("{ path: C:seed.mjs }")).toThrow(/must be relative to the flow file/);
expect(() => step("{ path: 'C:/seed.mjs' }")).toThrow(/must be relative to the flow file/);
});
it("refuses an uppercase extension, saying which spelling it wants", () => {
expect(() => step("{ path: scripts/SEED.MJS }")).toThrow(/lowercase .mjs extension/);
});
it("refuses any other extension, and never completes a bare name", () => {
for (const body of ["{ path: seed }", "{ path: seed.js }", "{ path: seed.cjs }"]) {
expect(() => step(body), body).toThrow(/must end in .mjs/);
}
});
it("refuses a basename outside the charset", () => {
for (const body of [
"{ path: 'my seed.mjs' }",
"{ path: seed.step.mjs }",
"{ path: 'sc/.mjs' }",
]) {
expect(() => step(body), body).toThrow(/filename must match/);
}
});
it("admits `..`, because shared code may sit outside the flow's directory", () => {
expect(step("{ path: ../../shared/seed.mjs }")).toEqual([
{ kind: "script", path: "../../shared/seed.mjs" },
]);
});
});
let root: string;
const registry = {
invokeTool: vi.fn(async () => ({ ok: true })),
getTool: vi.fn(() => ({ inputSchema: { properties: { udid: {} } } })),
} as unknown as Registry;
beforeEach(async () => {
root = await fs.mkdtemp(path.join(os.tmpdir(), "flow-script-parse-"));
});
afterEach(async () => {
await fs.rm(root, { recursive: true, force: true });
});
describe("a hand-written script step under the recorder", () => {
it("survives a flow-add-step append and the finish that re-reads the file", async () => {
const flowFile = path.join(root, ".argent", "flows", "seeded.yaml");
await flowStartRecordingTool.execute({}, { name: "seeded", project_root: root });
await fs.writeFile(
flowFile,
"steps:\n - script: { path: scripts/seed.mjs, timeout: 5000 }\n",
"utf8"
);
await createFlowAddStepTool(registry).execute(
{},
{
name: "seeded",
project_root: root,
command: "keyboard",
args: JSON.stringify({ text: "after" }),
}
);
const finished = await flowFinishRecordingTool.execute(
{},
{ name: "seeded", project_root: root }
);
expect(parseFlow(finished.flowFile).steps).toEqual([
{ kind: "script", path: "scripts/seed.mjs", timeout: 5000 },
{ kind: "tool", name: "keyboard", args: { text: "after" } },
]);
expect(finished.summary[0]).toBe("1. script: scripts/seed.mjs (timeout 5000ms)");
});
});