Ignacy Łątka
c58777fd57
`argent run` answers a missing required flag locally: exit 2, the tool's help block, and one object on stderr under `--json`. Relaxing `udid` moved that answer to the server, which replied with prose in `error` alone - `describeServerValidationFailure` recognises a rejected invocation structurally, found no issue list, and the call fell through to the generic runtime path (exit 1, nothing on the `--json` channel). The refusal now sends the three fields a schema rejection does: prose in `message`, the list in `issues`, that list as JSON in `error` for a CLI released before `issues` existed. The nested `list-devices` emits its own invoke/complete pair and was minting a fresh unattributed id, so every device-less call added an anonymous `list-devices` to telemetry - the very metric that would show whether this feature removed those calls. Dispatch it through `invokeSubTool` with a recorder bound to the request, the mechanism run-sequence and flow-execute already use. `invokeSubTool`'s context parameter narrows to the two fields it reads, so the HTTP layer can hand over attribution before it owns a `ToolContext`. Also: - `flow-add-step` still needs `udid` in nested args by design - a recorded step pins its own device - so say that in the `args` description - `scripts/e2e-full`'s README promised missing-required coverage for every tool, which no longer holds for the ten whose only required field was `udid`
Full Argent E2E harness
A release-gating end-to-end test that starts from nothing but a
swmansion-argent-*.tgz bundle and exercises the whole product: the install
flow, every CLI command, every tool's argument validation, and a happy-path run
of every tool that applies against real devices.
Run it on the real Linux box and the real Mac before a release.
Quick start
# from the repo root, with a swmansion-argent-*.tgz present there:
bash scripts/e2e-full/run-e2e.sh
# a subset of phases:
bash scripts/e2e-full/run-e2e.sh --phase install,introspection,validation
# offline core only, driving the unpacked bundle (no npm install, fast):
bash scripts/e2e-full/run-e2e.sh --skip-install --phase introspection,validation
A markdown report is written to scripts/e2e-full/results/report-<ts>.md and the
raw per-case log to results/e2e-<ts>.jsonl. The process exits non-zero if any
hard assertion failed (skips do not fail the run).
What it does (phases)
| phase | needs | covers |
|---|---|---|
install |
npm + network | npm i -g <tgz>, bundled binaries, init (global + --local), update, uninstall, telemetry, MCP-config generation |
introspection |
— | --version/--help, tools, tools describe for every published tool, feature flags, server start/status/logs/stop, link/unlink |
validation |
— | bad-enum / bad-type rejection for every tool, missing-required for every tool that still has one (deterministic, no hardware) |
android |
Android emulator | happy-path of every touch/gesture/screenshot/app-lifecycle tool |
chromium |
Electron (bundled optional dep) + a display | boots a generated Electron app; drives CDP tools (scroll/drag/tabs/cookies/storage) |
rn |
~/dev/bluesky + Android device |
debugger + react/native profiler + network chain against the real Bluesky app |
Tiers auto-skip (with a recorded reason) when their prerequisites are missing, so a partial run still produces a meaningful report. iOS / tvOS / Vega tiers are intentionally out of scope.
Isolation
Everything runs under a throwaway HOME and npm prefix ($(mktemp -d)), so the
real machine's ~/.argent, editor MCP configs, and global packages are never
touched — safe to run on a shared box. Add --keep to inspect the sandbox after.
Providing a device
The device tiers need a booted device. Two ways:
- Inject an already-booted one (recommended on shared/CI machines):
--android-serial emulator-5554(the harness attaches, doesn't boot/teardown it). - Let the harness boot it:
--android-avd Pixel_9a(usesboot-device). The booted serial is published to the RN tier, so it runs against the same device, and the emulator is shut down in the cleanup phase — after every tier that uses it, and on an aborted run too.
The Chromium tier needs no device — it generates and boots its own Electron app.
It requires DISPLAY; on a headless Linux box run the whole harness under
xvfb-run, which supplies one. Having xvfb-run merely installed is not enough,
because nothing wraps the Electron spawn in it.
RN (Bluesky) tier
E2E_RN_DIR=~/dev/bluesky E2E_RN_PKG=xyz.blueskyweb.app \
bash scripts/e2e-full/run-e2e.sh --phase rn --android-serial <serial>
Assumes the Bluesky dev-client is already built and installed on the device; the
tier skips itself if the package is absent. Pass E2E_RN_BUILD=1 to let it run
expo run:android first (slow). It starts Metro if nothing is serving the port
and tears down only a Metro it started — one you were already running is left
alone.
Flags
--tgz PATH tarball to test (default: newest swmansion-argent-*.tgz at repo root)
--phase a,b,c subset of: install introspection validation android chromium rn
--skip-install drive the unpacked bundle directly (offline phases only; skips `install`)
--system install to the REAL global prefix (dedicated release machine only)
--android-serial S use an already-booted Android device
--android-avd NAME boot this AVD via boot-device
--keep leave the sandbox dir for inspection
Layout
run-e2e.sh orchestrator: env setup, phase dispatch, report, exit code
lib/common.sh logging, argent_cli/run_tool, assert_* helpers, server + screenshot helpers
lib/discover-tools.sh parses `argent tools describe` into per-tool arg models
lib/report.py JSONL -> markdown (per-phase + per-tool coverage matrix + failures)
phases/*.sh one run_phase() per phase
results/ generated JSONL + report (gitignored)