mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
cd4190b80e
Now verified end-to-end on the real Mac: install/introspection/validation, the android tier (against a booted emulator), and the chromium tier all pass from a locally packed tgz; the rn tier auto-skips without `~/dev/bluesky` as designed. ## What this is A release-gating end-to-end harness under `scripts/e2e-full/` that starts from **only a `swmansion-argent-*.tgz` bundle** and exercises the whole product: the install flow, every CLI command, all 70 tools' argument validation, and a happy-path run of every tool against real devices. Intended to be run on the real Linux box and the real Mac before a release. Everything runs under a throwaway `HOME` + npm prefix, so it never touches the machine's real `~/.argent`, editor MCP configs, or global packages (safe on a shared box). It emits a JSONL log + a markdown coverage report and exits non-zero if any hard assertion fails. ## Phases | phase | needs | covers | |---|---|---| | `install` | npm + network | `npm i -g <tgz>`, bundled binaries, `init` (global + `--local`), `update`, `uninstall`, telemetry, MCP-config generation | | `introspection` | — | `--version/--help`, `tools describe` for **all 70** tools, feature flags, `server start/status/logs/stop`, `link/unlink` | | `validation` | — | per tool: missing-required / bad-enum / bad-type rejection (schema-driven, no hardware) | | `android` | Android emulator | happy-path of every touch/gesture/screenshot/app-lifecycle tool | | `chromium` | Electron (bundled optional dep) + a display | boots a generated Electron app; drives CDP tools (scroll/drag/tabs/cookies/storage) | | `rn` | `~/dev/bluesky` + Android device | debugger + react/native profiler + network chain against the real Bluesky app | Tiers auto-skip with a recorded reason when prerequisites are missing. iOS / tvOS / Vega are intentionally out of scope for now. ## What I actually verified (real runs, Linux) - **install** 18✓/0✗, **introspection** 85✓/0✗ (all 70 `tools describe`), **validation** 81✓/0✗, **chromium** 24✓/0✗ (real Electron app + CDP). - Holistic run: **190✓ / 0✗ / 34∼, exit 0.** Failure-gate confirmed (non-zero on fail), `bash -n` clean on all files, all 72 android/rn JSON arg literals linted. ## What I did NOT verify live (please scrutinize) - **`android` and `rn` tiers were never run against a live device.** The Android emulator would not stay up on my box (memory-constrained shared machine; QEMU threads hung seconds after boot across ~7 attempts). The code is complete and uses the same machinery the `chromium` tier exercised live, but it has **not** been proven end-to-end. This is the main reason for the "not certain how well it works" caveat. - The RN tier's exact debugger/profiler arg shapes are derived from `tools describe`, not confirmed against a running Metro. ## Run it ```bash bash scripts/e2e-full/run-e2e.sh # everything applicable to this OS bash scripts/e2e-full/run-e2e.sh --skip-install --phase introspection,validation # fast offline core bash scripts/e2e-full/run-e2e.sh --phase android --android-serial emulator-5554 # inject a device ```
102 lines
4.4 KiB
Bash
102 lines
4.4 KiB
Bash
#!/usr/bin/env bash
|
|
# Phase 1 — Introspection (offline, no device).
|
|
#
|
|
# Exercises the CLI surface itself and proves `argent tools describe` works for
|
|
# EVERY tool (this is where all 70 tools first get a recorded case). Also
|
|
# round-trips flags, the server lifecycle, and remote-link config — all against
|
|
# the sandbox HOME.
|
|
|
|
run_phase() {
|
|
local P=introspection
|
|
|
|
# --- help / version / unknown command ------------------------------------
|
|
if argent_cli --version && [ "$CLI_OUT" = "$TGZ_VERSION" ]; then
|
|
pass "$P" cli version
|
|
else
|
|
fail "$P" cli version "got '$CLI_OUT' want '$TGZ_VERSION'"
|
|
fi
|
|
|
|
argent_cli --help
|
|
case "$CLI_OUT" in *"Usage: argent"*) pass "$P" cli help;; *) fail "$P" cli help "no usage banner";; esac
|
|
|
|
argent_cli definitely-not-a-command
|
|
if [ $? -ne 0 ]; then pass "$P" cli unknown-command-exits-nonzero; else fail "$P" cli unknown-command-exits-nonzero "exited 0"; fi
|
|
|
|
# --- tool list: expect the full published set ----------------------------
|
|
local names n
|
|
names="$(list_tool_names)"
|
|
n="$(printf '%s\n' "$names" | grep -c .)"
|
|
if [ "$n" -ge 60 ]; then
|
|
pass "$P" tools "list ($n tools)"
|
|
else
|
|
fail "$P" tools "list ($n tools)" "suspiciously few tools"
|
|
fi
|
|
|
|
# --- describe EVERY tool (records a case per tool) ------------------------
|
|
local t model
|
|
while read -r t; do
|
|
[ -z "$t" ] && continue
|
|
model="$(parse_tool_model "$t")"
|
|
if [ -s "$model" ] || argent_cli tools describe "$t"; then
|
|
# a tool with no params is legitimate; require the describe call itself to succeed
|
|
if argent_cli tools describe "$t"; then
|
|
pass "$P" "$t" describe "$(model_flag_count "$model") flags"
|
|
else
|
|
fail "$P" "$t" describe "describe exited non-zero"
|
|
fi
|
|
fi
|
|
done <<< "$names"
|
|
|
|
# --- feature flags round-trip (uses a predefined flag name) --------------
|
|
argent_cli flags; [ $? -eq 0 ] && pass "$P" flags list || fail "$P" flags list "$CLI_OUT"
|
|
local PROBE_FLAG="disable-auto-screenshot"
|
|
if argent_cli enable "$PROBE_FLAG" --scope global; then
|
|
argent_cli flags
|
|
case "$CLI_OUT" in *"$PROBE_FLAG"*) pass "$P" flags enable;; *) fail "$P" flags enable "flag not shown after enable";; esac
|
|
argent_cli disable "$PROBE_FLAG" --scope global && pass "$P" flags disable || fail "$P" flags disable "$CLI_OUT"
|
|
else
|
|
fail "$P" flags enable "enable exited non-zero: $(printf '%s' "$CLI_OUT" | head -1)"
|
|
fi
|
|
|
|
# --- telemetry status -----------------------------------------------------
|
|
argent_cli telemetry status; [ $? -eq 0 ] && pass "$P" telemetry status || fail "$P" telemetry status "$CLI_OUT"
|
|
|
|
# --- server lifecycle (start/status/logs/stop) ---------------------------
|
|
# Start from a clean slate: kill any server auto-spawned by the calls above so
|
|
# we exercise an explicit `server start`.
|
|
argent_cli server stop >/dev/null 2>&1 || true
|
|
if argent_cli server start --detach --no-auth --port 0; then
|
|
pass "$P" server start
|
|
else
|
|
fail "$P" server start "$(printf '%s' "$CLI_OUT" | head -1)"
|
|
fi
|
|
local i ready=0
|
|
for i in $(seq 1 30); do server_running && { ready=1; break; }; sleep 1; done
|
|
[ "$ready" -eq 1 ] && pass "$P" server ready || fail "$P" server ready "status never reported healthy"
|
|
|
|
argent_cli server status --json
|
|
if printf '%s' "$CLI_OUT" | jq -e '.running==true' >/dev/null 2>&1; then
|
|
pass "$P" server status "port=$(printf '%s' "$CLI_OUT" | jq -r .port)"
|
|
else
|
|
fail "$P" server status "$(printf '%s' "$CLI_OUT" | head -1)"
|
|
fi
|
|
argent_cli server logs; [ $? -eq 0 ] && pass "$P" server logs || skip "$P" server logs "logs exited non-zero"
|
|
|
|
# Prove stop works, then bring one back for downstream phases.
|
|
if argent_cli server stop; then pass "$P" server stop; else fail "$P" server stop "$(printf '%s' "$CLI_OUT" | head -1)"; fi
|
|
ensure_server || warn "could not restart server after stop test"
|
|
|
|
# --- link / unlink round-trip (sandbox ~/.argent/link.json) --------------
|
|
# NB: setting a link overrides discovery; unset it immediately so downstream
|
|
# phases keep using ARGENT_TOOLS_URL.
|
|
if argent_cli link "http://127.0.0.1:${E2E_TOOLS_PORT}"; then
|
|
pass "$P" link set
|
|
if [ -f "$E2E_HOME/.argent/link.json" ]; then pass "$P" link persisted; else skip "$P" link persisted "no link.json"; fi
|
|
argent_cli unlink && pass "$P" link unset || fail "$P" link unset "$CLI_OUT"
|
|
else
|
|
fail "$P" link set "link exited non-zero: $(printf '%s' "$CLI_OUT" | head -1)"
|
|
fi
|
|
|
|
# leave the server running for downstream phases (validation reuses it)
|
|
}
|