mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
cd4190b80e
Now verified end-to-end on the real Mac: install/introspection/validation, the android tier (against a booted emulator), and the chromium tier all pass from a locally packed tgz; the rn tier auto-skips without `~/dev/bluesky` as designed. ## What this is A release-gating end-to-end harness under `scripts/e2e-full/` that starts from **only a `swmansion-argent-*.tgz` bundle** and exercises the whole product: the install flow, every CLI command, all 70 tools' argument validation, and a happy-path run of every tool against real devices. Intended to be run on the real Linux box and the real Mac before a release. Everything runs under a throwaway `HOME` + npm prefix, so it never touches the machine's real `~/.argent`, editor MCP configs, or global packages (safe on a shared box). It emits a JSONL log + a markdown coverage report and exits non-zero if any hard assertion fails. ## Phases | phase | needs | covers | |---|---|---| | `install` | npm + network | `npm i -g <tgz>`, bundled binaries, `init` (global + `--local`), `update`, `uninstall`, telemetry, MCP-config generation | | `introspection` | — | `--version/--help`, `tools describe` for **all 70** tools, feature flags, `server start/status/logs/stop`, `link/unlink` | | `validation` | — | per tool: missing-required / bad-enum / bad-type rejection (schema-driven, no hardware) | | `android` | Android emulator | happy-path of every touch/gesture/screenshot/app-lifecycle tool | | `chromium` | Electron (bundled optional dep) + a display | boots a generated Electron app; drives CDP tools (scroll/drag/tabs/cookies/storage) | | `rn` | `~/dev/bluesky` + Android device | debugger + react/native profiler + network chain against the real Bluesky app | Tiers auto-skip with a recorded reason when prerequisites are missing. iOS / tvOS / Vega are intentionally out of scope for now. ## What I actually verified (real runs, Linux) - **install** 18✓/0✗, **introspection** 85✓/0✗ (all 70 `tools describe`), **validation** 81✓/0✗, **chromium** 24✓/0✗ (real Electron app + CDP). - Holistic run: **190✓ / 0✗ / 34∼, exit 0.** Failure-gate confirmed (non-zero on fail), `bash -n` clean on all files, all 72 android/rn JSON arg literals linted. ## What I did NOT verify live (please scrutinize) - **`android` and `rn` tiers were never run against a live device.** The Android emulator would not stay up on my box (memory-constrained shared machine; QEMU threads hung seconds after boot across ~7 attempts). The code is complete and uses the same machinery the `chromium` tier exercised live, but it has **not** been proven end-to-end. This is the main reason for the "not certain how well it works" caveat. - The RN tier's exact debugger/profiler arg shapes are derived from `tools describe`, not confirmed against a running Metro. ## Run it ```bash bash scripts/e2e-full/run-e2e.sh # everything applicable to this OS bash scripts/e2e-full/run-e2e.sh --skip-install --phase introspection,validation # fast offline core bash scripts/e2e-full/run-e2e.sh --phase android --android-serial emulator-5554 # inject a device ```
117 lines
6.1 KiB
Bash
117 lines
6.1 KiB
Bash
#!/usr/bin/env bash
|
|
# Phase 3 — Android device tier (Linux + macOS).
|
|
#
|
|
# Drives a happy-path of every tool that applies to an Android emulator. The
|
|
# device is EITHER injected (E2E_ANDROID_SERIAL — e.g. a serial the device
|
|
# allocator handed us on a shared machine, already booted) OR booted here from
|
|
# E2E_ANDROID_AVD. If neither is available the whole tier is skipped with a
|
|
# reason so the report still makes sense.
|
|
|
|
# Verify a booted device with this serial is visible to the tool-server.
|
|
_android_present() { # serial
|
|
run_tool list-devices '{}'
|
|
printf '%s' "$RT_JSON" | jq -e --arg s "$1" 'any(.devices[]?; .serial==$s or .udid==$s)' >/dev/null 2>&1
|
|
}
|
|
|
|
# Grab a screenshot to a file and assert it has real pixels. Sets SHOT_PATH.
|
|
_shot_ok() { # phase udid case
|
|
local phase="$1" udid="$2" case="$3"
|
|
if capture_screenshot "$udid" "$E2E_WORK/android-$case.png"; then
|
|
pass "$phase" screenshot "$case (${SHOT_SIZE}B)"; return 0
|
|
else
|
|
fail "$phase" screenshot "$case" "size=${SHOT_SIZE:-0} rc=${SHOT_RC:-?} (blank framebuffer?)"; return 1
|
|
fi
|
|
}
|
|
|
|
run_phase() {
|
|
local P=android
|
|
ensure_server || { skip "$P" tier all "tool-server unavailable"; return 0; }
|
|
|
|
local DEV=""
|
|
if [ -n "${E2E_ANDROID_SERIAL:-}" ]; then
|
|
DEV="$E2E_ANDROID_SERIAL"
|
|
if _android_present "$DEV"; then
|
|
pass "$P" list-devices "injected serial $DEV present"
|
|
else
|
|
# not yet visible — try a boot-device against it in case the sim-server
|
|
# needs to attach; otherwise skip.
|
|
skip "$P" tier all "injected serial $DEV not visible to tool-server"; return 0
|
|
fi
|
|
elif [ -n "${E2E_ANDROID_AVD:-}" ]; then
|
|
log "booting AVD $E2E_ANDROID_AVD"
|
|
run_tool boot-device "{\"avdName\":\"$E2E_ANDROID_AVD\",\"bootTimeoutMs\":840000}"
|
|
if [ "$RT_RC" -eq 0 ] && printf '%s' "$RT_JSON" | jq -e '.booted==true' >/dev/null 2>&1; then
|
|
DEV="$(printf '%s' "$RT_JSON" | jq -r '.serial // .udid // empty')"
|
|
pass "$P" boot-device "booted $DEV"
|
|
else
|
|
fail "$P" boot-device "$(printf '%s' "$RT_OUT" | tr '\n' ' ' | cut -c1-160)"; return 0
|
|
fi
|
|
else
|
|
skip "$P" tier all "no Android device (set E2E_ANDROID_SERIAL or E2E_ANDROID_AVD)"; return 0
|
|
fi
|
|
local U="{\"udid\":\"$DEV\"}"
|
|
|
|
# --- discovery ------------------------------------------------------------
|
|
_shot_ok "$P" "$DEV" baseline || true
|
|
assert_field "$P" describe describe "$U" '(.description|length>0)' 'true'
|
|
assert_ok "$P" await-screen-idle idle "$U"
|
|
assert_ok "$P" await-ui-element exists-probe "{\"udid\":\"$DEV\",\"condition\":\"exists\",\"selector\":{\"text\":\"a\"},\"timeoutMs\":3000}"
|
|
|
|
# --- interaction ----------------------------------------------------------
|
|
assert_true "$P" gesture-tap tap "{\"udid\":\"$DEV\",\"x\":0.5,\"y\":0.5}" '.tapped'
|
|
assert_true "$P" gesture-swipe swipe "{\"udid\":\"$DEV\",\"fromX\":0.5,\"fromY\":0.8,\"toX\":0.5,\"toY\":0.2}" '.swiped'
|
|
assert_ok "$P" gesture-custom custom "{\"udid\":\"$DEV\",\"events\":[{\"type\":\"Down\",\"x\":0.5,\"y\":0.6},{\"type\":\"Move\",\"x\":0.5,\"y\":0.4},{\"type\":\"Up\",\"x\":0.5,\"y\":0.4}]}"
|
|
assert_true "$P" gesture-pinch pinch "{\"udid\":\"$DEV\",\"centerX\":0.5,\"centerY\":0.5,\"startDistance\":0.1,\"endDistance\":0.4}" '.pinched'
|
|
assert_true "$P" gesture-rotate rotate2 "{\"udid\":\"$DEV\",\"centerX\":0.5,\"centerY\":0.5,\"radius\":0.2,\"startAngle\":0,\"endAngle\":90}" '.rotated'
|
|
assert_ok "$P" button home "{\"udid\":\"$DEV\",\"button\":\"home\"}"
|
|
assert_ok "$P" button back "{\"udid\":\"$DEV\",\"button\":\"back\"}"
|
|
assert_ok "$P" rotate landscape "{\"udid\":\"$DEV\",\"orientation\":\"LandscapeLeft\"}"
|
|
assert_ok "$P" rotate portrait "{\"udid\":\"$DEV\",\"orientation\":\"Portrait\"}"
|
|
assert_ok "$P" keyboard text "{\"udid\":\"$DEV\",\"text\":\"hello e2e\"}"
|
|
assert_ok "$P" keyboard key "{\"udid\":\"$DEV\",\"key\":\"enter\"}"
|
|
assert_ok "$P" run-sequence seq "{\"udid\":\"$DEV\",\"steps\":[{\"tool\":\"button\",\"args\":{\"button\":\"home\"}},{\"tool\":\"gesture-tap\",\"args\":{\"x\":0.5,\"y\":0.5}}]}"
|
|
|
|
# --- url navigation -------------------------------------------------------
|
|
assert_true "$P" open-url url "{\"udid\":\"$DEV\",\"url\":\"https://example.com\"}" '.opened'
|
|
|
|
# --- screenshot-diff (two live captures) ----------------------------------
|
|
local b c
|
|
_shot_ok "$P" "$DEV" diff-baseline && b="$SHOT_PATH"
|
|
cp "$b" "$E2E_WORK/diff-base.png" 2>/dev/null || true
|
|
run_tool button "{\"udid\":\"$DEV\",\"button\":\"home\"}" >/dev/null 2>&1
|
|
_shot_ok "$P" "$DEV" diff-current && c="$SHOT_PATH"
|
|
if [ -n "${b:-}" ] && [ -n "${c:-}" ]; then
|
|
assert_ok "$P" screenshot-diff diff "{\"udid\":\"$DEV\",\"baselinePath\":\"$E2E_WORK/diff-base.png\",\"currentPath\":\"$c\"}"
|
|
else
|
|
skip "$P" screenshot-diff diff "could not capture two screenshots"
|
|
fi
|
|
|
|
# --- app lifecycle against a stock system app -----------------------------
|
|
local APP="com.android.settings"
|
|
assert_true "$P" launch-app launch "{\"udid\":\"$DEV\",\"bundleId\":\"$APP\"}" '.launched'
|
|
assert_true "$P" restart-app restart "{\"udid\":\"$DEV\",\"bundleId\":\"$APP\"}" '.restarted'
|
|
|
|
# --- state-only + platform-scoped ----------------------------------------
|
|
assert_ok "$P" dismiss-update dismiss "{\"hours\":1}"
|
|
|
|
# iOS-only tools: recorded as skips so coverage is explicit.
|
|
local iostool
|
|
for iostool in native-describe-screen native-devtools-status native-full-hierarchy \
|
|
native-find-views native-view-at-point native-user-interactable-view-at-point \
|
|
native-network-logs; do
|
|
skip "$P" "$iostool" happy-path "iOS-only (not applicable on Android)"
|
|
done
|
|
skip "$P" tv-remote happy-path "TV-only tier (skipped per scope)"
|
|
skip "$P" reinstall-app happy-path "covered in RN tier with the Bluesky apk"
|
|
|
|
# --- teardown for this device --------------------------------------------
|
|
# Only stop the sim-server if we booted the AVD ourselves; a device injected
|
|
# by the allocator is released by the caller.
|
|
if [ -z "${E2E_ANDROID_SERIAL:-}" ]; then
|
|
assert_ok "$P" stop-simulator-server stop "$U"
|
|
else
|
|
run_tool stop-simulator-server "$U" >/dev/null 2>&1 || true
|
|
skip "$P" stop-simulator-server stop "injected device released by caller"
|
|
fi
|
|
}
|