Files
Ignacy Łątka 05e619e55d ci: device E2E smoke tests for every hosted-CI platform (#371)
## What

`wayland-e2e.yml` boots an Android AVD on Linux under headless Weston
and asserts the full pipeline works without a display. It guards exactly
one cell of the support matrix. This adds the same
boot→screenshot→gesture-tap smoke test for every other (target × host)
cell that can run on GitHub-hosted runners.

| target | host | job | status |
|---|---|---|---|
| Android emulator | Linux (KVM) | `wayland-e2e.yml` (existing) | ✅ |
| iOS simulator | macOS | `ios-sim-macos` | ✅ |
| Chromium / Electron | Linux (Xvfb) | `chromium-linux` | ✅ |
| Chromium / Electron | macOS | `chromium-macos` | ✅ |

## How

- **`scripts/e2e/drive-device.sh`** — shared driver. Boots one device
through the tool-server, then asserts `booted:true` → screenshot has
real (non-blank) pixels → `gesture-tap` round-trips. Every tool takes
the device id as `udid` and screenshots come back as the same
`data.image.hostPath` envelope on all platforms, so the body is uniform;
each job just supplies the cell-specific boot JSON + device id.
- **`packages/tool-server/test/fixtures/electron-smoke-app/`** — minimal
self-contained Electron app the Chromium jobs point `electronAppPath`
at. It vendors its own electron via its own lockfile (kept out of the
repo root lockfile); CI `npm ci`s it so `boot-electron` resolves
`./node_modules/.bin/electron`.
- **`.github/workflows/e2e-device-smoke.yml`** — three jobs,
`workflow_dispatch` + path-filtered `pull_request`. iOS downloads the
darwin native binaries (injection must succeed for `bootIos` to report
`booted:true`); Chromium-Linux runs under Xvfb with `--no-sandbox`. The
tool-server is started with `TS_NODE_TRANSPILE_ONLY=1` so cold ts-node
startup doesn't flakily exceed the readiness poll on loaded macOS
runners (types are still enforced by the typecheck job).

## Known gaps (supported on real machines, not runnable on hosted
runners)

- **Android emulator on macOS** — the arm64 emulator needs HVF
(`-enable-hvf`), but hosted macOS runners are themselves VMs with no
nested virtualization. Confirmed by running it: qemu dies with `HVF
error: HV_UNSUPPORTED` ~18s into boot, regardless of GPU mode or RAM.
The Android boot/screenshot/tap path is already regression-guarded on
Linux via KVM (`wayland-e2e.yml`), so macOS adds no runnable coverage.
- **Physical devices** — real iOS (CoreDevice HID) and real Android
(adb-USB) have no attached hardware on hosted runners.

Both would need self-hosted runners (a Mac exposing HVF / wired-up
devices). Documented in the workflow header.

## Verification

- All checks green: the 3 device jobs + ESLint + Prettier + Unit tests.
- Locally (before CI): ran `drive-device.sh` end-to-end against a real
tool-server → boot `booted:true`, screenshot **142 KB** (floor 20 KB),
tap `tapped:true`, exit 0; and confirmed via the Argent stack that a tap
at (0.5, 0.5) lands on the fixture's centered button and increments its
counter.
2026-06-26 17:44:17 +02:00

186 lines
8.6 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name: Device E2E smoke
# Boot-and-drive smoke tests for every (target × host) cell of the support
# matrix that runs on GitHub-hosted runners. Each job boots ONE device through
# the tool-server and asserts the headless interaction pipeline end-to-end via
# scripts/e2e/drive-device.sh: booted:true → screenshot has real pixels →
# gesture-tap round-trips.
#
# ┌────────────────────┬───────────────┬──────────────────────────────────┐
# │ target │ host │ where │
# ├────────────────────┼───────────────┼──────────────────────────────────┤
# │ Android emulator │ Linux (KVM) │ wayland-e2e.yml (separate file: │
# │ │ │ also asserts gpu.mode=swiftshader)│
# │ iOS simulator │ macOS │ ios-sim-macos (this file) │
# │ Chromium / Electron│ Linux (Xvfb) │ chromium-linux (this file) │
# │ Chromium / Electron│ macOS │ chromium-macos (this file) │
# └────────────────────┴───────────────┴──────────────────────────────────┘
#
# KNOWN GAPS — cells Argent supports on real machines but that GitHub-hosted
# runners can't run, so they are intentionally not jobs here:
# - Android emulator on macOS: the arm64 emulator needs HVF (`-enable-hvf`),
# but hosted macOS runners are themselves VMs with no nested virtualization
# — qemu dies with "HVF error: HV_UNSUPPORTED" seconds into boot. The
# Android boot/screenshot/tap path is already regression-guarded on Linux
# via KVM (wayland-e2e.yml), so macOS adds no runnable coverage here.
# - Physical devices: real iOS (CoreDevice HID) and real Android (adb-USB)
# have no attached hardware on hosted runners.
# Both would need self-hosted runners (a Mac exposing HVF / wired-up devices).
#
# Manual + on-change trigger only — the macOS jobs are slow and billed.
on:
workflow_dispatch:
pull_request:
branches:
- main
paths:
- "packages/tool-server/src/tools/devices/**"
- "packages/tool-server/src/tools/gesture-tap/**"
- "packages/tool-server/src/tools/screenshot/**"
# Transport layer that drives screenshot/gesture-tap for each platform —
# where the #391 screenshot regression lived, so changes here must re-run
# these smoke jobs too.
- "packages/tool-server/src/blueprints/simulator-server.ts"
- "packages/tool-server/src/blueprints/chromium-cdp.ts"
- "packages/tool-server/src/utils/simulator-client.ts"
- "packages/tool-server/test/fixtures/electron-smoke-app/**"
- "scripts/e2e/drive-device.sh"
- ".github/workflows/e2e-device-smoke.yml"
# The shared composite action is part of the test harness too.
- ".github/actions/start-tool-server/action.yml"
# Cancel superseded runs on the same ref: rapid pushes to a PR would otherwise
# stack up the slow, billed macOS jobs (see header) with no cancellation.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# ── iOS simulator on macOS ────────────────────────────────────────────────
ios-sim-macos:
name: macOS + iOS E2E
runs-on: macos-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/start-tool-server
- name: Download signed native binaries (darwin)
# iOS needs the simulator-server (screenshot/gesture-tap backend) AND the
# dylibs + ax-service: bootIos() only returns booted:true when native-
# devtools injection does NOT give up (boot-device.ts), so missing dylibs
# would surface here as a hard failure rather than a silent skip. Only
# consumed at boot time, so it can run after the server starts — and after
# start-tool-server has set up Node (download-native-binaries.sh reads the
# Android manifest version with `node -p`).
run: |
bash scripts/download-simulator-server.sh
bash scripts/download-native-binaries.sh
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Pick an available simulator UDID
run: |
UDID=$(xcrun simctl list devices available --json | python3 -c "import json,sys; d=json.load(sys.stdin)['devices']; print(next(dev['udid'] for rt in d for dev in d[rt] if dev.get('isAvailable') and 'iPhone' in dev['name']))")
echo "Chosen simulator: $UDID"
echo "UDID=$UDID" >> "$GITHUB_ENV"
- name: Boot + drive
run: bash scripts/e2e/drive-device.sh
env:
BOOT_JSON: '{"udid":"${{ env.UDID }}"}'
DEVICE_ID: ${{ env.UDID }}
- if: always()
uses: actions/upload-artifact@v7
with:
name: e2e-ios-sim-macos
path: |
${{ runner.temp }}/smoke-shot.png
${{ runner.temp }}/smoke-shot.json
~/.argent/tool-server.log
if-no-files-found: warn
retention-days: 7
# ── Chromium / Electron on Linux (Xvfb) ───────────────────────────────────
chromium-linux:
name: Linux + Chromium E2E
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v6
- name: Start Xvfb
# boot-electron spawns the Electron child inheriting the tool-server's
# env, so DISPLAY must be set (in GITHUB_ENV) BEFORE the tool-server
# starts in the start-tool-server action below.
run: |
sudo apt-get update -y
sudo apt-get install -y --no-install-recommends xvfb
Xvfb :99 -screen 0 1280x1024x24 >/tmp/xvfb.log 2>&1 &
echo "DISPLAY=:99" >> "$GITHUB_ENV"
- uses: ./.github/actions/start-tool-server
- name: Vendor the fixture's electron
# The fixture brings its own electron (kept out of the root lockfile);
# boot-electron resolves ./node_modules/.bin/electron from the app dir.
# Standalone project (not a packages/* workspace member), so this ci is
# independent of the root install. Electron is only needed at boot time,
# so vendoring it after the server starts is fine.
run: npm ci
working-directory: packages/tool-server/test/fixtures/electron-smoke-app
- name: Boot + drive
run: bash scripts/e2e/drive-device.sh
env:
BOOT_JSON: '{"electronAppPath":"${{ github.workspace }}/packages/tool-server/test/fixtures/electron-smoke-app","electronPort":19222,"electronArgs":["--no-sandbox"]}'
DEVICE_ID: chromium-cdp-19222
- if: always()
uses: actions/upload-artifact@v7
with:
name: e2e-chromium-linux
path: |
${{ runner.temp }}/smoke-shot.png
${{ runner.temp }}/smoke-shot.json
/tmp/xvfb.log
~/.argent/tool-server.log
if-no-files-found: warn
retention-days: 7
# ── Chromium / Electron on macOS ──────────────────────────────────────────
chromium-macos:
name: macOS + Chromium E2E
runs-on: macos-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/start-tool-server
- name: Vendor the fixture's electron
run: npm ci
working-directory: packages/tool-server/test/fixtures/electron-smoke-app
- name: Boot + drive
# macOS runner has a window server and Electron's sandbox is fine there,
# so no Xvfb and no --no-sandbox.
run: bash scripts/e2e/drive-device.sh
env:
BOOT_JSON: '{"electronAppPath":"${{ github.workspace }}/packages/tool-server/test/fixtures/electron-smoke-app","electronPort":19222}'
DEVICE_ID: chromium-cdp-19222
- if: always()
uses: actions/upload-artifact@v7
with:
name: e2e-chromium-macos
path: |
${{ runner.temp }}/smoke-shot.png
${{ runner.temp }}/smoke-shot.json
~/.argent/tool-server.log
if-no-files-found: warn
retention-days: 7