skills/callstack__agent-device
1
0
Fork 0
mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
fix/web-snapshot-disabled-annotation
callstack__agent-device/.github/workflows/replays-manual.yml
T
Michał Pierzchała ccf64f6797 ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) (#1794) 
* ci: move parked device replay suites to a dispatch-only workflow (#1781 A1)

Both full-tier device jobs have failed every scheduled run since 2026-07-24: the
Android suite inside full-tier scenarios that had never executed end to end, the
iOS suite on varying steps. They move to .github/workflows/replays-manual.yml,
which has no `schedule:`, so the schedule stops emitting a guaranteed failure while
the suites stay runnable on demand.

A job-level `if: github.event_name == 'workflow_dispatch'` would have looked the
same and lied: `workflowLanes()` decides `qualifying` per workflow FILE and never
reads job-level `if:`, so the manifest kept reporting replay-android, replay-ios,
and replay-ios-device as scheduled-lane owners — the silent-owner-loss failure the
manifest exists to catch. A separate file is what the file-level model already
reads correctly.

Those three checks now have no pull_request/schedule owner, so they are declared as
MANUAL_ONLY_OWNERS rather than folded into UNPROVABLE_OWNERS, whose claim ("it runs,
this loader cannot see it") is no longer true for replay-android. check:gate-manifest
drops from 48 to 46 wired checks and names the three on every run. Two tests pin it:
a dispatch-only lane is non-qualifying however many gates it declares, and every
manual-only declaration must name a registered check that no qualifying lane owns, so
a re-scheduled lane cannot keep a stale exemption.

* ci: attest manual-only checks against their dispatch lane (#1781 A1)

Review P1: MANUAL_ONLY_OWNERS was a negative allowlist — it proved each entry named a
registered check no qualifying lane owned, but nothing tied the entry to a lane that can
still run it. Deleting a parked job, or its run-gate step, would have left the manifest
green and still printing the check as manual-only: parked coverage silently becoming
deleted coverage.

Each entry now names its dispatch lane, and a new 'manual-only' audit assertion resolves
that name against the derived model: the lane must exist, must still be dispatch-only, and
must still declare the gate. replay-android carries an explicit `opaque` flag because its
gate sits inside the third-party emulator action's `script:` (#1429), so the job's
existence is the whole attestation the model can make — and the flag says so rather than
letting an unreadable lane look like a declaring one.

Four regressions pin both directions: deleting a declaration reports the check as unowned;
deleting the parked job fails with 'no workflow defines'; re-scheduling the lane fails
until the entry is dropped; and a parked lane that loses its run-gate step fails unless the
entry is opaque.

* ci: make manual-only mean dispatch-only, not merely non-qualifying (#1781 A1)

Review follow-up: the attestation checked `qualifying === false`, which is true of any lane
that is not pull_request/schedule. Swapping `workflow_dispatch` for `push` in
replays-manual.yml would have kept the audit green and the checks printed as manual-only,
while the runs nobody starts by hand quietly started themselves on every push.

The lane model now keeps the trigger names instead of collapsing them into that one bit, and
the manual-only assertion requires `workflow_dispatch` and nothing else. Three planted
regressions cover the gap the review named: a parked lane re-triggered by `push` fails, a
parked lane with no trigger at all fails, and the loader test pins that trigger kinds survive
into the model (a push lane reads `[push]`, the nightly reads `[schedule, workflow_dispatch]`).
2026-08-18 09:59:34 +02:00

198 lines
8.2 KiB
YAML
Raw Permalink Blame History

name: Replay Manual
# Parked here by #1781 A1. Both suites failed every scheduled run from 2026-07-24 on — the
# Android suite inside full-tier scenarios that had never executed end to end, the iOS suite
# on varying steps — so on a schedule they only produced noise. Manual dispatch keeps them
# runnable while they are being repaired; `pnpm check:gate-manifest` reports the checks they
# declare as manual-only (scripts/gate/declarations.ts) so the missing owner stays visible.
# Move these jobs back into replays-nightly.yml once a dispatch run is green.
on:
workflow_dispatch:
permissions:
contents: read
actions: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
manual-android:
name: Android Full Emulator Suite
runs-on: ubuntu-latest
timeout-minutes: 15
env:
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/android-e2e-state
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Setup Android replay host
id: android-replay-host
uses: ./.github/actions/setup-android-replay-host
with:
package-helpers: 'true'
- name: Restore Android fixture APK
id: fixture-app
uses: ./.github/actions/setup-fixture-app
with:
platform: android
install: 'false'
require-artifact: 'true'
- name: Mark Android emulator setup complete
id: android-setup
run: echo "seconds=$(date +%s)" >> "$GITHUB_OUTPUT"
- name: Run Android full emulator suite
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0
with:
api-level: 36
arch: x86_64
profile: pixel_7
target: google_apis_playstore
emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -no-metrics
script: |
# android-emulator-runner executes this with /usr/bin/sh, which is dash on
# ubuntu runners and rejects `-o pipefail`. The serial assignment below is
# validated by the `test -n` guard instead.
set -eu
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
test -n "$ANDROID_SERIAL"
BOOT_FINISHED_AT="$(date +%s)"
pnpm gate build
pnpm clean:daemon
adb -s "$ANDROID_SERIAL" install -r "${{ steps.fixture-app.outputs.apk-path }}"
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag-android.ad --platform android --serial "$ANDROID_SERIAL" --session nightly-android-target-drag --json
pnpm clean:daemon
FULL_STARTED_AT="$(date +%s)"
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
FULL_FINISHED_AT="$(date +%s)"
echo "Android setup before emulator: $((BOOT_FINISHED_AT - ${{ steps.android-setup.outputs.seconds }}))s"
echo "Android full scenario wall time: $((FULL_FINISHED_AT - FULL_STARTED_AT))s"
pnpm gate replay-android --retries 2 --artifacts-dir test/artifacts/replays-android --report-junit test/artifacts/replays-android.junit.xml
- name: Upload Android artifacts
if: always()
uses: ./.github/actions/upload-agent-device-artifacts
with:
artifact-name: replay-nightly-android-artifacts
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
manual-ios:
name: iOS Replay Suite
runs-on: macos-26
timeout-minutes: 80
env:
IOS_RUNTIME_VERSION: '26.2'
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/agent-device-state
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS: '420000'
AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE: agent-device-test-app:///automation?event={event}&payload={payload}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
with:
cache-dependency-path: |
pnpm-lock.yaml
examples/test-app/pnpm-lock.yaml
- name: Restore and build iOS XCTest runner
uses: ./.github/actions/setup-apple-runner-build
with:
derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
cache-key-prefix: ios-runner-prebuilt
cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}
gate: swift-runner-ios
xcuitest-platform: ios
xcuitest-destination: generic/platform=iOS Simulator
- name: Boot iOS test simulator
id: ios-simulator
uses: ./.github/actions/boot-ios-test-simulator
with:
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
preferred-device-name: iPhone 17 Pro
- name: Preflight iOS runner through public CLI
run: |
pnpm clean:daemon
node --experimental-strip-types src/bin.ts prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json
pnpm clean:daemon
- name: Run iOS simulator replay suite
uses: ./.github/actions/run-gate
with:
gate: replay-ios
args: |
--udid
${{ steps.ios-simulator.outputs.simulator-udid }}
--retries
2
--artifacts-dir
test/artifacts/replays-ios-simulator
--report-junit
test/artifacts/replays-ios-simulator.junit.xml
- name: Fetch current fixture app
id: fixture-app
uses: ./.github/actions/setup-fixture-app
with:
install: 'false'
- name: Report fixture cache source
run: |
echo "Fixture app source: ${{ steps.fixture-app.outputs.source }}" >> "$GITHUB_STEP_SUMMARY"
- name: Prove selector drag reaches its destination on iOS
run: |
xcrun simctl install "${{ steps.ios-simulator.outputs.simulator-udid }}" "${{ steps.fixture-app.outputs.app-path }}"
pnpm clean:daemon
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag.ad --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --session nightly-ios-target-drag --json
pnpm clean:daemon
- name: Run full fixture-backed iOS simulator E2E
env:
AGENT_DEVICE_FIXTURE_APP_ID: ${{ steps.fixture-app.outputs.app-id }}
AGENT_DEVICE_FIXTURE_APP_PATH: ${{ steps.fixture-app.outputs.app-path }}
AGENT_DEVICE_IOS_E2E: '1'
AGENT_DEVICE_IOS_E2E_TIER: full
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
run: |
pnpm gate build
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
- name: Run iOS physical device replay suite
if: env.IOS_UDID != ''
env:
IOS_UDID: ${{ vars.IOS_UDID }}
uses: ./.github/actions/run-gate
with:
gate: replay-ios-device
args: |
--udid
${{ env.IOS_UDID }}
--retries
2
--artifacts-dir
test/artifacts/replays-ios-device
--report-junit
test/artifacts/replays-ios-device.junit.xml
- name: Upload iOS artifacts
if: always()
uses: ./.github/actions/upload-agent-device-artifacts
with:
artifact-name: replay-nightly-ios-artifacts
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
runner-derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
Reference in New Issue View Git Blame Copy Permalink