callstack__agent-device/.github/workflows/xctest-nightly.yml

name: XCTest Nightly

# The full iOS runner XCTest suite (#1781 A7). The PR lane (ios.yml) names a hand-written
# subset of the target's methods in an `-only-testing:` list; everything outside that list ran
# nowhere at all. This lane drops the filter and runs the test plan whole, skipping only
# `testCommand` — which is the runner's server entry point rather than a test (see the step
# below).
#
# No count is quoted here on purpose. `pnpm check:xctest-selection` prints the live split
# (declared / PR-selected / skipped / nightly-only) and is the only place those numbers are
# derived rather than remembered. A hand-copied figure in a comment is the same class of
# defect this lane exists to close: between this file being written and first review, the
# suite gained three methods and the PR list gained five entries, so every number originally
# written here was already wrong.
#
# Under the lane rule in #1781:
#
#   - Catches: regressions in the simulator-only tests (guard `… && os(iOS)`: they launch
#     the host app, route through SpringBoard, or assert an iOS-only branch) that ios.yml's
#     PR list leaves out. Since the A7 classification the pure-decision tests also run on
#     the macOS host lane on every PR (ci.yml "Swift Runner Host XCTests"), so this lane's
#     unique reach is the simulator-only remainder — `pnpm check:xctest-selection` prints
#     both numbers. `testCommand` is skipped, and the tests guarded to macOS/tvOS only
#     cannot execute on this lane's iOS Simulator destination (the host lane runs them).
#   - Evidence: the first execution found #1812 in the dark set (2 of 155 red, both real);
#     the other ~114 never-run tests all passed.
#   - Cost: this is a public repository, so nothing here is billed. The real cost is one macOS
#     concurrency slot per night. The Apple runner build shares ios.yml's cache (identical
#     inputs and build variant), but scheduled runs only see DEFAULT-BRANCH caches, so the hit
#     depends on ios.yml's `push: main` run having populated the cache at the same source
#     hash — on a miss this lane also pays a full runner build. Adds nothing to any PR.
#   - Kill criterion: what ONLY this lane reaches is 11 tests / ~42 s of simulator time
#     (measured from the first night's xcresult) — the app-launching, SpringBoard-probing
#     class. Once ios.yml runs the whole iOS-compiled bundle instead of its hand-written
#     list — the follow-up stacked on the classification — every test reaches a PR lane and
#     this lane goes. That trade is not free: those 11 are the flakiest tests in the suite,
#     and they would land on a blocking 15-minute gate that is already cancelled ~19% of the
#     time, so measure their flake rate here before moving them.
#
# Scheduled + manual only. It is deliberately NOT a PR gate: ios.yml is already over half of
# this repo's CI spend (#1781 A9), and moving the unrun set onto it would make that worse for
# tests whose flakiness on a shared runner is unknown.
#
# Note on `concurrency` below: it is keyed on `github.ref`, so a manual dispatch on the same
# ref cancels an in-flight scheduled run. That is the intended trade — an operator asking for
# a run wants the fresh one — but it means "dispatch while the nightly is running" loses the
# nightly's result rather than queueing behind it.

on:
  schedule:
    - cron: '30 4 * * *'
  workflow_dispatch:

permissions:
  contents: read
  actions: read

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  full-suite:
    name: iOS Runner Full XCTest Suite
    runs-on: macos-26
    # Measured, not guessed. A full local run of this exact command on matching toolchain
    # (Xcode 26.2, iOS 26.2 Simulator) executed 154 tests in 120s wall — the suite itself is
    # ~2 minutes, not the bulk of the job. What dominates the budget is the runner build on a
    # derived-data cache MISS, plus checkout and simulator boot. ios.yml carries 80 minutes
    # while doing strictly more than this lane (same build and boot, plus replays, prepare,
    # and the fixture E2E), so half of that is generous headroom for a cache-miss night and
    # still caps a wedged run at 45 minutes instead of two hours of held macOS slot.
    timeout-minutes: 45
    env:
      # Must match ios.yml. The runtime version rides in the derived-data cache key, and
      # AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS is BOTH an input to the cache's build-variant
      # hash and the `-D AGENT_DEVICE_RUNNER_UNIT_TESTS` compile flag that makes the RunnerTests
      # methods exist at all. Dropping it would build a different variant (cache miss) whose
      # bundle contains no tests, and `xcodebuild` reports running zero tests as success — the
      # "silent 0-test green" this lane would then be. The test-count assertion below is the
      # backstop for exactly that. AGENT_DEVICE_XCUITEST_ARCHS rides the cache key too: dropping
      # it silently reverts the runner to Xcode's x86_64 default under Rosetta.
      IOS_RUNTIME_VERSION: '26.2'
      AGENT_DEVICE_XCUITEST_ARCHS: 'arm64'
      AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS: '1'
      AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
      RESULT_BUNDLE_PATH: ${{ github.workspace }}/.tmp/xctest-nightly/RunnerTests.xcresult
      RESULT_SUMMARY_PATH: ${{ github.workspace }}/.tmp/xctest-nightly/test-results-summary.json
      XCTEST_LANE: nightly
    steps:
      - name: Checkout
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

      - name: Setup toolchain
        uses: ./.github/actions/setup-node-pnpm
        with:
          cache-dependency-path: |
            pnpm-lock.yaml
            examples/test-app/pnpm-lock.yaml

      # Byte-identical inputs to ios.yml's step, so both lanes restore the same cache entry
      # and only the first one to run on a commit pays for the build.
      - name: Restore and build iOS XCTest runner
        uses: ./.github/actions/setup-apple-runner-build
        with:
          derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
          cache-key-prefix: ios-runner-prebuilt
          cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}-arm64
          gate: swift-runner-ios
          xcuitest-platform: ios
          xcuitest-destination: generic/platform=iOS Simulator

      - name: Boot iOS test simulator
        id: ios-simulator
        uses: ./.github/actions/boot-ios-test-simulator
        with:
          runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
          preferred-device-name: iPhone 17 Pro

      # Same command as ios.yml's targeted step minus every `-only-testing:` flag, so the
      # xctestrun's own test plan decides what runs — with one exception.
      #
      # `RunnerTests/testCommand` is not a test. It is the runner's server entry point: it
      # opens an NWListener and blocks in `XCTWaiter.wait(timeout: 24 * 60 * 60)` until a
      # client closes the session, which is exactly how production drives it
      # (packages/platform-apple/src/runner/runner-session.ts always passes it as the sole
      # `-only-testing:`). It compiles unconditionally — the `#if AGENT_DEVICE_RUNNER_UNIT_TESTS`
      # block ends well above it — so an unfiltered run reaches it in alphabetical order and
      # hangs the job until `timeout-minutes`. Its one escape hatch,
      # AGENT_DEVICE_RUNNER_NOOP_STARTUP, is an environment variable, and the comment above
      # the flag records that env plumbing into a simulator test process is not dependable;
      # `-skip-testing:` is the lever that works from here.
      #
      # A typo in that identifier silently re-arms the hang, so
      # `pnpm check:xctest-selection` validates `-skip-testing:` exactly like `-only-testing:`.
      - name: Run the full iOS runner XCTest suite
        run: |
          set -euo pipefail
          XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)"
          test -n "$XCTESTRUN_PATH"
          mkdir -p "$(dirname "$RESULT_BUNDLE_PATH")"
          xcodebuild test-without-building \
            -xctestrun "$XCTESTRUN_PATH" \
            -destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \
            -skip-testing:AgentDeviceRunnerUITests/RunnerTests/testCommand \
            -resultBundlePath "$RESULT_BUNDLE_PATH"

      # Best-effort and never the job's verdict on its own; the step below is what asserts.
      # `--compact` first because a red night's summary is the large one, and the job summary
      # has a 1 MiB cap — the plain form is the fallback if an Xcode version rejects the flag.
      - name: Extract the test-results summary
        if: always()
        run: |
          set -uo pipefail
          [ -d "$RESULT_BUNDLE_PATH" ] || exit 0
          xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE_PATH" --compact \
            > "$RESULT_SUMMARY_PATH" 2>/dev/null ||
            xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE_PATH" \
              > "$RESULT_SUMMARY_PATH" 2>/dev/null ||
            rm -f "$RESULT_SUMMARY_PATH"
          exit 0

      # Reports the run AND is the lane's own liveness check. `xcodebuild` exits 0 when a
      # selection matches nothing, so "green" and "ran no tests" are the same log tail: a
      # build variant without the unit-test compile flag, an empty test plan, or a renamed
      # target would all read as a healthy night. The reporter asserts the executed count
      # equals what `pnpm check:xctest-selection` derives this lane reaches from the `#if`
      # guards (XCTEST_LANE above), so a guard that compiles a file of tests out of iOS reads
      # as red too. The failure list is capped so a badly red night cannot blow the 1 MiB
      # job-summary limit and lose the headline with it.
      - name: Report the run and assert it executed the source-derived set
        if: always()
        run: |
          set -euo pipefail
          if [ ! -s "$RESULT_SUMMARY_PATH" ]; then
            echo 'No usable test-results summary was produced, so nothing here can show a test ran.'
            echo 'The suite step above failed before or during the run; read its log first.'
            echo '### iOS runner full XCTest suite' >> "$GITHUB_STEP_SUMMARY"
            echo 'No result-bundle summary; see the run log and the uploaded bundle.' >> "$GITHUB_STEP_SUMMARY"
            exit 1
          fi
          node --experimental-strip-types scripts/xctest-run-summary.ts

      - name: Upload result bundle
        if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: xctest-nightly-results-${{ github.run_id }}-${{ github.run_attempt }}
          include-hidden-files: true
          path: .tmp/xctest-nightly
          if-no-files-found: warn