mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
9c22467832
* test(ci): prove every registered gate is owned and reachable (#1429) A check that silently stops running looks exactly like a green build. Two suites had already stopped: `check:tmpdir-leaks` (with its model tests) and `test:fixture-cache` are real package scripts that no workflow ran, reachable only through the `check:unit` aggregate CI never invokes. `CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the only way CI runs one, so finding what a lane runs is a scan for `pnpm gate` rather than an attempt to interpret shell. `pnpm check:gate-manifest` then asserts against the real workflows that every registered check is run by some qualifying lane (per unit, not per script name), that every check the real selector activates for a path is run by a lane that path would start (#1420's class), and that every Vitest project and suite script belongs to a check. The wiring that keeps those honest is asserted too: a gate id must name a registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false` unowns what it guards, an action declared to run a gate is proven to, and a job whose steps the loader cannot open fails closed. It deliberately does not try to prove CI runs project code only through `pnpm gate`. Whether a shell block executes project code is not decidable from its text, so shell this model does not recognise earns no ownership credit — the failure direction is a check reported unowned, never one waved through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * test(ci): update the two suites that assert on rewired workflow text `scripts/mutation/workflow.test.ts` and `test/ci/trusted-fixture-artifact.test.mjs` read the workflow and action files and assert on their command text, so routing those steps through `pnpm gate <id>` moved what they were matching. They are the two suites the manifest cannot help with: it proves a gate is still run, not that a test asserting on how CI spells a command was updated with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(ci): credit gates by execution shape, and keep every guard Three ways the manifest could report a gate as owned when it does not run. 1. Crediting was a substring scan over `run:`, which #1429 explicitly rules out — "do not infer reachability from a command name merely appearing in workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`, one named in a heredoc, and `echo pnpm gate x` all credited it. There is a live instance: conformance-regenerate.yml's "Fail if regeneration changed anything" step names `pnpm gate maestro-regenerate` inside an error message telling a human to run it, and that credited the gate. A gate now counts only as the first command segment of a line, and a body carrying shell structure earns nothing. Reachability inside a script is not decidable, so this does not try: unrecognised shape means no credit and the check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment form is unambiguous and the gate runs. 2. Job-level `if:` was not modelled at all, though six live jobs carry one, so a job that cannot run still credited every gate inside it. Two conditions on the mutation lanes are now declared. 3. A caller's `if:` REPLACED the guard on a nested composite-action step (`guard[0] ?? step.condition`), so an outer `always()` erased an inner `if: false`. Steps carry every guard between the lane and the step. Also corrects two source comments that still claimed project code run outside the runner fails the manifest. It does not: such a step earns no credit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * ci: add the run-gate action that names a gate structurally The seam the ownership proof will read instead of shell. A lane says which gate it runs in `with.gate`, a typed input the manifest reads straight out of the YAML and validates against CHECK_CATALOG. Nothing here is wired yet — the ~60 call sites and the model change follow. Added first so the target of that conversion is reviewable on its own. `args` cannot select which gate runs; it is appended after the id, so the worst a wrong value does is fail the gate it already named. There is no `|| true` and no output capture: the gate's exit code is the step's exit code, so a gate cannot run without being able to fail its lane. Part of #1429. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * merge: main (#1770) and route its three new steps through the runner #1770 landed the orphan-check fix on main, wiring `check:tmpdir-leaks`, `check:tmpdir-leaks:test` and `test:fixture-cache` into Coverage, Layering Guard and Integration Tests. This branch had wired the same three through `pnpm gate`, so the merge produced two steps per check rather than a conflict — each check ran twice. Kept main's steps, with the placement and reasoning reviewed on #1770, and changed only their `run:` line to the canonical runner. Dropped this branch's duplicates. Net effect on CI is unchanged: the same three checks, in the same three lanes, once each. Gate manifest green after the merge: 47 checks wired across 33 lanes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(ci): address review — suite detection, freerange, glob, vacuous skip-list Six review findings plus the mutation blocker. [bug] `registered` was shape-only, so a `test:*` script running `node src/bin.ts test <dir>` resolved to a `script:` leaf and was invisible. Four `test:replay:*` scripts were owned only because someone hand-registered them; `test:replay:android` was neither registered nor reported while the nightly ran the same six .ad files by inlining them. A `test:*` script is now a suite by name. `replay-android` is registered, and the nightly runs the script instead of re-listing its files so the two cannot drift. The nightly invokes it inside `reactivecircus/android-emulator-runner`'s `script:` input — shell handed to a third-party action this loader does not read — so the suite executes but cannot be credited. Recorded in UNPROVABLE_OWNERS with that exact reason rather than assumed. The fixed detector also found a second orphan the review did not name: `test:integration:progress`. That one is a reporter whose `--check` sibling is the registered gate, so it is declared in REPORTING_SCRIPTS — a declaration that itself fails when inert. [bug] `freerange` defaulted to localRunnable, so fail-open ran `fr` (a Bun binary) on the pre-push path. Now false. [suggestion] The `--run` skip-list asserted `build:android-snapshot-helper`, a name `android-helpers` no longer uses, so it could not fail. Derived from the catalog instead. [suggestion] `matchesGlob` joined `**` splits with `.*`, making the adjacent slash mandatory — GitHub's `**` matches zero directories, so `src/**/*.test.ts` did not match `src/a.test.ts`. Pinned against `packages/*/src/**/*.test.ts`. [suggestion] Deleted the unwired `run-gate` action. It had no callers, was absent from GATE_ACTIONS, and its comment described a system that had not shipped. It returns with the rewiring, not before. [suggestion] Collapsed the module headers that narrated discarded designs. Mutation: `daemon entrypoint publishes HTTP metadata and cleans up on shutdown` is the only test here that spawns a real daemon process. It takes ~1.1s alone but exceeds Vitest's 5s default inside Stryker's dry run, which aborts the sweep before a single mutant runs. Given 30s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(mutation): order sandbox aliases longest-first so subpaths resolve Every shard of the mutation sweep aborted in Stryker's dry run with: Cannot find package '@agent-device/selectors/engine' imported from .tmp/stryker/sandbox-*/src/core/selector-pipeline.ts The alias was generated correctly; it just never won. Vite matches a STRING alias by prefix and takes the first hit, and `workspaceSpecifierTargets` emitted the bare `@agent-device/selectors` ahead of the subpath entries. The bare entry therefore captured `@agent-device/selectors/engine` and rewrote it to `…/src/index.ts/engine`, which does not exist; Node fell back to real package resolution, could not find the subpath inside the sandbox, and the dry run failed before a single mutant ran — so the shard uploaded an empty envelope instead of a report and the ratchet failed for want of one. Sorting longest specifier first makes the most specific alias win: @agent-device/selectors/engine -> packages/selectors/src/engine.ts @agent-device/selectors/ast -> packages/selectors/src/ast.ts @agent-device/selectors -> packages/selectors/src/index.ts `/ast` never tripped this because nothing in a related test set imported it; `selector-pipeline.ts` introduced the first subpath import that mattered (#1744), so the mutation lane has been unable to run since that landed. Any PR touching `scripts/mutation/**` — which fails open into the full sweep — would have hit it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * refactor: derive gate ownership from workflow structure * fix: run gates without optional arguments * fix: resolve mutation workspace subpaths exactly --------- Co-authored-by: Claude <noreply@anthropic.com>
191 lines
6.8 KiB
YAML
191 lines
6.8 KiB
YAML
name: Mutation Affected
|
|
|
|
# PR-side half of the decision-kernel mutation lane (issue #1415).
|
|
#
|
|
# Graduation, not a flag day: this job runs on every PR that touches a kernel
|
|
# module, but `scripts/mutation/run.ts` only exits non-zero once the committed
|
|
# baseline reports `gating: true` — earned by two consecutive stable weekly
|
|
# sweeps (mutation-weekly.yml). Until then it is a report with the same numbers,
|
|
# so the gate's first failing day is not also its first running day.
|
|
#
|
|
# Cost control: before graduation an affected run is a report nobody acts on, so
|
|
# `select` returns an empty matrix and no mutants run — except when the diff
|
|
# touches the lane's own tooling, the one case where a pre-graduation run buys
|
|
# something (the gate has to be proven before it can bite).
|
|
#
|
|
# Scope is the AFFECTED modules only, and affectedness is DERIVED from the import
|
|
# graph (scripts/mutation/ownership.ts): a kernel source, or any test that reaches
|
|
# one. Reaching a kernel is a superset of killing its mutants, so the `select` job
|
|
# frequently returns several modules — sharded like the weekly sweep so the PR's
|
|
# wall clock is one module, not their sum. The weekly run stays the full sweep.
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
# Kernel sources, every src/package test (ownership is derived, so any
|
|
# test may own a kernel — `select` decides, not this filter), and the
|
|
# lane's own tooling. scripts/mutation/workflow.test.ts asserts this
|
|
# covers the registry.
|
|
- 'packages/kernel/src/errors.ts'
|
|
- 'src/daemon/ref-frame.ts'
|
|
- 'src/commands/interaction/runtime/settle.ts'
|
|
- 'src/utils/scroll-edge-state.ts'
|
|
- 'packages/selectors/src/**'
|
|
- 'packages/ad-script/src/internal/target-annotation-serde.ts'
|
|
- 'src/snapshot/snapshot-occlusion.ts'
|
|
- 'src/**/*.test.ts'
|
|
- 'packages/*/src/**/*.test.ts'
|
|
- 'scripts/mutation/**'
|
|
- 'scripts/lib/**'
|
|
- 'stryker.config.json'
|
|
- 'mutation-baselines/**'
|
|
- '.github/workflows/mutation-affected.yml'
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ci-${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
select:
|
|
name: Select affected kernels
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
outputs:
|
|
modules: ${{ steps.select.outputs.result }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
- name: Setup toolchain
|
|
uses: ./.github/actions/setup-node-pnpm
|
|
|
|
- name: Ratchet self-test
|
|
uses: ./.github/actions/run-gate
|
|
with: { gate: mutation-model }
|
|
|
|
- id: select
|
|
name: Derive the affected shard matrix
|
|
uses: ./.github/actions/run-gate
|
|
with:
|
|
gate: mutation-affected
|
|
args: |
|
|
--list-affected
|
|
--base
|
|
origin/${{ github.event.pull_request.base.ref }}
|
|
|
|
# The self-test and the derivation run before any mutant, so a failure here
|
|
# would leave the lane with no envelope at all (#1430).
|
|
- name: Record a failed lane envelope
|
|
if: failure()
|
|
run: |
|
|
pnpm gate mutation --affected --fail-envelope \
|
|
"affected selection failed before any mutant ran (run ${{ github.run_id }})" || true
|
|
|
|
- name: Upload selection envelope
|
|
if: failure()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
|
|
with:
|
|
name: mutation-affected-select
|
|
path: .tmp/mutation/lane-envelope.json
|
|
if-no-files-found: warn
|
|
|
|
mutants:
|
|
name: Mutants (${{ matrix.name }})
|
|
needs: select
|
|
if: needs.select.outputs.modules != '[]'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include: ${{ fromJSON(needs.select.outputs.modules) }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Setup toolchain
|
|
uses: ./.github/actions/setup-node-pnpm
|
|
|
|
- name: Run mutants for ${{ matrix.name }}
|
|
uses: ./.github/actions/run-gate
|
|
with:
|
|
gate: mutation
|
|
args: |
|
|
--modules
|
|
${{ matrix.module }}
|
|
${{ matrix.shard && '--shard' || '' }}
|
|
${{ matrix.shard || '' }}
|
|
|
|
- name: Record a failed shard envelope
|
|
if: failure()
|
|
run: |
|
|
pnpm gate mutation --affected --modules ${{ matrix.module }} --fail-envelope \
|
|
"shard ${{ matrix.name }} failed before producing a report (run ${{ github.run_id }})" || true
|
|
|
|
- name: Upload shard report
|
|
if: always()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
|
|
with:
|
|
name: mutation-affected-shard-${{ matrix.name }}
|
|
path: |
|
|
.tmp/mutation/mutation.json
|
|
.tmp/mutation/mutation.html
|
|
.tmp/mutation/lane-envelope.json
|
|
if-no-files-found: warn
|
|
|
|
ratchet:
|
|
name: Affected decision-kernel mutants
|
|
needs: [select, mutants]
|
|
if: always() && needs.select.result == 'success'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
- name: Setup toolchain
|
|
uses: ./.github/actions/setup-node-pnpm
|
|
|
|
- name: Download shard reports
|
|
if: needs.select.outputs.modules != '[]'
|
|
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
|
|
with:
|
|
pattern: mutation-affected-shard-*
|
|
path: .tmp/mutation/shards
|
|
|
|
# No shards means no affected kernel: run.ts reports "nothing to mutate"
|
|
# and still writes the envelope, so the lane is never silently absent.
|
|
- name: Ratchet the affected modules
|
|
run: |
|
|
if [ -d .tmp/mutation/shards ]; then
|
|
expected=$(echo '${{ needs.select.outputs.modules }}' | jq length)
|
|
pnpm gate mutation-check --report-dir .tmp/mutation/shards --affected \
|
|
--expect-shards "$expected" \
|
|
--base "origin/${{ github.event.pull_request.base.ref }}"
|
|
else
|
|
pnpm gate mutation-affected --base "origin/${{ github.event.pull_request.base.ref }}"
|
|
fi
|
|
|
|
- name: Record a failed lane envelope
|
|
if: failure()
|
|
run: |
|
|
pnpm gate mutation --affected --fail-envelope \
|
|
"affected ratchet failed before producing a verdict (run ${{ github.run_id }})" || true
|
|
|
|
- name: Upload mutation report
|
|
if: always()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
|
|
with:
|
|
name: mutation-affected
|
|
path: |
|
|
.tmp/mutation/shards
|
|
.tmp/mutation/lane-envelope.json
|
|
if-no-files-found: ignore
|