Files
callstack__agent-device/.github/workflows/mutation-affected.yml
Michał Pierzchała 423927fdd8 chore(mutation): shrink to report-only — drop the ratchet, baseline and graduation (#1457, #1781) (#1828)
* chore(mutation): shrink the lane to report-only (#1457, #1781 wave 2)

The mutation harness's two real catches (#1474, #1475) both came from humans
reading the weekly score report. The ratchet half never operated: the baseline
was committed exactly twice (8cce0ef6b, 60400d04b), both times with
`stableRuns: 0, gating: false`, and was never updated after the very fixes it
triggered — the weekly job computed a new baseline and then `git checkout --`d
it, uploading a proposal nobody applied in 3+ weeks. A gate nobody arms is
harness weight; the report is the part that paid.

Deletes ratchet.ts + ratchet.test.ts, mutation-baselines/, and every
baseline/graduation/gating path in run.ts (`--update`, `mutation:baseline`).
run.ts now exits non-zero only on a harness failure, never on a score. The
report renders the per-kernel table (kernel, score, killed, survived, total,
timeouts) plus the surviving mutants a strengthening PR works from.

Kernel scoping stays: stryker.config.json and KERNEL_MODULES are untouched.

* fix(mutation): restore denominator coverage and publish the table before judging the shard set

Review of #1828:
- `report.test.ts` re-asserts that Ignored/CompileError/RuntimeError leave the
  denominator — the one behaviour `ratchet.test.ts` covered and nothing replaced.
  A `tally()` edit that counted tool noise would have deflated every published
  score with a green `mutation:test`.
- `assertShardsCoverModules` now runs after `emit()`, so an incomplete shard set
  still publishes the kernels that completed instead of only an error string.
  This makes the workflow comments' claim about the job summary true rather than
  re-wording them down.

* chore(mutation): trigger the affected lane on exactly the paths that can select mutants

The PR lane returns an empty matrix unless the diff touches the harness, so the
kernel-source and `**/*.test.ts` triggers only bought a 1-4 min no-op job on
~96% of PRs. `on.pull_request.paths` is now exactly `LANE_TOOLING` plus the
workflow file, asserted in both directions by workflow.test.ts against the
exported constant — a missing path would let a harness change merge unproven,
an extra one starts a job that can only answer `[]`.

Also drops the workflow header's contradictory scope paragraph: it claimed the
lane selects on kernel sources and any test reaching one, which has not been
true since the ratchet went.

* fix(mutation): score and publish a short shard set before failing on the count

The expected-count check ran inside readShardedReports, before anything was
summarized, so on the weekly's real `--expect-shards 10` one dead shard threw
away the nine that had reported — the earlier reorder only moved the
zero-mutants check. The merge now returns the shard count, and both verdicts
run after emit() with the same exit code and `score` stage.

Regression uses the weekly argument shape (`--expect-shards 10`, one shard
present) and asserts the reporting kernel's row reaches stdout while the run
still fails.
2026-08-18 17:47:29 +02:00

182 lines
6.4 KiB
YAML

name: Mutation Affected
# PR-side half of the decision-kernel mutation lane (issue #1415).
#
# Advisory only — the lane reports and never gates (#1457). `scripts/mutation/run.ts`
# exits non-zero on a harness failure, never on a score.
#
# Cost control: the kernel report that pays is the weekly sweep, so this lane
# spends mutants on one thing only — proving the harness still runs end to end
# when the harness itself changes. `select` therefore returns an empty matrix
# unless the diff touches the lane's own sources. Selecting on derived kernel
# ownership instead would run the full ten-shard sweep on 24 of the last 40
# merged PRs.
#
# On a harness diff the matrix is the canary (`LANE_CANARY`, the registry's
# cheapest real sweep — lane sources own no kernel, so derivation alone would
# select nothing) plus any kernel that same diff reaches through the import
# graph (scripts/mutation/ownership.ts): a kernel source, or a test that reaches
# one. Selected modules are sharded like the weekly sweep, so the PR's wall
# clock is one module rather than their sum.
on:
pull_request:
paths:
# Exactly `LANE_TOOLING` in scripts/mutation/run.ts plus this file — the
# only diffs that can produce a non-empty matrix, asserted both ways by
# scripts/mutation/workflow.test.ts. A kernel or test path here would only
# start a job that selects nothing.
- 'scripts/mutation/**'
- 'scripts/lib/**'
- 'stryker.config.json'
- '.github/workflows/mutation-affected.yml'
permissions:
contents: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
select:
name: Select affected kernels
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
modules: ${{ steps.select.outputs.result }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Harness self-test
uses: ./.github/actions/run-gate
with: { gate: mutation-model }
- id: select
name: Derive the affected shard matrix
uses: ./.github/actions/run-gate
with:
gate: mutation-affected
args: |
--list-affected
--base
origin/${{ github.event.pull_request.base.ref }}
# The self-test and the derivation run before any mutant, so a failure here
# would leave the lane with no envelope at all (#1430).
- name: Record a failed lane envelope
if: failure()
run: |
pnpm gate mutation --affected --fail-envelope \
"affected selection failed before any mutant ran (run ${{ github.run_id }})" || true
- name: Upload selection envelope
if: failure()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-affected-select
path: .tmp/mutation/lane-envelope.json
if-no-files-found: warn
mutants:
name: Mutants (${{ matrix.name }})
needs: select
if: needs.select.outputs.modules != '[]'
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.select.outputs.modules) }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Run mutants for ${{ matrix.name }}
uses: ./.github/actions/run-gate
with:
gate: mutation
args: |
--modules
${{ matrix.module }}
${{ matrix.shard && '--shard' || '' }}
${{ matrix.shard || '' }}
- name: Record a failed shard envelope
if: failure()
run: |
pnpm gate mutation --affected --modules ${{ matrix.module }} --fail-envelope \
"shard ${{ matrix.name }} failed before producing a report (run ${{ github.run_id }})" || true
- name: Upload shard report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-affected-shard-${{ matrix.name }}
path: |
.tmp/mutation/mutation.json
.tmp/mutation/mutation.html
.tmp/mutation/lane-envelope.json
if-no-files-found: warn
report:
name: Affected decision-kernel mutants
needs: [select, mutants]
if: always() && needs.select.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Download shard reports
if: needs.select.outputs.modules != '[]'
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
pattern: mutation-affected-shard-*
path: .tmp/mutation/shards
# No shards means no affected kernel: run.ts reports "nothing to mutate"
# and still writes the envelope, so the lane is never silently absent.
# Either invocation writes the score table to $GITHUB_STEP_SUMMARY.
- name: Score the affected modules
run: |
if [ -d .tmp/mutation/shards ]; then
expected=$(echo '${{ needs.select.outputs.modules }}' | jq length)
pnpm gate mutation-check --report-dir .tmp/mutation/shards --affected \
--expect-shards "$expected" \
--base "origin/${{ github.event.pull_request.base.ref }}"
else
pnpm gate mutation-affected --base "origin/${{ github.event.pull_request.base.ref }}"
fi
- name: Record a failed lane envelope
if: failure()
run: |
pnpm gate mutation --affected --fail-envelope \
"affected report failed before producing a score table (run ${{ github.run_id }})" || true
- name: Upload mutation report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-affected
path: |
.tmp/mutation/shards
.tmp/mutation/lane-envelope.json
if-no-files-found: ignore