5 Commits

Author	SHA1	Message	Date
Michał Pierzchała	423927fdd8	chore(mutation): shrink to report-only — drop the ratchet, baseline and graduation (#1457, #1781) (#1828) ... * chore(mutation): shrink the lane to report-only (#1457, #1781 wave 2) The mutation harness's two real catches (#1474, #1475) both came from humans reading the weekly score report. The ratchet half never operated: the baseline was committed exactly twice (8cce0ef6b, 60400d04b), both times with `stableRuns: 0, gating: false`, and was never updated after the very fixes it triggered — the weekly job computed a new baseline and then `git checkout --`d it, uploading a proposal nobody applied in 3+ weeks. A gate nobody arms is harness weight; the report is the part that paid. Deletes ratchet.ts + ratchet.test.ts, mutation-baselines/, and every baseline/graduation/gating path in run.ts (`--update`, `mutation:baseline`). run.ts now exits non-zero only on a harness failure, never on a score. The report renders the per-kernel table (kernel, score, killed, survived, total, timeouts) plus the surviving mutants a strengthening PR works from. Kernel scoping stays: stryker.config.json and KERNEL_MODULES are untouched. * fix(mutation): restore denominator coverage and publish the table before judging the shard set Review of #1828: - `report.test.ts` re-asserts that Ignored/CompileError/RuntimeError leave the denominator — the one behaviour `ratchet.test.ts` covered and nothing replaced. A `tally()` edit that counted tool noise would have deflated every published score with a green `mutation:test`. - `assertShardsCoverModules` now runs after `emit()`, so an incomplete shard set still publishes the kernels that completed instead of only an error string. This makes the workflow comments' claim about the job summary true rather than re-wording them down. * chore(mutation): trigger the affected lane on exactly the paths that can select mutants The PR lane returns an empty matrix unless the diff touches the harness, so the kernel-source and `**/*.test.ts` triggers only bought a 1-4 min no-op job on ~96% of PRs. `on.pull_request.paths` is now exactly `LANE_TOOLING` plus the workflow file, asserted in both directions by workflow.test.ts against the exported constant — a missing path would let a harness change merge unproven, an extra one starts a job that can only answer `[]`. Also drops the workflow header's contradictory scope paragraph: it claimed the lane selects on kernel sources and any test reaching one, which has not been true since the ratchet went. * fix(mutation): score and publish a short shard set before failing on the count The expected-count check ran inside readShardedReports, before anything was summarized, so on the weekly's real `--expect-shards 10` one dead shard threw away the nine that had reported — the earlier reorder only moved the zero-mutants check. The merge now returns the shard count, and both verdicts run after emit() with the same exit code and `score` stage. Regression uses the weekly argument shape (`--expect-shards 10`, one shard present) and asserts the reporting kernel's row reaches stdout while the run still fails.	2026-08-18 17:47:29 +02:00
Michał Pierzchała	9c22467832	refactor(ci): make gate ownership structural (#1429) (#1753) ... * test(ci): prove every registered gate is owned and reachable (#1429) A check that silently stops running looks exactly like a green build. Two suites had already stopped: `check:tmpdir-leaks` (with its model tests) and `test:fixture-cache` are real package scripts that no workflow ran, reachable only through the `check:unit` aggregate CI never invokes. `CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the only way CI runs one, so finding what a lane runs is a scan for `pnpm gate` rather than an attempt to interpret shell. `pnpm check:gate-manifest` then asserts against the real workflows that every registered check is run by some qualifying lane (per unit, not per script name), that every check the real selector activates for a path is run by a lane that path would start (#1420's class), and that every Vitest project and suite script belongs to a check. The wiring that keeps those honest is asserted too: a gate id must name a registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false` unowns what it guards, an action declared to run a gate is proven to, and a job whose steps the loader cannot open fails closed. It deliberately does not try to prove CI runs project code only through `pnpm gate`. Whether a shell block executes project code is not decidable from its text, so shell this model does not recognise earns no ownership credit — the failure direction is a check reported unowned, never one waved through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * test(ci): update the two suites that assert on rewired workflow text `scripts/mutation/workflow.test.ts` and `test/ci/trusted-fixture-artifact.test.mjs` read the workflow and action files and assert on their command text, so routing those steps through `pnpm gate <id>` moved what they were matching. They are the two suites the manifest cannot help with: it proves a gate is still run, not that a test asserting on how CI spells a command was updated with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(ci): credit gates by execution shape, and keep every guard Three ways the manifest could report a gate as owned when it does not run. 1. Crediting was a substring scan over `run:`, which #1429 explicitly rules out — "do not infer reachability from a command name merely appearing in workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`, one named in a heredoc, and `echo pnpm gate x` all credited it. There is a live instance: conformance-regenerate.yml's "Fail if regeneration changed anything" step names `pnpm gate maestro-regenerate` inside an error message telling a human to run it, and that credited the gate. A gate now counts only as the first command segment of a line, and a body carrying shell structure earns nothing. Reachability inside a script is not decidable, so this does not try: unrecognised shape means no credit and the check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment form is unambiguous and the gate runs. 2. Job-level `if:` was not modelled at all, though six live jobs carry one, so a job that cannot run still credited every gate inside it. Two conditions on the mutation lanes are now declared. 3. A caller's `if:` REPLACED the guard on a nested composite-action step (`guard[0] ?? step.condition`), so an outer `always()` erased an inner `if: false`. Steps carry every guard between the lane and the step. Also corrects two source comments that still claimed project code run outside the runner fails the manifest. It does not: such a step earns no credit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * ci: add the run-gate action that names a gate structurally The seam the ownership proof will read instead of shell. A lane says which gate it runs in `with.gate`, a typed input the manifest reads straight out of the YAML and validates against CHECK_CATALOG. Nothing here is wired yet — the ~60 call sites and the model change follow. Added first so the target of that conversion is reviewable on its own. `args` cannot select which gate runs; it is appended after the id, so the worst a wrong value does is fail the gate it already named. There is no `|| true` and no output capture: the gate's exit code is the step's exit code, so a gate cannot run without being able to fail its lane. Part of #1429. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * merge: main (#1770) and route its three new steps through the runner #1770 landed the orphan-check fix on main, wiring `check:tmpdir-leaks`, `check:tmpdir-leaks:test` and `test:fixture-cache` into Coverage, Layering Guard and Integration Tests. This branch had wired the same three through `pnpm gate`, so the merge produced two steps per check rather than a conflict — each check ran twice. Kept main's steps, with the placement and reasoning reviewed on #1770, and changed only their `run:` line to the canonical runner. Dropped this branch's duplicates. Net effect on CI is unchanged: the same three checks, in the same three lanes, once each. Gate manifest green after the merge: 47 checks wired across 33 lanes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(ci): address review — suite detection, freerange, glob, vacuous skip-list Six review findings plus the mutation blocker. [bug] `registered` was shape-only, so a `test:*` script running `node src/bin.ts test <dir>` resolved to a `script:` leaf and was invisible. Four `test:replay:*` scripts were owned only because someone hand-registered them; `test:replay:android` was neither registered nor reported while the nightly ran the same six .ad files by inlining them. A `test:*` script is now a suite by name. `replay-android` is registered, and the nightly runs the script instead of re-listing its files so the two cannot drift. The nightly invokes it inside `reactivecircus/android-emulator-runner`'s `script:` input — shell handed to a third-party action this loader does not read — so the suite executes but cannot be credited. Recorded in UNPROVABLE_OWNERS with that exact reason rather than assumed. The fixed detector also found a second orphan the review did not name: `test:integration:progress`. That one is a reporter whose `--check` sibling is the registered gate, so it is declared in REPORTING_SCRIPTS — a declaration that itself fails when inert. [bug] `freerange` defaulted to localRunnable, so fail-open ran `fr` (a Bun binary) on the pre-push path. Now false. [suggestion] The `--run` skip-list asserted `build:android-snapshot-helper`, a name `android-helpers` no longer uses, so it could not fail. Derived from the catalog instead. [suggestion] `matchesGlob` joined `**` splits with `.*`, making the adjacent slash mandatory — GitHub's `**` matches zero directories, so `src/**/*.test.ts` did not match `src/a.test.ts`. Pinned against `packages/*/src/**/*.test.ts`. [suggestion] Deleted the unwired `run-gate` action. It had no callers, was absent from GATE_ACTIONS, and its comment described a system that had not shipped. It returns with the rewiring, not before. [suggestion] Collapsed the module headers that narrated discarded designs. Mutation: `daemon entrypoint publishes HTTP metadata and cleans up on shutdown` is the only test here that spawns a real daemon process. It takes ~1.1s alone but exceeds Vitest's 5s default inside Stryker's dry run, which aborts the sweep before a single mutant runs. Given 30s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(mutation): order sandbox aliases longest-first so subpaths resolve Every shard of the mutation sweep aborted in Stryker's dry run with: Cannot find package '@agent-device/selectors/engine' imported from .tmp/stryker/sandbox-*/src/core/selector-pipeline.ts The alias was generated correctly; it just never won. Vite matches a STRING alias by prefix and takes the first hit, and `workspaceSpecifierTargets` emitted the bare `@agent-device/selectors` ahead of the subpath entries. The bare entry therefore captured `@agent-device/selectors/engine` and rewrote it to `…/src/index.ts/engine`, which does not exist; Node fell back to real package resolution, could not find the subpath inside the sandbox, and the dry run failed before a single mutant ran — so the shard uploaded an empty envelope instead of a report and the ratchet failed for want of one. Sorting longest specifier first makes the most specific alias win: @agent-device/selectors/engine -> packages/selectors/src/engine.ts @agent-device/selectors/ast -> packages/selectors/src/ast.ts @agent-device/selectors -> packages/selectors/src/index.ts `/ast` never tripped this because nothing in a related test set imported it; `selector-pipeline.ts` introduced the first subpath import that mattered (#1744), so the mutation lane has been unable to run since that landed. Any PR touching `scripts/mutation/**` — which fails open into the full sweep — would have hit it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * refactor: derive gate ownership from workflow structure * fix: run gates without optional arguments * fix: resolve mutation workspace subpaths exactly --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-12 16:02:18 +02:00
Michał Pierzchała	60400d04b7	feat(mutation): add target-annotation-serde + snapshot-occlusion kernels (#1553) ... * feat(mutation): add target-annotation-serde + snapshot-occlusion kernels Both are pure decision kernels the lane's own membership rule covers (target-annotation-serde: parse/validate/normalize the .ad comment-line codec, zero I/O; snapshot-occlusion: pure covered/not-covered decision where a wrong answer silently blocks or mis-allows a tap) but were excluded from KERNEL_MODULES. Fixing the harness's packages/*/src blind spot was required, not optional: test-scope.ts, ownership.ts, and vitest.mutation.config.ts all hardcoded `src/` as the only place a kernel's tests could live. target-annotation-serde's own tests live under packages/ad-script/src/internal/__tests__/, so without this fix the module would score 0% from day one — not from weak tests, but because its test file was silently invisible to the lane. Widened the same three places, plus mutation-affected.yml's path filter and isTestFile/ownedTestFiles in ownership.ts, to also recognize packages/*/src/**/*.test.ts (mirroring vitest.config.ts's own unit-core project include list). Triaged every surviving mutant from the initial run: real coverage gaps got a new/adjusted test (kill-with-test), everything else is documented equivalent with an inline comment at the mutation site explaining the invariant that makes it unobservable (redundant early-returns, JSON.stringify dropping undefined-valued keys, Number.isFinite/isSafeInteger's total-function safety, caller-enforced positiveRect/candidate invariants, etc). Baseline recorded from the actual measured run, not inherited or guessed: 94.03% (315/335) and 89.74% (175/195). * style: run the formatter over the four files the gate flagged	2026-08-02 11:36:43 +02:00
devin-ai-integration[bot]	edca35d122	chore(deps): Renovate config, packageManager-derived pnpm in CI, repo-wide format (#1444) ... * chore(deps): add Renovate config and enforce packageManager pnpm version in CI Refs #1422 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: bump pnpm to 11.17.0 and format the whole repo with oxfmt format/format:check drop their hand-maintained path list: oxfmt already skips node_modules and honors .gitignore, so the only exclusion list is .oxfmtrc.json ignorePatterns. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mutation): accept either quote style in the affected-lane path filter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(deps): keep fixture-app runtime deps as individual Renovate PRs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-28 13:50:39 +02:00
devin-ai-integration[bot]	8cce0ef6b8	test: ratchet mutation score over enumerated decision kernels (#1441) ... * test: ratchet mutation score over enumerated decision kernels Adds a Stryker (vitest runner) mutation lane scoped to the decision kernels, a per-module baseline with tool/config provenance, and a ratchet that only lets scores rise. Non-gating until two consecutive stable weekly sweeps. Refs #1415 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: declare the mutation test-scope seam for production-export analysis Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: own kernel tests in the mutation registry and ship the #1430 lane envelope - restore bench:help-conformance, broken by a formatting-path edit - kernel test files select their module on PRs (registry `tests` + workflow paths), asserted to reach the kernel through the import graph - every mutation run writes the standard scheduled-lane artifact envelope - move src/utils/__tests__/errors.test.ts beside its source per the mirror rule Refs #1415, #1430 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: derive kernel test ownership and land the scheduled-lane health monitor Ownership of a kernel's tests is now computed from the static import graph (scripts/mutation/ownership.ts) instead of a hand-listed set, so a test that reaches a kernel indirectly -- src/__tests__/daemon-error.test.ts through src/daemon.ts -- selects that kernel on a PR. The PR lane triggers on every src test and shards the derived modules, keeping wall clock at one module. The lane envelope (#1430) is now written on every exit path with the stage it reached, so a crash before any mutant runs is distinguishable from a lane that never ran. Adds the derived cadence monitor (scripts/lane-health, daily workflow): scheduled lanes are enumerated from .github/workflows/ and reported dark, failing, or never-run against their own cron cadence. * fix: merge only Stryker reports from a shard directory The shard artifacts now carry the lane envelope beside mutation.json, and the merge globbed every .json under the download path, so the ratchet job fed the envelope to the report parser and died after the mutants had already run. * fix(mutation): fail on incomplete shard sets and envelope pre-run failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mutation): downgrade a passing envelope when a later lane step fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mutation): shard by registry, defer the PR lane, drop the bundled watcher Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: describe registry sharding and the deferred PR mutation lane Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mutation): make the pre-graduation tooling exception select real mutants Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mutation): give the worktree fixture commits their own identity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-28 10:07:14 +02:00