Files
callstack__agent-device/scripts/gate/model.test.ts
Michał Pierzchała 9c22467832 refactor(ci): make gate ownership structural (#1429) (#1753)
* test(ci): prove every registered gate is owned and reachable (#1429)

A check that silently stops running looks exactly like a green build. Two
suites had already stopped: `check:tmpdir-leaks` (with its model tests) and
`test:fixture-cache` are real package scripts that no workflow ran, reachable
only through the `check:unit` aggregate CI never invokes.

`CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the
only way CI runs one, so finding what a lane runs is a scan for `pnpm gate`
rather than an attempt to interpret shell. `pnpm check:gate-manifest` then
asserts against the real workflows that every registered check is run by some
qualifying lane (per unit, not per script name), that every check the real
selector activates for a path is run by a lane that path would start (#1420's
class), and that every Vitest project and suite script belongs to a check.

The wiring that keeps those honest is asserted too: a gate id must name a
registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false`
unowns what it guards, an action declared to run a gate is proven to, and a
job whose steps the loader cannot open fails closed.

It deliberately does not try to prove CI runs project code only through
`pnpm gate`. Whether a shell block executes project code is not decidable from
its text, so shell this model does not recognise earns no ownership credit —
the failure direction is a check reported unowned, never one waved through.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* test(ci): update the two suites that assert on rewired workflow text

`scripts/mutation/workflow.test.ts` and `test/ci/trusted-fixture-artifact.test.mjs`
read the workflow and action files and assert on their command text, so routing
those steps through `pnpm gate <id>` moved what they were matching.

They are the two suites the manifest cannot help with: it proves a gate is still
run, not that a test asserting on how CI spells a command was updated with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* fix(ci): credit gates by execution shape, and keep every guard

Three ways the manifest could report a gate as owned when it does not run.

1. Crediting was a substring scan over `run:`, which #1429 explicitly rules
   out — "do not infer reachability from a command name merely appearing in
   workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`,
   one named in a heredoc, and `echo pnpm gate x` all credited it. There is a
   live instance: conformance-regenerate.yml's "Fail if regeneration changed
   anything" step names `pnpm gate maestro-regenerate` inside an error message
   telling a human to run it, and that credited the gate.

   A gate now counts only as the first command segment of a line, and a body
   carrying shell structure earns nothing. Reachability inside a script is not
   decidable, so this does not try: unrecognised shape means no credit and the
   check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment
   form is unambiguous and the gate runs.

2. Job-level `if:` was not modelled at all, though six live jobs carry one, so
   a job that cannot run still credited every gate inside it. Two conditions on
   the mutation lanes are now declared.

3. A caller's `if:` REPLACED the guard on a nested composite-action step
   (`guard[0] ?? step.condition`), so an outer `always()` erased an inner
   `if: false`. Steps carry every guard between the lane and the step.

Also corrects two source comments that still claimed project code run outside
the runner fails the manifest. It does not: such a step earns no credit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* ci: add the run-gate action that names a gate structurally

The seam the ownership proof will read instead of shell. A lane says which
gate it runs in `with.gate`, a typed input the manifest reads straight out of
the YAML and validates against CHECK_CATALOG.

Nothing here is wired yet — the ~60 call sites and the model change follow.
Added first so the target of that conversion is reviewable on its own.

`args` cannot select which gate runs; it is appended after the id, so the
worst a wrong value does is fail the gate it already named. There is no
`|| true` and no output capture: the gate's exit code is the step's exit code,
so a gate cannot run without being able to fail its lane.

Part of #1429.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* merge: main (#1770) and route its three new steps through the runner

#1770 landed the orphan-check fix on main, wiring `check:tmpdir-leaks`,
`check:tmpdir-leaks:test` and `test:fixture-cache` into Coverage, Layering
Guard and Integration Tests. This branch had wired the same three through
`pnpm gate`, so the merge produced two steps per check rather than a conflict
— each check ran twice.

Kept main's steps, with the placement and reasoning reviewed on #1770, and
changed only their `run:` line to the canonical runner. Dropped this branch's
duplicates. Net effect on CI is unchanged: the same three checks, in the same
three lanes, once each.

Gate manifest green after the merge: 47 checks wired across 33 lanes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* fix(ci): address review — suite detection, freerange, glob, vacuous skip-list

Six review findings plus the mutation blocker.

[bug] `registered` was shape-only, so a `test:*` script running
`node src/bin.ts test <dir>` resolved to a `script:` leaf and was invisible.
Four `test:replay:*` scripts were owned only because someone hand-registered
them; `test:replay:android` was neither registered nor reported while the
nightly ran the same six .ad files by inlining them. A `test:*` script is now
a suite by name. `replay-android` is registered, and the nightly runs the
script instead of re-listing its files so the two cannot drift.

  The nightly invokes it inside `reactivecircus/android-emulator-runner`'s
  `script:` input — shell handed to a third-party action this loader does not
  read — so the suite executes but cannot be credited. Recorded in
  UNPROVABLE_OWNERS with that exact reason rather than assumed.

  The fixed detector also found a second orphan the review did not name:
  `test:integration:progress`. That one is a reporter whose `--check` sibling
  is the registered gate, so it is declared in REPORTING_SCRIPTS — a
  declaration that itself fails when inert.

[bug] `freerange` defaulted to localRunnable, so fail-open ran `fr` (a Bun
binary) on the pre-push path. Now false.

[suggestion] The `--run` skip-list asserted `build:android-snapshot-helper`,
a name `android-helpers` no longer uses, so it could not fail. Derived from
the catalog instead.

[suggestion] `matchesGlob` joined `**` splits with `.*`, making the adjacent
slash mandatory — GitHub's `**` matches zero directories, so
`src/**/*.test.ts` did not match `src/a.test.ts`. Pinned against
`packages/*/src/**/*.test.ts`.

[suggestion] Deleted the unwired `run-gate` action. It had no callers, was
absent from GATE_ACTIONS, and its comment described a system that had not
shipped. It returns with the rewiring, not before.

[suggestion] Collapsed the module headers that narrated discarded designs.

Mutation: `daemon entrypoint publishes HTTP metadata and cleans up on
shutdown` is the only test here that spawns a real daemon process. It takes
~1.1s alone but exceeds Vitest's 5s default inside Stryker's dry run, which
aborts the sweep before a single mutant runs. Given 30s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* fix(mutation): order sandbox aliases longest-first so subpaths resolve

Every shard of the mutation sweep aborted in Stryker's dry run with:

  Cannot find package '@agent-device/selectors/engine' imported from
    .tmp/stryker/sandbox-*/src/core/selector-pipeline.ts

The alias was generated correctly; it just never won. Vite matches a STRING
alias by prefix and takes the first hit, and `workspaceSpecifierTargets`
emitted the bare `@agent-device/selectors` ahead of the subpath entries. The
bare entry therefore captured `@agent-device/selectors/engine` and rewrote it
to `…/src/index.ts/engine`, which does not exist; Node fell back to real
package resolution, could not find the subpath inside the sandbox, and the dry
run failed before a single mutant ran — so the shard uploaded an empty
envelope instead of a report and the ratchet failed for want of one.

Sorting longest specifier first makes the most specific alias win:

  @agent-device/selectors/engine -> packages/selectors/src/engine.ts
  @agent-device/selectors/ast    -> packages/selectors/src/ast.ts
  @agent-device/selectors        -> packages/selectors/src/index.ts

`/ast` never tripped this because nothing in a related test set imported it;
`selector-pipeline.ts` introduced the first subpath import that mattered
(#1744), so the mutation lane has been unable to run since that landed. Any
PR touching `scripts/mutation/**` — which fails open into the full sweep —
would have hit it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* refactor: derive gate ownership from workflow structure

* fix: run gates without optional arguments

* fix: resolve mutation workspace subpaths exactly

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-12 16:02:18 +02:00

88 lines
2.8 KiB
TypeScript

// Unit-resolution tests, kept to the cases where a mistake would OVER-credit a lane.
//
// Under-credit corrects itself: the real tree is audited on every PR, so a command
// the model fails to read shows up as an unowned check within one run. Over-credit
// is the dangerous direction — it reads as coverage that is not there — so those are
// the shapes pinned here, plus the two real-tree facts the unit vocabulary exists for.
import { execFileSync } from 'node:child_process';
import assert from 'node:assert/strict';
import path from 'node:path';
import test from 'node:test';
import { loadModel, scriptUnits, unitCovers } from './model.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
const tracked = execFileSync('git', ['ls-files'], {
cwd: repoRoot,
encoding: 'utf8',
})
.split('\n')
.filter(Boolean);
const model = loadModel(repoRoot, tracked);
const scriptModel = (scripts: Record<string, string>) => ({
scripts,
vitestProjects: ['unit-core', 'subprocess-stub'],
opaque: {},
});
test('an env prefix does not hide the command behind it', () => {
assert.deepEqual(
scriptUnits(
'build:x',
scriptModel({
'build:x': 'AGENT_DEVICE_XCUITEST_PLATFORM=ios sh ./scripts/build.sh',
}),
),
['script:build:x'],
);
});
test('a filtered Vitest run does not credit the whole project', () => {
const units = scriptUnits(
'docs',
scriptModel({
docs: 'vitest run --project unit-core src/__tests__/command-doc-coverage.test.ts',
}),
);
assert.deepEqual(units, ['vitest:unit-core@src/__tests__/command-doc-coverage.test.ts']);
assert.equal(
unitCovers('vitest:unit-core', units[0] as string),
true,
'the whole project covers the file',
);
assert.equal(
unitCovers(units[0] as string, 'vitest:unit-core'),
false,
'the file does not cover the project',
);
});
test('a bare Vitest run spans every configured project', () => {
assert.deepEqual(scriptUnits('all', scriptModel({ all: 'vitest run --coverage' })), [
'vitest:unit-core',
'vitest:subprocess-stub',
]);
});
test('aggregates expand transitively, so a lane running the aggregate owns its parts', () => {
const units = scriptUnits(
'check:all',
scriptModel({
'check:all': 'pnpm lint && pnpm test:unit',
lint: 'oxlint .',
'test:unit': 'vitest run --project unit-core',
}),
);
assert.deepEqual(units, ['script:lint', 'vitest:unit-core']);
});
test('`node --test` globs expand against the tree, which is how test:smoke is owned', () => {
const smoke = scriptUnits('test:smoke', model);
const integration = scriptUnits('test:integration:node', model);
assert.ok(smoke.length > 1, 'the smoke glob must resolve to real files');
for (const unit of smoke) {
assert.ok(integration.includes(unit), `${unit} must be covered by the integration glob`);
}
});