Files
callstack__agent-device/scripts/gate/model.ts
Michał Pierzchała 9c22467832 refactor(ci): make gate ownership structural (#1429) (#1753)
* test(ci): prove every registered gate is owned and reachable (#1429)

A check that silently stops running looks exactly like a green build. Two
suites had already stopped: `check:tmpdir-leaks` (with its model tests) and
`test:fixture-cache` are real package scripts that no workflow ran, reachable
only through the `check:unit` aggregate CI never invokes.

`CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the
only way CI runs one, so finding what a lane runs is a scan for `pnpm gate`
rather than an attempt to interpret shell. `pnpm check:gate-manifest` then
asserts against the real workflows that every registered check is run by some
qualifying lane (per unit, not per script name), that every check the real
selector activates for a path is run by a lane that path would start (#1420's
class), and that every Vitest project and suite script belongs to a check.

The wiring that keeps those honest is asserted too: a gate id must name a
registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false`
unowns what it guards, an action declared to run a gate is proven to, and a
job whose steps the loader cannot open fails closed.

It deliberately does not try to prove CI runs project code only through
`pnpm gate`. Whether a shell block executes project code is not decidable from
its text, so shell this model does not recognise earns no ownership credit —
the failure direction is a check reported unowned, never one waved through.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* test(ci): update the two suites that assert on rewired workflow text

`scripts/mutation/workflow.test.ts` and `test/ci/trusted-fixture-artifact.test.mjs`
read the workflow and action files and assert on their command text, so routing
those steps through `pnpm gate <id>` moved what they were matching.

They are the two suites the manifest cannot help with: it proves a gate is still
run, not that a test asserting on how CI spells a command was updated with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* fix(ci): credit gates by execution shape, and keep every guard

Three ways the manifest could report a gate as owned when it does not run.

1. Crediting was a substring scan over `run:`, which #1429 explicitly rules
   out — "do not infer reachability from a command name merely appearing in
   workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`,
   one named in a heredoc, and `echo pnpm gate x` all credited it. There is a
   live instance: conformance-regenerate.yml's "Fail if regeneration changed
   anything" step names `pnpm gate maestro-regenerate` inside an error message
   telling a human to run it, and that credited the gate.

   A gate now counts only as the first command segment of a line, and a body
   carrying shell structure earns nothing. Reachability inside a script is not
   decidable, so this does not try: unrecognised shape means no credit and the
   check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment
   form is unambiguous and the gate runs.

2. Job-level `if:` was not modelled at all, though six live jobs carry one, so
   a job that cannot run still credited every gate inside it. Two conditions on
   the mutation lanes are now declared.

3. A caller's `if:` REPLACED the guard on a nested composite-action step
   (`guard[0] ?? step.condition`), so an outer `always()` erased an inner
   `if: false`. Steps carry every guard between the lane and the step.

Also corrects two source comments that still claimed project code run outside
the runner fails the manifest. It does not: such a step earns no credit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* ci: add the run-gate action that names a gate structurally

The seam the ownership proof will read instead of shell. A lane says which
gate it runs in `with.gate`, a typed input the manifest reads straight out of
the YAML and validates against CHECK_CATALOG.

Nothing here is wired yet — the ~60 call sites and the model change follow.
Added first so the target of that conversion is reviewable on its own.

`args` cannot select which gate runs; it is appended after the id, so the
worst a wrong value does is fail the gate it already named. There is no
`|| true` and no output capture: the gate's exit code is the step's exit code,
so a gate cannot run without being able to fail its lane.

Part of #1429.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* merge: main (#1770) and route its three new steps through the runner

#1770 landed the orphan-check fix on main, wiring `check:tmpdir-leaks`,
`check:tmpdir-leaks:test` and `test:fixture-cache` into Coverage, Layering
Guard and Integration Tests. This branch had wired the same three through
`pnpm gate`, so the merge produced two steps per check rather than a conflict
— each check ran twice.

Kept main's steps, with the placement and reasoning reviewed on #1770, and
changed only their `run:` line to the canonical runner. Dropped this branch's
duplicates. Net effect on CI is unchanged: the same three checks, in the same
three lanes, once each.

Gate manifest green after the merge: 47 checks wired across 33 lanes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* fix(ci): address review — suite detection, freerange, glob, vacuous skip-list

Six review findings plus the mutation blocker.

[bug] `registered` was shape-only, so a `test:*` script running
`node src/bin.ts test <dir>` resolved to a `script:` leaf and was invisible.
Four `test:replay:*` scripts were owned only because someone hand-registered
them; `test:replay:android` was neither registered nor reported while the
nightly ran the same six .ad files by inlining them. A `test:*` script is now
a suite by name. `replay-android` is registered, and the nightly runs the
script instead of re-listing its files so the two cannot drift.

  The nightly invokes it inside `reactivecircus/android-emulator-runner`'s
  `script:` input — shell handed to a third-party action this loader does not
  read — so the suite executes but cannot be credited. Recorded in
  UNPROVABLE_OWNERS with that exact reason rather than assumed.

  The fixed detector also found a second orphan the review did not name:
  `test:integration:progress`. That one is a reporter whose `--check` sibling
  is the registered gate, so it is declared in REPORTING_SCRIPTS — a
  declaration that itself fails when inert.

[bug] `freerange` defaulted to localRunnable, so fail-open ran `fr` (a Bun
binary) on the pre-push path. Now false.

[suggestion] The `--run` skip-list asserted `build:android-snapshot-helper`,
a name `android-helpers` no longer uses, so it could not fail. Derived from
the catalog instead.

[suggestion] `matchesGlob` joined `**` splits with `.*`, making the adjacent
slash mandatory — GitHub's `**` matches zero directories, so
`src/**/*.test.ts` did not match `src/a.test.ts`. Pinned against
`packages/*/src/**/*.test.ts`.

[suggestion] Deleted the unwired `run-gate` action. It had no callers, was
absent from GATE_ACTIONS, and its comment described a system that had not
shipped. It returns with the rewiring, not before.

[suggestion] Collapsed the module headers that narrated discarded designs.

Mutation: `daemon entrypoint publishes HTTP metadata and cleans up on
shutdown` is the only test here that spawns a real daemon process. It takes
~1.1s alone but exceeds Vitest's 5s default inside Stryker's dry run, which
aborts the sweep before a single mutant runs. Given 30s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* fix(mutation): order sandbox aliases longest-first so subpaths resolve

Every shard of the mutation sweep aborted in Stryker's dry run with:

  Cannot find package '@agent-device/selectors/engine' imported from
    .tmp/stryker/sandbox-*/src/core/selector-pipeline.ts

The alias was generated correctly; it just never won. Vite matches a STRING
alias by prefix and takes the first hit, and `workspaceSpecifierTargets`
emitted the bare `@agent-device/selectors` ahead of the subpath entries. The
bare entry therefore captured `@agent-device/selectors/engine` and rewrote it
to `…/src/index.ts/engine`, which does not exist; Node fell back to real
package resolution, could not find the subpath inside the sandbox, and the dry
run failed before a single mutant ran — so the shard uploaded an empty
envelope instead of a report and the ratchet failed for want of one.

Sorting longest specifier first makes the most specific alias win:

  @agent-device/selectors/engine -> packages/selectors/src/engine.ts
  @agent-device/selectors/ast    -> packages/selectors/src/ast.ts
  @agent-device/selectors        -> packages/selectors/src/index.ts

`/ast` never tripped this because nothing in a related test set imported it;
`selector-pipeline.ts` introduced the first subpath import that mattered
(#1744), so the mutation lane has been unable to run since that landed. Any
PR touching `scripts/mutation/**` — which fails open into the full sweep —
would have hit it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ

* refactor: derive gate ownership from workflow structure

* fix: run gates without optional arguments

* fix: resolve mutation workspace subpaths exactly

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-12 16:02:18 +02:00

218 lines
7.6 KiB
TypeScript

// Derive script units, structural workflow owners, and real selector path categories.
import fs from 'node:fs';
import path from 'node:path';
import { CHECK_CATALOG, type CheckSpec } from '../check-affected/checks.ts';
import { selectChecks, type CheckId } from '../check-affected/model.ts';
import { OPAQUE_RUNNERS } from './declarations.ts';
import { ENV_PREFIX, commandSegments } from './shell.ts';
import { loadLanes, triggersOnPath, type Lane } from './workflows.ts';
// Units distinguish whole Vitest projects, filtered files, node:test files, and scripts.
export type Unit = string;
export type Model = {
readonly scripts: Readonly<Record<string, string>>;
readonly vitestProjects: readonly string[];
readonly lanes: readonly Lane[];
readonly trackedFiles: ReadonlySet<string>;
readonly packageEntryFiles: readonly string[];
readonly opaque: Readonly<Record<string, readonly string[]>>;
};
// --- Units -------------------------------------------------------------------
function tokens(segment: string): string[] {
return segment.split(/\s+/).filter(Boolean);
}
/** `pnpm x`, `pnpm run x`, `pnpm --silent x` — the invoked script, or null. */
function invokedScript(segment: string, scripts: Readonly<Record<string, string>>): string | null {
const parts = tokens(segment);
if (parts[0] !== 'pnpm') return null;
for (const part of parts.slice(1)) {
if (part === 'run' || part.startsWith('-')) continue;
return part in scripts ? part : null;
}
return null;
}
function expandGlob(pattern: string): string[] {
if (!pattern.includes('*')) return [pattern];
const dir = path.dirname(pattern);
const rest = path.basename(pattern);
const matcher = new RegExp(
`^${rest.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '[^/]*')}$`,
);
if (!fs.existsSync(dir)) return [];
return fs
.readdirSync(dir)
.filter((entry) => matcher.test(entry))
.map((entry) => path.posix.join(dir, entry))
.sort();
}
function vitestArgs(parts: readonly string[]): {
named: string[];
files: string[];
} {
const named: string[] = [];
const files: string[] = [];
for (let i = 0; i < parts.length; i++) {
const part = parts[i] ?? '';
if (part === '--project') named.push(parts[++i] ?? '');
else if (part.startsWith('--project=')) named.push(part.slice('--project='.length));
else if (!part.startsWith('-') && /[./]/.test(part) && !RUNNER_TOKENS.test(part))
files.push(part);
}
return { named, files };
}
const RUNNER_TOKENS = /^(?:pnpm|exec|vitest|run)$/;
function vitestUnits(parts: readonly string[], projects: readonly string[]): Unit[] {
const { named, files } = vitestArgs(parts);
const selected = named.length > 0 ? named : projects;
const suffix = files.length > 0 ? `@${files.join(',')}` : '';
return selected.map((project) => `vitest:${project}${suffix}`);
}
function nodeTestUnits(parts: readonly string[]): Unit[] {
const targets = parts
.slice(parts.indexOf('--test') + 1)
.filter((part) => !part.startsWith('-'))
.flatMap(expandGlob);
return targets.map((file) => `node-test:${file}`);
}
export function scriptUnits(
script: string,
model: Pick<Model, 'scripts' | 'vitestProjects' | 'opaque'>,
seen: ReadonlySet<string> = new Set(),
): Unit[] {
const declared = model.opaque[script];
if (declared) return [...declared];
const body = model.scripts[script];
if (body === undefined || seen.has(script)) return [];
const next = new Set([...seen, script]);
return [
...new Set(
commandSegments(body).flatMap((segment) => segmentUnits(segment, script, model, next)),
),
];
}
function segmentUnits(
raw: string,
script: string,
model: Pick<Model, 'scripts' | 'vitestProjects' | 'opaque'>,
seen: ReadonlySet<string>,
): Unit[] {
const segment = raw.replace(ENV_PREFIX, '');
const nested = invokedScript(segment, model.scripts);
if (nested) return scriptUnits(nested, model, seen);
const parts = tokens(segment);
if (parts.includes('vitest')) return vitestUnits(parts, model.vitestProjects);
if (parts.includes('--test')) return nodeTestUnits(parts);
return [`script:${script}`];
}
/** A lane running `have` satisfies a need for `want`. Whole projects cover their files. */
export function unitCovers(have: Unit, want: Unit): boolean {
if (have === want) return true;
return want.startsWith(`${have}@`);
}
export function checkUnits(spec: CheckSpec, model: Model): Unit[] {
// `vitest-related` has no script: it is Vitest's own `related` command over the
// diff, so the lane that runs the whole suite is what owns it.
if (spec.kind.type === 'vitest-related') {
return model.vitestProjects.map((name) => `vitest:${name}`);
}
return scriptUnits(spec.kind.script, model);
}
// --- Coverage ----------------------------------------------------------------
function laneUnits(lane: Lane, model: Model): Unit[] {
const fromGates = lane.gates.flatMap((id) => {
const spec = CHECK_CATALOG.find((entry) => entry.id === id);
return spec ? checkUnits(spec, model) : [];
});
return [...fromGates, ...lane.verbatim.flatMap((name) => scriptUnits(name, model))];
}
export function covered(
spec: CheckSpec,
file: string | null,
model: Model,
): { covered: boolean; missing: Unit[]; lanes: string[] } {
const wanted = checkUnits(spec, model);
const usable = model.lanes.filter(
(lane) => lane.qualifying && (file === null || triggersOnPath(lane, file)),
);
const owners = new Map<Unit, string[]>();
for (const lane of usable) {
const have = laneUnits(lane, model);
for (const want of wanted) {
if (have.some((unit) => unitCovers(unit, want)))
owners.set(want, [...(owners.get(want) ?? []), lane.label]);
}
}
const missing = wanted.filter((unit) => !owners.has(unit));
return {
covered: wanted.length > 0 && missing.length === 0,
missing,
lanes: [...new Set([...owners.values()].flat())].sort(),
};
}
export type Category = {
readonly rule: string;
readonly path: string;
readonly checks: readonly CheckId[];
};
// One real tracked path per selector rule; fictional hand-written samples are impossible.
export function categories(model: Model): Category[] {
const found = new Map<string, Category>();
for (const path of [...model.trackedFiles].sort()) {
const plan = selectChecks({
changedFiles: [path],
packageEntryFiles: model.packageEntryFiles,
});
if (plan.failOpen) continue;
for (const { rule } of plan.reasons) {
if (!found.has(rule)) found.set(rule, { rule, path, checks: plan.checks });
}
}
return [...found.values()];
}
export function owningLanes(model: Model): Map<CheckId, string[]> {
return new Map(CHECK_CATALOG.map((spec) => [spec.id, covered(spec, null, model).lanes]));
}
export function loadModel(
repoRoot: string,
trackedFiles: readonly string[],
opaque: Readonly<Record<string, readonly string[]>> = OPAQUE_RUNNERS,
): Model {
const pkg = JSON.parse(fs.readFileSync(path.join(repoRoot, 'package.json'), 'utf8')) as {
scripts: Record<string, string>;
exports?: Record<string, { import?: string }>;
};
const config = fs.readFileSync(path.join(repoRoot, 'vitest.config.ts'), 'utf8');
return {
scripts: pkg.scripts,
packageEntryFiles: Object.values(pkg.exports ?? {})
.map((entry) => entry.import)
.filter((target): target is string => typeof target === 'string')
.map((target) => target.replace(/^\.\/dist\//, '').replace(/\.js$/, '.ts')),
vitestProjects: [...config.matchAll(/name:\s*'([^']+)'/g)].map((match) => match[1] as string),
lanes: loadLanes(path.join(repoRoot, '.github/workflows'), repoRoot, pkg.scripts),
trackedFiles: new Set(trackedFiles),
opaque,
};
}