mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
9c22467832
* test(ci): prove every registered gate is owned and reachable (#1429) A check that silently stops running looks exactly like a green build. Two suites had already stopped: `check:tmpdir-leaks` (with its model tests) and `test:fixture-cache` are real package scripts that no workflow ran, reachable only through the `check:unit` aggregate CI never invokes. `CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the only way CI runs one, so finding what a lane runs is a scan for `pnpm gate` rather than an attempt to interpret shell. `pnpm check:gate-manifest` then asserts against the real workflows that every registered check is run by some qualifying lane (per unit, not per script name), that every check the real selector activates for a path is run by a lane that path would start (#1420's class), and that every Vitest project and suite script belongs to a check. The wiring that keeps those honest is asserted too: a gate id must name a registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false` unowns what it guards, an action declared to run a gate is proven to, and a job whose steps the loader cannot open fails closed. It deliberately does not try to prove CI runs project code only through `pnpm gate`. Whether a shell block executes project code is not decidable from its text, so shell this model does not recognise earns no ownership credit — the failure direction is a check reported unowned, never one waved through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * test(ci): update the two suites that assert on rewired workflow text `scripts/mutation/workflow.test.ts` and `test/ci/trusted-fixture-artifact.test.mjs` read the workflow and action files and assert on their command text, so routing those steps through `pnpm gate <id>` moved what they were matching. They are the two suites the manifest cannot help with: it proves a gate is still run, not that a test asserting on how CI spells a command was updated with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(ci): credit gates by execution shape, and keep every guard Three ways the manifest could report a gate as owned when it does not run. 1. Crediting was a substring scan over `run:`, which #1429 explicitly rules out — "do not infer reachability from a command name merely appearing in workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`, one named in a heredoc, and `echo pnpm gate x` all credited it. There is a live instance: conformance-regenerate.yml's "Fail if regeneration changed anything" step names `pnpm gate maestro-regenerate` inside an error message telling a human to run it, and that credited the gate. A gate now counts only as the first command segment of a line, and a body carrying shell structure earns nothing. Reachability inside a script is not decidable, so this does not try: unrecognised shape means no credit and the check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment form is unambiguous and the gate runs. 2. Job-level `if:` was not modelled at all, though six live jobs carry one, so a job that cannot run still credited every gate inside it. Two conditions on the mutation lanes are now declared. 3. A caller's `if:` REPLACED the guard on a nested composite-action step (`guard[0] ?? step.condition`), so an outer `always()` erased an inner `if: false`. Steps carry every guard between the lane and the step. Also corrects two source comments that still claimed project code run outside the runner fails the manifest. It does not: such a step earns no credit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * ci: add the run-gate action that names a gate structurally The seam the ownership proof will read instead of shell. A lane says which gate it runs in `with.gate`, a typed input the manifest reads straight out of the YAML and validates against CHECK_CATALOG. Nothing here is wired yet — the ~60 call sites and the model change follow. Added first so the target of that conversion is reviewable on its own. `args` cannot select which gate runs; it is appended after the id, so the worst a wrong value does is fail the gate it already named. There is no `|| true` and no output capture: the gate's exit code is the step's exit code, so a gate cannot run without being able to fail its lane. Part of #1429. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * merge: main (#1770) and route its three new steps through the runner #1770 landed the orphan-check fix on main, wiring `check:tmpdir-leaks`, `check:tmpdir-leaks:test` and `test:fixture-cache` into Coverage, Layering Guard and Integration Tests. This branch had wired the same three through `pnpm gate`, so the merge produced two steps per check rather than a conflict — each check ran twice. Kept main's steps, with the placement and reasoning reviewed on #1770, and changed only their `run:` line to the canonical runner. Dropped this branch's duplicates. Net effect on CI is unchanged: the same three checks, in the same three lanes, once each. Gate manifest green after the merge: 47 checks wired across 33 lanes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(ci): address review — suite detection, freerange, glob, vacuous skip-list Six review findings plus the mutation blocker. [bug] `registered` was shape-only, so a `test:*` script running `node src/bin.ts test <dir>` resolved to a `script:` leaf and was invisible. Four `test:replay:*` scripts were owned only because someone hand-registered them; `test:replay:android` was neither registered nor reported while the nightly ran the same six .ad files by inlining them. A `test:*` script is now a suite by name. `replay-android` is registered, and the nightly runs the script instead of re-listing its files so the two cannot drift. The nightly invokes it inside `reactivecircus/android-emulator-runner`'s `script:` input — shell handed to a third-party action this loader does not read — so the suite executes but cannot be credited. Recorded in UNPROVABLE_OWNERS with that exact reason rather than assumed. The fixed detector also found a second orphan the review did not name: `test:integration:progress`. That one is a reporter whose `--check` sibling is the registered gate, so it is declared in REPORTING_SCRIPTS — a declaration that itself fails when inert. [bug] `freerange` defaulted to localRunnable, so fail-open ran `fr` (a Bun binary) on the pre-push path. Now false. [suggestion] The `--run` skip-list asserted `build:android-snapshot-helper`, a name `android-helpers` no longer uses, so it could not fail. Derived from the catalog instead. [suggestion] `matchesGlob` joined `**` splits with `.*`, making the adjacent slash mandatory — GitHub's `**` matches zero directories, so `src/**/*.test.ts` did not match `src/a.test.ts`. Pinned against `packages/*/src/**/*.test.ts`. [suggestion] Deleted the unwired `run-gate` action. It had no callers, was absent from GATE_ACTIONS, and its comment described a system that had not shipped. It returns with the rewiring, not before. [suggestion] Collapsed the module headers that narrated discarded designs. Mutation: `daemon entrypoint publishes HTTP metadata and cleans up on shutdown` is the only test here that spawns a real daemon process. It takes ~1.1s alone but exceeds Vitest's 5s default inside Stryker's dry run, which aborts the sweep before a single mutant runs. Given 30s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * fix(mutation): order sandbox aliases longest-first so subpaths resolve Every shard of the mutation sweep aborted in Stryker's dry run with: Cannot find package '@agent-device/selectors/engine' imported from .tmp/stryker/sandbox-*/src/core/selector-pipeline.ts The alias was generated correctly; it just never won. Vite matches a STRING alias by prefix and takes the first hit, and `workspaceSpecifierTargets` emitted the bare `@agent-device/selectors` ahead of the subpath entries. The bare entry therefore captured `@agent-device/selectors/engine` and rewrote it to `…/src/index.ts/engine`, which does not exist; Node fell back to real package resolution, could not find the subpath inside the sandbox, and the dry run failed before a single mutant ran — so the shard uploaded an empty envelope instead of a report and the ratchet failed for want of one. Sorting longest specifier first makes the most specific alias win: @agent-device/selectors/engine -> packages/selectors/src/engine.ts @agent-device/selectors/ast -> packages/selectors/src/ast.ts @agent-device/selectors -> packages/selectors/src/index.ts `/ast` never tripped this because nothing in a related test set imported it; `selector-pipeline.ts` introduced the first subpath import that mattered (#1744), so the mutation lane has been unable to run since that landed. Any PR touching `scripts/mutation/**` — which fails open into the full sweep — would have hit it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ * refactor: derive gate ownership from workflow structure * fix: run gates without optional arguments * fix: resolve mutation workspace subpaths exactly --------- Co-authored-by: Claude <noreply@anthropic.com>
218 lines
7.6 KiB
TypeScript
218 lines
7.6 KiB
TypeScript
// Derive script units, structural workflow owners, and real selector path categories.
|
|
|
|
import fs from 'node:fs';
|
|
import path from 'node:path';
|
|
import { CHECK_CATALOG, type CheckSpec } from '../check-affected/checks.ts';
|
|
import { selectChecks, type CheckId } from '../check-affected/model.ts';
|
|
import { OPAQUE_RUNNERS } from './declarations.ts';
|
|
import { ENV_PREFIX, commandSegments } from './shell.ts';
|
|
import { loadLanes, triggersOnPath, type Lane } from './workflows.ts';
|
|
|
|
// Units distinguish whole Vitest projects, filtered files, node:test files, and scripts.
|
|
export type Unit = string;
|
|
|
|
export type Model = {
|
|
readonly scripts: Readonly<Record<string, string>>;
|
|
readonly vitestProjects: readonly string[];
|
|
readonly lanes: readonly Lane[];
|
|
readonly trackedFiles: ReadonlySet<string>;
|
|
readonly packageEntryFiles: readonly string[];
|
|
readonly opaque: Readonly<Record<string, readonly string[]>>;
|
|
};
|
|
|
|
// --- Units -------------------------------------------------------------------
|
|
|
|
function tokens(segment: string): string[] {
|
|
return segment.split(/\s+/).filter(Boolean);
|
|
}
|
|
|
|
/** `pnpm x`, `pnpm run x`, `pnpm --silent x` — the invoked script, or null. */
|
|
function invokedScript(segment: string, scripts: Readonly<Record<string, string>>): string | null {
|
|
const parts = tokens(segment);
|
|
if (parts[0] !== 'pnpm') return null;
|
|
for (const part of parts.slice(1)) {
|
|
if (part === 'run' || part.startsWith('-')) continue;
|
|
return part in scripts ? part : null;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function expandGlob(pattern: string): string[] {
|
|
if (!pattern.includes('*')) return [pattern];
|
|
const dir = path.dirname(pattern);
|
|
const rest = path.basename(pattern);
|
|
const matcher = new RegExp(
|
|
`^${rest.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '[^/]*')}$`,
|
|
);
|
|
if (!fs.existsSync(dir)) return [];
|
|
return fs
|
|
.readdirSync(dir)
|
|
.filter((entry) => matcher.test(entry))
|
|
.map((entry) => path.posix.join(dir, entry))
|
|
.sort();
|
|
}
|
|
|
|
function vitestArgs(parts: readonly string[]): {
|
|
named: string[];
|
|
files: string[];
|
|
} {
|
|
const named: string[] = [];
|
|
const files: string[] = [];
|
|
for (let i = 0; i < parts.length; i++) {
|
|
const part = parts[i] ?? '';
|
|
if (part === '--project') named.push(parts[++i] ?? '');
|
|
else if (part.startsWith('--project=')) named.push(part.slice('--project='.length));
|
|
else if (!part.startsWith('-') && /[./]/.test(part) && !RUNNER_TOKENS.test(part))
|
|
files.push(part);
|
|
}
|
|
return { named, files };
|
|
}
|
|
|
|
const RUNNER_TOKENS = /^(?:pnpm|exec|vitest|run)$/;
|
|
|
|
function vitestUnits(parts: readonly string[], projects: readonly string[]): Unit[] {
|
|
const { named, files } = vitestArgs(parts);
|
|
const selected = named.length > 0 ? named : projects;
|
|
const suffix = files.length > 0 ? `@${files.join(',')}` : '';
|
|
return selected.map((project) => `vitest:${project}${suffix}`);
|
|
}
|
|
|
|
function nodeTestUnits(parts: readonly string[]): Unit[] {
|
|
const targets = parts
|
|
.slice(parts.indexOf('--test') + 1)
|
|
.filter((part) => !part.startsWith('-'))
|
|
.flatMap(expandGlob);
|
|
return targets.map((file) => `node-test:${file}`);
|
|
}
|
|
|
|
export function scriptUnits(
|
|
script: string,
|
|
model: Pick<Model, 'scripts' | 'vitestProjects' | 'opaque'>,
|
|
seen: ReadonlySet<string> = new Set(),
|
|
): Unit[] {
|
|
const declared = model.opaque[script];
|
|
if (declared) return [...declared];
|
|
const body = model.scripts[script];
|
|
if (body === undefined || seen.has(script)) return [];
|
|
const next = new Set([...seen, script]);
|
|
return [
|
|
...new Set(
|
|
commandSegments(body).flatMap((segment) => segmentUnits(segment, script, model, next)),
|
|
),
|
|
];
|
|
}
|
|
|
|
function segmentUnits(
|
|
raw: string,
|
|
script: string,
|
|
model: Pick<Model, 'scripts' | 'vitestProjects' | 'opaque'>,
|
|
seen: ReadonlySet<string>,
|
|
): Unit[] {
|
|
const segment = raw.replace(ENV_PREFIX, '');
|
|
const nested = invokedScript(segment, model.scripts);
|
|
if (nested) return scriptUnits(nested, model, seen);
|
|
const parts = tokens(segment);
|
|
if (parts.includes('vitest')) return vitestUnits(parts, model.vitestProjects);
|
|
if (parts.includes('--test')) return nodeTestUnits(parts);
|
|
return [`script:${script}`];
|
|
}
|
|
|
|
/** A lane running `have` satisfies a need for `want`. Whole projects cover their files. */
|
|
export function unitCovers(have: Unit, want: Unit): boolean {
|
|
if (have === want) return true;
|
|
return want.startsWith(`${have}@`);
|
|
}
|
|
|
|
export function checkUnits(spec: CheckSpec, model: Model): Unit[] {
|
|
// `vitest-related` has no script: it is Vitest's own `related` command over the
|
|
// diff, so the lane that runs the whole suite is what owns it.
|
|
if (spec.kind.type === 'vitest-related') {
|
|
return model.vitestProjects.map((name) => `vitest:${name}`);
|
|
}
|
|
return scriptUnits(spec.kind.script, model);
|
|
}
|
|
|
|
// --- Coverage ----------------------------------------------------------------
|
|
|
|
function laneUnits(lane: Lane, model: Model): Unit[] {
|
|
const fromGates = lane.gates.flatMap((id) => {
|
|
const spec = CHECK_CATALOG.find((entry) => entry.id === id);
|
|
return spec ? checkUnits(spec, model) : [];
|
|
});
|
|
return [...fromGates, ...lane.verbatim.flatMap((name) => scriptUnits(name, model))];
|
|
}
|
|
|
|
export function covered(
|
|
spec: CheckSpec,
|
|
file: string | null,
|
|
model: Model,
|
|
): { covered: boolean; missing: Unit[]; lanes: string[] } {
|
|
const wanted = checkUnits(spec, model);
|
|
const usable = model.lanes.filter(
|
|
(lane) => lane.qualifying && (file === null || triggersOnPath(lane, file)),
|
|
);
|
|
const owners = new Map<Unit, string[]>();
|
|
for (const lane of usable) {
|
|
const have = laneUnits(lane, model);
|
|
for (const want of wanted) {
|
|
if (have.some((unit) => unitCovers(unit, want)))
|
|
owners.set(want, [...(owners.get(want) ?? []), lane.label]);
|
|
}
|
|
}
|
|
const missing = wanted.filter((unit) => !owners.has(unit));
|
|
return {
|
|
covered: wanted.length > 0 && missing.length === 0,
|
|
missing,
|
|
lanes: [...new Set([...owners.values()].flat())].sort(),
|
|
};
|
|
}
|
|
|
|
export type Category = {
|
|
readonly rule: string;
|
|
readonly path: string;
|
|
readonly checks: readonly CheckId[];
|
|
};
|
|
|
|
// One real tracked path per selector rule; fictional hand-written samples are impossible.
|
|
export function categories(model: Model): Category[] {
|
|
const found = new Map<string, Category>();
|
|
for (const path of [...model.trackedFiles].sort()) {
|
|
const plan = selectChecks({
|
|
changedFiles: [path],
|
|
packageEntryFiles: model.packageEntryFiles,
|
|
});
|
|
if (plan.failOpen) continue;
|
|
for (const { rule } of plan.reasons) {
|
|
if (!found.has(rule)) found.set(rule, { rule, path, checks: plan.checks });
|
|
}
|
|
}
|
|
return [...found.values()];
|
|
}
|
|
|
|
export function owningLanes(model: Model): Map<CheckId, string[]> {
|
|
return new Map(CHECK_CATALOG.map((spec) => [spec.id, covered(spec, null, model).lanes]));
|
|
}
|
|
|
|
export function loadModel(
|
|
repoRoot: string,
|
|
trackedFiles: readonly string[],
|
|
opaque: Readonly<Record<string, readonly string[]>> = OPAQUE_RUNNERS,
|
|
): Model {
|
|
const pkg = JSON.parse(fs.readFileSync(path.join(repoRoot, 'package.json'), 'utf8')) as {
|
|
scripts: Record<string, string>;
|
|
exports?: Record<string, { import?: string }>;
|
|
};
|
|
const config = fs.readFileSync(path.join(repoRoot, 'vitest.config.ts'), 'utf8');
|
|
return {
|
|
scripts: pkg.scripts,
|
|
packageEntryFiles: Object.values(pkg.exports ?? {})
|
|
.map((entry) => entry.import)
|
|
.filter((target): target is string => typeof target === 'string')
|
|
.map((target) => target.replace(/^\.\/dist\//, '').replace(/\.js$/, '.ts')),
|
|
vitestProjects: [...config.matchAll(/name:\s*'([^']+)'/g)].map((match) => match[1] as string),
|
|
lanes: loadLanes(path.join(repoRoot, '.github/workflows'), repoRoot, pkg.scripts),
|
|
trackedFiles: new Set(trackedFiles),
|
|
opaque,
|
|
};
|
|
}
|