Files
callstack__agent-device/scripts/gate/model.test.ts
Michał Pierzchała 104fe75248 fix(ci): run the fuzz corpus replay outside the coverage lane (#1994)
The Coverage job intermittently ends with no failing test and one file's
results missing:

    Test Files  1070 passed (1071)
    Errors      1 error
    Error: [vitest-pool]: Worker forks emitted error.
    Caused by: Error: Worker exited unexpectedly

This is shape (B) of #1824 — the half #1854 did not fix. Scanning every
failed Coverage job across the 120 CI runs since #1854 merged finds the
signature five times, and the vanished file is
scripts/fuzz/corpus-replay.test.ts all five (six for six with #1866's
occurrence): 23% of Coverage failures in that window, ~4% of all CI runs.

The ~40s gap before the error is coverage report generation, not test
time — the pool surfaces its AggregateError only once every task settles.
Control, from a green attempt of the same run: the file passes in 3152ms
at 09:37:35.9 and the summary prints at 09:38:12.5. So the file is not
slow in CI, nothing else is in flight when it dies, and neither a missed
per-case budget nor STARTUP_BUDGET_MS is implicated. Partial test counts
(3/11 and 9/11 reported) place the death mid-file, inside runCases.

So the corpus replay gets its own serialized project that the coverage
run skips, and a second uninstrumented Vitest invocation in
`test:coverage:ci` runs it, keeping the tests on every PR. Measured
against two full runs, this costs zero coverage: the cases execute in
worker threads, a separate isolate the fork's inspector never
instruments, so the lines reported are identical with and without it.

Membership is by demonstrated failure, not by a property of the code:
`session-replay-runtime-maestro.test.ts` also constructs a
node:worker_threads Worker and stays in unit-core, instrumented and
green, so "nests a Worker" is explicitly not the criterion.

The second leg goes through `test:fuzz-worker`, which blanks
AGENT_DEVICE_COVERAGE_SHARD and AGENT_DEVICE_COVERAGE_MERGE. ci.yml sets
those as job-level env over a single `gate: unit-ci` step, so both legs
would otherwise inherit them and the shard would die: Vitest refuses
`--shard=1/2` over this one-file project, and the blob reporter
overwrites the instrumented shard's report on its way out. Verified on
the merged tree — shard 1/2 (549 files), shard 2/2 (548), and the merge
job (1097 files, 90.38% lines) all pass, and the leg still fails without
the blanking.

Refs #1824
2026-08-24 17:03:25 +02:00

113 lines
3.8 KiB
TypeScript

// Unit-resolution tests, kept to the cases where a mistake would OVER-credit a lane.
//
// Under-credit corrects itself: the real tree is audited on every PR, so a command
// the model fails to read shows up as an unowned check within one run. Over-credit
// is the dangerous direction — it reads as coverage that is not there — so those are
// the shapes pinned here, plus the two real-tree facts the unit vocabulary exists for.
import { execFileSync } from 'node:child_process';
import assert from 'node:assert/strict';
import path from 'node:path';
import test from 'node:test';
import { loadModel, scriptUnits, unitCovers } from './model.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
const tracked = execFileSync('git', ['ls-files'], {
cwd: repoRoot,
encoding: 'utf8',
})
.split('\n')
.filter(Boolean);
const model = loadModel(repoRoot, tracked);
const scriptModel = (scripts: Record<string, string>) => ({
scripts,
vitestProjects: ['unit-core', 'subprocess-stub'],
opaque: {},
});
test('an env prefix does not hide the command behind it', () => {
assert.deepEqual(
scriptUnits(
'build:x',
scriptModel({
'build:x': 'AGENT_DEVICE_XCUITEST_PLATFORM=ios sh ./scripts/build.sh',
}),
),
['script:build:x'],
);
});
test('a filtered Vitest run does not credit the whole project', () => {
const units = scriptUnits(
'docs',
scriptModel({
docs: 'vitest run --project unit-core src/__tests__/command-doc-coverage.test.ts',
}),
);
assert.deepEqual(units, ['vitest:unit-core@src/__tests__/command-doc-coverage.test.ts']);
assert.equal(
unitCovers('vitest:unit-core', units[0] as string),
true,
'the whole project covers the file',
);
assert.equal(
unitCovers(units[0] as string, 'vitest:unit-core'),
false,
'the file does not cover the project',
);
});
test('a bare Vitest run spans every configured project', () => {
assert.deepEqual(scriptUnits('all', scriptModel({ all: 'vitest run --coverage' })), [
'vitest:unit-core',
'vitest:subprocess-stub',
]);
});
test('a negated --project subtracts from the configured set, so the skipped one is not credited', () => {
assert.deepEqual(
scriptUnits('cov', scriptModel({ cov: 'vitest run --coverage --project=!subprocess-stub' })),
['vitest:unit-core'],
);
});
// The real `test:coverage:ci` shape: the second leg is a nested script whose body carries an env
// prefix (it blanks the coverage-shard switches). Both indirections have to survive, or the lane
// stops owning the project it hands to that leg.
test('the two halves of test:coverage:ci together still own every project', () => {
assert.deepEqual(
scriptUnits(
'test:coverage:ci',
scriptModel({
'test:coverage:ci':
'vitest run --coverage --project=!subprocess-stub && pnpm test:subprocess-stub',
'test:subprocess-stub':
'AGENT_DEVICE_COVERAGE_SHARD= AGENT_DEVICE_COVERAGE_MERGE= vitest run --project subprocess-stub',
}),
),
['vitest:unit-core', 'vitest:subprocess-stub'],
);
});
test('aggregates expand transitively, so a lane running the aggregate owns its parts', () => {
const units = scriptUnits(
'check:all',
scriptModel({
'check:all': 'pnpm lint && pnpm test:unit',
lint: 'oxlint .',
'test:unit': 'vitest run --project unit-core',
}),
);
assert.deepEqual(units, ['script:lint', 'vitest:unit-core']);
});
test('`node --test` globs expand against the tree, which is how test:smoke is owned', () => {
const smoke = scriptUnits('test:smoke', model);
const integration = scriptUnits('test:integration:node', model);
assert.ok(smoke.length > 1, 'the smoke glob must resolve to real files');
for (const unit of smoke) {
assert.ok(integration.includes(unit), `${unit} must be covered by the integration glob`);
}
});