Files
callstack__agent-device/scripts/fuzz/options.ts
devin-ai-integration[bot] 006c4cadc9 test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414) (#1438)
* test: nightly parser fuzz lane with typed-AppError invariant (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): run envelope, artifact promotion, and harness self-check tests (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): shared scheduled-lane envelope on every terminal path, watchdog after ready (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): envelope for malformed options; add scheduled-lane health consumer (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): actions:read scope, terminal error envelope, first-due grace (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): anchor first-run grace to schedule registration, use exec helper in tests (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): portable POSIX pickaxe pattern for schedule registration (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(fuzz): fast-check generators over the shared hazard list, drop the bundled lane-health work (#1414)

- Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own
  deliverable and collides with PR #1439's implementation of the same lane. What this lane owes
  (a per-run envelope) moves into scripts/fuzz/envelope.ts.
- Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries
  sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a
  failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string.
- Route harness.test.ts into the serialized subprocess-stub project.
- Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a
  case budget far below real worker startup.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): replay the regression corpus through the worker watchdog (#1414)

A promoted hang case used to wedge the unit job until the CI timeout, because corpus replay called
checkCase in-process. It now goes through the same worker-backed watchdog the nightly lane uses, so
such a case fails against a 5s per-case budget; the file moves to the serialized subprocess-stub
project with the rest of the worker-driven fuzz tests.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): let the watchdog outlive vitest's default case timeout (#1414)

A wedged parser was surfacing as a bare 'Test timed out in 5000ms' instead of the named hang:
failure that says which input wedged, because the file's vitest timeout was shorter than the
watchdog budget times the number of replayed cases.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): complete drift provenance in the lane envelope (#1414)

configHash now covers every input that decides what a seed generates (generate.ts and the shared property arbitraries, not just the arbitraries/targets/invariant), and tool records fast-check's installed version. A generation-loop edit or a fast-check upgrade previously changed the case set while the envelope looked unchanged. A test recomputes the hash with each input omitted so a future omission fails.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-28 11:29:25 +02:00

100 lines
3.8 KiB
TypeScript

// Command-line surface for `pnpm fuzz:parsers` (#1414).
import { parseArgs } from 'node:util';
export type FuzzOptions = {
target?: string;
iterations: number;
seed: number;
caseTimeoutMs: number;
artifactDir: string;
inputFile?: string;
appendCorpus: boolean;
replayCorpus: boolean;
selfCheck: boolean;
};
export const FUZZ_USAGE = `Usage: pnpm fuzz:parsers [options]
--target <name> Fuzz one target only (default: all)
--iterations <n> Cases per target (default: 2000)
--seed <n> fast-check seed (default: 1). Same seed = same cases.
--case-timeout-ms <n> Per-case watchdog budget (default: 2000)
--artifact-dir <dir> Where failing cases are written (default: .tmp/fuzz)
--input-file <file> Replay a single saved failing case (JSON artifact) and exit
--append-corpus Promote failures (incl. an --input-file artifact) into the corpus
--replay-corpus Replay the checked-in corpus instead of generating cases
--self-check Run the broken-on-purpose targets and require each to be caught
`;
const DEFAULTS = {
iterations: '2000',
seed: '1',
caseTimeoutMs: '2000',
artifactDir: '.tmp/fuzz',
} as const;
function positiveInt(raw: string | undefined, name: string): number {
const value = Number(raw);
if (!Number.isInteger(value) || value <= 0) {
throw new Error(`--${name} must be a positive integer (got ${String(raw)}).`);
}
return value;
}
/** `{ key: value }`, or nothing when the flag was not passed (exactOptionalPropertyTypes). */
function optional<K extends string>(key: K, value: string | undefined): Record<K, string> | object {
return value === undefined ? {} : ({ [key]: value } as Record<K, string>);
}
/**
* Options to fall back on when argv itself is unusable, so a malformed dispatch input still
* produces an envelope. `--artifact-dir` is recovered positionally: the value that decides *where*
* monitoring looks must survive a rejected flag elsewhere in argv.
*/
export function fallbackFuzzOptions(argv: readonly string[]): FuzzOptions {
const flag = argv.indexOf('--artifact-dir');
const dir = flag === -1 ? undefined : argv[flag + 1];
return {
iterations: Number(DEFAULTS.iterations),
seed: Number(DEFAULTS.seed),
caseTimeoutMs: Number(DEFAULTS.caseTimeoutMs),
artifactDir: dir !== undefined && !dir.startsWith('--') ? dir : DEFAULTS.artifactDir,
appendCorpus: false,
replayCorpus: false,
selfCheck: argv.includes('--self-check'),
};
}
/** Parses argv into options; `null` means usage was requested and nothing should run. */
export function readFuzzOptions(argv: readonly string[]): FuzzOptions | null {
const { values } = parseArgs({
args: [...argv],
options: {
target: { type: 'string' },
iterations: { type: 'string', default: DEFAULTS.iterations },
seed: { type: 'string', default: DEFAULTS.seed },
'case-timeout-ms': { type: 'string', default: DEFAULTS.caseTimeoutMs },
'artifact-dir': { type: 'string', default: DEFAULTS.artifactDir },
'input-file': { type: 'string' },
'append-corpus': { type: 'boolean', default: false },
'replay-corpus': { type: 'boolean', default: false },
'self-check': { type: 'boolean', default: false },
help: { type: 'boolean', short: 'h', default: false },
},
allowPositionals: false,
});
if (values.help === true) return null;
return {
...optional('target', values.target),
...optional('inputFile', values['input-file']),
iterations: positiveInt(values.iterations, 'iterations'),
seed: positiveInt(values.seed, 'seed'),
caseTimeoutMs: positiveInt(values['case-timeout-ms'], 'case-timeout-ms'),
artifactDir: String(values['artifact-dir']),
appendCorpus: values['append-corpus'] === true,
replayCorpus: values['replay-corpus'] === true,
selfCheck: values['self-check'] === true,
};
}