Files
callstack__agent-device/scripts/fuzz/corpus.ts
devin-ai-integration[bot] 006c4cadc9 test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414) (#1438)
* test: nightly parser fuzz lane with typed-AppError invariant (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): run envelope, artifact promotion, and harness self-check tests (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): shared scheduled-lane envelope on every terminal path, watchdog after ready (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): envelope for malformed options; add scheduled-lane health consumer (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): actions:read scope, terminal error envelope, first-due grace (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): anchor first-run grace to schedule registration, use exec helper in tests (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): portable POSIX pickaxe pattern for schedule registration (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(fuzz): fast-check generators over the shared hazard list, drop the bundled lane-health work (#1414)

- Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own
  deliverable and collides with PR #1439's implementation of the same lane. What this lane owes
  (a per-run envelope) moves into scripts/fuzz/envelope.ts.
- Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries
  sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a
  failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string.
- Route harness.test.ts into the serialized subprocess-stub project.
- Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a
  case budget far below real worker startup.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): replay the regression corpus through the worker watchdog (#1414)

A promoted hang case used to wedge the unit job until the CI timeout, because corpus replay called
checkCase in-process. It now goes through the same worker-backed watchdog the nightly lane uses, so
such a case fails against a 5s per-case budget; the file moves to the serialized subprocess-stub
project with the rest of the worker-driven fuzz tests.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): let the watchdog outlive vitest's default case timeout (#1414)

A wedged parser was surfacing as a bare 'Test timed out in 5000ms' instead of the named hang:
failure that says which input wedged, because the file's vitest timeout was shorter than the
watchdog budget times the number of replayed cases.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): complete drift provenance in the lane envelope (#1414)

configHash now covers every input that decides what a seed generates (generate.ts and the shared property arbitraries, not just the arbitraries/targets/invariant), and tool records fast-check's installed version. A generation-loop edit or a fast-check upgrade previously changed the case set while the envelope looked unchanged. A test recomputes the hash with each input omitted so a future omission fails.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-28 11:29:25 +02:00

67 lines
2.7 KiB
TypeScript

// The checked-in regression corpus for the parser fuzz lane (#1414).
//
// Every input the fuzzer ever catches is appended here and replayed by the unit lane
// (scripts/fuzz/corpus-replay.test.ts), so a fixed parser stays fixed without waiting for
// the nightly to rediscover the case. Entries are sorted and deduplicated on write, which
// keeps the diff of an append small and the replay order deterministic.
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import type { FuzzTargetName } from './target-types.ts';
export type CorpusEntry = {
target: FuzzTargetName;
/** The failing input, verbatim. */
input: string;
/** Why it was added — the invariant it broke, or where it came from. */
note: string;
};
// AGENT_DEVICE_FUZZ_CORPUS retargets the corpus file so the harness's own tests can exercise
// the real promotion path against a scratch file instead of mutating the checked-in one.
const CORPUS_PATH =
process.env.AGENT_DEVICE_FUZZ_CORPUS ??
path.join(path.dirname(fileURLToPath(import.meta.url)), 'corpus', 'regressions.json');
export function readCorpus(corpusPath = CORPUS_PATH): CorpusEntry[] {
const raw = fs.readFileSync(corpusPath, 'utf8');
const parsed: unknown = JSON.parse(raw);
if (!Array.isArray(parsed)) throw new Error(`${corpusPath} must contain a JSON array.`);
return parsed.map((entry, index) => readEntry(entry, index, corpusPath));
}
function readEntry(entry: unknown, index: number, corpusPath: string): CorpusEntry {
if (entry === null || typeof entry !== 'object') {
throw new Error(`${corpusPath}[${index}] must be an object.`);
}
const record = entry as Record<string, unknown>;
const { target, input, note } = record;
if (typeof target !== 'string' || typeof input !== 'string' || typeof note !== 'string') {
throw new Error(`${corpusPath}[${index}] needs string target, input, and note fields.`);
}
return { target: target as FuzzTargetName, input, note };
}
function entryKey(entry: CorpusEntry): string {
return `${entry.target}\u0000${entry.input}`;
}
/** Merges `additions` into the corpus file. Returns the entries actually added. */
export function appendToCorpus(
additions: readonly CorpusEntry[],
corpusPath = CORPUS_PATH,
): CorpusEntry[] {
const existing = readCorpus(corpusPath);
const seen = new Set(existing.map(entryKey));
const added = additions.filter((entry) => {
if (seen.has(entryKey(entry))) return false;
seen.add(entryKey(entry));
return true;
});
if (added.length === 0) return [];
const merged = [...existing, ...added].sort((a, b) => entryKey(a).localeCompare(entryKey(b)));
fs.writeFileSync(corpusPath, `${JSON.stringify(merged, null, 2)}\n`);
return added;
}