mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Jordan Ritter
edf109b6cb
Across two 12-reviewer rounds on PR #6156 the most effective review technique was mutation testing — deliberately breaking the implementation to check whether the tests notice. Three different agents wrote those mutations by hand, independently, and threw them away each time, while the findings they produced stayed true. This commits them. mutation-gate/mutants.json is a declarative manifest: one entry per mutation, carrying the target file, the exact find/replace, the test scope, the tests that MUST fail, and the defect class the mutation guards. Entries that are KNOWN to slip past their scope declare `expect: "survive"` with a mandatory `knownGap` (reason + what closes it), so the manifest is a ledger of guard strength rather than a list of mutations someone already knew would fail. mutation-gate/mutation-gate.ts applies each mutation in isolation, runs the declared scope, asserts the declared tests fail, reverts, and reports. It fails loudly when a mutation does NOT produce its expected failure — that is the signal a guard has gone toothless — and equally when a declared gap has been closed, because a stale ledger is the same disease one level up. It edits tracked source by design, so: it refuses to run when any tracked file is modified; it never uses `git stash` (the stash stack is shared repo-wide across worktrees) and never commits; it verifies `origin` resolves to CopilotKit/CopilotKit before the first write; originals are backed up inside the worktree's git dir and journalled, so SIGINT/SIGTERM/SIGHUP and uncaught errors revert synchronously and a SIGKILL is recovered from the journal on the next invocation. __tests__/mutation-gate-manifest.test.ts is the cheap half, and rides the showcase/scripts vitest step that already runs in showcase_validate.yml: it asserts the manifest is valid and that every anchor still occurs exactly once at HEAD, which is the failure mode that would otherwise rot silently — a refactor that moves an anchor breaks no test, it just means the guard is no longer measured.