mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
4acfd31fc9
* fix(audit): coverage caveats printed by lint never reached the audit report `formatUntestedReport` grew two qualifier lines — a test file still carrying the retired `vigiles:covers` marker, and a run record older than the text it measured. `audit` assembles its own fact block from the SAME UntestedReport and carried neither, so the note that exists to explain a silent migration was itself silent for anyone whose habit is `audit` rather than `lint`. Measured before the fix on a fixture whose only harness carries the marker: lint → "1 test file(s) still carry the retired `vigiles:covers` marker …" audit → "Untested surfaces: 0" and nothing else The caveats now come from one exported builder (`coverageCaveats`) that both renderers read, so a third qualifier reaches both callers or neither — a renderer can no longer carry half the list. `ScanReport.coverageCaveats` is absent rather than `[]` when empty, because the browser twin structurally cannot compute these (no `.vigiles/coverage.json` without a filesystem) and byte-parity between the two engines is asserted field-for-field. Both halves tested at the product level (`node dist/cli.js audit`): it fires on a repo carrying the marker and stays silent on one that does not. Proven by two mutations, each with the patched source grepped first: dropping the render line fails the "fires" assertion; emitting a caveat unconditionally fails the "silent" one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the suffix that stopped counting now says so `*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run collects exactly those names and a skill test spawns a model — so a misnamed harness was being executed, and paid for, by somebody else's runner. The removal was announced to nobody. Its own comment argues "the migration is a rename, and the untested finding prints the exact path", but the path printed is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore told to write the test they wrote, with neither the file nor the reason named. The sibling half of the same migration — the retired `vigiles:covers` marker — got its explanatory line at the time. This is the half that did not. Scoped to surfaces that are otherwise UNCOVERED: that is the only position where the count contradicts what the author can see in the directory. A stray `*.test.*` beside a properly covered surface is somebody's ordinary unit test and none of our business — asserted, because an advisory line that fires on the innocent case gets muted within a day. Nothing about coverage changes: the name still grants nothing. Measured silent on all seven real corpora available here (this repo, six vendored dogfood plugins) and on the `mine` knowledge base; it fires on a planted fixture in both `lint` and `audit`, the latter through the shared caveat list added in the previous commit rather than a second wiring. Three mutations, each with the patched source grepped before the run and each dying on its own assertion: dropping the note from the caveat list · widening the scope from untested to every considered surface · dropping the suffix test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * refactor(scan): compute the caveat list once Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was called twice in the same object literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the retired-suffix note claimed a cost the guard already closes The note said a `*.test.*` harness "spawns a model" under someone else's runner. `refuseUnderForeignRunner` guards every spawn door and refuses under vitest/jest — measured here against the detector: argv of `node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model: this process is running under vitest." What actually happens is that the foreign run FAILS on the file, which is the true and smaller claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it Importing a `*.eval.*` file spent real money. These files did their work in the module body, and in ESM `import` IS execution — so the cheapest imaginable question, "does this file even parse?", launched a paid run against a live model when it was asked with `import()`. That happened, and was paid for. The one guard that exists does not close the door and cannot be made to. Under `node -e 'import(f)'` there is no `process.argv[1]` at all, so `foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays silent; and no process fact distinguishes that from a legitimate runner doing the same import, so an argv-shaped guard would have to refuse the correct invocation to catch the wrong one — the exact failure that got `process.env.VITEST` rejected in `core/foreign-runner.ts`. So the shape changed instead. An eval file default-exports a description: export default defineEval({ measureTriggerRate: { …spec… }, assert: (report) => assertTriggerRate(report, { min: 0.8 }), }); `defineEval` builds a plain value; there is no call in it to run. The five paid runners are reached only by `dist/eval-entry.js`, the program `vigiles eval` now spawns per file — not a new verb, the interpreter the existing verb uses. `defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and that is the point: after this change no eval file imports a runner at all. Three loud edges, because a silent no-op would be the same defect in a new place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares nothing (or two measurements) is reported by the runner instead of exiting green; and a half-migrated file — a description plus a leftover top-level `measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and throws a migration message instead of billing. `node --check` still works and is what the free syntax check should be. Subtractions: every eval file loses its `process.env.VIGILES_TRIALS || process.argv[2]` line, its `console.log(format…Report(report))`, its `if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval` import. The runner does all five, once. BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})` instead of calling a runner at the top level, and `node <file>.eval.mjs` no longer runs an eval. To migrate: move the runner call into a key named after that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` · `measureSelectionMatrix` — declare exactly one), move anything you did with the report into `assert(report)`, and delete the trials line, the `format…` print and the `report.n === 0` check — `vigiles eval` does those. A capability probe that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s directory argument becomes a `pluginDir` field; and `measureTriggerRate(spec, { evalDriver })`'s second argument becomes a top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials. Run with `vigiles eval <file>`; a file left on the old shape fails with a message naming docs/harness-testing.md, never silently. All 19 eval files in this repository were migrated in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture and generated types The eval-definition change rewrote one report hint (`paid_measureTriggerRate` → `export default defineEval({ measureTriggerRate: … })`), which is exactly the divergence the parity test is built to catch. Regenerated per that test's own instruction for an intentional engine change; the diff is that one line — grade and every other field are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>