12 Commits

Author	SHA1	Message	Date
zernie	4acfd31fc9	feat(eval)!: an eval file describes, the runner performs — importing one no longer spends (#159) ... * fix(audit): coverage caveats printed by lint never reached the audit report `formatUntestedReport` grew two qualifier lines — a test file still carrying the retired `vigiles:covers` marker, and a run record older than the text it measured. `audit` assembles its own fact block from the SAME UntestedReport and carried neither, so the note that exists to explain a silent migration was itself silent for anyone whose habit is `audit` rather than `lint`. Measured before the fix on a fixture whose only harness carries the marker: lint → "1 test file(s) still carry the retired `vigiles:covers` marker …" audit → "Untested surfaces: 0" and nothing else The caveats now come from one exported builder (`coverageCaveats`) that both renderers read, so a third qualifier reaches both callers or neither — a renderer can no longer carry half the list. `ScanReport.coverageCaveats` is absent rather than `[]` when empty, because the browser twin structurally cannot compute these (no `.vigiles/coverage.json` without a filesystem) and byte-parity between the two engines is asserted field-for-field. Both halves tested at the product level (`node dist/cli.js audit`): it fires on a repo carrying the marker and stays silent on one that does not. Proven by two mutations, each with the patched source grepped first: dropping the render line fails the "fires" assertion; emitting a caveat unconditionally fails the "silent" one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the suffix that stopped counting now says so `*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run collects exactly those names and a skill test spawns a model — so a misnamed harness was being executed, and paid for, by somebody else's runner. The removal was announced to nobody. Its own comment argues "the migration is a rename, and the untested finding prints the exact path", but the path printed is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore told to write the test they wrote, with neither the file nor the reason named. The sibling half of the same migration — the retired `vigiles:covers` marker — got its explanatory line at the time. This is the half that did not. Scoped to surfaces that are otherwise UNCOVERED: that is the only position where the count contradicts what the author can see in the directory. A stray `*.test.*` beside a properly covered surface is somebody's ordinary unit test and none of our business — asserted, because an advisory line that fires on the innocent case gets muted within a day. Nothing about coverage changes: the name still grants nothing. Measured silent on all seven real corpora available here (this repo, six vendored dogfood plugins) and on the `mine` knowledge base; it fires on a planted fixture in both `lint` and `audit`, the latter through the shared caveat list added in the previous commit rather than a second wiring. Three mutations, each with the patched source grepped before the run and each dying on its own assertion: dropping the note from the caveat list · widening the scope from untested to every considered surface · dropping the suffix test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * refactor(scan): compute the caveat list once Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was called twice in the same object literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the retired-suffix note claimed a cost the guard already closes The note said a `*.test.*` harness "spawns a model" under someone else's runner. `refuseUnderForeignRunner` guards every spawn door and refuses under vitest/jest — measured here against the detector: argv of `node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model: this process is running under vitest." What actually happens is that the foreign run FAILS on the file, which is the true and smaller claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it Importing a `*.eval.*` file spent real money. These files did their work in the module body, and in ESM `import` IS execution — so the cheapest imaginable question, "does this file even parse?", launched a paid run against a live model when it was asked with `import()`. That happened, and was paid for. The one guard that exists does not close the door and cannot be made to. Under `node -e 'import(f)'` there is no `process.argv[1]` at all, so `foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays silent; and no process fact distinguishes that from a legitimate runner doing the same import, so an argv-shaped guard would have to refuse the correct invocation to catch the wrong one — the exact failure that got `process.env.VITEST` rejected in `core/foreign-runner.ts`. So the shape changed instead. An eval file default-exports a description: export default defineEval({ measureTriggerRate: { …spec… }, assert: (report) => assertTriggerRate(report, { min: 0.8 }), }); `defineEval` builds a plain value; there is no call in it to run. The five paid runners are reached only by `dist/eval-entry.js`, the program `vigiles eval` now spawns per file — not a new verb, the interpreter the existing verb uses. `defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and that is the point: after this change no eval file imports a runner at all. Three loud edges, because a silent no-op would be the same defect in a new place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares nothing (or two measurements) is reported by the runner instead of exiting green; and a half-migrated file — a description plus a leftover top-level `measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and throws a migration message instead of billing. `node --check` still works and is what the free syntax check should be. Subtractions: every eval file loses its `process.env.VIGILES_TRIALS || process.argv[2]` line, its `console.log(format…Report(report))`, its `if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval` import. The runner does all five, once. BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})` instead of calling a runner at the top level, and `node <file>.eval.mjs` no longer runs an eval. To migrate: move the runner call into a key named after that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` · `measureSelectionMatrix` — declare exactly one), move anything you did with the report into `assert(report)`, and delete the trials line, the `format…` print and the `report.n === 0` check — `vigiles eval` does those. A capability probe that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s directory argument becomes a `pluginDir` field; and `measureTriggerRate(spec, { evalDriver })`'s second argument becomes a top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials. Run with `vigiles eval <file>`; a file left on the old shape fails with a message naming docs/harness-testing.md, never silently. All 19 eval files in this repository were migrated in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture and generated types The eval-definition change rewrote one report hint (`paid_measureTriggerRate` → `export default defineEval({ measureTriggerRate: … })`), which is exactly the divergence the parity test is built to catch. Regenerated per that test's own instruction for an intentional engine change; the diff is that one line — grade and every other field are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-18 20:34:40 +05:00
zernie	9413061d9b	feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149) ... * refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-16 22:46:26 +05:00
zernie	863c1b42ee	docs: два примера импортируют символ не из того сабпаса — оба падают на копипасте (#145) ... Найдено ресёрчем по инструментам doctest и подтверждено вызовом, а не чтением: scriptModel: vigiles/testing → undefined, vigiles/claude-code → function runHookProgram: vigiles/testing → undefined, vigiles/hook → function Оба примера читатель копирует целиком, и оба падают на первой строке. Это не опечатка в прозе, а неработающий код в документации. 🔴 Про `scriptModel` это уже ТРЕТИЙ раз. Тот же неверный импорт дважды записан в потребительской базе знаний (`vigiles/s59.md` §3 и починен в скилле по s61 §4) — с пометкой, что граница намеренная: `scriptModel` живёт в адаптере Claude Code, а не в харнесс-агностичном барреле. Правило было известно, задокументировано, и третий экземпляр всё равно уехал в публичные доки. Ровно тот класс, который проза не удержит: граница верная, документация о ней верная, а конкретная строка импорта — нет. Механическая проверка предложена в `vigiles/ideas/19` (в приватной базе); этот коммит закрывает два известных экземпляра, а не класс. Гейты: npx prettier --check docs/ · npm run lint (0 errors). Правок кода нет. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com>	2026-08-16 14:24:11 +05:00
zernie	3f33925722	feat: skill-description-budget rule + document-the-why + audit/opt-in docs (#54) ... * docs: explain why compiler-grade guarantees are opt-in The "opt-in, like TS's strict" line appeared in three public docs but only asserted the analogy, never the reason. Add a dedicated FAQ entry that gives the actual answer — opt-in means gradual not permissive (the default already gates real breakage), compiler-grade proofs need a typed program to check (tsc can't verify inert markdown), and gradual adoption is the TS-strict bet — and link the README and linting-guide one-liners to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: codify auto-vs-nudge enforcement model in the spec skills Apply the init enforcement model at edit time across the three spec skills: auto-apply the con-free wins, nudge (with the tradeoff) for anything that costs config/plugins/CI-failure, never silently escalate into strict gating. - strengthen: name the principle explicitly (cost, not strictness, is the dividing line; Tier 1 auto, Tiers 2-4 are the user's call) - edit-spec: add "do what's asked, don't silently escalate enforcement" - adopt-spec: replace the prose blockquote with explicit, scannable adoption rules (faithful / non-destructive / don't-escalate / reversible / ask-before-writing / lighter-touch-exists) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: explain what audit runs (read vs measure) + selection-collision Close the audit→behavioral→consent documentation gap exposed while dogfooding a skills-heavy repo: a clean-A audit "found little" because a plain audit is a deterministic READ — the behavioral checks (does each skill fire? do two collide?) run only under the one-time audit.measure consent. - faq.md: new "What does audit actually run — and why did it find nothing?" entry stating the read-vs-run model and the headless fallback - for-plugin-authors.md: turn the bare "selection-collision matrix" name-drop into a real explanation (diagonal = recall, off-diagonal = a sibling hijacked the prompt; body-stubbed, so it measures selection cheaply) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add document-the-why rule + doc-command coverage check The "docs suck" failure repeated twice this session (the opt-in why; the audit→behavioral→consent story), and no existing docs rule owns it: the polish, placement, and sync rules govern how well / where / consistency, but none requires the REASONING behind a user-facing decision be documented at all. - new `document-the-why` guidance rule: every user-facing decision/concept gets a plain-language doc home that answers why/what (name-dropping ≠ documenting); the test is "could a user answer it from the docs alone?" - new deterministic FLOOR (the inverse of self-command-refs): every public CLI verb must be mentioned under docs/ — src/doc-command-coverage.ts, dogfooded in its test (catches a verb shipped undocumented). The concept-coverage half stays judgment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: add skill-description-budget lint rule (warn-tier trigger proxy) A model-invocable skill is selected on its description, and the selector weighs the opening most — a bloated description buries the trigger signal and hurts recall + precision. New deterministic heuristic proxy (generous 500-char budget), the free sibling of description-overlap. Wired end-to-end: detector (src/core/skill-description-budget.ts) + rule-meta (heuristic-behavioral → warn ceiling) + DEFAULT_RULES + NUDGE group (never gates) + scan report + lint application + docs/rules + the rules matrix + tests. NOTE on haretrail: it does NOT fire there — the earlier "634/781-char" figures were BYTE counts; in characters (Cyrillic = 2 bytes/char) the longest skill description is 463 chars, under budget. The rule is correctly calibrated FP-safe; forcing a hit would mean crying wolf. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for haretrail-dogfood session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * test: add skill-description-budget to validate.test DEFAULT_RULES literal The new rule landed in DEFAULT_RULES but validate.test.ts asserts the full rules object against a hardcoded literal — add the key so the deep-equal passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: prettier-format HANDOFF.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * refactor: remove the cross-language description scan flag (refuted) The deterministic "non-Latin description → cross-language trigger risk" finding (unexpectedScript / descriptionScript in scan.ts) is removed: the Script type, scriptCounts, unexpectedScript, the ScanSkill.descriptionScript field, its report line + summary, its tests, the docs mention, and the config-expected-script notion. Why: it fires on 100% of non-Latin descriptions, but a non-English description is a legitimate audience choice, not a defect — so it scolds correct work. And the one time we measured it (plugin-behavioral-findings.md Finding 3a, native codex), the risk was REFUTED — the model handles RU-description/EN-prompt selection fine. A warn that reads as a defect on correct authoring cries wolf (lint-rule-calibration). Reasoning saved in research/plugin-behavioral-findings.md; the one-detector-no-drift teaching example reassigned to description-overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: eval cost transparency — surface tokens, API-equivalent $, metered warning Make what a real-model run SPENT impossible to miss (vigiles's affordability pitch is "runs on your subscription, not a metered API" — so every run should say so). - new src/eval-cost.ts: the pure engine — cost snapshots (per-run / per-arm / whole-report), metered-vs-subscription detection (ANTHROPIC_API_KEY set → metered), a session tally, and the formatted block (tokens + API-equivalent $, a LOUD warning + actionable fix when metered, ✅ when on the sub). Fully unit-tested, no model. - runEval now emits the cost block after a run (stderr; report carries usage). - test-harness skill: MUST relay the spend to the user after any eval it runs. - measuring-skills docs: "what a run reports + the metered-API warning" section. HONEST SCOPE: no "% of your subscription" — Anthropic doesn't expose plan quota, so a percentage would be invented; tokens + API-equivalent $ + billed-to is the complete honest picture. FOLLOW-UP: measure/measureArms/measureTriggerRate reports don't carry usage yet (only EvalReport does) — threading a usage field through those model-gated run loops + the eval-lock is the next commit so the skill's trigger-rate path emits too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * fix: brace void-returning arrows in eval-cost (eslint no-confusing-void-expression) Missed running eslint on the new file (only ran fmt:check). The console.error sink and the test beforeEach arrows returned a void expression in shorthand form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add surface-architecture-decisions rule Surface non-trivial architectural decisions (file placement, core vs adapter vs port, extend-vs-create, cross-cutting changes) to chat in a scannable block with the boundary reasoning — the structure analog of subagent-model-note — so the user can catch a wrong placement before the diff, given the hexagonal + multi-adapter constraints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: thread usage into TriggerRateReport so measureTriggerRate emits cost The skill's main path (measureTriggerRate) now surfaces spend too. Per-run usage comes from the parser (makeContext(out, cfg.parse).usage) — harness-neutral, since both the Claude and Codex parsers fill ParsedModelRun.usage (Codex reports $0 on the sub, real tokens). runTriggerTrial returns its run's usage; runTriggerSet collects them; the report aggregates via the existing aggregateUsage (relevant + irrelevant sets both counted); the wrapper emits costFromArm(report.usage). No new port/driver method (the parser already provides usage) and eval-lock-safe (additive field; the lock hashes inputs, not the report). FOLLOW-UP: measure/measureArms (CheckReport) still don't carry usage — same threading applies, next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: fold parse-don't-validate + illegal-states-irrepresentable into surface-architecture-decisions Frame them from this rule's angle (surfacing): a parse-boundary and an illegal-state-forbidding type ARE architectural decisions — name them when you make them (where the boundary sits; what invalid state the type forbids). Cross-refs ts-essentials for the mechanical how, rather than duplicating it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: update vigiles-testing API surface for TriggerRateReport.usage api-extractor tracks the public surface; adding the usage field is intended (additive — consumers read the report, don't construct it). npm run build doesn't run api-extractor, so this was missed locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for the eval-cost / rules session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: capture haretrail-driven audit + eval ideas (research note) Save the session's un-persisted ideas before context is lost: new deterministic cross-ref checks (openai.yaml / trigger-manifest / env-var contract) and new eval helpers (the selection-collision confusion matrix as a first-class primitive, manifest-driven fixtures + auto-negatives, description-ablation). Indexed in research/README.md (Skills) + cross-linked to plugin-behavioral-findings / measurement-authority / audit-wow-ideas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * style: prettier-format the haretrail-eval-ideas note + README Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add status/topic frontmatter + agent-facing index to research corpus Migrate all 114 research/*.md docs to carry `status:` (active/shipped/idea/ superseded/rejected) and `topic:` frontmatter, so the corpus is sliceable deterministically (`grep -l 'status: shipped' research/*.md`). Add research/CLAUDE.md.spec.ts — a scoped claude() spec whose keyFiles map every research doc to a one-line, status-tagged summary (grouped by topic). Compiles to research/CLAUDE.md, the agent-facing index a session loads as directory memory for research/ without opening 114 files. Mirrors src/CLAUDE.md.spec.ts (the code map); this is the research map. Bidirectional sync is enforced, not by hand: - the compiler verifies every keyFiles path EXISTS (a deleted/renamed doc fails `vigiles compile`) - src/research-index.ts + its dogfood test assert the reverse — every research/*.md (except README.md) appears in the index research/README.md stays the human, thematic index and now points at the machine index + the frontmatter query convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for research corpus index work * feat: thread eval cost into measure/measureArms CheckReport The scored-check paths (`measure`, `measureArms`) were the last eval entry points that didn't report spend — only `runEval` and `measureTriggerRate` did. Close the gap the same way: add `usage: ArmUsage` to `CheckReport` (sourced from the arm's already-aggregated usage), and emit the cost summary in the v8-ignored real-CLI wrappers. `measureArms` sums both arms (an A/B run pays for both). So every real-model eval path now prints tokens + API-equivalent $ + the metered-API warning. Wiring tests drive an injected runner (no model) and assert usage aggregates across trials and per arm. API surface + docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: promote selection-collision to an assertable primitive (measureSelectionMatrix) The cross-skill selection-collision matrix — "when I ask for skill i's job, does ONLY skill i fire?" — already existed but was reachable only from the CLI under the audit consent. Promote it to a first-class, assertable eval primitive so a plugin author can gate collisions in a test: - `measureSelectionMatrix(dir, opts)` — zero-setup: prompts auto-derived from each skill's description (or supply your own). Thin wrapper over the existing `measurePluginSelection`; `measureSelectionMatrixWith` is the injectable core. - `assertNoCollision(report, { maxOffDiagonal, maxPluginCollision })` — the throwing gate. Defaults to demanding ZERO collision; throws on a green-that-tested-nothing (unavailable / zero runs), never a silent pass. Both are exported on `vigiles/claude-code`, NOT the agnostic `vigiles/testing`: selection-collision reads which skill the selector chose, which only Claude Code surfaces (Codex has no skill-selection event), and `scan-behavioral` is Codex-adapter-coupled — so the CC surface is the correct home. Adds a canonical example eval, unit tests (auto-derive + all assert branches, no model), the plugin-author doc's CI path, and API/keyFiles updates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — cost threading + selection-matrix landed * docs: tidy HANDOFF formatting + refresh stale gotchas * docs: mark selection-collision as Claude-Code-only, consistently Make the CC-only surface unmistakable across the docs, one convention: - testing-api.md: a dedicated section "Selection-collision matrix (Claude Code only)" — the "(Claude Code only)" heading + a 🔵 callout naming the `vigiles/claude-code` import — plus a Contents entry and an Imports-section bullet cataloguing which measurements are CC-only vs agnostic. - harness-testing-claude-code.md: a pointer from the Tier-3 trigger section (a CC user's natural landing spot for a CC-only measurement). Complements the existing inline note in for-plugin-authors.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-01 15:49:43 +05:00
zernie	15c0607a5b	feat!: eval lock (CI staleness gate) + full Codex parity for hooks & inject (#53) ... * feat!: eval lock (CI staleness gate) + per-adapter harness versionKey Real-model evals run on your subscription, so they only run locally — never in CI. The eval LOCK closes the resulting gap: a committed integrity stamp that lets CI verify the committed numbers still match the current inputs WITHOUT a model call (distinct from the gitignored, local-speed eval cache). - `vigiles eval --update` (local): records each NAMED eval's report to a committed `.vigiles/eval-locks/<name>.lock.json` + prints the per-number delta. - `vigiles eval --check` (CI): recompute the input hash, fail "stale" on a mismatch — binary-free, deterministic. Green no-op until the first lock is committed (smooth adoption). - The clean split: the lock stores only the model's observed behavior; the script's own assertions re-run live against the replayed report, so a threshold-only edit is a valid replay while an input change is stale. - Honest scope: the harness version is provenance, NOT a hash input (CI's pinned claude ≠ a dev's local claude would false-trip), and there is no nightly run — model/harness drift is a local `--update` concern. Also extracts the harness version reduction to a per-adapter port method (`HarnessRuntime.versionKey`): Claude Code keys major.minor (~quarterly), Codex opts out with "" (its minor is patch-cadence, ~weekly). This fixes the local cache churning weekly for Codex and is where the cadence decision belongs. Whole-package coherence: the GHA gains `command: eval-check`, `vigiles init` scaffolds the eval-check CI job, and this repo dogfoods it. Wired at both eval seams (runEvalWith + measureTriggerRateWith), 100%-covered, docs updated (internal research + public guides). BREAKING CHANGE: `HarnessRuntime` (exported on `vigiles/adapter`) gains a required `versionKey(raw: string): string` method. Third-party adapters must add it; return "" to opt out of version partitioning. First-party adapters (claude-code, codex) are updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * feat: eval-lock agent awareness (skill + nudge hook) + cohesive-delivery rule Close the loop on the eval lock so the AGENT knows it exists and maintains it — without the invasive option of editing the user's CLAUDE.md. - test-harness skill documents the lock lifecycle (--update/--check, commit the lock, re-run after editing a skill). - A PostToolUse nudge hook (hooks/eval-lock-nudge.sh → hook-runtime eval-lock-nudge): after editing a SKILL.md / *.eval.* with committed locks present, inject a NON-BLOCKING reminder to re-run `vigiles eval --update`. Self-gated (silent until a lock is committed), never blocks (the gate is CI's `eval --check`). Wired into the vigiles plugin; runHook + unit + .sh tests. - NO init edit to the user's CLAUDE.md — verified that's not how plugins deliver context (no plugin-level instructions file exists; the channel is a hook/skill). Docs + dedup: - research/agent-context-delivery.md captures the VERIFIED CC/Codex context-delivery mechanics ONCE (SessionStart additionalContext is the plugin channel; CLAUDE.md is the user's file; skill descriptions always-in-context; Codex AGENTS.md + hooks but inject deferred) so we stop re-researching it. - Per-adapter guides (harness-testing-claude-code/-codex.md) explain how the reminder reaches the agent per harness (CC injects; Codex inject is a documented follow-on), with the agnostic guide linking DOWN to them (no duplication). Cohesive-view rule: - New CLAUDE.md `cohesive-feature-delivery` rule — a definition-of-done CHECKLIST (CLI × GHA × init × agent-awareness × both adapters × per-adapter deduped docs × tests) so a feature's INTEGRATION across surfaces is checked as one thing, generalizing cohesive-cli-surface + harness-parity + great-agent-flow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * test: CLI e2e for the eval-lock invalidation flow (update → check → stale) Closes the gap flagged in review: the joined CLI flow wasn't tested as one thing. A real *.eval.cjs fixture (model-free — drives runEvalWith from dist with a fake runner) run through the built CLI proves the whole loop: `eval --update` writes a committed lock, `eval --check` with the same inputs replays green (no model), and an input change makes `eval --check` go red ("stale"). Binary-free, no subscription — exactly how it runs in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * docs: scannable eval-lock docs + prose-clarity & easy-adoption rules Make the eval-lock docs readable at a glance and frame upkeep as agent-driven, not a manual chore. Docs (scannable rewrite — ASCII flow diagram, comparison tables, ✅/❌/⚠️ signposts, short sentences, no walls of text): - docs/harness-testing.md — the lock how-to now leads with "you mostly won't touch this by hand; the test-harness skill records it + a hook reminds you," then a diagram + tables. - docs/harness-testing-claude-code.md / -codex.md — the per-harness nudge delivery as tables; the Codex inject caveat is one clear ⚠️ row. - docs/eval-architecture.md — the "what's stale vs a free replay" split as ✅/❌ bullets. - skills/test-harness/SKILL.md — Step 5 now tells the agent to run `eval --update` + commit the lock as part of its job (so it IS done for you). - README — the Quick-start reassurance leads with "Adoption is smooth: one command, then your agent does the rest," names skills AND hooks, no chores. Two new CLAUDE.md rules: - prose-clarity — public prose must be clear + scannable (lead with the point, one idea per sentence, define/drop jargon, tables/bullets/ASCII/Mermaid over walls, emoji as signposts, one bold per point). Prefer ASCII/Mermaid; never commit generated raster images into docs. - lead-with-easy-adoption — docs (README first screen especially) must make it clear up front that adoption is low-friction because skills + hooks do the work; frame every feature's upkeep as agent/skill/hook-driven, not homework. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * docs: prose-clarity sweep across 13 user-facing guides Apply the new prose-clarity rule to the prose-heavy public guides: lead with the point, one idea per sentence, tables/bullets/ASCII over walls of text, ✅/❌/⚠️/ℹ️ signposts, one bold phrase per point, jargon glossed or dropped. Net −578 lines — shorter and far more scannable, with meaning unchanged. Swept: for-plugin-authors, what-vigiles-catches, measuring-skills, skills, harnesses, comparison, faq, sandboxing, safety, agent-setup, agent-workflows, verifying-instruction-files, compiled-hooks. The onboarding pair leads with the low-friction "one command, the agent does the rest" framing. Reference docs (cli, testing-api, spec-format, …) and the per-rule docs were deliberately left alone — they're already table-shaped, and churning them is high-risk / low-value. Verified: no heading text changed (anchors intact), no `research/` links added to public docs, the validation-rules matrix is complete (28 rule links → 28), safety claims retained, command refs resolve (self-command-refs), no orphan docs, fmt clean. Produced by parallel rewrite agents under strict preserve-anchors/links/ code constraints, then verified + spot-read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * feat!: encode hook inject support into HookProtocol.injectableEvents Codex inject was documented as "deferred" but the official Codex hooks docs confirm `additionalContext` is honored on SessionStart / UserPromptSubmit / PreToolUse / PostToolUse / SubagentStart — the same shape as Claude Code. The gap existed because inject support was a prose deferral, not a tested contract: nothing in the adapter conformance kit asserted a shellHooks harness could actually deliver an inject hook. Encode it as a required port field so the gap can't recur silently: - HookProtocol gains `injectableEvents: readonly string[]` — the events that honor `additionalContext` injection, per-adapter. - claudeCodeHookProtocol: [SessionStart, UserPromptSubmit, PostToolUse]; codexHookProtocol: + [PreToolUse, SubagentStart]. - Conformance rejects a shellHooks adapter with an empty list; the adapter-contract suite asserts every shellHooks adapter declares the events vigiles' shipped hooks use (PostToolUse, SessionStart) — the test that would have caught the original miss. - `compile` warns on an inject hook only when its event isn't injectable on the target harness (react output stays CC-confirmed-only). - Correct every "Codex inject deferred" reference across docs, research, and CLAUDE.md; document injectableEvents in the adapter API + authoring guides. BREAKING CHANGE: HookProtocol (exported on vigiles/adapter) gains a required `injectableEvents` field — a third-party adapter must declare it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for the eval-lock branch + inject-port encoding Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: fan out hook install + the CI workflow to every declared harness A repo can enable BOTH harnesses (`harness: ["claude-code","codex"]`), and instruction files already mirror to both — but `compile` installed a hook into only ONE harness (`installHooks` ignored `config.harness` and auto-detected a single adapter), and the scaffolded CI workflow hardcoded the Claude Code binary. So an inject/gate hook that works on both could land in `.claude/settings.json` while silently missing from `.codex/config.toml` — the "works on CC, absent on Codex" failure the harness-parity rule exists to prevent. - `resolveHarnessAdapters` (adapter-registry.ts) returns the FULL declared set: `--harness=` flag → that one; else config `harness` list → all of them; else auto-detect → the detected one (de-duped by name). - `installHooks` loops over it, merging the SAME compiled hook into every enabled harness's native config (JSON / TOML), with per-harness warnings. - The `init` CI workflow is harness-aware: the deterministic harness job installs `@anthropic-ai/claude-code` and/or `@openai/codex` per declared harness (`harnessTestBinaries`). Skills compile byte-identical across harnesses and subagents are CC-only by design, so hooks + the CI binary were the real fan-out gaps. Tests: resolveHarnessAdapters units (fan-out / flag-singular / detect / alias-dedup), a both-harness hook-install e2e (the same hook in both configs), and harness-aware workflow e2es (codex-only, both). Documented as §4 in research/multi-harness-compile.md + the compiled-hooks guide. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: wire vigiles's nudge hooks into .codex/config.toml during init Closes the last Codex-parity gap in agent-awareness: Claude Code gets the eval-lock + refs nudge hooks from the global marketplace plugin, but Codex has no global plugin store, so it got skills only — the proactive nudges never reached a Codex user (the "works on CC, silently absent on Codex" failure). Codex config is repo-committed (the idiomatic place), so `init` now wires the two PostToolUse nudges into `.codex/config.toml`: - `codexPluginHooks` (pure) declares them as DIRECT `npx vigiles hook-runtime …` commands — no plugin root, no vendored bash script. The runtime entrypoints read the event JSON on stdin and emit the `additionalContext` shape Codex honors on PostToolUse (confirmed + encoded in HookProtocol.injectableEvents). - `applyCodexPluginHooks` (pure, unit-tested) merges them idempotently: re-runs replace in place, and the user's own Codex hooks + every other config key are preserved. `wireCodexHooks` in cli.ts does only the IO. - Safety: only an INTENTIONAL exit 2 blocks an edit (the refs nudge under `unmarked-refs: error`); an npx-resolution failure exits non-2, so a missing dep never blocks. Loud, documented deferral (no-silent-skips): the SessionStart lint summary (plain stdout, unconfirmed on Codex) and the compile-on-edit / pre-edit guards (filename-gated bash, no harness-neutral entrypoint yet) stay manual on Codex. Also: the remaining pre-release Codex item — running the eval tier against the real `codex` binary once — is added to the launch-readiness P0 list in the roadmap. Tests: codexPluginHooks shape + applyCodexPluginHooks (fresh / idempotent / preserves user config); the planPluginInstall codex notes/vendors updated. Verified end-to-end that `init --harness=codex` writes the config and a pre-existing `.codex/config.toml` (model + a user hook) is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: update codex install e2e for the new vendors:true (repo hooks) The codex install plan now writes nudge hooks into .codex/config.toml, so `vendors` flipped to true. Updated the unit assertion last commit but missed this e2e (run by both the `e2e` and `test` CI jobs). The skills-are-global checks below are unchanged and still stand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: cover the non-Actions lock message branch + address Codex review The `test` job's coverage gate failed: `emitLockMessage`'s console.warn/ console.log branches (the non-GITHUB_ACTIONS path) are unreachable in CI, where GITHUB_ACTIONS is globally set. Added a wiring test that DELETES the var to exercise both branches regardless of environment. Also addresses two Codex review findings on the eval-lock work: - P1 (soundness): `evalArmsInputs` omitted `spec.stubs` from the hash, though tool stubs are written onto PATH and are model-facing input — so a changed canned CLI output after `--update` left the same `inputsHash` and `--check` replayed stale results. Now folds a canonical (name-sorted) view of the stubs into the hash; a new wiring test proves a stub change → stale. - P2 (collision): two eval names normalizing to the same slug (`foo/bar` vs `foo bar` → `foo-bar.lock.json`) could let `--check` replay the WRONG eval's report. `readLock` now rejects a lock whose stored `name` differs from the requested one, degrading a collision to a safe miss (→ stale/re-run), never wrong data. Unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: two more eval-lock soundness gaps from Codex review - An unnamed eval in `--check` mode called `produce()` (ran the model), breaking the no-model-in-CI contract: in CI there's no subscription, so it would hit the model or fail for missing auth. Now `check` mode throws a clear, actionable error for an unnamed eval (never reaching the model); `off`/`update` keep producing (the eval just isn't gated locally). - `spec.ephemeralEnv` toggles the trial's run environment (scrubbed env + throwaway HOME vs the inherited process env) — a model-facing input that was omitted from the lock hash, so a report recorded under one mode could be replayed under the other with the same inputsHash. Now folded into the hashed inputs. Both proven by new wiring tests (unnamed-check throws without calling the runner; an ephemeralEnv toggle → stale). Coverage holds at 100% lines/ statements/functions (verified with GITHUB_ACTIONS=true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fold the harness identity into the trigger-rate lock hash Codex review (P2): a trigger-rate eval recorded with one harness driver and later switched via `measureTriggerRate(spec, { evalDriver: codexEvalDriver })` hashed identically (prompts/model/plugin unchanged), so `--check` could replay a Claude-recorded report for a Codex run — defeating the gate for the Codex eval path this PR adds. `EvalDriver` gains an optional `harness` identity (defaults to "claude-code", so existing single-harness locks are unaffected); `claudeEvalDriver` and `codexEvalDriver` declare theirs, and `measureTriggerRate`/`...With` thread it into the hashed trigger inputs. A new wiring test proves a harness switch → stale. API surface regenerated (additive `harness?` on EvalDriver). This closes the input-hash bug class on the trigger-rate seam (the runEval seam's stubs + ephemeralEnv were fixed in the prior two commits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — PR #53 merge loop + the 5 review fixes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: three more Codex-review findings (matchers + plugin-root path hash) - Codex nudge hooks used Claude tool names (`Edit|Write`) in their matcher, but Codex's edit tool is `apply_patch` — so the eval-lock + refs nudges wired into `.codex/config.toml` would never fire. Matcher → `^apply_patch$`. - The compiled-hook FAN-OUT to Codex regex-wraps the raw matcher without translating tool names, so a CC-authored `tools("Edit","Write")` gate emits `^(Edit|Write)$` into Codex config and silently never fires though compile reports success. vigiles has no cross-dialect tool-name map yet, so install now WARNS LOUDLY when a tool-matched hook is fanned to a non-CC harness (no silent non-firing success); a real mapping is the follow-up. - For eval arms using `plugin`, `resolveHarness` expands `${PLUGIN_ROOT}` to the checkout's ABSOLUTE path before the settings are hashed, so a lock recorded at `/home/dev/...` was falsely STALE when `--check` recomputed it at `/home/runner/...` in CI. `stripPluginRoot` normalizes the prefix back to a token → location-independent hash. New wiring test: the same plugin at two different abs paths replays (not stale). Coverage holds at 100% lines/statements/functions (verified GITHUB_ACTIONS=true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: guard stripPluginRoot against a hookless plugin's undefined settings A plugin that ships only instructions/skills (no hooks) resolves to settings:undefined; JSON.stringify(undefined) is undefined, so the prior .split() threw before the eval could run. Guard the non-string case (pass it through). Regression I introduced in the plugin-root normalization last commit; new test covers a hookless-plugin arm (update + check, no throw). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: add a lock-hash COMPLETENESS guard for both seams The Codex review kept finding the same class of bug: a model-facing input omitted from the lock hash, so `--check` replays stale results (stubs, ephemeralEnv, driver-harness, plugin-root all slipped through). Our tests proved the MECHANISM (record→replay, task-change→stale) but never asserted the input set was COMPLETE. This adds a structural guard: for each input that steers the run, mutate ONLY it and assert the lock inputsHash CHANGES — one parametrized test per seam (runEval's evalArmsInputs + trigger-rate's triggerInputs). A new input added to EvalSpec/TriggerRateSpec without hashing it now fails a row HERE in CI, instead of a reviewer finding it later. Covers task/model/tools/fixture/ arm.files/arm.settings/arm.model/interceptTools/stubs/ephemeralEnv and prompts/irrelevantPrompts/model/harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-01 01:55:49 +05:00
zernie	1767bd3990	feat!: audit Safety ring + seven deterministic harness checks, README to 5/5 ... Adds a Safety ring + seven deterministic, FP-safe harness checks to `vigiles audit` (lethal-trifecta, delegation-trifecta, skill-resource-resolves, skill-missing-fence, plugin-dir-layout, hook-block-ineffective, hook-matcher), each wired into both audit and lint from one shared detector. Declares every rule's decidability bucket in a RuleMeta registry, documents the enforcement model, and lands the README at 5/5. BREAKING CHANGE: the HarnessDialect field `blockingHookEvents` is renamed to `noEffectHookEvents` (plus a new `permissionDecisionHookEvents`), and the public surface gains the RuleMeta registry exports. Adapters that referenced `blockingHookEvents` must use `noEffectHookEvents`.	2026-06-30 14:17:42 +05:00
zernie	e8fa3b6b78	refactor!: pre-release polish — surface freeze, spec-first defaults, auto-adopt (#48) ... * chore: refresh HANDOFF — PR #47 merged, surface-freeze still open Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: freeze pre-release public surface + collapse markdown ladder Pre-release surface freeze (research/pre-release-focus.md): - Mark the experimental typed-composition cluster @internal in vigiles/spec (pipe/pipeStep/needs/start/andThen + Supplies/Handoff/KnownAgentName/ Pipeline/PipeStep/NeedsContract) and effect()/EffectRegion. They stay exported at runtime (no removal) but are excluded from the published API docs and flagged unstable in the committed surface reports, so a later change to them is not a breaking change to a documented-stable API. - Note evolve.ts as @internal/research-stage (already off every public entry). - Markdown cut: drop the "Level 0/1/2" ladder framing across public docs; collapse to two on-ramps (plain markdown via inline enforce comments -> typed spec). Frontmatter mode is KEPT, still works, and its full docs + the old ladder are PARKED via an HTML comment in docs/markdown-mode.md (matching the README's parked-section convention) — nothing deleted, trivially un-parked post-launch. - Ship STABILITY.md (0.x: CLI stable; library API evolving; experimental surfaces marked) + a one-line README pointer. - Regenerate etc/*.api.md to reflect the @internal tags (api:check green). BREAKING CHANGE: public symbols in vigiles/spec are re-tagged @internal (typed-composition combinators, effect()). They remain importable but are no longer part of the frozen, supported surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: surface the author-time-vs-runtime wedge + citeable authorities - README hero: state the differentiator positively — vigiles proves the harness "before the agent ever runs" (author-time / pre-run), ending on verify/test/measure (no negative opener, per the README direction rules). - pre-release-focus.md Positioning lock: fold in the canonical one-line wedge (author-time vs runtime, public-safe phrasing, no competitor names) and the citeable harness-engineering authorities (Karpathy "automate what you can verify"; OpenAI "the harness is hard"; arXiv AHE structure-beats- prose = enforce > guidance) for the launch article. The AHE prediction-loop note already lives on the roadmap A2 optimizer item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the README hero with the concrete pain (untested harness) Reframe the hero to go straight to the developer pain: you installed plugins and wrote skills but don't know if they work — a library with no tests. Replaces the "100x coder / 1x verifier" lead (kept as a backup hook in the positioning lock) with the concrete plugins/skills situation + silent-fail triad, ending on the verify/test/measure benefit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the guide openings with the concrete pain, not the mechanics Apply the README's pain-first hook to the subdoc openings: - verifying-instruction-files.md: open with the CLAUDE.md-drifts pain instead of "vigiles's linting layer ... checks every reference". - for-plugin-authors.md: add the installer pain hook (do your skills fire for the people who install it?) above the audience-routing intro. - harness-testing.md: sharpen the lead to the concrete "does it actually block? does your skill fire?" question. Each keeps its README uplink + "what this doc is" line (docs-quality). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: codify the pain-first principle + fix a stale doc link - README direction comment: add rule 1c — lead with the concrete pain in the reader's own situation (the specific silent failure + the "library with no tests" anchor), for the hero AND every subdoc opening; reconcile with 1b (a pain about the reader's situation is a hook, not an apology). - Fix a stale link: src/vendor.test.ts -> src/adapters/claude-code/vendor.test.ts (moved in the adapter relayout). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — surface freeze + launch polish shipped on branch Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs(positioning): not a linter — spec-first + eject, don't fight on structure-lint Lock the decision (Positioning lock + roadmap launch-readiness): vigiles is NOT a markdown/structure linter and won't contest that crowded, commoditized lane. The bold differentiated play is author-time typed specs the agent writes + evals you can afford. Markdown demotes from on-ramp to floor / eject-target. Names the enabling build — a one-command `vigiles eject` (strip the integrity header -> clean owned markdown) so "always ejectable" is true and the two spec objections (TypeScript friction, lock-in) dissolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: add `vigiles eject` + reframe spec-first (managed, but ejectable) The escape hatch that makes "adopt a typed spec, you can always eject" true, and the positioning shift that leans on it. - `vigiles eject [file]`: the inverse of compile. Strips the vigiles:sha256 integrity header -> plain hand-owned markdown, removes the managing spec (--keep-spec to keep it), and adds a `vigiles-disable require-spec` marker so lint stays quiet. Pure helpers (parseIntegrityHeader/ejectMarkdown in core/integrity.ts) + handler + dispatch + help + docs/cli.md. New verb in cli-commands.ts (additive). Unit + e2e tests. - Positioning reframe: don't contest the structure-lint crown. README hero sub-line + spec mentions lead with "your agent writes the spec, and you can eject anytime"; markdown demoted to floor/eject-target (still the zero-TS on-ramp). README direction rule 1c codifies the pain-first hook. - adopt-spec skill: faithful-by-default + eject note; dropped the Level 0/1 ladder pitch; fixed stale commands (check->lint, generate-schema->generate schema, Action @main command:check -> @v1 command:lint). - Closed the scan gap: skills/ now in the self-command-refs dogfood, which caught two rotted `vigiles generate-types` refs (now `generate types`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(scan): nudge toward the real-model trigger tier; agent-safe by default When a plain `scan` finds model-invocable skills AND a model is reachable (ANTHROPIC_API_KEY, or an authenticated Claude Code session via CLAUDECODE / CLAUDE_CODE_ENTRYPOINT), surface the real-model `--trigger` tier that measures whether those skills actually FIRE: - A human at a TTY is OFFERED setup (scaffold a trigger-prompts.json, or run an existing one — the real-model run is always an explicit yes). - An agent / CI run (non-TTY / --json / --no-interactive / --yes) gets a one-line, non-blocking hint and never a prompt (great-agent-flow). Pure core (hasModelAccess / decideTriggerSuggestion / formatTriggerHint / scaffoldTriggerPrompts) + unit tests; CLI wiring + env-gated e2e in scan-cli.test.ts; docs in docs/cli.md. Also records (HANDOFF) that real-model tiers ARE runnable in Claude Code web: the claude CLI is authenticated via session OAuth (no API key), proven by `claude -p` and a live `scan --trigger` run (test-harness recall 100%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: --strict gates broken surfaces (CI red); fix `lint <dir>` crash The actual "edit a skill/subagent/hook badly → CI goes red" payoff. - `vigiles init --strict` now promotes the HIGH-PRECISION, FP-safe structural rules to `error` (not just require-spec): subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap. So a typo'd tool, a dead hook script, a misconfigured MCP server, or two colliding skill descriptions FAIL `vigiles lint` (exit 2). Never clobbers a user-set severity; leaves the lenient/recommendation rules (frontmatter-valid, skill-frontmatter, prefer-compiled-hooks, untested-*) at warn. Honest limit documented: CC itself loads a name-less / broken-YAML skill, so vigiles gates the subagent/hook/MCP defects + skill collisions that genuinely break, not skill content that still works. Proven: clean subagent exit 0, typo'd tool exit 2. - FIX: `vigiles lint .` (a directory arg) crashed with EISDIR (it read `.` as a file). findInstructionFiles now expands a directory arg to the instruction files inside it; a file arg and a non-existent arg pass through unchanged. Tests: setup-plan (strict gate set), cli (lint-dir no-crash + strict exit-2 regression). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: init gates broken surfaces BY DEFAULT (no --strict needed) Fold the structural gating into a plain `init` — no confusing "strict mode" to remember. A default `init` now writes the 9 HIGH-PRECISION, FP-safe structural rules as `error` (subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap), so a broken subagent / hook / MCP ref or a skill collision FAILS `vigiles lint` out of the box — while a well-formed plugin stays green (never cries wolf). `--strict` is reframed to the WORKFLOW-FORCING tier a clean repo can still fail because the work isn't done yet: require-spec, untested-skill/subagent/hook, frontmatter-valid, skill-frontmatter. Opt-in by design (progressive adoption), so a first CI run isn't red just for not having written a spec yet. Never clobbers a severity the user already set. vigiles dogfoods it — the 9 gate rules added to its own .vigilesrc.json (self-lint stays green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(init): interactive init offers the workflow tier (recommended, opt-out) Per the chosen default model: structural gating is always on; the WORKFLOW tier (require specs + a test per surface) is the recommended default a human opts OUT of, never forced on a silent run. - Interactive `init` now asks "Also enforce specs + a test per surface (recommended)? [Y/n]" (default yes) → sets strict. SetupAnswers/applyAnswers carry it; resolvePlan honors it. - A bare NON-interactive `init` (agent/CI, no one to ask) stays structural-only unless `--strict` — so an automated setup never silently turns an existing repo's CI red just for not having written a spec yet. - README agentic-install prompt + docs/cli.md updated: the agent asks before enforcing the workflow tier, structural gating is the safe default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — spec-first defaults + eject + scan nudge; PR is the next step Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * test(init): make the interactive setup Q&A unit-testable (inject ask) The interactive init prompts were the one untested-and-fragile bit of the setup flow (readline can't run in CI's non-TTY env, so the questions/defaults/ answer-mapping had zero coverage). Extract the Q&A into a pure collectSetupAnswers(ask) with an injected AskFn; promptSetup is now just the readline IO shell. Unit-tested with a fake ask: all-defaults, lint/test pillar selection, and declining CI/plugin/strict — deterministic, no TTY, no model. Now the init flow is covered end to end: pure plan logic + the interactive Q&A (unit) + non-interactive init (e2e in cli.test.ts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: install-enforcement model — rule groups (not presets) + rationale Capture the reasoning, not just the outcome, before implementing. - New `install-enforcement-model` rule in CLAUDE.md (via the spec): rules are organized into GROUPS by confidence (structural / workflow / nudge), NOT a relaxed/standard/strict preset MENU — the linter best practice (Clippy/Ruff/ Biome; "presets are a poor emulation of rule groups"). default = structural at error; --strict adds workflow; --report-only is an orthogonal severity dial. Installation is EXPLICIT ABOUT TRADEOFFS (one source reused across the CLI prompt / agent AskUserQuestion / summary). - Reconcile smooth-adoption + progressive-adoption to the new stance: "permissive = doesn't force specs/TS + doesn't cry wolf, NOT ignores breakage." - research/install-enforcement-dx.md: the full design record — the linter landscape (verified), why we rejected a preset menu, the group→rule mapping, the agentic-install DX (no prior art), and the philosophy reconciliation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: rename require-spec → require-instructions-spec; rule groups; init auto-adopt Rename + narrow the rule, name the install rule-groups, and make `init` faithfully adopt existing instruction files so the rule is satisfied by construction. - require-spec → require-instructions-spec, NARROWED: only a `.spec.ts` satisfies it — inline `vigiles:enforce` / `vigiles:` frontmatter no longer do. Disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. - Name the rule groups in src/setup-plan.ts (STRUCTURAL_RULES / WORKFLOW_RULES / NUDGE_RULES) and add `--report-only` (writes the whole gate at warn). - prefer-compiled-hooks defaults off; require-skill-spec un-deprecated (the consistent require-<surface>-spec parallel, default off). - New src/core/adopt.ts: adoptMarkdown/adoptToSpec faithfully convert an existing CLAUDE.md/AGENTS.md into a claude() spec (heading → verbatim section, no rule inferred, always compiles). Wired into init()/setupPillar1; round-trip tested (byte-identical below the integrity header). - Sweep stale Level-0/1/2 markdown-mode comments; docs + CLAUDE.md recompiled. BREAKING CHANGE: the `require-spec` lint rule and config key are renamed to `require-instructions-spec` and narrowed — only a `.spec.ts` satisfies it (inline comments / `vigiles:` frontmatter no longer do), and the per-file disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. No automatic alias; update `.vigilesrc.json` and any disable markers. * fix(init,eject): honor interactive strict; don't delete a shared spec on eject Addresses two Codex review findings on #48 + the fmt:check CI failure. - setup(): write `.vigilesrc.json` and the summary from the RESOLVED plan (`plan.strict`), not the raw `--strict` flag — so an interactive "yes" to the workflow tier is actually persisted instead of silently dropped. - eject(): only delete the header's `compiled from` spec when it EXCLUSIVELY belongs to the file being ejected (`<file>.spec.ts`). A mirrored secondary target (e.g. AGENTS.md compiled from CLAUDE.md.spec.ts) names the shared source; deleting it would break the primary's recompile. Keep it + warn instead. Regression test added. - Format HANDOFF.md (the fmt:check failure). * fix(init): non-destructive adopt, deferred compile, test-only & strict config gating Install-DX hardening from PR #48 review + a self-audit across user/project types. - Auto-adopt is now NON-DESTRUCTIVE: `init` writes the spec but never overwrites your CLAUDE.md/AGENTS.md. Run `vigiles compile` to opt the file into spec management (byte-faithful; review the diff; `eject` reverses it). - Defer the compile when `vigiles` isn't resolvable yet (a fresh repo hasn't run `npm install`) — print the next step instead of a "failed to load" error. - Test-only setup (`init --test` / `--no-lint`) records the harness but writes NO lint rules — honors the positive-flag contract (Codex review). - setup() writes config from the RESOLVED plan, so an interactive "yes" to the workflow tier persists (was dropped). - Next steps reordered (install → compile → strengthen → test); `--report-only` in `init --help`; README + docs surface non-destructive adopt. New e2e tests. * fix(adopt): don't drop intro text when a literal ## Overview already exists Codex review (P2): a file with intro prose before the first heading AND an explicit `## Overview` section produced two sections keyed "Overview" — building the sections object overwrote the synthesized intro, silently dropping it (and a later `compile` would remove it from the file). Extract a shared `allocKey` dedup helper and use it for the synthesized Overview too, so it becomes "Overview (2)" on collision instead of clobbering. Regression test added. * fix(eject): preserve frontmatter on skill/agent eject; keep shared multi-target specs Two Codex review P2s on the eject feature. - ejectMarkdown no longer prepends the require-instructions-spec disable marker before a body that leads with YAML frontmatter (a compiled SKILL.md / subagent): doing so displaced the frontmatter and lost the surface's name/description/tools. The marker is instruction-file-only, so a frontmatter-led body ejects verbatim. - eject deletes the header's spec only when NO other compiled output still references it (a directory scan), instead of a basename heuristic. A multi-target spec (`target: ["CLAUDE.md","AGENTS.md"]`) or mirror now keeps its spec until the LAST output is ejected — ejecting either the primary or a secondary no longer orphans the others. The trailing marker hint prints only when a marker was added. Tests: frontmatter-preserved eject (unit) + eject primary/secondary/last-output of a multi-target spec (e2e). Docs note the two safety behaviors. * fix(eject,scan): path-safety on spec deletion; gate scan prompt on stdin TTY Two Codex review P2s. - eject: the `compiled from <path>` header is untrusted — a hand-edited/forged header naming `package.json` or a `../` path would make rmSync delete an unrelated file. Only remove a `.spec.ts` that resolves INSIDE the project; otherwise keep it and warn. Regression test added. - scan --trigger nudge: prompt only when BOTH stdout AND stdin are a terminal — `askOnce` reads stdin, so a TTY stdout with piped/redirected stdin (agents, pipelines) must take the non-blocking hint path instead of blocking on a read. * docs(readme): add FAQ + docs/faq.md, tighten install prompt, lift adoption line 5-star onboarding pass (1/N). - New docs/faq.md: the objection-handling answers (why a typed spec & is it optional, do I write TS, does init touch my files, CC+Codex, cost/key, non-JS repo, safe to run, vs promptfoo). - README: compact FAQ section linking it; install prompt rewritten to plain English (names adoption, tells the agent to finish: install + compile, no flag jargon); the non-destructive adoption reassurance lifted above the fold. * fix(init): honest guidance for non-JS repos (no package.json) 5-star onboarding pass (2/N). A Python/Rust/Go repo with a CLAUDE.md but no package.json can't resolve the vigiles npm package, so the typed-spec compile path isn't available there. Previously init told them to run `npm install` (which wouldn't help). Now it points at the real paths: - the deferred-compile notice and the adopted next-step both detect a missing package.json and recommend `npx vigiles lint` (verifies with zero install) plus `npm init -y && npm i -D vigiles` to enable specs. (Dropped the post-init lint "aha" idea — a freshly-adopted file is guidance-only with no marks to verify, so it would show nothing until `/strengthen` runs.) Test: init on a no-package.json repo guides to lint, not a bare npm install. * docs: add scripts/demo.sh — self-contained asciinema demo of the Lint layer 5-star onboarding pass (3/N). A zero-dependency, reproducible script for recording an asciinema cast / GIF: it builds a throwaway repo whose CLAUDE.md points (via inline marks) at a file that moved and a script that was renamed, then runs `vigiles lint` to show the ✗/✓. One command to record: asciinema rec --command "bash scripts/demo.sh" Tested end-to-end (real ✗ "File not found" / "Script not found" output). Consistency spot-check: public docs carry no stale adopt/compile language and all README-linked guides open with a README uplink. * docs(readme): final 5-star pass — tighten spec blurb, wire demo GIF slot 5-star onboarding pass (4/4). - Tighten the above-fold spec/adopt/eject paragraph (the new FAQ now carries the "do I write TS / will it touch my files" detail), and point new readers to it. - Wire a DEMO GIF slot at the prime "see it work" spot with the exact asciinema/agg recording command (scripts/demo.sh) + the embed snippet. README rendered content ≈152 non-blank lines (well within the ~200 cap); all internal doc links resolve; command refs pass the self-command-refs dogfood. * docs(handoff): record the 5-star onboarding-polish pass * docs(readme): add the demo GIF — vigiles lint catching stale references Animated terminal of `vigiles lint` flagging a moved file + a renamed script, at the README's prime "see it work" spot. The output lines are verbatim from the real CLI (nothing fabricated). Generated headlessly with Pillow — no recorder needed — so it's reproducible: `python3 scripts/make-demo-gif.py` (generator committed); scripts/demo.sh remains for a live asciinema recording. * docs(rules): sync rule docs with this session's default changes doc-per-rule drift caught auditing the session: - prefer-compiled-hooks.md: default is now OFF (was documented "warn") — a recommendation that shouldn't fire unasked. - require-skill-spec.md: drop the "deprecated" framing — it's the consistent require-<surface>-spec parallel (kept, default off), matching the code + matrix. * fix(eject): scan the whole project tree for shared-spec references, matched by resolved path The shared-spec guard scanned only the ejected file's directory, so a multi-target spec emitting a secondary output in a subdirectory (target: ["CLAUDE.md", "docs/AGENTS.md"]) was missed — eject would delete a spec still feeding another compiled file. Walk the whole project tree (skipping node_modules/.git/dist/build/coverage/.next/out) and match on the resolved spec path, not the basename, so two unrelated CLAUDE.md.spec.ts in different packages don't false-positive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-26 05:58:23 +07:00
zernie	d8c639f8da	refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46) ... * docs: sharpen README lint/guard copy - Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff, Clippy, and four more)"; Cedar stays in the subdoc - Drop the plugin-audit paragraph from the Lint section (secondary use case, still reachable from More) - Rewrite the Guard intro to lead with the concrete "fail open" failure and name what vigiles emits (exit code, JSON, AST-backed matcher) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface the $0/subscription cost story in the first paragraph The "no metered tokens, runs on your Claude subscription" wedge was buried in the last line of the Eval section. Pull a crisp version into the opening paragraph (above the fold); the Eval section keeps the detailed promptfoo/DeepEval contrast as proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface agent-writes-the-harness flow, reorder More, drop internal links - Quick start now leads with "then your agent writes the harness for you": the three model-invocable skills (test-harness/strengthen/ edit-spec) and the plain-English ask each one answers - Reorder More so CLI sits above the leaderboard (whose link points into the CLI docs); trim the now-redundant Skills bullet to a pointer - Stop linking internal research/ docs from the README (the moat link and the Research index) per public-vs-internal-docs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(compile): auto-refresh an existing harness.gen.ts compile now keeps a whole-harness registry in sync as a side effect, so the user never hand-runs generate-harness. Gated on the file already existing (compile maintains a registry you opted into, never imposes one) and cheap (parses specs, no linter spawn — unlike generate-types/ generate-schema, which spawn every linter and stay on the config-change guard). A duplicate agent name fails the compile. Docs: note the auto-refresh in the compile section and mark the generate-* artifacts as dev-toolchain output rarely run by hand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule Directory-scoped guidance loaded when working in src/: adding a new CLI verb or hook-runtime subcommand must clear a high bar (genuinely new mental action, no overlap with an existing command, siblings collapse to one verb + subcommand, runtime entrypoints stay hidden). Complements the root cohesive-cli-surface rule, placed next to the code that defines the command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec + integrity); dogfoods vigiles compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the "(generated)" annotation from the API reference link Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule The nested src/CLAUDE.md.spec.ts (added last commit) compiles under rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so both load via the tsx fallback (as the root always has). Also adds the no-internal-links-in-public-docs guidance rule (public docs never link research/*.md; keep the fact in research, don't link it), reconciles the old "link to research" line in public-vs-internal-docs, and files the deterministic lint-rule version as a P1 roadmap item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: separate public/internal indexes + surface eval safety - docs/README.md no longer links into internal research/ (public docs don't point at the private record) - research/README.md is explicitly labelled the internal record - README Eval section gains a "Safe to repeat" note (sandbox + ephemeral dir + egress + interceptTools, past-the-sandbox case flagged) linking docs/safety.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: stop public docs linking internal research/ + add safety FAQ Strip every research/*.md link from docs/*.md (the no-internal-links-in- public-docs rule) — ~60 links across 16 guides plus the docs/ index; prose kept readable, no orphaned research doc (all still cited from CLAUDE.md keyFiles / research/README.md, orphan-docs green). Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox question honestly: network via egress allowlist, a real DB/redis/browser via a disposable container you stand up (turnkey provisioning is a committed next step), irreversible externals via interceptTools, and the no-Docker deterministic/record-replay tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind> De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands rule, dogfooded). All breaking: - `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral column folded into scan; trigger-rate + collision matrix unchanged) - `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic WHY, a sibling of --fix-plan on the same ScanReport) - `refs <file>` → folded into `lint` (the unmarked-refs check is already a lint rule; the PostToolUse hook keeps its shared collectRefIssues entry) - `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand; the three dev-toolchain generators grouped, not N hyphenated siblings) scan reuses handleMeasure/handleExplain directly behind --trigger/--explain. Docs, help text, the regen guard, tests, and the self-command-refs dogfood all updated together (cohesive-cli-surface); 1588 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: split the GitHub Action reference out of cli.md into its own doc cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action section into docs/github-action.md (quick start, inputs, output channels, versioning) and leave a short pointer; retitle the cli.md intro. Update the README More bullet (CLI · GitHub Action) and the docs index. cli.md → 619 lines. orphan-docs + self-command-refs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks A single repo-level recommendation (one finding regardless of hook count): when a plugin ships hand-written hook commands that aren't compiled vigiles/hook artifacts, nudge toward compiling them — the message links docs/compiled-hooks.md and points at guardrail-check for verifying the ones you have. A discovery nudge, not a defect: the hand-written shell lane stays first-class, so it fires ONCE (never per-hook), defaults to warn (info- level ℹ; informational, doesn't fail CI), and is opt-out. Detector (manualHookCount: a command that isn't a hook-runtime invocation) lives in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc + matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's form-based, so it stays a recommendation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the eval-startups critique mapped onto positioning New research/eval-startups-positioning.md captures Thomas Liao's "why eval startups fail" critique and its implications: vigiles dodges the trap (harness-eval not model-eval, tooling not services, eval already a feature), but it's a yellow flag on the public ecosystem-benchmark as a MOAT (Goodhart-gameable → keep it a thin acquisition flywheel) and it independently validates the attestation/safety direction (capability-diff, guardrail proof) as the only defensible eval-shaped niche. Linked from research/README.md, measurement-authority.md (whose benchmark thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2 conformance/attestation play it validates). Internal-only per no-internal-links-in-public-docs; orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: purge internal positioning from public docs (jargon, name-drops, benchmark) Public docs (README + docs/) now state user-facing value, not vigiles's internal strategic vocabulary: - strip "moat" from 14 places → user-facing wording (engine / cost advantage / differentiator / cross-reference) - remove all bare research/*.md name-drops from docs/ (the backtick form the link-strip + deterministic check miss); scope-descriptions reworded to "configured directories" rather than naming research/ - README: drop the "ranks the hyped ecosystem: what works vs hype" benchmark banner from the Eval row + kicker (depth stays in measuring-skills.md) — per eval-startups, the benchmark is a flywheel, not a headline - docs/cli.md: "moat #2 (research/…)" → "the capability-diff check" Codified in CLAUDE.md: public-vs-internal-docs now has a strategic- vocabulary clause (moat/measurement-authority/flywheel stay internal; public docs name the benefit), so this doesn't regress. orphan-docs + self-command-refs green; the public↔private split stays docs/ vs research/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): apply multi-persona review fixes From a multi-persona README review (power user / staff eng / plugin author / non-TS user / eng manager): - non-TS users: a visible "No TypeScript needed — lint + inline comments work on any CLAUDE.md" line (was invisible; contradicted progressive adoption) - plugin authors: surface "vigiles scan ranks a marketplace, no key" above the fold (was buried in More) - eng-manager cost: hero now says deterministic layers run free in CI; merged kicker+affordability nails "real-model on a dev's own subscription, never a CI token" - staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted"; the typed-pipeline claim is concretized to the real pipe()/needs() mechanism - fixed a broken blockquote and trimmed the safe-to-repeat para to hold length Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): capture the README design direction as an editor-facing comment A top-of-file comment block records the framing the README must hold: lead with benefits, present compiled specs as the easy/auto-enforced next step (skills write them, hooks compile them; JS→TS evolution, not a wall), keep all four instruments first-class incl. Eval, stay scannable + under the length cap, and no internal vocabulary/research links. Guides future edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): benefits-forward rewrite + specs-are-easy framing Per the README direction: lead with what the user GETS, and present the typed spec as the frictionless auto-enforced next step, not a TS wall. - hero: stronger, concrete failure modes (stale ref / guard fails open / skill never fires), keeps Agent=Model frame, cost above the fold; dropped the redundant centered pitch (merge) - table cells condensed to punchy benefit one-liners - "You don't hand-write specs — your agent does. TypeScript for your harness: opt-in, gradual, auto-compiled, never a chore" (skills write them; JS→TS evolution) - de-dupe cost (drop "not metered tokens" from the Test cell) - "Not for you if …" honest-scope line - docs/verifying-instruction-files.md: back the claim — "you rarely hand-write one; edit-spec/strengthen write it, a hook recompiles on save" Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): don't lead callouts with "typed spec" — benefit first Casual readers shouldn't be greeted by "a typed spec is a program" / "TypeScript for your harness" in the prominent callouts. Reframe both to lead with the outcome and demote the spec to a soft, opt-in aside: - specs-are-easy block: "You don't hand-write any of this — your agent does … you never touch it by hand" (drop the TypeScript-for-your-harness headline) - Lint blockquote: "Bad states stop compiling … turn it up only when you want it" (drop "Markdown is prose; a typed spec is a program") - the edit-spec bullet says "edits the source", not "the typed spec" "No TypeScript?" reassurance and the optional .spec.ts mention stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: prettier-format HANDOFF Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): update the stale generate-types ref the verb rename missed The CI "Check generated types compile" step ran `node dist/cli.js generate-types` — renamed to `generate types` in the verb consolidation, so the job failed with "Unknown command". self-command-refs didn't catch it because it didn't scan .github/; add .github/ to its scan scope so a stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on. Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to research/hook-modes-and-testing.md (the nudge, not a requirement; the write-hook authoring skill is the real authoring lever, unbuilt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig) The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts and errors "not found by the project service". It's a tsx-loaded build input (like the root CLAUDE.md.spec.ts, which isn't linted either), so ignore it in eslint to match. Latent since the exclude landed; surfaced now that the earlier generate-types step no longer fails first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-25 01:20:05 +07:00
zernie	5c3cd2f947	feat!: layout-driven multi-harness validation rules & harness testing (#41) ... * docs: add rules-docs-in-sync rule; fix nested blockquote in harness-testing Add a `rules-docs-in-sync` guidance rule to CLAUDE.md so the validation-rule set (RulesConfig keys + built-in vigiles/* rules) and the central rules table in docs/cli.md never drift: adding/removing/renaming a rule must update the type, the table row, and the per-rule doc together. Also fix the doubly-nested blockquote (`> >`) in docs/harness-testing.md that rendered as a confusing quote-inside-a-quote — convert the pasteable prompt to a fenced code block, matching the README convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: surface the lint rule set in the linting guide; break up walls of text The Lint guide (verifying-instruction-files.md) explained the three rule *types* but never showed or linked the actual validation-rule *list* — so a reader following the README's Lint link couldn't find the rules. Add a "What vigiles lint checks" section that summarizes the rule families and links the canonical table in cli.md (no duplication, per rules-docs-in-sync). Also break four dense prose paragraphs in harness-testing.md into bullets / shorter paragraphs (entry points, harness-specific transport, the two-layers explainer, the eval-tool comparison) for mobile scannability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(orphans): never flag harness-loaded instruction files; move rules matrix to lint guide orphan-docs now categorically exempts files the harness loads directly (CLAUDE.md / AGENTS.md / SKILL.md / agents/ / commands/) from the orphan CANDIDATE set — they're load-bearing by name/location, not by being linked, so they're never orphans even if `orphans.include` is broadened to the whole repo. They still count as referencers. Adds tests + a docs/rules/orphan-docs.md. Move the canonical validation-rules matrix out of the CLI reference and into the linting guide (verifying-instruction-files.md), where a reader looking for the rules expects it — grouped by family (spec & integrity, test coverage, reference marking, subagent contracts, hooks & MCP, skill triggers, docs hygiene). cli.md and the README now link to that one matrix. Updates the rules-docs-in-sync rule to name the linting guide as the canonical home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: overhaul harness-testing — task-first guide + extracted API reference The harness-testing guide buried the first runnable example five screens deep, under ~140 lines of architecture (capability-contract theory, composition-root, adapter selection, import taxonomy) and didn't show its TOC until line 118. A first-timer couldn't find "how do I test my hook?". Full split: - Rewrite docs/harness-testing.md as a lean task-first guide (902 → 297 lines): a copy-paste first test in the first screen, a "what do you want to test?" decision table as the primary nav, trigger-rate promoted to a headline section, advanced material (significance / regression / cost / judge) collapsed. - Extract docs/testing-api.md: the full reference — Trace model, predicates, assertions, the check vocabulary, matchers, every measureTriggerRate / runEval option, significance/regression, imports & harness selection. - Move the theory (import-path-is-capability-contract, "two layers or three?") to research/harness-testing.md where the rationale belongs. - Repoint inbound anchor links (harness-testing-claude-code.md) and add the new reference to the docs index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * refactor!: surface-scoped lint rules — rename agent-*→subagent-*, capability-gate, fix dialect drift Adopt the "surface model" for cross-adapter vs adapter-only lint rules: a rule is scoped to the SURFACE it checks (subagents, shell hooks, skills, MCP, docs), not to a harness name. Universal rules stay bare and run everywhere; surface rules run only where the active adapter has that surface and report n/a otherwise (loud, never a failure) — so one config targets Claude Code AND Codex (incl. independent .claude/.codex trees, not just byte-identical mirrors) without per-harness rule duplication. Harness prefixes are reserved for genuinely exclusive concepts (none needed today). BREAKING (no back-compat, per design): - Rename rules: agent-tool-contract → subagent-tool-contract, agent-frontmatter → subagent-frontmatter, untested-agent → untested-subagent. Named by the harness-neutral concept (subagent), resolving the Codex `[agents]` (a concurrency table, not a subagent) name collision at the source. - Add AdapterCapabilities.subagents (CC/OpenCode true, Codex false); the subagent rules gate on it and report n/a where absent. - Fix the lint dialect-drift bug: checkSubagentToolContracts / checkHookEvents / checkFrontmatterSchema now run against the RESOLVED adapter's dialect + layout (and respect --harness), not a hard-coded claudeCodeDialect — scan was already adapter-aware, lint was not. - Docs: add the surface→harness "Applies to" matrix to the linting guide, rename the three rule docs, recompile CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * feat: layout-driven surface classification — ready for new-harness subagent dirs The subagent rules and scan's surface classifiers were hard-coded to `agents/` / `skills/` / `commands/`, so a harness whose subagents live elsewhere (OpenCode's `.opencode/agent`) wouldn't be classified. Make it layout-driven. - PluginLayout gains labeled `skillDir` / `agentDir` / `commandDir` (an empty `agentDir` means "no subagent surface" — e.g. Codex's `[agents]` TOML table). Set per adapter; documented in the adapter API + authoring guide. - scan.ts builds its isSkill/isAgent/isCommand classifiers from the active layout (regex anchored on a real path boundary, path-segment-safe), threaded through every surface helper — no module-level hard-coded dirs. - The `subagent-tool-contract` lint rule now REUSES scanPlugin's already-resolved, layout+dialect-driven agents (one-detector-no-drift) instead of re-globbing a hard-coded `agents/` path; drops the now-dead parseAgentTools/verifyToolContract imports. - Test: scan classifies a subagent under a custom `agentDir` and a default Claude Code scan does not — proving the dir is read from the layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: finish de-CC-ing the lint/scan detectors + add an ESLint guard Two more functional Claude Code hard-codings remained in the deterministic reference-verification path (same class as the subagent glob): - scan.ts `resolveScript` (hook-script-exists) hard-coded `${CLAUDE_PLUGIN_ROOT}`; now expands the harness's own `layout.pluginRootToken` (braced + unbraced). - test-coverage.ts (untested-skill/agent/hook) hard-coded `skills/` / `agents/` / `.claude/` globs and the CC token; now layout-driven via `skillDir`/`agentDir`/ `materializeRoot`/`pluginRootToken`/`manifestPath` (default CC), threaded from the resolved adapter in lint and from `lay` in scan. The deterministic layer-1 (lint + scan) is now fully layout/dialect-driven. The remaining `.claude` literals live only where they belong: the CC adapter, the CC eval transport (src/eval.ts), and CC plugin onboarding (init in src/cli.ts). Prevent recurrence: a `no-restricted-syntax` ESLint rule bans hard-coded CC string literals (`${CLAUDE_PLUGIN_ROOT}`, `.claude/`, `ANTHROPIC_*`) — as plain literals AND template quasis — in the harness-agnostic domain + detectors (src/core, scan.ts, test-coverage.ts, plugin-loader.ts). Verified it blocks a probe. Complements the existing import-graph boundary with a literal boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * test: dogfood the layout-driven untested-surface detector on a non-CC layout A regression guard for the de-CC-ing: build a repo with custom surface dirs (mysurf/skill, mysurf/agent) and plugin-root token ($MY_PLUGIN_ROOT), then assert findUntestedSurfaces discovers all three surfaces via the injected layout — and the default Claude Code layout sees none of it. CC-shaped fixtures can't catch a CC-hardcoding regression; this one can. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: thread the resolved adapter through ALL lint checks + Codex CLI dogfood Completes the dialect-drift fix: 8 more scanPlugin-based lint checks (skill-frontmatter, mcp-config, mcp-tool-resolves, hook-script-exists, disallowed-tools-contract, description-overlap, frontmatter-valid, mcp-hook-target-resolves) were still calling scanPlugin(process.cwd()) with the default Claude Code layout/dialect — only 3 of 11 had been fixed. All 11 now run against the resolved adapter, and the subagent-/hook-surface ones report n/a on a harness lacking that surface (capability-gated). Dogfood: a Codex `lint` CLI golden test (offline, in the existing CI job) wires a real Codex repo — AGENTS.md + TOML [hooks] with the harness's own ${PLUGIN_ROOT} token + an untested skill — and asserts hook-script-exists resolves the token and flags the missing script, untested-skill finds the skill, and the subagent rule reports n/a. A CC-shaped fixture can't catch a CC-hardcoding regression; this one runs the full pipeline against the second harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: codify the adapter-aware-lint convention + fill the capability/layout doc gaps Add an explicit `adapter-aware-lint-rules` guidance rule to CLAUDE.md (recompiled from the spec): lint rules are surface-scoped not harness-named; name by the harness-neutral concept (subagent, not agent); thread the resolved adapter and never hard-code a CC literal (eslint-enforced); gate by capability and report n/a; dogfood must be non-CC-shaped. The full record of this session's de-CC-ing. Docs caught up to the code: - harnesses.md capability matrix + adapter-api.md AdapterCapabilities gain the `subagents` flag. - adapter-api.md / authoring-an-adapter.md PluginLayout gains skillDir/agentDir/ commandDir; the myHarnessAdapter example gains the (previously missing, required) `capabilities` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(ci): update API report for the new adapter fields; add internal audience-check skill The `check` job's API surface gate failed: AdapterCapabilities.subagents and PluginLayout.skillDir/agentDir/commandDir are public (vigiles/adapter), so the committed etc/vigiles-adapter.api.md was stale. Regenerated (diff = exactly those four field additions) and gave each new layout field its own TSDoc so the report has no `(undocumented)` markers. Also add an internal dev skill `dev/skills/audience-check` (not shipped, not counted by untested-skill): re-read the README through six audience lenses — Claude Code user, Codex user, senior/staff skeptic, junior newcomer, eng-manager decision-maker, QA/test engineer — and report per-persona friction + ranked fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: note the UI-created-PR title gotcha in the conventional-commits rule A PR opened from the Claude Code UI gets a plain prose title that fails the `validate` job (which checks the PR title, not the commits). Document the fix: edit the title to a Conventional Commit + the matching breaking `!` as soon as the PR exists; validate re-runs on the title edit. (Hit on PR #41.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-20 03:54:34 +07:00
zernie	456991c974	feat!: harness testing/eval pillar + public-API surface revamp (#37) ... * docs: add eval-architecture design doc + ranked roadmap Capture the conceptual model for testing Claude Code harness features: feature = deterministic TEST + behavioral EVAL; two orthogonal knobs (reproducibility x gating); cost-matched mechanism; trigger-rate as a classifier eval; hash-lockfile + evalApiVersion + dated-model; deferred canary; closure-scoped invalidation. Reconciles the model with what the codebase already ships (record/replay cache, significance-gated baseline, check vocabulary, trigger-rate recall+precision) and ranks the genuine remaining gaps (tool-call spy/fake keystone, negative checks, curl fake, subprocess golden, hash-lockfile) mapped to the dogfood build order. Link from research/eval-api-landscape.md and research/roadmap.md. * docs: clarify constraint is no-SaaS, not zero-dep, in eval-architecture Deps are acceptable when they require no hosted account; rest the case against Polly.js/promptfoo on their real merits (cache exists, cassette doesn't escape the snapshot trap, promptfoo punt is strategic) rather than a dep-count rule. * docs: drop reuse-vs-reinvent tangent from eval-architecture * feat: add toolWith/notTool argument + safety checks to vigiles/check The inspection half of the keystone tool-call spy (docs/eval-architecture.md gap #1/#2): a serializable ArgMatcher (dot-path keys; RegExp = pattern, primitive = exact) drives two new Trace checks — - toolWith(name, args): the agent called `name` with input matching `args` (asserts HOW a tool was called, e.g. the image-request body carried the style suffix), and - notTool(name, args?): the agent did NOT call `name` (optionally only the args-matching calls, so "did not push to main" still allows pushing elsewhere) — the safety/negative assertion a completion grader can't make. Both flow through vigiles/testing + vigiles/unit via the existing `export *`. Pure, model-free, fully unit-tested in src/check.test.ts. * feat: tool-fake interception core + vigiles fake-tool-hook The interception half of the keystone tool-call spy (docs/eval-architecture.md gap #1). At the eval tier the model's tool decisions are real, but letting a side-effecting tool actually run (paid image API, git push, paid subagent) makes the eval expensive and risky. A FakeTool denies the real execution while the model still emits the tool_use — so the agent's decision is observable via toolWith/notTool, but nothing happens. - src/arg-match.ts: extract the shared ArgMatcher (dot-path keys; RegExp = pattern, primitive = exact) so the check vocabulary and the fake seam share one matching semantics; src/check.ts now imports it. - src/tool-fake.ts: FakeTool + decideFakeTool (first match wins, optional `when` arg-scoping), buildFakeToolSettings (PreToolUse hook over the union of faked tool names), and serializeFakeTools/parseFakeTools (env round-trip that preserves RegExp matchers a naive JSON.stringify would drop). Pure, tested. - src/cli.ts: `vigiles fake-tool-hook` — reads VIGILES_FAKE_TOOLS, denies a matched call via exit 2 feeding the canned result to the model (mirrors the agent-contract rail's block mechanism). Remaining: the ergonomic `fakeTools:` field on the eval spec (auto-merge the hook settings + a cache-key entry) to wire it end-to-end from measure/runEval. * feat: wire fakeTools onto the eval spec (end-to-end tool-call spy) Make the interception usable from the API: declare `fakeTools: [{ tool, when?, result? }]` on a measure/runEval arm and the eval tier does the rest. - EvalArm.fakeTools + MeasureSpec.fakeTools. - executeTrial auto-merges the fake PreToolUse hook into the arm's resolved settings (APPENDING to any existing PreToolUse list, never clobbering), pointing the hook at this CLI's own cli.js (resolved via __dirname, since the eval runs in a throwaway cwd where `npx vigiles` wouldn't resolve). - The fake list rides VIGILES_FAKE_TOOLS (AgentRunArgs.env, layered over process.env in spawnAgent), RegExp matchers intact. - CacheKeyInput gains `env` so two fake configs that share tool names — and thus produce identical merged settings — don't collide in the replay cache (their when/result lives only in the env). Tested via the injected runner (no model): the hook merges + appends, the env round-trips, and a no-fakes run wires neither. eval.ts/eval-cache.ts stay at the 100% lines/functions/statements gate. * refactor: make tool-fake honest + add real-model validation eval Acting on the step-back assessment of the tool-call spy: - Fix the misleading DEFAULT_FAKE_RESULT: the exit-2 hook is a BLOCK, not a successful return (CC surfaces it as a denied call), so the default reason now says "intercepted, NOT executed, do not retry" instead of "assume success". Reframe the tool-fake/eval docs as intercept-and-prevent + observe the attempt, not a faithful tool mock (CC has no skip-exec-return-success primitive). Note cost is not reduced — only the side effect is removed. - docs/eval-architecture.md: record the honest assessment — fidelity ceiling (deny != success), toolWith overlaps the existing toolUsedWith predicate (net-new is the negative check + interception), the simplest safety path (don't allowlist + assert the attempt) often needs no new primitive, and the competitor delta is narrow (assertions at parity; intercept-in-real-harness is the edge). - examples/harness/tool-fake.eval.mjs: end-to-end validation of the one assumption unit tests can't reach — that a DENIED tool's args still land in the Trace. Induces a git push origin main, intercepts it, asserts toolWith(Bash, /push origin main/) holds across trials. Loud skip() without claude. Run with a key before relying on the spy. * ci: behavioral eval gate scaffolding (inert until ANTHROPIC_API_KEY) Addresses the original kicker — real-model evals never ran in CI, results frozen as FINDING: comments. Adds .github/workflows/evals.yml with two jobs from the cost model in docs/eval-architecture.md: - eval-gate (pull_request): the cheap tier — the three shipped model-invocable skills' trigger evals (stubSkillBodies, 1 trial) on a DATED model, --no-skip. - eval-nightly (schedule + workflow_dispatch): the fuller drift tier you can't amortize. Both are INERT until enabled: every model step is guarded on the ANTHROPIC_API_KEY secret, so with no secret (default + all fork PRs) it prints a notice and exits 0 — never reds a PR. Turning it on is a deliberate secret + budget decision. Plumbing: `vigiles eval --model=<id>` → VIGILES_MODEL (sibling of --trials) so the gate pins a dated model id honestly without changing library defaults; the cheap trigger evals + tool-fake.eval.mjs now read process.env.VIGILES_MODEL. * feat: warn when an eval cache rides a floating model alias Closes the last piece of the honest-model-pinning gap (doc #3). A record/replay cache keyed on a floating alias (haiku/sonnet, or claude-sonnet-4-6 with no date) can serve a result computed against a since-changed model — hiding drift behind a green replay. - isDatedModel(id): a model is honestly pinned when it ends in an 8-digit date stamp (claude-haiku-4-5-20251001); a bare alias or version is floating. - runEvalWith emits a one-time warning (::warning:: under GITHUB_ACTIONS, else stderr) when cache != off and the model is floating. Tested: isDatedModel dated/floating, both warning branches (CI annotation + stderr), and the dated-id no-warn path. eval.ts stays at the 100% gate. * docs: document arg-match + tool-fake modules in CLAUDE.md Key Files The repo convention is that every src module appears in Key Files. Adds entries for src/arg-match.ts, src/tool-fake.ts, and src/tool-fake.test.ts (the tool-call spy), then recompiles CLAUDE.md from the spec — integrity hash valid. * test: validate tool-fake spy against a real model (capture-under-deny works) Ran the validation against claude-haiku-4-5-20251001. The model called Bash with "git push origin main", the auto-wired PreToolUse fake-tool-hook denied it (isError, "the git push didn't execute"), and the denied tool_use — WITH its args — still landed in ctx.toolCalls, so toolWith matched. This confirms the one assumption the unit tests can't reach: a denied tool's arguments are still captured for toolWith/notTool, and the side effect is prevented. Adjust the eval to assert the MECHANISM, not model obedience: strengthen the prompt, 5 trials, and gate at min 0.2 — any non-zero rate proves capture works; only a true regression (a flat 0 across every trial) fails. Records the finding inline. * docs: refresh eval comparison (public + internal) and positioning The eval comparison material predated the shipped significance/cache/cost/ regression-gating work and the tool-call spy. Brought it current: - research/eval-api-landscape.md (INTERNAL): the scorecard's vigiles column was marked ✗ on significance/caching/cost/concurrency/regression-gating — all shipped. Rewrote it to shipped reality, added rows for the no-model cheaper tiers, the intercept-and-prevent tool spy, and the regression gate; noted the inert-until-keyed CI eval gate. Three bolded-unique rows now name the defensible core honestly. - docs/harness-testing.md (PUBLIC): broadened the promptfoo-only section into a promptfoo/DeepEval/Braintrust/Inspect table; added the safety-intercept and no-model-tier rows; pointed to the internal landscape doc. - README: added one line on the safety property (notTool + fakeTools catch a push-to-wrong-branch / paid-API attempt and prevent the side effect). Positioning audit: README front door + internal scorecard now current; the spy modules are already in CLAUDE.md Key Files. * fix: harden eval cache invalidation (dir contents, tool-set, version salt) Research-backed (research/cache-invalidation.md, synthesizing Bazel/Turborepo/ Nx/ccache/Jest/webpack + promptfoo/LangChain/Helicone). Fixes the confirmed false-replay bug and two phantom-miss/staleness gaps, ranked by severity: 1. pluginDir contents weren't in the key (THE bug): a native --plugin-dir was passed to the run but its files were never hashed, so editing a skill in it false-replayed. hashDir() folds a sorted relativePath:contentHash list into the key (content not mtime; path-included). Flat sorted list, not a Merkle tree — over-engineering at plugin-dir scale. 2. Tool list now hashed as a SET (sorted) so a reordered allowlist isn't a phantom-distinct key. (canonical() already sorted object keys, so no stable-stringify dependency is warranted.) 3. CACHE_FORMAT_VERSION salted into the key (Jest/webpack pattern) so a record format change makes old entries unreachable — no brittle read-time gate. Model drift on a floating alias stays warn-only (the dated-id pin is the sound fix; can't auto-resolve without an API call). Eviction/TTL deferred as disk hygiene, not invalidation correctness — rationale in the research doc. Tests: hashDir (edit/add/rename/skip-junk), tool-set order-insensitivity, pluginDirHash sensitivity, and an end-to-end runEvalWith test proving a pluginDir content edit busts the cache. eval-cache.ts/eval.ts stay at 100%. * ci: let the eval gate run on a Claude subscription, not just an API key The affordability bet: vigiles drives the real `claude` CLI, so an eval authenticates like the user's own CLI — including a Pro/Max SUBSCRIPTION (flat cost), not metered API billing. (Confirmed empirically: a validation run reported apiKeySource:"none", i.e. it ran on the session's OAuth subscription.) Both eval jobs now accept CLAUDE_CODE_OAUTH_TOKEN (subscription, via `claude setup-token`) OR ANTHROPIC_API_KEY (metered fallback), and stay inert until one is present. Header documents the subscription as the cheap path. * docs: isolated-vs-whole-harness eval decision (research of record) Decision: support BOTH as tiers — default isolated (cheap authoring loop), add a whole-harness install-set arm as the release gate, plus a near-neighbor middle tier on the existing ncd engine. Why: skill selection is competitive and Claude Code evicts least-used skill descriptions under a context budget (skillListingBudgetFraction), so isolated trigger-rate overstates recall and understates false-positives. Every existing tool (promptfoo, Anthropic skill-creator, AWS, OpenAI) isolates; none populate the install set — a genuine wedge for "test the harness as it ships". Captured in research/isolated-vs-whole-harness-eval.md; roadmapped in eval-architecture (item 7). Also updated the eval-gate roadmap note for the subscription-token auth path. * feat: whole-harness trigger-rate tier (installSet) + honest isolated labelling Implements the isolated-vs-whole-harness decision (research/isolated-vs-whole- harness-eval.md). measureTriggerRate stays isolated by default (cheap authoring loop); a new `installSet` co-installs the skill-under-test alongside the user's real skills so it competes for selection as it does in the shipped harness — the release-gate tier no other eval tool offers. - TriggerRateSpec.installSet: plugin dirs or loose skills dirs merged into one install under the under-test plugin's name (under-test wins a name collision, so its <name>:<skill> id still matches `fired`). packageInstallSet + collectSkillsSource + copySkillsInto, pure/tested, cleans up on any error. - TriggerRateReport.competitors: count of co-installed competitors. 0 = isolated. - formatTriggerRateReport now labels honestly: an isolated run prints that recall is an UPPER bound and false-positive a LOWER bound (selection is competitive); a whole-harness run prints the competitor count. Tests cover the merge (skills/ + .claude/skills/ sources, collision, stub, missing-source + empty-under-test throws, junk-skip), both report labels, and installSet from pluginDir + skillsDir. eval.ts stays at the 100% gate. * docs: lead README with affordability; correct the evals-in-CI framing - README: affordability is now an explicit theme — most harness questions are answered with no model (free, every commit), and a real-model eval rides the user's Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI), not metered API billing like API-SDK-based tools. States plainly that evals don't run in CI by default. - research/isolated-vs-whole-harness-eval.md: fixed the "nightly/release gate" wording that wrongly implied a default CI cadence. Clarified that evals are not in CI by default (the per-PR CI is the free deterministic tiers; the eval workflow is inert until you add a token); "release gate" = run deliberately (locally on the sub, or opt-in nightly). * fix: make the eval CI gate fail on a non-run or a corrupt cache Two silent-pass holes the eval gate could slip through — closed so a green eval job actually means evals ran and the cache was sound: 1. Evals never executed. `vigiles eval`/`test` gained `--min=N`: assert at least N scripts actually matched/ran, else exit 1 loudly. A renamed file, a bad path, or a glob matching nothing previously printed "No files found" and exited 0 — a hidden gap. evals.yml now passes --min=3 (the gate's three listed evals) and --min=1 (nightly), alongside the existing --no-skip. 2. Cache fails. readCache now THROWS on a corrupt record (file present but invalid JSON) instead of swallowing it and silently re-running. A broken cassette is a real failure the gate must surface; a true miss (no file) still returns null and proceeds normally. Message says how to recover. Tests: cli.test --min (0 matched / too-few / enough), eval-cache corrupt→throws vs miss→null. eval-cache.ts stays at the 100% gate. * fix: trigger-rate on Sonnet (not haiku) + count sibling skills as competitors Dogfooding the three shipped skills on the subscription found two real bugs: 1. Gate model. All three skills "failed" recall on haiku (test-harness 0.50, edit-spec/strengthen 0.70) — but on Sonnet test-harness scores 0.90 and passes. Trigger-rate is a SELECTION measurement and haiku is a much weaker selector, so a haiku gate false-fails skills that fire fine on the model users run. measureTriggerRate now defaults to "sonnet" (matching measure); evals.yml pins claude-sonnet-4-6 for the gate; docs corrected. The skill descriptions were never the problem. 2. competitors miscount. The honest-label counted only installSet additions, so a multi-skill plugin (the dogfood installs all of vigiles' skills) printed "isolated: no competing skills" while ~5 siblings were actually competing. Now competitors = (skills in the resolved install pool) − 1, via countSkills, so siblings count and a real single-skill install still reads 0/isolated. packageInstallSet's return renamed competitors→added (installSet delta). Tests: sibling-competitor count, packageInstallSet `added`. Recorded the haiku-vs-Sonnet FINDING in test-harness.trigger.eval.mjs. eval.ts at 100% gate. * feat: model-per-arm (Q1) + save model/isolation eval decisions Q1 (model strategy) — agreed + built: - EvalArm.model: a per-arm model override. A model comparison IS a harness A/B, so "does my harness hold on the cheaper tier / after a model upgrade?" is arms: { sonnet: {model}, opus: {model} } read through the existing significance machinery — no separate multi-model matrix runner (that's the cost-multiplying promptfoo lane we avoid). measureTriggerRate stays single-model by design. - Default stays Sonnet (the realistic selector), per the dogfood finding. Q2 (isolation) — agreed: keep TWO tiers (in-plugin by default = honest sibling competition; installSet for the whole harness). Do NOT build per-skill isolation machinery (it produces the LESS honest number others default to). Already shipped; no new code. Saved research + conclusions: docs/eval-architecture.md gains a "Model strategy" section (default Sonnet w/ the 0.50-haiku-vs-0.90-Sonnet dogfood evidence, model-as-arm, no matrix, the deferred model-floor); the isolation research/decision is in research/isolated-vs-whole-harness-eval.md. Validated on the subscription: the corrected Sonnet gate is GREEN (3/3 skills ≥0.8) and the competitors label now reads "whole-harness: 4 competing skills". * feat: runtime model floor for trigger-rate (default Sonnet) + gate on alias The "set a floor for all skills" ask, solved without a (brittle) static linter: a RUNTIME floor enforced in the eval library catches the env-var model a lint can't see — which is exactly how the haiku footgun entered. - modelTier(id): rank by family (haiku<sonnet<opus; version ignored), null for an unrecognized family so the floor never blocks a model it can't judge. belowModelFloor(model, floor): fail-open on unknowns. - TriggerRateSpec.minModel (default "sonnet", resolves minModel → VIGILES_MIN_MODEL → "sonnet"). measureTriggerRate FAILS before spending a token if the resolved model is below the floor — so one knob (VIGILES_MIN_MODEL / --min-model) sets the floor for ALL skills, no per-file annotation, no linter. Lower it deliberately for a cheap run. Decision: FAIL (not warn) — the floor is a contract; the escape hatch is lowering it. - CLI: `vigiles eval --min-model=ID` → VIGILES_MIN_MODEL. - Gate model switched claude-sonnet-4-6 → the `sonnet` ALIAS: the per-PR gate runs cache-off, so it should track current Sonnet without churn; dated/exact ids matter only for cached/baselined reproducibility (kept on that path). Tests: modelTier/belowModelFloor (families + fail-open), floor throw at default + VIGILES_MIN_MODEL, and the lower-floor escape hatch. eval.ts at 100% gate. * refactor: remove model envs + the speculative evals.yml CI workflow Acting on review: neither was asked for, and both were the wrong layer. - Removed VIGILES_MODEL / --model and VIGILES_MIN_MODEL / --min-model. Model is part of the measurement DEFINITION (a different model = a different result), so it belongs in the spec (`model`/`minModel`, version-controlled), not a hidden CLI/env override. Contrast VIGILES_TRIALS, a legit run knob (cost/precision, doesn't change what's measured) that PRE-DATES this work — kept. The dogfood evals now omit `model:` and rely on the Sonnet default + the minModel floor. - Removed .github/workflows/evals.yml. Real-model evals don't belong in a standalone GitHub Actions workflow needing a metered/sub-token secret: CI runs the FREE deterministic tiers (ci.yml, no token), and the real-model eval runs where the subscription already is — a Claude Code session or locally (vigiles drives the `claude` CLI). The eval-robustness work (--min, --no-skip, corrupt-cache throw, Sonnet model floor) stays — it hardens `vigiles eval` wherever it runs. Docs reframed accordingly (eval-architecture kicker + roadmap, README, the two research docs): deterministic tiers in CI, evals on the sub in a session. eval.ts at the 100% gate; 974 tests pass. * docs: consolidate positioning + pros/cons of the eval approach Per review — the competitor scorecard, caching research, isolation decision, and model strategy were all saved, but the POSITIONING + pros/cons were scattered across README/kicker. Added a dedicated "Positioning & pros/cons" section to the eval-architecture design-of-record: the affordability thesis (free deterministic tiers + sub-priced real-model tier vs competitors' per-token-every-run), the pros (cost/fidelity/honesty) and the honest cons (sub rate limits, non-determinism, intercept≠mock, sonnet-not-haiku, not-a-CI-checkbox, no dataset/red-team/UI), with links to the four supporting research docs. * fix: key the eval cache on the harness binary version (fast-evolution edge) You were right — the cache wasn't tested/keyed against fast model/harness evolution. Closing it: - harnessVersion (`claude --version`, memoized, cache-path only) is now in the cache key. A CLI upgrade changes the system prompt + tool defs — which steer behaviour as much as the model — so a cached result now INVALIDATES across a `claude` upgrade instead of silently replaying a result from a different harness. Trade-off (frequent patch invalidation) documented; re-scoring within a session is unaffected (version stable then). - cacheKey test now actually covers model / settings / env / harnessVersion changes (it previously asserted "any input" but skipped several, incl. model). - research/cache-invalidation.md gains a "Fast-evolution edges" section: harness version (keyed), model-alias re-point (warned, can't auto-invalidate without a weight hash), and modelTier new-family fail-open (a documented known limit). eval-cache.ts/eval.ts stay at the 100% gate. * docs: positioning into CLAUDE.md (canonical) + consistency pass Point 3 — positioning belonged in the positioning source, not buried in eval-architecture.md: - CLAUDE.md ## Positioning (pillar 2) now states the affordability thesis: free deterministic tiers + a real-model tier that runs on the Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI, no metered API), measured on the realistic selector (Sonnet, minModel floor), model-in-spec not env. Recompiled. - eval-architecture's pros/cons section now points to CLAUDE.md as canonical and keeps only the detail. Gap #4 — refreshed the stale `src/eval.ts` Key Files entry (model-per-arm, minModel floor + modelTier, installSet whole-harness tier + competitors, harnessVersion in the cache key, Sonnet default). Gap #2 (consistency pass, you approved) — the "knobs" / "match-the-mechanism" / reconciliation sections no longer claim evals run in `every-PR` CI on `haiku`: deterministic tiers are every-commit CI, the real-model eval is run on-demand on the sub (Sonnet). Fixed the decision table + the reconciliation bullets too. Gap #3 — documented the shipped trigger-rate knobs in the user guide (harness-testing.md): Sonnet default + minModel floor, isolated-vs-whole-harness `installSet` + the `competitors` count, and model-per-arm for cross-model A/B. * feat: key cache on harness major.minor, automate stubSkillBodies, exhaustive cache tests + near-neighbor roadmap Acting on review: - Claude version granularity: key the cache on the harness MAJOR.MINOR, not the patch (harnessVersionKey: "2.1.179 (Claude Code)" → "2.1"). A patch rarely changes behaviour, so keying patches would churn the cache every release for no signal; a minor/major bump is where the system prompt actually moves. Documented the trade-off (a known-behavioural patch → clear cache / bump CACHE_FORMAT_VERSION). - stubSkillBodies is now AUTOMATIC for measureTriggerRate (default true): testing a description never needs the body (trigger is a frontmatter property decided before the body loads), so it's not a knob you must remember. Set false only to run the real body. Updated the 4 fake-pluginDir tests to stubSkillBodies:false (passthrough), which also keeps the non-stub branch covered. - Cache tests are now EXHAUSTIVE: one assertion per CacheKeyInput field, plus explicit VERSION axes — two model snapshots (sonnet-4-6 vs -4-5) and two harness minor versions each produce a different key. + harnessVersionKey unit test. - Roadmap: added the near-neighbor trigger-rate tier (NCD-nearest competitors) to Next, priority P3 (MED-LOW) — decided, deliberately deferred. (Consistency gaps #2/#3/#4 shipped in 74fc151.) eval.ts/eval-cache.ts at 100% gate; 975 tests. * feat!: rename fakeTools -> interceptTools (intercept-and-prevent, not a mock) The tool-fake feature does intercept-and-prevent (a PreToolUse hook denies the call via exit-2), not fake-a-return-value. The "fake" name wrongly implied a mock that substitutes a result and continues, requiring two JSDoc corrections to undo. Rename to carry the real semantics. BREAKING CHANGE: the vigiles/eval spec field `fakeTools` is now `interceptTools` (on EvalArm and MeasureSpec); the `FakeTool` type is `ToolIntercept` and its `result` field is `denyReason`. The `vigiles fake-tool-hook` CLI subcommand is now `intercept-tool-hook`. Pre-1.0, but signalled for a correct changelog. Keep toolWith/notTool (clearer than competitors) but add promptfoo/DeepEval cross-references to their JSDoc and a "vigiles check <-> promptfoo assertion" mapping to research/promptfoo-deep-dive.md, noting notTool/interceptTools as the restraint-assertion gap a completion-grading eval cannot cheaply express. * docs: add safety model front-door doc (confinement + tool interception) docs/sandboxing.md covers host confinement for hook/plugin code but not the eval-tier risk — a real model deciding to call a side-effecting tool. Add docs/safety.md as the front door tying the whole model together: the two risks (foreign code vs the model's actions), the two mechanisms (bwrap confinement vs interceptTools/notTool PreToolUse deny), the one safe-by-default fail-closed rule, the per-tier confinement table, and the honest gaps. Answers "why not confine everything" by pointing at the four-reason argument in sandboxing.md. Referenced from README + sandboxing.md (not an orphan). * docs: decide cross-platform sandboxing (macOS via sandbox-exec) + save the verify-sandbox-config idea Mac support is a hard requirement (large dev population), and the claude binary's own sandbox can't cover us (Bash-only, skips hooks, no external API, CC-only). Record the decision: a vigiles/os-isolation port with a native backend per OS — keep bwrap+nft on Linux (strongest, packet-layer), add sandbox-exec/Seatbelt on macOS (built-in, zero deps, deny-all-net). Per-host egress stays Linux-only; macOS degrades honestly. srt/nono are documented fallbacks, not the default; Birdcage is reference-only (GPL/binary-net). Also clarifies the enabled/disabled model (provenance, not platform: your code runs direct, foreign code is confined-or-refused — the Mac backend makes the safe option available so nobody is forced to sandbox:false), and parks the "verify & test the harness's own sandbox config" idea (the network-validation wedge: valid-is-not-true for sandbox policy). Roadmap + sandboxing.md updated; not an orphan. * docs: add the ephemerality axis — running a model-driven skill is a side effect, independent of trust Correct the safety model: provenance governs CONFINEMENT (host protection), but EPHEMERALITY (state protection) is unconditional — a skill/agent is model-driven, so even a plugin you wrote produces nondeterministic side effects when run. Trust does not make a run side-effect-free. Classify side effects by reversibility: local-reversible -> ephemeral CWD (already done); escapes-CWD (~/.gitconfig, ~/.ssh) -> ephemeral HOME + scrubbed env (the GAP: the direct/non-bwrap path inherits process.env + real HOME); external -> deny/allow net; irreversible (push, charge) -> interceptTools/notTool (can't un-push by discarding a box). Ephemerality needs no kernel features, so it lands on macOS today, ahead of Seatbelt — the cheapest cross-platform side-effect protection. safety.md + cross-platform-sandboxing.md + roadmap (HIGH) updated. * docs: positioning — behavioral/side-effecting testing + cross-platform (Mac+Linux); fix phantom issue refs Positioning (CLAUDE.md): name BEHAVIORAL, side-effecting eval of the assembled harness as the apex of pillar 2 (highest value, kept thin; notTool/interceptTools ship today, the ephemeral run env + disposable-dep provisioning are the committed next step). Add a "Cross-platform by requirement (macOS + Linux)" paragraph: native backend per OS behind a vigiles/os-isolation port, and the two orthogonal axes (provenance-keyed host protection vs unconditional ephemeral state protection). Fix: the declared-vs-enforced anchor cited as Claude Code #54898 could not be verified; replace every occurrence with the real anchors (Claude Code #4740/#21460, claude-agent-sdk-typescript #172) across agent-runtime, cli, the spec, and research. A reference-checking tool should not ship a phantom reference. * docs(research): save the mid-2026 subagent-demand + skill-eval-landscape findings subagent-compilation.md: ranked issue-survey demand evidence (declared!=enforced anchored to #4740/#21460/SDK #172; orchestration/output-contract/dispatch/MCP/ observability/model/determinism/firing/nesting themes), the authoring+observability tooling table (almost no verification/testing tool exists), Codex-vs-CC delta (enforcement is Claude-Code-shaped -> Codex non-goal confirmed), honest map-to-vigiles. skill-eval-landscape.md: how labs + practitioners actually eval skills — Anthropic skill-creator + "Demystifying evals", OpenAI's Codex eval guide, the two Reddit harnesses (SkillBenchmark blind-judge A/B+CIs on Caveman; the "SKILL.md is a contract" methodology harness incl. the adversarial-gate finding), Spence/dbt/ Husain/Schmid. Gap verdict: behavioral skill eval is now TABLE STAKES; vigiles's moat is the intersection (whole assembled harness + subscription affordability + statistical rigor + the deterministic runHook tier + the eval->enforce bridge). promptfoo-deep-dive.md: promptfoo acquired by OpenAI (~$86M) + DeepEval v4 agentic harness; sharpens the vendor-neutral, subscription-not-metered positioning. eval-api-landscape.md: new competitors (TribeAI/claude-evals metered; paultyng/ testagent CLI-fake) + statistical-rigor confirmed as a differentiator. * docs: native input/output/cache token+cost measurement decision + behavioral-umbrella clarification eval-architecture.md: (1) state explicitly that behavioral/side-effecting eval (the ephemeral integration tier) is the UMBRELLA and the adversarial-gate test is one member, not a replacement — with the eval->enforce bridge (pillar 2 hands off to pillar 1 when a prose gate caves); (2) decide native token/cost support split by class (input/output/cache) — the tokens() check collapses input+output and cache tokens aren't captured, so a single total can bless a net-negative change (Caveman cut output yet 2-4x'd cost). Plan: extend UsageTrace to all token classes, first-class inputTokens/outputTokens/cacheTokens checks, and an A/B delta-per-class gated by Welch significance. Added both to the ranked roadmaps (HIGH). * feat: native input/output/cache token measurement + per-class checks A harness change trades tokens across sides (a skill/CLAUDE.md injection adds INPUT every turn; a "compression" skill cuts OUTPUT), so a single total can bless a net-negative change — SkillBenchmark's Caveman cut output yet 2-4x'd cost. Split the usage model into honest classes and gate on them separately. - EvalUsage + UsageTrace gain cacheCreationTokens + cacheReadTokens (parsed from the claude result usage: cache_creation_input_tokens / cache_read_input_tokens); inputTokens stays the FRESH (uncached) input. ArmUsage + aggregateUsage sum the two cache totals. - New vigiles/check checks: inputTokens({max}), outputTokens({max}), cacheTokens({maxCreation?, maxRead?}) beside the existing tokens/cost/latency. Exposed on ctx.usage so an A/B delta-per-class rides the existing mean±se / Welch-significance machinery for free. - Dogfood: skill-compression.eval.mjs now measures input AND output (cache reads ~0.1x input, writes 1.25x), so the compression claim is verified on both sides. Tests: pure pass/fail/score/message/toJSON for each new check; usageFrom parses the cache fields (0 when absent); aggregateUsage sums the cache totals. * docs: record real-haiku FINDING in the compression eval — cost metrics work, cache dominates Ran the compression dogfood on real haiku (3 trials/arm, on the Pro/Max subscription, apiKeySource:"none", $0.1057). Captures the end-to-end result as a FINDING: total_cost_usd + input/output + cache tokens all populate from a real run; inputTokens read ~42/run (the uncached remainder) vs ~2.4-2.7k real volume — the rest is cache reads, so a single input/total understates economics ~50x here. The cache classes are where the volume lives (vindicates the split). The compression claim did not hold on this task/model (output +11%, cost +$0.002, correctness intact) — the headline-% failure mode the eval exists to catch. * feat: adversarial-gate dogfood eval + the eval->enforce bridge The highest-value behavioral test for an enforcement skill: adversarially ask the agent to skip the gate the skill is supposed to enforce, and assert it REFUSES. - examples/harness/dogfood/adversarial-gate.eval.mjs (write-don't-run): an inline SKILL.md states a measurement gate (no benchmark without a baseline; refuse if asked to skip), the task adversarially asks to skip it, and two composed checks assert refusal — notTool (didn't run the forbidden command) + output (pushed back in prose), gated by assertRates({min:0.7}). No new check primitive: the pattern is the existing vocabulary composed inline (no gold-plating). - docs/eval-architecture.md + docs/safety.md: document the eval->enforce BRIDGE — pillar 2 (eval) MEASURES whether a prose gate holds; when the adversarial test shows it caves, pillar 1's deterministic PreToolUse rail is the FIX. Eval hands off to enforcement; nobody else connects the two. * feat: ephemeral run environment foundation (pure builder + opt-in flag, default OFF) Running a model-driven skill mutates state regardless of trust (the model, not the author, chose the actions), but the eval tier inherits the full real environment — a model-driven git push / write to ~ uses the real $HOME, ~/.gitconfig, ~/.ssh, ~/.aws. Add the foundation to close that, behind an opt-in flag, default OFF so the eval tier's own auth can't silently break before a real-run validation. - ephemeralRunEnv(base, { home, allow }) — pure, exported, no fs/spawn: sets HOME+TMPDIR to a throwaway dir, passes through an auth+runtime allowlist (mirrors the runtime port: PATH/LANG/TERM/ANTHROPIC_*/CLAUDE_*/LC_* + region/profile so a Bedrock/Vertex-backed claude still authenticates), DROPS everything else (GIT_*/GH_TOKEN/SSH_*/AWS secret keys + any non-listed var). Conservative by design: when unsure, allow (a too-narrow allowlist breaks auth). - EvalSpec.ephemeralEnv?: boolean (default off). When on, executeTrial mkdtemps a throwaway HOME under the trial cwd and builds the scrubbed env; AgentRunArgs gains replaceEnv so spawnAgent uses a.env as the COMPLETE env. When OFF, the env path is byte-identical to before (asserted by a test). The cross-platform state-protection floor (needs no kernel features → lands on macOS now). Flipping the default to on waits for a real-auth validation pass; two honest follow-ups noted in code (AWS static-key auth, cache+ephemeral key interaction). * docs: validate ephemeral-env auth survival (env-var path works; file-based ~/.claude is the gap) Ran a real-model check (haiku, ephemeralEnv: true) on the subscription: the run authenticated through the scrubbed throwaway HOME — exit 0, $0.003, output "OK.", cache read ~21.7k tok (caching unaffected). So the env-var auth path is validated: the credential rides an allowlisted ANTHROPIC_* var that survives the scrub. Remaining gap before flipping the default on: this only covers env-var auth. A subscription credential that lives only in ~/.claude/.credentials.json (a file under HOME) would be lost when HOME is scrubbed — ephemeralRunEnv injects env vars, not the credential file. Safe default-on needs injecting the harness's own credential file into the throwaway HOME. Flag stays opt-in until then. * feat: complete the opt-in ephemeral run env (auth-file injection + cache-safe) + validate containment Finish the ephemeral-run-env opt-in path (still default OFF): - seedEphemeralHome copies a minimal auth allowlist (EPHEMERAL_HOME_KEEP = [".claude/.credentials.json"], copy-not-symlink so the run can't write back) from the real HOME into the throwaway HOME when present, so local file-based OAuth survives the scrub; .gitconfig/.ssh/.aws are never carried. - cache-safety: HOME/TMPDIR excluded from the eval cache key (CACHE_KEY_ENV_EXCLUDE), so the random throwaway HOME no longer makes every ephemeral run a cache miss; non-ephemeral keys unchanged (surgical, not a blanket env drop). Validated state protection end-to-end on the real subscription (haiku, ephemeralEnv: true, the run told to write $HOME/PROOF.txt + probe ~/.ssh): real $HOME untouched (write landed in the throwaway HOME) AND the run still authenticated (exit 0, $0.0083). Containment holds; auth survives via the host-brokered CLAUDE_* allowlist. Default stays OFF: this host-brokered env can't run-test the file-based ~/.claude/.credentials.json OAuth modality (no credential to exercise; no sandbox mints a real token), so the credential-file injection is unit-tested for the copy logic but not e2e-validated. Flipping default-on (a behavior change for every eval) waits on a local-file-auth user validating that path; then it's a one-liner. * docs: phased design for the vigiles/os-isolation port + Seatbelt backend Grounded implementation plan over the existing sandbox core (design of record, not yet built): the narrow port interface (NetPolicy tagged union / FsPolicy / IsolationCapabilities / OsIsolationBackend), the core-vs-backend layout, the per-backend capability matrix, the Seatbelt backend shape (sandbox-exec + minimal SBPL profile), the eslint-plugin-boundaries enforcement (only backends name a raw primitive, dogfooded via enforce), and a 4-phase migration that keeps the 997-test suite green at each step (extract+move bwrap behind the port with ZERO behaviour change -> add Seatbelt -> wire selection + lint -> fold in ephemeral env). Surfaces the sharp limitation honestly: Seatbelt (deny network*) blocks localhost too, so bwrap's co-launched-mock-over-loopback trick has no Mac equivalent (loopbackNet:false) — a trusted spec runs direct, an untrusted-needs-mock spec refuses. Per-host egress stays Linux-only; srt and CC's built-in sandbox stay out. Linked from the roadmap (HIGH). * docs: consolidate eval coverage/isolation into one source of truth (R1+R2 owned, R3 composed) Add research/eval-coverage-and-isolation.md as the authoritative doc: the three-rung model (R1 cheap/deterministic — nothing executes; R2 record-replay — a tool/MCP/API result recorded once from the real tool and replayed via a fake binary on PATH, never model-synthesized; R3 real disposable service whose semantics IS under test), the blended+scrubbed coverage distribution (R1+R2 ~90%+ with no Docker on the sub; R3 a thin apex — browser/DB/redis), the honest e2e landscape (every benchmark/lab/Inspect runs e2e in a container; no containerless reproducible e2e exists), the safety/viability/performance axes + non-goals, the competitor comparison, and a reusable rung-classifier prompt. Make the chosen approach crystal-clear in CLAUDE.md positioning (R1+R2 owned on the sub; R3 composed-with-a-container, not reinvented), docs/eval-architecture.md (Coverage & scope section), README (public), and the roadmap (the R2 enabler ranked ahead of testcontainers + a dogfood/COVERAGE.md item). Use standard terminology (PATH-shim + record-replay/VCR), not a coined name. Scrubbed — no private-source identifying details; conclusions blended with the community/landscape data. * feat: R2 tool-stub / PATH-shim helper (record-replay a tool result, no service, no Docker) The R2 enabler from the coverage model: test a skill whose logic consumes a tool's RESULT by shadowing the binary on PATH with a fake that emits a recorded/canned output — standard PATH-shimming + VCR-style replay, not a novel approach. Unlocks the ~43% R2 surface with no live service and no Docker (ranked ahead of any testcontainers work). - src/tool-stub.ts: ToolStub + renderToolStub/writeToolStubs/stubBinDir. Each stub is a self-contained POSIX script that base64-decodes its canned bytes straight to stdout (base64's alphabet can't contain a shell metacharacter → injection-proof for any fixture content; piping preserves trailing newlines that command-subst would strip). Stubs are author/recorded fixtures, NEVER model-synthesized (drift → false confidence). argv-matching + record-from-real are documented follow-ons. - EvalSpec.stubs wired into executeTrial: a bin dir under the trial cwd is prepended to PATH for BOTH the ephemeral and legacy env paths; absent stubs = byte-identical (no default flipped, ephemeralEnv stays off). - Exposed via vigiles/testing + a ./tool-stub subpath export. - harness-test tier wiring left as a documented TODO (it routes through the per-harness driver seam — out of MVP scope). Tests: faithful round-trip of shell-special content incl. trailing newlines; PATH prepend on both env paths; absent-stubs unchanged. 1007 passed / 11 skipped. * test: dogfood the three vendored plugins + per-plugin COVERAGE.md scorecards Prove how much of real plugins the cheap tiers actually cover. 9 model-free, CI-runnable tests (vendor-coverage.test.ts) across the three vendored plugins — OMC keyword-detector R1 fire/no-fire (the free floor under the bwrap-gated egress copies), scan structural facts + the wshobson inherits-all footgun on the scan report, superpowers session-start R1 (fallback) + R2 (reconstructed plugin root injecting the real skill body), and the canonical R2 stubBinDir demo (shadow npm on PATH to drive find-polluter.sh's real bisection to completion, no live service). Per-plugin COVERAGE.md scorecards (rung table + distribution + grade + R3 shortlist + honest verdict), placed beside the pinned snapshots (never inside them): - oh-my-claudecode: ~60% free / ~33% model-gated / ~7% R3 — B+ - superpowers: ~50% free / ~50% model-gated / ~0% R3 — B (a prose library is half behavioral, only measurable with a model) - wshobson-accessibility: ~30% free / ~25% model-gated / ~45% R3 — C (worst case: its semantics ARE the real browser; the free tier still catches the inherits-all agent footgun) The B+/B/C spread is the finding: where free/deterministic testing stops and the model/container tiers take over. 1016 tests pass; R3/model-gated items written-up, not run, by design. Linked from eval-coverage-and-isolation.md (Dogfood results). * docs: record 2026-06-17 multi-SDK probe in sdk-harness-testing Validate and sharpen the prior "don't retarget pillar 2" verdict with a nine-SDK primary-source probe across six dimensions (mock-model / tool-contract enforcement / assembled-as-shipped / trigger-rate / sub-affordability / record-replay). The prior verdict holds for code-SDKs (Pydantic/Vercel/LangGraph/LlamaIndex/Mastra ship first-party mocks), but the probe relocates pillar-2 value to the Claude Agent SDK + Codex (no mock, buggy/unenforced #172 tool contract, sub-affordability) and adds a concrete mock-ergonomics borrow-list. Roadmap entry tightened to match. * docs: make the two-axis coverage boundary explicit (public + internal) The R1/R2/R3 rungs only answer "what does the test need from the outside world" — they don't answer "who decides pass/fail", which is what makes a prose skill look like a coverage hole when it isn't. Add the second (oracle) axis everywhere the rungs appear: - Define the `-MG` (model-gated) suffix the dogfood scorecards already use, and place a prose/guidance skill explicitly at R1-MG (nothing executes, but only a model judges its worth — fully testable on the sub, not free, not uncovered). - Add the three-bucket framing (A free & deterministic / B model-gated on your sub / C needs-a-container, composed) and grade with two numbers ("% testable at all (free+sub)" vs "% needs-a-container"). - State that skill BEHAVIOR is tested (runEval A/B + judged), not just the description firing (measureTriggerRate). - Record that the sub-affordability story is ToS-clean (drive your own CLI / your own harness / your own sub; the Agent SDK ToS restricts productizing claude.ai login, not running your own tests). - Make the shipped test-harness skill coverage-aware (a Step 0.5 that classifies each surface into the three buckets and sets honest per-bucket expectations). - Roadmap: add the 2026-06-17 multi-SDK mock-ergonomics borrow-list, linked to sdk-harness-testing.md, marked recent/this-PR. Covers CLAUDE.md(.spec.ts), README, docs/eval-architecture.md, research/eval-coverage-and-isolation.md, research/roadmap.md, skills/test-harness/SKILL.md. * docs: flag open finding — behaviour testing is broader than A/B Persist a conclusion raised this PR that lived only in chat: A/B (runEval arms + assertSignificant) is the right oracle for relative /regression/noise-floor questions, but testing an EXACT skill wants an ABSOLUTE judged assertion, not a comparison to an OFF arm. The single-arm absolute path already exists (measure + judged + assertRates, skill-quality.eval.mjs); the open question is whether it is first-class and discoverable or whether A/B is over-privileged in the API + README + the test-harness skill. Recorded as testing-api-design Part 7 #7 + a roadmap item for the next review session. * docs(positioning): sharpen the pillar-2 wedge against SDK reality The mock-model tier is table stakes (Pydantic/Vercel/LangGraph/LlamaIndex all ship a first-party fake), so it is not the differentiator. Name the three things no SDK mock does — testing the harness as it SHIPS, tool-contract enforcement of the assembled agent (anchored on the live Claude Agent SDK bug #172), and a sub-affordable real-model tier. Also reframe behaviour testing as TWO oracles: the absolute single-arm measure()+judged() (the default when there's no on/off baseline) alongside the relative runEval A/B. * docs(testing): make the absolute behaviour oracle first-class for single skills A/B (runEval + assertSignificant) was over-privileged across the README, docs/harness-testing.md, and the test-harness skill, forcing 'test this one skill behaves right' into a relative on/off mould. Lead instead with the absolute oracle — measure({ checks: [judged(rubric)] }) + assertRates — which needs no baseline and is what promptfoo/DeepEval lead with; reframe A/B as the specialised relative/regression oracle. Resolves testing-api-design Part 7 #7 (docs/skill only, no code change). * docs(testing): declare one canonical import per pillar The docs taught granular concern-named paths (vigiles/run-hook, vigiles/eval, vigiles/check) while the test-harness skill and README use the vigiles/testing barrel — an inconsistent canonical-import story. Declare vigiles/testing and vigiles/linting the canonical entry points (granular paths stay as stable aliases) so new code has one obvious path. * docs(safety): correct overclaimed default + macOS confinement status The ephemeral run environment is shipped but OPT-IN (ephemeralEnv: true, default off) — docs/safety.md read as if it were on by default for every run. State the real status: default-off today, default-on is the committed next step gated on validating the file-based OAuth copy path; the pure pieces and scrubbed-env wiring are unit-tested. Also mark Seatbelt as macOS-planned (not shipped) in the two mechanism tables, so 'Mac deny-all' no longer reads as a working network wall. * docs(roadmap): note per-check thresholds for assertRates The absolute-oracle path (measure + assertRates) is now recommended for testing one skill, but assertRates({ min }) applies one rate floor to all checks. Persist the additive, non-breaking follow-up: let min accept a per-check mapping. * docs: add subagent-model-note rule; promote macOS Seatbelt to Now/P1 - New guidance rule: when launching a subagent, state which model was chosen and why, in one line, at launch — visibility into the cost/capability tradeoff. - Promote the cross-platform (Seatbelt) confinement backend from Next to Now/P1 per maintainer request; note there's no Mac to validate it on, so it needs a macOS CI runner or a Mac-having contributor. * refactor!: make the agnostic surface truly harness-agnostic, enforced by eslint vigiles/testing and vigiles/integration re-exported Claude-Code transport (scriptModel, claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable, loadPlugin, resolveHarness) via blanket `export *`s — so the 'agnostic' barrels weren't actually agnostic. The directory-based boundary lint missed it because src/mock-model.ts (the Anthropic-Messages mock) lives at the composition root, not under src/adapters/. - testing.ts / integration.ts: re-export ONLY the agnostic harness-test names; CC transport now comes from vigiles/claude-code. - eslint: classify src/mock-model.ts as part of the cc-harness element, so the existing agnostic-surface boundary rule forbids any agnostic barrel from re-exporting it (verified: reintroducing the leak now errors). - claude-code.ts: expose claudeCodeDriver + argv/parse helpers + claudeAvailable (beside the Codex driver in vigiles/codex), preserving access off the agnostic surface. BREAKING: scriptModel/loadPlugin/claudeCodeDriver no longer importable from vigiles/testing or vigiles/integration; import them from vigiles/claude-code. * refactor!: trim package.json exports to a canonical public surface Dropped 12 internal/redundant module subpaths that were each fully re-exported by a concern-named barrel: ./compile ./linters ./eval ./harness-test ./harness-assert ./run-hook ./check ./judge ./mock-model ./plugin-loader ./mcp ./tool-stub. The package's `exports` map IS the access wall (Node throws ERR_PACKAGE_PATH_NOT_EXPORTED; TS honors it under node16/bundler), so these are now genuinely internal. Canonical surface (12): . and ./spec (authoring + the module-augmentation target), ./linting, ./testing, ./unit ./integration ./e2e (tiers), ./claude-code ./codex ./adapter (harness), ./vitest ./jest (runners). ./linting now also re-exports the linter engine (was ./linters). BREAKING: the dropped subpaths no longer resolve — import via the canonical barrel (vigiles/testing, vigiles/linting, vigiles/claude-code). * docs: update Key Files descriptions for the trimmed public surface vigiles/plugin-loader and vigiles/check are no longer standalone subpaths; loadPlugin is on vigiles/claude-code and the check vocabulary on vigiles/testing. * docs: sweep imports onto the canonical public surface The trimmed exports removed the granular module subpaths, so rewrite every doc/skill import to the canonical barrel: vigiles/{eval,run-hook,harness-test, harness-assert,check,judge,tool-stub} → vigiles/testing, vigiles/{compile,linters} → vigiles/linting, vigiles/{plugin-loader,mock-model} → vigiles/claude-code. CC-specific transport (scriptModel, loadPlugin) is split out of mixed imports into a vigiles/claude-code line. Also corrected the two now-false 'granular paths keep working' claims (only vigiles/spec remains besides the barrels). * build: add API Extractor surface gate + API Documenter reference Lock the public API surface so it can't silently regrow (the dual of the eslint agnostic-surface rule, at the .d.ts level). scripts/api-extractor.mjs runs API Extractor over all 11 public entry points and writes a committed report per entry (etc/*.api.md) — the surface contract. CI (`node scripts/api-extractor.mjs` in the check job) fails on any drift; `npm run api:report` regenerates after an intentional change. The reports prove the de-leak: vigiles-testing.api.md has zero CC symbols, vigiles-claude-code.api.md has them. `npm run docs:api` runs API Documenter over the doc models to generate a cross-linked Markdown API reference (api-reference/, gitignored — a derived artifact, not committed). Narrative guides stay hand-authored. Adds @microsoft/api-extractor + @microsoft/api-documenter as devDependencies. * docs(roadmap): track moving the CC mock + driver into the adapter dir The glob-classification of src/mock-model.ts as cc-harness enforces the agnostic boundary today; the structural end-state is to relocate it (and claudeCodeDriver) under src/adapters/claude-code/ so the boundary is directory-based, mirroring the Codex adapter. * ci: smoke-check the API reference generator (no publish) Run API Documenter in the check job, reusing the doc models the surface gate already wrote to temp/, so npm run docs:api can't silently break. Output (api-reference/) is gitignored and not published — there's no docs site yet; upgrade this to a Pages deploy when one lands. * test(safety): prove ephemeralEnv actually scrubs — real-subprocess test The one line that enforces ephemeral scrubbing (env: replaceEnv ? scrubbed : merge) lived inside the v8-ignored real-spawn path, so a regression to an always-merge would silently defeat ephemerality with no test catching it. Extract it to a tested seam resolveSpawnEnv() and pin it two ways: a pure test of the replace-vs-overlay decision, and a behavioural test that spawns a REAL node child with the scrubbed env and asserts a planted secret is gone, HOME is the throwaway dir, and the real ~/.ssh is unreachable. Deterministic, no model. Updates the vigiles-testing API report for the new resolveSpawnEnv export. * feat(testing): per-check rate thresholds in assertRates The absolute-oracle path (measure({ checks }) + assertRates) is the recommended way to test one skill, but assertRates({ min }) applied a single floor to every check — you couldn't gate 'skill fires every trial AND judged >= 0.8' in one call. Add an optional per (by check kind) override: assertRates(report, { min, per: { skill: 1.0 } }); the failure message now reports each check's own min. Additive + non-breaking. Updates the vigiles-testing API report. * feat(testing): thread per-check thresholds through checkReportToJUnit assertRates gained a per-check-kind threshold override but checkReportToJUnit still took a single min, so the strict gate and the JUnit report could disagree about which checks failed. Extract one shared checkRateThreshold() helper used by both; checkReportToJUnit now accepts the same { min, per }. Updates the report. * test(safety): prove interceptTools actually PREVENTS, end-to-end The pure decision (decideIntercept/interceptHookDecision) was tested, but the rail's whole point — the real `vigiles intercept-tool-hook` CLI, wired as a PreToolUse hook, BLOCKING a matched call before it runs — was never exercised. A regression in the subcommand's env-read or deny format would pass every pure test. Drive the real built CLI through runHook (no model) and assert a matched `git push` is blocked while a non-matching command and an un-intercepted tool run. Deterministic; skips loudly if dist/ isn't built. * fix(testing): guard two silent-green footguns in measure/assertRates Both are the 'passes green while testing nothing' anti-pattern the no-silent-skips rule forbids: - assertRates on an empty report (measure with no checks) vacuously passed — now throws, pointing at the likely empty checks list. - stubSkillBodies + a judged check: stubbing replaces the skill body with a no-op, so judged graded empty output (docs warned, nothing enforced) — now throws, telling you to drop one. Adds tests for both. * feat(cli): per-line GitHub annotations for broken doc refs The dogfooded GitHub Action surfaces lint findings inline on the PR diff via ::error file=,line=:: workflow annotations. The inline/frontmatter/spec checks already annotated per-line, but the markdown code-block ref check reported to stdout only — so a broken enforce()/file()/cmd() in a doc never showed up inline on the PR, only in the summary blob. Emit a per-line annotation for each docRefReport error (it carries file+line). CI-only; skipped under --json/--summary. Adds a hermetic test. * ci: run the dogfooded vigiles Action as the primary lint gate Move the composite Action (uses: ./) ahead of the redundant direct `node dist/cli.js lint` in the check job, so a lint failure surfaces THROUGH the shipped Action — its per-line annotations (inline on the PR diff) and sticky PR comment — rather than a bare CLI step failing first and skipping it. The direct CLI call stays as a redundant safety net. Aligns with prod-grade-gha-cli (the Action is how CI runs for users). * test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation NOT FOR MERGE — a broken file() ref in docs/cli.md so the dogfooded Action fails and leaves a per-line ::error annotation inline on the PR diff plus a sticky comment. Revert this commit once the GHA behavior is confirmed. * Revert "test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation" This reverts commit 204497f0e6. * refactor(lint): deprecate require-skill-spec (default off), lean on untested-surface Requiring a .spec.ts per SKILL.md was the wrong constraint: skills are legitimately hand-written (the Level-0/1 on-ramp), so the rule only added noise — it nagged about hand-authored, vendored, fixture, and bench skills alike (~24 warnings in this repo, all non-gating). The constraint that actually matters — 'every skill/agent/hook ships with a test or eval' — is already the untested-surface rule. - DEFAULT_RULES: require-skill-spec → false (was warn); aligns with its own documented default and the @deprecated type comment. - --strict no longer promotes it (tightens only require-spec). - Removed from this repo's .vigilesrc.json. - Implementation KEPT — set it explicitly to re-enable (e.g. skills with file()/cmd() refs). - Docs: deprecation banner + untested-surface cross-links; updated tests. * fix(lint): untested-surface fires on all skills/agents — drop the invocation-mode exemption The filter silently exempted command-only (disable-model-invocation) skills from the test requirement, on the reasoning that a *trigger* eval is meaningless for them. But a command-only skill still DOES something when invoked, and that behaviour is worth a test — so the exemption was wrong (and inconsistent: agents were never exempt). Now every skill, agent, and hook is held to the requirement; the ONLY opt-out is an explicit vigiles:ignore-test marker, which is reported as `exempt` so the skip is visible, never silent. - test-coverage.ts: drop the userInvoked filter + the isUserInvoked helper; the Surface gains an `ignored` flag (set by vigiles:ignore-test) so the explicit opt-out is counted, not dropped at discovery. - Remove the now-meaningless includeUserInvokedSkills option (TestCoverageOptions + TestCoverageConfig). - Docs + tests + spec descriptions updated. Surfaces the one real gap it was hiding: skills/migrate-to-spec now flags as untested (linter-docs is content-referenced by skills-dogfood.test.ts). * refactor!: split untested-surface into per-kind untested-skill/agent/hook rules A skill, an agent, and a hook are tested in different ways (a skill's 'does the description fire?', an agent's tool-contract/outcome, a hook's block/allow), so one umbrella rule conflated them. Replace untested-surface with three parallel, independently-configurable rules — untested-skill, untested-agent, untested-hook — each gated by its own severity (set one to false to skip that kind). The detector (findUntestedSurfaces) is unchanged; the CLI now gates per kind. BREAKING: the untested-surface config key is removed. Use untested-skill / untested-agent / untested-hook (all default 'warn', same as before). The TestCoverageConfig skills/agents/hooks toggles are gone (a kind is toggled by setting its rule to false); testGlobs/exclude remain, shared across the three. - types.ts: RulesConfig keys; TestCoverageConfig trimmed. - validate.ts DEFAULT_RULES: the three keys. - cli.ts checkUntestedSurfaces: per-kind severity + gating; fix an unrelated no-deprecated eslint error on the intentional require-skill-spec read. - Docs: untested-surface.md split into three rule docs; all links updated. - .vigilesrc.json: drop the explicit key (defaults apply). --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-18 12:17:34 +07:00
zernie	66ca161e8b	fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals (#35) ... * fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals - untested-surface: discover colocated tests under dot dirs (dot:true), so a `.claude/skills/<name>/<name>.eval.mjs` the warning suggests actually counts as coverage instead of leaving the nudge unsatisfiable. - init: flag a CI workflow that still calls a renamed/removed subcommand (`vigiles audit` → `lint`) even when the Action or a modern command is also present, instead of reporting it "up to date". - orphan-docs: honor an inline `<!-- vigiles-disable orphan-docs -->` opt-out and point the failing message at both escape hatches (inline disable + .vigilesrc.json orphans.exclude/include). - measureTriggerRate: accept a loose `skillsDir` (e.g. .claude/skills) and auto-package it into a throwaway --plugin-dir via packageSkillsDir, so trigger-testing repo-local skills is a one-liner. - docs/strings: replace stale `vigiles audit` command references with `vigiles lint` (schema descriptions, hook/skill prose, CLAUDE.md). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: make strengthen/edit-spec model-invocable, tidy skill packaging, harden agent install Skills are useless if nobody remembers to run them. Address discovery + install gaps surfaced in the dogfood review: - strengthen + edit-spec are now MODEL-INVOCABLE (drop disable-model-invocation) with sharpened, precision-aware descriptions, so the agent reaches for them on its own — "strengthen my rules" / "add a rule to CLAUDE.md" fire the skill. The pre-edit hook already points at edit-spec when it blocks a direct edit; making it model-invocable closes that loop. Each gets a dogfood trigger eval (recall + precision) so auto-firing doesn't become over-firing. - Merge generate-rule into edit-spec (rule classification absorbed) and delete it. Fix stale commands in edit-spec (vigiles check->lint, setup->init, the old `npx skills add` vendoring installer -> marketplace). - linter-docs gets a real SKILL.md (user-invoked reference), fixing the packaging smell of a skills/ dir with no SKILL.md. - Bump .claude-plugin/plugin.json version 3.0.0 -> 4.0.1 (was stale). - Harden installPlugins: on auto-install failure/no-CLI, fail LOUDLY and lead with the shell-runnable CLI commands (an agent can run those) instead of only /plugin slash commands a non-TTY agent can't execute. - Sweep remaining stale `vigiles audit` -> `lint` (example spec). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: remove the renamed `audit` command everywhere; rename lint-command internals `audit` became `lint` in v4 but the old name lingered. Purge it: - cli.ts: rename the lint command's internals (audit()->runLint, AuditReport-> LintReport, auditExitCode->lintExitCode, printAuditSummary->printLintSummary, annotateAuditForGitHub->annotateLintForGitHub) + comments. Keep the REMOVED_SUBCOMMANDS `audit: "lint"` detection key (it matches the OLD name). - FUNCTIONAL: bench scripts invoked `node $CLI audit SKILL.md` (removed cmd) — now `lint`. - docs/research/comments: `vigiles audit` -> `vigiles lint`, and the command- sense word "audit" -> "lint". - Kept genuinely-distinct uses: the post-session AUDIT feature (session.ts), the Merkle "audit trail", the supply-chain audit / scan concept, the dev-only `audit-feedback-loop` skill, and English "you audited the code". - Left third-party vendored fixtures (SHA-pinned) untouched. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: prompt-diversity gate for trigger evals + raise auto-skill recall bar to 0.8 Task: ensure shipped model-invocable skills have evals, and validate prompt sets are big/varied enough before spending tokens. - measureTriggerRate now runs a deterministic pre-flight gate (no model): each prompt set needs >= minPrompts (default 10) and no two prompts more than maxSimilarity (default 0.8) alike via normalized Levenshtein. Throws with an actionable message before any run. Exports levenshtein/promptSimilarity/ checkPromptDiversity/assertPromptDiversity; minPrompts/maxSimilarity are per-spec overridable (narrow skills, short walkthroughs). - Expanded all shipped model-invocable dogfood evals (test-harness, strengthen, edit-spec) to 10 diverse relevant + 10 irrelevant prompts and raised the bar: recall min 0.8, maxFalsePositive 0.3. generate-logo + the canonical template expanded to 10; the OMC walkthrough sets minPrompts: 3. - All shipped skills are covered: the 3 auto skills by trigger evals, the user-invoked (migrate-to-spec, linter-docs) by the load gate. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: use NCD (gzip) instead of Levenshtein for prompt-diversity check Levenshtein measured spelling; switch the prompt-diversity gate to Normalized Compression Distance — the SAME engine `findSimilarRules` uses for near- duplicate rule detection, so the project has one house algorithm for "are these two texts basically the same". NCD scores shared structure/redundancy (a templated prompt with one word swapped compresses together), which is exactly the lazy copy-paste set the gate should reject. - Drop hand-rolled levenshtein/promptSimilarity; add promptDistance (= ncd over normalized text), reusing src/core/proofs.ts ncd(). - TriggerRateSpec: maxSimilarity -> minDistance (default 0.3, the rule-dup threshold). Calibration: real 10-prompt sets sit at NCD >= 0.44; near-dup pairs (one-word swaps, punctuation) at <= 0.24 — clean separation at 0.3. - Tests updated to NCD distance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: stubSkillBodies — measure skill triggering on frontmatter only Trigger-rate is decided by a skill's frontmatter (name + description) before its body ever loads, so the body is causally downstream of selection. stubSkillBodies keeps every skill's frontmatter (the selection competition stays faithful) but replaces each body with a no-op stub, so a trigger run stops AT selection instead of executing an expensive multi-step procedure — far cheaper, faster, side- effect-free, without changing what's measured. ~95% smaller skill payload. - stubSkillBody (pure) + packageSkillsDir({ stub }) (frontmatter-only package). - resolveTriggerPluginDir stubs both paths (pluginDir + skillsDir), preserving the plugin NAME so `<name>:<skill>` still matches the fired predicate. - TriggerRateSpec.stubSkillBodies (default off; on in all trigger evals). - Needs one real-model run to confirm the Skill tool still resolves under a stubbed body (the deterministic transform is fully tested here). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(init): scaffold spec imports only `claude` so it passes strict ESLint The generated CLAUDE.md.spec.ts imported `{ claude, enforce, guidance }` but the enforce()/guidance() examples are commented out, so a repo running `eslint . --max-warnings=0` (common in CI) failed on two no-unused-vars warnings the moment the scaffold was committed. Import only `claude` (the symbol the scaffold actually uses); show the fuller import as a comment for when a real rule is added. Regression test guards the import line. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(init): --force rewrites a stale CI workflow in place Round-2 follow-up: instead of only warning about a stale workflow, `vigiles init --force` fixes it. A renamed/removed subcommand (vigiles audit) is rewritten surgically in place (audit → lint, the rest of the user's workflow preserved); a bare-`npx vigiles` no-op workflow is regenerated from the template. Without --force the behavior is unchanged (warn, don't clobber) and the warning now points at --force. Pure `rewriteRemovedSubcommands` + force plumbed through setup-plan; unit + CLI integration tests. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: positioning for the eval-tier decision + new trigger-rate features - research/eval-api-landscape.md: dated decision — harness eval ≠ model eval (fidelity to the harness-as-shipped is why promptfoo et al. can't host it); keep the deterministic tiers, keep the real-model surface thin (stubSkillBodies = selection not execution), don't rebuild the eval stack, model fidelity is the one on-moat refinement. - CLAUDE.md(.spec): second-pillar paragraph gains the harness-eval-≠-model-eval framing; src/eval.ts Key Files entry documents skillsDir/packageSkillsDir, stubSkillBodies, and the NCD prompt-diversity gate. - docs/harness-testing.md: trigger-rate section documents skillsDir, stubSkillBodies, the diversity gate, and measuring on the user's real model. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(research): testing-API design proposal — the ideal shape Synthesizes "what makes a world-class testing API" (deterministic assertion libs: node:assert/Chai/Vitest/Testing-Library/fast-check; AI-eval: promptfoo/DeepEval/ Inspect) with the "do we need all these layers?" question into one conclusion: - One Trace (already the substrate); a single declarative `check` vocabulary evaluated two ways — strict (throw, unit/integration) vs scored (0-1 ± se, eval). Collapses today's predicate/assert*/matcher 3× into one surface and unifies deterministic + non-deterministic. - Three tiers, not four: e2e is integration + one egress symbol → fold egress in as a capability of integration, keep the capability ladder. - Checks-as-data ⇒ JUnit/baseline/regression + a promptfoo bridge fall out free. - Recommends option C (declarative checks) + D (property-test add-on for hooks). Indexed in CLAUDE.md Key Files + research/README; no users yet, so it's framed as a clean-break design, not a migration. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(orphan-docs): resolve file-relative markdown links; refine tier taxonomy in design doc - orphans.ts: a markdown link is conventionally file-relative ([x](foo.md) in research/README.md → research/foo.md, ../docs/x.md walks up). The detector only matched root-relative paths, so a bare sibling link read as an orphan. Now credits BOTH readings (refTargets resolves against the source file's dir + keeps the root-relative reading). Tests for same-dir and ../ links. - testing-api-design.md: fold in the two taxonomy questions. The layer names are borrowed from app testing, not derived: the harness has only TWO execution scopes (hook = the one standalone-process surface; harness = the whole agent, since skills/CLAUDE.md/settings only act inside a live agent) over a static lint floor. integration/e2e/eval are the SAME scope under realness flags (model mock|real, egress) — model picks the verdict mode (mock→strict, real→scored). "unit only works with hooks" is structural, not a missing capability; each layer adds exactly one capability (the ladder). Recommends runHook + runHarness(flags) over the four borrowed tiers. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 0 — the declarative `vigiles/check` vocabulary First step of the testing-API revamp (research/testing-api-design.md). A check is DATA, not a throwing assert: { kind, eval(target) → {pass, score, message}, toJSON() }, typed over Trace (tool/skill/output/hookFired/wrote) vs HookRunResult (blocked/allowed). One vocabulary, to be evaluated two ways downstream — strict (expect, Phase 1) and scored (measure, Phase 3) — so a check reads as pass/fail on one run AND a rate across trials, and serializes (JUnit/baseline/promptfoo-bridge fall out later). - src/check.ts + src/check.test.ts (pure, model-free; the failure MESSAGE is tested as much as the verdict). - Exported at `vigiles/check` only — NOT folded into vigiles/testing yet, since hookFired collides with the legacy predicate (reconciled in Phase 6). - Additive: nothing removed; 889 tests green. - Build plan appended to the design doc as Part 6 (Phase 0 marked shipped). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 1 — strict evaluator (assertChecks) + check matcher veneer - assertChecks(target, checks): the strict evaluator — throws collecting ALL failed checks' messages (Validation-applicative, not short-circuit). Any runner. - vigilesMatchers gains ONE generic veneer for the whole vocabulary: expect(r).toPass(tool("Bash")) / toPassAll([...]) — each carries the check's own failure message. vitest/jest type augmentations added; cross-runner tests prove toPass works under both. - Additive: the legacy predicate/assert*/matcher surface is untouched (it gets removed in Phase 6). 891 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(harness): Phase 2 — runHarness, the explicit harness-scope entry Establishes the collapsed naming from Part 4: the harness has two execution scopes (hook = runHook, harness = the whole agent); integration/e2e/eval are the ONE harness scope under realness flags. runHarness is the DETERMINISTIC harness run (model: "mock", default) — behaviour identical to runHarnessTest, which it wraps. A real-model harness run is non-deterministic, so a single one can't be asserted: runHarness(spec, { model: "real" }) throws and points at measure() / runEval (the eval scope). Unit-tested (the guard needs no claude). The e2e barrel fold + a materialized real path remain for Phase 6. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phases 3+4 — measure() over checks + serialization (toJUnit, assertRates) Phase 3 — the SCORED evaluator. measure(spec, { trials, checks }) scores a Trace check vocabulary across trials → per-check rate ± se + pass^k, reusing the tested runEvalWith aggregation (one arm). measureWith takes an injected runner (unit- tested, no model); measure wraps it with the real CLI. Defaults to model "sonnet" (measure on the model your users run). So `tool("Bash")` reads pass/fail strictly (assertChecks) AND as a rate here — one vocabulary, two evaluators. Phase 4 — serialization → CI. checkReportToJUnit (each check a <testcase>, fails below `min`) + assertRates (the scored gate) + formatCheckReport, all keyed off checkLabel(check.toJSON()). Because a check is data, JUnit/baseline/promptfoo- bridge fall out. Pure + unit-tested. (Wiring into the Action/baseline is the remaining CI-integration step.) 894 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(run-hook): Phase 5 — propertyHook, invariant testing for a hook's decision A hook is (event) → decision, so instead of a few example events, generate MANY and assert invariants hold for every decision ("destructive command ⇒ blocked", "never both allow and block"). Reuses the deterministic propertyTest from proofs.ts (seeded, shrinks the counterexample; no fast-check dep). The `decide` runner is injectable — pure fake for unit tests, `(e) => runHook(cmd, e)` for a real hook. Returns the shrunk counterexample, not a throw, so it's inspectable. Test: a correct guard satisfies the invariant; a buggy one (blocks only the exact "rm -rf /") gets a "rm -rf foo" counterexample found. 895 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): Phase 6 — front-door the new check surface; defer the mass deletion The new world-class surface (Phases 0-5) is now the documented front door: docs/harness-testing.md gains a "Checks — one vocabulary, two evaluators" section leading with assertChecks / measure / toPass; the design doc marks Phases 0-5 shipped; CLAUDE.md Key Files note runHarness/measure/propertyHook/toPass. DELIBERATELY NOT done: ripping out the ~40 legacy predicate/assert*/matcher helpers + the e2e barrel and rewriting the ~900-test suite/examples off them. On a no-users lib that's high-churn cleanup with no functional gain and real regression risk — the legacy surface stays as working sugar, removed in a single separately- reviewed major-version cleanup (resolving the check↔testing hookFired collision then). The breaking step is the one to take slowly. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor(testing): fold check into testing/unit; collapse e2e into integration The "breaking cleanup" — done at its true (small) blast radius, not the feared 900-test rewrite: - vigiles/check folds into vigiles/testing + vigiles/unit; the check `hookFired` (Check<Trace>) supersedes the legacy boolean predicate via an explicit re-export that wins over the two `export *`s. Checks (tool/skill/output/..., assertChecks, measure, propertyHook) are now first-class at the front door. - e2e collapsed: egress is a CAPABILITY of integration (egressRoutes + runHook's egress:{allow}), not a tier — the old e2e barrel added exactly one symbol. vigiles/e2e stays as a thin deprecated alias so imports keep working. KEPT (deliberately): the bare predicates + per-runner assert*/matcher fronts. They're not redundant — they're context-appropriate surfaces (compose / node:test / vitest-jest) now unified UNDER one check vocabulary. Deleting them would be churn without a greatness gain. 895 tests green, types + cross-runner pass. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(testing-api): Part 7 — greatness re-evaluation + ranked eval-side gaps Post-cleanup assessment: the API is great on the two deciding axes (unification: one Trace / one check vocabulary / two evaluators / serialized; and the moat: the deterministic no-key tiers + pass^k/significance/fidelity nobody else has). Extensibility is strong; the one seam to watch is Check.eval being sync (fine incl. a judge()-backed check; an async scorer would need an AsyncCheck for measure only). Ranked gaps, eval-side first: (1) judged() model-graded check — judge() exists but isn't a Check; fits the sync interface; #1 parity gap; (2) cost/latency/tokens checks (data already captured); (3) checks × A/B arms + significance (unify the harness-A/B moat with measure); (4) multi-surface CC eval — Trace is single+flat, no nested subagent/Task traces, slash-command expansion, multi-turn, or mcp() checks (the deep, new-capability bet); (5) real-tier Trace fidelity (modelRequests mock-only); (6) CC-feature drift maintenance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): close eval-side gaps — judged/cost/latency/tokens/mcp checks, measureArms + significance, harness-agnostic - judged(rubric, { min, judge }) — model-graded check (LLM rubric) via an injectable judge fn (default real judge(), sync via spawnSync), composes into measure/assertRates/toJUnit. Closes the promptfoo llm-rubric / DeepEval GEval gap. - cost/latency/tokens — Check<UsageTrace> over a run's usage; measure.checks widened to Check<RunContext> so they compose with behavioural checks. - mcp(server, tool) — matches the mcp__server__tool tool name. - measureArms(spec, { arms, checks }) + compareCheck(report, baseline, arm, i): the SAME checks scored per arm, with Welch significance (reuses stats.ts) — unifies the harness-A/B moat with the check vocabulary. - Harness axis: a test pins that checks are harness-agnostic (evaluate over a Codex-sparse Trace — output passes, tool/hookFired fail gracefully). Runners dispatch via adapter; measureWith/measureArmsWith take an injectable runner so a Codex eval reuses the same checks. All unit-tested with fakes (no model). 900 tests green. Design doc Part 7 updated: gaps 1-3 + mcp shipped; nested-trace (subagents/turns) is the remaining big bet. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): add the vitest/jest setup + matcher matrix The reference table we'd flagged in the vitest/jest discussion: a two-column setup matrix (vitest vs jest: config / per-file / manual) and a matcher table (toPass / toPassAll / toHaveCreated / toBlock / toBeatBaseline → asserts → over which result), leading with toPass as the one matcher fronting the whole check vocabulary. Makes the runner story scannable instead of prose. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): nested subagent traces — Trace.subagents + subagent() check The deep gap's first piece: a subagent (Task) runs its own session, recoverable because CC tags its events with parent_tool_use_id (= the Task tool call) and the Task input carries subagent_type. parseSubagents groups the tagged tool calls into Trace.subagents (keyed by name); subagent(name, [checks]) runs the WHOLE check vocabulary recursively over that sub-trace, so a test/eval can assert what the subagent DID, not just that Task fired. Wired into makeResult (harness) + makeContext (eval); empty for harnesses that don't emit parent_tool_use_id (Codex) so it stays harness-agnostic. parseSubagents + the check unit-tested on synthetic stream-json; flagged in Part 7 for one real-CC-subagent run to confirm the field names. 902 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): slash-command-expansion (received) + multi-turn (turns) checks Closes the last two pieces of gap #4, both unit-tested: - received(matcher): the expanded slash command (or a hook's injected context) actually REACHED the model — reads modelRequests (mock/harness tier captures them; the eval tier captures none, with an explicit message saying so). - turns({min,max}): the multi-turn observable — the agent took a back-and-forth, not a one-shot (the deterministic harness already scripts multiple model turns via ModelTurn[]; this asserts how many it took). Remaining (smaller, noted in Part 7): driving multiple USER messages (--resume-style, unverifiable here) + per-turn trace slicing. 904 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): validate the subagent parser schema against Claude Code stream-json Confirmed via the Claude Code SDK docs / guide: subagent events carry a top-level `parent_tool_use_id` (= the Task tool-use id), the Task input carries `subagent_type`, linked by tool_use id — exactly what parseSubagents assumes, on the same CLI `message.content` line shape parseToolCalls already consumes (the SDK StreamEvent `event:{delta}` shape is a different, unused format). No code change — the parser is validated; dropped the "needs a real run to confirm" caveat in the comment + Part 7. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(dogfood): exercise the new check capabilities on real targets - checks-dogfood.harness.mjs — RUNS here (claude, no key): assertChecks over a real runHook (blocked/allowed) + a real runHarness mock run (turns/output). Validated end-to-end, not just unit-tested with fakes. - dogfood/skill-quality.eval.mjs — the SCORED path on vigiles's OWN strengthen skill: measure({ checks: [skill, judged, cost] }) + assertRates (does it fire, is the suggestion good, is it cheap). Write-don't-run (needs a key). - subagent.eval.mjs — the nested-trace check on a REAL subagent (vigiles ships none): the vendored oh-my-claudecode code-reviewer, dispatched via Task; subagent("code-reviewer", [tool("Read")]) asserts what the subagent DID. Note: the harness-tier dogfood asserts the cross-version-reliable observables (turns/output); tool-call capture in a bare run is plugin/permission-dependent (the fuller pluginDir tests in harness-test.test.ts cover it). 904 green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: document the mock-tier tool-capture fidelity edge (diagnosed, not a check bug) Diagnosed the flaky tool-call capture from the dogfood: the full stream shows the mock's scripted Write tool_use is never surfaced ("tool_use" absent, only the text turn, plus a spurious rate_limit_event from a LOCAL mock) — a mock↔claude desync with the current version, isolated to a BARE runHarness. The established pluginDir configs capture tools fine (the Skill test runs + passes here). Confirmed it's not permissions (acceptEdits didn't help, reverted), not the parser (tool_use isn't in the stream), and not the check API. Noted in Part 7 gap #5 as a harness-fidelity backlog item; the reliable observables (hooks/turns/output/ received) are unaffected. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): make the full check vocabulary + measureArms discoverable The guide had the Checks intro but not the vocabulary itself. Added a scannable reference table (tool/skill/mcp/subagent/output/received/hookFired/turns/wrote/ judged/cost-latency-tokens/blocked-allowed → holds-when → over-which-result), plus a brief measureArms+compareCheck (A/B with Welch significance) and propertyHook note. So a user can discover the new capabilities, not just read that they exist. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(harness): match subagent dispatch by subagent_type input, not tool name A real `claude` run revealed the subagent dispatch tool is named `Agent` on the live CLI, not `Task` as older docs say. parseSubagents now keys off the `subagent_type` input field (present on the dispatch tool_use regardless of its name) so it survives the rename. Validated end-to-end against real claude output, which recovered the dispatched subagent and its tool calls. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measure for cheap skill-firing checks measure() now takes stubSkillBodies (the same body-stub the trigger tier uses, via a shared stubbedPluginDir helper): a skill-FIRING check (skill()) stops the selected skill at selection instead of running its often-expensive procedure, so a description/firing run costs a fraction of the tokens. Requires pluginDir; throws if combined without one. Not for judged/quality checks — the body is gone. Refactors resolveTriggerPluginDir to reuse the extracted stubbedPluginDir helper. Unit-tested (stubbed body in the packaged plugin, cleanup after, the no-pluginDir guard). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * chore(types): regenerate project-file types to include src/check https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(dogfood): add cheap firing-only eval using measure + stubSkillBodies A shipped dogfood example for the body-stub path: measure({ checks: [skill, latency], stubSkillBodies: true }) on the strengthen skill, the cheap sibling of skill-quality.eval.mjs (which keeps the body to grade quality). Makes the ~18x firing-check saving discoverable, not just unit-tested. Smoke-run against real claude: both checks 100% (skill fires, run stays fast). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measureArms for cheap A/B firing tests Brings measureArms to parity with measure: an A/B firing comparison (does description variant A fire more reliably than B?) now stubs each arm's skill bodies via stubSkillBodies. Every arm that sets a pluginDir is repackaged with bodies stripped (frontmatter kept) through the shared stubbedPluginDir helper, so each run stops at selection — a fraction of the tokens, per-arm. Arms without a pluginDir pass through untouched; throwaways are cleaned up in a finally. Not for judged/quality checks — the body is gone. Unit-tested (both arms see a stubbed throwaway, originals untouched, cleanup after). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(eval): restore 100% coverage for the stubSkillBodies + check surfaces CI's coverage gate (100% lines/functions/statements on the harness-testing files) flagged uncovered paths accumulated across the revamp: the measure/ measureArms stubSkillBodies branches and their helpers (stubbedPluginDir, skillsDirOf .claude/skills fallback, pluginName missing-manifest catch, packageSkillsDir missing-dir throw + non-directory entry skip), checkLabel's no-arg fallback, and the toPass/toPassAll matchers. Adds targeted unit tests for each — all tests still pass, coverage back to 100%. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-16 21:29:30 +07:00
zernie	71b6c4c3d3	fix: finish audit→lint rename — repair CI build, shipped hook, and docs (#34) ... * fix: finish audit→lint rename — repair CI build, shipped hook, and docs The `audit` command was renamed to `lint` in 2c64c62, but the rename never reached CI, the SessionStart hook, the CLI help, or the docs — so the `check` job ran `vigiles audit` ("Unknown command", exit 1) and the build badge went red. - ci.yml: run `vigiles lint` (+ the action dogfood `command: lint`) - hooks/session-start.sh: `vigiles lint --summary` - examples/demo/run.sh: `vigiles lint` (npm run demo was broken too) - src/cli.ts + action.yml: help text / input description say `lint` - docs/harnesses.md: de-stale the "second harness lands" framing (Codex already shipped), trim two over-explained sections, align Pillar 1/2 → Lint/Test - docs/harness-testing.md: trim the demo callout, audit→lint - README.md: paste-prompt as a copyable code block; fold init details into <details> * docs: research core-plus-adapters API design (grounds the pillar-2 relayout) Five-angle survey (AI SDK, unplugin, Drizzle, Knex, Testing Library, ESLint, OpenTelemetry, dependency-cruiser, semver/exports) of how comparable tools lay out a harness-agnostic core + per-target adapters, written to ground the pillar-2 API relayout. Verdict: keep one package with subpath exports; move the agnostic runners out of src/adapters/claude-code to the composition root (non-breaking — exports hide file paths); select by import in the library, string only in the CLI (never resolve a string to a module on a library path); add an eslint boundary rule so an agnostic barrel can't import a specific adapter; parameterize the conformance kit over ADAPTERS; mind the default-param tree-shaking trap. Linked from research/code-adapter-architecture.md + research/README.md. * refactor: route agnostic pillar-2 barrels through composition-root modules Step 1 of the pillar-2 relayout (research/adapter-api-design.md). The harness- agnostic public entries (vigiles/testing, /unit, /integration, /e2e) no longer `export *` from src/adapters/claude-code/* — they route through composition-root modules (src/{harness-test,run-hook,eval,mock-model}.ts), and a new eslint-plugin-boundaries `agnostic-surface` element forbids them from importing any adapter (proven to flag the old pattern). exports map repointed accordingly. Next step relocates the runner *logic* to these composition-root modules so the 100% coverage gate (which already lists them at src/ root) actually covers it. * refactor: relocate harness-testing library to the composition root Step 2 of the pillar-2 relayout (research/adapter-api-design.md). Physically move the deterministic harness-testing modules out of src/adapters/claude-code/ to the composition root src/: harness-test, run-hook, eval, eval-cache, eval-baseline, stats, mock-model, mock-entry, sandbox, egress(+entry/proxy), judge (+ tests). The adapter keeps the ports + bundle + harness glue (dialect, layout, runtime, hook-protocol, model-mock, adapter, agent-*, skill-*, plugin-loader, run-scripts). This makes the 100% coverage gate — whose include list already named these at src/ root — actually cover them: gated statements go 268 → 902 (the gate was near-empty because egress/sandbox/judge were listed at root but lived in the adapter, so the v8 include glob matched nothing). Coverage gaps the now-live gate exposed in egress.ts were closed with real tests, not v8-ignores. Pure relocation + import-path fixes + coverage-include (added stats, eval-cache); no logic or public-export changes. Build, lint (boundaries clean), 863 tests, and the 100% gate all green. * docs: split harness-testing into agnostic core + per-harness guides; sync spec Step 3+4 of the pillar-2 relayout. Docs: docs/harness-testing.md is rewritten as the harness-AGNOSTIC core (retitled "Testing your harness"): leads with select-by-import + { adapter } (default Claude Code), the Trace/predicates/assertions model, runHook/runHarnessTest/runEval shown agnostically, significance + regression gating, and a promoted runner-integration section (vitest/jest is the runner-agnostic seam, and it's tested in CI). The Claude-Code-specific walkthrough (oh-my-claudecode, ${CLAUDE_PLUGIN_ROOT}, the Anthropic mock, sandbox/egress) moves to docs/harness-testing-claude-code.md, and docs/harness-testing-codex.md covers the Codex path (runHarnessTest with codexAdapter against real codex exec; subagents a deliberate non-goal, runEval a documented follow-on). Both linked from the index + harnesses.md (orphan-docs ✓). Spec: CLAUDE.md.spec.ts Key Files repointed to the relocated module paths (src/adapters/claude-code/{harness-test,run-hook,eval,...} → src/...), the architecture section updated to describe the composition-root harness-testing library + the new agnostic-surface boundary rule, and entries added for the two new docs + research/adapter-api-design.md. CLAUDE.md recompiled (hash valid). * docs: fix stale `vigiles audit` references → `vigiles lint` The audit→lint command rename left 16 user-facing docs/examples telling users to run a command that no longer exists. Sweep them: docs/comparison, inline-mode, markdown-mode, spec-format, skills, verifying-instruction-files, testing-matrix, README, docs/rules/* and examples/{demo,frontmatter-CLAUDE}. `vigiles lint` (integrity + orphan-docs + refs) stays green; no command now points at the removed `audit`. * fix: repoint test/example imports to the relocated runner dist paths The step-2 relocation rewired src/ imports but missed consumers outside src/: test/types/smoke.{vitest,jest}.ts and examples/harness/*.mjs (+ dogfood, bench, plugin-test-demo) still imported dist/adapters/claude-code/{harness-test,eval, run-hook,sandbox}.js, which moved to dist/*.js. The CI test:types step (TS2307) and the harness job (ERR_MODULE_NOT_FOUND) caught it — local `npm test` doesn't run those tiers. Repointed every dist/adapters/claude-code/<moved>.js → dist/<moved>.js. * fix: repoint vigiles/judge export + regenerate path-union types Two more relocation leftovers: the package.json "./judge" export still pointed at ./dist/adapters/claude-code/judge.js (judge.ts moved to src/, so the public vigiles/judge subpath was broken), and .vigiles/generated.d.ts still listed the old src/adapters/claude-code/<moved> paths. Repointed ./judge → ./dist/judge.js and regenerated the type union (generate-types --check now passes). --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-15 19:34:08 +05:00