2 Commits

Author	SHA1	Message	Date
zernie	e78841174d	feat: sweep the disaster battery across a whole plugin, and render it honestly (#212) ... * feat: sweep the disaster battery across every hook a repo declares `verifyGuardrail` takes a command string. #211 taught it to honour a hook's `if` condition, but only when the caller passes it — its own comment has to ask. That left "forgot to pass the condition" reachable, and reaching it reproduces the exact false 7/7 #211 existed to kill: a narrow force-push guard certified as stopping `rm -rf /` and `cat ~/.ssh/id_rsa`. `experimental_verifyPluginGuards(dir, opts?)` takes a DIRECTORY instead. It reads the command, event, matcher and condition off the same registration (loadPlugin -> normalizeHooks), so the pairing cannot be made wrongly. Per hook it returns a discriminated union, not a score with nullable fields: measured the battery reached it; results/blocked/allowed/notRun not-applicable another event, or a matcher selecting no battery tool unresolved the command names a variable nothing sets Reading the score of a hook that never ran is a tsc error rather than a 0/7 someone quotes. A repo with no hooks — or none reachable — says so in `notes`, because `hooks: []` on its own reads as a clean bill of health, which is the same false confidence this battery exists to remove. It reports and does not judge: no throwing sibling, because a repo's config never declares which of its hooks is meant to be a guard. `assertBlocksDisasters` stays the gate once you have said so yourself. Harness-agnostic with Claude Code as the default, the runHarnessTest shape — every piece it composes already takes its harness by injection, and a shellHooks:false adapter reports n/a in words rather than an empty pass. Also exports `hookMatcherSelects` from core/hook-matcher.ts so the measured matcher semantics have one home; it fails open in the same direction `decideHookCondition` does, so a wrong answer can only ever say "ran and did not block", never invent a skip. Tested in both directions over a fixture plugin carrying all five shapes, the real vendored davila7 guard, a Codex-shaped repo, and the opencode prototype. Two mutations verified: dropping each hook's own condition brings the 7/7 back (3 assertions red), and ignoring the hook's event turns a Stop hook into a measured 0/7 (4 red). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: render a plugin-guard sweep, without giving an unmeasured hook a number `experimental_verifyPluginGuards` returned a discriminated union and nothing to print it with, so the one use case it exists for — "point the battery at YOUR hooks" — was a six-line fold over that union at every call site. That is a worse code block than the two-liner it replaces, and `cohesive-feature-delivery` row 9 (output parity) could not be claimed. `experimental_formatPluginGuardReport(report)` renders it as terminal text, and its contract is that the union's honesty survives becoming a string: measured prints `blocks n/7` and one row per battery event not-applicable prints its REASON under a ⊘ heading, and no count unresolved likewise notes printed FIRST and in full A rendered `0/7` beside a hook the battery never reached would reintroduce, one layer above the type system, exactly the false confidence the union was built to make unrepresentable. So it is asserted directly and in both directions: for a sweep that measured nothing the output must match no `n/m` at all, and must carry the sentence saying nothing was measured. Mutating the formatter to print `blocks 0/7` for a not-applicable hook fails that assertion and three others. MANY HOOKS stay readable by grouping the unmeasured half BY REASON — a repo with thirty hooks usually has two or three — and naming at most three hooks per reason before counting the rest. That is the only lossy step, and it is confined to the half whose payload is the reason rather than the hook; the measured half is never collapsed, since those are the hooks you came for. The per-event rows come from a shared `guardrailRow`, so this renderer and `formatGuardrailReport` cannot drift into two vocabularies for the same three outcomes; `formatGuardrailReport`'s output is byte-identical. The prefix is on the name because the report SHAPE is documented as likely to move, and a stable formatter pinned to an unstable shape is a lie. It comes off with the same change that takes it off the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: four ways the guard sweep answered for a hook it could not run An automated review of #212 found four, and all four move the same way: toward measuring less. Each has a test that failed before the fix. 1. An UNCOMPILABLE matcher scored 7/7. `hookMatcherSelects` failed open on every uncertain input, reasoning that imprecision can only add a run the harness makes. That reasoning holds where the harness fails open, and this is where it fails closed: the same module already reports `Bash(` as invalid-regex, "the harness can't compile it, so the hook never fires". So the sweep ran the battery against a hook the harness never spawns, and an unconditional-deny body came back blocking all seven disasters — the exact false 7/7 #211 removed, arriving through the matcher instead of the condition. The predicate is now `hookMatcherReach`, three-valued, and an uncompilable matcher is `not-applicable` with the matcher named. 2. Codex matcher semantics were ignored. `HookProtocol.matcherStyle` already says all Codex matchers are regexes; the helper applied Claude Code's literal-equality rule globally, so `ash` — which regex-matches Bash — was classified n/a and never tested. The style is threaded through. 3. Variables the command sets for itself read as unresolved. A `$NAME` regex over shell called two self-contained commands unresolvable: one that assigns the name before expanding it, and one written inside single quotes, where the shell expands nothing. Both are real guards that went unmeasured for a reason untrue of them. Reads now come from the shell's own parser (core/shell-vars.ts, the mvdan-sh parse bash-effects uses); a parse failure falls back to the regex and says so. 4. Adapter-declared event variables were not supplied. `eventEnvVars` exists to say what a synthesized hook event carries, and the sweep synthesizes one, so a Codex hook reading `$hook_event_name` or `$cwd` was unresolvable for lack of a variable the harness would have set. Now populated from the declaration. Deliberately NOT populated: `permission_mode`, `session_id`, `model` — the sweep does not know those, and a guard branching on an invented permission mode is a measurement of a program we made up. The direction is the same in the ambiguous places: refuse to score rather than score optimistically. `*` and `**` on a regex-matcher harness come back uncompilable rather than assumed to be special-cased; an expansion with a default counts as a dependency; a parse failure over-reports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: five ways the guard sweep answered for a hook it never ran Two came from pointing the instrument at a real OSS corpus; three more from an automated review of #212. All five are one class — we run a DIFFERENT program than the harness runs, and record a score anyway — and all five move the same way: refuse rather than answer. Each has a test that failed first. 1. A hook whose SCRIPT IS MISSING scored 7/7. `python3 <missing>.py` exits 2, and 2 is Claude Code's deny code, so a guard that does not exist was certified as stopping every disaster in the battery. Worse than the uncompilable-matcher door fixed last round, which at least needs a malformed config: a relative script path is the commonest hook shape in the wild, and the campaign hit this three times in one run. THE EXIT CODE CANNOT DECIDE IT, and this repo already knew: run-script.ts says so in prose for its own coverage attribution ("`sh <missing>` exits 2 ... indistinguishable from a gate legitimately denying, so it cannot be encoded"). The fact was written down one module away and the sweep scored on it regardless. So the question is asked in two halves that no single signal covers. BEFORE the run, core/command-files.ts names the files the command hands a program to execute, and an absent one is `unresolved` -- no score at all, which is the assertion the mutation targets. AFTER it, a shell exit of 126/127 means the program never launched (missing interpreter, no execute bit, bad shebang); that is the shell's own code, not a language's convention and not a guess about stderr text, and `shellNeverLaunched` is now shared with run-script.ts rather than copied. The pre-flight narrowing is MEASURED, not taste. Over davila7's 107 real hook registrations, "any path-shaped operand" hits 31 with 8 wrong -- files a hook writes or later reads, plus `echo N/A`, which contains a slash -- while "interpreter head OR script extension" hits 23 with 0 wrong. 2. `**` was in MATCH_ALL on no evidence, and that direction MANUFACTURES AN ACCUSATION: a guard registered under it came back "allows 0/7", reported as letting seven disasters through, about a hook the harness never spawned -- the exact mirror of scoring one that could not start. Measured here against claude 2.1.263 with a marker file as the oracle: `*`, `.*` and `""` fire on a Bash call and `**` does not, with `.*` as the in-run control and a hand-run `*` swap ruling out "only the first hook per event fires". Pinned by src/hook-matcher-delivery.test.ts, the shape subagent-delivery already uses: prose cannot notice that someone else's product moved. 3. Variables were checked against `process.env`, which a CONFINED run does not have. `trusted: false` starts from --clearenv and restores only HOME/TMPDIR/PATH plus the caller's env, so a foreign hook reading an ambient variable was measured with it missing. The availability set now follows the execution MODE, decided by the same `decideSandbox` policy the runner uses, and `hookEnvironment` returns three fields because they answer three different questions: what is passed, what the hook will find set, and the subset whose value is known here (a confined HOME points at a sandbox temp dir, so it is present without being resolvable). 4. `layout.projectRootTokens` were never supplied, so `$CLAUDE_PROJECT_DIR` -- what an ordinary project hook reads, including this repo's own -- read as unresolvable while the harness supplies exactly the root being swept. 5. Assignment subtraction ignored execution order, and this one is worth naming plainly: the parser rewrite an hour earlier removed two ways to be wrong and added a third, in the worse direction. Subtracting every assigned name globally excused a read the assignment never reached. Measured against /bin/sh with the name exported first, `echo "$X"; X=1` and `FOO=1 sh -c "echo $FOO"` both print the AMBIENT value -- so both really do read the environment. Two existing tests asserted the opposite; they were the defect written down, and are corrected with the measurement in place of the argument. The rule is now dominance, not membership, and a prefix, subshell or function-body assignment excuses nothing. Where a claim about someone else's product is load-bearing it is pinned by a test against the real binary, not restated: #2 was re-measured here rather than taken from the report that found it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: three more doors to a score for a hook that never ran Review found five; four were real and one measurement had to be redone before the fifth could be. All four are the same class the branch exists to close: `sh` exits 2 both for a hook that legitimately denied and for a hook that never ran, so anything reaching the spawn that should not have comes back as a perfect 7/7. SYNTAX ERROR SCORED AS A BLOCK. `commandFileRefs` returns `{ parsed: false, refs: [] }` for a command the shell parser rejects, and the caller read only `refs` — so an unparseable command looked exactly like a clean one. Measured on the build before this commit: hook `echo "unterminated` → MEASURED blocks=7/7 exits=2,2,2,2,2,2,2 The parse verdict is now decided first, ahead of the variable check too: `shellVarReads` falls back to a regex when the parse fails, so it still named `$FOO` in `echo "$FOO` and the sweep blamed an unset variable for a syntax error, telling the caller to pass `env` for a command no `env` could have saved. PRE-FLIGHT AND RUNNER LOOKED IN DIFFERENT DIRECTORIES. `sandboxedSpawn` chdirs into a `work/` directory it has just created and left empty when no `cwd` is given, while the missing-file check resolved against `process.cwd()`. Same guard, same command, two directories: project dir → exit 0 (the guard's real verdict: allow) fresh empty dir → exit 2 (cannot open its script = DENY = a block) So a guard that allows was reported as blocking, out of nothing but the mismatch. `effectiveCwd` now derives the run's directory from the same policy the runner uses; a relative ref under confinement-without-`cwd` is unresolvable by construction, and an absolute one is still tested on disk because `/` stays ro-bound in there. THE PROJECT IS NOT THE PLUGIN. `$CLAUDE_PROJECT_DIR` and `$CLAUDE_PROJECT` were bound to the swept dir, which is right only while the two coincide. Sweeping an installed plugin against a host project — `dir` the plugin, `cwd` the project — pointed an ordinary project hook at the wrong tree: measured before this commit, a hook reading `$CLAUDE_PROJECT_DIR` came back `unresolved — not on disk here` for a script sitting in the project all along. The plugin-root token still follows `dir`; the project tokens and Codex's `$cwd` follow the run. SOURCE ORDER IS NOT CONTROL FLOW. An assignment excused a later read whenever its offset came first, so `false && MODE=safe; [ "$MODE" = safe ]` suppressed `MODE` although the shell goes on reading the environment — and under confinement the sweep clears it, so it would have measured the other arm of the guard. The denylist of non-persisting assignments is now an allowlist of unconditional ones (a top-level `NAME=value`, or an `export`/`readonly`/`declare`), which makes the unlisted construct default to excusing nothing rather than to excusing everything. The old rule was wrong on four of five measured shapes: `&&`, `if`, `&`, and a pipeline. AN EXTENSION IS NOT A LICENCE TO EXECUTE. `rm -f /tmp/stale.sh` — an ordinary cleanup hook whose file is meant to be absent — was reported as running a missing script, because the extension test ran independently of the head. It is evidence about the FILE, not about the command, so it now requires a head that could execute: a known data-only head silences it, an unknown head still carries it. Re-measured before narrowing, since narrowing that loses a true positive would be worse than the false positive it fixes. The pinned corpus behind the original table (107 registrations, 23 hits, 0 wrong) has moved, so this is a second sample rather than a reproduction: today's davila7/claude-code-templates gives 134 registrations / 120 distinct commands and 5 hits, all true positives reached by an interpreter head or a head that is itself a path — 5 before the clause, 5 after. Both numbers are recorded in the module header, labelled as different corpora. Each fix is pinned by a test proven with a mutation: dropping the parse gate, ignoring confinement in the pre-flight, collapsing project onto plugin, restoring source-order dominance, and removing the head check each fail on their own assertion, with the patch grepped in source and dist. Two existing tests caught the allowlist being too narrow while it was being written — `export NAME=value` is a DeclClause, not a CallExpr — and that gap is now handled and covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: three hand-kept copies of a fact that already had one source Review's second round left five findings. Three are the same defect wearing different clothes — a fact with ONE authoritative source, restated by hand somewhere else, where the copy fell behind — so they are closed as classes rather than patched as places. Two are their own bugs. DERIVE, DON'T RESTATE CONFINEMENT. The sweep kept its own copy of the runner's mode expression and did not know `recordEgress` confines. So a `recordEgress` sweep pre-flighted a relative script against the host's cwd and env, accepted it, and handed it to a run that starts in a fresh empty directory with a cleared environment — where `sh` cannot open the script and exits 2, this harness's DENY code, scored as a block. `routeScriptRun` is now the one decision: `runScriptWith` dispatches on it and owns no policy of its own, and the pre-flight asks it. There is no list here left to fall behind — a future option that selects confinement is added once, in the runner, and both readers inherit it. WRAPPERS. `commandFileRefs` read `args[0]` as the head, so `env python3 guard` named no file at all and the same exit-2 door opened. `stripWrappers` already owns the wrapper list for the effect classifier, so it is exported and ASKED, not copied: env / command / timeout / sudo / nice / xargs / nohup, recursively, skipping each wrapper's own options and `NAME=value` words. NOT fixed, and said out loud in the code and pinned by a test so it is recorded rather than rediscovered: `exec` is a pass-through that list does not carry, and `stripWrappers` has its own recorded remainder (an escaped head behind a wrapper, `sudo g\it push`). Both belong in that module, where the safety matcher reads the same list and a change deserves the measuring a matcher change deserves — not in a private copy here. EXECUTION CWD. The pre-flight tested files against the CALLER's directory while the project-root variables named the swept root: two trees, one run. A guard on disk exactly where its config says was reported `unresolved`; and where the caller happened to hold the same relative path, the sweep ran and SCORED the caller's file under the swept repo's name. `effectiveCwd(opts, root)` answers once, defaults to the swept repository, and the runner is handed that same answer. An explicit `cwd` still overrides — the installed-plugin case, where hooks are READ from the plugin and RUN against a project elsewhere. TWO ORDERING BUGS, THEIR OWN A read inside an assignment's own initializer is never dominated by that assignment: the shell expands the RHS and only then binds the name. Keyed on the assignment's START offset it was, because the walk is preorder — the `Assign` was recorded before its own `ParamExp` was visited. So `MODE="$MODE"; [ "$MODE" = strict ]` reported NO dependency, and under confinement the sweep cleared an ambient value without reporting it unresolved and scored whichever branch the empty value took. Keyed on the END, the containment falls out of the arithmetic instead of needing a special case. `prompt` / `http` / `mcp_tool` / `agent` are real hook actions. `normalizeHooks` drops them correctly — nothing that spawns a shell can drive one — but the sweep read the empty list as "No hooks are declared", which is an accusation about the repository rather than a limit of the tier, written in the one sentence this report exists to get right. They are carried now as `report.unmeasurable` and reported declared-and-NOT-measured in the notes, the census and the rendering; the mixed case is counted too, because a reader who sees six measured hooks cannot otherwise know a seventh went unexamined. MUTATIONS Six, each reverting a fix to its old shape, each failing on its own assertion. Two passed GREEN first, and both were findings about the tests: - `routeScriptRun`'s unit tests pinned the ROUTER, but nothing pinned that the SWEEP asks it, so restoring the hand-kept copy went green. Added. - the cwd fixture's guard DENIED — and `sh missing.sh` also exits 2, so a correct measurement and a total failure to find the file both read 7/7. The fixture now ALLOWS, making the right answer 0/7: the one thing the false 7/7 cannot be confused with. That is this branch's own bug class, found inside its own test. No published number moves. `WRAPPER_HEADS` is untouched (the bash-effects diff is a doc comment and an `export` keyword): 7 seeds, 136 alternate spellings and the 2/7 -> 7/7 dogfood all unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: name the one class five review findings belong to `blocked` is decided by exit code 2, and an interpreter that cannot start its script also exits 2 — so a guard that never ran and a guard that denied are the same observation. The preflight that resolves a hook's script is a proxy for "did it run", and review has defeated that proxy five times. Two are fixed on this branch; three are not, and patching them individually would buy one spelling each while leaving the class open — extracting a script path from an arbitrary shell command is the same undecidable problem bash-effects.ts already documents for itself. Names the three open spellings and points at the fix that closes the class: a control probe, one benign command the guard must allow, run beside the battery. Blocked too means the program is failing to start, not blocking, and the hook is unmeasurable rather than scored. Comment only; no behaviour change. Follow-up: #213 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: a hook rejected by its own condition was scored, not excused Matcher `Bash` with `if: "Bash(terraform apply*)"` reaches the battery by matcher and matches none of it by condition. Every entry in `ran` came back `ran: false`, and the hook was still returned `measured` — so the renderer printed `blocks 0/7` and `sweepNotes`, seeing a measured hook, stayed silent. A guard the harness never spawned, reported as a guard that blocks nothing. That is the same false verdict as the other three non-verdicts and in the unsafe direction, so it gets the same shape: `not-applicable` with its own reason naming the condition, and no score anywhere. The module header said THREE things are not a verdict; the condition is the fourth and it now says so. Decided AFTER the run rather than before, deliberately: `ran: false` IS the runner's own condition verdict, so this reads the decision the harness would make instead of re-deciding it beside `decideHookCondition`. It costs nothing, because a rejected condition never spawns a process either. `ran` is non-empty (an empty `reachable` returned earlier) and a launch failure already returned `unresolved`, so every `ran: false` left is a condition rejection. The sole-hook path falls out: a repo whose only hook was that one now measures nothing, and `sweepNotes` reaches its "none of them reachable / Nothing was measured" branch instead of rendering an empty score. Four tests, and the boundary is pinned in both directions — a condition matching PART of the battery must still score. Mutations: removing the classification reddens three assertions (`expected 'measured' to be 'not-applicable'`); widening `every` to `some` reddens seven, including the partial-condition guard and the vendored davila7 hooks. Also records two review findings as KNOWN REMAINDER (#213) rather than patching them. Both fail toward UNDER-reporting — a real guard reads `unresolved`, nothing is falsely scored — and both were reproduced: an unexpanded `~` probes `<cwd>/~/.claude/…` so a guard present at $HOME is called missing, and `$PWD` is absent from the confined name set though /bin/sh sets it after bubblewrap's --chdir (same command: 7/7 unconfined, `unresolved` confined). Fixing either by resolving harder is the reflex the block already argues against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-09-09 02:21:38 +05:00
zernie	41169e379a	fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129) ... * feat(audit): split Tested into Tested + Evaluated, with a `not measured` state `Tested` collapsed deterministic harnesses and real-model evals into one number, and its finding read "N surfaces with no vigiles test/eval" — one sentence for two prescriptions three orders of magnitude apart in cost. A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on every push, and asks whether a gate still catches what it claims. An eval (`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one question a deterministic read cannot: does this skill fire at all? Folded together, a repo with complete deterministic coverage and no evals scored identically to a repo with neither. - test-coverage: one discovery pass, partitioned by suffix into a harness tier and an eval tier; the union `covered`/`untested` fields are unchanged. Mirrored in the browser-safe twin. - scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and additive; a producer predating the split falls back to the union). - audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both advisory. `Evaluated` has THREE states — a number, a measured `0`, and `not measured` when the executing checks were skipped and no eval exists. Scoring 0 there would report the absence of a check as its result, which is the pattern this tool exists to name elsewhere. In that state the ring names the command that would measure it. - cli: threads the existing execute-skip signal into the ring. - report/HTML/JSON: both rings render; the inventory carries both counts. Grade weighting is unchanged — both rings are advisory and neither enters `reportDeductions`. Audited on this repo before and after: A (100/100) both times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * docs(site): the audit grades six categories, not five Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * fix(lint): a clean union must not read as 'firing verified' Every surface can have a deterministic harness while nothing has ever measured that a skill fires. The gate is unchanged (still the union), but the ✓ now says so instead of implying the question was asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * test(site): regenerate the browser-parity fixture for the Evaluated ring The audit now emits a sixth category, so the Node-generated expected report and the in-browser one differ until the fixture is regenerated — which is what the test's own header prescribes for an intentional engine change. Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the Evaluated ring and its two inventory counts and nothing else: grade and overall are untouched, which is the check that the split did not move the weighting. * fix(coverage): `Tested` measured a STRING, not a test DEFECT. A surface counted as covered when its path or namespace appeared as a SUBSTRING anywhere in any discovered test file (`content-reference`, in src/test-coverage.ts). Two halves, and the second was worse. REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an existing harness file: // probe: skills/argument-arc moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the line put it back. So `Tested` was trivially gameable and the gaming was indistinguishable from real work. The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely asserts over all 21 of its pipeline skills, but builds each path at runtime (`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING. Generality was PENALISED: 21 trivial files each naming their own subject would have scored better than the one harness that actually tests all 21. This is the exact substitution vigiles names in other people's repos — presence of a surface taken for presence of the property, the way `noExplicitAny` in a config is taken for the rule being enforced. The tool was committing it in its own scoring. FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the disk detector and its in-browser twin cannot drift on what decides a grade): 1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring the existing `vigiles:ignore-test` marker convention on the other side. A runtime-path harness can now say what it covers; an accidental mention cannot. Declaration beats inference — the same move as declaring `allowed-tools` instead of inferring capability. 2. PROVENANCE. Every coverage decision now carries HOW it was reached — declared / colocated / mention — and the counts are printed by `formatUntestedReport`, by `formatScanReport`, and carried in `AuditReport.inventory.coverageEvidence` (additive; schema version unchanged). A repo whose coverage rests entirely on substring matches can finally see that about itself; "28 covered" and "28 names that happen to appear in a file" used to print identically. 3. The bare substring detector is KEPT, deliberately, and said so in the code: it is the zero-config path that makes an ordinary `foo.test.ts` naming `skills/foo` count without anyone learning a marker, and dropping it would flip honestly-covered surfaces to "untested" overnight. What it no longer reads is COMMENTS. A path in a comment is prose about a test; a path in executable code is the test reaching for the surface. Prose isn't policy, including ours. It is also now visibly the weakest tier in the report, and when it is the ONLY evidence the report says so in words. Grade weighting is untouched. TESTS prove both halves, each watched failing first: with the fix reverted, "a surface named ONLY in a comment is NOT covered", "a runtime-path harness that DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail (3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE still counts, and a `"https://..."` string literal survives comment-stripping. Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously recommended "add the skill's path in a comment", which was advice to exploit the defect. * fix(site): key the browser-test prebundle to dist, so a stale cache can't lie DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako divergence by comparing an in-browser AuditReport against one generated in Node. `site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on `@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite invalidates on config/lockfile changes, not on the mtimes of an aliased path outside node_modules. So the browser side can compute against a PREVIOUS engine while Node computes against the current one. REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately). Accidentally: an engine change added a report field; the parity test failed showing the field present on the Node side and absent in the browser. I read that aloud as a genuine engine divergence and started hunting a pako bug that does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass. Deliberately, to prove the fix: plant an extra `inventory` field in src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the browser suite with a warm cache. - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1` on one side only — a field-level diff that reads exactly like a real divergence and says nothing about the cache. - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` -> `.vite-engine-a52a27e0137e`. Reverting the plant returns the fixture to its committed bytes. FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/` (`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`), sweeping the directories keyed to other fingerprints so they don't pile up. WHY a key and not the alternatives. Comparing the cache's mtime against `dist/` and forcing a re-bundle, or clearing `.vite` in the test script, both work and both leave the bad state REACHABLE — a comparison someone can get wrong, or a script step bypassed by running `vitest` directly. Keying the directory removes the state instead: a different `dist/` is a different path, so "prebundle built from another engine" cannot be reached, and an unchanged `dist/` still gets a warm cache. By construction rather than by check — the argument this project makes everywhere else about removing a capability rather than policing its use. The test's header instruction ("pako and node:zlib disagree — do NOT paper over it") was correct for one of the two states and actively misleading for the other; it now says which state is no longer reachable and why, and the assertion carries the same note in its failure message. * fix(site): make the parity-fixture generator emit prettier-clean JSON DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with `JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it keeps short arrays exploded across three lines where prettier collapses them. CI gates on `prettier --check .`, so every regeneration of the fixture committed a file that reddened CI later and far from the cause. It just did: branch head dec140a failed the `test` job on exactly this file after the Evaluated-ring change required a regeneration. REPRODUCTION: `node site/scripts/gen-parity-expected.mjs` then `npx prettier --check site/src/demo/__fixtures__/sample-repo.expected.json` → exit 1, six array literals reflowed. FIX: the generator formats through prettier before writing. Chosen over adding the file to `.prettierignore` because the fixture is committed and reviewed, so its diffs should look like every other file's — and because ignoring it would silence the alarm while leaving the state that trips it. Same shape as the stale-prebundle defect this fixture's own test hit: an artifact whose regeneration leaves a bad state is fixed at the regeneration, not downstream. Also adds `site/checks/` to `.prettierignore` — generated per-rule pages from the site prebuild, gitignored by `site/.gitignore`, but not ignored by prettier, so building the site locally reddened `fmt:check` on files nobody commits. Same class as the `site/test-results/` entry already there. `npm run fmt:check` is clean. * fix(cli): refuse unknown flags; `<verb> --help`; nothing-to-audit ≠ grade F DEFECT, in three parts. Every verb parsed its own flags with `args.includes("--x")` and ignored everything else. 1. AN UNKNOWN FLAG WAS SWALLOWED. Measured: $ npx vigiles audit --this-flag-does-not-exist Detected harness: claude-code Harness audit … EXIT=0 A full, confident audit as if the flag had never been typed. Not cosmetic: audit's flags govern what LEAVES the machine — `--no-html`, `--no-json`, `--out=<dir>`, `--serve`. `--no-htlm` wrote the HTML report while the author believed they had suppressed it. For a tool whose thesis is "prose isn't policy", accepting an argument it did not understand and continuing is the same class it names elsewhere: the argument is PRESENT, therefore the property is assumed enforced. The hole was on EVERY verb, not just audit — checked and closed uniformly (`lint`, `test`, `eval`, `compile`, `eject`, `init`, `generate`). `hook-runtime` is deliberately exempt: its argv is emitted into hooks configs by the harness, never typed by a human, so validating it would break the runtime contract. 2. `vigiles audit --help` RAN AN AUDIT. Help existed only on the bare `vigiles --help`. Asking what a flag is called executed the command. Now each verb answers `--help` with its own entry plus its complete flag list, and `printUsage` renders from the same table, so the banner and the per-verb help cannot drift. 3. "FOUND NOTHING" WAS INDISTINGUISHABLE FROM "FOUND SOMETHING BAD". How it bit: `npx vigiles audit --json /some/path`. `--json` is real but takes no value, so the PATH fell through to the positional scan directory. The audit ran over a directory containing no harness and reported "No loadable harness surface — nothing to grade yet", `skills: 0`, grade F, EXIT 0 — byte-wise indistinguishable from a genuine audit of a repo that scores badly. A confident measurement OF THE WRONG OBJECT, in silence. An operator who did not already know the repo had 37 skills would take it as fact. `score.empty` (no instruction file AND no surface) is now a distinct, loud outcome on stderr that says what was scanned, that nothing was measurable, and that a flag may have swallowed the path. No grade is printed for it. Note the flag check does NOT catch that invocation — `--json ./dir` is legitimate — so this branch is what makes it visible, and it covers every other way of pointing at the wrong place. EXIT CODE, chosen against the CLI's existing convention rather than invented: 1 already means "I ran, and the thing you asked about is bad" (lint findings, a failing harness script); 2 already means "I could not do what you asked" (the top-level error handler, `eval`'s refusal-to-fan-out, `generate` with an unknown kind). Both new outcomes are the second. A script that greps for a grade needs "unhealthy" and "I was pointed at the wrong directory" to differ, and the second has no grade at all. `--json` still emits the report for an empty target (it self-describes with `score.empty: true`), so the machine contract is kept. TESTS, each watched failing first. With both guards reverted, 8 of src/scan-cli.test.ts fail (unknown flag on audit, the typo suggestion, the per-verb sweep, `audit --help`, every-verb `--help`, the empty-target outcome, its `--json` form, and the codex-config case below); with them, 48 pass. Plus 7 pure tests in src/cli-flag-check.test.ts, including the over-rejection guard that asserts EVERY flag in the registry is accepted by its verb, and a dogfood test that drives the real `vigiles --help` and fails if it advertises a flag the registry doesn't know. TWO EXISTING TESTS CHANGED, both because the behaviour they pinned WAS the defect: - cli.test.ts "old --pillars / --verify flags no longer scope a pillar" asserted that a REMOVED flag is silently ignored and `init` proceeds with defaults. It now asserts the removed flag is rejected — a cleaner break, and the exact "silently do something other than what was asked" this fixes. - scan-cli.test.ts "honors the .vigilesrc.json `harness` key" audits a CLAUDE.md-only fixture AS CODEX, which finds no AGENTS.md and no codex surface. That is now exit 2 rather than a silent grade F at exit 0; the harness-resolution assertion the test exists for is unchanged. Both `--help`/unknown-flag sweeps run from a THROWAWAY cwd: while proving the fix by reverting it, `init --definitely-not-a-flag` ran a real `vigiles init` and wrote 21 specs, a workflow and a config into the repo. The test now cannot do that to whatever directory it runs in. docs/cli.md documents the rejection, the `=`-only value form, and the exit-code table. * fix(report-view): mirror the split inventory fields, and let CI see the project The Evaluated ring added `untestedHarness` and `unevaluated` to the engine's inventory and not to the view's mirror, so `sample.ts` set two fields the type did not have. `tsc -p site/tsconfig.json` failed the whole time. It went unnoticed because CI runs three typecheck steps — root, `test/types`, and generated types — and none of them covers site/ or packages/report-view. Three green checks, none over the project that broke. That is the same shape as every defect in this PR: a check that exists, passes, and does not cover the thing that failed. So both halves. The fields are declared optional, because a report produced before the split carries neither and the view must still render an older artifact. And CI now typechecks that project, so the next field added on one side of the mirror fails where someone will see it instead of waiting for whoever next runs tsc by hand. Found by the agent that fixed #10-#12 while reporting what it had NOT fixed. * feat(audit): say so when a repo's vigiles skills are unreachable vigiles ships six user-facing skills, and `test-harness` alone answers the question the testing docs are otherwise organized around ("which tier do I want?"). They are wired by the GLOBAL plugin install; `npm install vigiles` ALSO drops them at `node_modules/vigiles/skills/`, which Claude Code never scans. A repo that took the dependency but never ran the plugin install has all six present on disk, unselectable, and silent — observed for a full day in a consumer repo whose author was writing harness tests by hand. `vigiles audit` now checks reachability and warns, following the `dialect-drift` shape: pure parsers + a best-effort local read that never throws + a formatter that returns null when there is nothing to say. No new verb, no new flag. Advisory only, never scored: reachability is a property of the machine, not the repo, and a score that differed between a laptop and CI for identical source would be a lie. The subtle part, and the reason this was misdiagnosed as an npm packaging bug: the authoritative record of `claude plugin install` is the global ~/.claude/plugins/installed_plugins.json, NOT the repo's .claude/settings.json (which carries PROJECT-level enabledPlugins, where a correctly-installed user-scope plugin does not appear). Judged from settings.json alone, a working install reads as broken. Verified live: with this box's real HOME the check is silent because vigiles@vigiles genuinely IS installed; with a fresh HOME the same repo warns. Also pins the trap that started this. A hand-rolled execFileSync runner returns stdout ALONE on success, so advisory output — including vigiles's own compiled hook notice() — vanishes and a healthy react hook reports as dead (hit three times in one repo). runHook already carries both streams and is the right tier for a plain helper script; test-harness now says so, including the honest limit that filesWritten is recorded only under confinement, so assertNoWrite passes vacuously on an unconfined run. Three tests hold that contract. Co-Authored-By: Claude <noreply@anthropic.com> * fix(audit): a project plugin declaration is not an install Investigating whether `init` should wire the plugin per-repo turned up the opposite of the premise, and a bug in the check shipped in a138826. Claude Code's team-marketplace docs, as of CC v2.1.195: "A plugin that only the project's .claude/settings.json enables, and that comes from an external source such as a GitHub repository or npm package, doesn't load until the team member installs it." vigiles ships from a GitHub marketplace, so that is our case. Committing extraKnownMarketplaces + enabledPlugins makes Claude Code PROMPT a collaborator to install; it does not install, and a fresh clone or CI job still has no plugin. Confirmed empirically on one machine: a repo whose committed settings.json declares an external plugin project-level had NO marketplace entry, NO cache dir and NO install record for it in ~/.claude/plugins/, while a plugin installed the normal way on that same machine had all three. Network was not the difference — the second install succeeded there, commit SHA recorded. The bug: the check treated a project `enabledPlugins` entry as proof of reachability, so it would fall silent for a repo that declares the plugin and has never installed it — precisely the state it exists to surface, and precisely the state observed. Project declaration is now a THIRD state, `declaredNotInstalled`, which still warns but with the correct fix line: the install is per machine, and the repo cannot perform it on the collaborator's behalf. Four tests were watched failing against the previous implementation before the fix. Advisory-and-unscored stands, and for a stronger reason than before: this is machine state that no repo-committed file can determine, by deliberate security design — plugins execute arbitrary code with user privileges. Co-Authored-By: Claude <noreply@anthropic.com> * feat(testing)!: split runScript out as the primitive under runHook `runHook` already WAS a general program runner — arbitrary shell command, stdin fed, exit code + both streams + writes + egress back — with a thin hook layer on top. The specialized name owned the general machinery, so someone testing an ordinary helper script had no name to look for. One did: he hand-rolled an execFileSync runner and hit the same bug three times, because execFileSync returns stdout ALONE on success while advisory output (including vigiles's own compiled-hook notice()) goes to stderr, so healthy react hooks reported as dead. The layering is now the right way up: runHook = runScript + event-to-stdin + exit-code-to-decision src/run-script.ts owns spawning, confinement, egress and write recording; none of that is about hooks. src/run-hook.ts is 250 lines of hook protocol. These are two questions, not two ways to ask one: a HOOK has a decision, a SCRIPT has effects. So ScriptRunResult carries no `decision` field rather than an always-undefined one, which would teach the reader the field means nothing. A test asserts its absence, so re-adding it fails. Also fixes a vacuous assertion, which is the same defect class this repo exists to catch in other people's harnesses. `filesWritten` is recorded by diffing the work dir, which only a confined run does — so an unconfined run reported `[]` and `assertNoWrite` returned GREEN having inspected nothing. "Nobody looked" and "nothing happened" are now structurally distinct: `undefined` vs `[]`. assertNoWrite/assertWroteOnly throw on the former and name the fix (`sandbox: "auto"`). Documenting this was too weak; the code distinguishes it now instead of the reader having to remember. BREAKING CHANGE: `HookRunResult.filesWritten` is optional (`readonly string[] | undefined`) — `undefined` when writes were never recorded, rather than a misleading `[]`. Code reading it directly needs `?? []`; code using assertNoWrite/assertWroteOnly gets a real assertion instead of a vacuous pass. Everything else is additive or structurally identical: HookRunResult now extends ScriptRunResult with the same members, and RunHookOptions is Omit<RunScriptOptions, "stdin">. runHook's signature and behaviour are unchanged. Watched failing first: 3 vacuity tests (assert.throws saw no throw — the vacuous pass itself), then 3 deliberate breaks — dropping stderr fails 3 tests, re-adding a decision field fails 1, collapsing unrecorded writes to [] fails 2. Co-Authored-By: Claude <noreply@anthropic.com> * feat(init): declare the plugin in the repo, turning silent absence into a prompt `vigiles init` now merges a two-key declaration into the repo's .claude/settings.json: an `extraKnownMarketplaces.vigiles` source and `enabledPlugins["vigiles@vigiles"]`. WHAT THIS BUYS, precisely: a collaborator who clones and never runs `init` currently gets SILENCE — the npm package is in node_modules, its six skills are unreachable, and nothing says so. That silence cost a full day. With the declaration, Claude Code tells them the project wants this plugin and prints the install command. WHAT IT DOES NOT BUY: the plugin. An external-source plugin declared project-level does not load until each person installs it on their own machine (CC docs, v2.1.195+), and the boundary is deliberate — plugins execute arbitrary code with the user's privileges, so a repo may not install one on your behalf. A fresh clone and a CI job still have no plugin. The README, agent-setup docs and the CLI output all say this in those terms; none of them claim it wires anything up. Nothing is vendored: the entry is a reference. Plugin content stays in the global cache, one copy shared across repos. MERGE, never clobber — that file holds the user's hooks, permissions and other plugins. The merge rules are pure and unit-tested: unrelated keys survive byte-identical; an existing `vigiles` marketplace entry is left alone (it may point at a fork); an explicit `false` is respected as a deliberate disable; the input object is never mutated; a second run is a no-op. Verified end to end against a synthetic repo whose settings.json carries hooks, permissions, a $schema and another marketplace — the diff was exactly the two additions. Invalid JSON is a loud skip, never a rewrite. Gated by the existing `--no-plugin`; skipped in the vigiles repo itself. `removeVigilesDeclaration` ships tested alongside it, deliberately not wired to `eject`: `eject [file]` is the per-file inverse of `compile`, and repo-scoped plugin wiring is not a property of any one file. Reasoning and the two-key manual removal are documented. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-08 14:30:39 +05:00