mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
fix/hook-runtime-unused-err
132 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
e759e1a028 |
fix(hook-runtime): name the thrown cause when a hook fails to load
`main` is red: `catch (err)` at src/hook-runtime.ts:498 never used `err`, so
`@typescript-eslint/no-unused-vars` fails `lint` and the whole `check` job with
it. Arrived in the merge
|
||
|
zernie
|
236615e978 | merge: fix inject hook resilience from fix/inject-hook-resilience | ||
|
zernie
|
b7458a6526 |
fix(ci): remove the root↔site cross-reads the path filter assumes away, and check the assumption (#223)
* fix(ci): delete the root test that read site/ instead of patching around it #219 was a site-only PR that deleted a file a ROOT test read off disk. The `changes` filter set root=false, the root jobs were SKIPPED, the PR merged GREEN and main broke on an ENOENT — a skipped job and a passing job are the same tick in the checks list, so nothing announced it. That was patched twice before anyone asked whether the test should exist. It should not. What it asserted: expect(hero).toMatch(/\bLANGUAGES\b[^;]*from ["']@\/lib\/linters["']/s); expect(heroCode).not.toMatch(/Any language/); Both are properties of the SITE — an import line and a piece of copy — checked by regexing the SOURCE TEXT of two site files from a test about the engine's linter registry. Three defects, only the third visible: it asserts the shape of the text rather than the value, so a rename or a reformat breaks it; a substring search cannot tell an assertion from a comment discussing one, and it had already fired on exactly that (its own comment says so); and it made root depend on site, which is the dependency that produced the green-merge-red-main above. Deleted. The property moves to where it belongs and gets stronger on the way: site/src/lib/linters.browser.test.ts now RECOMPUTES the derivation from the engine and compares values — expect(LINTER_NAMES).toEqual(BUILTIN_LINTERS.map((l) => LINTER_LABELS[l] ?? l)); expect(LANGUAGES).toEqual(LANGUAGE_ORDER.filter((lang) => …)); so a hand-typed array is no longer forbidden, it is made HARMLESS: it must equal what the engine implies and goes red the moment BUILTIN_LINTERS moves under it. That test runs in the `site` job, which is gated on site changes — the job that would have run for #219. With the read gone, the filter's own invariant is true again, and now it is CHECKED rather than asserted in a comment: src/ci-path-filter.test.ts fails if any root test reads site/ off disk. Mutation-proven — reinstating one read fails exactly that test. ⚠️ Two of my own errors on the way here, both mine and both worth the record. The first fix I wrote for this declared the cross-read in ci.yml instead of removing it — treating the symptom a third time. The second was in the guard itself: `grep -rl` exits 1 when it finds nothing and execFileSync throws on a non-zero exit, so the check went RED precisely when the property HELD. It now reads the exit code instead of letting it decide. Gates: npm run check 18/18. npm run coverage 3959 passed, 0 failed; the 99.94% statement gate is src/mock-model.ts:147,339,374, untouched here and green on CI (25 tests skip locally and never reach those lines). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(site): assert the derivation's properties, not a copy of its formula The two tests added in |
||
|
zernie
|
aee337a975 |
fix: strip YAML wrapper quotes on the salvage path before tokenizing (#222)
A REGRESSION shipped in #221, found by the Codex review bot and reproduced on merged main before changing anything: malformed block, tools: "Bash(git status *), Read" -> ["Bash(git", "status", "*)", "Read"] Three phantom tool names. The opening YAML quote was read as a SHELL quote by the new quote-aware depth counter, so the grant's parentheses stopped counting as structure and its own spaces split it — the exact shape #217 argued against, and in the false-EXPOSED direction, since `bashGrantIsUnbounded()` answers "unbounded" when it cannot recognise a grant. The older comma-only parser recovered this input correctly, which is what makes it a regression and not a pre-existing gap. THE CAUSE IS PLACEMENT, NOT LOGIC, so the fix is at the boundary rather than in the tokenizer. On valid YAML js-yaml removes a scalar's surrounding quotes before `splitList` ever runs; only the regex salvage handed them through, so the two paths fed the tokenizer DIFFERENT shapes and only one of them carried YAML syntax the tokenizer had no business interpreting. `salvageList` now strips one MATCHED wrapping pair (parse-don't-validate), and both paths hand `splitList` the same thing. An unbalanced leading quote is left alone rather than guessed at, and still tokenizes. Mutation-proven, patch verified landed: remove the wrapper strip (the shipped regression) -> the new test fails quotes suppress splitting too (the over-fix) -> the new test fails 🔴 AND A STALE CLAIM IN THE TESTS IS CORRECTED IN THE SAME PASS. #221's counterweight comment said its quoted-list row proved "quotes gate the depth counter, not the splitting". That was true only while the salvage path leaked wrapper quotes — this fix removes them, so no quote reaches the splitter on that input and the widening mutation can no longer bite there. The live counterweight is now the unbalanced-quote row, where a lone `"` genuinely does reach the tokenizer. A test whose comment outlives what it proves is the same defect class as the one that let this regression through. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
c4e75ae538 |
fix: read a space-separated tool list, so a working fence stops auditing as ineffective (#221)
* fix: read a space-separated tool list, so a working fence stops auditing as ineffective Claude Code documents three equivalent spellings of `allowed-tools:` / `disallowed-tools:` / a subagent's `tools:` — a comma-separated string, a SPACE-separated string, and a YAML list. `splitList` in core/frontmatter-read.ts split on `,` alone, so the space-separated form arrived as ONE token matching no built-in name. Reported with a reproduction by @vlad-ryzhkov in #217: on a 22-skill harness, changing only the separator moved Safety 81 -> 80 -> 81, reversibly. Reproduced here before changing anything — three of the seven documented shapes came back as a single token: `WebFetch WebSearch`, `Bash(git add *) Bash(git commit *)`, and `"Read Write Glob"` (a YAML-quoted scalar whose VALUE is a space-separated list, which is what most skills in the wild write). TWO SURFACES, AND THE SECOND IS WORSE THAN THE REPORTED ONE. #217 asks about a skill's fence, where `allowed-tools:` only pre-approves, so a mis-split MISLEADS: a fence Claude Code really enforces was reported as closing no lethal-trifecta leg, naming as still-supplied the very tools the author had just denied. But a SUBAGENT's `tools:` is the allowlist the PreToolUse rail denies against, and it reads the same `splitList`. Measured on `tools: Read Grep Glob`, pre-fix: Read DENY · Grep DENY · Bash DENY — the joined token matched nothing, so the agent was denied EVERY tool, including the three it was explicitly granted. #217 raised this path as an open question ("I have not worked out whether anything downstream depends on that"); it did. THE NAIVE FIX IS WORSE THAN THE BUG. `split(/[,\s]+/)` shreds `Bash(git push *)` into `Bash(git`, `push`, `*)`, and `bashGrantIsUnbounded()` answers "unbounded" when it cannot recognise a grant — so a BOUNDED grant would begin reading as an unbounded one. That trades a false-clean verdict for a false-exposed one, which is the worse side of the trade core/lethal-trifecta.ts already reasons about, for a reason the author cannot see anywhere in their file. So this is a tokenizer: whitespace separates only at paren depth 0. Quotes are stripped per token AFTER splitting, never treated as a delimiter — the outer quotes of `"Read Write Glob"` are YAML syntax, and treating them as token boundaries would keep exactly this bug for every skill that quotes its list. An unclosed `(` keeps the remainder as one token rather than fragmenting it into garbage tool names. The tokenizer is the one @vlad-ryzhkov proposed, with his six-row table plus irregular spacing and an unbalanced paren as tests. Every new test is mutation-proven with the patch verified landed: separator back to `,` only (pre-fix) -> the 4 new tests fail whitespace splits at ANY depth (the naive fix) -> the parens test fails One home, so the surfaces cannot drift: `frontmatterList` is the single reader behind scan, adopt, and the rail. Closes #217 Co-Authored-By: vlad-ryzhkov <211849729+vlad-ryzhkov@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: a parenthesis inside a quoted string is a character, not a bracket Follow-up within #217, found by the Codex review bot on the first cut of the tokenizer. The depth counter read every `(` as structure, so a grant carrying a literal unmatched paren in its own command left depth at 1 after the grant closed and swallowed everything after it: tools: Bash(printf '( %s' foo) Read -> ["Bash(printf '( %s' foo) Read"] one token On a subagent that is the same failure this PR exists to fix: the rail denies `Read`, a tool the author granted. Reproduced before changing anything. Quote state now gates the DEPTH COUNTER ONLY, never the splitting. That asymmetry is the fix, not an accident: if being inside quotes also suppressed whitespace splitting, a quoted list would collapse back to one token and reintroduce #217 itself for every skill that writes `allowed-tools: "Read Write Glob"`. Backslash escapes are honoured outside single quotes, per shell rules. MUTATION-PROVEN BOTH WAYS, each with the patch verified landed: depth counted inside quotes (the pre-fix state) -> the new test fails quotes suppress splitting too (the over-fix) -> the new test fails 🔴 AND THE SECOND MUTATION EXPOSED A DEFECT IN THE TEST FIRST. Its original counterweight asserted the quoted list on a VALID YAML block and stayed GREEN under the widening mutation — because js-yaml strips the quotes before `splitList` ever runs, so no quote character was present and the mutation could not bite. Measured, then moved onto a MALFORMED block, where the regex salvage path hands the quotes through: that is the only path on which the property is observable at all. A green mutation is a finding about the test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: vlad-ryzhkov <211849729+vlad-ryzhkov@users.noreply.github.com> |
||
|
zernie
|
25c91c8ea2 |
fix(test): stop reading a site section that #219 deleted (#220)
`src/core/linter-contract.test.ts` asserted that TWO site components read the
BUILTIN_LINTERS-derived lists. One of them, `Wedge.tsx`, was deleted by the
landing-page rework in #219 — the test kept `readFileSync`ing it, so `main`
has been red on ENOENT since that merge.
Measured, not assumed: the file is absent from `origin/main`, `git log
--diff-filter=D` names
|
||
|
zernie
|
848538d888 |
feat(site): sell the test and eval tiers with measured OSS findings (#218)
Render the test and eval tiers as measured findings rather than feature bullets: a real vendored OSS guard's own verdicts through runHook, and measureTriggerRate on superpowers' TDD skill at 83% over 30 runs. Both render from committed fixtures, one regenerated from the engine and pinned by a browser test, the other copied from the lock so CI spends no model quota. Adds a /comparison page rendered from a probe snapshot, retires four unmeasured claims plus two shipped-but-"planned" rows, and type-checks rendered snippets against the real .d.ts instead of scanning them with a regex. Committing this repo's first eval lock also revealed that the CI eval-check step had never once executed — it short-circuited on anyLocksCommitted and exited 0. Two defects behind it are fixed: a quota consent gate that refused a model-free verify, and a dogfood step that now names the one eval holding a lock. |
||
|
zernie
|
45a7690ec2 |
feat: cheap hook-runtime startup, a docs-drift gate, and claims that were measured (#215)
* fix(site): replace unmeasured claims about other tools with a measurement The hero opened with "Nothing checks that the tools, events, files and rules your skills and hooks name actually exist." Nobody had ever run the other tool. Two things were wrong with the sentence: it is a universal negative over every tool that exists, which is unmeasurable, and on its own terms it is false for one of its four nouns. Measured instead, and the probe is committed as tools/measure-validate-overlap.mjs so the next reader reruns it rather than trusting prose. Against Claude Code 2.1.263, seven planted defects, both harness shapes: repo-local .claude/ 0 of 7 flagged, default AND --strict packaged plugin 1 of 7 — "hooks.PreToolUze: unknown hook event" The silence is real silence: the same binary errors on invalid manifest JSON and on a hooks.json missing its root key, and warns on an agent with no description, so it does read these files. That measurement moved claims in BOTH directions, which is the point: - Hero drops the universal negative and describes only what vigiles does. - Wedge credits `claude plugin validate` with the hook-event check it really makes, scoped to plugin form, and stops listing that check among the ones it misses. A draft of this paragraph had it in the wrong column. - docs/rules/hook-script-exists.md said the rule "matches Anthropic's own `claude plugin validate`, making vigiles a superset on the hook surface". Measured false — validate does not check hook-script existence at all. We were under-crediting our own rule. Corrected in place rather than deleted, so it does not read as a wording change and get added back. - Same false credit removed from the subagent-frontmatter keyFile: validate flags neither `model: sonnnet` nor `color: purpel`, and cclint was never run. Every claim of exclusivity, scope or comparison now carries a comment above it with the claim, the command that backs it and the version, what the earlier version said and why it was wrong, and what would invalidate it. The record of the mistake is the part that prevents the repeat. Two live falsehoods found on the page while verifying the above: - The first failure card and the `audit` example both used `event: "Setup"` as a hook event that does not exist. `Setup` IS a real Claude Code event, with its own docs sections; this repo's own catalog learned that on 2026-08-17 and the site never followed. The page's opening example of "config that parses and does the wrong thing" was config that works. Now `PreToolUze`, verified wrong in both directions against the real binary. - The `eval` row quoted the debunked claim as 65%. Three places in this repo and the linked post all say 65%. A debunk that misquotes its own target is the one number a reader checks in ten seconds. Also: the hero's language chip said "Any language · 11 linter catalogs" — an overclaim wrapped around a term a cold visitor cannot gloss, which never told a Python developer that Ruff and Pylint are covered. It now names the languages, still derived from BUILTIN_LINTERS, with a test that fails if a shipped linter has no language decision or a covered language is missing from the display order. Both directions proved by mutation. The landing-site skill is committed in this public repo and carried a "backend (roadmap)" design for an unbuilt capability plus a growth-loop framing, which no-product-strategy-here forbids here. The technical guidance stays; the roadmap and the growth narrative are gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: replace the unmeasured exclusivity claim with the mechanism "No other tool resolves rules across 11 catalog APIs" stood in five places. It rested on a competitor matrix whose cells were checked against DOCUMENTATION rather than against a run — the same matrix that put a false claim on the landing page, corrected in the previous commit after measuring the real binary. Two independent reasons to drop it, not one: 1. It was never measured. Verifying it means enumerating 11 catalog APIs across every comparable tool, which nobody did. 2. This repository is public, and its own `no-product-strategy-here` rule forbids competitive positioning here outright. "The core moat" and "no other tool" are positioning, so those lines were a rule violation independent of whether they happened to be true. Replaced with what the engine actually does, which a reader can check by running it: the rule name is resolved against the linter's own catalog rather than matched as a string, and the enabled state is read from the project's config. The removal is recorded in the source rather than performed silently. A silent deletion reads as a wording change and invites the claim back; the comment in linters.ts says what stood there and why it went. Sites: docs/linter-support.md, docs/what-vigiles-catches.md, src/core/linters.ts, CLAUDE.md.spec.ts, examples/CLAUDE.md.spec.ts (the two CLAUDE.md artifacts recompiled). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(compile): the section size guard measured the wrong unit `DEFAULT_MAX_SECTION_LINES = 200` reads as "a section may not sprawl" and never fired on the section that actually sprawled. Two reasons, both measured on this repo 2026-09-08: 1. The cap counts LINES. A `keyFiles` entry is ONE line of any length, so `## Key Files` reached 173 KB — 59% of the compiled CLAUDE.md — while staying far under 200 lines. Longest single description: 4782 chars. 2. `keyFiles` never reached the check at all. `compileKeyFilesSection` is a separate path that calls `validateFileRef` and nothing else, so the one section with no length control was the one that grew. Both closed: a character budget beside the line budget (whichever runs out first reports), and a per-entry cap on a key-file description, since an entry is a pointer and the reasoning belongs in that file's own header where a reader meets it on opening the file. RATCHETED, not bare. Introduced bare these caps open with 60 findings and get switched off the same day. `.vigiles/keyfile-debt.json` records the existing oversize once; the caps report only what is NEW or has GROWN, and a repo without the file gets every finding, which is the right default for everyone else. Lowering a number is the fix; raising one is the edit the file exists to make visible in review. Verified in three directions, not one: silent on the recorded debt, fires on an entry whose recorded size was lowered, fires on an entry removed from the debt entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(docs): withdraw the unmeasured #30519 matcher claim, and record the probe `docs/compiled-hooks.md` and CLAUDE.md both said the native `if:` matcher misses a compound `cd repo && git push -f`, citing #30519, and offered that as a reason to prefer the AST-backed `runs()`. Measured against Claude Code 2.1.263: the native matcher CAUGHT the compound command, in both spellings. Either the issue was fixed the way #34692 was, or the claim was wrong when written. Withdrawn rather than repeated; a focused re-measure decides. What the same run DID establish is a real gap, and a sharper one: form input hook spawned prefix git push --force origin main YES prefix /usr/bin/git push --force origin main NO glob git push --force origin main YES glob /usr/bin/git push --force origin main NO An ABSOLUTE program head is matched by neither spelling. That holds for `Bash(git push:*)`, the form Anthropic ships in its own security-guidance plugin, as much as for the glob form — so a guard written either way does not run on `/usr/bin/git push --force`, which is the whole guard when the hook body is unconditional. The probe is committed as tools/measure-if-matcher-forms.mjs so the next person re-runs it after a Claude Code minor instead of trusting this note. It measures someone else's matcher, and prose cannot notice when that moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(claude-md): key files are pointers again — 73k tokens to 43k `## Key Files` was 59% of the compiled CLAUDE.md. Its 266 entries had grown from index lines into paragraphs, median 371 characters, longest 4782 — and every one of them is loaded on every turn of every session in this repo. Measured before cutting anything: 220 of the 266 files ALREADY carry the same reasoning in their own header comment, usually in a fuller form. So this is mostly not a move, it is deletion of a duplicate that had drifted from the copy people actually read when they open the file. For the 17 entries whose file had no header, the prose was moved into one rather than dropped; 29 short entries were left alone. compiled CLAUDE.md 291579 -> 173682 chars ~72900 -> ~43400 tokens AND THE RATCHET IS GONE. The cap landed an hour earlier with a `.vigiles/keyfile-debt.json` recording 60 existing violations so it would not open loud enough to be switched off. That was the wrong order of work: a mechanism to tolerate the problem, invented instead of spending one pass fixing it. With the entries pruned the debt file is empty, so both it and the code reading it are deleted and the cap stands bare — 0 findings on a clean tree, fires on a lengthened entry. The section CHARACTER cap goes with it, for the same reason it should never have shipped ratcheted: `positioning` is 24877 characters and would fail it today, and a guard that cannot fire is the defect this product is about. That section is a real and separate piece of work, named here rather than recorded as debt. Verified in both directions on a clean tree: silent at 0 findings, and a mutation that lengthens one description fails on its own assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(tools): record the WIDENED if: forms, which are the ones that are sound The probe measured the two `if:` spellings found in the wild and stopped there, leaving the useful half unmeasured: what a native `if:` would have to look like for a compiler to be allowed to emit one. Measured on Claude Code 2.1.263: emitted if: plain absolute unrelated Bash(git push *--force*) fires NO no Bash(git push:*) fires NO no Bash(*git*) fires fires no Bash(*) fires fires fires The natural translation of a typed predicate is strictly NARROWER than the body it guards — it misses an absolute program head — so emitting it turns a guard off on exactly the input a careful author wrote it for. Widening to the PROGRAM NAME alone keeps every case the body would catch and still filters unrelated commands, which is the property an emission needs: it may only ever add spawns, never remove one. Flags and subcommand refinement are dropped on purpose. Including them is what makes the natural translation too narrow, and the body re-checks them anyway. Left explicitly unmeasured so nobody assumes it: the widening for `touches()` and `pipesToShell()`. Either may have none, and the honest answer for such a primitive is to emit nothing rather than something that looks like a filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(claude-md): the two "layer" paragraphs point at their guides again `## Positioning` was 24891 characters, and two of its twelve paragraphs were 64% of it: Second layer — testing the harness 7859 chars Third layer — COMPILED HOOKS 8108 chars Both are summaries of a document that already exists, is far richer, and was already linked from the same paragraph: docs/harness-testing.md 37114 chars 4.8x the summary docs/compiled-hooks.md 62167 chars 7.8x the summary Same shape as the key-files entries in the previous commit: a copy that duplicates the text people actually read, drifts from it, and is loaded into every session's context whether or not the turn is about hooks. What was KEPT in each is the part that constrains what gets built or claimed, because that has to be in context and cannot live behind a link: the tier discipline (push a question down a tier; only firing and behaviour-movement touch a real model), that the mock-model tier is table stakes and must not be sold as distinctive, the `experimental_*` marking, that compile and verify fix authoring and logic and never delivery, and that a gate is a strong default and never an unbypassable wall. What went to the guides is the mechanics: the tier APIs, the role family, the vocabulary, the stamp. ## Positioning 24891 -> 11361 chars CLAUDE.md 43420 -> 40038 tokens (72894 before this branch) The section CHARACTER cap is still not restored, and this says why rather than leaving it looking forgotten: `## Rules` is 82586 characters and would fail any cap that means anything. Rules are a different problem — measured across 53 of them, median 1449 characters, and unlike a key-files entry or a layer summary there is no richer copy elsewhere to point at, because a rule's text IS the instruction. Trimming it loses information rather than removing a duplicate, so it is judgement, not a mechanical pass, and a 500-char cap would open on 40 of 53 and be switched off the same day. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(hook)!: a bash gate is Bash by construction — delete its match field `HookProgram.match: { tool: string }` was a representable illegal state, and the same false-confidence class the module header says this subsystem exists to eliminate: if (rawEvent.tool_name !== program.match.tool) return allow(); `match: tool("Edit")` type-checked, compiled, wired a PreToolUse matcher `Edit`, fired on edits, built `commandView("")` from a missing `tool_input.command`, found every predicate false and returned allow() — a silently dead guard. The comparison was `!==` while every sibling role routes through `matchesTool`; that runtime/emit disagreement is the one MEASURED and fixed for reacts on 2026-08-12 and documented in this very file's header. It survived on the flagship role. The field carried no information either: `BashToolEvent.tool` is the literal "Bash", and all 34 call sites across both repos wrote `match: tool("Bash")`. So the fix is deletion rather than unification: the matcher moves to `hookRouting`, the runtime keeps a defensive `!== "Bash"` on a literal nobody can mis-set, and the author's file loses a line it could only get wrong. BREAKING CHANGE: `tool` is no longer exported from `vigiles/hook`, and `HookProgram` no longer accepts `match`. Both are part of the compiled-hook vocabulary, whose entry points are `experimental_*`; taken now precisely because the vocabulary is still young. `tools()` — the plural, for file gates and reacts — is untouched, and so is the `tool()` CHECK in `vigiles` (a different function that happens to share the name). Found by an architecture review that was asked to break a different proposal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(hook): restore the CHECK `tool()` a careless sweep took with it The previous commit removed `tool` from the compiled-hook vocabulary, and the regex that cleared its call sites also stripped `tool,` from `src/test.ts` — the `vigiles` ROOT barrel, where a DIFFERENT `tool()` lives: the check from `src/check.ts`, used as `assertChecks(run, [tool("Bash")])`. Two unrelated functions that happen to share a name, and the sweep could not tell them apart. That silently removed a stable public export. Nothing in the change was about it, the commit message did not mention it, and the api-surface diff that would have shown it was regenerated only afterwards. Caught by `docs-imports`, which type-checks the code blocks in `docs/` against the real surface and failed on `docs/testing-api.md:210` importing `tool` from `"vigiles"`. Worth noting which gate did it: the unit tests stayed green, because no test imports that symbol from the root barrel. A doc example was the only consumer standing in for every external one. Also completes the docs half of the previous commit, which shipped without it: `docs/compiled-hooks.md` still imported `tool` from `vigiles/hook` in three places, wrote `match: tool("Bash")` in four, and listed `tool(name)` in the vocabulary — all naming a function that no longer exists. And `api-surface/vigiles-hook.api.md` is regenerated, so the removal is visible as a reviewed diff rather than a surprise. Gates: `npm run check` — the only failure is `cli-lint`, whose 480 errors are all inside other sessions' gitignored worktrees (0 outside; CI never sees them). `docs-imports` 0 findings. `src/hook.test.ts` + `src/core/hook-program.test.ts` 120/120. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(examples): drop the `tool` import the sweep left behind Two example hooks kept importing `tool` from the hook vocabulary after the sweep removed their `match: tool("Bash")` line — and `tool` no longer exists, so the module failed to load: HookCompileError: Cannot load hook "examples/harness/protect-main.hook.mjs": Named export 'tool' not found. `compiled-hook-inprocess.harness.mjs` loads that file BY PATH and asserts over its decisions, so it went red. The runtime behaved correctly — a hook that cannot load must not pass traffic — which is why the failure surfaced as a refusal rather than a silent skip. HOW IT REACHED CI, which is the part worth recording: `npm run check` prints the CI jobs it does not cover, and `harness` was on that list. The change touched hooks throughout. I read the list, decided the harness tier was the known-environment-red one (`claude` 2.1.187 pinned vs 2.1.263 installed) and skipped it — while the note that records that redness says in as many words that redness IN CI is no longer environmental. The printed list did its job; I did not do mine with it. The unit tiers could not have caught this either: `src/hook.test.ts` writes its fixtures as source text and was updated with the sweep, so 120/120 stayed green while a shipped example was unloadable. Verified: `node examples/harness/compiled-hook-inprocess.harness.mjs` → 6/6. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(test): the `tool` import inside a fixture STRING, which no sweep reached `hook-state-store.test.ts:387` expected a throttled gate to stay quiet on a ten-minute-old fact and got `blocked: true`. The gate was not misbehaving — it could not load: const THROTTLED_GATE = `import { experimental_defineHook, tool, state, … }` `tool` was removed two commits ago, so the module threw on import and the runtime failed closed, which is correct behaviour for a gate that cannot run. Same in `hook-load-wedge.test.ts` and `core/hook-program.test.ts`. WHY THIS TOOK THREE ROUNDS TO FINISH, since that is the reusable part. The reference lived in four forms and a regex over source reaches one: real import in an example .mjs caught by the CI `harness` job re-export in a barrel caught by `docs-imports` import inside a template STRING caught by the CI `test` job code block in a doc caught by `docs-imports` That is verbatim the class already written down in this project's own notes after `.claude/pipeline/` was relocated — "four forms of a reference, of which a path search sees ONE" — including its conclusion that the only check which covers all four is one that RESOLVES the reference, not one that greps for it. I could have quoted the rule and still edited by regex and verified by a subset, three times running. Verified this time by the whole suite rather than the files I expected to break. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(test): a FIFTH spelling of the same import — built by concatenation `cli-coverage-record.test.ts` builds its fixture's import line with string concatenation: `import { experimental_defineHook, tool, deny, allow } from ` + JSON.stringify(HOOK_ENTRY) + … so no search for `from "…"` reaches it. Two tests failed because the fixture hook could not load and the runtime failed closed — correct behaviour, stale fixture. That is the fifth form this one removed symbol lived in. The running tally, because the tally is the finding: real import in an example .mjs caught by the CI `harness` job re-export in a barrel caught by `docs-imports` import inside a fixture template caught by the CI `test` job code block in a doc caught by `docs-imports` import assembled by concatenation caught by the full local suite Five spellings, four different gates, and every one of them found by a gate rather than by the sweep that created the problem. The lesson already written down in this project — that a reference search sees one form and only RESOLVING the reference sees all of them — held on every round. Verified: `npx vitest run src/cli-coverage-record.test.ts` 16/16. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: say why the emitted hook block has a matcher and no native `if:` Claude Code lets a hook action carry an `if:` precondition, checked before the hook process spawns. `vigiles compile` does not emit one, and the guide never said why — the first question a reader who knows Claude Code will ask, with no public home. That fails `document-the-why`. The rule the decision rests on: a prefilter may only ever be a proven SUPERSET of the typed predicate. Narrower and the process never starts, nothing is logged, and the guard reads as passing — the silent-guard bug this whole page exists to make unwritable. The safe superset is not a useful filter: `command.runs("git push", {force})` is AST-backed (compound, subshell, pipeline, absolute program head), and the narrowest string condition provably covering that is about `Bash(*git*)`. Same measurement as the "Matcher bypass" row (tools/measure-if-matcher-forms.mjs, 2.1.263). Two smaller reasons: `if:` is Claude Code only (Codex has no equivalent field, so emitting it would make one typed source mean different scopes per harness), and it would not buy the latency it appears to — same-event hooks run concurrently, so a filtered-out hook does not shorten the turn by its own duration. Per-event cost is the runtime's startup, which is where it belongs. Also cross-referenced from the "Runtime cost" limitation, which previously left `if:` looking like the obvious escape hatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: `tool` is no longer in the hook vocabulary — the Status list still named it `tool("Bash")` was removed from `vigiles/hook` earlier today: a bash gate is Bash BY CONSTRUCTION, the matcher is emitted for the author, and every call site in both repos wrote the same literal. The removal swept the imports, the `match: tool("Bash")` lines and the vocabulary bullet — and missed this one. The Status/pending section lists the vocabulary as prose to explain why only the entry points carry the `experimental_` prefix. `tool` sat in that list, naming a symbol that no longer exists, in a section about a different subject. Measured limit this exposes: nothing catches it. self-command-refs resolves `vigiles <cmd>` invocations; doc-refs resolves marks inside fenced ts blocks. A vocabulary word named in PROSE is a form neither resolves, and a grep for the symbol also matches the CHECK `tool()` in docs/testing-api.md, which is a different, live function — so the sweep that would have found it also had to tell two same-named symbols apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: reading an `if:` and emitting one are different acts — say so The new "why no native `if:`" section explained the refusal to EMIT a condition and left a reader free to conclude vigiles ignores the field entirely. That inference is wrong in a way that costs trust in a different feature: the test tiers already honour a hook's own `if:` when measuring it (`decideHookCondition`, src/core/hook-condition.ts, on main since #211), precisely so a guard is scored as the harness would really run it rather than as an unconditional stand-in that flatters it. One paragraph, stated as the asymmetry it is: we read yours, we do not generate one for you, because a generated filter can be narrower than the program behind it and a hand-written one is the author's own declared intent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: cut a paragraph that took five lines to carry one fact "None of this means vigiles ignores `if:`. Reading one and emitting one are different acts." — an aphorism standing in for a point, opening on a negation of an objection nobody raised, then five lines to say one thing. It broke this repo's own prose-clarity rule at the first sentence: lead with the point. Two lines now, the fact first: an `if:` you wrote is honoured, `runHook` reads it, only generating one is refused. Also dropped "a bad trade at any price" from the paragraph above — the preceding sentence already establishes that the failure is silent, so the flourish added emphasis to a point that was already made. Correction to |
||
|
zernie
|
e78841174d |
feat: sweep the disaster battery across a whole plugin, and render it honestly (#212)
* feat: sweep the disaster battery across every hook a repo declares `verifyGuardrail` takes a command string. #211 taught it to honour a hook's `if` condition, but only when the caller passes it — its own comment has to ask. That left "forgot to pass the condition" reachable, and reaching it reproduces the exact false 7/7 #211 existed to kill: a narrow force-push guard certified as stopping `rm -rf /` and `cat ~/.ssh/id_rsa`. `experimental_verifyPluginGuards(dir, opts?)` takes a DIRECTORY instead. It reads the command, event, matcher and condition off the same registration (loadPlugin -> normalizeHooks), so the pairing cannot be made wrongly. Per hook it returns a discriminated union, not a score with nullable fields: measured the battery reached it; results/blocked/allowed/notRun not-applicable another event, or a matcher selecting no battery tool unresolved the command names a variable nothing sets Reading the score of a hook that never ran is a tsc error rather than a 0/7 someone quotes. A repo with no hooks — or none reachable — says so in `notes`, because `hooks: []` on its own reads as a clean bill of health, which is the same false confidence this battery exists to remove. It reports and does not judge: no throwing sibling, because a repo's config never declares which of its hooks is meant to be a guard. `assertBlocksDisasters` stays the gate once you have said so yourself. Harness-agnostic with Claude Code as the default, the runHarnessTest shape — every piece it composes already takes its harness by injection, and a shellHooks:false adapter reports n/a in words rather than an empty pass. Also exports `hookMatcherSelects` from core/hook-matcher.ts so the measured matcher semantics have one home; it fails open in the same direction `decideHookCondition` does, so a wrong answer can only ever say "ran and did not block", never invent a skip. Tested in both directions over a fixture plugin carrying all five shapes, the real vendored davila7 guard, a Codex-shaped repo, and the opencode prototype. Two mutations verified: dropping each hook's own condition brings the 7/7 back (3 assertions red), and ignoring the hook's event turns a Stop hook into a measured 0/7 (4 red). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: render a plugin-guard sweep, without giving an unmeasured hook a number `experimental_verifyPluginGuards` returned a discriminated union and nothing to print it with, so the one use case it exists for — "point the battery at YOUR hooks" — was a six-line fold over that union at every call site. That is a worse code block than the two-liner it replaces, and `cohesive-feature-delivery` row 9 (output parity) could not be claimed. `experimental_formatPluginGuardReport(report)` renders it as terminal text, and its contract is that the union's honesty survives becoming a string: measured prints `blocks n/7` and one row per battery event not-applicable prints its REASON under a ⊘ heading, and no count unresolved likewise notes printed FIRST and in full A rendered `0/7` beside a hook the battery never reached would reintroduce, one layer above the type system, exactly the false confidence the union was built to make unrepresentable. So it is asserted directly and in both directions: for a sweep that measured nothing the output must match no `n/m` at all, and must carry the sentence saying nothing was measured. Mutating the formatter to print `blocks 0/7` for a not-applicable hook fails that assertion and three others. MANY HOOKS stay readable by grouping the unmeasured half BY REASON — a repo with thirty hooks usually has two or three — and naming at most three hooks per reason before counting the rest. That is the only lossy step, and it is confined to the half whose payload is the reason rather than the hook; the measured half is never collapsed, since those are the hooks you came for. The per-event rows come from a shared `guardrailRow`, so this renderer and `formatGuardrailReport` cannot drift into two vocabularies for the same three outcomes; `formatGuardrailReport`'s output is byte-identical. The prefix is on the name because the report SHAPE is documented as likely to move, and a stable formatter pinned to an unstable shape is a lie. It comes off with the same change that takes it off the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: four ways the guard sweep answered for a hook it could not run An automated review of #212 found four, and all four move the same way: toward measuring less. Each has a test that failed before the fix. 1. An UNCOMPILABLE matcher scored 7/7. `hookMatcherSelects` failed open on every uncertain input, reasoning that imprecision can only add a run the harness makes. That reasoning holds where the harness fails open, and this is where it fails closed: the same module already reports `Bash(` as invalid-regex, "the harness can't compile it, so the hook never fires". So the sweep ran the battery against a hook the harness never spawns, and an unconditional-deny body came back blocking all seven disasters — the exact false 7/7 #211 removed, arriving through the matcher instead of the condition. The predicate is now `hookMatcherReach`, three-valued, and an uncompilable matcher is `not-applicable` with the matcher named. 2. Codex matcher semantics were ignored. `HookProtocol.matcherStyle` already says all Codex matchers are regexes; the helper applied Claude Code's literal-equality rule globally, so `ash` — which regex-matches Bash — was classified n/a and never tested. The style is threaded through. 3. Variables the command sets for itself read as unresolved. A `$NAME` regex over shell called two self-contained commands unresolvable: one that assigns the name before expanding it, and one written inside single quotes, where the shell expands nothing. Both are real guards that went unmeasured for a reason untrue of them. Reads now come from the shell's own parser (core/shell-vars.ts, the mvdan-sh parse bash-effects uses); a parse failure falls back to the regex and says so. 4. Adapter-declared event variables were not supplied. `eventEnvVars` exists to say what a synthesized hook event carries, and the sweep synthesizes one, so a Codex hook reading `$hook_event_name` or `$cwd` was unresolvable for lack of a variable the harness would have set. Now populated from the declaration. Deliberately NOT populated: `permission_mode`, `session_id`, `model` — the sweep does not know those, and a guard branching on an invented permission mode is a measurement of a program we made up. The direction is the same in the ambiguous places: refuse to score rather than score optimistically. `*` and `**` on a regex-matcher harness come back uncompilable rather than assumed to be special-cased; an expansion with a default counts as a dependency; a parse failure over-reports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: five ways the guard sweep answered for a hook it never ran Two came from pointing the instrument at a real OSS corpus; three more from an automated review of #212. All five are one class — we run a DIFFERENT program than the harness runs, and record a score anyway — and all five move the same way: refuse rather than answer. Each has a test that failed first. 1. A hook whose SCRIPT IS MISSING scored 7/7. `python3 <missing>.py` exits 2, and 2 is Claude Code's deny code, so a guard that does not exist was certified as stopping every disaster in the battery. Worse than the uncompilable-matcher door fixed last round, which at least needs a malformed config: a relative script path is the commonest hook shape in the wild, and the campaign hit this three times in one run. THE EXIT CODE CANNOT DECIDE IT, and this repo already knew: run-script.ts says so in prose for its own coverage attribution ("`sh <missing>` exits 2 ... indistinguishable from a gate legitimately denying, so it cannot be encoded"). The fact was written down one module away and the sweep scored on it regardless. So the question is asked in two halves that no single signal covers. BEFORE the run, core/command-files.ts names the files the command hands a program to execute, and an absent one is `unresolved` -- no score at all, which is the assertion the mutation targets. AFTER it, a shell exit of 126/127 means the program never launched (missing interpreter, no execute bit, bad shebang); that is the shell's own code, not a language's convention and not a guess about stderr text, and `shellNeverLaunched` is now shared with run-script.ts rather than copied. The pre-flight narrowing is MEASURED, not taste. Over davila7's 107 real hook registrations, "any path-shaped operand" hits 31 with 8 wrong -- files a hook writes or later reads, plus `echo N/A`, which contains a slash -- while "interpreter head OR script extension" hits 23 with 0 wrong. 2. `**` was in MATCH_ALL on no evidence, and that direction MANUFACTURES AN ACCUSATION: a guard registered under it came back "allows 0/7", reported as letting seven disasters through, about a hook the harness never spawned -- the exact mirror of scoring one that could not start. Measured here against claude 2.1.263 with a marker file as the oracle: `*`, `.*` and `""` fire on a Bash call and `**` does not, with `.*` as the in-run control and a hand-run `*` swap ruling out "only the first hook per event fires". Pinned by src/hook-matcher-delivery.test.ts, the shape subagent-delivery already uses: prose cannot notice that someone else's product moved. 3. Variables were checked against `process.env`, which a CONFINED run does not have. `trusted: false` starts from --clearenv and restores only HOME/TMPDIR/PATH plus the caller's env, so a foreign hook reading an ambient variable was measured with it missing. The availability set now follows the execution MODE, decided by the same `decideSandbox` policy the runner uses, and `hookEnvironment` returns three fields because they answer three different questions: what is passed, what the hook will find set, and the subset whose value is known here (a confined HOME points at a sandbox temp dir, so it is present without being resolvable). 4. `layout.projectRootTokens` were never supplied, so `$CLAUDE_PROJECT_DIR` -- what an ordinary project hook reads, including this repo's own -- read as unresolvable while the harness supplies exactly the root being swept. 5. Assignment subtraction ignored execution order, and this one is worth naming plainly: the parser rewrite an hour earlier removed two ways to be wrong and added a third, in the worse direction. Subtracting every assigned name globally excused a read the assignment never reached. Measured against /bin/sh with the name exported first, `echo "$X"; X=1` and `FOO=1 sh -c "echo $FOO"` both print the AMBIENT value -- so both really do read the environment. Two existing tests asserted the opposite; they were the defect written down, and are corrected with the measurement in place of the argument. The rule is now dominance, not membership, and a prefix, subshell or function-body assignment excuses nothing. Where a claim about someone else's product is load-bearing it is pinned by a test against the real binary, not restated: #2 was re-measured here rather than taken from the report that found it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: three more doors to a score for a hook that never ran Review found five; four were real and one measurement had to be redone before the fifth could be. All four are the same class the branch exists to close: `sh` exits 2 both for a hook that legitimately denied and for a hook that never ran, so anything reaching the spawn that should not have comes back as a perfect 7/7. SYNTAX ERROR SCORED AS A BLOCK. `commandFileRefs` returns `{ parsed: false, refs: [] }` for a command the shell parser rejects, and the caller read only `refs` — so an unparseable command looked exactly like a clean one. Measured on the build before this commit: hook `echo "unterminated` → MEASURED blocks=7/7 exits=2,2,2,2,2,2,2 The parse verdict is now decided first, ahead of the variable check too: `shellVarReads` falls back to a regex when the parse fails, so it still named `$FOO` in `echo "$FOO` and the sweep blamed an unset variable for a syntax error, telling the caller to pass `env` for a command no `env` could have saved. PRE-FLIGHT AND RUNNER LOOKED IN DIFFERENT DIRECTORIES. `sandboxedSpawn` chdirs into a `work/` directory it has just created and left empty when no `cwd` is given, while the missing-file check resolved against `process.cwd()`. Same guard, same command, two directories: project dir → exit 0 (the guard's real verdict: allow) fresh empty dir → exit 2 (cannot open its script = DENY = a block) So a guard that allows was reported as blocking, out of nothing but the mismatch. `effectiveCwd` now derives the run's directory from the same policy the runner uses; a relative ref under confinement-without-`cwd` is unresolvable by construction, and an absolute one is still tested on disk because `/` stays ro-bound in there. THE PROJECT IS NOT THE PLUGIN. `$CLAUDE_PROJECT_DIR` and `$CLAUDE_PROJECT` were bound to the swept dir, which is right only while the two coincide. Sweeping an installed plugin against a host project — `dir` the plugin, `cwd` the project — pointed an ordinary project hook at the wrong tree: measured before this commit, a hook reading `$CLAUDE_PROJECT_DIR` came back `unresolved — not on disk here` for a script sitting in the project all along. The plugin-root token still follows `dir`; the project tokens and Codex's `$cwd` follow the run. SOURCE ORDER IS NOT CONTROL FLOW. An assignment excused a later read whenever its offset came first, so `false && MODE=safe; [ "$MODE" = safe ]` suppressed `MODE` although the shell goes on reading the environment — and under confinement the sweep clears it, so it would have measured the other arm of the guard. The denylist of non-persisting assignments is now an allowlist of unconditional ones (a top-level `NAME=value`, or an `export`/`readonly`/`declare`), which makes the unlisted construct default to excusing nothing rather than to excusing everything. The old rule was wrong on four of five measured shapes: `&&`, `if`, `&`, and a pipeline. AN EXTENSION IS NOT A LICENCE TO EXECUTE. `rm -f /tmp/stale.sh` — an ordinary cleanup hook whose file is meant to be absent — was reported as running a missing script, because the extension test ran independently of the head. It is evidence about the FILE, not about the command, so it now requires a head that could execute: a known data-only head silences it, an unknown head still carries it. Re-measured before narrowing, since narrowing that loses a true positive would be worse than the false positive it fixes. The pinned corpus behind the original table (107 registrations, 23 hits, 0 wrong) has moved, so this is a second sample rather than a reproduction: today's davila7/claude-code-templates gives 134 registrations / 120 distinct commands and 5 hits, all true positives reached by an interpreter head or a head that is itself a path — 5 before the clause, 5 after. Both numbers are recorded in the module header, labelled as different corpora. Each fix is pinned by a test proven with a mutation: dropping the parse gate, ignoring confinement in the pre-flight, collapsing project onto plugin, restoring source-order dominance, and removing the head check each fail on their own assertion, with the patch grepped in source and dist. Two existing tests caught the allowlist being too narrow while it was being written — `export NAME=value` is a DeclClause, not a CallExpr — and that gap is now handled and covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: three hand-kept copies of a fact that already had one source Review's second round left five findings. Three are the same defect wearing different clothes — a fact with ONE authoritative source, restated by hand somewhere else, where the copy fell behind — so they are closed as classes rather than patched as places. Two are their own bugs. DERIVE, DON'T RESTATE CONFINEMENT. The sweep kept its own copy of the runner's mode expression and did not know `recordEgress` confines. So a `recordEgress` sweep pre-flighted a relative script against the host's cwd and env, accepted it, and handed it to a run that starts in a fresh empty directory with a cleared environment — where `sh` cannot open the script and exits 2, this harness's DENY code, scored as a block. `routeScriptRun` is now the one decision: `runScriptWith` dispatches on it and owns no policy of its own, and the pre-flight asks it. There is no list here left to fall behind — a future option that selects confinement is added once, in the runner, and both readers inherit it. WRAPPERS. `commandFileRefs` read `args[0]` as the head, so `env python3 guard` named no file at all and the same exit-2 door opened. `stripWrappers` already owns the wrapper list for the effect classifier, so it is exported and ASKED, not copied: env / command / timeout / sudo / nice / xargs / nohup, recursively, skipping each wrapper's own options and `NAME=value` words. NOT fixed, and said out loud in the code and pinned by a test so it is recorded rather than rediscovered: `exec` is a pass-through that list does not carry, and `stripWrappers` has its own recorded remainder (an escaped head behind a wrapper, `sudo g\it push`). Both belong in that module, where the safety matcher reads the same list and a change deserves the measuring a matcher change deserves — not in a private copy here. EXECUTION CWD. The pre-flight tested files against the CALLER's directory while the project-root variables named the swept root: two trees, one run. A guard on disk exactly where its config says was reported `unresolved`; and where the caller happened to hold the same relative path, the sweep ran and SCORED the caller's file under the swept repo's name. `effectiveCwd(opts, root)` answers once, defaults to the swept repository, and the runner is handed that same answer. An explicit `cwd` still overrides — the installed-plugin case, where hooks are READ from the plugin and RUN against a project elsewhere. TWO ORDERING BUGS, THEIR OWN A read inside an assignment's own initializer is never dominated by that assignment: the shell expands the RHS and only then binds the name. Keyed on the assignment's START offset it was, because the walk is preorder — the `Assign` was recorded before its own `ParamExp` was visited. So `MODE="$MODE"; [ "$MODE" = strict ]` reported NO dependency, and under confinement the sweep cleared an ambient value without reporting it unresolved and scored whichever branch the empty value took. Keyed on the END, the containment falls out of the arithmetic instead of needing a special case. `prompt` / `http` / `mcp_tool` / `agent` are real hook actions. `normalizeHooks` drops them correctly — nothing that spawns a shell can drive one — but the sweep read the empty list as "No hooks are declared", which is an accusation about the repository rather than a limit of the tier, written in the one sentence this report exists to get right. They are carried now as `report.unmeasurable` and reported declared-and-NOT-measured in the notes, the census and the rendering; the mixed case is counted too, because a reader who sees six measured hooks cannot otherwise know a seventh went unexamined. MUTATIONS Six, each reverting a fix to its old shape, each failing on its own assertion. Two passed GREEN first, and both were findings about the tests: - `routeScriptRun`'s unit tests pinned the ROUTER, but nothing pinned that the SWEEP asks it, so restoring the hand-kept copy went green. Added. - the cwd fixture's guard DENIED — and `sh missing.sh` also exits 2, so a correct measurement and a total failure to find the file both read 7/7. The fixture now ALLOWS, making the right answer 0/7: the one thing the false 7/7 cannot be confused with. That is this branch's own bug class, found inside its own test. No published number moves. `WRAPPER_HEADS` is untouched (the bash-effects diff is a doc comment and an `export` keyword): 7 seeds, 136 alternate spellings and the 2/7 -> 7/7 dogfood all unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: name the one class five review findings belong to `blocked` is decided by exit code 2, and an interpreter that cannot start its script also exits 2 — so a guard that never ran and a guard that denied are the same observation. The preflight that resolves a hook's script is a proxy for "did it run", and review has defeated that proxy five times. Two are fixed on this branch; three are not, and patching them individually would buy one spelling each while leaving the class open — extracting a script path from an arbitrary shell command is the same undecidable problem bash-effects.ts already documents for itself. Names the three open spellings and points at the fix that closes the class: a control probe, one benign command the guard must allow, run beside the battery. Blocked too means the program is failing to start, not blocking, and the hook is unmeasurable rather than scored. Comment only; no behaviour change. Follow-up: #213 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: a hook rejected by its own condition was scored, not excused Matcher `Bash` with `if: "Bash(terraform apply*)"` reaches the battery by matcher and matches none of it by condition. Every entry in `ran` came back `ran: false`, and the hook was still returned `measured` — so the renderer printed `blocks 0/7` and `sweepNotes`, seeing a measured hook, stayed silent. A guard the harness never spawned, reported as a guard that blocks nothing. That is the same false verdict as the other three non-verdicts and in the unsafe direction, so it gets the same shape: `not-applicable` with its own reason naming the condition, and no score anywhere. The module header said THREE things are not a verdict; the condition is the fourth and it now says so. Decided AFTER the run rather than before, deliberately: `ran: false` IS the runner's own condition verdict, so this reads the decision the harness would make instead of re-deciding it beside `decideHookCondition`. It costs nothing, because a rejected condition never spawns a process either. `ran` is non-empty (an empty `reachable` returned earlier) and a launch failure already returned `unresolved`, so every `ran: false` left is a condition rejection. The sole-hook path falls out: a repo whose only hook was that one now measures nothing, and `sweepNotes` reaches its "none of them reachable / Nothing was measured" branch instead of rendering an empty score. Four tests, and the boundary is pinned in both directions — a condition matching PART of the battery must still score. Mutations: removing the classification reddens three assertions (`expected 'measured' to be 'not-applicable'`); widening `every` to `some` reddens seven, including the partial-condition guard and the vendored davila7 hooks. Also records two review findings as KNOWN REMAINDER (#213) rather than patching them. Both fail toward UNDER-reporting — a real guard reads `unresolved`, nothing is falsely scored — and both were reproduced: an unexpanded `~` probes `<cwd>/~/.claude/…` so a guard present at $HOME is called missing, and `$PWD` is absent from the confined name set though /bin/sh sets it after bubblewrap's --chdir (same command: 7/7 unconfined, `unresolved` confined). Fixing either by resolving harder is the reflex the block already argues against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
6349d2c123 |
fix: three places where a function promised more than it could see (#211)
verifyGuardrail certified guards it never ran. Claude Code hooks carry an `if:` permission-rule pattern deciding whether the hook runs at all; vigiles never read it, so a real vendored hook whose body is an unconditional deny scored 7/7 blocked — certifying it stops rm -rf /, git reset --hard, an SSH-key read and curl | sh. It now reports 1/7 blocked and 6 not run, each with its reason. The condition is evaluated rather than reported unknown because Claude Code's own matcher fails open, so mirroring that asymmetry can only add a run, never invent a skip. clear() erased other hooks' state. Hooks in one directory deliberately share a store — that is what lets one read a fact another recorded — so removing the directory reset unrelated hooks. It now deletes only entries whose `by` names this hook, verified to agree between the runtime writer and seed() across three path shapes. The site showed numbers nobody measured: a blurred `recall 92% · precision 100%` sat beside a real grade of the repo a visitor types in, two screens above copy mocking unverified numbers in other people's READMEs. Replaced with em-dash placeholders, as this repo's own landing-site skill had already ruled in writing. Wedge no longer claims breadth that `claude plugin validate` covers. Each fix carries a mutation proof with the patch grepped in source and in dist before the run. |
||
|
zernie
|
195c1e14b7 |
feat: launch blockers, a hook-state test API, and the site's missing proof (#210)
Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes. Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody. Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:. Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/. |
||
|
zernie
|
7dacd8f402 |
docs: finish the #34692 correction, and give every site the same narrow scope (#196)
The subagent-bypass fix was recorded in one place and left stale in four others, so the docs understated the product while contradicting themselves. Corrected — these stated the old limit as CURRENT: - CLAUDE.md keyFiles for hook-program.ts, guardrail-check.ts and guards.ts, each contradicting the already-corrected header of the very file it describes. - docs/compiled-hooks.md's cons summary listed "delivery (#34692)" as a live limit eight lines above the section explaining it was fixed. Scope added where the correction had none. src/subagent-delivery.test.ts measures headless `claude -p` only; interactive is unmeasured and depth-2 nesting does not occur there. STABILITY.md, docs/cli.md, src/hook.ts, src/core/hook-program.ts and src/guardrail-check.ts said "fixed as of 2.1.241" without that limit — one overclaim replacing another. Unchanged because it still holds: a model can route around a tool entirely (#45427 / #32376), so a gate is a strong default, never an unbypassable wall. Every dated record of what was once true stays. Not refactored, flagged only: agent-runtime.ts reconstructs the active subagent from a dispatch-time stack, while the event now carries `agent_type` directly — a simplification opportunity in a rail already marked experimental and parked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
f52fa12c4f |
fix(lint): spec-refs reaches every compiled surface, and compile emits prettier-clean artifacts (#195)
Closes #190. spec-refs re-derives a compiled file's references from its spec, but called compileClaude directly and skipped every spec whose _specType was not "claude". Four of the five compiled surfaces — skills, subagents, railway orchestrators, generated harness files — got no re-derivation at lint time, so an artifact committed while its refs were live stayed green forever after the target was deleted. Measured here: compile reported `Skill not found` on examples/SKILL.md while lint reported `hash valid`. Smaller than the issue estimated. #190 expected a per-type ref accessor because each compiler reaches its refs differently; they do not — all four already return the same errors: CompileError[], and only the options differ. One exhaustive dispatcher, so a fifth spec type is a tsc error rather than a silent skip. compile also emitted artifacts that prettier --check then rejected, on all seven examples/ artifacts — which is why they sat un-recompiled carrying a retired marker position, and which blocked the fix above. Two independent defects: both frontmatter renderers padded inside the --- fence, and the frontmatter/body join added a blank line placeIntegrityHeader then added again. Fixed at the join, not in the stamper: seal hashes the content it receives, so trimming there hashes one string and writes another (measured — it broke integrity on three artifacts). Also: check.mjs now declares and prints the CI jobs it does NOT run, with a test asserting that list against ci.yml. It covered one job of six while its name and its "18/18 passed" read like all of them, and that cost a cycle. The test earned itself immediately — the hand-written list missed e2e and named two commands that do not exist. Also: @openai/codex is pinned. latest was 0.153.1, whose tarball 404s while its metadata still resolves, so the install step failed on every branch until pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
d6b1ad06aa |
fix!: exclude reaches every pass, and the guard stops missing re-spelled commands (#194)
`.vigilesrc.json#exclude` was honoured by 3 of 11 discovery walks, and the walks that did honour it disagreed on the dialect: glob's string `ignore` treated bare `bench` and `bench/` as matching nothing, while the minimatch helper accepted them. One `ExcludeSet` is now parsed where `loadConfig()` runs and taken as a REQUIRED parameter by every in-scope walk in cli.ts — the shape that let `findSpecs` go a year without it was optional-with-default, which compiles silently. Closes #192. The disaster battery now generates its own inputs: `experimental_alternateSpellings` (renamed from `equivalentDisasters`) re-spells hook test cases so the shell executes them identically — nine families, 7 seeds to 136 spellings. Five of those came from a normalizer fix: `g\it push --force` runs git but was not matched by `runs("git push")`. That fix then had to be scoped, because CI caught the regression it caused: escapes are resolved for the HEAD and FLAG-shaped operands (what the command IS) and never for a path operand (where it WRITES) — the same bytes are a resolved escape to `sh` and a real Windows UNC path on Windows, and fail-closed keeps the path. Both directions pinned by mutation. Also: `spec-refs` and `duplicate-rules` had a meta and a doc but no row in the canonical rules matrix, and nothing compared either against it. A test now asserts every rule has a matrix ROW (matched on the table row, not a prose mention). BREAKING CHANGE: `findSpecs` changed signature; `orphans` gained a `repoExclude` option; `equivalentDisasters` was renamed to `experimental_alternateSpellings` and now takes a required argument. Any repo with an `exclude` key sees it applied to every pass, so an excluded spec is no longer compiled and an excluded harness/eval script is no longer discovered. |
||
|
zernie
|
4f0e6e8037 |
feat(eval): pin --effort as a measurement input, hashed in lock and cache (#191)
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7693c71b96 |
fix(compile)!: a failed compile has no stamp to write — in all five compilers (#188)
Two constructions, and one live bug that a red-team of #186/#187 turned up. **THE BUG: #173 was fixed in one compiler out of five.** `compileClaudeToFile` stopped writing on error. The identical three lines — write, THEN check errors — sat unchanged in the skill, subagent, railway and generator compilers, and all four stamped unconditionally. Worse than the original: `spec-refs` skips non-`claude` specs, so there was no lint-side backstop either. Reproduced on the built CLI: $ vigiles compile skills/demo/SKILL.md.spec.ts ✗ [stale-file] File not found: "docs/never-existed.md" skills/demo/SKILL.md — 256 bytes, stamped $ vigiles lint . ✓ skills/demo/SKILL.md — hash valid # exit 0 So the comment closing #173 was right about `CLAUDE.md` and wrong about scope. Guarding four more write sites would leave the class writable — the sixth compiler gets written the same way. `addHash` now mints a branded `StampedMarkdown`, `seal()` hands it out only when `errors` is empty, and `writeArtifact` accepts nothing else. `markdown` stays a plain string because `adoptDiff` legitimately wants the draft to diff against even when a ref is stale. An erroring compile has no stamp to give the writer. **Two constructions from the earlier analysis, kept:** `RepoRelativePath` — an out-of-repo `.gitignore` entry (#176.8) cannot be constructed, so the write-site guard is deleted rather than duplicated. `Record<BlockMechanism, …>` — the block mechanisms are a closed table. `decideHook` was three `||` terms while `HookOutput` declared a fourth nothing read (#174); a `||` chain has no shape that can be incomplete. `blockedBy` reports what fired and `haltsTurn` derives from it, so the next mechanism needs no new boolean. Also removed a stale `continue` in `rule-meta.test.ts` — `orphan-docs` was exempted as "built-in, not in RulesConfig", true when written and false since it moved into `RulesConfig`, and it excused the rule from the severity comparison. **A third construction was written, measured, and REMOVED.** A textual check for "every declared rule's severity is read somewhere" — the #181 class. Five iterations, each fixing what the last missed, and the third is why it is gone: with every real read of `orphan-docs` deleted, the suite stayed GREEN, kept alive by an unrelated `new Set(["orphan-docs"])` in `linters.ts`. It was green-and-dead for the rule that motivated it. The fourth iteration's own comment-stripper then ate real code, because `cli.ts` holds glob strings whose slash-star opens a comment that closes somewhere else. The reasoning is recorded in the file, along with the structural fix that belongs in its own PR: the ~30 `check*` functions already share one signature and are a registry nobody wrote down. `Record<RuleName, RuleRunner>` makes a rule without a runner a tsc error and collapses `lintExitCode`'s hand-maintained 31-line `||` list — which no textual check can see anyway. BREAKING CHANGE: `addHash` returns `StampedMarkdown` rather than `string`, and the four compile results gain a required `artifact`. Reading a result is unaffected; code that constructs one must supply it. Four mutations, each landing and failing on its own assertion: write-before-check restored in the skill compiler; `seal` handing out a stamp on errors; `repoRelative` no longer refusing escapes; a mechanism dropped from the table. ⚠️ The first of those came back green on its first run — it landed on the first of four identical blocks rather than the skill compiler. Third time this session that "prove the patch landed" earned its keep. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
85c54e7e78 |
fix(lint)!: make the lint report complete, tierable and machine-readable (#187)
Closes #181, closes #182, closes #183, closes #184, closes #185, and the residual half of #173. #185 — every per-surface check read ONE root, so a monorepo with `skills/` plus `plugins/*/skills/` scored only the first: 4 skills over budget, 2 reported, exit 0. The default now NAMES the bundles it did not score; `--bundles=all` scores them in one pass with one exit code. Root-only stays the default so a vendored third-party corpus is not scored as if it were yours. #181 — `orphan-docs` had a documented severity nothing read, and the duplicate-rules check had no id at all, so neither could be tiered. Both are real rules now and the exit code respects their severity. #183 — the log and `--json` disagreed (21 lines against 88 findings) because some checks print per finding and others per group. One computed total, printed and carried in `--json`. #182 — `--json-out=<file>` writes the JSON while stdout stays human-readable, so a CI job stops scanning twice. #184 — `vigiles test` resolves a harness's bare `vigiles` import from the CLI's own install, so `npx vigiles@<version> test` needs nothing in the project. NODE_PATH was measured, not assumed: Node ignores it for ESM. #173 residual — `lint` verified the integrity HASH, which says nothing about whether the things a file NAMES still exist, so an artifact committed while its refs were live stayed green after the target was deleted. The new `spec-refs` rule re-derives them. It immediately found four dead references in this repo's own `examples/CLAUDE.md.spec.ts`, left behind by the hexagonal refactor. BREAKING CHANGE: `orphan-docs` and `duplicate-rules` now default to "warn" — a repo relying on either to fail CI must set "error" explicitly. Registering them surfaced that both are heuristic proxies, and this repo's calibration rule is that a heuristic never defaults to error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
94af23a5b7 |
feat(hook,cli)!: close five adopter-reported issues — halted-turn guards, dead-ref artifacts, centralized tests (#186)
Closes #173, #174, #175, #176 and #170. #174 — decideHook read exit 2, `decision` and `permissionDecision` but not the harness's halt-the-turn field, so `assertBlocksDisasters` reported a guard that stopped every disaster as blocking none. The field is read from the PORT, not hard-coded: `continue` is documented for Claude Code and unverified for Codex. The issue's adjacent suggestion was deliberately NOT implemented — adding the halt to `hook-block-ineffective` as a block-attempt signal would fire on hooks that WORK, since a halt is not event-scoped. It reads as a suppressor instead, fixing a real false alarm. A mutation implementing the original suggestion literally is pinned by a test that fails on it. #173 — compile printed its errors, exited 1, and wrote a hash-valid CLAUDE.md anyway, so `lint` went green over refs it had just called dead. Nothing is written on a failed compile now. #176 — eight small defects, including `audit --out` outside the repo appending unreachable `../../..` entries to .gitignore (a documented read-only report editing a tracked file), an unknown command exiting 1 against a documented 2, and STABILITY.md promising the removed `scan` verb. #175 + #176.2 — three missing capabilities, each opt-in and additive: `{surface}` in testGlobs, `--single`, and `rulesDir` so `.claude/rules/**` stops being invisible. #170 — one enumeration of export forms written against the grammar, both naming checks driven off it, and `check-internal-tag.mjs` asking the TypeScript compiler instead of a regex. It immediately found the issue's open case, 13 tagged types the regex had never seen, and a live gap in the ESLint rule. BREAKING CHANGE: `HookRunResult` gains a required `haltsTurn`, so code that CONSTRUCTS one (a test fake) must set it. Reading a result is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
1df264ce6b |
feat(cli)!: load specs in a vigiles-owned spec host, not a two-loader guess (#178)
Replaces the native-import/npx-tsx pair with ONE loader: a child process that registers vigiles' own module hooks and streams results as NDJSON. The measured bug: a consuming repo without tsx installed, where `npx tsx` went to the registry — >60s against a 15s budget, so all 50 specs failed at once with advice to run `npm run build`, a step that does not exist in a consumer install. Why a redesign and not more guards: the seam between two loaders cannot be made correct. The fallthrough guard had to answer "did the module body already run?" and Node does not expose that bit — ERR_MODULE_NOT_FOUND and SyntaxError each occur both before and during evaluation. The native path also had no time bound, and an in-flight evaluation cannot be cancelled. Three facts made it cheap: typescript is already a runtime dependency and already used at runtime; specs already round-trip through JSON (the old tsx path did exactly that); and the .js -> .ts rewrite is a rule to define, not inherit. Fixes at once: no npx and no network; a stalled spec is killed and NAMED; double evaluation is impossible by construction; resolution no longer varies by Node version or by which loader won; the Windows quoting bug dies with `npx -e`. Nine review findings closed — six by fixes, three by removing the seam. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014LVXMw2PJtPPtV8BoTANAd |
||
|
zernie
|
7144826614 |
fix(audit): cap the score on a confident breakage, split test-harness, retract an unverified claim (#177)
A confident breakage now caps the health score at 89, one point below the healthy band, so a definitively-broken harness can never render a green dot or grade A. Measured: Structure scored 92 with a green dot while carrying a dead tool contract, because a summed score dilutes one real breakage among clean siblings. confidentBreakages() enumerates what qualifies — decidable AND breakage, never the heuristic rings, since capping on those is how a gate gets switched off. The Safety ring stopped repeating one paragraph per unit: past three, ineffective disallowed-tools fences collapse into a single line that still names every skill. Measured 4,500 -> 810 characters. test-harness split into SKILL.md + references/, 5,534 -> 2,633 tokens loaded on activation, measured with a real tokenizer rather than chars/4. Retracted "no other plugin linter has it" on description-overlap: Cisco's skill-scanner ships --check-overlap. Edited the spec, not the artifact. Blast radius measured: this repo unchanged, clean vendored plugins still A. Gates: vitest 3288 passed, lint, prettier, api:check, build — plus the site browser-parity fixture regenerated for the new expected value. |
||
|
zernie
|
681edf7673 |
fix(hook): the subagent-delivery caveat is stale, and now a test says so (#172)
* fix(hook): the subagent-delivery caveat is stale, and now a test says so Claude Code #34692 — a PreToolUse hook not firing for a subagent's tool calls — was closed not-planned and quoted across these docs for months as a standing limit on what a gate can promise. It is fixed. Measured against a stock @anthropic-ai/claude-code@2.1.241 installed from the registry, not the container's binary: a subagent's own Bash reaches the hook and an exit-2 deny stops it, with the parent's identical command in the same run as the control and the marker files on disk as ground truth. The event also carries agent_type, naming which subagent made the call. The claim is about someone else's product, so prose cannot keep it honest — a doc cannot notice that the platform moved. src/subagent-delivery.test.ts pins both directions and goes red on a regression; removing the deny from its hook fails it on its own assertion. What did NOT change is stated at every site that was edited: a model can still route around a tool entirely, so a gate remains a strong default and is never unbypassable. The measurement's scope is stated too — headless only, with interactive sessions unmeasured and depth-2 nesting absent there. Gates run, in order: vitest (3282 passed), lint (0 errors), prettier --check, docs:check, internal:check, api:check, build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: regenerate the committed project-file types for the new test CI's generated-types:committed gate compares a fresh `generate types` run against the committed file, and adding a source file changes that list. Caught by CI rather than locally because I ran the individual commands instead of `npm run check`, which is the aggregate CI actually runs — the same shortcut this repo has already written down as a recurring miss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7c97e4ed7d |
refactor(spec)!: one experimental root per feature (#171)
The prefix answers "is this name stable?" and cannot answer "is the name my stable symbol depends on stable?". That gap shipped: railway(), delegate() and result() carried stable names while being meaningless without the experimental builder, and the lint rule cannot see it — it compares tag and name within one declaration, not the dependency between two. So the subagent vocabulary now hangs off one marked root: result, railway, delegate, pipe, pipeStep, needs, start and andThen become members of the builder. Eight names leave the top level. Two further gains the prefix alone could not reach: collision-prone words stop owning package-level names, and the warning survives destructuring, since the binding site still names the root — unlike a namespace object or an import subpath, which mark the import line. Object.assign rather than a TypeScript namespace: namespaces are an error in this repo's lint config, and one merges only with function declarations while the skill builder is a const. Types stay top-level, for the same reason the prefix rule excludes them. No deprecation window: nothing outside this repository imports these, and a window subtracts nothing. The container-rollback argument for a window does not reach them — they take no part in loading hooks. Found along the way, and worth more than the refactor: two example specs had not compiled since the previous rename, because the examples directory was in no tsconfig at all. Both fixed, and the class is closed — examples now type-check. Gates run, in order: vitest (3281 passed), lint (0 errors), prettier --check, api:check, build, exports:check, internal:check, docs:check, and tsc over the types project now that it includes examples. Claude-Session: https://claude.ai/code/session_017aUJEoaEQ3DxtvoSVWAWqe Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
900dc7e06a |
refactor(spec)!: name things for what they are; retire the vigiles/experimental subpath (#169)
Three exported symbols were named for something other than what they are, and
two mechanisms were doing one job.
RENAMES (old names kept one major, @deprecated, removed one major AFTER this one):
claude() → instructionFile() compiles CLAUDE.md AND AGENTS.md, so a
name from one harness was never right
instructions`` → prose`` builds a prose FRAGMENT; the plural read
as "the instruction file", which is what
instructionFile() builds
agent() → experimental_agent() the shape is not settled, and the tag was
never there (git log -S empty)
SkillSpec.result: → postcondition: `result` already meant the subagent output
contract one screen down; the doc comment
had to spend a line disambiguating them
The alias window is not politeness. Measured: the previous rename shipped with
ZERO overlap (18.1.1 old names only, 19.0.0 new only). A container rollback put
the old node_modules under new sources, PreToolUse stopped loading, and a
PreToolUse that does not load denies EVERY Bash command — including the one that
fixes it. One window per major makes that pair unrepresentable.
SUBPATH RETIRED. vigiles/experimental is gone; its members moved to the door
their feature already uses. Of two mechanisms for one promise the subpath was the
weaker: it marks the import line, out of view by the time anyone reads the call.
Measured on the only user-facing example — with the prefix aliased away at
import, the marker survived at 0 of 5 call sites. So the prefix stays, the advice
to alias it away became an explicit prohibition, and the six define* entry points
are DECLARED prefixed rather than gaining it at re-export.
That de-aliasing is what unblocked moving the naming check into ESLint, and the
ORDER was load-bearing: measured before it, dropping the public/internal
exemption produced 14 findings, all from the re-export indirection. Measured
after, across all of src/ with no exemption: 22 tagged declarations, 0
unprefixed. The rule turns on silent.
EXPORT CLEANUP. vigiles/linting export * → curated, 103 → 79 symbols; its header
claimed to be curated with an `export *` one line below, and claimed the builders
were also at the package root (191 exports there, zero matches). ClaudeTool and
HookEvent deleted — zero references, and both had drifted off the dialect they
claimed to mirror; HookEvent listed 5 events, two of which do not exist, against
the 31 the vendor documents.
DOCS. experimental.md deleted and distributed per feature: the skill half into
skills.md §Status, the hook half into a new compiled-hooks.md §Status, and the
emit half — 111 of its 177 lines, a topic doc wearing a stability page's clothes
— into docs/emit-channel.md, which gains the "which channel when" comparison it
never had. The one cross-cutting sentence went to the README.
TWO GATES WERE GREEN AND WRONG, both found by this work:
- api:check reported "verified for 11 entries — no drift" while reading
dist/experimental.d.ts from a PREVIOUS build (tsc does not clean dist/) for a
source file that no longer existed. Now asserted against package.json
exports, with a mutation proving exit 1 on a phantom entry.
- check-export-prefixes caught MY error: the R3 services first landed on
vigiles/eval, where `paid_` means "this call can bill you" — and
startServices takes a ContainerRuntime and calls no model. Naming it
paid_experimental_startServices would have redefined paid_ as "expensive in
some sense". They went to the package root.
SEVENTEEN review findings landed on this branch, all real. Two mechanical
closures came out of them: src/example-imports.test.ts resolves every relative
import under examples/ instead of matching a string, and the naming rule now
ENUMERATES the three export paths it visits rather than describing its coverage
in the abstract. One attempt is recorded as REFUTED: teaching the doc gate to
flag our own exports as free variables measured 66 findings across 93 blocks, so
that gate's original scoping was right.
BREAKING CHANGE: vigiles/experimental is removed. Its exports are unchanged in
name and behaviour and are now reached from `vigiles` (the emit channel, the R3
disposable-service tier). Every renamed symbol keeps a working @deprecated alias
for this major.
|
||
|
zernie
|
ec455214c7 |
fix(hook-install): recompiling appended a duplicate beside a hand-wired hook (#168)
Measured in a consumer repo 2026-08-21: `vigiles compile` on twelve hooks
produced twelve DUPLICATE wirings, because `managesHook` could not see that the
existing entry pointed at the same file.
The existing entries were wired the way Claude Code's own docs recommend:
node "$CLAUDE_PROJECT_DIR/node_modules/vigiles/dist/cli.js" \
hook-runtime run-program "$CLAUDE_PROJECT_DIR/.claude/hooks/x.hook.ts"
`managesHook` splits the command on whitespace and canonicalises each token, so
it compared the literal `"$CLAUDE_PROJECT_DIR/.claude/hooks/x.hook.ts"` —
quotes included — against `.claude/hooks/x.hook.ts`. Two independent reasons it
could never match, and they compound:
- QUOTES are not an exotic spelling. A path containing a space MUST be quoted,
so the quoted form is the correct one and was unmatchable.
- `$CLAUDE_PROJECT_DIR` IS the project root — exactly what `normalizeHookRef`
resolves relative paths against — so the two spellings name the same file by
definition, not by guess.
The duplicate is the worse of the two copies: it spells the path relative to the
cwd, so it exits 2 the moment the agent runs from a subdirectory. The healthy
copy beside it keeps working, which is why this went a full day unnoticed —
the harness looked fine until someone `cd`'d.
Only `$CLAUDE_PROJECT_DIR` is stripped. Any other variable names an unknown
location (a sibling checkout, a plugin dir), and treating it as the root would
DELETE a hook pointing somewhere else.
Mutation-proven, each with evidence the patch landed:
| mutation | fails on |
| --- | --- |
| stop stripping quotes | the hand-wired replace case |
| stop stripping $CLAUDE_PROJECT_DIR | the hand-wired replace case |
| strip ANY $VAR/ | "a hook under a DIFFERENT variable" |
⚠️ Two mutations I ran first were WORTHLESS and are recorded so nobody repeats
them: a `command.includes(ref)` over-match passed green (a full-path ref makes
`includes` nearly as strict as `===`, so the mutation changes no behaviour), and
a hand-written regex mutation produced a syntax error, which reads as "no tests"
rather than as a failure. A green mutation is a finding about the TEST, not a
verdict about the code.
Gates, in order: `npm run check` 18/18 · `npx vitest run`.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
7b8cdecd5e |
feat!: doc-refs is opt-in, because it measured 0 true positives (#167)
* fix(loader): a surface that exists on disk was never read, and the report named it anyway
The loader read the repo-root `skills/` **or** the project-level
`.claude/skills/` — never both — and materialized whichever it picked under the
SAME canonical `<materializeRoot>/<surface>/…` key. Two real files, one key: the
loser was never opened, and the winner's bytes sat under the loser's name.
The vendor settles which one the harness loads: BOTH, in two namespaces.
> Plugin skills use a `plugin-name:skill-name` namespace, so they can't
> conflict with other levels. For example, `my-plugin/skills/deploy/SKILL.md`
> becomes `/my-plugin:deploy` and loads alongside a `deploy` skill in your
> project's `.claude/skills/`.
> — https://code.claude.com/docs/en/skills § "Where skills live"
So "pick one" was never a tie-break to get right; it was a question with no
answer, asked because the key shape forced one. The fix removes the question:
every scope present is read, and a scope's key prefix is DERIVED FROM THE SCOPE,
so two files can no longer claim one key — there is no ordering, no
"if already taken", and no last-write-wins left to get wrong.
Measured (36 SHA-pinned public repos, 2026-08-18):
* 573 → 493 surface files that exist on disk and are never opened.
The 80 recovered are the shadowed ones: nyldn/claude-octopus 71 → 0,
hyhmrright/brooks-lint 8 → 0, thedotmack/claude-mem 1 → 0.
* The 493 that remain are a DIFFERENT, pre-existing gap, unchanged by this
commit: directory symlinks under `skills/`, which `entryOf` deliberately
does not descend (the ELOOP containment) but Claude Code does follow.
* nyldn/claude-octopus ships each skill twice — 61 under `skills/`, 57 under
`.claude/skills/`, 50 names in both, and ALL 50 pairs differ. One pair is
vendored as a true-positive fixture.
* vigiles' own repo: 6 skills → 19, lethal-trifecta units 4 → 12. Its own
audit was blind to two thirds of its own machine.
* Grades moved on 1 of 36: brooks-lint C(70) → B(82). NO finding was deleted
(6 before, 6 after) and the COUNT of dangerous units rose (6 → 7); the
Safety ring is a ratio, and this moves its denominator from 6 to the real
12. See the report for the full justification.
Also fixed, because reading both scopes caused it: a name no longer identifies a
surface, so `description-overlap` would have printed `skills "dup" and "dup"` —
50 times on claude-octopus. Overlap labels now carry the real path.
SUBTRACTS: the `root` | `user` | `none` variants of `SurfaceSource` (the either/or
is gone, so "we picked the wrong one" is unrepresentable), the duplicated
precedence rules in `scan-files.ts` (the decision now lives once, IO-free), and
one net lint warning in the two touched files (6 → 4).
Gates: npx vitest run · npm run lint · npx prettier --check . · npm run api:check
· npm run build — all clean, in that order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
* chore: regenerate the browser-parity fixture for the two-scope disambiguation
The overlap finding now names each skill by PATH, not by bare name — which is
the point of the two-scope work: with root skills/ and .claude/skills both
present, a bare name does not identify a surface. Intentional engine change,
regenerated per the parity test's own instruction.
* chore: regenerate generated.d.ts for the merged tree
Единственный конфликт мёрджа был одной строкой в СГЕНЕРИРОВАННОМ файле — счётчик файлов
проекта, 388 против 387. Разрешено не выбором стороны, а регенерацией: у слитого дерева
счёт 389, то есть ОБЕ стороны были неверны для него. Выбор любой из них дал бы зелёный
мёрдж с неправдой в артефакте.
* fix: four defects found by dogfooding vigiles on a real consumer repo
Все четыре найдены за один день догфуда на `zernie/mine` (43 скилла, 48 харнесс-файлов),
каждый — с замером, а не с рассуждением.
1. `exclude` не доходил до прохода по markdown-рефам
`findDocRefs({ basePath })` вызывался без `ignore`, хотя поле `exclude` документировано
ровно под этот случай («vendored or benchmark fixtures the repo's own lint shouldn't
police»). Замер: 9 битых рефов, 8 из них внутри каталога, явно перечисленного в
`exclude`, и все 9 репортились. Следствие дороже причины: выйти в 0 стало нечем, поэтому
шаг пометили `continue-on-error: true` — а lint, чей код возврата выбрасывают, не гейтит
ничего, и рядом выросли самодельные CI-шаги, чтобы гейтить вместо него. Один непереданный
аргумент — вся эта цепочка. После починки: 9 → 0.
2. Рефы искались ТЕКСТОМ, а не разбором
`/\b(enforce|file|cmd|ref)\(\s*["']…/` — четыре из шести входов классифицировались
неверно (замер 19.08): `ctx.file("OUT")` (метод чужого объекта — живой ложный
срабатыв в реальной репе), `obj.cmd(...)`, `// cmd(...)` в комментарии,
`'cmd("x")'` внутри строки. `\b` спокойно стоит после точки, а у регулярки вообще нет
понятия комментария и строкового литерала. Разбор делает все четыре НЕВЫРАЗИМЫМИ: AST
отдаёт только вызов, у которого callee — голый идентификатор. `typescript` и так
runtime-зависимость пакета — парсер лежал в коробке всё это время.
3. Раннер скриптов был строгой очередью
`files.forEach(... spawnSync ...)` со `stdio: "inherit"`. Замер на 48 харнессах:
270s → 77s (3.5×), результат ПОБАЙТОВО тот же (84 passed, 1 with 0 checks, 7 failed).
Пул по умолчанию выводится из `entry`, а не из нового флага: `test` не передаёт entry и
разъезжается по ядрам (файлы бесплатны и детерминированы по построению — мок-модель, без
API-ключа), `eval` передаёт и остаётся на 1, потому что параллель там — это одновременные
оплачиваемые вызовы модели. Вывод буферизуется на ребёнка: с `inherit` 48 отчётов
шинкуются друг о друга, и это настоящая цена параллели, а не сам цикл.
4. Путь агента в находке брался по ИМЕНИ
`new Map(agents.map(a => [a.name, a.path]))` оставляет последнюю запись, поэтому при
`agents/foo.md` и `.claude/agents/foo.md` (разные неймспейсы у Claude) находка о первом
репортилась против файла второго. Находка lethal-trifecta, указывающая не на тот файл,
хуже отсутствующей: она отправляет читателя проверять невиновный код. До полной
scope-qualified идентичности (правка крупнее, `finding.name` несёт только голое имя)
неоднозначное имя даёт ПУСТОЙ путь вместо произвольного.
Плюс: все 10 джобов в 6 воркфлоу этого репозитория были без `timeout-minutes` — то есть
зависший джоб шёл до шестичасового потолка GitHub. Бонды ~3-4× ЗАМЕРЕННОЙ длительности
собственных прогонов репы, замер процитирован над каждой строкой.
Тесты: два новых на пул, обе половины. ⚠️ Первая версия фикстуры считала маркеры, которые
только СОЗДАВАЛИСЬ, — то есть «сколько стартовало всего», а это достигает N и при строго
последовательном прогоне; тест проходил по причине, не связанной с параллелью. Исправлено
освобождением слота на выходе. Обе мутации (форсировать serial · распараллелить eval)
валят каждая свой ассерт, каждая с доказательством, что патч лёг.
Гейты: `npm run lint` 0 errors · `npx prettier --check .` чисто · `npm run api:check` без
дрейфа · `npm run build` · `npx vitest run` — падает только пре-существующий
`scripts/release-preset.test.ts` (не находит `@semantic-release/commit-analyzer`, объявленный
в package.json, но не установленный в этом окружении; на CI со свежей установкой зелено).
* docs: record what each of the four defects was, in the pages that own them
comparison.md — что СЧИТАЕТСЯ рефом теперь (вызов, у которого callee — голый идентификатор,
а первый аргумент строковый литерал) и почему три формы больше не рефы: метод чужого объекта,
упоминание в комментарии, строка с текстом вызова. Плюс область: проход читает `**/*.md` и
теперь уважает `exclude`, чего не делал — из-за чего репозиторий с вендоренной чужой
разметкой не мог получить чистый lint вообще.
docs/rules/delegation-trifecta.md — новая секция про два агента с одним именем: находка
теперь показывается БЕЗ пути, а не с произвольно выбранным из двух. Явно сказано, почему
это лучше: неверный путь отправляет читателя проверять невиновный файл.
docs/cli.md — почему `test` гоняет несколько скриптов разом, а `eval` по одному, и что
это следует из яруса, а не из флага. С замером 270s → 77s при побайтово одинаковом
результате, и с ценой: вывод буферизуется, живого стрима больше нет.
Все три — то, что было НАЙДЕНО, а не только что изменилось: через месяц полезно именно это.
* fix(docs): mark two illustrative enforce() blocks the old regex could not see
Разбор нашёл на два рефа больше, чем регулярка, и оба оказались СЛОМАННЫМИ — годами,
незаметно: `enforce("rubocop/Custom/NoDirectDbQuery")` и `enforce("pylint/no-direct-db-query")`
в справочных страницах скилла. Это примеры того, как объявить СВОЁ правило, то есть имена
заведомо несуществующие — им место под `<!-- vigiles:ignore -->`.
🔴 Почему регулярка их не видела, и это отдельный класс: она применялась ПОСТРОЧНО
(`for (const {lineNo, text} of blockLines) text.matchAll(CALL_RE)`), поэтому вызов,
разнесённый на несколько строк — а оба именно такие — не совпадал никогда. То есть у прежнего
детектора были не только ложные срабатывания (метод чужого объекта, комментарий, строка), но и
ложные ПРОПУСКИ на самой обычной форматировке. Разбор снимает оба класса разом: у AST нет
понятия «строка файла».
Итог на этом репозитории: 21 реф → 25 (после двух маркеров), все валидны.
* fix(compile): quote frontmatter scalars YAML would otherwise mis-read
`description: ${spec.description}` подставлялся сырым, поэтому описание с двоеточием и
пробелом — «…ловит ошибки, где замер врёт: неаналоги в выборке» — давало frontmatter,
который НЕ является валидным YAML. Замер 19.08 на двух реальных скиллах: сразу после
`vigiles compile` метод `skillContract()` отдаёт `malformed: true` и `declared: []`. Файл
ВЫГЛЯДИТ объявляющим `allowed-tools`, строгий парсер не читает ничего, скилл молча
наследует все инструменты сессии.
🔴 Это ровно тот дефект, который правило `frontmatter-valid` существует чтобы репортить —
то есть благословлённый путь производил то, что продукт ищет. Хуже: он НАКАЗЫВАЛ за
починку. Человек, закавычивший значение руками, получал на следующем линте расхождение
хеша («manually edited after compilation»), а перекомпиляция молча откатывала правку.
Проверка — КРУГОВОЙ ПРОГОН, а не список опасных символов: эмитим, читаем обратно тем же
загрузчиком, которым пользуется линтер, и квотим, только если вернулось не то, что ушло.
Два следствия этого выбора:
· уже безопасное значение эмитится побайтово как раньше — ни один скомпилированный файл не
меняется, ни один интегрити-хеш не двигается (подтверждено: все 1303 core-теста зелены);
· множество «опасных» входов не надо перечислять и поддерживать — решает сам YAML, поэтому
`#`, `[`, `&`, `*`, ведущий пробел, `yes`, `null` покрыты, не будучи названными.
Применено и к субагентам: описание там пишется так же свободно и ломает frontmatter так же.
Тесты — обе половины, и обе несущие (мутации с доказательством, что патч лёг):
· вернуть сырую подстановку → падает «compile quotes a description YAML would mis-read»
· квотить всегда → падает «leaves an already-safe description bare — no hash churn»
Вторая не менее важна первой: избыточное квотирование переписало бы каждый файл в каждой
репе и сдвинуло каждый хеш, что читается как «vigiles переписал мне весь репозиторий».
* feat!: doc-refs is opt-in, because it measured 0 true positives
The markdown code-block ref pass ran unconditionally and could not be turned
off. Measured across two real repositories on 2026-08-19 — 2 582 markdown
files, 52 builder refs — it produced ZERO true positives, and every error it
had ever raised was false: design prose sketching an API that does not exist
yet (`cmd("npm test")` for a package with no such script), and a third-party
CLAUDE.md captured verbatim as benchmark data.
The cause is structural rather than a threshold miss. A fenced block in prose
is a DRAWING of config, and this pass read it as config. No calibration fixes
that, because the block is not wrong — it is not config.
The consumer repo could only reach a passing lint by listing whole directories
in `exclude`, silencing every other check over a third of the tree; measured
afterwards the pass walked 604 files and found 0 refs — inert, still paying for
the walk. Off by default replaces that blunt instrument with an honest default.
BREAKING CHANGE: the markdown code-block ref pass no longer runs by default.
Enable it with {"rules": {"doc-refs": "error"}} — appropriate where markdown
genuinely is the source (a docs site whose fences are copied into live specs).
At "warn" findings print marked ℹ and the exit code is untouched; at "error"
they gate with exit 2 (they previously contributed exit 1, the soft tier,
because the pass fired unasked).
Known gap recorded for whoever raises the default again: the walker globs
without `dot`, so .claude/** — where deployed skills and agents live — has
never been scanned. The rule has never once judged a real instruction file.
* test(doc-refs): the GitHub annotation follows the configured severity
The inline-annotation test drove the pass with no config, which no longer runs
it. Extended to all three tiers rather than just re-enabled: "error" annotates
::error, "warn" annotates ::warning, and absent annotates nothing at all. The
middle case is the one worth asserting — a ::error rendered next to an exit code
of 0 is how an advisory rule gets read as a gate.
* chore: regenerate generated.d.ts, and stop two docs lines claiming doc-refs is on
CI caught what my five local gates could not: `npm run check` runs 18 stages
and I ran 5 of them. The new test file was missing from the generated project-
file union — 17/18, failing on `generated-types:committed`.
Also two sentences in docs/comparison.md still described the pass as
unconditional: the section's opening claim ("vigiles validates … calls inside
fenced TS/JS code blocks") and the Scope paragraph. Both now say opt-in; the
reader hits the opening line four paragraphs before the new note explaining the
default.
* ci(site): stop the browser install from eating the job
The site job was cancelled at its 15-minute bond twice in a row (runs
32300064395 and 32303343213), both times inside `playwright install --with-deps
chromium`, and neither failure was in the diff under review. The log shows the
cause plainly: `--with-deps` shells out to apt, apt hung on
azure.archive.ubuntu.com (`Ign:` on every line, then twelve minutes of silence),
and the orphan reaped at the end is `npm exec playwright install --with-deps`.
Three changes, one per cause:
- Drop `--with-deps`. ubuntu-latest already ships the libraries Chromium needs;
the flag exists for bare containers. Removing it removes the only part of this
step that touches a distro mirror. If a library really is missing, the browser
fails to LAUNCH in the next step within seconds and names the missing .so —
loud, and far better than a silent 15-minute hang.
- Cache ~/.cache/ms-playwright, keyed on the lockfiles. The step's floor was a
full ~150 MB download on every run, cache or no cache, good day or bad.
- Bound the step at 5 minutes. The job bond cannot tell "this step wedged" from
"the suite is slow", so a wedge cost the whole job; now it costs five minutes
and names the step that did it.
Measured baseline for the job: 3m35s.
* ci: run a job only when the commit can reach its inputs
This workflow had no path filter at all, so every PR paid for all five jobs
whatever it touched. Measured on this very PR (16 files across src/ and docs/,
none under site/): the site job built the engine, downloaded a browser and ran
the demo suite against code the diff could not reach — twice, at ~15 minutes
each, because the browser install wedged.
A workflow-level `paths:` cannot express this. It decides whether to run the
workflow AT ALL and says nothing about individual jobs. So the decision moves
into a `changes` job: one API call, no checkout, ~10s, feeding an `if:` on the
jobs whose inputs are disjoint from what changed.
Two flags, each conservative in the direction that costs minutes rather than
correctness:
root false ONLY when every changed file is under site/ — nothing outside
site/ imports anything inside it. Gates test, harness, e2e.
site false ONLY when every changed file is prose or agent config
(docs/, .claude/, root *.md). Gates site.
`check` stays unconditional: it runs types:site and cli-lint, so both halves of
the repo are among its inputs.
The site flag is a measurement, not an assumption. The demo imports the built
engine through three aliases, so any src/ change can reach it and must run it.
What cannot: grepping the three bundles it actually loads for the lint rule table
(DEFAULT_RULES, frontmatter-valid, doc-refs) returns zero in all three, and its
check pages are generated from site/src/checks/checks.ts rather than from docs/.
Tested in src/ci-path-filter.test.ts, which EXTRACTS the patterns from the
workflow rather than restating them — a test that agrees with its own copy of a
rule proves nothing about the rule that runs. A skipped job and a passed job look
identical in the checks list, so both directions are asserted. Three mutations,
each failing on its own assertion with the patch verified in place:
prose arm widened to any *.md → 'a markdown file NESTED in src'
prose arm widened to include src/ → 'runs everything for a normal src change'
empty-list bail flipped to false → 'an empty diff is not a licence to skip'
* test(ci): a job added later cannot silently escape the path gate
The filter itself was tested; the SHAPE of the workflow was not. In the checks
list a skipped job and a passed job are the same tick, so a job added tomorrow
with no `if:` looks exactly like a correctly-gated one that happened to run —
the filter quietly stops meaning anything and nothing goes red.
Exemptions are named with their reason rather than inferred, so granting one is
a visible edit: `changes` is the classifier itself (gating it on itself is a
cycle), `check` runs types:site and cli-lint so both halves of the repo are its
inputs. Plus a vacuity guard — if every job ends up exempt the assertion can no
longer fail, and an empty list would read as health.
Mutation, patch verified in place: stripping the gate off `site` fails with
"job(s) run on every commit with no path gate: site".
Mirrors the same guard added to mine's paper-gates.harness.mjs, where an ungated
19m42s job was what prompted it.
Gates, in order: npx vitest run (8 in this file) · npm run check 18/18.
* Revert "test(ci): a job added later cannot silently escape the path gate"
This reverts commit
|
||
|
zernie
|
6532dbf48d |
fix: five defects found dogfooding vigiles on a real consumer repo, plus CI job bounds (#162)
Five defects found dogfooding vigiles on a real consumer repo (zernie/mine — 43 skills,
48 harness files), each measured rather than reasoned about:
1. `exclude` never reached the markdown-refs pass. Measured: 9 broken refs, 8 of them
inside a directory explicitly listed in `exclude`, all 9 still reported. After: 9 → 0.
The consequence was larger than the cause — with no way to reach exit 0 the consumer
set `continue-on-error: true`, and a lint whose exit code is discarded gates nothing.
2. Refs were found by TEXT, per line. Four of six probe inputs classified wrongly
(`ctx.file("OUT")`, `obj.cmd(...)`, a comment, a string), and because the regex ran
per line, any call split across lines was invisible — which had hidden two genuinely
broken refs in this repo's own docs. Now parsed: a ref is a call whose callee is a
bare identifier with a string-literal first argument. `typescript` was already a
runtime dependency. Net: 21 refs → 25, all valid.
3. The script runner was a strict queue. Measured on 48 harness files: 270s → 77s
(3.5x), byte-identical results. The default follows `entry` rather than a new flag:
`test` fans out (free and deterministic by construction — mock model, no API key),
`eval` stays at 1 (real model quota). `stdio: "inherit"` was the actual blocker.
4. Agent path attribution was keyed by name, so with `agents/foo.md` and
`.claude/agents/foo.md` a lethal-trifecta finding was displayed against the wrong
file. Ambiguous names now yield no path instead of an arbitrary one.
5. `compile` emitted invalid YAML. A description containing `: ` produced frontmatter
where `skillContract()` reports `malformed: true, declared: []` — the blessed path
producing the exact defect `frontmatter-valid` exists to report, and punishing anyone
who fixed it by hand with a hash mismatch. Fixed by round-tripping through the same
loader the linter uses, so already-safe values stay byte-identical and no integrity
hash moves.
Plus: all 10 jobs across 6 workflows were unbounded and now carry `timeout-minutes`
sized to measured durations; docs updated in the pages that own each defect.
Gates: vitest 3186 passed - lint 0 errors - prettier clean - api:check no drift - build.
CI: all 7 jobs green.
|
||
|
zernie
|
04225f9980 |
feat: an errored emit is not an emission, and a contract field can be an enum (#165)
Two locked design items (vigiles/ideas/23), plus the public docs brought level with
what has actually been measured.
1. A CALL THAT ERRORED IS NOT AN EMISSION. Measured on the shipped build: a
permission-denied call carrying a valid payload returned {"kind":"ok"} — the reader
filtered by name and never looked at ToolCall.isError, a field that was on the type
all along. The server never received that call. This is the exact class of lie the
emit channel exists to remove from the fenced rail, reproduced inside the channel.
Errored calls now get their own branch, because all three collapses lie in a
different direction: counting them reports success for a call nobody received;
dropping them silently says "no tool call", sending the reader to the skill's
instructions when the fault is in permissions; lumping them with "called twice"
gives a true signal under a false name when a model retries a denial. A successful
call alongside a denied one is one successful emission — the contract was met.
Four tests, each naming its collapse. Mutation-checked: removing the split fails
three of the four.
2. ENUM, the one extension to OutputFieldType. A readonly tuple of literals declares
a field whose value must be one of them; it renders as a JSON-Schema enum in the
tool definition and as "CUT" | "MERGE" in the fenced rail, so both deliveries show
a worker the same permitted values. The measurement behind it: a verdict field
declared as `string` held three mutually incomparable invented categories over three
runs, with 3/19 = 16% vocabulary compliance. `string` cannot express "one of these".
Nothing relational follows it — no object[], no tuples. On 14 real payloads, 23
scalar values carried every assertion anyone made while 80,981 characters of prose
carried none, so the body stays prose by decision rather than by omission.
3. DOCS. docs/experimental.md gains the outcome table — every way an emission can fail
to satisfy the contract, with the reason text — which was the one thing a reader
could not find. Measurements updated to the breadth study (15/15 on sonnet; a zero
on haiku that was the skill stopping early, not the channel dropping); the serving
question is marked answered; the prefix-removal conditions now say which of the
three is met. Also states plainly what the channel does NOT check: whether the
emitted result is true. Gating on an emitted verdict was measured and rejected —
self-report disagreed with a judged check in 45-75.8% of runs.
Gates: npx vitest run, npm run lint, npx prettier --check ., npm run api:check,
npm run build — via the check script, 18 of 18. The api-surface diff is the intended
widening of OutputFieldType and EmitFieldSchema, reviewed and committed.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
4f0a01b2f9 |
CLI clarity, an execution-tier finding, and the Action's boundary documented (#163)
Five changes from a measured review of the CLI surface and of a consumer's CI. 1. BEHAVIOUR — a positional target that matches nothing now FAILS. Bare discovery finding nothing still exits 0: an empty repo is a legitimate place to stand, but a path or glob you NAMED that matched nothing is stale, wrong, or never reached. Collapsing the two cost a real repo three days of green CI verifying zero files. Jest and Vitest fail on no-tests-found and make you opt in with --passWithNoTests; pytest exits 5. --min=0 is the opt-out here, so no flag is added. 2. HELP — the four checkers (audit/lint/test/eval) all read as "check my stuff". They are now grouped by READ vs RUN, and each line states its own consequence positively. The success criterion was checkable and is met: "NOT a CI step — use vigiles lint in CI" is deleted, because nothing needs defending. A help line that says what a command ISN'T is a description that failed. Per-verb flag detail moves into the per-verb help; audit's four wrapped lines naming nine flags were most of the felt crowding. Verb count is unchanged at 8 — the smallest of every tool measured (vitest 8 verbs and 164 flags, cargo 48, docker 57, git 166), and none of them thinned their surface by removing verbs. 3. DOCBLOCK — src/cli.ts listed four of the eight commands, having quietly stopped growing. Replaced by a pointer to the single source; a new test asserts every verb appears in the RENDERED help exactly once, because a constant is what drifted. 4. AUDIT — new Tested finding: "N surfaces counted as covered by PLACEMENT only — no run on record ever exercised one". Colocated evidence is a claim about the filesystem and the provenance line already admits it; nobody reads a provenance line. Measured in a consumer: 26 colocated and 0 executed, while its CI skipped every one of those harnesses and reported success. The corpus-wide threshold is a stated limit in the code, not a silence. 5. DOCS + SKILL — the Action reference now explains why there is no test input (the Action never runs npm ci; harnesses import the library), and the test-harness skill gained a six-line CI pointer: an Action exists, init wires it, inputs are documented there. It REFERENCES rather than explains, because copied explanations rot — three measured proofs, one of them inside this product. Gates, in order: vitest, lint, prettier, api:check, build — via the check script, 18 of 18. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
ad413b0ac5 |
fix(scan): the fix-scan work, merged onto current main (#161)
* fix(scan): two ways a surface that is there was invisible to audit
A marketplace listing suppressed the audit of the directory it sat in, and
`agents/` was read one level deep though the harness reads it recursively.
Both end the same way: vigiles reported nothing, at exit 0, about surfaces
that were on disk — and silence is this tool's success signal, so neither
failure could be seen from the outside.
MARKETPLACE (`audit`, cli.ts). A `marketplace.json` whose members are all
external expanded to zero targets, and a branch printed "nothing to scan"
and returned BEFORE looking at the directory. But that file is a statement
about OTHER directories: a repo may list external members and also be a
plugin. Measured on nyldn/claude-octopus @57cfb9b0 (3,979 stars): 59 skills,
50 subagents, 40 hooks, 54 commands — 203 surfaces — suppressed, exit 0.
The branch is deleted; the `targets` fallback that already existed does the
work. Proof the suppression is gone: deleting the marketplace.json now
yields the byte-identical report, so the file no longer changes the outcome.
A genuinely curated marketplace reaches the existing `score.empty` branch,
which carries the "clone a member plugin" advice and exits 2 — this repo's
own rule that 2 means "I could not do what you asked". `--json` still emits
`kind:"marketplace"` there, for exactly the case it describes.
AGENTS RECURSION (core/layout.ts + its three readers). The vendor documents
`agents/` as scanned recursively, with the subfolder forming the identifier
(`agents/review/security.md` -> `review:security`). vigiles matched
`agents/[^/]+\.md` — in THREE places that each spelled the rule themselves:
the scan classifier and both coverage discoverers. Measured on
rsmdt/the-startup @88d447c7: 16 agent files, 2 read, and the plugin was
still graded B (80/100) over that 12.5%. The rule now has one home,
AGENT_FILE_LEAF_RE, quoted by all three, so the subagent count and the
untested count can no longer be computed over different sets of files.
Agent names become the documented scoped identifier. That is not cosmetic:
a basename cannot be unique once a dir is read recursively, and the
delegation graph keys agents BY NAME, so `agents/a/review.md` and
`agents/b/review.md` would have swallowed each other. Top-level agents keep
their bare filename, which is why no existing report moves.
Two defects the recursion itself introduced, both closed: a `SKILL.md` under
`agents/` matched both classifiers and was counted twice (the harness loads
it as an agent, so it is excluded from skills — the mirror of the existing
skills/<x>/agents/ rule); and the on-disk discoverer normalized separators
for the pattern but not for the name, which on Windows would have silently
fallen back to a basename.
Folded in: a nonexistent path now says it does not exist rather than "no
surface was found there" — one existsSync, its own test.
GRADES MOVE, in the harsher direction, on 2 of the 32 dogfood repos (the
other 30 are byte-identical):
nyldn/claude-octopus suppressed, no grade -> D (60/100), 203 surfaces
rsmdt/the-startup member `team`: B (80) -> D (60), 2 -> 16 agents,
and 2 -> 14 malformed-frontmatter findings
No exit code moved anywhere in the corpus. bench/leaderboard/RESULTS.md is
unaffected: both its pinned repos have zero nested agents (measured).
Tests: 10 new cases across scan/parity/layout/e2e, each proven by mutation
with the patch asserted to have landed. The parity fixture is load-bearing —
reverting only the browser-side discoverer survives the entire pre-existing
suite and dies only on the new nested-agent parity case.
Gates: vitest 2991 passed - lint 0 errors - prettier clean - api:check no
drift - build ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
The same two bytes as on main. Both sides of a merge must be text before git
will do a three-way merge at all — normalising only one side leaves the blob
pair binary.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
34a4f7a556 |
fix: no raw NUL bytes in source, and a gate that keeps it that way (#160)
src/scan-files.test.ts carried two literal NUL bytes inside a template literal where the escape was meant. Git decides "binary" per blob by looking for a NUL, and a binary blob gets no line diff and NO THREE-WAY MERGE — so every branch touching the file conflicted, unresolvably, because there were no hunks to pick. Measured: with the bytes escaped, `main <- fix-scan` and `main <- fix-shadow` each merge with ZERO conflicts. They had been parked for a day as "conflicting with each other"; they never were. The escape is the form the rest of the codebase already uses (skill-contract, scan-core, coverage-artifact, coverage-probe — eight sites), and the string it produces is byte-identical, so scan-files.test.ts still passes its 21 tests. Adds scripts/check-text-sources.mjs as an 18th `check` gate. It refuses to pass on an empty scan, since a scan finding nothing is indistinguishable from a scan looking in the wrong place. Proven by mutation: planting one raw NUL fails it with the file, the count and the offset. Not a .gitattributes marker: that restores diff and merge while leaving the byte in the source, hiding the defect instead of removing it. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
4acfd31fc9 |
feat(eval)!: an eval file describes, the runner performs — importing one no longer spends (#159)
* fix(audit): coverage caveats printed by lint never reached the audit report `formatUntestedReport` grew two qualifier lines — a test file still carrying the retired `vigiles:covers` marker, and a run record older than the text it measured. `audit` assembles its own fact block from the SAME UntestedReport and carried neither, so the note that exists to explain a silent migration was itself silent for anyone whose habit is `audit` rather than `lint`. Measured before the fix on a fixture whose only harness carries the marker: lint → "1 test file(s) still carry the retired `vigiles:covers` marker …" audit → "Untested surfaces: 0" and nothing else The caveats now come from one exported builder (`coverageCaveats`) that both renderers read, so a third qualifier reaches both callers or neither — a renderer can no longer carry half the list. `ScanReport.coverageCaveats` is absent rather than `[]` when empty, because the browser twin structurally cannot compute these (no `.vigiles/coverage.json` without a filesystem) and byte-parity between the two engines is asserted field-for-field. Both halves tested at the product level (`node dist/cli.js audit`): it fires on a repo carrying the marker and stays silent on one that does not. Proven by two mutations, each with the patched source grepped first: dropping the render line fails the "fires" assertion; emitting a caveat unconditionally fails the "silent" one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the suffix that stopped counting now says so `*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run collects exactly those names and a skill test spawns a model — so a misnamed harness was being executed, and paid for, by somebody else's runner. The removal was announced to nobody. Its own comment argues "the migration is a rename, and the untested finding prints the exact path", but the path printed is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore told to write the test they wrote, with neither the file nor the reason named. The sibling half of the same migration — the retired `vigiles:covers` marker — got its explanatory line at the time. This is the half that did not. Scoped to surfaces that are otherwise UNCOVERED: that is the only position where the count contradicts what the author can see in the directory. A stray `*.test.*` beside a properly covered surface is somebody's ordinary unit test and none of our business — asserted, because an advisory line that fires on the innocent case gets muted within a day. Nothing about coverage changes: the name still grants nothing. Measured silent on all seven real corpora available here (this repo, six vendored dogfood plugins) and on the `mine` knowledge base; it fires on a planted fixture in both `lint` and `audit`, the latter through the shared caveat list added in the previous commit rather than a second wiring. Three mutations, each with the patched source grepped before the run and each dying on its own assertion: dropping the note from the caveat list · widening the scope from untested to every considered surface · dropping the suffix test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * refactor(scan): compute the caveat list once Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was called twice in the same object literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the retired-suffix note claimed a cost the guard already closes The note said a `*.test.*` harness "spawns a model" under someone else's runner. `refuseUnderForeignRunner` guards every spawn door and refuses under vitest/jest — measured here against the detector: argv of `node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model: this process is running under vitest." What actually happens is that the foreign run FAILS on the file, which is the true and smaller claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it Importing a `*.eval.*` file spent real money. These files did their work in the module body, and in ESM `import` IS execution — so the cheapest imaginable question, "does this file even parse?", launched a paid run against a live model when it was asked with `import()`. That happened, and was paid for. The one guard that exists does not close the door and cannot be made to. Under `node -e 'import(f)'` there is no `process.argv[1]` at all, so `foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays silent; and no process fact distinguishes that from a legitimate runner doing the same import, so an argv-shaped guard would have to refuse the correct invocation to catch the wrong one — the exact failure that got `process.env.VITEST` rejected in `core/foreign-runner.ts`. So the shape changed instead. An eval file default-exports a description: export default defineEval({ measureTriggerRate: { …spec… }, assert: (report) => assertTriggerRate(report, { min: 0.8 }), }); `defineEval` builds a plain value; there is no call in it to run. The five paid runners are reached only by `dist/eval-entry.js`, the program `vigiles eval` now spawns per file — not a new verb, the interpreter the existing verb uses. `defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and that is the point: after this change no eval file imports a runner at all. Three loud edges, because a silent no-op would be the same defect in a new place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares nothing (or two measurements) is reported by the runner instead of exiting green; and a half-migrated file — a description plus a leftover top-level `measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and throws a migration message instead of billing. `node --check` still works and is what the free syntax check should be. Subtractions: every eval file loses its `process.env.VIGILES_TRIALS || process.argv[2]` line, its `console.log(format…Report(report))`, its `if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval` import. The runner does all five, once. BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})` instead of calling a runner at the top level, and `node <file>.eval.mjs` no longer runs an eval. To migrate: move the runner call into a key named after that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` · `measureSelectionMatrix` — declare exactly one), move anything you did with the report into `assert(report)`, and delete the trials line, the `format…` print and the `report.n === 0` check — `vigiles eval` does those. A capability probe that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s directory argument becomes a `pluginDir` field; and `measureTriggerRate(spec, { evalDriver })`'s second argument becomes a top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials. Run with `vigiles eval <file>`; a file left on the old shape fails with a message naming docs/harness-testing.md, never silently. All 19 eval files in this repository were migrated in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture and generated types The eval-definition change rewrote one report hint (`paid_measureTriggerRate` → `export default defineEval({ measureTriggerRate: … })`), which is exactly the divergence the parity test is built to catch. Regenerated per that test's own instruction for an intentional engine change; the diff is that one line — grade and every other field are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
effed2afcc |
fix(vocab): the validator knew 9 of 31 hook events, and the other 21 passed by failing open (#158)
* fix(dialect): vocabulary is a status per name, not membership in two lists Three checks answered "does this string look like something I recognise?" instead of "is this string valid?", and decided what to SAY about an unrecognised name by its EDIT DISTANCE to a hand-kept list. Measured against the vendor docs (claude-code 2.1.233, read 2026-08-17): - `Setup`, a documented hook event, was reported as "a hook here never fires" and told to become `Stop` — a different event with different semantics. It was caught only because `setup` sits 2 edits from `stop`. - The catalog held 9 of the vendor's 31 events. The other 22 were not "accepted"; 21 were unrecognised and >2 away from anything, so nothing was said. `SubagentStart` and `PostCompact` sat at 3 — one character from becoming the next `Setup`. - `Agent` was on the never-available denylist while its own deprecated alias `Task` sat in the built-in catalog. The vendor renamed it the other way in 2.1.63 and ships `tools: Agent, Read, Bash` as a worked example, so vigiles told delegating subagents to drop the one tool they exist to use. - `MultiEdit`, `BashOutput`, `KillBash`, `LS` appear zero times across six vendor doc pages. They were in the catalog; four real tools were not. So a valid name got an accusation or silence by spelling luck. The fix is not a longer array: `classify` returns one of four verdicts — available / withheld / conditional / unrecognised — and no near-match is in scope where the report-or-skip decision is made. `unrecognised` is the third option between silence and accusation: an advisory that names vigiles's own capture as the possibly-stale party, surfaced and never scored. The ≤2 threshold was not merely loose, it was set at the width of the collision: over 465 event pairs and 741 tool pairs, ZERO are 1 edit apart and exactly one pair in each is 2 apart — one of them being `Setup`/`Stop`. So distance 1 is scored as a typo and distance 2 stays advisory, and a test pins that measurement so a future catalog entry landing 1 edit from an existing name fails here rather than in someone's repo. Subtracts `confidentToolIssues` / `confidentHookEventIssues`: severity now travels on the issue, so scan, lint and compileAgent cannot disagree about which findings count. `builtinAgentTools` / `neverAvailableTools` / `sideEffectingTools` become projections of one authored vocabulary, and the conformance kit rejects a dialect that contradicts itself — the invariant whose absence let `Agent` sit in two lists while dialect-drift read it out of the vendor's own sdk-tools.d.ts every run. Also fixes an unrelated defect found on the way: the dialect-drift hook-event test never applied the stale-leftover reconciliation its sibling applies, so it only ever ran against whatever old package was lying around. Grades on faithful reductions that reproduce the recorded before-state exactly: SethGammon/Citadel F(57) -> C(73), thedotmack/claude-mem F(55) -> C(70). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(site): keep the browser engine's graph node-free; retire LS in the type test collectVocabularyNotes lived in scan.ts and scan-files.ts imported it as a VALUE, which pulled the node-only graph into the browser bundle: dist/scan-files.js → dist/scan.js → core/mcp.js → core/refs.js → core/symbols.js → @ast-grep/napi (a native .node binding esbuild has no loader for) Measured by walking the built CJS require graph, and confirmed by mutation: restoring the scan.js import makes the chain above reappear, removing it makes @ast-grep/napi unreachable. The functions are pure data transforms and now sit in scan-core.ts, the module whose stated job is to be node-free. test/types/purity.ts still declared LS, which left the catalog on 2026-08-17. It is now a negative case instead: a retired name has to FAIL at tsc time, since its absence from a list proves nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture for the vocabulary message The one changed line is the finding this PR exists to fix: the old rationale dumped the whole catalog at the reader — including MultiEdit, BashOutput, KillBash and LS, which are not Claude Code tools — instead of saying what is wrong with the name it was given. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7afa00f00a |
fix(refs): 17 false "file is missing" accusations, none of them true (#157)
* fix(scan): the reference extractor accused files that were present
Three false-accusation defects found by a 32-repo OSS dogfood sweep, all the
same shape — vigiles telling a maintainer a file they have is missing. For a
tool whose pitch is deterministic precision, a false positive costs more than a
miss: it is the failure mode that gets the tool uninstalled.
Measured on the corpus (91 audit units): 74 reference findings before, 57
after. All 17 removed are verified false; zero true positives lost, zero new
findings anywhere. Eight units change grade, every one upward — rsmdt/the-startup
F(54)→C(70), ayghri/i-have-adhd B(85)→A(100), fcakyon fable-advisor B(82)→A(90),
jjanczur/tyran D(64)→C(72), and Truthfulness 47→100 / 25→100 on two Microsoft
plugins.
One extractor was being run over five grammars. It now reads three, each through
that grammar's parser, and deliberately stops reading the other two:
markdown markdownRefs() in core/markdown.ts — a link's DESTINATION, and code
spans that are NOT inside link text. The old line-scanner reported a
link's LABEL as a missing resource while its destination resolved.
shell commandWords() in core/bash-effects.ts — the words a shell would
resolve, with inline program text (`node -e`, `python -c`) removed
and a nested `sh -c` parsed rather than discarded. The old regex cut
a character run out of a JavaScript payload and reported it as the
hook's script name.
source core/source-refs.ts — the one place a path-token pattern is built,
with both boundaries un-omittable. `js` could claim the head of
`json`; `(?:agents)/` could claim the tail of `claude-agents/`. Both
were live.
NOT READ: frontmatter (already stripped) and full-line comments in every
language, not only shell — both remaining corpus false positives here came out
of JSDoc.
Subtracted: four hand-written extension alternations (two of them duplicated
across the disk detector and its browser twin), a hand-rolled markdown link
regex, a hand-rolled inline-code regex, a whitespace split standing in for shell
words, and `stripShellComments`.
No new CLI verb, no new flag, no public API change (`api:check`: no drift).
Tests: 10 mutations, each proven applied by grep before running and each killed
by its own assertion, across the whole relevant suite. Every new case carries
both halves — fires on the planted defect, silent on the clean shape beside it.
* fix(scan): drop the redundant start-of-content guard in isPluginRooted
`startsAtSeparator` already answers idx <= 0 with true, and the next line
reaches it for idx === 0 (content[-1] is undefined, so never "/"). The guard
was a second copy of a boundary rule this module deliberately does not own —
the same duplication the module comment warns about — and it was the one
statement the 100% gate could not reach.
Also regenerates .vigiles/generated.d.ts for the new source files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
41b4da49a2 |
fix(coverage): the runner executed colocated tests and discarded the result (#156)
* fix(coverage): a colocated test that RAN is execution evidence, not a directory listing
Coverage answered "tested?" with two tiers: a recorded run (`executed`) and a
file sitting beside the surface (`colocated`). The second printed its own
caveat — "this says the file EXISTS, not that it ran" — and on a 52-surface
consumer repo it carried 23 of the 27 covered surfaces. Those 23 are ordinary
`*.harness.mjs` that `vigiles test` runs on every push. The run happened, the
runner watched it happen, and the metric described it as `ls` output, because a
harness asserting through `node:assert` reports a CHECK COUNT and no probe —
and `runsFromResults` discarded any run whose probe list was empty.
Measured on that corpus, before → after:
27 covered = 4 MEASURED BY A RUN + 23 colocated
27 covered = 27 MEASURED BY A RUN + 0 colocated
Coverage TOTAL is unchanged and untested stays 30: nothing gains or loses
credit, the evidence behind it moves. Zero attributions were downgraded.
The bar is a REPORTED CHECK, not exit zero. `touch <skill>/<skill>.harness.mjs`
— zero bytes — already drops the untested count by one, and `vigiles test` then
runs it and prints a tick, because an empty script exits 0. Attributing on
"ran and exited 0" would promote that file to "MEASURED BY A RUN": the same
emptiness wearing the stronger label. An empty script cannot report a check, so
it keeps its colocated credit and gains nothing — `statusFor`'s existing
undefined/0 distinction reused, not a new one.
`colocated` is a fourth RunAttribution and deliberately NOT a ProbeOrigin.
Origins are the wire vocabulary `parseCheckReport` reads out of a file the
CHILD writes; putting it there would let a harness hand-write
`{"how":"colocated"}` and mint coverage by declaring it — `vigiles:covers`
returning through the back door. Off `ProbeOrigin`, a script cannot spell it.
Colocation is the FALLBACK attribution, not an additional one. Emitting both
downgraded records: `mergeRuns` dedupes on (surface, tier, script) WITHOUT
`how`, so the second attribution replaces the first — measured, it turned
`.claude/hooks/paper-lint.mjs` from `command` into `colocated` and took three
`fired` activations with it.
SUBTRACTS a mirror: `isColocated` was written twice, and the copy's own
docstring said "Mirror of test-coverage.ts `isColocated`". One body
(`isColocatedTest`), three callers — disk detector, browser twin, run-record
builder. Colocation and execution were answering the same question with two
different pieces of code, which is why a script vigiles had just RUN could not
be attributed by the rule already willing to credit it for existing.
Also corrects a stale claim in coverage-probe.ts's header: it advertised the
in-process tier as recording nothing, which `harness-assert.ts` closed in
August. A header that advertises a gap somebody already closed sends the next
reader to build a second recorder.
Tests: 12 new, each with both halves (fires on a planted defect, silent on the
clean corpus), proven by 8 mutations each grepped for application first —
including one that came back green and exposed that the artifact's `how`
validation was pinned by nothing at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: regenerate generated.d.ts for the new colocated-execution test
The CI gate generated-types:committed compares the checked-in file against a
fresh generation; adding a source file without regenerating leaves them apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
51d439200a |
feat(spec): a skill could not declare the one key that actually removes a tool (#155)
* fix(compile): integrity header displaced a skill's frontmatter, so the harness lost its contract `addHash` prepended `<!-- vigiles:sha256:… -->` to every compiled file. For a SKILL.md that pushes the YAML frontmatter out of first position, and a reader anchored at `^---` then finds NO frontmatter at all — no name, no description, no allowed-tools. 🔴 THE INVARIANT WAS ALREADY WRITTEN DOWN, IN THE MODULE THE FIX LANDS IN. `ejectMarkdown` in integrity.ts has documented since it was written that a compiled SKILL.md "begins with YAML frontmatter (`---`) that MUST stay in first position — prepending an HTML comment there would push the frontmatter out of the lead block and the harness would lose the skill's name/description/tools". It honours that. `addHash`, two modules over, did the opposite to the same files. MEASURED, not reasoned. Five real skills from a live corpus (145–256 lines) were translated to minimal specs and compiled. The header was the ENTIRE delta: body byte-identical section order untouched diff 5 added lines, 1–2 removed — identical on all five regardless of size frontmatter original: found by a `^---` parser · compiled: NOT FOUND So one misplaced line was the whole reason a compiled skill could not be adopted. Everything else people feared about adoption (reordered sections, a generated `## Arguments`) turned out to be opt-in and absent unless `input()` is used. THE REAL DEFECT WAS DUPLICATION, and that is what this removes. Four sites independently encoded "the header is the first line": compile.ts HASH_RE, integrity.ts HASH_LINE_RE, coverage.ts, and cli.ts `targetHasHash`. Position now lives in ONE place — `findIntegrityHeader` / `placeIntegrityHeader` in integrity.ts — and the other three ask it. `targetHasHash` was the dangerous one: it decides whether a file is ours and safe to overwrite. Reading only the first line, it would have called every compiled skill hand-written prose and refused to update it. BACKWARD COMPATIBLE BY CONSTRUCTION. Every SKILL.md compiled before today carries the header first; such a file has no leading frontmatter (the header displaced it), so the same function finds it at the top. Without this, upgrading would report every previously-compiled file as hand-edited. Pinned by a test. Tests: 6 new cases in integrity.test.ts — placement with and without frontmatter, round-trip, tamper detection, the OLD layout, and plain markdown. Mutation-proved: reverting `placeIntegrityHeader` to prepend fails exactly the placement case (1 failed / 14 passed) and restoring passes 15/15. ⚠️ One defect was caught by these tests BEFORE shipping, not by review: the header is written a blank line below the closing `---`, and the finder anchored at position 0 of the body missed it. The file looked correct and `checkIntegrity` failed on it. The `\s*` in HASH_LINE_IN_BODY_RE is what makes it hold, and it is commented as load-bearing. Gates, in order: npx vitest run · npm run lint · npx prettier --check . · npm run api:check · npm run build — via `npm run check`, 17/17 in 77s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(spec): a spec value that was an object shipped as "[object Object]" with no error Second defect from the same 2026-08-17 measurement. Calling `input({ name, description })` — the object form a reader reasonably guesses — compiled with NO error and wrote this into the SKILL.md that would have been shipped: argument-hint: <[object Object]> - `$1` **[object Object]** — undefined WHY TYPES DID NOT CATCH IT, which is the part worth knowing. `input(name: string, hint: string)` is typed correctly. But `vigiles compile` loads a `.spec.ts` through tsx in a child process: transpiled, types erased, never checked. vigiles's OWN specs are cross-checked by a separate `tsc --noEmit` over a generated registry — and that safety net does not travel to the people compiling their own specs. So for every user, the type is documentation. Two guards, one concept each. 1. `input()` refuses a non-string name or hint and names the real signature. "expected string" does not tell the caller what to write instead, so the message carries the call shape. 2. `addHash` refuses any compiled body containing "[object Object]". This is the LAST GATE before a compiled file is written — all four compile paths return through it — so one check covers the whole family (file/cmd/ref/dir/glob/symbol/guidance are all unguarded the same way, and this catches them without eight new checks). Hard error, not a warning: measured zero occurrences of that string across every `.md` in this repository, so there are no false positives to trade against, and a file carrying it is broken in a way its author cannot see by reading the spec. Tests: 5 new cases. Mutation-proved, both with proof the patch landed — neutralising the `[object Object]` check fails exactly its case (1 failed / 19 passed); neutralising the `input()` check fails exactly its two (2 failed / 18 passed); restoring passes 20/20. Gates, in order: npx vitest run · npm run lint · npx prettier --check . · npm run api:check · npm run build — via `npm run check`, 17/17 in 76s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(adopt): the header move missed a caller, and a round-trip grew a stamp each pass Three tests went red on the placement change. Two were stale assertions; the third was a real defect in the fix itself, and only enumerating every caller found it. `splitFrontmatterBody` in adopt.ts consumes the leading frontmatter with a regex that tolerates an HTML comment BEFORE the block — precisely the old placement. Once the stamp moved below the frontmatter it landed inside what that function calls "the body", so adopt → compile → re-adopt carried the stamp into the spec's body, and every further pass added another one. Fixed by delegating to findIntegrityHeader, which is the one site that knows where the header can be — the reason it exists, and this was the caller the move missed. Both of its callers are covered by the change. The two stale assertions are rewritten rather than relocated. They anchored `/^<!-- vigiles:sha256/`, which is the OLD contract; simply moving the anchor would have restored a test that could not see the defect either way, because `name: reviewer` appears in the file under both placements. They now assert through readFrontmatter — a reader must FIND the block and read name and description out of it — which is the property the whole change is about. railway.test.ts is untouched on purpose: its fixture has no frontmatter, so the stamp is legitimately first there, and it now covers the other half of the contract. Also corrects a comment in frontmatter-read.ts that described the pre-move layout as current. Its optional-leading-comment branch is backward compatibility for files compiled before today, not what the compiler emits. Gates, in order: npx vitest run (2964 passed, 0 failures; scripts/release-preset fails to import @semantic-release/commit-analyzer, which is not a dependency — same local-only gap noted in #153) · npm run lint (0 errors, 206 pre-existing warnings) · npx prettier --check . (clean) · npm run api:check (11 entries, no drift) · npm run build (ok). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(spec): a skill could not declare the ONE key that actually removes a tool `SkillSpec` had `tools` and no fence. `AgentSpec` had both. So adopting a skill into a spec meant LOSING the ability to declare `disallowed-tools:` — and that is the only key measured to take a tool away: `allowed-tools:` is a pre-approval, it waives the prompt for what it lists and removes nothing from the pool. The result was that `audit` reported every compiled skill as holding all three lethal-trifecta legs, and the spec offered no way to answer it. The tool asked for a fence its own authoring surface could not express. 🔴 The key is HYPHENATED for skills. A subagent's is `disallowedTools:` (camelCase) and a different parser reads it, so emitting the agent spelling on a skill produces a key nothing looks at — inert, in the direction that reads as protection. That is the same class as the #107 defect two lines up, where `tools:` on a skill silently lost the contract, so the test asserts the exact bytes rather than a loose match. Writer and reader verified end to end, not assumed: `parseAgentToolList` was run against the actual compiled output and returns ["Bash","WebFetch"] for the emitted flow style, and null when the key is absent (the "no contract" signal the rail honors). A fence that compiles but does not parse would have been worse than none. Entries are typo-checked like a subagent's: "Wrte" removes nothing while the file reads as though Write were blocked, so it is an error, not a fence. Both new tests proved by mutation, each dying on its own assertion: emit `disallowedTools:` instead of `disallowed-tools:` → the key test fails disable the validation block → the typo test fails Gates in order: npx vitest run (2966 passed; scripts/release-preset fails to import @semantic-release/commit-analyzer, not a dependency — the known local-only gap) · npm run lint (0 errors, 209 pre-existing warnings) · npx prettier --check . (clean) · npm run api:check (11 entries, no drift after api:report — the surface diff is exactly the one new field in two reports) · npm run build (ok). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
4fef3779fa |
fix(compile): integrity header displaced a skill's frontmatter — the harness lost its contract (#154)
The header was prepended to EVERY compiled file, which for a SKILL.md pushes the YAML frontmatter out of first position — a reader anchored at `^---` then finds no frontmatter at all: no name, no description, no allowed-tools. The invariant was already written down in the very module this lands in; `addHash`, two modules over, did exactly the prohibited thing to exactly those files. Measured on five real skills (145-256 lines): the header was the ENTIRE delta. Body byte-identical, section order untouched, 5 added / 1-2 removed on all five regardless of size. So one misplaced line was the whole reason a compiled skill could not be adopted. The real defect was duplication — four sites independently encoded "the header is the first line". They now share one resolver in core/integrity.ts, which is the only place that knows where the header can be, and it reads both placements so files compiled before today still verify. Enumerating the callers found a fifth site the move had missed: splitFrontmatterBody in adopt.ts consumed a comment BEFORE the frontmatter only, so adopt -> compile -> re-adopt carried the stamp into the spec's body and grew another one each pass. Two stale assertions were rewritten, not relocated: they anchored the old contract, and simply moving the anchor would have restored a test blind to the defect either way, since `name: reviewer` appears under both placements. They now assert through readFrontmatter — a reader must FIND the block and read the fields out of it. Also: a spec value that was an object shipped as "[object Object]" with no error. Merged with `describe` red on an unrelated GitHub GraphQL 503; every substantive check (test, check, harness, site, e2e, validate) is green. |
||
|
zernie
|
1ebef8320a |
ci: one list of commands instead of two — CI now calls npm run check (#151)
* ci: один список команд вместо двух — CI зовёт `npm run check` Сегодня три PR уехали в `main` красными: список гейтов гонялся по памяти (пять команд), а джоб `check` гонял тринадцать. Упавшая — `tsc -p test/types/tsconfig.json` — живёт в ОТДЕЛЬНОМ tsconfig, которого корневой тайпчек не видит, и три `@ts-expect-error`-ассерта тихо перестали срабатывать. Длинный список, который надо помнить, это не чинит. Чинит то, что список СТАЛ ОДИН и CI его ВЫЗЫВАЕТ: `scripts/check.mjs` + `run: npm run check` вместо тринадцати шагов. Это стандартный паттерн поля (pre-commit / just / mise): определение задач — источник правды, CI его дёргает. Прежняя моя идея «завести гейт на совпадение двух списков» снята: она была подпоркой под дубль, который правильнее убрать. Стадии несущие, а не косметика. Часть команд ПИШЕТ файлы, которые другие ЧИТАЮТ (`generate types` → `.vigiles/generated.d.ts`, `api-extractor` → `api-surface/*`, `build` → `dist/`), поэтому писатели сериализованы, а параллельно идёт только read-only группа. 17/17 за 70 с против ~3 мин последовательно. Имена падающих шагов не потеряны — раннер печатает, что именно упало, и `--list` показывает команды без запуска. 🔴 Два шага `uses: ./` НЕ входят в `npm run check` намеренно: они догфудят собственный composite Action, у которого нет shell-эквивалента. Один из них (`eval-check`) при первой правке был снесён случайно — он стоял между двумя command-шагами, и правка по диапазону его проглотила. Возвращён, причина записана прямо над ним. Раннер сразу поймал два собственных промаха: неотформатированный `check.mjs` и протухший `generated.d.ts` после добавления скрипта `check`. Проверено: `npm run check` 17/17 ✓ · ci.yml парсится. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: словарь скилла уехал на сам билдер — experimental_skill.input/.step BREAKING: `input` и `step` больше не самостоятельные экспорты. Достаются как `experimental_skill.input(…)` / `.step(…)`. Пользователей нет. Зачем. Префикс `experimental_` — привычка, а привычка уже один раз протекла: `skill()` отгрузился со стабильным именем при доке «`skill()` is experimental». На самих `input`/`step` тега и префикса не было вообще, то есть словарь экспериментальной фичи выглядел стабильным. Повесив их на билдер, маркировку делаем СТРУКТУРНОЙ: до `input()` не добраться, не назвав `experimental_skill`. Забрано ровно два символа, и это замер, а не вкус. Перепроверено независимо от вердикта субагента: `input` и `step` встречаются только в скилл-спеках, а `cmd` (5 агентских спек), `file`, `ref`, `result` (8 агентских) общие — их спрятать в неймспейс скилла нельзя. 🔴 По дороге закрыта асимметрия, которую правка иначе бы УХУДШИЛА: дверь `vigiles/claude-code` своих `input`/`step` не экспортировала никогда, и типизированный автор брал их из второй двери. После снятия standalone- экспортов он остался бы без них совсем. Теперь CC-билдер несёт те же хелперы. Форма — `Object.assign`, не `export namespace`: последний запрещён нашим же линтом (`no-namespace: error`). Генерик пережил: в api-репорте `typeof skillSpec & { input; step }`, `ae-forgotten-export` нет. Честная граница записана и в коде, и в доке: `const { input } = experimental_skill` снимает маркер внутри файла. Гарантия у́же — немаркированное имя не пересекает границу пакета. Доки: `docs/skills.md` — пример на алиасе (`experimental_skill as skill`), маркер держится на строке импорта. `docs/experimental.md` — вступление утверждало, что префикс носит всё из `vigiles/experimental`, что после переезда `experimental_skill` в `vigiles/spec` стало неверным; плюс своя секция. `docs/README.md` — доиндексированы `testing-matrix.md` и `non-js-harnesses.md`. `safety.md` намеренно НЕ индексирован: он сам пишет «INTERNAL / WIP — not surfaced in the public docs yet». Гейты: npm run check 17/17 ✓ · vitest 2954 ✓. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b9302db4e6 |
feat!: one instability marker — the experimental_ name prefix, enforced (#147)
* docs: генераторный режим убран из публичной доки и припаркован `skill()` помечен экспериментальным. Из `docs/skills.md` убраны §3 (генератор) и секция тестов `vigiles/skill-test`, которая на нём стоит: `genSkill`/`act`/`checkpoint`/`finish`/`runSkill` не достижимы ни из одного сабпаса пакета — то есть снаружи репозитория такой скилл не написать. Ровно эти два фантомных сабпаса и заставляли держать в шапке рукописный баннер; баннер снят вместе с ними. Замерено при этом, и записано в §Status как причины экспериментального статуса: - `vigiles compile` генераторы ПОДДЕРЖИВАЕТ (`cli.ts` ловит `genSkill(`, прогнано end-to-end) — прежняя строчка §Status «wiring it into `vigiles compile` … is the next integration step» устарела; - скомпилированный `SKILL.md` ни разу не был показан загружающимся как скилл: `vigiles:sha256:`-шапка отодвигает YAML-фронтматтер с первой строки, и читатель, привязанный к `^---`, не находит фронтматтера вовсе. Все работающие `SKILL.md`, включая 13 своих, начинаются с `---`; шапку несут только два файла в `examples/`, которые никто не грузит; - принятие всё-или-ничего: `renderSkillSections` собирает документ целиком в фиксированном порядке, пути «моя проза + один проверяемый гейт» нет. Паркинг помечен в шапках трёх модулей, чтобы его не подняли молча. Публично экспортировать нечего: замер показал ноль достижимых символов. `scriptModel` НЕ паркуется — он публичен из `vigiles/claude-code` независимо от генератора. Заодно — дефект гейта из #146, найденный этой же правкой: он переиспользовал маркер `vigiles:ignore-file`, который принадлежит резолверу ссылок. Из-за этого единственный документ с двумя доказуемо битыми импортами был единственным, которого гейт не видел (91 блок сканировался, этот — нет). У гейта теперь свой маркер `vigiles:ignore-imports`; на дореформенном тексте он выдаёт ровно те два TS2307, ради которых писался баннер, на текущем корпусе молчит (92 блока, 0 находок). Тест на разделение маркеров прогнан мутацией: возврат общего маркера роняет оба ассерта. Гейты: vitest 2928 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · check-doc-imports 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: skill() → experimental_skill(), и правило, которое это держит BREAKING: `skill()` переименован в `experimental_skill()` на `vigiles/spec`, `vigiles`, `vigiles/linting` и `vigiles/claude-code`. Пользователей нет, мажор бампаем. Почему: дока открывалась словами «`skill()` is experimental», а имя на экспорте говорило «стабильный». Проза против механики — ровно тот дефект, который vigiles и ищет, у vigiles. Соглашение о префиксе существовало (`experimental_emitTool`, `experimental_parseEmitted`, `experimental_withServices`), нигде не было записано и ничем не проверялось. Побочно рейм снимает КОЛЛИЗИЮ, которой не должно было быть: `skill()` экспортировался из ВОСЬМИ сабпасов и был ДВУМЯ разными функциями — билдером спеки (`vigiles/spec`) и `Check<Trace>` по трассе (`vigiles/testing`, принимает id-строку, отвечает «сработал ли скилл»). Одно слово, два понятия, различимые только дверью импорта. Префикс несёт только билдер; `Check` не тронут (`vigiles-testing.api.md` без изменений). Правило: `scripts/check-experimental-naming.mjs` + шаг CI + `npm run experimental:check`. Публичное объявление с TSDoc-тегом `@experimental` обязано начинаться с `experimental_`. Ни одного нового маркера: тег уже используется в `experimental-emit.ts` и `services.ts`, публичность берётся из закоммиченных `api-surface/*.api.md`, которые `api:check` и так держит честными. Типы вне области: на дореформенной поверхности префикс несли только функции, а `ServiceSpec`/`ServiceHandle`/`ContainerRuntime` — нет; с типами гейт открылся бы 6 косметическими переименованиями против 1 настоящей находки и был бы заглушен в тот же день. Настоящая находка одна и закрыта: `makeDockerRuntime` → `experimental_makeDockerRuntime`. 8 тестов гейта, обе половины у каждого. Один нашёл баг в самом гейте: `\s+\S` в маркере-исключении удовлетворялся переводом строки и звёздочкой продолжения JSDoc, то есть голый маркер без причины проходил. Поправка к предыдущему коммиту: §Status утверждал, что sha256-шапка мешает загрузке скилла. Замер это ОПРОВЕРГАЕТ для соседней поверхности — в `reviewer-ab.eval.mjs` живой Claude Code загрузил скомпилированный `agents/code-reviewer.md` С шапкой через `--plugin-dir` и отдиспатчил в него, 100% прогонов на реальном sonnet (2026-06-20). Формулировка сужена до «непроверено», а не «сломано». Гейты: vitest 2936 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал устаревший импорт в docs/railway-subagents.md — правка внесена) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * docs: STABILITY.md объявлял стабильным то, что переименовано и помечено экспериментальным Найдено субагентом при разборе неймспейс-идеи, подтверждено мной. `STABILITY.md` — документ под заголовком «What's stable — depend on it» — перечислял `skill` среди стабильных билдеров `vigiles/spec`. После переименования такого символа нет вообще, а то, во что он переименован, в предыдущем коммите объявлено экспериментальным. То есть контракт стабильности обещал стабильность несуществующему имени. Третий экземпляр «проза против механики» в этой же области за два дня, и мой гейт его НЕ ловит: он сверяет TSDoc-тег с именем, а не документ с API. Заодно записано то, что вскрылось при проверке: - `input()` и `step()` используются ТОЛЬКО скилл-спеками, но экспортируются без префикса и без тега — то есть словарь экспериментальной фичи выглядит стабильным. Слепая зона гейта: он энфорсит «тег ⇒ имя», но не «на всём, что должно, стоит тег». - Маркеров нестабильности в проекте ТРИ и правила выбора между ними нет: `@internal` (39 объявлений), `@experimental` (18), префикс `experimental_` (8 публичных символов). Секция больше не утверждает, что маркер один. - У `pipe` тег `@internal` стоит только на ПЕРВОЙ перегрузке, поэтому остальные в api-репорте читаются как `@public (undocumented)`. Гейты: prettier ✓ · docs:check 0 · experimental:check 0 · lint 0 errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: один маркер нестабильности вместо трёх — префикс имени BREAKING: `pipe` `pipeStep` `needs` `start` `andThen` `effect` → `experimental_*` на `vigiles` и `vigiles/linting`. Пользователей нет. ## Выбрано **Экспортируем и не стабильно — имя начинается с `experimental_`.** Тег `@experimental` объявляет это один раз, `experimental:check` роняет CI, если у помеченного публичного символа нет префикса — тег и имя разъехаться не могут. `@internal` перестал означать «нестабильно»: он отвечает на ДРУГОЙ вопрос — входит ли символ в API вообще — и верен только на том, что мы не экспортируем. Почему имя, а не тег: тег не виден там, где решают. Имя видно в каждом вызове, дифе, ревью и автокомплите. Ровно так протёк `skill()` — стабильное имя при доке «`skill()` is experimental». ## Исправлено всё Было три маркера без правила выбора: `@internal` 39 объявлений, `@experimental` 18, префикс 8 публичных символов. Найдено 6 противоречий — символы с `@internal`, которые ЭКСПОРТИРУЮТСЯ; все шесть переименованы, теги переведены на `@experimental`. ## Два дефекта в гейте, отгруженном час назад 1. 🔴 Тег читался только если стоял на строке ОДИН (`@experimental\s*$`). Реальные теги несут прозу — `@internal Experimental typed-composition surface — …` у `pipe`, два в `services.ts`. Замер: **2 из 8** `@experimental` и **31 из 39** `@internal` объявлений были гейту не видны. Строгий паттерн выглядел безопаснее и был наоборот. 2. Между JSDoc и объявлением может стоять комментарий (`eslint-disable` перед перегрузками `pipe`) — привязка к следующему символу его теряла. После починки гейт видит 14 объявлений вместо 6. ## Записано `STABILITY.md` — секция «The one rule» + список; `CLAUDE.md.spec.ts` → перекомпилированный `CLAUDE.md` (правило №51). Плюс честно записанная дыра: гейт энфорсит «помечено ⇒ названо», но НЕ «всё, что должно быть помечено, помечено» — живой пример `input()`/`step()`, словарь только скилл-спеков, без тега и префикса. ## Третий раз за сессию Фикстура-спека в шаблонной строке (`generate-harness.test.ts:505`) не видна ни tsc, ни грепу по импорту — ловится только прогоном тестов. До этого то же в `cli.test.ts` (дважды). 5 новых тестов гейта (13 всего), включая закрепление обоих дефектов выше. Гейты: vitest 2941 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал `docs/cli.md:646`) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
a2af365682 |
docs: четыре утверждения о собственной публичной поверхности были ложными (#144)
Все четыре проверены командой, а не чтением, и каждое обещало читателю то, чего пакет не делает. 1. ШЕСТЬ SUBPATH, КОТОРЫЕ ПАКЕТ РЕКЛАМИРУЕТ САМ СЕБЕ И НЕ ОБСЛУЖИВАЕТ. `vigiles/eval`, `vigiles/run-hook`, `vigiles/harness-test`, `vigiles/harness-assert`, `vigiles/skill`, `vigiles/skill-test` — каждый даёт ERR_PACKAGE_PATH_NOT_EXPORTED. Они стояли в комментариях одиннадцати файлов examples/, в докстринге `harness-assert.ts` как готовый пример импорта, и в `e2e.ts`, который сам объясняет депрекейт и по дороге называет несуществующий путь. Заменены на существующие. 2. `src/testing.ts` УТВЕРЖДАЛ, ЧТО ЭТИ ПУТИ «still work». Обещание, что импорт продолжает работать, — ровно то, что читатель не станет перепроверять. 3. `vigiles/unit` ОБЕЩАЕТ «no `claude`, no model, no bubblewrap, no network». Из четырёх верны три: `judged` re-экспортируется отсюда, а его судья по умолчанию — реальный вызов модели (`check.ts`: `opts.judge ?? runJudge(o)`). Тест, импортирующий ТОЛЬКО из этого барреля, может тратить деньги, и единственная строка, по которой читатель заключил бы обратное, — эта. Честная починка — увезти `judged` с барреля, но это правка публичной поверхности, а поверхность сейчас пересматривается отдельно (замеренное предложение режет по СТОИМОСТИ, и там дефект становится невыразимым, а не задокументированным). До тех пор утверждение исправлено, а не оставлено. 4. `docs/skills.md` УЧИТ ИМПОРТИРОВАТЬ ТО, ЧЕГО НЕТ НИГДЕ. `genSkill` / `act` / `checkpoint` / `finish` / `runSkill` не экспортируются НИ ИЗ ОДНОГО subpath — проверено перебором всех. Генераторный API компилируется из исходника командой `vigiles compile`, и точки входа ему не завели. Форма скилла в документе описана верно; строки `import` — намерение, а не рабочий код, и теперь документ говорит это сам. Плюс `site/index.html`: alt-текст og-карточки называл показатель «Truthful refs», которого у продукта нет (`audit-score.ts`: Truthfulness · Triggering · Structure · Safety · Tested). Инструмент, который ловит расхождение прозы с механикой, показывал в собственном превью несуществующее имя своего же показателя. Гейты, поимённо и в порядке: npx vitest run (2926 passed, 173 файла) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
10e528e8db |
refactor(testing)!: изъять runMutations — она не про домен харнессов (#143)
* docs(testing): runMutations was in the reference only — the guide never mentioned it Measured after shipping it, not guessed: `runMutations` appeared in exactly ONE file, `testing-api.md`. Zero mentions in `harness-testing.md` (the task-first guide), `testing-matrix.md` and `docs/README.md`. The reference says of itself "the guide has the task-first how-to; this is the reference" — so anyone following the guide would never learn the API exists. And the runner table still opened with "Four runners" while listing five. That is the same defect the API was built to catch, produced by its own docs: prose and content disagreeing, with nothing to notice. - harness-testing.md — a routing row in "What do you want to test?" plus a full section, including the two things the reference did not say at all: WHEN NOT TO reach for it (a one-off script, an already-red test, or wanting a coverage number — mutation counts here say how many cases you wrote, not how good the suite is), and CI, where the honest answer is usually "don't": a run forks a process per case over full files, so it belongs in a hand-run pass after editing the test or the checker, or in its own job if wired at all. - testing-api.md — "Four runners" now says four for the harness plus a fifth that asks about your tests, which is what the table actually shows. - testing-matrix.md — a row for src/mutations.test.ts, naming the properties it covers, so the coverage map matches the suite. - docs/README.md — an entry under the API reference pointing at both. Gates: prettier · lint 0 errors · the 10 mutation tests still pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * revert(testing): withdraw runMutations — it does not belong in this package Shipped in #142 and 15.3.0; withdrawn a day later on the owner's question: «how is this vigiles' responsibility, it's just nodejs?» He is right, and the measurement is what settles it rather than taste. Against the rest of this package: runHook knows the Claude Code hook protocol, measureTriggerRate knows what a skill is, skillContract reads allowed-tools out of frontmatter. This module knows none of that. It replaces a substring, spawns `node`, and reads stdout — lift it into any Node repository and it works unchanged. A newcomer reading the testing docs hits exactly the owner's question, which is the cost this surface was charging. The confirming measurement, and the one I should have taken before merging: NO TEST IN THIS REPOSITORY USED IT. The only references were its own test file, its export line and its docs. The package did not dogfood the API it published. What actually fixed the problem it was extracted for was CONSOLIDATION, not RELOCATION. Ten hand-written copies of one driver had drifted apart in a consumer repository — the no-op guard in 4 of 10, the retry in 1 of 10. One shared copy buys all of that. Moving that copy into a product for harness authors was a second and separate step, and nothing justified it. The engine survives where its callers are, with the reasoning recorded in its header so it is not re-proposed. The shape that WOULD belong here is a different function that does not exist yet: mutate the HOOK — remove one branch of a guard — and require the assembled harness test to notice. Only vigiles knows what a hook is. Removed: src/mutations.ts, src/mutations.test.ts, the export block in src/testing.ts, the api-surface entry, and the docs sections in testing-api.md (from #142) plus the guide, matrix and README entries this branch had added. Gates, in order: npx vitest run (2916 passed, 172 files) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: перезапуск проверки заголовка PR Тип `revert` не входит в список, который принимает линтер заголовков (feat/fix/docs/chore/refactor/test/perf/ci/build). Заголовок переименован в `refactor(testing)!` — восклицательный знак несёт то, что `revert` нёс именем: изъятие опубликованного API ломающее. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b5da1c5604 |
feat(testing): runMutations — prove a test can fail, by breaking what it watches (#142)
A green test says the checker passed. It does not say the test would notice if the check were DELETED: an assertion can be vacuous, a fixture can be wrong in the same direction as the code, and both look exactly like a pass. `runMutations` plants a defect, runs the test that owns it, and requires that test to go red WITH THE MESSAGE THAT NAMES the defect. ## Why this belongs in the package It arrived as ten hand-written copies of one driver in a dogfooding repo (1799 lines), and the copies had drifted. Counted against the committed versions: - the NO-OP guard (a replacement equal to the original leaves a green test that proves nothing) was in 4 of 10; - the RETRY on a non-kill (a flake reported as a survivor sends the next reader to rewrite a working assertion) was in 1 of 10; - the strict rule — killed by its OWN named assertion, not merely by something going red — was in 1 of 10. Each was written AFTER it caught something, in whichever copy caught it, and never travelled. Two copies also named a test file a refactor had deleted, and the runner they used exits 0 on a path matching nothing, so those cases reported SURVIVED on every run for three days. ## Five verdicts, one of which is proof killed · wrong-assertion (a neighbour caught it) · survived · unjudgeable (the test was already red before the run, so "red" says nothing) · not-applied (find matched 0 or many, or the replacement equalled the original). The baseline matters more than it looks: without it, a test that is red ON PURPOSE — an open finding it exists to report — makes every run announce a failed restore about a working one, and makes every case against it look killed. ## What refuses rather than reports An empty case list and a named test file that does not exist both throw BEFORE any file is touched. A runner with nothing to run reports success, and that is the one claim this API exists to prevent. ## Verification vitest 2924 passed · lint 0 errors · prettier · api:check (additive only, +53 lines to vigiles-testing) · build. The new file has 10 tests, each building a real checker and a real test in a temp dir — a mocked runner would be judging a mock. Mutation-tested itself: removing the strict-kill rule, the no-op guard, or the existence check each turns the suite red on its own assertion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7a2f2c8def |
fix(experimental-emit): unsupported field type throws; the example finds a numbered heading (#141)
Two defects found by dogfooding this surface across five real pipeline skills. 1. `fieldSchema` was a switch over four literals with NO default, so a fifth type became `undefined` — and every later step read that as a field. `"actions" in properties` is TRUE for an undefined value, so nothing noticed; JSON.stringify then DROPPED the key while `required` kept it and `additionalProperties: false` forbade it. The served schema demanded a property it also banned, silently, while `.instruction` still told the model to send it. Unreachable from TypeScript, reachable from JavaScript — and the one shipped example of this API is `.mjs`, which is why it needed a runtime test. Verified by mutation: removing the new `default:` arm turns the regression test red on its own assertion. 2. The example matched its target section with a substring, so a numbered heading was a different section to it. Measured on the corpus: 11 of 15 eligible recorders reached, 4 refused for formatting alone (`## 6. Record the verdict`). Now the heading is read as a HEADING through markdown-it — a string search has no notion of heading level and cannot tell a heading from the same words inside a fence. 15 of 15 after. Two defects in that fix, both caught by running it over the real corpus rather than an invented example: the separator class needed whitespace (`Step 8 — record` has a space before the dash), and markdown-it reads YAML frontmatter as a SETEXT h2, so every skill produced a phantom heading whose text is the description — a description containing these words would have spliced the emit into the frontmatter. Gates, in order: npx vitest run (2915 passed) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift). Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cce1936460 |
docs: экспериментальная поверхность и именованная память (#140)
* wip: branch anchor
* feat(experimental): experimental_emitTool/parseEmitted/assertEmittedOk — a typed result carried by a TOOL CALL
An unforked skill has no call→return boundary, so compile hard-errors
output-without-fork on 31 of our 38 skills. A tool call needs no return
boundary: the skill EMITS mid-conversation and the call lands in
Trace.toolCalls. Same OutputContract, different delivery.
Marked experimental in the API itself — experimental_ prefix on every
runtime export plus a JSDoc block listing, by number, what is unproven
(N=3, no compile-time consumer, transport untested, runtime does not
enforce required) and what would have to be true to drop the prefix.
No new CLI verbs or flags.
shapeError is exported from the CC agent-result parser so both rails
validate a contract with the SAME code — a second copy would drift.
* examples(experimental-emit): the runnable measurement — MCP server + runner
emit-server.mjs serves ONE tool, read from JSON that experimental_emitTool
produced; hand-copying the schema would test the copy, not the API.
run-emit.mjs takes a real skill dir, refuses one with context: fork, swaps
exactly the '## Record the verdict' section for the emit instruction, runs
it through the real CLI and reads the result back from ctx.toolCalls.
Records are appended per trial, not buffered to the end. VIGILES_EMIT_DRY
rehearses the swap for $0. Not named *.eval.mjs so a CI sweep can never
reach a paid run.
* measure: 8 runs of an UNFORKED skill, 8 emits, 8/8 parse — plus a $3 defect in isRateLimited
The measurement. paper-status (no context: fork, so it can carry no output:
contract today) emitted a typed verdict by calling the tool, and the call
lands in Trace.toolCalls next to the Skill activation that spliced the skill
in inline. 8 runs, 8 emits, all ok, none repeated; scored offline for $0 by
score-emits.mjs through the same experimental_parseEmitted a test would use.
Two independent witnesses (MCP server log + CLI transcript) agree 8/8 on
track, count, verdict and report hash.
The run also exposed a real defect: the CLI emits an informational
{"type":"rate_limit_event",...,"status":"allowed"} line on EVERY run, and
isRateLimited's /rate.?limit/i matched it because . matches _. So every eval
trial silently ran retries+1 = 4 times, and maxCostUsd only ever saw the LAST
attempt: 2 trials became 8 model runs and a $0.60 cap was crossed at roughly
$3. Separator is now [ -]? plus an explicit rate_limit_error clause; the
telemetry line is captured verbatim as a fixture and asserted both ways.
Records are redacted (the long free-text report bodies are replaced by
head+length+sha256) and the raw files are gitignored: a run replays a skill
from a private corpus into a public repo.
* measure: run the TODAY-assertion against the same 8 outputs — green 8/8 while its number is NaN 8/8
The write-up asserted that the prose regex would break on real output. Asserting
that is the failure mode the standing rules name, so it is executed instead:
/BLOCKED/ matches 8 of 8 and /(\d+)\s+blockers?/ matches 0 of 8 — the assertion is
green and the count it exists to guard is NaN every time. Only the derived booleans
are published; the model prose describes a private corpus.
(Pushed via the GitHub API: git-over-HTTPS is failing in this container because
/root/.ccr/agent-proxy-ca.crt was truncated to 2 bytes mid-session.)
* docs: раздел про record()/state() + публичная страница экспериментальной поверхности
Обе фичи отгружены без единой строки в docs/. record()/state() приехали в #135,
experimental_emitTool — на этой ветке; ни то, ни другое из публичных доков не читалось.
docs/experimental.md намеренно отдельным файлом, а не абзацем в основных гайдах:
недоказанное не должно стоять в одном ряду с semver-стабильным. На странице —
что померено (N=8, 8/8), что НЕ померено (одна модель, один скилл, нет compile-time
пути, транспорт не проверен), чего оно НЕ покупает (схему рантайм не энфорсит,
Bash не уходит — allowed-tools это пред-одобрение), что должно стать правдой,
чтобы снять префикс, и стабильная альтернатива.
Прогнано: prettier --check docs/ · npm run lint · vitest по experimental-emit ·
vigiles lint (21 ссылка в код-блоках, все валидны).
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
82d89ae4de |
fix(agent-result): закрывающая ограда обязана начинать свою строку (#139)
* spike: closing fence must own its line, so a payload can carry a code block Measured 2026-08-13 (N=28 real sonnet runs): a result() contract whose string field is asked to hold a code snippet made every such answer "malformed". The model's JSON was VALID (newlines correctly escaped) but the old pattern's non-greedy body matched up to the first closing fence it found, which sat inside the JSON string. Fences in the PROSE before the block were never the problem. Not a feature; a spike branch. Raw records + the eval that produced them live in the private KB under vigiles/repro/output-contract-2026-08-13/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * spike: correct the measured-numbers comment (5/28, exact correspondence) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
f80a6333f6 |
feat(hook): writeTargets(prefixes) — API перестаёт выбрасывать, какой файл пишется (#138)
`commandView` already builds the list of write targets and collapses it to a boolean on the way out, so every consumer that needed a FILENAME rebuilt it from the raw command string. The shipped paper guard carries a 69-line character scan with its own quote tracking for exactly that, and that scan has three documented silent bypasses (a quoted path, an absolute path, a path eaten by a wrapper). - `writeTargets(prefixes)` returns the matched targets; `prefixes` is required and there is no unfiltered form, because a consumer holding the raw list has to re-implement the prefix matching that produced every one of those bypasses. - `writesTo` becomes `writeTargets(prefixes).length > 0` — one code path, so the boolean and the list cannot drift the way `runs()` and `writesTo` did. - `NormalizedLeaf.chdir`: `stripWrappers` read the `env -C` / `sudo -D` value in order to skip past it and then discarded it, so `env -C migratsiya sed -i papers/x.tex` reported NO write under `migratsiya/papers` — measured false before this change. A leaf's own chdir wrapper now resolves. A redirection is deliberately NOT joined onto it: the shell opens `> out.txt` in its own directory before `env` ever runs. Out of scope and pinned by a characterization assertion: cross-statement cwd (`cd x && …`, `pushd`, subshells). Connectors do not survive leaf extraction and `cd x; cmd` writes into the OLD directory when the `cd` fails — a model with failure semantics, not a field on a leaf. Both halves tested for every new behaviour, and both mutations verified to land before running: returning the unfiltered list fails 6 tests; restoring `writesTo` as its own extraction fails 2, including the dedicated derivation test. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
d6801511d5 |
fix(core): .. зажимается на корне Windows, а не съедает его (#137)
`resolveRef` collapses dot segments itself (core stays dependency-free, no node:path) with an unconditional `out.pop()`. That pops through the ROOT when the root lives inside the segment list — which is exactly the Windows case: C:/../repo/src/x.ts -> repo/src/x.ts (the DRIVE LETTER eaten) //server/share/../../repo/x -> //repo/x (share, then server, eaten) Real filesystems clamp: `C:/..` is `C:/`, `/..` is `/`. The damage is silent — a path naming the repo's own file came back relative-looking, stopped resolving against `C:/repo`, and an allowlist gate/react hook simply stopped recognising it. `isStampRepairEvent` refused the write that unwedges the repo. POSIX was already correct BY ACCIDENT, not by design: its leader `/` is held OUTSIDE `out`, so popping an empty array is already the clamp. Named in a comment so nobody "simplifies" it back. The floor comes from WINDOWS_ROOT — this file's single answer to "what is a Windows root", where both forms and the `//?/` spelling are already enumerated. Consistent with the two decisions the surrounding comments record: a `//` leader is the ROOT's answer (a POSIX stutter is not a share, round 37), while a drive letter names its filesystem BY ITSELF, as `isAtOrUnder` already says of a base. That second half is load-bearing rather than theoretical: `absoluteSpelling` resolves a ROOTLESS absolute path against the literal "/", so a floor gated on the root would have left `C:/../repo/x` losing its drive at that sibling call site — the one a fix written only where the defect was found would have missed. The `windows-fs?` predicate the UNC leader used inline is now shared, so the two questions cannot disagree about the same root. Tests, both halves each: fires on the defect (drive kept, UNC clamped at the share, rootless drive, repair door) and quiet on what already worked (ordinary `a/b/../c` still collapses under every root, POSIX `/..` unchanged, `//repo` + POSIX root still collapses, UNC without `..` untouched). MUTATION (proven landed by grep before each run, whole suite not one file): - restore the unconditional `out.pop()` -> 4 failures, all mine, e.g. "`..` clamps at a DRIVE root — the drive letter is not poppable", false !== true at hook-program.test.ts:1862. - drop the root gate so the floor is read off the operand -> 1 failure, "whether a `//` leader is a SHARE is the ROOT's answer, not the path's". No pre-existing test depended on the old behaviour. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
8c34e2096c |
fix(sidecar): классифицировать записи обхода через entryOf, а не голый statSync (#136)
* fix(sidecar): classify walk entries through entryOf, not raw statSync
`walk` in src/core/sidecar.ts decided "is this a directory" with a bare
`statSync(fullPath).isDirectory()`. `statSync` FOLLOWS symlinks, so a link
inside `.vigiles/` pointing at a directory outside it was descended into,
and a cycle re-entered the same tree lap after lap.
This is the THIRD instance of the class in this repo (after the loader's
`readTree` and the deleted surface enumerator), so the fix is the canonical
one: `entryOf` from src/fs-walk.ts, which refuses a symlinked DIRECTORY,
still reads a symlinked FILE (it cannot recurse), returns "skip" for
anything unreadable — exactly what the removed try/continue did — and
terminates by construction rather than a visited-set. No new helper, no
bookkeeping, no change to entryOf.
Measured, not assumed. Reverting the call site (grep-verified that the
mutation landed before running) fails both new tests:
does not descend into a directory symlink inside .vigiles/
AssertionError: + 'linked/Foreign.md'
reports a manifest once under a symlink cycle, not once per lap
AssertionError: + 'loop/Real.md', 'loop/loop/Real.md', … (41 entries)
The cycle result corrected the write-up: unlike the loader's uncaught walk,
this one never threw. Its own `catch` swallowed the ELOOP, so the defect
showed up as ONE manifest audited 41 times rather than as a crash — quieter,
not milder. The assertion is on the list of targets for that reason; an
"it didn't throw" check stays green under the defect and proves nothing.
Quiet halves included, since a fix that skipped every symlink would pass the
firing half alone and silently shrink the audit: nested sidecars
(`.vigiles/.github/copilot-instructions.md.inputs.json`) are still found, and
a symlink to a manifest FILE is still read.
Gates, in order: npx vitest run (2884 passed, 25 skipped) · npm run lint
(0 errors) · npx prettier --check . (clean) · npm run api:check (no drift) ·
npm run build (ok).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(sidecar): pin the symlink policy at the walk — both halves per shape
Tests half of the previous commit, split only because the local `git push`
could not run (the sandbox's proxy CA file was truncated to 2 bytes
mid-session, so TLS to github.com failed and the branch had to be published
through the API). The resulting tree is identical to the local commit's.
FIRES on the defect (both fail when the raw statSync is put back):
does not descend into a directory symlink inside .vigiles/
AssertionError: + 'linked/Foreign.md'
reports a manifest once under a symlink cycle, not once per lap
AssertionError: + 'loop/Real.md', 'loop/loop/Real.md', … (41 entries)
QUIET on the clean case, so a fix that skipped every symlink cannot pass:
still finds nested sidecars and follows a symlinked manifest FILE
(.vigiles/.github/copilot-instructions.md.inputs.json is still found,
and a symlink to a manifest FILE is still read)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
zernie
|
7af6f21e43 |
feat(hook): runtime-owned named state — a hook can remember a fact without reaching for the filesystem (#135)
* feat(hook): runtime-owned named state — record() + state(), and the two routing defects it exposed A compiled hook could read declared context but could not remember anything, so every advisory hook in the dogfood repo stayed hand-written shell. A hook now names a fact, the runtime stores it, and any hook in the same directory declares that name in needs and reads it back with its age. Deliberately NO throttle: field — throttling is this primitive used twice, and two measurements rejected the field. See the design note in core/hook-state.ts. Also fixes two routing defects the dogfood exposed, both measured against the real runtime: a react's tool filter disagreed with the matcher the compiler emits (a family matcher was silently dropped), and a react on a tool-less event (Stop) could never fire at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(hook): an empty tool list must match nothing, not the empty tool name Found by a mutation that was meant to disable tool-less reacts and did not: with no tools the joined pattern is ^()$, which matches the empty string, and a tool-less event's tool name IS the empty string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore(hook): api surface report, lint and prettier for runtime-owned state Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore(hook): regenerate the API surface after the rebase onto main --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cfa2fc04af |
fix(hook-runtime): resolve hook paths against a project root, so file and command gates stop failing on the spelling the harness actually sends (#134)
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since
|