mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
fix/hook-runtime-unused-err
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
45a7690ec2 |
feat: cheap hook-runtime startup, a docs-drift gate, and claims that were measured (#215)
* fix(site): replace unmeasured claims about other tools with a measurement The hero opened with "Nothing checks that the tools, events, files and rules your skills and hooks name actually exist." Nobody had ever run the other tool. Two things were wrong with the sentence: it is a universal negative over every tool that exists, which is unmeasurable, and on its own terms it is false for one of its four nouns. Measured instead, and the probe is committed as tools/measure-validate-overlap.mjs so the next reader reruns it rather than trusting prose. Against Claude Code 2.1.263, seven planted defects, both harness shapes: repo-local .claude/ 0 of 7 flagged, default AND --strict packaged plugin 1 of 7 — "hooks.PreToolUze: unknown hook event" The silence is real silence: the same binary errors on invalid manifest JSON and on a hooks.json missing its root key, and warns on an agent with no description, so it does read these files. That measurement moved claims in BOTH directions, which is the point: - Hero drops the universal negative and describes only what vigiles does. - Wedge credits `claude plugin validate` with the hook-event check it really makes, scoped to plugin form, and stops listing that check among the ones it misses. A draft of this paragraph had it in the wrong column. - docs/rules/hook-script-exists.md said the rule "matches Anthropic's own `claude plugin validate`, making vigiles a superset on the hook surface". Measured false — validate does not check hook-script existence at all. We were under-crediting our own rule. Corrected in place rather than deleted, so it does not read as a wording change and get added back. - Same false credit removed from the subagent-frontmatter keyFile: validate flags neither `model: sonnnet` nor `color: purpel`, and cclint was never run. Every claim of exclusivity, scope or comparison now carries a comment above it with the claim, the command that backs it and the version, what the earlier version said and why it was wrong, and what would invalidate it. The record of the mistake is the part that prevents the repeat. Two live falsehoods found on the page while verifying the above: - The first failure card and the `audit` example both used `event: "Setup"` as a hook event that does not exist. `Setup` IS a real Claude Code event, with its own docs sections; this repo's own catalog learned that on 2026-08-17 and the site never followed. The page's opening example of "config that parses and does the wrong thing" was config that works. Now `PreToolUze`, verified wrong in both directions against the real binary. - The `eval` row quoted the debunked claim as 65%. Three places in this repo and the linked post all say 65%. A debunk that misquotes its own target is the one number a reader checks in ten seconds. Also: the hero's language chip said "Any language · 11 linter catalogs" — an overclaim wrapped around a term a cold visitor cannot gloss, which never told a Python developer that Ruff and Pylint are covered. It now names the languages, still derived from BUILTIN_LINTERS, with a test that fails if a shipped linter has no language decision or a covered language is missing from the display order. Both directions proved by mutation. The landing-site skill is committed in this public repo and carried a "backend (roadmap)" design for an unbuilt capability plus a growth-loop framing, which no-product-strategy-here forbids here. The technical guidance stays; the roadmap and the growth narrative are gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: replace the unmeasured exclusivity claim with the mechanism "No other tool resolves rules across 11 catalog APIs" stood in five places. It rested on a competitor matrix whose cells were checked against DOCUMENTATION rather than against a run — the same matrix that put a false claim on the landing page, corrected in the previous commit after measuring the real binary. Two independent reasons to drop it, not one: 1. It was never measured. Verifying it means enumerating 11 catalog APIs across every comparable tool, which nobody did. 2. This repository is public, and its own `no-product-strategy-here` rule forbids competitive positioning here outright. "The core moat" and "no other tool" are positioning, so those lines were a rule violation independent of whether they happened to be true. Replaced with what the engine actually does, which a reader can check by running it: the rule name is resolved against the linter's own catalog rather than matched as a string, and the enabled state is read from the project's config. The removal is recorded in the source rather than performed silently. A silent deletion reads as a wording change and invites the claim back; the comment in linters.ts says what stood there and why it went. Sites: docs/linter-support.md, docs/what-vigiles-catches.md, src/core/linters.ts, CLAUDE.md.spec.ts, examples/CLAUDE.md.spec.ts (the two CLAUDE.md artifacts recompiled). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(compile): the section size guard measured the wrong unit `DEFAULT_MAX_SECTION_LINES = 200` reads as "a section may not sprawl" and never fired on the section that actually sprawled. Two reasons, both measured on this repo 2026-09-08: 1. The cap counts LINES. A `keyFiles` entry is ONE line of any length, so `## Key Files` reached 173 KB — 59% of the compiled CLAUDE.md — while staying far under 200 lines. Longest single description: 4782 chars. 2. `keyFiles` never reached the check at all. `compileKeyFilesSection` is a separate path that calls `validateFileRef` and nothing else, so the one section with no length control was the one that grew. Both closed: a character budget beside the line budget (whichever runs out first reports), and a per-entry cap on a key-file description, since an entry is a pointer and the reasoning belongs in that file's own header where a reader meets it on opening the file. RATCHETED, not bare. Introduced bare these caps open with 60 findings and get switched off the same day. `.vigiles/keyfile-debt.json` records the existing oversize once; the caps report only what is NEW or has GROWN, and a repo without the file gets every finding, which is the right default for everyone else. Lowering a number is the fix; raising one is the edit the file exists to make visible in review. Verified in three directions, not one: silent on the recorded debt, fires on an entry whose recorded size was lowered, fires on an entry removed from the debt entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(docs): withdraw the unmeasured #30519 matcher claim, and record the probe `docs/compiled-hooks.md` and CLAUDE.md both said the native `if:` matcher misses a compound `cd repo && git push -f`, citing #30519, and offered that as a reason to prefer the AST-backed `runs()`. Measured against Claude Code 2.1.263: the native matcher CAUGHT the compound command, in both spellings. Either the issue was fixed the way #34692 was, or the claim was wrong when written. Withdrawn rather than repeated; a focused re-measure decides. What the same run DID establish is a real gap, and a sharper one: form input hook spawned prefix git push --force origin main YES prefix /usr/bin/git push --force origin main NO glob git push --force origin main YES glob /usr/bin/git push --force origin main NO An ABSOLUTE program head is matched by neither spelling. That holds for `Bash(git push:*)`, the form Anthropic ships in its own security-guidance plugin, as much as for the glob form — so a guard written either way does not run on `/usr/bin/git push --force`, which is the whole guard when the hook body is unconditional. The probe is committed as tools/measure-if-matcher-forms.mjs so the next person re-runs it after a Claude Code minor instead of trusting this note. It measures someone else's matcher, and prose cannot notice when that moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(claude-md): key files are pointers again — 73k tokens to 43k `## Key Files` was 59% of the compiled CLAUDE.md. Its 266 entries had grown from index lines into paragraphs, median 371 characters, longest 4782 — and every one of them is loaded on every turn of every session in this repo. Measured before cutting anything: 220 of the 266 files ALREADY carry the same reasoning in their own header comment, usually in a fuller form. So this is mostly not a move, it is deletion of a duplicate that had drifted from the copy people actually read when they open the file. For the 17 entries whose file had no header, the prose was moved into one rather than dropped; 29 short entries were left alone. compiled CLAUDE.md 291579 -> 173682 chars ~72900 -> ~43400 tokens AND THE RATCHET IS GONE. The cap landed an hour earlier with a `.vigiles/keyfile-debt.json` recording 60 existing violations so it would not open loud enough to be switched off. That was the wrong order of work: a mechanism to tolerate the problem, invented instead of spending one pass fixing it. With the entries pruned the debt file is empty, so both it and the code reading it are deleted and the cap stands bare — 0 findings on a clean tree, fires on a lengthened entry. The section CHARACTER cap goes with it, for the same reason it should never have shipped ratcheted: `positioning` is 24877 characters and would fail it today, and a guard that cannot fire is the defect this product is about. That section is a real and separate piece of work, named here rather than recorded as debt. Verified in both directions on a clean tree: silent at 0 findings, and a mutation that lengthens one description fails on its own assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(tools): record the WIDENED if: forms, which are the ones that are sound The probe measured the two `if:` spellings found in the wild and stopped there, leaving the useful half unmeasured: what a native `if:` would have to look like for a compiler to be allowed to emit one. Measured on Claude Code 2.1.263: emitted if: plain absolute unrelated Bash(git push *--force*) fires NO no Bash(git push:*) fires NO no Bash(*git*) fires fires no Bash(*) fires fires fires The natural translation of a typed predicate is strictly NARROWER than the body it guards — it misses an absolute program head — so emitting it turns a guard off on exactly the input a careful author wrote it for. Widening to the PROGRAM NAME alone keeps every case the body would catch and still filters unrelated commands, which is the property an emission needs: it may only ever add spawns, never remove one. Flags and subcommand refinement are dropped on purpose. Including them is what makes the natural translation too narrow, and the body re-checks them anyway. Left explicitly unmeasured so nobody assumes it: the widening for `touches()` and `pipesToShell()`. Either may have none, and the honest answer for such a primitive is to emit nothing rather than something that looks like a filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(claude-md): the two "layer" paragraphs point at their guides again `## Positioning` was 24891 characters, and two of its twelve paragraphs were 64% of it: Second layer — testing the harness 7859 chars Third layer — COMPILED HOOKS 8108 chars Both are summaries of a document that already exists, is far richer, and was already linked from the same paragraph: docs/harness-testing.md 37114 chars 4.8x the summary docs/compiled-hooks.md 62167 chars 7.8x the summary Same shape as the key-files entries in the previous commit: a copy that duplicates the text people actually read, drifts from it, and is loaded into every session's context whether or not the turn is about hooks. What was KEPT in each is the part that constrains what gets built or claimed, because that has to be in context and cannot live behind a link: the tier discipline (push a question down a tier; only firing and behaviour-movement touch a real model), that the mock-model tier is table stakes and must not be sold as distinctive, the `experimental_*` marking, that compile and verify fix authoring and logic and never delivery, and that a gate is a strong default and never an unbypassable wall. What went to the guides is the mechanics: the tier APIs, the role family, the vocabulary, the stamp. ## Positioning 24891 -> 11361 chars CLAUDE.md 43420 -> 40038 tokens (72894 before this branch) The section CHARACTER cap is still not restored, and this says why rather than leaving it looking forgotten: `## Rules` is 82586 characters and would fail any cap that means anything. Rules are a different problem — measured across 53 of them, median 1449 characters, and unlike a key-files entry or a layer summary there is no richer copy elsewhere to point at, because a rule's text IS the instruction. Trimming it loses information rather than removing a duplicate, so it is judgement, not a mechanical pass, and a 500-char cap would open on 40 of 53 and be switched off the same day. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(hook)!: a bash gate is Bash by construction — delete its match field `HookProgram.match: { tool: string }` was a representable illegal state, and the same false-confidence class the module header says this subsystem exists to eliminate: if (rawEvent.tool_name !== program.match.tool) return allow(); `match: tool("Edit")` type-checked, compiled, wired a PreToolUse matcher `Edit`, fired on edits, built `commandView("")` from a missing `tool_input.command`, found every predicate false and returned allow() — a silently dead guard. The comparison was `!==` while every sibling role routes through `matchesTool`; that runtime/emit disagreement is the one MEASURED and fixed for reacts on 2026-08-12 and documented in this very file's header. It survived on the flagship role. The field carried no information either: `BashToolEvent.tool` is the literal "Bash", and all 34 call sites across both repos wrote `match: tool("Bash")`. So the fix is deletion rather than unification: the matcher moves to `hookRouting`, the runtime keeps a defensive `!== "Bash"` on a literal nobody can mis-set, and the author's file loses a line it could only get wrong. BREAKING CHANGE: `tool` is no longer exported from `vigiles/hook`, and `HookProgram` no longer accepts `match`. Both are part of the compiled-hook vocabulary, whose entry points are `experimental_*`; taken now precisely because the vocabulary is still young. `tools()` — the plural, for file gates and reacts — is untouched, and so is the `tool()` CHECK in `vigiles` (a different function that happens to share the name). Found by an architecture review that was asked to break a different proposal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(hook): restore the CHECK `tool()` a careless sweep took with it The previous commit removed `tool` from the compiled-hook vocabulary, and the regex that cleared its call sites also stripped `tool,` from `src/test.ts` — the `vigiles` ROOT barrel, where a DIFFERENT `tool()` lives: the check from `src/check.ts`, used as `assertChecks(run, [tool("Bash")])`. Two unrelated functions that happen to share a name, and the sweep could not tell them apart. That silently removed a stable public export. Nothing in the change was about it, the commit message did not mention it, and the api-surface diff that would have shown it was regenerated only afterwards. Caught by `docs-imports`, which type-checks the code blocks in `docs/` against the real surface and failed on `docs/testing-api.md:210` importing `tool` from `"vigiles"`. Worth noting which gate did it: the unit tests stayed green, because no test imports that symbol from the root barrel. A doc example was the only consumer standing in for every external one. Also completes the docs half of the previous commit, which shipped without it: `docs/compiled-hooks.md` still imported `tool` from `vigiles/hook` in three places, wrote `match: tool("Bash")` in four, and listed `tool(name)` in the vocabulary — all naming a function that no longer exists. And `api-surface/vigiles-hook.api.md` is regenerated, so the removal is visible as a reviewed diff rather than a surprise. Gates: `npm run check` — the only failure is `cli-lint`, whose 480 errors are all inside other sessions' gitignored worktrees (0 outside; CI never sees them). `docs-imports` 0 findings. `src/hook.test.ts` + `src/core/hook-program.test.ts` 120/120. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(examples): drop the `tool` import the sweep left behind Two example hooks kept importing `tool` from the hook vocabulary after the sweep removed their `match: tool("Bash")` line — and `tool` no longer exists, so the module failed to load: HookCompileError: Cannot load hook "examples/harness/protect-main.hook.mjs": Named export 'tool' not found. `compiled-hook-inprocess.harness.mjs` loads that file BY PATH and asserts over its decisions, so it went red. The runtime behaved correctly — a hook that cannot load must not pass traffic — which is why the failure surfaced as a refusal rather than a silent skip. HOW IT REACHED CI, which is the part worth recording: `npm run check` prints the CI jobs it does not cover, and `harness` was on that list. The change touched hooks throughout. I read the list, decided the harness tier was the known-environment-red one (`claude` 2.1.187 pinned vs 2.1.263 installed) and skipped it — while the note that records that redness says in as many words that redness IN CI is no longer environmental. The printed list did its job; I did not do mine with it. The unit tiers could not have caught this either: `src/hook.test.ts` writes its fixtures as source text and was updated with the sweep, so 120/120 stayed green while a shipped example was unloadable. Verified: `node examples/harness/compiled-hook-inprocess.harness.mjs` → 6/6. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(test): the `tool` import inside a fixture STRING, which no sweep reached `hook-state-store.test.ts:387` expected a throttled gate to stay quiet on a ten-minute-old fact and got `blocked: true`. The gate was not misbehaving — it could not load: const THROTTLED_GATE = `import { experimental_defineHook, tool, state, … }` `tool` was removed two commits ago, so the module threw on import and the runtime failed closed, which is correct behaviour for a gate that cannot run. Same in `hook-load-wedge.test.ts` and `core/hook-program.test.ts`. WHY THIS TOOK THREE ROUNDS TO FINISH, since that is the reusable part. The reference lived in four forms and a regex over source reaches one: real import in an example .mjs caught by the CI `harness` job re-export in a barrel caught by `docs-imports` import inside a template STRING caught by the CI `test` job code block in a doc caught by `docs-imports` That is verbatim the class already written down in this project's own notes after `.claude/pipeline/` was relocated — "four forms of a reference, of which a path search sees ONE" — including its conclusion that the only check which covers all four is one that RESOLVES the reference, not one that greps for it. I could have quoted the rule and still edited by regex and verified by a subset, three times running. Verified this time by the whole suite rather than the files I expected to break. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(test): a FIFTH spelling of the same import — built by concatenation `cli-coverage-record.test.ts` builds its fixture's import line with string concatenation: `import { experimental_defineHook, tool, deny, allow } from ` + JSON.stringify(HOOK_ENTRY) + … so no search for `from "…"` reaches it. Two tests failed because the fixture hook could not load and the runtime failed closed — correct behaviour, stale fixture. That is the fifth form this one removed symbol lived in. The running tally, because the tally is the finding: real import in an example .mjs caught by the CI `harness` job re-export in a barrel caught by `docs-imports` import inside a fixture template caught by the CI `test` job code block in a doc caught by `docs-imports` import assembled by concatenation caught by the full local suite Five spellings, four different gates, and every one of them found by a gate rather than by the sweep that created the problem. The lesson already written down in this project — that a reference search sees one form and only RESOLVING the reference sees all of them — held on every round. Verified: `npx vitest run src/cli-coverage-record.test.ts` 16/16. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: say why the emitted hook block has a matcher and no native `if:` Claude Code lets a hook action carry an `if:` precondition, checked before the hook process spawns. `vigiles compile` does not emit one, and the guide never said why — the first question a reader who knows Claude Code will ask, with no public home. That fails `document-the-why`. The rule the decision rests on: a prefilter may only ever be a proven SUPERSET of the typed predicate. Narrower and the process never starts, nothing is logged, and the guard reads as passing — the silent-guard bug this whole page exists to make unwritable. The safe superset is not a useful filter: `command.runs("git push", {force})` is AST-backed (compound, subshell, pipeline, absolute program head), and the narrowest string condition provably covering that is about `Bash(*git*)`. Same measurement as the "Matcher bypass" row (tools/measure-if-matcher-forms.mjs, 2.1.263). Two smaller reasons: `if:` is Claude Code only (Codex has no equivalent field, so emitting it would make one typed source mean different scopes per harness), and it would not buy the latency it appears to — same-event hooks run concurrently, so a filtered-out hook does not shorten the turn by its own duration. Per-event cost is the runtime's startup, which is where it belongs. Also cross-referenced from the "Runtime cost" limitation, which previously left `if:` looking like the obvious escape hatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: `tool` is no longer in the hook vocabulary — the Status list still named it `tool("Bash")` was removed from `vigiles/hook` earlier today: a bash gate is Bash BY CONSTRUCTION, the matcher is emitted for the author, and every call site in both repos wrote the same literal. The removal swept the imports, the `match: tool("Bash")` lines and the vocabulary bullet — and missed this one. The Status/pending section lists the vocabulary as prose to explain why only the entry points carry the `experimental_` prefix. `tool` sat in that list, naming a symbol that no longer exists, in a section about a different subject. Measured limit this exposes: nothing catches it. self-command-refs resolves `vigiles <cmd>` invocations; doc-refs resolves marks inside fenced ts blocks. A vocabulary word named in PROSE is a form neither resolves, and a grep for the symbol also matches the CHECK `tool()` in docs/testing-api.md, which is a different, live function — so the sweep that would have found it also had to tell two same-named symbols apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: reading an `if:` and emitting one are different acts — say so The new "why no native `if:`" section explained the refusal to EMIT a condition and left a reader free to conclude vigiles ignores the field entirely. That inference is wrong in a way that costs trust in a different feature: the test tiers already honour a hook's own `if:` when measuring it (`decideHookCondition`, src/core/hook-condition.ts, on main since #211), precisely so a guard is scored as the harness would really run it rather than as an unconditional stand-in that flatters it. One paragraph, stated as the asymmetry it is: we read yours, we do not generate one for you, because a generated filter can be narrower than the program behind it and a hand-written one is the author's own declared intent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: cut a paragraph that took five lines to carry one fact "None of this means vigiles ignores `if:`. Reading one and emitting one are different acts." — an aphorism standing in for a point, opening on a negation of an objection nobody raised, then five lines to say one thing. It broke this repo's own prose-clarity rule at the first sentence: lead with the point. Two lines now, the fact first: an `if:` you wrote is honoured, `runHook` reads it, only generating one is refused. Also dropped "a bad trade at any price" from the paragraph above — the preceding sentence already establishes that the failure is silent, so the flourish added emphasis to a point that was already made. Correction to |
||
|
zernie
|
7afa00f00a |
fix(refs): 17 false "file is missing" accusations, none of them true (#157)
* fix(scan): the reference extractor accused files that were present
Three false-accusation defects found by a 32-repo OSS dogfood sweep, all the
same shape — vigiles telling a maintainer a file they have is missing. For a
tool whose pitch is deterministic precision, a false positive costs more than a
miss: it is the failure mode that gets the tool uninstalled.
Measured on the corpus (91 audit units): 74 reference findings before, 57
after. All 17 removed are verified false; zero true positives lost, zero new
findings anywhere. Eight units change grade, every one upward — rsmdt/the-startup
F(54)→C(70), ayghri/i-have-adhd B(85)→A(100), fcakyon fable-advisor B(82)→A(90),
jjanczur/tyran D(64)→C(72), and Truthfulness 47→100 / 25→100 on two Microsoft
plugins.
One extractor was being run over five grammars. It now reads three, each through
that grammar's parser, and deliberately stops reading the other two:
markdown markdownRefs() in core/markdown.ts — a link's DESTINATION, and code
spans that are NOT inside link text. The old line-scanner reported a
link's LABEL as a missing resource while its destination resolved.
shell commandWords() in core/bash-effects.ts — the words a shell would
resolve, with inline program text (`node -e`, `python -c`) removed
and a nested `sh -c` parsed rather than discarded. The old regex cut
a character run out of a JavaScript payload and reported it as the
hook's script name.
source core/source-refs.ts — the one place a path-token pattern is built,
with both boundaries un-omittable. `js` could claim the head of
`json`; `(?:agents)/` could claim the tail of `claude-agents/`. Both
were live.
NOT READ: frontmatter (already stripped) and full-line comments in every
language, not only shell — both remaining corpus false positives here came out
of JSDoc.
Subtracted: four hand-written extension alternations (two of them duplicated
across the disk detector and its browser twin), a hand-rolled markdown link
regex, a hand-rolled inline-code regex, a whitespace split standing in for shell
words, and `stripShellComments`.
No new CLI verb, no new flag, no public API change (`api:check`: no drift).
Tests: 10 mutations, each proven applied by grep before running and each killed
by its own assertion, across the whole relevant suite. Every new case carries
both halves — fires on the planted defect, silent on the clean shape beside it.
* fix(scan): drop the redundant start-of-content guard in isPluginRooted
`startsAtSeparator` already answers idx <= 0 with true, and the next line
reaches it for idx === 0 (content[-1] is undefined, so never "/"). The guard
was a second copy of a boundary rule this module deliberately does not own —
the same duplication the module comment warns about — and it was the one
statement the 100% gate could not reach.
Also regenerates .vigiles/generated.d.ts for the new source files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
ff542b7c4b |
refactor!: vigiles audit — Lighthouse-style harness report (rename scan→audit)
Rename the read-only scan verb to `vigiles audit` — a local, zero-config Lighthouse-style report scoring a harness across four deterministic rings (Truthfulness/Triggering/Structure/Tested) with each finding's fix inline + a shareable HTML report and a versioned AuditReport JSON.
audit is a local report, not a CI step (CI uses `vigiles lint`). The executing checks (live MCP resolution + trigger-rate + the adoption preview) run only behind one interactive consent; there is no execution flag. The safety battery is not an audit ring — it lives in the vigiles/testing API.
Adds the adoption-gateway preview ("what would vigiles catch in your repo?"): the model drafts a spec from the instruction file, the cross-reference engine verifies every drafted ref deterministically.
BREAKING CHANGE: the `scan` verb is renamed to `audit` (no alias); the `--check-hooks`/`--verify-mcp`/`--trigger`/`--fix-plan`/`--explain`/`--deep`/`--measure`/`--fast`/`--no-measure` flags are removed.
|
||
|
zernie
|
1e798a2a4b |
feat: deterministic scan/lint rule suite for agent plugins (the cross-referencing moat) + scanner hardening (#40)
* fix: anchor scan surface regexes so a '-agents' skill dir isn't a phantom agent Pointing `vigiles scan` at popular OSS plugins (obra/superpowers, wshobson/agents marketplace) surfaced a real misclassification bug: the skill/agent/command path matchers were unanchored, so any directory whose NAME merely ends in the keyword was misread. superpowers ships `skills/dispatching-parallel-agents/SKILL.md`, and the `agents/[^/]+\.md$` matcher hit the `-agents/SKILL.md` substring — reporting a phantom agent named "SKILL" with no tool contract. That both corrupts the scan report and unfairly deducts from the leaderboard score (no-contract penalty). Anchor `skills/`, `agents/`, and `commands/` on a real path boundary (start-of-path or `/`). Add regression coverage for `-agents`/`-commands` skill directories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan traverses marketplaces and counts dangling refs as structural issues Two gaps surfaced while pointing `vigiles scan` at popular OSS plugins: 1. Marketplace traversal. `wshobson/agents` ships 80+ plugins under a single `.claude-plugin/marketplace.json`; scanning the repo root found nothing. `expandMarketplace` resolves each member's relative `source` (skipping external git sources and missing dirs), and `vigiles scan <root>` now expands a marketplace and ranks every member through the leaderboard. 2. Dangling intra-plugin references are now a first-class structural finding, not a buried free-text warning. `ScanReport.danglingRefs` exposes the broken-path / partial-vendor refs (obra/superpowers ships three); the report prints them under a "Broken references" ✗ section, the verdict counts them, and the leaderboard deducts -8 each. Previously superpowers reported "✓ no structural issues found" despite three missing files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: kill scan false positives that running against real plugins exposed Validating the marketplace scan against wshobson/agents (80+ plugins) and obra/superpowers showed the two heuristic findings were almost all FALSE POSITIVES — the exact doc-mention trap the leaderboard avoids. Promoting noisy signals to scored "bugs" would discredit the whole pitch, so tighten both detectors to fire only on real defects: - Dangling refs: scan only EXECUTABLE source files (hook/helper scripts, incl. extensionless ones like superpowers' hooks/session-start), never prose. Every hit across the corpus came from markdown — example task text, a literal "wc -w skills/path/SKILL.md" template, "❌ Bad" samples, React `hooks/*.ts` code comments — none real. The true positive (a script that cats a missing plugin file) is preserved. - Skill descriptions: parse YAML block scalars (`description: >` / `>-`). The regex captured only the `>` indicator, mislabeling four richly- described wshobson skills as "can't trigger". Net: the marketplace now reports 0 broken-ref and 0 missing-description false positives; remaining deductions are real (no tool contract, untested). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: log first behavioral findings — superpowers brainstorming under-triggers Ran the model-gated column (measureTriggerRate, real claude on the subscription) against the full obra/superpowers plugin to find bugs the deterministic scan structurally cannot see. Headline: the `brainstorming` skill — structurally green (good description, no scan finding) — fires on only ~20–30% of its own open-ended use-case prompts (recall 3/10 then 2/10 across two runs; precision a perfect 100% on mechanical edits). A diagnostic showed it isn't losing to a competitor: on 8/10 prompts no skill fired at all. The "You MUST use this before any creative work" imperative reads strong but fires weak — exactly the class of bug only a real model reveals. Records method, the over-trigger hypothesis the data refuted, an actionable fix for the author, the egress column we couldn't run here (no bwrap/slirp4netns), and honest n=10/1-trial caveats. Linked from distribution-strategy (E1) and divergent-bets (#9) so it isn't an orphan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empty-context validity + competitor landscape + auto-context idea Capture the methodology analysis from probing brainstorming: - Is empty context valid? Confirmed each trigger run is a cold single prompt (no fixture/history; TriggerRateSpec has no such field). It's the field STANDARD (AWS sample-agent-skill-eval does single relevant/irrelevant queries), faithful for opening-move skills (brainstorming qualifies — finding stands), biased-low for state-dependent ones (a shared blind spot, not vigiles-specific). - Landscape table: AWS (query, same), promptfoo (user-injected vars/hooks), DeepEval (ConversationalTestCase), Inspect/SWE-bench (seeded sandbox for stateful tasks). Nobody auto-reconstructs a skill's trigger state. - New idea — auto-apply known context: (1) add fixture/prior-turn to TriggerRateSpec (parity with measure()/runEval), (2) auto-select a curated preset context from the skill's declared trigger phrases so it's measured in the state it claims to fire on. Preset selection on explicit cues, not prose synthesis. Logged in roadmap.md (Next, P3-MED). The empty-vs-seeded sensitivity probe is noted as deferred (40 serial runs overran the budget here; re-run with concurrency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: roadmap — per-model trigger-rate + context-rot curve (MED) (a) Report trigger-rate per model — already a capability (model-as-arm), make it a standard column. (b) Study how recall rots as the skill roster grows and whether a stronger model rots slower — the concrete form of divergent-bets #11 (measure model × harness). Opt-in matrix, not a default gate; cheap first probe is brainstorming recall at 2 roster sizes × 2 models. Measure, don't claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: trigger-rate gets fixture (context) + concurrency for plugin testing at scale Two enablers for behavioral testing of OSS plugins, both surfaced by this session's probing limits: - `TriggerRateSpec.fixture` — seed files into each run's cwd so a skill is measured in the repo STATE it claims to trigger on. The default empty cwd is faithful for opening-move skills but under-measures state-dependent ones; this closes that gap (parity with measure()/runEval, which already had fixture). - `TriggerRateSpec.concurrency` — run the prompts × trials grid through the existing runPool instead of fully serial. Default 1 (rate-limit polite); raise it to cut wall-clock on a large prompt set or roster sweep. The serial runner is what made the earlier sensitivity probe overrun. Refactors runTriggerSet to flatten the grid into jobs + aggregate per-prompt (order preserved); extracts runTriggerTrial (also drops the function under the complexity ceiling). Unit tests with a fake runner prove fixture lands in the cwd before the agent runs, and that concurrency>1 actually parallelizes while aggregation stays correct. Docs + roadmap updated (step 1 of auto-context: done). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan --trigger behavioral column + record empty-vs-seeded sensitivity (a) scan --trigger — the model-gated behavioral column that stacks on the structural scan (not a separate command). For each model-invocable skill in a plugin it measures recall (+ precision) via measureTriggerRate and prints it as a "Behavioral (trigger-rate)" section. Author-supplied prompts (a JSON map of skill → { prompts, irrelevant }), never model-generated. Degrades honestly to "unavailable" with no claude/auth; a thin prompt set is surfaced per-skill, not a crash. Flags: --prompts= --concurrency= --model= --min-prompts=. New src/scan-behavioral.ts (probePluginTriggers[With] + formatBehavioralReport), keeps scanPlugin pure/sync; spawnAgent exported from eval.ts so the real runner is shared. Unit tests with a fake runner; verified live against superpowers (brainstorming recall 20%, rest unmeasured). (b) Sensitivity recorded — empty 0.00 vs seeded 0.20 (10 runs each, via the new fixture + concurrency). Inconclusive at n=10: empty itself spans 0.0/0.2/0.3 across three samples, so the +20pt delta is within the noise floor. The real lesson — n=10 can't resolve a 20pt effect — is documented. Docs: scan --trigger + marketplace + the -8 dangling-ref weight in cli.md; cross-link from harness-testing.md; sensitivity result in plugin-behavioral-findings.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: behavioral findings 2 & 3 — superpowers recall sweep + haretrail scan Finding 2 — superpowers recall sweep (vigiles scan --trigger across 6 opening-move skills, 10 prompts each): concrete imperative-task skills fire reliably (test-driven-development 100%, writing-skills 100%) while exploratory / orchestration skills barely fire (writing-plans 40%, brainstorming 10%, systematic-debugging 10%, dispatching-parallel-agents 0%). State-dependent skills left unmeasured on purpose (cold prompt = artifact). The 0-10% vs 100% gap on structurally-green skills is the finding. Finding 3 — fleytman/haretrail (Codex/AGENTS.md, 8 skills): structurally clean (no agents/hooks/broken refs). Description-level risks worth a trigger probe: Russian descriptions (cross-language trigger risk on English prompts), a literal {data-repo} placeholder, and overlapping domains (debrief/lessons/postmortem, task/research) = precision risk; good practice in the explicit trigger-phrase lists + postmortem's negative cue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan flags non-Latin skill descriptions as a cross-language trigger risk Operationalizes the haretrail catch as a deterministic, no-model check: each skill description's dominant script is detected via Unicode \p{Script=…} (no dependency), and a description ≥20% non-Latin (Cyrillic/CJK/Arabic/…) is flagged "cross-language trigger risk" — the selector is English-centric, so it may under-fire on English prompts. A RISK flag, not a structural defect (a language-matched audience is fine), reported separately from the verdict and pointing at the model-gated `scan --trigger` to measure the real gap. ScanSkill.descriptionScript carries the detected script; formatScanReport notes it per-skill + a summary line. Latin/English descriptions (and mostly-English with a stray accented word like "résumé") are not flagged. Verified on the real fleytman/haretrail (all 8 Cyrillic skills tripped) and superpowers (0 flagged). Also extracts skillLine() (clears a pre-existing nested-ternary warning). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — one shared detector for lint+scan; behavioral column is the exception Records the convention this session arrived at: a deterministic structural check has ONE home (a single pure detector reused by both `lint` and `scan`, never reimplemented in each, so they can't drift) — untested-surface, dangling-ref, and non-Latin-description detection already work this way. The model-gated behavioral column (scan --trigger / trigger-rate, observed egress) is the deliberate exception: it's LLM-based (auth, tokens, non-deterministic), so it lives only in scan/eval and is NEVER a lint rule. Cheap deterministic detection can be a lint rule; expensive model-gated confirmation cannot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: description-script check is expected-relative (default Latin) + Script union Per review: Latin isn't special, it's a configurable DEFAULT. unexpectedScript(text, expected="Latin") now flags the dominant script only when it DIFFERS from the expected one — so a Cyrillic-targeted pack sets expected="Cyrillic", its Russian descriptions pass, and an English one is flagged instead. Adds a `Script` string union (ts-essentials: literal union over bare string) used by the detector, the SCRIPTS table, and ScanSkill.descriptionScript. unexpectedScript is exported as the single shared detector the future lint rule will reuse (one-detector-no-drift). Reworded scan output + the CLAUDE.md rule from "non-Latin" to "unexpected script". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: eval-tier trace-parser seam (increment 1 of native Codex eval) + docs Native Codex eval (measureTriggerRate/runEval for Codex) needs three things; this lands the env-independent foundation and documents the rest. - Increment 1 (shipped): makeContext's Claude stream-json parsing is extracted as parseClaudeRun and injected as a ModelOutputParser (default = Claude, zero behavior change). measureTriggerRateWith takes one as a 3rd arg, threaded via TriggerRunConfig.parse. A unit test drives a non-Claude line protocol + a custom parser end-to-end and detects firing — the exact slot a Codex `codex exec --json` parser plugs into. - Spike recorded (no live binary here): the Codex driver's parseCodexRun returns toolCalls:[] by design, so the eval tier can't yet detect a Codex tool/skill call; and whether a Codex skill activation is even a discrete `codex exec --json` event is the open question only the binary can settle. Docs (internal + public): research/codex-prototype-findings.md gets the spike + increment plan + env-validation checklist; CLAUDE.md (eval.ts entry) notes the parser seam; docs/harness-testing-codex.md and docs/harnesses.md mark Codex runEval/trigger-rate as in-progress (not over-promised) with the three remaining steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scaffold Codex eval runner + tolerant trace parser (increment 2) src/adapters/codex/eval.ts (exported on vigiles/codex): - parseCodexEvalRun(out): ParsedModelRun — a TOLERANT, multi-shape parser of `codex exec --json` JSONL. No codex binary in the build env, so the line schema is unverified: it probes BOTH plausible Codex event models (older {msg:{type,…}} stream, newer {type:"item.*", item:{…}} thread/item stream) and degrades to empty / output=trimmed-stdout on anything unrecognised. Each extractor (assistantText / toolCall / usage) is isolated so finishing against captured JSONL is a field-name edit. Returns the Claude-shaped ParsedModelRun so it slots straight into the increment-1 ModelOutputParser seam. - codexEvalRunner — spawns real `codex exec --json` (v8-ignored; flags + skill-install wiring best-guess pending the binary). Unit-tested over synthetic fixtures for both shapes + junk tolerance + the output contract (src/adapters/codex/eval.test.ts). Deliberately NOT wired into the public measureTriggerRate({adapter}) dispatch (increment 3) — gated until the schema and the skill-selection question are confirmed against the live binary, so an unvalidated parser can't silently report recall 0. Findings doc + checklist updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: how to authenticate Codex for the eval tier (device-auth for headless) The deterministic tier is keyless; only the eval tier (real model) needs Codex auth. Documents the three flows in order of fit: `codex login --device-auth` (URL + one-time code, the only one that works in a remote/sandbox box — no localhost callback to forward), plain `codex login` (browser, local dev only), and `printenv OPENAI_API_KEY | codex login --with-api-key`. Notes the network- egress requirement, the `npm i -g @openai/codex` install (validated 0.141.0), and the version-sensitivity of the `codex exec --json` schema. Ticks the install/auth box on the env-validation checklist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: confirm Codex eval schema against the live binary + the skill-firing finding Authed real codex (ChatGPT) and captured real `codex exec --json` turns, turning increment 2 from a tolerant guess into a confirmed, live-validated parser. Findings (now documented internal + public): - Schema CONFIRMED (codex-cli 0.139.0): a thread/item JSONL stream — assistant = item.completed/agent_message→text; tool = command_execution→command (+ aggregated_output/exit_code); usage on turn.completed (incl. cached_input_tokens → cacheReadTokens). Count item.completed ONLY (item.started repeats the id). - THE skill question ANSWERED: Codex has NO discrete skill-selection event (no Skill tool). A trigger surfaces as the model READING skills/<name>/SKILL.md via a command_execution — so codexSkillFired() detects that read (best-effort; pair with a judged check). Validated live: a marker skill fired and was detected end-to-end through codexEvalRunner → parseCodexEvalRun → codexSkillFired. - Version regression: pin codex 0.139.0 (0.141.0 broke the keyless mock recipe). - Spawn gotcha: stdin must be /dev/null or codex blocks on stdin. parseCodexEvalRun rewritten to the confirmed schema; fixtures replaced with REAL captures; codexSkillFired added. Docs updated: harness-testing-codex.md (schema + skill-firing sections), harnesses.md footnote, CLAUDE.md (eval.ts clause + new Key Files entries), codex-prototype-findings.md (2026-06-19 live-capture update + checklist ticked). Remaining: increment 3 — {adapter} dispatch into measureTriggerRate/runEval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim public Codex eval docs to high-level (full findings stay internal) The previous commit over-shared internal detail into the public docs (the full codex exec --json JSONL schema, item.started/completed dedup, the version regression specifics, the stdin gotcha). Those belong in research/codex-prototype-findings.md (where they remain). Public docs (harness-testing-codex.md, harnesses.md) now keep only the user-relevant level: runEval-for-Codex is in progress, Codex has no skill-selected event (firing detected via the SKILL.md read), pin 0.139.0 — and link out for the schema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — match doc detail to audience (internal full record vs public link-out) Adds the public-vs-internal-docs guidance rule: research/*.md + CLAUDE.md hold the full record (schemas, parse mechanics, version regressions, spike notes, checklists); public docs (docs/, README) state only what a user needs to act + link out for the rest. Don't paste schemas/internal mechanics/version quirks into a user-facing guide unless a user needs them to use the feature. Codifies the trim just made to the Codex eval docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: haretrail native-Codex behavioral probe — cross-language risk refuted First native-Codex behavioral eval (codexEvalRunner → parseCodexEvalRun → codexSkillFired) on the real fleytman/haretrail skills, RU vs EN prompts: research 3/3 RU & 3/3 EN, summary 3/3 & 3/3, debrief 3/3 & 2/3. The Russian descriptions fire reliably on English prompts too — the deterministic "non-Latin description → cross-language risk" flag is a correct RISK flag that the model-gated measurement CLEARS here, so no "switch to English" fix is warranted. The single debrief/en miss confirms codexSkillFired discriminates. Logged in research (internal) per the audience-detail rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: codexRunError — an errored/rate-limited Codex turn isn't a "skill didn't fire" Dogfooding the haretrail precision probe hit a Codex usage limit, and every run came back all-miss: parseCodexEvalRun left the errored turn an empty trace, so codexSkillFired read it as a clean recall-0 miss — a rate-limit silently corrupting trigger-rate with false negatives. codexRunError(out) now detects the `error` / `turn.failed` event (incl. usage limits) so a caller skips/retries instead of scoring it (the Codex analog of the Claude path's isRateLimited backoff). Increment 3's dispatch must consult it. Test uses the real usage-limit capture. Precision/overlap question remains unmeasured (retry after quota resets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: native-Codex trigger-rate dispatch via { evalDriver } (increment 3) Wire the eval-tier harness dispatch so measureTriggerRate (and the scan --trigger behavioral column) can route through a non-Claude harness, with Codex as the first second-harness driver. - eval.ts: EvalDriver { runner, parse, runError? } bundles a harness's AgentRunner + ModelOutputParser + an optional errored-run detector; claudeEvalDriver (default) + measureTriggerRate({ evalDriver }) select one. A trial whose runError fires is excluded from the denominator (an errored/rate-limited turn isn't a clean miss) and counted as report.errored. - adapters/codex/eval.ts: codexEvalDriver (codexEvalAgentRunner + parseCodexEvalRun + codexRunError) + installCodexSkills (the Codex analog of --plugin-dir: copies skills into <cwd>/.codex/skills). - scan-behavioral.ts: a HarnessProbe (buildProbe(dir, harness)) selects the driver + per-harness "fired" predicate (skillResolved/stubbed for Claude, codexSkillFired for Codex), so scan --trigger --harness=codex routes through the Codex driver. - cli.ts: scan --trigger passes the detected harness through. Built + fake-tested (injected driver, no binary); a live native eval run remains, gated on Codex quota. Docs + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: live-validate the native-Codex trigger dispatch (numbers still quota-blocked) Re-ran the haretrail debrief/lessons/postmortem precision probe through the shipped public path (`vigiles scan --trigger --harness=codex`) now that the `{ evalDriver }` dispatch is wired. The Codex usage limit is still capped, so the recall/precision NUMBERS remain unmeasured — but the run live-validated increment 3's plumbing against the real binary: - the CLI dispatched through the Codex EvalDriver and spawned real `codex exec --json`; - `codexRunError` correctly EXCLUDED every errored trial (report showed `recall 0% (0 runs)` = n=0, not scored as misses) — the errored-run exclusion proven against reality, not just fakes; - the diversity gate fired live (rejected a too-similar irrelevant pair). So the only remaining gap is non-errored trials for actual numbers — a quota wall, not a code gap. Saves the eval-ready cluster prompt set at research/fixtures/haretrail-cluster.json for the retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: command-only / MCP-only plugins are a real surface, not score 0 scoreReport treated any plugin with no skills/agents/hooks as "no loadable plugin surface" (score 0/F) — but a command-only plugin (commands/*.md with a valid plugin.json) or an MCP-only plugin (.mcp.json) is legitimate. Anthropic itself ships command-only plugins (code-review, commit-commands) in its marketplace, and the leaderboard graded both 0/F. Count commands + MCP toward the loadable-surface check so they score on their real structural health (a clean command-only plugin → 100/A); a truly empty directory still scores 0. Found by scanning anthropics/claude-code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: close the two dogfood untested-surface gaps + fix a migrate-to-spec drift vigiles lint flagged two untested surfaces in our OWN plugin. Closing them also surfaced a real skill bug: - hooks/refs-nudge.sh: the sibling refs-hook.test.ts drives the CLI directly, so the SHELL WRAPPER (its not-a-node-project guard + exit-2-only propagation) was untested. Add a runHook unit test that drives the real script, pointing npx at a local node_modules/.bin/vigiles shim so it resolves offline. - skills/migrate-to-spec: a user-invoked prose skill (model-driven, nothing deterministic to test), but it SHOWS agents an exact `import { … } from "vigiles/spec"` template. The template imported `check` and `every`, which vigiles/spec does NOT export — following the skill produced specs that fail to compile. Fixed the template and added a dogfood test asserting every taught import is a real vigiles/spec export (catches skill ⇄ API drift; names the surface so it counts as covered). `vigiles lint` now reports all 9 surfaces tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface `scan` (audit any plugin) in README + Codex/command-only in cli.md - README: add a high-signal "Audit any plugin" bullet — the zero-setup, no-key scan/leaderboard is the strongest adoption hook and wasn't mentioned at all. Kept to one bullet per readme-brevity (links to the cli.md depth). - cli.md: note that command-only / MCP-only plugins score as real surfaces (not 0), and that `scan --trigger --harness=codex` routes the trigger probe through the native Codex driver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan reports the instruction file on any cc/codex repo + lint-vs-scan docs scan was plugin-oriented — on a plain repo (just a CLAUDE.md/AGENTS.md, no skills/agents/hooks) it printed an empty inventory and didn't even mention the instruction file, so it looked broken on the most common repo shape. - scan: report the detected instruction file (CLAUDE.md/AGENTS.md) with spec-managed vs hand-written, so it's useful on ANY cc/codex repo. This is a read-only informational fact, NOT the require-spec gate (that stays lint's). - docs/cli.md: add a "Lint vs scan — gate vs report" section with the full capability + target matrices (the answer to "do we need both / does it work on normal & Codex repos"). lint = config-driven gate (refs, 7 catalogs, exit codes); scan = zero-config read-only report (any plugin, leaderboard, + the paid --trigger column). They share one detector for the few common structural checks; everything else is intentionally asymmetric. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: e2e scan over the cc/codex/mixed/marketplace repo matrix Drive the real built CLI (node dist/cli.js scan) across every repo shape, in the free unit tier (deterministic, model-free, offline — like vendor.test.ts). Two fixture sources by design: - DOGFOOD pinned OSS plugins (examples/harness/vendor/*) for the Claude Code surface — real shapes (oh-my-claudecode multi-surface + MCP; the wshobson-accessibility inherits-all agent). - ARTIFICIAL tmp fixtures for shapes we have no vendored example of: a Codex repo (AGENTS.md + TOML [mcp_servers] + skill), a mixed CLAUDE.md+AGENTS.md repo (detection-ambiguity warning + --harness override), a CC instruction-only repo (the new instruction-file line), and a marketplace (leaderboard expansion). Covers the CLI scan WIRING the library-level scan.test.ts can't reach: harness auto-detection, the ambiguity warning, --harness=, the leaderboard branch, --json, and the read-only exit-0 contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan honest on curated marketplaces + dedupe name-aliased plugin dirs Two real bugs surfaced by sweeping ~13 popular plugin marketplaces (~650 plugins) + a variant mix of cc/codex/both repos: 1. A CURATED marketplace — every member an external git/url plugin, nothing vendored in-tree (obra/superpowers-marketplace, anthropics/claude-plugins- community with 2200 entries) — scanned as "nothing was loaded / no structural issues found", indistinguishable from a broken empty repo. Now inspectMarketplace classifies members on-disk vs external and scan reports `Marketplace "X": N plugin(s), all external (…not on disk)` honestly. 2. A marketplace that maps many NAMES onto the same dir (TheBushidoCollective/ han: 338 names → 159 dirs) made the leaderboard double-scan and rank the same plugin repeatedly ("(338 scanned)" with duplicate rows). expandMarket- place now dedupes by resolved path → "(159 scanned)", no dup rows. inspectMarketplace is the new source of truth (MarketplaceInfo: name/onDisk/ external/total); expandMarketplace delegates to it. The sweep also confirmed robustness: no crash across all marketplaces, and correct harness detection + instruction-file reporting across claude-only / codex-only / both-files repos (openai/codex, sst/opencode, wshobson, obra, …). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: eliminate scanner false positives found by sweeping real plugins Sweeping ~13 marketplaces + a variant mix of cc/codex/both repos surfaced that the scanner's "bugs" were mostly FALSE POSITIVES — a scanner that cries wolf on working plugins can't drive adoption. Fixed four classes (all verified against the real plugin that exposed them): 1. Multi-line QUOTED description (`description:\n "…"`, trailofbits/react-pdf) was mislabeled "no description / can't trigger". readField now gathers an indented quoted scalar, not just block scalars. 2. Relative hook path (`./hooks/x.sh`, ananddtyagi/cc-marketplace) was reported MISSING because existsSync was cwd-relative. resolveScript now resolves a relative hook against the PLUGIN ROOT. 3. Existence-guarded hook (`[ ! -f x ] || x`, gmickel/flow-next) — an optional run-if-present — was flagged MISSING. Now treated as a conditional one-liner. 4. A project/runtime path in a script (`$CLAUDE_PROJECT_DIR/.claude/hooks/…`, flow-next) matched mid-path as a plugin ref. danglingRefs/isPluginRooted now only flags PLUGIN-rooted paths (bare or ${PLUGIN_ROOT}/…), skipping literal nested dirs and project/home vars; a genuine plugin-rooted miss still flags. After hardening, the sweep's honest tally over 444 unique plugins: ONE real bug (ananddtyagi/cc-marketplace's claude-dev-infrastructure ships crisis-debugging- advisor + meta-skill-router with NO frontmatter → can't be model-triggered), and the popular plugins are otherwise structurally clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record the structural sweep + public-disclosures / FPs-fixed list New research/plugin-structural-findings.md (sibling of plugin-behavioral- findings.md) logs the deterministic scan sweep of ~13 marketplaces / 444 unique plugins: - Public disclosures: the one genuine external-plugin bug found — ananddtyagi/cc-marketplace's claude-dev-infrastructure ships two skills with no YAML frontmatter (can't be model-triggered). Status: found, not yet reported upstream + disclosure etiquette note. - Scanner false positives found by the sweep and fixed (the dogfood value): command-only 0/F, curated-marketplace "empty", han dedup, multi-line quoted description, relative ./hooks resolution, existence-guarded hook, project- rooted dangling ref — each with the plugin that exposed it + the fix commit. Linked from plugin-behavioral-findings.md and distribution-strategy.md (E1) so it's discoverable and not an orphan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: tool-contract validation — the cross-referencing moat in scan + lint The deterministic scanner described tool contracts but never VERIFIED them. Add the "valid is not true" moat to a subagent's `tools:` rail, as ONE shared detector (src/core/tool-contract.ts) reused by three callers (one-detector-no- drift): compileAgent (strict), scan (read-only), and a new agent-tool-contract lint rule (severity-gated). Calibrated HIGH-PRECISION after sweeping real plugins (the cry-wolf lesson): - flags only never-available tools (curated denylist) + close typos (edit distance ≤2, with did-you-mean) — the two confident cases. - a bare unrecognized tool is NOT flagged: a 280★ plugin uses TaskCreate/TaskGet/… consistently (plugin/MCP-provided), and our catalog can't know every tool — flagging those would be a false-positive flood. Two real bugs this surfaced and fixed along the way: - parseAgentTools didn't handle the inline ARRAY form (`tools: [Read, "Bash"]`) → it returned `[Read` / `Bash]`, which BOTH broke the PreToolUse rail (it would block Read) and tripped the check. Fixed at the source. - builtinAgentTools was incomplete (missing LS/MultiEdit/BashOutput/KillBash) — a latent false "unknown tool" error in compileAgent itself. Catalog fixed. Sweep result after calibration: 5 real never-available dead-contract entries across 444 plugins (wshobson team-lead `Agent`, developer-kit/ananddtyagi `AskUserQuestion`/`ExitPlanMode`), no flood. Leaderboard penalizes a dead tool like a broken ref. New rule defaults to warn (no-op on repos without agents/); doc + README + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: hook-event validation (#2) — typo'd hook events in scan + lint Cross-reference each hook's event name against the harness event catalog: a hook registered under a name the platform doesn't define silently never fires. Shared detector (src/core/hook-events.ts) → scan + the hook-events lint rule. Caught/fixed two things before it could ship: - dialect.hookEvents was WRONG and unused (PreSession/PostSession don't exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact were missing). Corrected against the events real plugins register. - Validating ALL unknown events flooded FPs: TheBushidoCollective/han ships a custom hook runtime (TeammateIdle, WorktreeRemove, …) — the event set is NOT closed. And ananddtyagi/sugar ships hooks as an ARRAY of {event:"tool-use"} objects (a non-CC format → Object.keys gave indices). So: high-precision (close typos only, via confidentHookEventIssues) + object-keyed shape only. After calibration: 0 on 444 swept plugins (FP-safe), catches PreToolUSe→ PreToolUse at authoring. Default warn (no-op on this repo); doc + README + spec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: frontmatter-schema validation (#3) — missing name/description in scan + lint A SKILL.md needs a `name` to load (CC keys a skill on its frontmatter name); a subagent needs `name` + `description` to register. A missing field is a structurally broken surface that silently never works. Computed in scanPlugin (frontmatterIssuesFor) and consumed by scan + the frontmatter-schema lint rule. Verified real on the sweep: 12 issues across 444 plugins, mostly ananddtyagi/ cc-marketplace — the headline bug (crisis-debugging-advisor / meta-skill-router ship with no frontmatter) is now caught precisely as "missing name", plus subagents (changelog-generator, …) that are pure prose with no frontmatter → never register. Confirmed by hand (changelog-generator.md opens with prose, no --- block). FP-safe (name is required; skill description stays on the skill line to avoid double-counting). Default warn; doc + README + spec + tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: MCP-config validation (#4) — unstartable MCP servers in scan + lint A declared MCP server must say how to reach it: a `command` (stdio) or a `url` (http/sse). An entry with neither can't start, and its tools silently go missing. Shared detector (src/core/mcp-config.ts) → scan + the mcp-config lint rule. FP-safe by construction: command-or-url is an unambiguous requirement, not a catalog that could be incomplete. Reads .mcp.json + manifest mcpServers (JSON CC shape); Codex TOML [mcp_servers] is a documented gap. Found 0 on the 444 swept plugins (they all configure command/url correctly) — survivor bias, as expected; the value is catching a malformed server at authoring time. Default warn; doc + README + spec + detector/scan tests. This completes the deterministic cross-referencing moat across the four agent surfaces: tool contracts (#1), hook events (#2), frontmatter schema (#3), and MCP config (#4) — each a shared detector reused by scan + lint (+ compileAgent for tools), all calibrated high-precision against the real-plugin sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: tool-contract treats `tools: *` as a wildcard (inherits all), not unknown A subagent declaring `tools: "*"` (all tools) was flagged `unknown "*" — did you mean "LS"?` (editDistance("*","LS")=2). A wildcard isn't a tool to validate. Skip "*" like the empty contract. Found on ananddtyagi/cc-marketplace's problem-solver-specialist. One more cry-wolf case closed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: frontmatter-schema is subagents-only — skills need no frontmatter (correction) Verified against the Claude Code docs (prompted by skepticism that "those skills never fire"): a SKILL.md requires NO frontmatter — `name` falls back to the directory name, `description` to the first body paragraph — so a frontmatter-less skill STILL loads and can fire. My earlier claim that ananddtyagi's crisis-debugging-advisor / meta-skill-router "never trigger" was WRONG. A subagent, by contrast, REQUIRES `name` + `description` with no fallback, or it won't register — confirmed by docs/en/sub-agents. So: - frontmatterIssuesFor now checks AGENTS ONLY (skill checks were false positives). - Corrected the disclosures doc: the real ananddtyagi bug is ~9 prose-only SUBAGENTS that never register (changelog-generator, content-creator, …), NOT the skills. Added a loud correction note. - Updated the rule doc (skills explicitly not checked + why), cli.md row, spec. Re-tally after correction: 9 agent-frontmatter bugs in ananddtyagi (real, docs-confirmed), the 2 skills + han's name-less skill correctly drop off. The "verify before you claim" discipline catching my own overclaim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empirically confirm the ananddtyagi agent finding (real claude CLI) Cloned ananddtyagi/cc-marketplace and drove the real claude CLI to verify the prose-only subagents really are inert (not just doc-inferred): listing subagents omits the prose-only one, dispatching it returns NO_SUCH_AGENT, while a control agent with frontmatter runs. Mirror test confirms a no-frontmatter SKILL still loads (dir-name fallback). Both the agent bug and the earlier skill correction now have ground-truth backing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: save superpowers trigger-probe fixture + session state before probe Persists (compaction-safe): the recall-probe prompt set (research/fixtures/superpowers-trigger-probe.json, 3 skills x 10 prompts) and a durable session-state note in plugin-behavioral-findings.md — the 4-rule moat shipped this session + the probe plan (results to be filled in after the run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: superpowers trigger probe results — systematic-debugging fires 10% Ran the recall probe on the SHIPPED scan --trigger path (claude 2.1.183, 10 prompts/skill on the subscription). Control test-driven-development fires 100% (validates the measurement); systematic-debugging fires only 10% and brainstorming 50%. A structurally-green skill silently never triggering on its own use case — the behavioral bug the deterministic column structurally can't see. Reconfirms the method-vs-situation trigger pattern on the shipped path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: skill-frontmatter rule — recommend explicit skill name+description (warn) Skills require NO frontmatter (name←dir, description←first paragraph), so a missing one isn't a load failure — but relying on fallbacks is fragile (the first paragraph is often a heading → a weak, accidental trigger surface). Add a best-practice NUDGE: scanPlugin.skillMetaIssues (a soft ℹ note, never counted as a structural defect or scored) + the `skill-frontmatter` lint rule. This is the correct form of the earlier "two rules" idea, split by severity/ intent: `frontmatter-schema` = subagents (REQUIRED → won't register, correctness) vs `skill-frontmatter` = skills (RECOMMENDED → reliable trigger surface, quality). Default warn (the skill works without it); set "error" to enforce on your own skills. vigiles's own skills pass. Doc + README + spec + tests. Note (follow-up): the pre-existing scan `hasDescription` "can't trigger" line is the same overclaim class — a skill with no frontmatter description still triggers via the first-paragraph fallback. Left as-is for now (pre-existing, load-bearing on the leaderboard); worth softening separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename frontmatter-schema rule → agent-frontmatter (symmetric pair) With two frontmatter rules now, `frontmatter-schema` (agents) vs `skill-frontmatter` (skills) was lopsided — one generic, one specific. Rename to `agent-frontmatter` so the pair is parallel and self-describing: agent-frontmatter (required → won't register) + skill-frontmatter (recommended → trigger surface). Unreleased on this branch, so no back-compat cost. Config key + doc (git mv to docs/rules/agent-frontmatter.md) + cli.md table + spec + research refs all updated; internal symbol names left as-is (not user-facing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: soften skill 'can't trigger' overclaim — honor the first-paragraph fallback The trigger-surface check flagged any skill without a FRONTMATTER description as 'can't trigger' — but Claude Code falls back to the first body paragraph, so a skill with prose still triggers. Now hasDescription = frontmatter description OR first body paragraph (firstBodyParagraph); only a skill with NEITHER is flagged. The last instance of the frontmatter overclaim class. The explicit-metadata best-practice stays the separate skill-frontmatter rule. (nodesc — name only, body just a heading — is still correctly flagged.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: ranked next-deterministic-rule ideas, grounded in the OSS sweep New research/deterministic-rule-ideas.md — the curated backlog of next moat surfaces, each the same 'valid is not true' cross-reference on a new surface, high-precision by design (the session's recurring FP lesson). Tier 1: mcp-tool-resolves (an mcp__server__tool whose server isn't declared), hook-matcher (typo'd tool in a hook matcher), frontmatter-valid (malformed YAML). Tier 2: duplicate-names, hook-shape, and the showpiece description-overlap — a DETERMINISTIC precision-collision proxy reusing the NCD engine (catches a --trigger-class bug with no model), grounded in haretrail's debrief/lessons/ postmortem overlap. Linked from roadmap Now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-tool-resolves rule (the MCP half of the tool moat) Cross-reference an `mcp__server__tool` in a subagent's contract against the plugin's declared `mcpServers` — a server the plugin doesn't declare can't resolve. Completes the tool moat: agent-tool-contract verifies built-in tools but passes any mcp__* token unchecked; this verifies the MCP half. One pure detector (src/core/mcp-tool.ts: verifyMcpToolServers) reused by scan + the mcp-tool-resolves lint rule (one-detector-no-drift). Calibrated high-precision via three guards, each grounded in a real plugin from the mid-2026 sweep: - GATE on a declared set — no .mcp.json → reaches global/project servers, flag nothing (ananddtyagi's mcp__ide__* shape). - ALLOWLIST harness built-ins via a new optional dialect.knownMcpServers (Claude Code's `ide`); optional/additive, non-breaking for adapters. - SKIP the plugin-namespaced mcp__plugin_<plugin>_<server>__ form (han's playwright-mcp — ambiguous join, the plugin's own server). Validated FP-free across the whole sweep (0 hits) and fires correctly on a synthetic undeclared-server plugin. Wired into scan (per-agent mcpToolIssues), lint (warn default), and the leaderboard score. Doc + detector tests + scan tests + CLAUDE.md/cli.md + regenerated API reports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: competitor rule matrix + verified OSS bug PR drafts Save the 3-agent competitor landscape sweep (eval tools, plugin/skill validators, MCP/security scanners) as research/competitor-rule-matrix.md: the rule-by-rule matrix vs agnix / claudelint / cclint / Anthropic's own `claude plugin validate`, the positioning finding (structural rules are converging to table stakes; the moat is semantic cross-reference + harness testing), and the ranked poach backlog. Add research/oss-pr-drafts.md with ready-to-file PR/issue text for two hand-verified third-party bugs the sweep found: subagents listing AskUserQuestion/ExitPlanMode (silently dropped — Anthropic #12890/#18721) and ananddtyagi's prose-only/blank-description subagents (won't register). Record the verified non-bugs (test fixtures, codex source tree) so they aren't re-chased. Cross-linked from plugin-structural-findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add hook-script-exists lint rule Promote scan's missing-hook detection to a lint rule: a hook command referencing a script file that doesn't exist on disk (with ${CLAUDE_PLUGIN_ROOT} resolved) silently never runs. Reuses scanPlugin's hooks status (one-detector-no-drift) — FP-safe by construction (skips unresolved $VARs, existence-guarded one-liners, inline commands). Matches Anthropic's own `claude plugin validate`, making vigiles a superset on the hook surface (poach #1 from the competitor matrix). Wired into lint (warn default) + exit code; doc + types/validate + cli.md table + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add disallowed-tools-contract lint rule (deny-side tool moat) Cross-reference a subagent's `disallowedTools:` block-list against the catalog — the mirror of agent-tool-contract. A close typo there blocks NOTHING (you meant to deny `Bash`, wrote `Bsh`), so the tool stays available silently — worse than no deny-list. New detector disallowedToolIssues in tool-contract.ts, calibrated high-precision: only a close typo of a real built-in flags; a real tool is legitimately blocked, a never-available tool is harmless to list, a bare unknown is likely a plugin tool (no cry-wolf). A check no other validator does (poach from SkillCheck). Generalizes parseAgentTools → parseAgentToolList(md, key) for the disallowedTools parse. Wired into scan (per-agent disallowedToolIssues) + lint (warn) + exit code + leaderboard; doc + detector/scan tests + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: agent-frontmatter also flags invalid model/color values Extend the agent-frontmatter rule from "missing required field" to also catch an invalid VALUE: a model: that's a close typo of an alias (inherit/sonnet/opus/ haiku) silently falls back; a color: typo is ignored. High-precision — close-typo only (≤2 edits), a full/dated model id (claude-sonnet-4-5) is an explicit form left alone, a far-off unknown is suppressed (no cry-wolf). Matches Anthropic's own `claude plugin validate` + cclint (poach #3). New frontmatterValueIssuesFor detector + frontmatterValueIssues on ScanReport, folded into the same agent-frontmatter lint rule + scan Frontmatter section + leaderboard; scan test; agent-frontmatter doc + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add description-overlap lint rule (deterministic precision proxy) The showpiece: two model-invocable skills with near-identical descriptions can't be told apart by the selector, so the wrong one fires (a precision collision). findDescriptionOverlaps (core/description-overlap.ts) reuses proofs.ts ncd to catch a --trigger-class behavioral bug with NO model — bridging the deterministic and behavioral columns, a check no other plugin linter has. Calibrated against the sweep: across 4678 within-plugin description pairs the most-similar legitimately-distinct pair (create-issue/create-pr) sits at NCD 0.25, nothing below it — so the cutoff 0.2 fires only on basically-identical text (FP-free on the whole sweep). User-invoked skills excluded (picked by command, can't collide). Wired into scan (descriptionOverlaps, reported as a precision risk) + lint (warn) + exit code; detector + scan tests; doc + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: mark second-wave rules shipped in the backlog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scan): section headers count entities, not output lines The "Agents (N)" / "Hooks (N)" headers used the rendered line count, so an agent's indented issue lines (and the inline-hooks summary line) inflated the number — "Agents (4)" for one agent with three findings. section() now takes an explicit count; Agents passes agents.length, Hooks passes hooks + inline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: one lenient frontmatter reader (real YAML + regex salvage) Replace the two divergent hand-parsers (scan's readField, agent-runtime's regex) with a single shared reader, core/frontmatter-read.ts: try js-yaml (block scalars, quoted multi-line, flow arrays parse correctly for free), fall back to a regex salvage on malformed input and record malformed:true. readFrontmatter never throws — it audits arbitrary third-party files — and frontmatterScalar / frontmatterList preserve the exact semantics the PreToolUse rail depends on (absent → null = inherits all, present-empty → [] = no tools). Anchored at file start (+ optional BOM + optional leading vigiles integrity comment) so a body --- hr is never read as frontmatter. Behavior-preserving (agent-runtime/scan/vendor/dogfood suites green); js-yaml was already a dependency. core/frontmatter.ts (the Level-1 vigiles: block) is a different concern, untouched. Sets up frontmatter-valid (the malformed flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add frontmatter-valid rule (malformed-YAML frontmatter) The payoff of the lenient-reader refactor: readFrontmatter's `malformed` flag becomes a rule — a skill/agent `---` block that exists but isn't valid YAML, so fields may not parse as intended. Calibrated and shipped CONSERVATIVELY per the verify-before-claiming discipline: a real YAML parser is stricter than some loaders, so a one-line description: with a `: ` colon / <example> block is flagged though it may still load. The sweep showed 7% of blocks flagged, concentrated in 2 repos (ananddtyagi, MadAppGang), 0 in the other 11 — a real authoring smell, but NOT empirically confirmed to fail loading. So: default WARN (not error), scan surfaces it as an informational ℹ note (not a structural defect, no leaderboard penalty), doc states the caveat and says verify before setting error. The unambiguous cases (unclosed bracket, tab, duplicate key) are real regardless. Same detector as scan (malformedFrontmatter); detector reuses the shared reader. Wired into lint + scan note; doc + scan test + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: dogfood the scan rules against real vendored plugins (FP-guard + bug fixture) Lock the deterministic rules against reality, offline + model-free, alongside the existing loader dogfood. src/scan-vendor.test.ts asserts golden scan verdicts: - FP-GUARD: oh-my-claudecode / wshobson-accessibility / superpowers must stay CLEAN — every high-precision rule field zero (the regression that catches a rule going noisy; the don't-cry-wolf bet). - TRUE-POSITIVE: a minimal MIT slice vendored BECAUSE it reproduces a real bug — madappgang-frontend@6097ad4 tester.md (a single agent file) — must keep firing agent-tool-contract (AskUserQuestion never-available) AND frontmatter-valid (the one-line description: is invalid YAML). Only the structural file is vendored (not the repo), only from a permissive (MIT) upstream — ananddtyagi has no LICENSE, so its files are NOT committed (its bug stays in research/oss-pr-drafts.md). Provenance + licensing in examples/harness/vendor/SOURCES.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: make the OSS bug drafts filing-ready (recipe + worked diffs) Add a per-repo filing recipe (branch / mechanical edit / conventional commit / gh pr create) plus worked before→after diffs for both verified bugs — the comma-form and array-form AskUserQuestion removal, and the missing-frontmatter prepend. States plainly these can't be filed from a vigiles-scoped session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add PR #40 handoff (resume-here state + next-session prompt) Single consolidated handoff for the branch: what shipped (the deterministic rule suite, scanner hardening, the lenient-reader refactor, the vendor dogfood, the research + OSS PR drafts), the disciplines that must carry forward (verify-before-claim, high-precision, calibrate-against-the-sweep, one-detector-no-drift, check-license-before-vendoring), the ranked open follow-ups (file the OSS PRs, README positioning, the remaining backlog rules), and a paste-ready next-session prompt. Linked from deterministic-rule-ideas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-hook-target-resolves rule (the moat on the hook surface) Cross-reference a `type: "mcp_tool"` HOOK action against the plugin's declared MCP servers (verified real: CC hooks support command/http/mcp_tool/prompt/agent, and mcp_tool requires server+tool). Flags two silent-no-op cases: an incomplete action missing server/tool (unambiguous, always — like mcp-config), and a server the plugin doesn't declare (gated on a declared mcpServers set + built-ins like `ide` allowlisted, mirroring mcp-tool-resolves). New detector core/mcp-hook.ts (verifyMcpHookTargets), wired into scan (mcpHookIssues) + lint (warn) + exit code + leaderboard; detector/scan tests; doc + types/validate + cli.md + CLAUDE.md. FP-free across the sweep (0 mcp_tool hooks in the wild — advanced feature; value is authoring-time + moat completion). The regex matcher surface is left to a future hook-matcher rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(readme): lead with the cross-reference moat; surface plugin audit Per research/competitor-rule-matrix.md: the structural lints (hook-events, agent-frontmatter, mcp-config) are now table stakes — Anthropic's own `claude plugin validate` ships them — so don't headline them. Elevate what's actually the moat: cross-referencing a declared reference against LIVE tooling. Extend the Lint section to the plugin surface — `vigiles scan` checks a plugin's contracts (subagent tools, mcp__server__tool, mcp_tool hooks, hook events, script paths) actually exist AND resolve, "a superset of Anthropic's `claude plugin validate`" (valid YAML ≠ a tool that's real). Retarget the old "Audit any plugin" More bullet to the leaderboard + --trigger so it doesn't duplicate. Stays within the readme-brevity budget (150 lines). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |