3 Commits

Author	SHA1	Message	Date
zernie	7afa00f00a	fix(refs): 17 false "file is missing" accusations, none of them true (#157) ... * fix(scan): the reference extractor accused files that were present Three false-accusation defects found by a 32-repo OSS dogfood sweep, all the same shape — vigiles telling a maintainer a file they have is missing. For a tool whose pitch is deterministic precision, a false positive costs more than a miss: it is the failure mode that gets the tool uninstalled. Measured on the corpus (91 audit units): 74 reference findings before, 57 after. All 17 removed are verified false; zero true positives lost, zero new findings anywhere. Eight units change grade, every one upward — rsmdt/the-startup F(54)→C(70), ayghri/i-have-adhd B(85)→A(100), fcakyon fable-advisor B(82)→A(90), jjanczur/tyran D(64)→C(72), and Truthfulness 47→100 / 25→100 on two Microsoft plugins. One extractor was being run over five grammars. It now reads three, each through that grammar's parser, and deliberately stops reading the other two: markdown markdownRefs() in core/markdown.ts — a link's DESTINATION, and code spans that are NOT inside link text. The old line-scanner reported a link's LABEL as a missing resource while its destination resolved. shell commandWords() in core/bash-effects.ts — the words a shell would resolve, with inline program text (`node -e`, `python -c`) removed and a nested `sh -c` parsed rather than discarded. The old regex cut a character run out of a JavaScript payload and reported it as the hook's script name. source core/source-refs.ts — the one place a path-token pattern is built, with both boundaries un-omittable. `js` could claim the head of `json`; `(?:agents)/` could claim the tail of `claude-agents/`. Both were live. NOT READ: frontmatter (already stripped) and full-line comments in every language, not only shell — both remaining corpus false positives here came out of JSDoc. Subtracted: four hand-written extension alternations (two of them duplicated across the disk detector and its browser twin), a hand-rolled markdown link regex, a hand-rolled inline-code regex, a whitespace split standing in for shell words, and `stripShellComments`. No new CLI verb, no new flag, no public API change (`api:check`: no drift). Tests: 10 mutations, each proven applied by grep before running and each killed by its own assertion, across the whole relevant suite. Every new case carries both halves — fires on the planted defect, silent on the clean shape beside it. * fix(scan): drop the redundant start-of-content guard in isPluginRooted `startsAtSeparator` already answers idx <= 0 with true, and the next line reaches it for idx === 0 (content[-1] is undefined, so never "/"). The guard was a second copy of a boundary rule this module deliberately does not own — the same duplication the module comment warns about — and it was the one statement the 100% gate could not reach. Also regenerates .vigiles/generated.d.ts for the new source files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-18 20:13:34 +05:00
zernie	ff542b7c4b	refactor!: vigiles audit — Lighthouse-style harness report (rename scan→audit) ... Rename the read-only scan verb to `vigiles audit` — a local, zero-config Lighthouse-style report scoring a harness across four deterministic rings (Truthfulness/Triggering/Structure/Tested) with each finding's fix inline + a shareable HTML report and a versioned AuditReport JSON. audit is a local report, not a CI step (CI uses `vigiles lint`). The executing checks (live MCP resolution + trigger-rate + the adoption preview) run only behind one interactive consent; there is no execution flag. The safety battery is not an audit ring — it lives in the vigiles/testing API. Adds the adoption-gateway preview ("what would vigiles catch in your repo?"): the model drafts a spec from the instruction file, the cross-reference engine verifies every drafted ref deterministically. BREAKING CHANGE: the `scan` verb is renamed to `audit` (no alias); the `--check-hooks`/`--verify-mcp`/`--trigger`/`--fix-plan`/`--explain`/`--deep`/`--measure`/`--fast`/`--no-measure` flags are removed.	2026-06-28 02:34:50 +05:00
zernie	1e798a2a4b	feat: deterministic scan/lint rule suite for agent plugins (the cross-referencing moat) + scanner hardening (#40) ... * fix: anchor scan surface regexes so a '-agents' skill dir isn't a phantom agent Pointing `vigiles scan` at popular OSS plugins (obra/superpowers, wshobson/agents marketplace) surfaced a real misclassification bug: the skill/agent/command path matchers were unanchored, so any directory whose NAME merely ends in the keyword was misread. superpowers ships `skills/dispatching-parallel-agents/SKILL.md`, and the `agents/[^/]+\.md$` matcher hit the `-agents/SKILL.md` substring — reporting a phantom agent named "SKILL" with no tool contract. That both corrupts the scan report and unfairly deducts from the leaderboard score (no-contract penalty). Anchor `skills/`, `agents/`, and `commands/` on a real path boundary (start-of-path or `/`). Add regression coverage for `-agents`/`-commands` skill directories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan traverses marketplaces and counts dangling refs as structural issues Two gaps surfaced while pointing `vigiles scan` at popular OSS plugins: 1. Marketplace traversal. `wshobson/agents` ships 80+ plugins under a single `.claude-plugin/marketplace.json`; scanning the repo root found nothing. `expandMarketplace` resolves each member's relative `source` (skipping external git sources and missing dirs), and `vigiles scan <root>` now expands a marketplace and ranks every member through the leaderboard. 2. Dangling intra-plugin references are now a first-class structural finding, not a buried free-text warning. `ScanReport.danglingRefs` exposes the broken-path / partial-vendor refs (obra/superpowers ships three); the report prints them under a "Broken references" ✗ section, the verdict counts them, and the leaderboard deducts -8 each. Previously superpowers reported "✓ no structural issues found" despite three missing files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: kill scan false positives that running against real plugins exposed Validating the marketplace scan against wshobson/agents (80+ plugins) and obra/superpowers showed the two heuristic findings were almost all FALSE POSITIVES — the exact doc-mention trap the leaderboard avoids. Promoting noisy signals to scored "bugs" would discredit the whole pitch, so tighten both detectors to fire only on real defects: - Dangling refs: scan only EXECUTABLE source files (hook/helper scripts, incl. extensionless ones like superpowers' hooks/session-start), never prose. Every hit across the corpus came from markdown — example task text, a literal "wc -w skills/path/SKILL.md" template, "❌ Bad" samples, React `hooks/*.ts` code comments — none real. The true positive (a script that cats a missing plugin file) is preserved. - Skill descriptions: parse YAML block scalars (`description: >` / `>-`). The regex captured only the `>` indicator, mislabeling four richly- described wshobson skills as "can't trigger". Net: the marketplace now reports 0 broken-ref and 0 missing-description false positives; remaining deductions are real (no tool contract, untested). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: log first behavioral findings — superpowers brainstorming under-triggers Ran the model-gated column (measureTriggerRate, real claude on the subscription) against the full obra/superpowers plugin to find bugs the deterministic scan structurally cannot see. Headline: the `brainstorming` skill — structurally green (good description, no scan finding) — fires on only ~20–30% of its own open-ended use-case prompts (recall 3/10 then 2/10 across two runs; precision a perfect 100% on mechanical edits). A diagnostic showed it isn't losing to a competitor: on 8/10 prompts no skill fired at all. The "You MUST use this before any creative work" imperative reads strong but fires weak — exactly the class of bug only a real model reveals. Records method, the over-trigger hypothesis the data refuted, an actionable fix for the author, the egress column we couldn't run here (no bwrap/slirp4netns), and honest n=10/1-trial caveats. Linked from distribution-strategy (E1) and divergent-bets (#9) so it isn't an orphan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empty-context validity + competitor landscape + auto-context idea Capture the methodology analysis from probing brainstorming: - Is empty context valid? Confirmed each trigger run is a cold single prompt (no fixture/history; TriggerRateSpec has no such field). It's the field STANDARD (AWS sample-agent-skill-eval does single relevant/irrelevant queries), faithful for opening-move skills (brainstorming qualifies — finding stands), biased-low for state-dependent ones (a shared blind spot, not vigiles-specific). - Landscape table: AWS (query, same), promptfoo (user-injected vars/hooks), DeepEval (ConversationalTestCase), Inspect/SWE-bench (seeded sandbox for stateful tasks). Nobody auto-reconstructs a skill's trigger state. - New idea — auto-apply known context: (1) add fixture/prior-turn to TriggerRateSpec (parity with measure()/runEval), (2) auto-select a curated preset context from the skill's declared trigger phrases so it's measured in the state it claims to fire on. Preset selection on explicit cues, not prose synthesis. Logged in roadmap.md (Next, P3-MED). The empty-vs-seeded sensitivity probe is noted as deferred (40 serial runs overran the budget here; re-run with concurrency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: roadmap — per-model trigger-rate + context-rot curve (MED) (a) Report trigger-rate per model — already a capability (model-as-arm), make it a standard column. (b) Study how recall rots as the skill roster grows and whether a stronger model rots slower — the concrete form of divergent-bets #11 (measure model × harness). Opt-in matrix, not a default gate; cheap first probe is brainstorming recall at 2 roster sizes × 2 models. Measure, don't claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: trigger-rate gets fixture (context) + concurrency for plugin testing at scale Two enablers for behavioral testing of OSS plugins, both surfaced by this session's probing limits: - `TriggerRateSpec.fixture` — seed files into each run's cwd so a skill is measured in the repo STATE it claims to trigger on. The default empty cwd is faithful for opening-move skills but under-measures state-dependent ones; this closes that gap (parity with measure()/runEval, which already had fixture). - `TriggerRateSpec.concurrency` — run the prompts × trials grid through the existing runPool instead of fully serial. Default 1 (rate-limit polite); raise it to cut wall-clock on a large prompt set or roster sweep. The serial runner is what made the earlier sensitivity probe overrun. Refactors runTriggerSet to flatten the grid into jobs + aggregate per-prompt (order preserved); extracts runTriggerTrial (also drops the function under the complexity ceiling). Unit tests with a fake runner prove fixture lands in the cwd before the agent runs, and that concurrency>1 actually parallelizes while aggregation stays correct. Docs + roadmap updated (step 1 of auto-context: done). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan --trigger behavioral column + record empty-vs-seeded sensitivity (a) scan --trigger — the model-gated behavioral column that stacks on the structural scan (not a separate command). For each model-invocable skill in a plugin it measures recall (+ precision) via measureTriggerRate and prints it as a "Behavioral (trigger-rate)" section. Author-supplied prompts (a JSON map of skill → { prompts, irrelevant }), never model-generated. Degrades honestly to "unavailable" with no claude/auth; a thin prompt set is surfaced per-skill, not a crash. Flags: --prompts= --concurrency= --model= --min-prompts=. New src/scan-behavioral.ts (probePluginTriggers[With] + formatBehavioralReport), keeps scanPlugin pure/sync; spawnAgent exported from eval.ts so the real runner is shared. Unit tests with a fake runner; verified live against superpowers (brainstorming recall 20%, rest unmeasured). (b) Sensitivity recorded — empty 0.00 vs seeded 0.20 (10 runs each, via the new fixture + concurrency). Inconclusive at n=10: empty itself spans 0.0/0.2/0.3 across three samples, so the +20pt delta is within the noise floor. The real lesson — n=10 can't resolve a 20pt effect — is documented. Docs: scan --trigger + marketplace + the -8 dangling-ref weight in cli.md; cross-link from harness-testing.md; sensitivity result in plugin-behavioral-findings.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: behavioral findings 2 & 3 — superpowers recall sweep + haretrail scan Finding 2 — superpowers recall sweep (vigiles scan --trigger across 6 opening-move skills, 10 prompts each): concrete imperative-task skills fire reliably (test-driven-development 100%, writing-skills 100%) while exploratory / orchestration skills barely fire (writing-plans 40%, brainstorming 10%, systematic-debugging 10%, dispatching-parallel-agents 0%). State-dependent skills left unmeasured on purpose (cold prompt = artifact). The 0-10% vs 100% gap on structurally-green skills is the finding. Finding 3 — fleytman/haretrail (Codex/AGENTS.md, 8 skills): structurally clean (no agents/hooks/broken refs). Description-level risks worth a trigger probe: Russian descriptions (cross-language trigger risk on English prompts), a literal {data-repo} placeholder, and overlapping domains (debrief/lessons/postmortem, task/research) = precision risk; good practice in the explicit trigger-phrase lists + postmortem's negative cue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan flags non-Latin skill descriptions as a cross-language trigger risk Operationalizes the haretrail catch as a deterministic, no-model check: each skill description's dominant script is detected via Unicode \p{Script=…} (no dependency), and a description ≥20% non-Latin (Cyrillic/CJK/Arabic/…) is flagged "cross-language trigger risk" — the selector is English-centric, so it may under-fire on English prompts. A RISK flag, not a structural defect (a language-matched audience is fine), reported separately from the verdict and pointing at the model-gated `scan --trigger` to measure the real gap. ScanSkill.descriptionScript carries the detected script; formatScanReport notes it per-skill + a summary line. Latin/English descriptions (and mostly-English with a stray accented word like "résumé") are not flagged. Verified on the real fleytman/haretrail (all 8 Cyrillic skills tripped) and superpowers (0 flagged). Also extracts skillLine() (clears a pre-existing nested-ternary warning). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — one shared detector for lint+scan; behavioral column is the exception Records the convention this session arrived at: a deterministic structural check has ONE home (a single pure detector reused by both `lint` and `scan`, never reimplemented in each, so they can't drift) — untested-surface, dangling-ref, and non-Latin-description detection already work this way. The model-gated behavioral column (scan --trigger / trigger-rate, observed egress) is the deliberate exception: it's LLM-based (auth, tokens, non-deterministic), so it lives only in scan/eval and is NEVER a lint rule. Cheap deterministic detection can be a lint rule; expensive model-gated confirmation cannot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: description-script check is expected-relative (default Latin) + Script union Per review: Latin isn't special, it's a configurable DEFAULT. unexpectedScript(text, expected="Latin") now flags the dominant script only when it DIFFERS from the expected one — so a Cyrillic-targeted pack sets expected="Cyrillic", its Russian descriptions pass, and an English one is flagged instead. Adds a `Script` string union (ts-essentials: literal union over bare string) used by the detector, the SCRIPTS table, and ScanSkill.descriptionScript. unexpectedScript is exported as the single shared detector the future lint rule will reuse (one-detector-no-drift). Reworded scan output + the CLAUDE.md rule from "non-Latin" to "unexpected script". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: eval-tier trace-parser seam (increment 1 of native Codex eval) + docs Native Codex eval (measureTriggerRate/runEval for Codex) needs three things; this lands the env-independent foundation and documents the rest. - Increment 1 (shipped): makeContext's Claude stream-json parsing is extracted as parseClaudeRun and injected as a ModelOutputParser (default = Claude, zero behavior change). measureTriggerRateWith takes one as a 3rd arg, threaded via TriggerRunConfig.parse. A unit test drives a non-Claude line protocol + a custom parser end-to-end and detects firing — the exact slot a Codex `codex exec --json` parser plugs into. - Spike recorded (no live binary here): the Codex driver's parseCodexRun returns toolCalls:[] by design, so the eval tier can't yet detect a Codex tool/skill call; and whether a Codex skill activation is even a discrete `codex exec --json` event is the open question only the binary can settle. Docs (internal + public): research/codex-prototype-findings.md gets the spike + increment plan + env-validation checklist; CLAUDE.md (eval.ts entry) notes the parser seam; docs/harness-testing-codex.md and docs/harnesses.md mark Codex runEval/trigger-rate as in-progress (not over-promised) with the three remaining steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scaffold Codex eval runner + tolerant trace parser (increment 2) src/adapters/codex/eval.ts (exported on vigiles/codex): - parseCodexEvalRun(out): ParsedModelRun — a TOLERANT, multi-shape parser of `codex exec --json` JSONL. No codex binary in the build env, so the line schema is unverified: it probes BOTH plausible Codex event models (older {msg:{type,…}} stream, newer {type:"item.*", item:{…}} thread/item stream) and degrades to empty / output=trimmed-stdout on anything unrecognised. Each extractor (assistantText / toolCall / usage) is isolated so finishing against captured JSONL is a field-name edit. Returns the Claude-shaped ParsedModelRun so it slots straight into the increment-1 ModelOutputParser seam. - codexEvalRunner — spawns real `codex exec --json` (v8-ignored; flags + skill-install wiring best-guess pending the binary). Unit-tested over synthetic fixtures for both shapes + junk tolerance + the output contract (src/adapters/codex/eval.test.ts). Deliberately NOT wired into the public measureTriggerRate({adapter}) dispatch (increment 3) — gated until the schema and the skill-selection question are confirmed against the live binary, so an unvalidated parser can't silently report recall 0. Findings doc + checklist updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: how to authenticate Codex for the eval tier (device-auth for headless) The deterministic tier is keyless; only the eval tier (real model) needs Codex auth. Documents the three flows in order of fit: `codex login --device-auth` (URL + one-time code, the only one that works in a remote/sandbox box — no localhost callback to forward), plain `codex login` (browser, local dev only), and `printenv OPENAI_API_KEY | codex login --with-api-key`. Notes the network- egress requirement, the `npm i -g @openai/codex` install (validated 0.141.0), and the version-sensitivity of the `codex exec --json` schema. Ticks the install/auth box on the env-validation checklist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: confirm Codex eval schema against the live binary + the skill-firing finding Authed real codex (ChatGPT) and captured real `codex exec --json` turns, turning increment 2 from a tolerant guess into a confirmed, live-validated parser. Findings (now documented internal + public): - Schema CONFIRMED (codex-cli 0.139.0): a thread/item JSONL stream — assistant = item.completed/agent_message→text; tool = command_execution→command (+ aggregated_output/exit_code); usage on turn.completed (incl. cached_input_tokens → cacheReadTokens). Count item.completed ONLY (item.started repeats the id). - THE skill question ANSWERED: Codex has NO discrete skill-selection event (no Skill tool). A trigger surfaces as the model READING skills/<name>/SKILL.md via a command_execution — so codexSkillFired() detects that read (best-effort; pair with a judged check). Validated live: a marker skill fired and was detected end-to-end through codexEvalRunner → parseCodexEvalRun → codexSkillFired. - Version regression: pin codex 0.139.0 (0.141.0 broke the keyless mock recipe). - Spawn gotcha: stdin must be /dev/null or codex blocks on stdin. parseCodexEvalRun rewritten to the confirmed schema; fixtures replaced with REAL captures; codexSkillFired added. Docs updated: harness-testing-codex.md (schema + skill-firing sections), harnesses.md footnote, CLAUDE.md (eval.ts clause + new Key Files entries), codex-prototype-findings.md (2026-06-19 live-capture update + checklist ticked). Remaining: increment 3 — {adapter} dispatch into measureTriggerRate/runEval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim public Codex eval docs to high-level (full findings stay internal) The previous commit over-shared internal detail into the public docs (the full codex exec --json JSONL schema, item.started/completed dedup, the version regression specifics, the stdin gotcha). Those belong in research/codex-prototype-findings.md (where they remain). Public docs (harness-testing-codex.md, harnesses.md) now keep only the user-relevant level: runEval-for-Codex is in progress, Codex has no skill-selected event (firing detected via the SKILL.md read), pin 0.139.0 — and link out for the schema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — match doc detail to audience (internal full record vs public link-out) Adds the public-vs-internal-docs guidance rule: research/*.md + CLAUDE.md hold the full record (schemas, parse mechanics, version regressions, spike notes, checklists); public docs (docs/, README) state only what a user needs to act + link out for the rest. Don't paste schemas/internal mechanics/version quirks into a user-facing guide unless a user needs them to use the feature. Codifies the trim just made to the Codex eval docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: haretrail native-Codex behavioral probe — cross-language risk refuted First native-Codex behavioral eval (codexEvalRunner → parseCodexEvalRun → codexSkillFired) on the real fleytman/haretrail skills, RU vs EN prompts: research 3/3 RU & 3/3 EN, summary 3/3 & 3/3, debrief 3/3 & 2/3. The Russian descriptions fire reliably on English prompts too — the deterministic "non-Latin description → cross-language risk" flag is a correct RISK flag that the model-gated measurement CLEARS here, so no "switch to English" fix is warranted. The single debrief/en miss confirms codexSkillFired discriminates. Logged in research (internal) per the audience-detail rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: codexRunError — an errored/rate-limited Codex turn isn't a "skill didn't fire" Dogfooding the haretrail precision probe hit a Codex usage limit, and every run came back all-miss: parseCodexEvalRun left the errored turn an empty trace, so codexSkillFired read it as a clean recall-0 miss — a rate-limit silently corrupting trigger-rate with false negatives. codexRunError(out) now detects the `error` / `turn.failed` event (incl. usage limits) so a caller skips/retries instead of scoring it (the Codex analog of the Claude path's isRateLimited backoff). Increment 3's dispatch must consult it. Test uses the real usage-limit capture. Precision/overlap question remains unmeasured (retry after quota resets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: native-Codex trigger-rate dispatch via { evalDriver } (increment 3) Wire the eval-tier harness dispatch so measureTriggerRate (and the scan --trigger behavioral column) can route through a non-Claude harness, with Codex as the first second-harness driver. - eval.ts: EvalDriver { runner, parse, runError? } bundles a harness's AgentRunner + ModelOutputParser + an optional errored-run detector; claudeEvalDriver (default) + measureTriggerRate({ evalDriver }) select one. A trial whose runError fires is excluded from the denominator (an errored/rate-limited turn isn't a clean miss) and counted as report.errored. - adapters/codex/eval.ts: codexEvalDriver (codexEvalAgentRunner + parseCodexEvalRun + codexRunError) + installCodexSkills (the Codex analog of --plugin-dir: copies skills into <cwd>/.codex/skills). - scan-behavioral.ts: a HarnessProbe (buildProbe(dir, harness)) selects the driver + per-harness "fired" predicate (skillResolved/stubbed for Claude, codexSkillFired for Codex), so scan --trigger --harness=codex routes through the Codex driver. - cli.ts: scan --trigger passes the detected harness through. Built + fake-tested (injected driver, no binary); a live native eval run remains, gated on Codex quota. Docs + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: live-validate the native-Codex trigger dispatch (numbers still quota-blocked) Re-ran the haretrail debrief/lessons/postmortem precision probe through the shipped public path (`vigiles scan --trigger --harness=codex`) now that the `{ evalDriver }` dispatch is wired. The Codex usage limit is still capped, so the recall/precision NUMBERS remain unmeasured — but the run live-validated increment 3's plumbing against the real binary: - the CLI dispatched through the Codex EvalDriver and spawned real `codex exec --json`; - `codexRunError` correctly EXCLUDED every errored trial (report showed `recall 0% (0 runs)` = n=0, not scored as misses) — the errored-run exclusion proven against reality, not just fakes; - the diversity gate fired live (rejected a too-similar irrelevant pair). So the only remaining gap is non-errored trials for actual numbers — a quota wall, not a code gap. Saves the eval-ready cluster prompt set at research/fixtures/haretrail-cluster.json for the retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: command-only / MCP-only plugins are a real surface, not score 0 scoreReport treated any plugin with no skills/agents/hooks as "no loadable plugin surface" (score 0/F) — but a command-only plugin (commands/*.md with a valid plugin.json) or an MCP-only plugin (.mcp.json) is legitimate. Anthropic itself ships command-only plugins (code-review, commit-commands) in its marketplace, and the leaderboard graded both 0/F. Count commands + MCP toward the loadable-surface check so they score on their real structural health (a clean command-only plugin → 100/A); a truly empty directory still scores 0. Found by scanning anthropics/claude-code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: close the two dogfood untested-surface gaps + fix a migrate-to-spec drift vigiles lint flagged two untested surfaces in our OWN plugin. Closing them also surfaced a real skill bug: - hooks/refs-nudge.sh: the sibling refs-hook.test.ts drives the CLI directly, so the SHELL WRAPPER (its not-a-node-project guard + exit-2-only propagation) was untested. Add a runHook unit test that drives the real script, pointing npx at a local node_modules/.bin/vigiles shim so it resolves offline. - skills/migrate-to-spec: a user-invoked prose skill (model-driven, nothing deterministic to test), but it SHOWS agents an exact `import { … } from "vigiles/spec"` template. The template imported `check` and `every`, which vigiles/spec does NOT export — following the skill produced specs that fail to compile. Fixed the template and added a dogfood test asserting every taught import is a real vigiles/spec export (catches skill ⇄ API drift; names the surface so it counts as covered). `vigiles lint` now reports all 9 surfaces tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface `scan` (audit any plugin) in README + Codex/command-only in cli.md - README: add a high-signal "Audit any plugin" bullet — the zero-setup, no-key scan/leaderboard is the strongest adoption hook and wasn't mentioned at all. Kept to one bullet per readme-brevity (links to the cli.md depth). - cli.md: note that command-only / MCP-only plugins score as real surfaces (not 0), and that `scan --trigger --harness=codex` routes the trigger probe through the native Codex driver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan reports the instruction file on any cc/codex repo + lint-vs-scan docs scan was plugin-oriented — on a plain repo (just a CLAUDE.md/AGENTS.md, no skills/agents/hooks) it printed an empty inventory and didn't even mention the instruction file, so it looked broken on the most common repo shape. - scan: report the detected instruction file (CLAUDE.md/AGENTS.md) with spec-managed vs hand-written, so it's useful on ANY cc/codex repo. This is a read-only informational fact, NOT the require-spec gate (that stays lint's). - docs/cli.md: add a "Lint vs scan — gate vs report" section with the full capability + target matrices (the answer to "do we need both / does it work on normal & Codex repos"). lint = config-driven gate (refs, 7 catalogs, exit codes); scan = zero-config read-only report (any plugin, leaderboard, + the paid --trigger column). They share one detector for the few common structural checks; everything else is intentionally asymmetric. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: e2e scan over the cc/codex/mixed/marketplace repo matrix Drive the real built CLI (node dist/cli.js scan) across every repo shape, in the free unit tier (deterministic, model-free, offline — like vendor.test.ts). Two fixture sources by design: - DOGFOOD pinned OSS plugins (examples/harness/vendor/*) for the Claude Code surface — real shapes (oh-my-claudecode multi-surface + MCP; the wshobson-accessibility inherits-all agent). - ARTIFICIAL tmp fixtures for shapes we have no vendored example of: a Codex repo (AGENTS.md + TOML [mcp_servers] + skill), a mixed CLAUDE.md+AGENTS.md repo (detection-ambiguity warning + --harness override), a CC instruction-only repo (the new instruction-file line), and a marketplace (leaderboard expansion). Covers the CLI scan WIRING the library-level scan.test.ts can't reach: harness auto-detection, the ambiguity warning, --harness=, the leaderboard branch, --json, and the read-only exit-0 contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan honest on curated marketplaces + dedupe name-aliased plugin dirs Two real bugs surfaced by sweeping ~13 popular plugin marketplaces (~650 plugins) + a variant mix of cc/codex/both repos: 1. A CURATED marketplace — every member an external git/url plugin, nothing vendored in-tree (obra/superpowers-marketplace, anthropics/claude-plugins- community with 2200 entries) — scanned as "nothing was loaded / no structural issues found", indistinguishable from a broken empty repo. Now inspectMarketplace classifies members on-disk vs external and scan reports `Marketplace "X": N plugin(s), all external (…not on disk)` honestly. 2. A marketplace that maps many NAMES onto the same dir (TheBushidoCollective/ han: 338 names → 159 dirs) made the leaderboard double-scan and rank the same plugin repeatedly ("(338 scanned)" with duplicate rows). expandMarket- place now dedupes by resolved path → "(159 scanned)", no dup rows. inspectMarketplace is the new source of truth (MarketplaceInfo: name/onDisk/ external/total); expandMarketplace delegates to it. The sweep also confirmed robustness: no crash across all marketplaces, and correct harness detection + instruction-file reporting across claude-only / codex-only / both-files repos (openai/codex, sst/opencode, wshobson, obra, …). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: eliminate scanner false positives found by sweeping real plugins Sweeping ~13 marketplaces + a variant mix of cc/codex/both repos surfaced that the scanner's "bugs" were mostly FALSE POSITIVES — a scanner that cries wolf on working plugins can't drive adoption. Fixed four classes (all verified against the real plugin that exposed them): 1. Multi-line QUOTED description (`description:\n "…"`, trailofbits/react-pdf) was mislabeled "no description / can't trigger". readField now gathers an indented quoted scalar, not just block scalars. 2. Relative hook path (`./hooks/x.sh`, ananddtyagi/cc-marketplace) was reported MISSING because existsSync was cwd-relative. resolveScript now resolves a relative hook against the PLUGIN ROOT. 3. Existence-guarded hook (`[ ! -f x ] || x`, gmickel/flow-next) — an optional run-if-present — was flagged MISSING. Now treated as a conditional one-liner. 4. A project/runtime path in a script (`$CLAUDE_PROJECT_DIR/.claude/hooks/…`, flow-next) matched mid-path as a plugin ref. danglingRefs/isPluginRooted now only flags PLUGIN-rooted paths (bare or ${PLUGIN_ROOT}/…), skipping literal nested dirs and project/home vars; a genuine plugin-rooted miss still flags. After hardening, the sweep's honest tally over 444 unique plugins: ONE real bug (ananddtyagi/cc-marketplace's claude-dev-infrastructure ships crisis-debugging- advisor + meta-skill-router with NO frontmatter → can't be model-triggered), and the popular plugins are otherwise structurally clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record the structural sweep + public-disclosures / FPs-fixed list New research/plugin-structural-findings.md (sibling of plugin-behavioral- findings.md) logs the deterministic scan sweep of ~13 marketplaces / 444 unique plugins: - Public disclosures: the one genuine external-plugin bug found — ananddtyagi/cc-marketplace's claude-dev-infrastructure ships two skills with no YAML frontmatter (can't be model-triggered). Status: found, not yet reported upstream + disclosure etiquette note. - Scanner false positives found by the sweep and fixed (the dogfood value): command-only 0/F, curated-marketplace "empty", han dedup, multi-line quoted description, relative ./hooks resolution, existence-guarded hook, project- rooted dangling ref — each with the plugin that exposed it + the fix commit. Linked from plugin-behavioral-findings.md and distribution-strategy.md (E1) so it's discoverable and not an orphan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: tool-contract validation — the cross-referencing moat in scan + lint The deterministic scanner described tool contracts but never VERIFIED them. Add the "valid is not true" moat to a subagent's `tools:` rail, as ONE shared detector (src/core/tool-contract.ts) reused by three callers (one-detector-no- drift): compileAgent (strict), scan (read-only), and a new agent-tool-contract lint rule (severity-gated). Calibrated HIGH-PRECISION after sweeping real plugins (the cry-wolf lesson): - flags only never-available tools (curated denylist) + close typos (edit distance ≤2, with did-you-mean) — the two confident cases. - a bare unrecognized tool is NOT flagged: a 280★ plugin uses TaskCreate/TaskGet/… consistently (plugin/MCP-provided), and our catalog can't know every tool — flagging those would be a false-positive flood. Two real bugs this surfaced and fixed along the way: - parseAgentTools didn't handle the inline ARRAY form (`tools: [Read, "Bash"]`) → it returned `[Read` / `Bash]`, which BOTH broke the PreToolUse rail (it would block Read) and tripped the check. Fixed at the source. - builtinAgentTools was incomplete (missing LS/MultiEdit/BashOutput/KillBash) — a latent false "unknown tool" error in compileAgent itself. Catalog fixed. Sweep result after calibration: 5 real never-available dead-contract entries across 444 plugins (wshobson team-lead `Agent`, developer-kit/ananddtyagi `AskUserQuestion`/`ExitPlanMode`), no flood. Leaderboard penalizes a dead tool like a broken ref. New rule defaults to warn (no-op on repos without agents/); doc + README + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: hook-event validation (#2) — typo'd hook events in scan + lint Cross-reference each hook's event name against the harness event catalog: a hook registered under a name the platform doesn't define silently never fires. Shared detector (src/core/hook-events.ts) → scan + the hook-events lint rule. Caught/fixed two things before it could ship: - dialect.hookEvents was WRONG and unused (PreSession/PostSession don't exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact were missing). Corrected against the events real plugins register. - Validating ALL unknown events flooded FPs: TheBushidoCollective/han ships a custom hook runtime (TeammateIdle, WorktreeRemove, …) — the event set is NOT closed. And ananddtyagi/sugar ships hooks as an ARRAY of {event:"tool-use"} objects (a non-CC format → Object.keys gave indices). So: high-precision (close typos only, via confidentHookEventIssues) + object-keyed shape only. After calibration: 0 on 444 swept plugins (FP-safe), catches PreToolUSe→ PreToolUse at authoring. Default warn (no-op on this repo); doc + README + spec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: frontmatter-schema validation (#3) — missing name/description in scan + lint A SKILL.md needs a `name` to load (CC keys a skill on its frontmatter name); a subagent needs `name` + `description` to register. A missing field is a structurally broken surface that silently never works. Computed in scanPlugin (frontmatterIssuesFor) and consumed by scan + the frontmatter-schema lint rule. Verified real on the sweep: 12 issues across 444 plugins, mostly ananddtyagi/ cc-marketplace — the headline bug (crisis-debugging-advisor / meta-skill-router ship with no frontmatter) is now caught precisely as "missing name", plus subagents (changelog-generator, …) that are pure prose with no frontmatter → never register. Confirmed by hand (changelog-generator.md opens with prose, no --- block). FP-safe (name is required; skill description stays on the skill line to avoid double-counting). Default warn; doc + README + spec + tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: MCP-config validation (#4) — unstartable MCP servers in scan + lint A declared MCP server must say how to reach it: a `command` (stdio) or a `url` (http/sse). An entry with neither can't start, and its tools silently go missing. Shared detector (src/core/mcp-config.ts) → scan + the mcp-config lint rule. FP-safe by construction: command-or-url is an unambiguous requirement, not a catalog that could be incomplete. Reads .mcp.json + manifest mcpServers (JSON CC shape); Codex TOML [mcp_servers] is a documented gap. Found 0 on the 444 swept plugins (they all configure command/url correctly) — survivor bias, as expected; the value is catching a malformed server at authoring time. Default warn; doc + README + spec + detector/scan tests. This completes the deterministic cross-referencing moat across the four agent surfaces: tool contracts (#1), hook events (#2), frontmatter schema (#3), and MCP config (#4) — each a shared detector reused by scan + lint (+ compileAgent for tools), all calibrated high-precision against the real-plugin sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: tool-contract treats `tools: *` as a wildcard (inherits all), not unknown A subagent declaring `tools: "*"` (all tools) was flagged `unknown "*" — did you mean "LS"?` (editDistance("*","LS")=2). A wildcard isn't a tool to validate. Skip "*" like the empty contract. Found on ananddtyagi/cc-marketplace's problem-solver-specialist. One more cry-wolf case closed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: frontmatter-schema is subagents-only — skills need no frontmatter (correction) Verified against the Claude Code docs (prompted by skepticism that "those skills never fire"): a SKILL.md requires NO frontmatter — `name` falls back to the directory name, `description` to the first body paragraph — so a frontmatter-less skill STILL loads and can fire. My earlier claim that ananddtyagi's crisis-debugging-advisor / meta-skill-router "never trigger" was WRONG. A subagent, by contrast, REQUIRES `name` + `description` with no fallback, or it won't register — confirmed by docs/en/sub-agents. So: - frontmatterIssuesFor now checks AGENTS ONLY (skill checks were false positives). - Corrected the disclosures doc: the real ananddtyagi bug is ~9 prose-only SUBAGENTS that never register (changelog-generator, content-creator, …), NOT the skills. Added a loud correction note. - Updated the rule doc (skills explicitly not checked + why), cli.md row, spec. Re-tally after correction: 9 agent-frontmatter bugs in ananddtyagi (real, docs-confirmed), the 2 skills + han's name-less skill correctly drop off. The "verify before you claim" discipline catching my own overclaim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empirically confirm the ananddtyagi agent finding (real claude CLI) Cloned ananddtyagi/cc-marketplace and drove the real claude CLI to verify the prose-only subagents really are inert (not just doc-inferred): listing subagents omits the prose-only one, dispatching it returns NO_SUCH_AGENT, while a control agent with frontmatter runs. Mirror test confirms a no-frontmatter SKILL still loads (dir-name fallback). Both the agent bug and the earlier skill correction now have ground-truth backing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: save superpowers trigger-probe fixture + session state before probe Persists (compaction-safe): the recall-probe prompt set (research/fixtures/superpowers-trigger-probe.json, 3 skills x 10 prompts) and a durable session-state note in plugin-behavioral-findings.md — the 4-rule moat shipped this session + the probe plan (results to be filled in after the run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: superpowers trigger probe results — systematic-debugging fires 10% Ran the recall probe on the SHIPPED scan --trigger path (claude 2.1.183, 10 prompts/skill on the subscription). Control test-driven-development fires 100% (validates the measurement); systematic-debugging fires only 10% and brainstorming 50%. A structurally-green skill silently never triggering on its own use case — the behavioral bug the deterministic column structurally can't see. Reconfirms the method-vs-situation trigger pattern on the shipped path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: skill-frontmatter rule — recommend explicit skill name+description (warn) Skills require NO frontmatter (name←dir, description←first paragraph), so a missing one isn't a load failure — but relying on fallbacks is fragile (the first paragraph is often a heading → a weak, accidental trigger surface). Add a best-practice NUDGE: scanPlugin.skillMetaIssues (a soft ℹ note, never counted as a structural defect or scored) + the `skill-frontmatter` lint rule. This is the correct form of the earlier "two rules" idea, split by severity/ intent: `frontmatter-schema` = subagents (REQUIRED → won't register, correctness) vs `skill-frontmatter` = skills (RECOMMENDED → reliable trigger surface, quality). Default warn (the skill works without it); set "error" to enforce on your own skills. vigiles's own skills pass. Doc + README + spec + tests. Note (follow-up): the pre-existing scan `hasDescription` "can't trigger" line is the same overclaim class — a skill with no frontmatter description still triggers via the first-paragraph fallback. Left as-is for now (pre-existing, load-bearing on the leaderboard); worth softening separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename frontmatter-schema rule → agent-frontmatter (symmetric pair) With two frontmatter rules now, `frontmatter-schema` (agents) vs `skill-frontmatter` (skills) was lopsided — one generic, one specific. Rename to `agent-frontmatter` so the pair is parallel and self-describing: agent-frontmatter (required → won't register) + skill-frontmatter (recommended → trigger surface). Unreleased on this branch, so no back-compat cost. Config key + doc (git mv to docs/rules/agent-frontmatter.md) + cli.md table + spec + research refs all updated; internal symbol names left as-is (not user-facing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: soften skill 'can't trigger' overclaim — honor the first-paragraph fallback The trigger-surface check flagged any skill without a FRONTMATTER description as 'can't trigger' — but Claude Code falls back to the first body paragraph, so a skill with prose still triggers. Now hasDescription = frontmatter description OR first body paragraph (firstBodyParagraph); only a skill with NEITHER is flagged. The last instance of the frontmatter overclaim class. The explicit-metadata best-practice stays the separate skill-frontmatter rule. (nodesc — name only, body just a heading — is still correctly flagged.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: ranked next-deterministic-rule ideas, grounded in the OSS sweep New research/deterministic-rule-ideas.md — the curated backlog of next moat surfaces, each the same 'valid is not true' cross-reference on a new surface, high-precision by design (the session's recurring FP lesson). Tier 1: mcp-tool-resolves (an mcp__server__tool whose server isn't declared), hook-matcher (typo'd tool in a hook matcher), frontmatter-valid (malformed YAML). Tier 2: duplicate-names, hook-shape, and the showpiece description-overlap — a DETERMINISTIC precision-collision proxy reusing the NCD engine (catches a --trigger-class bug with no model), grounded in haretrail's debrief/lessons/ postmortem overlap. Linked from roadmap Now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-tool-resolves rule (the MCP half of the tool moat) Cross-reference an `mcp__server__tool` in a subagent's contract against the plugin's declared `mcpServers` — a server the plugin doesn't declare can't resolve. Completes the tool moat: agent-tool-contract verifies built-in tools but passes any mcp__* token unchecked; this verifies the MCP half. One pure detector (src/core/mcp-tool.ts: verifyMcpToolServers) reused by scan + the mcp-tool-resolves lint rule (one-detector-no-drift). Calibrated high-precision via three guards, each grounded in a real plugin from the mid-2026 sweep: - GATE on a declared set — no .mcp.json → reaches global/project servers, flag nothing (ananddtyagi's mcp__ide__* shape). - ALLOWLIST harness built-ins via a new optional dialect.knownMcpServers (Claude Code's `ide`); optional/additive, non-breaking for adapters. - SKIP the plugin-namespaced mcp__plugin_<plugin>_<server>__ form (han's playwright-mcp — ambiguous join, the plugin's own server). Validated FP-free across the whole sweep (0 hits) and fires correctly on a synthetic undeclared-server plugin. Wired into scan (per-agent mcpToolIssues), lint (warn default), and the leaderboard score. Doc + detector tests + scan tests + CLAUDE.md/cli.md + regenerated API reports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: competitor rule matrix + verified OSS bug PR drafts Save the 3-agent competitor landscape sweep (eval tools, plugin/skill validators, MCP/security scanners) as research/competitor-rule-matrix.md: the rule-by-rule matrix vs agnix / claudelint / cclint / Anthropic's own `claude plugin validate`, the positioning finding (structural rules are converging to table stakes; the moat is semantic cross-reference + harness testing), and the ranked poach backlog. Add research/oss-pr-drafts.md with ready-to-file PR/issue text for two hand-verified third-party bugs the sweep found: subagents listing AskUserQuestion/ExitPlanMode (silently dropped — Anthropic #12890/#18721) and ananddtyagi's prose-only/blank-description subagents (won't register). Record the verified non-bugs (test fixtures, codex source tree) so they aren't re-chased. Cross-linked from plugin-structural-findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add hook-script-exists lint rule Promote scan's missing-hook detection to a lint rule: a hook command referencing a script file that doesn't exist on disk (with ${CLAUDE_PLUGIN_ROOT} resolved) silently never runs. Reuses scanPlugin's hooks status (one-detector-no-drift) — FP-safe by construction (skips unresolved $VARs, existence-guarded one-liners, inline commands). Matches Anthropic's own `claude plugin validate`, making vigiles a superset on the hook surface (poach #1 from the competitor matrix). Wired into lint (warn default) + exit code; doc + types/validate + cli.md table + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add disallowed-tools-contract lint rule (deny-side tool moat) Cross-reference a subagent's `disallowedTools:` block-list against the catalog — the mirror of agent-tool-contract. A close typo there blocks NOTHING (you meant to deny `Bash`, wrote `Bsh`), so the tool stays available silently — worse than no deny-list. New detector disallowedToolIssues in tool-contract.ts, calibrated high-precision: only a close typo of a real built-in flags; a real tool is legitimately blocked, a never-available tool is harmless to list, a bare unknown is likely a plugin tool (no cry-wolf). A check no other validator does (poach from SkillCheck). Generalizes parseAgentTools → parseAgentToolList(md, key) for the disallowedTools parse. Wired into scan (per-agent disallowedToolIssues) + lint (warn) + exit code + leaderboard; doc + detector/scan tests + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: agent-frontmatter also flags invalid model/color values Extend the agent-frontmatter rule from "missing required field" to also catch an invalid VALUE: a model: that's a close typo of an alias (inherit/sonnet/opus/ haiku) silently falls back; a color: typo is ignored. High-precision — close-typo only (≤2 edits), a full/dated model id (claude-sonnet-4-5) is an explicit form left alone, a far-off unknown is suppressed (no cry-wolf). Matches Anthropic's own `claude plugin validate` + cclint (poach #3). New frontmatterValueIssuesFor detector + frontmatterValueIssues on ScanReport, folded into the same agent-frontmatter lint rule + scan Frontmatter section + leaderboard; scan test; agent-frontmatter doc + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add description-overlap lint rule (deterministic precision proxy) The showpiece: two model-invocable skills with near-identical descriptions can't be told apart by the selector, so the wrong one fires (a precision collision). findDescriptionOverlaps (core/description-overlap.ts) reuses proofs.ts ncd to catch a --trigger-class behavioral bug with NO model — bridging the deterministic and behavioral columns, a check no other plugin linter has. Calibrated against the sweep: across 4678 within-plugin description pairs the most-similar legitimately-distinct pair (create-issue/create-pr) sits at NCD 0.25, nothing below it — so the cutoff 0.2 fires only on basically-identical text (FP-free on the whole sweep). User-invoked skills excluded (picked by command, can't collide). Wired into scan (descriptionOverlaps, reported as a precision risk) + lint (warn) + exit code; detector + scan tests; doc + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: mark second-wave rules shipped in the backlog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scan): section headers count entities, not output lines The "Agents (N)" / "Hooks (N)" headers used the rendered line count, so an agent's indented issue lines (and the inline-hooks summary line) inflated the number — "Agents (4)" for one agent with three findings. section() now takes an explicit count; Agents passes agents.length, Hooks passes hooks + inline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: one lenient frontmatter reader (real YAML + regex salvage) Replace the two divergent hand-parsers (scan's readField, agent-runtime's regex) with a single shared reader, core/frontmatter-read.ts: try js-yaml (block scalars, quoted multi-line, flow arrays parse correctly for free), fall back to a regex salvage on malformed input and record malformed:true. readFrontmatter never throws — it audits arbitrary third-party files — and frontmatterScalar / frontmatterList preserve the exact semantics the PreToolUse rail depends on (absent → null = inherits all, present-empty → [] = no tools). Anchored at file start (+ optional BOM + optional leading vigiles integrity comment) so a body --- hr is never read as frontmatter. Behavior-preserving (agent-runtime/scan/vendor/dogfood suites green); js-yaml was already a dependency. core/frontmatter.ts (the Level-1 vigiles: block) is a different concern, untouched. Sets up frontmatter-valid (the malformed flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add frontmatter-valid rule (malformed-YAML frontmatter) The payoff of the lenient-reader refactor: readFrontmatter's `malformed` flag becomes a rule — a skill/agent `---` block that exists but isn't valid YAML, so fields may not parse as intended. Calibrated and shipped CONSERVATIVELY per the verify-before-claiming discipline: a real YAML parser is stricter than some loaders, so a one-line description: with a `: ` colon / <example> block is flagged though it may still load. The sweep showed 7% of blocks flagged, concentrated in 2 repos (ananddtyagi, MadAppGang), 0 in the other 11 — a real authoring smell, but NOT empirically confirmed to fail loading. So: default WARN (not error), scan surfaces it as an informational ℹ note (not a structural defect, no leaderboard penalty), doc states the caveat and says verify before setting error. The unambiguous cases (unclosed bracket, tab, duplicate key) are real regardless. Same detector as scan (malformedFrontmatter); detector reuses the shared reader. Wired into lint + scan note; doc + scan test + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: dogfood the scan rules against real vendored plugins (FP-guard + bug fixture) Lock the deterministic rules against reality, offline + model-free, alongside the existing loader dogfood. src/scan-vendor.test.ts asserts golden scan verdicts: - FP-GUARD: oh-my-claudecode / wshobson-accessibility / superpowers must stay CLEAN — every high-precision rule field zero (the regression that catches a rule going noisy; the don't-cry-wolf bet). - TRUE-POSITIVE: a minimal MIT slice vendored BECAUSE it reproduces a real bug — madappgang-frontend@6097ad4 tester.md (a single agent file) — must keep firing agent-tool-contract (AskUserQuestion never-available) AND frontmatter-valid (the one-line description: is invalid YAML). Only the structural file is vendored (not the repo), only from a permissive (MIT) upstream — ananddtyagi has no LICENSE, so its files are NOT committed (its bug stays in research/oss-pr-drafts.md). Provenance + licensing in examples/harness/vendor/SOURCES.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: make the OSS bug drafts filing-ready (recipe + worked diffs) Add a per-repo filing recipe (branch / mechanical edit / conventional commit / gh pr create) plus worked before→after diffs for both verified bugs — the comma-form and array-form AskUserQuestion removal, and the missing-frontmatter prepend. States plainly these can't be filed from a vigiles-scoped session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add PR #40 handoff (resume-here state + next-session prompt) Single consolidated handoff for the branch: what shipped (the deterministic rule suite, scanner hardening, the lenient-reader refactor, the vendor dogfood, the research + OSS PR drafts), the disciplines that must carry forward (verify-before-claim, high-precision, calibrate-against-the-sweep, one-detector-no-drift, check-license-before-vendoring), the ranked open follow-ups (file the OSS PRs, README positioning, the remaining backlog rules), and a paste-ready next-session prompt. Linked from deterministic-rule-ideas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-hook-target-resolves rule (the moat on the hook surface) Cross-reference a `type: "mcp_tool"` HOOK action against the plugin's declared MCP servers (verified real: CC hooks support command/http/mcp_tool/prompt/agent, and mcp_tool requires server+tool). Flags two silent-no-op cases: an incomplete action missing server/tool (unambiguous, always — like mcp-config), and a server the plugin doesn't declare (gated on a declared mcpServers set + built-ins like `ide` allowlisted, mirroring mcp-tool-resolves). New detector core/mcp-hook.ts (verifyMcpHookTargets), wired into scan (mcpHookIssues) + lint (warn) + exit code + leaderboard; detector/scan tests; doc + types/validate + cli.md + CLAUDE.md. FP-free across the sweep (0 mcp_tool hooks in the wild — advanced feature; value is authoring-time + moat completion). The regex matcher surface is left to a future hook-matcher rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(readme): lead with the cross-reference moat; surface plugin audit Per research/competitor-rule-matrix.md: the structural lints (hook-events, agent-frontmatter, mcp-config) are now table stakes — Anthropic's own `claude plugin validate` ships them — so don't headline them. Elevate what's actually the moat: cross-referencing a declared reference against LIVE tooling. Extend the Lint section to the plugin surface — `vigiles scan` checks a plugin's contracts (subagent tools, mcp__server__tool, mcp_tool hooks, hook events, script paths) actually exist AND resolve, "a superset of Anthropic's `claude plugin validate`" (valid YAML ≠ a tool that's real). Retarget the old "Audit any plugin" More bullet to the leaderboard + --trigger so it doesn't duplicate. Stays within the readme-brevity budget (150 lines). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-20 01:13:32 +07:00