mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
fix/hook-runtime-unused-err
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
1e798a2a4b |
feat: deterministic scan/lint rule suite for agent plugins (the cross-referencing moat) + scanner hardening (#40)
* fix: anchor scan surface regexes so a '-agents' skill dir isn't a phantom agent Pointing `vigiles scan` at popular OSS plugins (obra/superpowers, wshobson/agents marketplace) surfaced a real misclassification bug: the skill/agent/command path matchers were unanchored, so any directory whose NAME merely ends in the keyword was misread. superpowers ships `skills/dispatching-parallel-agents/SKILL.md`, and the `agents/[^/]+\.md$` matcher hit the `-agents/SKILL.md` substring — reporting a phantom agent named "SKILL" with no tool contract. That both corrupts the scan report and unfairly deducts from the leaderboard score (no-contract penalty). Anchor `skills/`, `agents/`, and `commands/` on a real path boundary (start-of-path or `/`). Add regression coverage for `-agents`/`-commands` skill directories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan traverses marketplaces and counts dangling refs as structural issues Two gaps surfaced while pointing `vigiles scan` at popular OSS plugins: 1. Marketplace traversal. `wshobson/agents` ships 80+ plugins under a single `.claude-plugin/marketplace.json`; scanning the repo root found nothing. `expandMarketplace` resolves each member's relative `source` (skipping external git sources and missing dirs), and `vigiles scan <root>` now expands a marketplace and ranks every member through the leaderboard. 2. Dangling intra-plugin references are now a first-class structural finding, not a buried free-text warning. `ScanReport.danglingRefs` exposes the broken-path / partial-vendor refs (obra/superpowers ships three); the report prints them under a "Broken references" ✗ section, the verdict counts them, and the leaderboard deducts -8 each. Previously superpowers reported "✓ no structural issues found" despite three missing files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: kill scan false positives that running against real plugins exposed Validating the marketplace scan against wshobson/agents (80+ plugins) and obra/superpowers showed the two heuristic findings were almost all FALSE POSITIVES — the exact doc-mention trap the leaderboard avoids. Promoting noisy signals to scored "bugs" would discredit the whole pitch, so tighten both detectors to fire only on real defects: - Dangling refs: scan only EXECUTABLE source files (hook/helper scripts, incl. extensionless ones like superpowers' hooks/session-start), never prose. Every hit across the corpus came from markdown — example task text, a literal "wc -w skills/path/SKILL.md" template, "❌ Bad" samples, React `hooks/*.ts` code comments — none real. The true positive (a script that cats a missing plugin file) is preserved. - Skill descriptions: parse YAML block scalars (`description: >` / `>-`). The regex captured only the `>` indicator, mislabeling four richly- described wshobson skills as "can't trigger". Net: the marketplace now reports 0 broken-ref and 0 missing-description false positives; remaining deductions are real (no tool contract, untested). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: log first behavioral findings — superpowers brainstorming under-triggers Ran the model-gated column (measureTriggerRate, real claude on the subscription) against the full obra/superpowers plugin to find bugs the deterministic scan structurally cannot see. Headline: the `brainstorming` skill — structurally green (good description, no scan finding) — fires on only ~20–30% of its own open-ended use-case prompts (recall 3/10 then 2/10 across two runs; precision a perfect 100% on mechanical edits). A diagnostic showed it isn't losing to a competitor: on 8/10 prompts no skill fired at all. The "You MUST use this before any creative work" imperative reads strong but fires weak — exactly the class of bug only a real model reveals. Records method, the over-trigger hypothesis the data refuted, an actionable fix for the author, the egress column we couldn't run here (no bwrap/slirp4netns), and honest n=10/1-trial caveats. Linked from distribution-strategy (E1) and divergent-bets (#9) so it isn't an orphan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empty-context validity + competitor landscape + auto-context idea Capture the methodology analysis from probing brainstorming: - Is empty context valid? Confirmed each trigger run is a cold single prompt (no fixture/history; TriggerRateSpec has no such field). It's the field STANDARD (AWS sample-agent-skill-eval does single relevant/irrelevant queries), faithful for opening-move skills (brainstorming qualifies — finding stands), biased-low for state-dependent ones (a shared blind spot, not vigiles-specific). - Landscape table: AWS (query, same), promptfoo (user-injected vars/hooks), DeepEval (ConversationalTestCase), Inspect/SWE-bench (seeded sandbox for stateful tasks). Nobody auto-reconstructs a skill's trigger state. - New idea — auto-apply known context: (1) add fixture/prior-turn to TriggerRateSpec (parity with measure()/runEval), (2) auto-select a curated preset context from the skill's declared trigger phrases so it's measured in the state it claims to fire on. Preset selection on explicit cues, not prose synthesis. Logged in roadmap.md (Next, P3-MED). The empty-vs-seeded sensitivity probe is noted as deferred (40 serial runs overran the budget here; re-run with concurrency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: roadmap — per-model trigger-rate + context-rot curve (MED) (a) Report trigger-rate per model — already a capability (model-as-arm), make it a standard column. (b) Study how recall rots as the skill roster grows and whether a stronger model rots slower — the concrete form of divergent-bets #11 (measure model × harness). Opt-in matrix, not a default gate; cheap first probe is brainstorming recall at 2 roster sizes × 2 models. Measure, don't claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: trigger-rate gets fixture (context) + concurrency for plugin testing at scale Two enablers for behavioral testing of OSS plugins, both surfaced by this session's probing limits: - `TriggerRateSpec.fixture` — seed files into each run's cwd so a skill is measured in the repo STATE it claims to trigger on. The default empty cwd is faithful for opening-move skills but under-measures state-dependent ones; this closes that gap (parity with measure()/runEval, which already had fixture). - `TriggerRateSpec.concurrency` — run the prompts × trials grid through the existing runPool instead of fully serial. Default 1 (rate-limit polite); raise it to cut wall-clock on a large prompt set or roster sweep. The serial runner is what made the earlier sensitivity probe overrun. Refactors runTriggerSet to flatten the grid into jobs + aggregate per-prompt (order preserved); extracts runTriggerTrial (also drops the function under the complexity ceiling). Unit tests with a fake runner prove fixture lands in the cwd before the agent runs, and that concurrency>1 actually parallelizes while aggregation stays correct. Docs + roadmap updated (step 1 of auto-context: done). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan --trigger behavioral column + record empty-vs-seeded sensitivity (a) scan --trigger — the model-gated behavioral column that stacks on the structural scan (not a separate command). For each model-invocable skill in a plugin it measures recall (+ precision) via measureTriggerRate and prints it as a "Behavioral (trigger-rate)" section. Author-supplied prompts (a JSON map of skill → { prompts, irrelevant }), never model-generated. Degrades honestly to "unavailable" with no claude/auth; a thin prompt set is surfaced per-skill, not a crash. Flags: --prompts= --concurrency= --model= --min-prompts=. New src/scan-behavioral.ts (probePluginTriggers[With] + formatBehavioralReport), keeps scanPlugin pure/sync; spawnAgent exported from eval.ts so the real runner is shared. Unit tests with a fake runner; verified live against superpowers (brainstorming recall 20%, rest unmeasured). (b) Sensitivity recorded — empty 0.00 vs seeded 0.20 (10 runs each, via the new fixture + concurrency). Inconclusive at n=10: empty itself spans 0.0/0.2/0.3 across three samples, so the +20pt delta is within the noise floor. The real lesson — n=10 can't resolve a 20pt effect — is documented. Docs: scan --trigger + marketplace + the -8 dangling-ref weight in cli.md; cross-link from harness-testing.md; sensitivity result in plugin-behavioral-findings.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: behavioral findings 2 & 3 — superpowers recall sweep + haretrail scan Finding 2 — superpowers recall sweep (vigiles scan --trigger across 6 opening-move skills, 10 prompts each): concrete imperative-task skills fire reliably (test-driven-development 100%, writing-skills 100%) while exploratory / orchestration skills barely fire (writing-plans 40%, brainstorming 10%, systematic-debugging 10%, dispatching-parallel-agents 0%). State-dependent skills left unmeasured on purpose (cold prompt = artifact). The 0-10% vs 100% gap on structurally-green skills is the finding. Finding 3 — fleytman/haretrail (Codex/AGENTS.md, 8 skills): structurally clean (no agents/hooks/broken refs). Description-level risks worth a trigger probe: Russian descriptions (cross-language trigger risk on English prompts), a literal {data-repo} placeholder, and overlapping domains (debrief/lessons/postmortem, task/research) = precision risk; good practice in the explicit trigger-phrase lists + postmortem's negative cue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan flags non-Latin skill descriptions as a cross-language trigger risk Operationalizes the haretrail catch as a deterministic, no-model check: each skill description's dominant script is detected via Unicode \p{Script=…} (no dependency), and a description ≥20% non-Latin (Cyrillic/CJK/Arabic/…) is flagged "cross-language trigger risk" — the selector is English-centric, so it may under-fire on English prompts. A RISK flag, not a structural defect (a language-matched audience is fine), reported separately from the verdict and pointing at the model-gated `scan --trigger` to measure the real gap. ScanSkill.descriptionScript carries the detected script; formatScanReport notes it per-skill + a summary line. Latin/English descriptions (and mostly-English with a stray accented word like "résumé") are not flagged. Verified on the real fleytman/haretrail (all 8 Cyrillic skills tripped) and superpowers (0 flagged). Also extracts skillLine() (clears a pre-existing nested-ternary warning). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — one shared detector for lint+scan; behavioral column is the exception Records the convention this session arrived at: a deterministic structural check has ONE home (a single pure detector reused by both `lint` and `scan`, never reimplemented in each, so they can't drift) — untested-surface, dangling-ref, and non-Latin-description detection already work this way. The model-gated behavioral column (scan --trigger / trigger-rate, observed egress) is the deliberate exception: it's LLM-based (auth, tokens, non-deterministic), so it lives only in scan/eval and is NEVER a lint rule. Cheap deterministic detection can be a lint rule; expensive model-gated confirmation cannot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: description-script check is expected-relative (default Latin) + Script union Per review: Latin isn't special, it's a configurable DEFAULT. unexpectedScript(text, expected="Latin") now flags the dominant script only when it DIFFERS from the expected one — so a Cyrillic-targeted pack sets expected="Cyrillic", its Russian descriptions pass, and an English one is flagged instead. Adds a `Script` string union (ts-essentials: literal union over bare string) used by the detector, the SCRIPTS table, and ScanSkill.descriptionScript. unexpectedScript is exported as the single shared detector the future lint rule will reuse (one-detector-no-drift). Reworded scan output + the CLAUDE.md rule from "non-Latin" to "unexpected script". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: eval-tier trace-parser seam (increment 1 of native Codex eval) + docs Native Codex eval (measureTriggerRate/runEval for Codex) needs three things; this lands the env-independent foundation and documents the rest. - Increment 1 (shipped): makeContext's Claude stream-json parsing is extracted as parseClaudeRun and injected as a ModelOutputParser (default = Claude, zero behavior change). measureTriggerRateWith takes one as a 3rd arg, threaded via TriggerRunConfig.parse. A unit test drives a non-Claude line protocol + a custom parser end-to-end and detects firing — the exact slot a Codex `codex exec --json` parser plugs into. - Spike recorded (no live binary here): the Codex driver's parseCodexRun returns toolCalls:[] by design, so the eval tier can't yet detect a Codex tool/skill call; and whether a Codex skill activation is even a discrete `codex exec --json` event is the open question only the binary can settle. Docs (internal + public): research/codex-prototype-findings.md gets the spike + increment plan + env-validation checklist; CLAUDE.md (eval.ts entry) notes the parser seam; docs/harness-testing-codex.md and docs/harnesses.md mark Codex runEval/trigger-rate as in-progress (not over-promised) with the three remaining steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scaffold Codex eval runner + tolerant trace parser (increment 2) src/adapters/codex/eval.ts (exported on vigiles/codex): - parseCodexEvalRun(out): ParsedModelRun — a TOLERANT, multi-shape parser of `codex exec --json` JSONL. No codex binary in the build env, so the line schema is unverified: it probes BOTH plausible Codex event models (older {msg:{type,…}} stream, newer {type:"item.*", item:{…}} thread/item stream) and degrades to empty / output=trimmed-stdout on anything unrecognised. Each extractor (assistantText / toolCall / usage) is isolated so finishing against captured JSONL is a field-name edit. Returns the Claude-shaped ParsedModelRun so it slots straight into the increment-1 ModelOutputParser seam. - codexEvalRunner — spawns real `codex exec --json` (v8-ignored; flags + skill-install wiring best-guess pending the binary). Unit-tested over synthetic fixtures for both shapes + junk tolerance + the output contract (src/adapters/codex/eval.test.ts). Deliberately NOT wired into the public measureTriggerRate({adapter}) dispatch (increment 3) — gated until the schema and the skill-selection question are confirmed against the live binary, so an unvalidated parser can't silently report recall 0. Findings doc + checklist updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: how to authenticate Codex for the eval tier (device-auth for headless) The deterministic tier is keyless; only the eval tier (real model) needs Codex auth. Documents the three flows in order of fit: `codex login --device-auth` (URL + one-time code, the only one that works in a remote/sandbox box — no localhost callback to forward), plain `codex login` (browser, local dev only), and `printenv OPENAI_API_KEY | codex login --with-api-key`. Notes the network- egress requirement, the `npm i -g @openai/codex` install (validated 0.141.0), and the version-sensitivity of the `codex exec --json` schema. Ticks the install/auth box on the env-validation checklist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: confirm Codex eval schema against the live binary + the skill-firing finding Authed real codex (ChatGPT) and captured real `codex exec --json` turns, turning increment 2 from a tolerant guess into a confirmed, live-validated parser. Findings (now documented internal + public): - Schema CONFIRMED (codex-cli 0.139.0): a thread/item JSONL stream — assistant = item.completed/agent_message→text; tool = command_execution→command (+ aggregated_output/exit_code); usage on turn.completed (incl. cached_input_tokens → cacheReadTokens). Count item.completed ONLY (item.started repeats the id). - THE skill question ANSWERED: Codex has NO discrete skill-selection event (no Skill tool). A trigger surfaces as the model READING skills/<name>/SKILL.md via a command_execution — so codexSkillFired() detects that read (best-effort; pair with a judged check). Validated live: a marker skill fired and was detected end-to-end through codexEvalRunner → parseCodexEvalRun → codexSkillFired. - Version regression: pin codex 0.139.0 (0.141.0 broke the keyless mock recipe). - Spawn gotcha: stdin must be /dev/null or codex blocks on stdin. parseCodexEvalRun rewritten to the confirmed schema; fixtures replaced with REAL captures; codexSkillFired added. Docs updated: harness-testing-codex.md (schema + skill-firing sections), harnesses.md footnote, CLAUDE.md (eval.ts clause + new Key Files entries), codex-prototype-findings.md (2026-06-19 live-capture update + checklist ticked). Remaining: increment 3 — {adapter} dispatch into measureTriggerRate/runEval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim public Codex eval docs to high-level (full findings stay internal) The previous commit over-shared internal detail into the public docs (the full codex exec --json JSONL schema, item.started/completed dedup, the version regression specifics, the stdin gotcha). Those belong in research/codex-prototype-findings.md (where they remain). Public docs (harness-testing-codex.md, harnesses.md) now keep only the user-relevant level: runEval-for-Codex is in progress, Codex has no skill-selected event (firing detected via the SKILL.md read), pin 0.139.0 — and link out for the schema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — match doc detail to audience (internal full record vs public link-out) Adds the public-vs-internal-docs guidance rule: research/*.md + CLAUDE.md hold the full record (schemas, parse mechanics, version regressions, spike notes, checklists); public docs (docs/, README) state only what a user needs to act + link out for the rest. Don't paste schemas/internal mechanics/version quirks into a user-facing guide unless a user needs them to use the feature. Codifies the trim just made to the Codex eval docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: haretrail native-Codex behavioral probe — cross-language risk refuted First native-Codex behavioral eval (codexEvalRunner → parseCodexEvalRun → codexSkillFired) on the real fleytman/haretrail skills, RU vs EN prompts: research 3/3 RU & 3/3 EN, summary 3/3 & 3/3, debrief 3/3 & 2/3. The Russian descriptions fire reliably on English prompts too — the deterministic "non-Latin description → cross-language risk" flag is a correct RISK flag that the model-gated measurement CLEARS here, so no "switch to English" fix is warranted. The single debrief/en miss confirms codexSkillFired discriminates. Logged in research (internal) per the audience-detail rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: codexRunError — an errored/rate-limited Codex turn isn't a "skill didn't fire" Dogfooding the haretrail precision probe hit a Codex usage limit, and every run came back all-miss: parseCodexEvalRun left the errored turn an empty trace, so codexSkillFired read it as a clean recall-0 miss — a rate-limit silently corrupting trigger-rate with false negatives. codexRunError(out) now detects the `error` / `turn.failed` event (incl. usage limits) so a caller skips/retries instead of scoring it (the Codex analog of the Claude path's isRateLimited backoff). Increment 3's dispatch must consult it. Test uses the real usage-limit capture. Precision/overlap question remains unmeasured (retry after quota resets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: native-Codex trigger-rate dispatch via { evalDriver } (increment 3) Wire the eval-tier harness dispatch so measureTriggerRate (and the scan --trigger behavioral column) can route through a non-Claude harness, with Codex as the first second-harness driver. - eval.ts: EvalDriver { runner, parse, runError? } bundles a harness's AgentRunner + ModelOutputParser + an optional errored-run detector; claudeEvalDriver (default) + measureTriggerRate({ evalDriver }) select one. A trial whose runError fires is excluded from the denominator (an errored/rate-limited turn isn't a clean miss) and counted as report.errored. - adapters/codex/eval.ts: codexEvalDriver (codexEvalAgentRunner + parseCodexEvalRun + codexRunError) + installCodexSkills (the Codex analog of --plugin-dir: copies skills into <cwd>/.codex/skills). - scan-behavioral.ts: a HarnessProbe (buildProbe(dir, harness)) selects the driver + per-harness "fired" predicate (skillResolved/stubbed for Claude, codexSkillFired for Codex), so scan --trigger --harness=codex routes through the Codex driver. - cli.ts: scan --trigger passes the detected harness through. Built + fake-tested (injected driver, no binary); a live native eval run remains, gated on Codex quota. Docs + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: live-validate the native-Codex trigger dispatch (numbers still quota-blocked) Re-ran the haretrail debrief/lessons/postmortem precision probe through the shipped public path (`vigiles scan --trigger --harness=codex`) now that the `{ evalDriver }` dispatch is wired. The Codex usage limit is still capped, so the recall/precision NUMBERS remain unmeasured — but the run live-validated increment 3's plumbing against the real binary: - the CLI dispatched through the Codex EvalDriver and spawned real `codex exec --json`; - `codexRunError` correctly EXCLUDED every errored trial (report showed `recall 0% (0 runs)` = n=0, not scored as misses) — the errored-run exclusion proven against reality, not just fakes; - the diversity gate fired live (rejected a too-similar irrelevant pair). So the only remaining gap is non-errored trials for actual numbers — a quota wall, not a code gap. Saves the eval-ready cluster prompt set at research/fixtures/haretrail-cluster.json for the retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: command-only / MCP-only plugins are a real surface, not score 0 scoreReport treated any plugin with no skills/agents/hooks as "no loadable plugin surface" (score 0/F) — but a command-only plugin (commands/*.md with a valid plugin.json) or an MCP-only plugin (.mcp.json) is legitimate. Anthropic itself ships command-only plugins (code-review, commit-commands) in its marketplace, and the leaderboard graded both 0/F. Count commands + MCP toward the loadable-surface check so they score on their real structural health (a clean command-only plugin → 100/A); a truly empty directory still scores 0. Found by scanning anthropics/claude-code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: close the two dogfood untested-surface gaps + fix a migrate-to-spec drift vigiles lint flagged two untested surfaces in our OWN plugin. Closing them also surfaced a real skill bug: - hooks/refs-nudge.sh: the sibling refs-hook.test.ts drives the CLI directly, so the SHELL WRAPPER (its not-a-node-project guard + exit-2-only propagation) was untested. Add a runHook unit test that drives the real script, pointing npx at a local node_modules/.bin/vigiles shim so it resolves offline. - skills/migrate-to-spec: a user-invoked prose skill (model-driven, nothing deterministic to test), but it SHOWS agents an exact `import { … } from "vigiles/spec"` template. The template imported `check` and `every`, which vigiles/spec does NOT export — following the skill produced specs that fail to compile. Fixed the template and added a dogfood test asserting every taught import is a real vigiles/spec export (catches skill ⇄ API drift; names the surface so it counts as covered). `vigiles lint` now reports all 9 surfaces tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface `scan` (audit any plugin) in README + Codex/command-only in cli.md - README: add a high-signal "Audit any plugin" bullet — the zero-setup, no-key scan/leaderboard is the strongest adoption hook and wasn't mentioned at all. Kept to one bullet per readme-brevity (links to the cli.md depth). - cli.md: note that command-only / MCP-only plugins score as real surfaces (not 0), and that `scan --trigger --harness=codex` routes the trigger probe through the native Codex driver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan reports the instruction file on any cc/codex repo + lint-vs-scan docs scan was plugin-oriented — on a plain repo (just a CLAUDE.md/AGENTS.md, no skills/agents/hooks) it printed an empty inventory and didn't even mention the instruction file, so it looked broken on the most common repo shape. - scan: report the detected instruction file (CLAUDE.md/AGENTS.md) with spec-managed vs hand-written, so it's useful on ANY cc/codex repo. This is a read-only informational fact, NOT the require-spec gate (that stays lint's). - docs/cli.md: add a "Lint vs scan — gate vs report" section with the full capability + target matrices (the answer to "do we need both / does it work on normal & Codex repos"). lint = config-driven gate (refs, 7 catalogs, exit codes); scan = zero-config read-only report (any plugin, leaderboard, + the paid --trigger column). They share one detector for the few common structural checks; everything else is intentionally asymmetric. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: e2e scan over the cc/codex/mixed/marketplace repo matrix Drive the real built CLI (node dist/cli.js scan) across every repo shape, in the free unit tier (deterministic, model-free, offline — like vendor.test.ts). Two fixture sources by design: - DOGFOOD pinned OSS plugins (examples/harness/vendor/*) for the Claude Code surface — real shapes (oh-my-claudecode multi-surface + MCP; the wshobson-accessibility inherits-all agent). - ARTIFICIAL tmp fixtures for shapes we have no vendored example of: a Codex repo (AGENTS.md + TOML [mcp_servers] + skill), a mixed CLAUDE.md+AGENTS.md repo (detection-ambiguity warning + --harness override), a CC instruction-only repo (the new instruction-file line), and a marketplace (leaderboard expansion). Covers the CLI scan WIRING the library-level scan.test.ts can't reach: harness auto-detection, the ambiguity warning, --harness=, the leaderboard branch, --json, and the read-only exit-0 contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan honest on curated marketplaces + dedupe name-aliased plugin dirs Two real bugs surfaced by sweeping ~13 popular plugin marketplaces (~650 plugins) + a variant mix of cc/codex/both repos: 1. A CURATED marketplace — every member an external git/url plugin, nothing vendored in-tree (obra/superpowers-marketplace, anthropics/claude-plugins- community with 2200 entries) — scanned as "nothing was loaded / no structural issues found", indistinguishable from a broken empty repo. Now inspectMarketplace classifies members on-disk vs external and scan reports `Marketplace "X": N plugin(s), all external (…not on disk)` honestly. 2. A marketplace that maps many NAMES onto the same dir (TheBushidoCollective/ han: 338 names → 159 dirs) made the leaderboard double-scan and rank the same plugin repeatedly ("(338 scanned)" with duplicate rows). expandMarket- place now dedupes by resolved path → "(159 scanned)", no dup rows. inspectMarketplace is the new source of truth (MarketplaceInfo: name/onDisk/ external/total); expandMarketplace delegates to it. The sweep also confirmed robustness: no crash across all marketplaces, and correct harness detection + instruction-file reporting across claude-only / codex-only / both-files repos (openai/codex, sst/opencode, wshobson, obra, …). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: eliminate scanner false positives found by sweeping real plugins Sweeping ~13 marketplaces + a variant mix of cc/codex/both repos surfaced that the scanner's "bugs" were mostly FALSE POSITIVES — a scanner that cries wolf on working plugins can't drive adoption. Fixed four classes (all verified against the real plugin that exposed them): 1. Multi-line QUOTED description (`description:\n "…"`, trailofbits/react-pdf) was mislabeled "no description / can't trigger". readField now gathers an indented quoted scalar, not just block scalars. 2. Relative hook path (`./hooks/x.sh`, ananddtyagi/cc-marketplace) was reported MISSING because existsSync was cwd-relative. resolveScript now resolves a relative hook against the PLUGIN ROOT. 3. Existence-guarded hook (`[ ! -f x ] || x`, gmickel/flow-next) — an optional run-if-present — was flagged MISSING. Now treated as a conditional one-liner. 4. A project/runtime path in a script (`$CLAUDE_PROJECT_DIR/.claude/hooks/…`, flow-next) matched mid-path as a plugin ref. danglingRefs/isPluginRooted now only flags PLUGIN-rooted paths (bare or ${PLUGIN_ROOT}/…), skipping literal nested dirs and project/home vars; a genuine plugin-rooted miss still flags. After hardening, the sweep's honest tally over 444 unique plugins: ONE real bug (ananddtyagi/cc-marketplace's claude-dev-infrastructure ships crisis-debugging- advisor + meta-skill-router with NO frontmatter → can't be model-triggered), and the popular plugins are otherwise structurally clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record the structural sweep + public-disclosures / FPs-fixed list New research/plugin-structural-findings.md (sibling of plugin-behavioral- findings.md) logs the deterministic scan sweep of ~13 marketplaces / 444 unique plugins: - Public disclosures: the one genuine external-plugin bug found — ananddtyagi/cc-marketplace's claude-dev-infrastructure ships two skills with no YAML frontmatter (can't be model-triggered). Status: found, not yet reported upstream + disclosure etiquette note. - Scanner false positives found by the sweep and fixed (the dogfood value): command-only 0/F, curated-marketplace "empty", han dedup, multi-line quoted description, relative ./hooks resolution, existence-guarded hook, project- rooted dangling ref — each with the plugin that exposed it + the fix commit. Linked from plugin-behavioral-findings.md and distribution-strategy.md (E1) so it's discoverable and not an orphan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: tool-contract validation — the cross-referencing moat in scan + lint The deterministic scanner described tool contracts but never VERIFIED them. Add the "valid is not true" moat to a subagent's `tools:` rail, as ONE shared detector (src/core/tool-contract.ts) reused by three callers (one-detector-no- drift): compileAgent (strict), scan (read-only), and a new agent-tool-contract lint rule (severity-gated). Calibrated HIGH-PRECISION after sweeping real plugins (the cry-wolf lesson): - flags only never-available tools (curated denylist) + close typos (edit distance ≤2, with did-you-mean) — the two confident cases. - a bare unrecognized tool is NOT flagged: a 280★ plugin uses TaskCreate/TaskGet/… consistently (plugin/MCP-provided), and our catalog can't know every tool — flagging those would be a false-positive flood. Two real bugs this surfaced and fixed along the way: - parseAgentTools didn't handle the inline ARRAY form (`tools: [Read, "Bash"]`) → it returned `[Read` / `Bash]`, which BOTH broke the PreToolUse rail (it would block Read) and tripped the check. Fixed at the source. - builtinAgentTools was incomplete (missing LS/MultiEdit/BashOutput/KillBash) — a latent false "unknown tool" error in compileAgent itself. Catalog fixed. Sweep result after calibration: 5 real never-available dead-contract entries across 444 plugins (wshobson team-lead `Agent`, developer-kit/ananddtyagi `AskUserQuestion`/`ExitPlanMode`), no flood. Leaderboard penalizes a dead tool like a broken ref. New rule defaults to warn (no-op on repos without agents/); doc + README + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: hook-event validation (#2) — typo'd hook events in scan + lint Cross-reference each hook's event name against the harness event catalog: a hook registered under a name the platform doesn't define silently never fires. Shared detector (src/core/hook-events.ts) → scan + the hook-events lint rule. Caught/fixed two things before it could ship: - dialect.hookEvents was WRONG and unused (PreSession/PostSession don't exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact were missing). Corrected against the events real plugins register. - Validating ALL unknown events flooded FPs: TheBushidoCollective/han ships a custom hook runtime (TeammateIdle, WorktreeRemove, …) — the event set is NOT closed. And ananddtyagi/sugar ships hooks as an ARRAY of {event:"tool-use"} objects (a non-CC format → Object.keys gave indices). So: high-precision (close typos only, via confidentHookEventIssues) + object-keyed shape only. After calibration: 0 on 444 swept plugins (FP-safe), catches PreToolUSe→ PreToolUse at authoring. Default warn (no-op on this repo); doc + README + spec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: frontmatter-schema validation (#3) — missing name/description in scan + lint A SKILL.md needs a `name` to load (CC keys a skill on its frontmatter name); a subagent needs `name` + `description` to register. A missing field is a structurally broken surface that silently never works. Computed in scanPlugin (frontmatterIssuesFor) and consumed by scan + the frontmatter-schema lint rule. Verified real on the sweep: 12 issues across 444 plugins, mostly ananddtyagi/ cc-marketplace — the headline bug (crisis-debugging-advisor / meta-skill-router ship with no frontmatter) is now caught precisely as "missing name", plus subagents (changelog-generator, …) that are pure prose with no frontmatter → never register. Confirmed by hand (changelog-generator.md opens with prose, no --- block). FP-safe (name is required; skill description stays on the skill line to avoid double-counting). Default warn; doc + README + spec + tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: MCP-config validation (#4) — unstartable MCP servers in scan + lint A declared MCP server must say how to reach it: a `command` (stdio) or a `url` (http/sse). An entry with neither can't start, and its tools silently go missing. Shared detector (src/core/mcp-config.ts) → scan + the mcp-config lint rule. FP-safe by construction: command-or-url is an unambiguous requirement, not a catalog that could be incomplete. Reads .mcp.json + manifest mcpServers (JSON CC shape); Codex TOML [mcp_servers] is a documented gap. Found 0 on the 444 swept plugins (they all configure command/url correctly) — survivor bias, as expected; the value is catching a malformed server at authoring time. Default warn; doc + README + spec + detector/scan tests. This completes the deterministic cross-referencing moat across the four agent surfaces: tool contracts (#1), hook events (#2), frontmatter schema (#3), and MCP config (#4) — each a shared detector reused by scan + lint (+ compileAgent for tools), all calibrated high-precision against the real-plugin sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: tool-contract treats `tools: *` as a wildcard (inherits all), not unknown A subagent declaring `tools: "*"` (all tools) was flagged `unknown "*" — did you mean "LS"?` (editDistance("*","LS")=2). A wildcard isn't a tool to validate. Skip "*" like the empty contract. Found on ananddtyagi/cc-marketplace's problem-solver-specialist. One more cry-wolf case closed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: frontmatter-schema is subagents-only — skills need no frontmatter (correction) Verified against the Claude Code docs (prompted by skepticism that "those skills never fire"): a SKILL.md requires NO frontmatter — `name` falls back to the directory name, `description` to the first body paragraph — so a frontmatter-less skill STILL loads and can fire. My earlier claim that ananddtyagi's crisis-debugging-advisor / meta-skill-router "never trigger" was WRONG. A subagent, by contrast, REQUIRES `name` + `description` with no fallback, or it won't register — confirmed by docs/en/sub-agents. So: - frontmatterIssuesFor now checks AGENTS ONLY (skill checks were false positives). - Corrected the disclosures doc: the real ananddtyagi bug is ~9 prose-only SUBAGENTS that never register (changelog-generator, content-creator, …), NOT the skills. Added a loud correction note. - Updated the rule doc (skills explicitly not checked + why), cli.md row, spec. Re-tally after correction: 9 agent-frontmatter bugs in ananddtyagi (real, docs-confirmed), the 2 skills + han's name-less skill correctly drop off. The "verify before you claim" discipline catching my own overclaim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empirically confirm the ananddtyagi agent finding (real claude CLI) Cloned ananddtyagi/cc-marketplace and drove the real claude CLI to verify the prose-only subagents really are inert (not just doc-inferred): listing subagents omits the prose-only one, dispatching it returns NO_SUCH_AGENT, while a control agent with frontmatter runs. Mirror test confirms a no-frontmatter SKILL still loads (dir-name fallback). Both the agent bug and the earlier skill correction now have ground-truth backing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: save superpowers trigger-probe fixture + session state before probe Persists (compaction-safe): the recall-probe prompt set (research/fixtures/superpowers-trigger-probe.json, 3 skills x 10 prompts) and a durable session-state note in plugin-behavioral-findings.md — the 4-rule moat shipped this session + the probe plan (results to be filled in after the run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: superpowers trigger probe results — systematic-debugging fires 10% Ran the recall probe on the SHIPPED scan --trigger path (claude 2.1.183, 10 prompts/skill on the subscription). Control test-driven-development fires 100% (validates the measurement); systematic-debugging fires only 10% and brainstorming 50%. A structurally-green skill silently never triggering on its own use case — the behavioral bug the deterministic column structurally can't see. Reconfirms the method-vs-situation trigger pattern on the shipped path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: skill-frontmatter rule — recommend explicit skill name+description (warn) Skills require NO frontmatter (name←dir, description←first paragraph), so a missing one isn't a load failure — but relying on fallbacks is fragile (the first paragraph is often a heading → a weak, accidental trigger surface). Add a best-practice NUDGE: scanPlugin.skillMetaIssues (a soft ℹ note, never counted as a structural defect or scored) + the `skill-frontmatter` lint rule. This is the correct form of the earlier "two rules" idea, split by severity/ intent: `frontmatter-schema` = subagents (REQUIRED → won't register, correctness) vs `skill-frontmatter` = skills (RECOMMENDED → reliable trigger surface, quality). Default warn (the skill works without it); set "error" to enforce on your own skills. vigiles's own skills pass. Doc + README + spec + tests. Note (follow-up): the pre-existing scan `hasDescription` "can't trigger" line is the same overclaim class — a skill with no frontmatter description still triggers via the first-paragraph fallback. Left as-is for now (pre-existing, load-bearing on the leaderboard); worth softening separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename frontmatter-schema rule → agent-frontmatter (symmetric pair) With two frontmatter rules now, `frontmatter-schema` (agents) vs `skill-frontmatter` (skills) was lopsided — one generic, one specific. Rename to `agent-frontmatter` so the pair is parallel and self-describing: agent-frontmatter (required → won't register) + skill-frontmatter (recommended → trigger surface). Unreleased on this branch, so no back-compat cost. Config key + doc (git mv to docs/rules/agent-frontmatter.md) + cli.md table + spec + research refs all updated; internal symbol names left as-is (not user-facing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: soften skill 'can't trigger' overclaim — honor the first-paragraph fallback The trigger-surface check flagged any skill without a FRONTMATTER description as 'can't trigger' — but Claude Code falls back to the first body paragraph, so a skill with prose still triggers. Now hasDescription = frontmatter description OR first body paragraph (firstBodyParagraph); only a skill with NEITHER is flagged. The last instance of the frontmatter overclaim class. The explicit-metadata best-practice stays the separate skill-frontmatter rule. (nodesc — name only, body just a heading — is still correctly flagged.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: ranked next-deterministic-rule ideas, grounded in the OSS sweep New research/deterministic-rule-ideas.md — the curated backlog of next moat surfaces, each the same 'valid is not true' cross-reference on a new surface, high-precision by design (the session's recurring FP lesson). Tier 1: mcp-tool-resolves (an mcp__server__tool whose server isn't declared), hook-matcher (typo'd tool in a hook matcher), frontmatter-valid (malformed YAML). Tier 2: duplicate-names, hook-shape, and the showpiece description-overlap — a DETERMINISTIC precision-collision proxy reusing the NCD engine (catches a --trigger-class bug with no model), grounded in haretrail's debrief/lessons/ postmortem overlap. Linked from roadmap Now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-tool-resolves rule (the MCP half of the tool moat) Cross-reference an `mcp__server__tool` in a subagent's contract against the plugin's declared `mcpServers` — a server the plugin doesn't declare can't resolve. Completes the tool moat: agent-tool-contract verifies built-in tools but passes any mcp__* token unchecked; this verifies the MCP half. One pure detector (src/core/mcp-tool.ts: verifyMcpToolServers) reused by scan + the mcp-tool-resolves lint rule (one-detector-no-drift). Calibrated high-precision via three guards, each grounded in a real plugin from the mid-2026 sweep: - GATE on a declared set — no .mcp.json → reaches global/project servers, flag nothing (ananddtyagi's mcp__ide__* shape). - ALLOWLIST harness built-ins via a new optional dialect.knownMcpServers (Claude Code's `ide`); optional/additive, non-breaking for adapters. - SKIP the plugin-namespaced mcp__plugin_<plugin>_<server>__ form (han's playwright-mcp — ambiguous join, the plugin's own server). Validated FP-free across the whole sweep (0 hits) and fires correctly on a synthetic undeclared-server plugin. Wired into scan (per-agent mcpToolIssues), lint (warn default), and the leaderboard score. Doc + detector tests + scan tests + CLAUDE.md/cli.md + regenerated API reports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: competitor rule matrix + verified OSS bug PR drafts Save the 3-agent competitor landscape sweep (eval tools, plugin/skill validators, MCP/security scanners) as research/competitor-rule-matrix.md: the rule-by-rule matrix vs agnix / claudelint / cclint / Anthropic's own `claude plugin validate`, the positioning finding (structural rules are converging to table stakes; the moat is semantic cross-reference + harness testing), and the ranked poach backlog. Add research/oss-pr-drafts.md with ready-to-file PR/issue text for two hand-verified third-party bugs the sweep found: subagents listing AskUserQuestion/ExitPlanMode (silently dropped — Anthropic #12890/#18721) and ananddtyagi's prose-only/blank-description subagents (won't register). Record the verified non-bugs (test fixtures, codex source tree) so they aren't re-chased. Cross-linked from plugin-structural-findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add hook-script-exists lint rule Promote scan's missing-hook detection to a lint rule: a hook command referencing a script file that doesn't exist on disk (with ${CLAUDE_PLUGIN_ROOT} resolved) silently never runs. Reuses scanPlugin's hooks status (one-detector-no-drift) — FP-safe by construction (skips unresolved $VARs, existence-guarded one-liners, inline commands). Matches Anthropic's own `claude plugin validate`, making vigiles a superset on the hook surface (poach #1 from the competitor matrix). Wired into lint (warn default) + exit code; doc + types/validate + cli.md table + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add disallowed-tools-contract lint rule (deny-side tool moat) Cross-reference a subagent's `disallowedTools:` block-list against the catalog — the mirror of agent-tool-contract. A close typo there blocks NOTHING (you meant to deny `Bash`, wrote `Bsh`), so the tool stays available silently — worse than no deny-list. New detector disallowedToolIssues in tool-contract.ts, calibrated high-precision: only a close typo of a real built-in flags; a real tool is legitimately blocked, a never-available tool is harmless to list, a bare unknown is likely a plugin tool (no cry-wolf). A check no other validator does (poach from SkillCheck). Generalizes parseAgentTools → parseAgentToolList(md, key) for the disallowedTools parse. Wired into scan (per-agent disallowedToolIssues) + lint (warn) + exit code + leaderboard; doc + detector/scan tests + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: agent-frontmatter also flags invalid model/color values Extend the agent-frontmatter rule from "missing required field" to also catch an invalid VALUE: a model: that's a close typo of an alias (inherit/sonnet/opus/ haiku) silently falls back; a color: typo is ignored. High-precision — close-typo only (≤2 edits), a full/dated model id (claude-sonnet-4-5) is an explicit form left alone, a far-off unknown is suppressed (no cry-wolf). Matches Anthropic's own `claude plugin validate` + cclint (poach #3). New frontmatterValueIssuesFor detector + frontmatterValueIssues on ScanReport, folded into the same agent-frontmatter lint rule + scan Frontmatter section + leaderboard; scan test; agent-frontmatter doc + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add description-overlap lint rule (deterministic precision proxy) The showpiece: two model-invocable skills with near-identical descriptions can't be told apart by the selector, so the wrong one fires (a precision collision). findDescriptionOverlaps (core/description-overlap.ts) reuses proofs.ts ncd to catch a --trigger-class behavioral bug with NO model — bridging the deterministic and behavioral columns, a check no other plugin linter has. Calibrated against the sweep: across 4678 within-plugin description pairs the most-similar legitimately-distinct pair (create-issue/create-pr) sits at NCD 0.25, nothing below it — so the cutoff 0.2 fires only on basically-identical text (FP-free on the whole sweep). User-invoked skills excluded (picked by command, can't collide). Wired into scan (descriptionOverlaps, reported as a precision risk) + lint (warn) + exit code; detector + scan tests; doc + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: mark second-wave rules shipped in the backlog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scan): section headers count entities, not output lines The "Agents (N)" / "Hooks (N)" headers used the rendered line count, so an agent's indented issue lines (and the inline-hooks summary line) inflated the number — "Agents (4)" for one agent with three findings. section() now takes an explicit count; Agents passes agents.length, Hooks passes hooks + inline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: one lenient frontmatter reader (real YAML + regex salvage) Replace the two divergent hand-parsers (scan's readField, agent-runtime's regex) with a single shared reader, core/frontmatter-read.ts: try js-yaml (block scalars, quoted multi-line, flow arrays parse correctly for free), fall back to a regex salvage on malformed input and record malformed:true. readFrontmatter never throws — it audits arbitrary third-party files — and frontmatterScalar / frontmatterList preserve the exact semantics the PreToolUse rail depends on (absent → null = inherits all, present-empty → [] = no tools). Anchored at file start (+ optional BOM + optional leading vigiles integrity comment) so a body --- hr is never read as frontmatter. Behavior-preserving (agent-runtime/scan/vendor/dogfood suites green); js-yaml was already a dependency. core/frontmatter.ts (the Level-1 vigiles: block) is a different concern, untouched. Sets up frontmatter-valid (the malformed flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add frontmatter-valid rule (malformed-YAML frontmatter) The payoff of the lenient-reader refactor: readFrontmatter's `malformed` flag becomes a rule — a skill/agent `---` block that exists but isn't valid YAML, so fields may not parse as intended. Calibrated and shipped CONSERVATIVELY per the verify-before-claiming discipline: a real YAML parser is stricter than some loaders, so a one-line description: with a `: ` colon / <example> block is flagged though it may still load. The sweep showed 7% of blocks flagged, concentrated in 2 repos (ananddtyagi, MadAppGang), 0 in the other 11 — a real authoring smell, but NOT empirically confirmed to fail loading. So: default WARN (not error), scan surfaces it as an informational ℹ note (not a structural defect, no leaderboard penalty), doc states the caveat and says verify before setting error. The unambiguous cases (unclosed bracket, tab, duplicate key) are real regardless. Same detector as scan (malformedFrontmatter); detector reuses the shared reader. Wired into lint + scan note; doc + scan test + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: dogfood the scan rules against real vendored plugins (FP-guard + bug fixture) Lock the deterministic rules against reality, offline + model-free, alongside the existing loader dogfood. src/scan-vendor.test.ts asserts golden scan verdicts: - FP-GUARD: oh-my-claudecode / wshobson-accessibility / superpowers must stay CLEAN — every high-precision rule field zero (the regression that catches a rule going noisy; the don't-cry-wolf bet). - TRUE-POSITIVE: a minimal MIT slice vendored BECAUSE it reproduces a real bug — madappgang-frontend@6097ad4 tester.md (a single agent file) — must keep firing agent-tool-contract (AskUserQuestion never-available) AND frontmatter-valid (the one-line description: is invalid YAML). Only the structural file is vendored (not the repo), only from a permissive (MIT) upstream — ananddtyagi has no LICENSE, so its files are NOT committed (its bug stays in research/oss-pr-drafts.md). Provenance + licensing in examples/harness/vendor/SOURCES.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: make the OSS bug drafts filing-ready (recipe + worked diffs) Add a per-repo filing recipe (branch / mechanical edit / conventional commit / gh pr create) plus worked before→after diffs for both verified bugs — the comma-form and array-form AskUserQuestion removal, and the missing-frontmatter prepend. States plainly these can't be filed from a vigiles-scoped session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add PR #40 handoff (resume-here state + next-session prompt) Single consolidated handoff for the branch: what shipped (the deterministic rule suite, scanner hardening, the lenient-reader refactor, the vendor dogfood, the research + OSS PR drafts), the disciplines that must carry forward (verify-before-claim, high-precision, calibrate-against-the-sweep, one-detector-no-drift, check-license-before-vendoring), the ranked open follow-ups (file the OSS PRs, README positioning, the remaining backlog rules), and a paste-ready next-session prompt. Linked from deterministic-rule-ideas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-hook-target-resolves rule (the moat on the hook surface) Cross-reference a `type: "mcp_tool"` HOOK action against the plugin's declared MCP servers (verified real: CC hooks support command/http/mcp_tool/prompt/agent, and mcp_tool requires server+tool). Flags two silent-no-op cases: an incomplete action missing server/tool (unambiguous, always — like mcp-config), and a server the plugin doesn't declare (gated on a declared mcpServers set + built-ins like `ide` allowlisted, mirroring mcp-tool-resolves). New detector core/mcp-hook.ts (verifyMcpHookTargets), wired into scan (mcpHookIssues) + lint (warn) + exit code + leaderboard; detector/scan tests; doc + types/validate + cli.md + CLAUDE.md. FP-free across the sweep (0 mcp_tool hooks in the wild — advanced feature; value is authoring-time + moat completion). The regex matcher surface is left to a future hook-matcher rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(readme): lead with the cross-reference moat; surface plugin audit Per research/competitor-rule-matrix.md: the structural lints (hook-events, agent-frontmatter, mcp-config) are now table stakes — Anthropic's own `claude plugin validate` ships them — so don't headline them. Elevate what's actually the moat: cross-referencing a declared reference against LIVE tooling. Extend the Lint section to the plugin surface — `vigiles scan` checks a plugin's contracts (subagent tools, mcp__server__tool, mcp_tool hooks, hook events, script paths) actually exist AND resolve, "a superset of Anthropic's `claude plugin validate`" (valid YAML ≠ a tool that's real). Retarget the old "Audit any plugin" More bullet to the leaderboard + --trigger so it doesn't duplicate. Stays within the readme-brevity budget (150 lines). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
8fb3dbad8d |
feat!: harness adapter architecture — extract Claude Code behind ports, ship Codex (#31)
* Add compose-with-sync-tools rule (Ruler/rulesync/AGENTS.md)
vigiles is the author+verify layer, not the fan-out layer. Codify
composability with the top rule-sync/interop tools rather than
reimplementing native multi-format emission.
* Add sync-tool compatibility research + collision detector
Research the real mid-2026 formats of Ruler, rulesync, and AGENTS.md and
write down the requirements to stay composable. Finding: most are already
met (AGENTS.md is a first-class target; frontmatter mode ignores foreign
keys). The one genuine gap is file-ownership collision — vigiles stamps a
SHA-256 integrity header that Ruler/rulesync silently stale when they
regenerate the same file.
Ship src/compose.ts (detectSyncTools/composeCollisions): a pure filesystem
detector that flags a compile target a detected sync tool also owns and
names the source-slot redirect (Topology A: compile upstream, let the tool
distribute). Mirrors the orphans.ts/test-coverage.ts detector pattern.
* Enforce hexagonal core/adapter boundary + harness docs
Establish the harness-adapter architecture: the reference-verification
domain (verify-core) must never import the Claude Code transport/harness
adapter (cc-harness), so the core stays harness-agnostic for a future
second adapter (Codex likely next).
- eslint-plugin-boundaries classifies modules by path and forbids the
inward edge (boundaries/dependencies, error mode, zero violations today);
eslint-import-resolver-typescript maps NodeNext .js specifiers to .ts so
dependencies classify correctly.
- Dogfood it: enforce("boundaries/dependencies") in the spec, so vigiles
verifies the boundary rule exists and is enabled (5 linter-verified rules).
- research/code-adapter-architecture.md: the two-axis port design, import-
named adapter selection, and the step-by-step recipe to add a harness.
- docs/harnesses.md: user-facing how-to — pick a harness by importing
vigiles/claude-code beside the harness-agnostic vigiles/testing core.
* Reshape src into core / adapters/claude-code / app (hexagonal)
Physically split the tree so a second harness (Codex next) is an additive
src/adapters/<harness>/ rather than a core rewrite:
- src/core/ — harness-agnostic reference-verification domain
- src/adapters/claude-code/ — the Claude Code runtime/transport adapter
- src/ root — application/composition (cli, scan, barrels)
Mechanical move via codemod (81 files, 87 import specifiers rewritten),
plus package.json exports remap, path-based eslint-plugin-boundaries
elements (src/core ⊄ src/adapters/claude-code), and fixture-path /
moved-source-ref fixes across the test suite. Build, full vitest suite
(755 passed), boundary lint, and spec compile all green. Architecture
section of CLAUDE.md updated to describe the layout.
* Extract HarnessDialect port (format-axis decoupling, Stage 1)
The subagent tool catalog (built-in / never-available / MCP shape) that was
hard-coded as literals in compile.ts is now the HarnessDialect interface in
src/core/dialect.ts. claudeCodeDialect is the reference impl + defaultDialect;
compileAgent(spec, { dialect }) takes an injected one, so the core no longer
hard-codes one harness's vocabulary. dialect.test.ts proves a second dialect
swaps the catalog (a tool valid under an alt dialect is flagged under Claude
Code) — the seam a Codex adapter plugs into: define codexDialect, inject it,
no core edit. Re-exported on the vigiles/claude-code surface; ClaudeTool /
HookEvent types documented as the dialect's typed mirror. Full suite green.
* Extract PluginLayout port (layout-axis decoupling, Stage 3)
The plugin/repo layout literals in loadPlugin (.claude-plugin/plugin.json,
.claude/settings.json, hooks/hooks.json, the skills/agents/commands surfaces,
${CLAUDE_PLUGIN_ROOT}, .mcp.json) are now the PluginLayout interface in
src/core/layout.ts, implemented by claudeCodeLayout in the adapter. loadPlugin
reads every path/token/surface from the injected descriptor (default =
claudeCodeLayout), so a Codex adapter reuses the same loader with its own
layout. layout.test.ts loads a Codex-shaped plugin (AGENTS.md, prompts/ surface,
.codex/ settings, ${CODEX_PLUGIN_ROOT}) through the same loadPlugin and proves
the default CC layout sees none of it. 100% coverage gate held.
* Extract HarnessRuntime port (transport-axis decoupling, Stage 2)
The transport literals — spawn "claude", and the ANTHROPIC_BASE_URL /
ANTHROPIC_API_KEY mock env — are now the HarnessRuntime interface in
src/core/runtime.ts, implemented by claudeCodeRuntime in the adapter.
harness-test.ts, eval.ts and the sandbox WRAPPER read the agent binary +
mock env from it (via the pure, unit-tested mockModelEnv helper) instead of
hard-coding them, so a Codex adapter supplies its own runtime (codex binary +
its model env) and the runners drive it the same way. runtime.test.ts proves
an alternate runtime maps the URL onto its own env var. 100% coverage held.
Completes Stages 1-3 of the adapter extraction: HarnessDialect (format),
PluginLayout (layout) and HarnessRuntime (transport) ports — adding Codex is
now 'implement three value objects', no core edit. Research doc + Key Files
updated.
* Fix CLAUDE.md prettier-stability (backtick ANTHROPIC_* and *.md)
Two Key Files descriptions contained bare markdown emphasis chars
(`ANTHROPIC_*`, `*.md`) that prettier escapes, so `npm run fmt` rewrote the
compiled CLAUDE.md after compile and broke its integrity hash (vigiles audit
caught it). Backtick the fragments so compile output is prettier-stable —
fmt:check and the audit hash check both pass. Also fixes a doc code-block
ref (src/compile.ts → src/core/compile.ts) the reshape moved.
* Add harness-landscape research + Codex extraction map
Multi-source mid-2026 research (Codex docs + repo, 14-tool landscape sweep).
Key result: the two seams deferred in code-adapter-architecture.md (HookProtocol,
ModelMock) are now extractable — Codex is the concrete second implementation.
Codex shipped real hooks WITH veto (permissionDecision/exit-2, near-1:1 with
Claude Code) and is mockable via [model_providers] + the OpenAI Responses-API
SSE. The research also surfaces two wrong-abstraction traps the wait avoided:
the layout port needs a settings-format axis (JSON vs TOML), and the runtime
port's point-at-mock is an env var for CC but a config-block for Codex. Plus a
scored landscape table and a next-adapter shortlist gated on mockability
(pillar-2-capable vs pillar-1-only).
* Adapter kit: bundle + registry + conformance + authoring lib (HookProtocol + ModelMock ports)
Make adding a harness 'write one object', backwards compatible:
- Finish the two deferred ports as descriptors: HookProtocol (run-hook block
exit code + deny values) and ModelMock (mock wire format + endpoints),
consumed by decideHook / startMock. 100% coverage gate held.
- HarnessAdapter bundles the five ports + a detect(root) predicate;
claudeCodeAdapter is the reference impl.
- adapter-registry: detectAdapter(root) defaults to Claude Code (backwards
compatible); CLI compile/scan now auto-detect (scan prints the harness).
- adapter-conformance: assertAdapterConformance — the reusable check a new
adapter (incl. third-party) runs.
- vigiles/adapter: the documented authoring kit (ports + bundle + conformance
+ registry); docs/authoring-an-adapter.md + README callout (custom adapters
welcome); CLAUDE.md positioning gains the multi-harness goal.
* Close adapter-readiness gaps #1/#4/#6 (settings-format, detection, conformance)
The concrete, testable, de-risking subset of the gaps adding adapter #2 exposes:
- #1 settings-format axis: PluginLayout.settingsFormat (json|toml); loadPlugin
parses Codex-style config.toml [hooks] via @iarna/toml instead of silently
reading zero hooks. (smol-toml is ESM-only → CJS build needs @iarna/toml.)
- #4 detection: detect(root) now returns a specificity score (manifest 3 >
settings 2 > bare CLAUDE.md 1); detectAdapterResult reports ambiguousWith for
a repo matching several; resolveAdapter honours a --harness override; scan
prints the harness + an ambiguity warning.
- #6 conformance depth: cross-port invariants (names agree, instructionFile is a
declared target, plugin-root tokens match) + assertAdapterLoadsHooks (a real
settings round-trip that catches the JSON-vs-TOML trap).
Deferred to the actual Codex build (need the real second impl to shape right):
runtime wireMock, the Responses SSE renderer, instruction renderers, config-based
surfaces — documented in research/code-adapter-architecture.md.
Adds ts-essentials (queued for the branded-types standardization). Docs updated:
authoring-an-adapter (skeleton + --harness + behavioural conformance),
code-adapter-architecture (gaps fixed/deferred), harness-landscape, CLAUDE.md
Key Files. Coverage 100%, 774 tests.
* fix: update harness/eval example imports to reshaped dist paths
The reshape codemod only touched src/; the examples/ + bench/ *.harness.mjs /
*.eval.mjs import the built dist by relative path, so the moved modules (eval,
harness-test, plugin-loader, run-hook, sandbox → dist/adapters/claude-code/)
broke. Passed locally only against a stale pre-reshape dist (tsc doesn't clean);
CI's fresh build surfaced it (8/8 harness examples ERR_MODULE_NOT_FOUND).
harness-assert stayed at dist root (unchanged). Verified: clean rebuild + node
dist/cli.js test → 8/8 pass.
* fix: update cli.test synthesized-spec import path to reshaped dist (dist/core/spec.js)
The cli.test compile/audit cases write synthesized specs that import the built
spec module via resolve(cwd, "dist", "spec.js"); spec moved to core/ in the
reshape → dist/core/spec.js, so the synthesized specs failed to load (the
codemod rewrote import statements, not string-built paths). 8 occurrences fixed;
cli.test 55/55, full unit suite 772 green.
Note: the 2 sandbox *e2e* failures in CI are the known bwrap/slirp4netns runner
network flakiness (research/egress-sandbox-tooling.md), not this change — the
sandbox binds the whole FS (--ro-bind / /) and the mock-model endpoint logic is
byte-identical to before the ModelMock descriptor was introduced.
* docs: add vigiles/adapter API reference (docs/adapter-api.md)
The authoring kit had a tutorial but no API reference. Add docs/adapter-api.md:
every export of vigiles/adapter — the five port interfaces field-by-field with
Claude Code AND Codex example values, HarnessAdapter + the detect specificity
contract, the conformance functions (signatures + what each checks/throws), the
registry API, the programmatic apply path (compileAgent/loadPlugin with your
ports), an honest third-party-adapter status (programmatic supported now; CLI
auto-detect of external packages is planned), and a stability/semver note.
Linked from the tutorial, README, and Key Files.
* test: isolate sandbox e2e failure — revert in-bwrap mock-model import to literals
The 2 failing CI tests are ONLY the sandbox:"strict" e2e path (the
non-sandboxed harness examples pass 8/8), i.e. code that runs INSIDE bwrap. The
only in-bwrap code I'd changed is mock-model.ts, which gained an
'import { claudeCodeModelMock } from ./model-mock.js' + a module-load COUNT_ENDPOINT
const. Under bwrap's --clearenv/--ro-bind that's the one thing that could make
claudeCodeModelMock.modelEndpoint throw at mock startup → 0 turns served.
Revert that consumption to the original Anthropic literals (mock-model.ts is now
byte-identical to main in the hot path). The ModelMock port + claudeCodeModelMock
descriptor stay — still read by the adapter bundle + conformance — just not by
the in-sandbox server. Clean experiment: if CI goes green this import was the
cause; if still red, my code is fully cleared and it's the known bwrap-netns
runner flake. Coverage 100%, all non-bwrap tests green locally.
* test: self-skip sandbox e2e when the bwrap netns loopback is unavailable
Definitive finding from the isolation experiment (
|