15 Commits

Author	SHA1	Message	Date
zernie	e13d219ad0	chore: rename vitest.config.mjs to .ts, so both vitest configs are TypeScript (#229) ... The root config was plain JS while site/vitest.config.ts was TypeScript — an inconsistency, not a constraint. The file was already ESM with a typed defineConfig, so nothing inside it changed. The landmine was removed first (#223): the path guard DISCOVERS the filename via readdirSync().find(f => f.startsWith("vitest.config.")) instead of spelling the extension, so this rename touches its comment, not its code. Nine references updated; one was a live relative link in docs/testing-matrix.md that would have broken. CLAUDE.md was regenerated from its spec, not hand-edited. Mutation-proven three ways: green on .mjs, green on .ts, and a VALID config not named vitest.config.* fails by the guard's own message ("no vitest.config.* at the repo root") on exactly one assertion. Gates: npm run check 18/18 · npm run coverage 3976 passed, 0 failed — byte-identical to the pre-rename run. Merged without an automated review: the Codex reviewer reported its code-review usage limit was exhausted, so the signal was unavailable rather than pending. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-10 06:43:46 +05:00
zernie	195c1e14b7	feat: launch blockers, a hook-state test API, and the site's missing proof (#210) ... Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes. Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody. Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:. Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/.	2026-09-08 00:52:20 +05:00
zernie	9413061d9b	feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149) ... * refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-16 22:46:26 +05:00
zernie	a87d0cf2bc	fix(audit,testing): three dogfood defects — a heading, a salvage, and a check count (#130) ... A bundled script named in a markdown heading was invisible to skill-resource-resolves, because the use-directive gate reads prose and a heading is not prose. An ATX heading now counts; the widening stops there, measured at 1 finding before and after over 28 SKILL.md files. Malformed frontmatter was regex-salvaged into the tool contract the lethal-trifecta and Safety ring read, so a unit whose allowed-tools a strict loader rejects scored as the narrow contract its author meant. A salvage may now only make the verdict worse, never better — the vendored corpus forced that nuance by going red when the first pass refused salvages outright. vigiles test reported a clean exit with zero assertions as a pass. vigiles/testing now counts checks per tier and the runner reports a fourth, non-fatal state. Silence stays a pass: undefined means nobody counted, 0 means counted and found nothing. No new CLI verb or flag. Every mechanism is deterministic. Direction is strictly more conservative: nothing can now pass that could not before.	2026-08-09 02:25:39 +05:00
zernie	c442f4b523	feat: faithful caveman ecosystem benchmark + --plugin-dir/skillsDir install support (#61) ... Land the completed ecosystem-benchmark work that was stranded on the gate-release branch: the real caveman plugin install (pluginDir + SessionStart hook) with the codified bare-SKILL.md warning, the 7-task corpus, the analysis scripts, and the canonical 2026-07-07 runs (7 tasks × 5 trials, $10.41; compress input test) plus corrected FINDINGS. This is the data behind the published token-savings article and makes `npx vigiles eval` reproduce the real thing.	2026-07-07 18:19:18 +05:00
zernie	d8c639f8da	refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46) ... * docs: sharpen README lint/guard copy - Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff, Clippy, and four more)"; Cedar stays in the subdoc - Drop the plugin-audit paragraph from the Lint section (secondary use case, still reachable from More) - Rewrite the Guard intro to lead with the concrete "fail open" failure and name what vigiles emits (exit code, JSON, AST-backed matcher) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface the $0/subscription cost story in the first paragraph The "no metered tokens, runs on your Claude subscription" wedge was buried in the last line of the Eval section. Pull a crisp version into the opening paragraph (above the fold); the Eval section keeps the detailed promptfoo/DeepEval contrast as proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface agent-writes-the-harness flow, reorder More, drop internal links - Quick start now leads with "then your agent writes the harness for you": the three model-invocable skills (test-harness/strengthen/ edit-spec) and the plain-English ask each one answers - Reorder More so CLI sits above the leaderboard (whose link points into the CLI docs); trim the now-redundant Skills bullet to a pointer - Stop linking internal research/ docs from the README (the moat link and the Research index) per public-vs-internal-docs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(compile): auto-refresh an existing harness.gen.ts compile now keeps a whole-harness registry in sync as a side effect, so the user never hand-runs generate-harness. Gated on the file already existing (compile maintains a registry you opted into, never imposes one) and cheap (parses specs, no linter spawn — unlike generate-types/ generate-schema, which spawn every linter and stay on the config-change guard). A duplicate agent name fails the compile. Docs: note the auto-refresh in the compile section and mark the generate-* artifacts as dev-toolchain output rarely run by hand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule Directory-scoped guidance loaded when working in src/: adding a new CLI verb or hook-runtime subcommand must clear a high bar (genuinely new mental action, no overlap with an existing command, siblings collapse to one verb + subcommand, runtime entrypoints stay hidden). Complements the root cohesive-cli-surface rule, placed next to the code that defines the command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec + integrity); dogfoods vigiles compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the "(generated)" annotation from the API reference link Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule The nested src/CLAUDE.md.spec.ts (added last commit) compiles under rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so both load via the tsx fallback (as the root always has). Also adds the no-internal-links-in-public-docs guidance rule (public docs never link research/*.md; keep the fact in research, don't link it), reconciles the old "link to research" line in public-vs-internal-docs, and files the deterministic lint-rule version as a P1 roadmap item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: separate public/internal indexes + surface eval safety - docs/README.md no longer links into internal research/ (public docs don't point at the private record) - research/README.md is explicitly labelled the internal record - README Eval section gains a "Safe to repeat" note (sandbox + ephemeral dir + egress + interceptTools, past-the-sandbox case flagged) linking docs/safety.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: stop public docs linking internal research/ + add safety FAQ Strip every research/*.md link from docs/*.md (the no-internal-links-in- public-docs rule) — ~60 links across 16 guides plus the docs/ index; prose kept readable, no orphaned research doc (all still cited from CLAUDE.md keyFiles / research/README.md, orphan-docs green). Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox question honestly: network via egress allowlist, a real DB/redis/browser via a disposable container you stand up (turnkey provisioning is a committed next step), irreversible externals via interceptTools, and the no-Docker deterministic/record-replay tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind> De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands rule, dogfooded). All breaking: - `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral column folded into scan; trigger-rate + collision matrix unchanged) - `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic WHY, a sibling of --fix-plan on the same ScanReport) - `refs <file>` → folded into `lint` (the unmarked-refs check is already a lint rule; the PostToolUse hook keeps its shared collectRefIssues entry) - `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand; the three dev-toolchain generators grouped, not N hyphenated siblings) scan reuses handleMeasure/handleExplain directly behind --trigger/--explain. Docs, help text, the regen guard, tests, and the self-command-refs dogfood all updated together (cohesive-cli-surface); 1588 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: split the GitHub Action reference out of cli.md into its own doc cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action section into docs/github-action.md (quick start, inputs, output channels, versioning) and leave a short pointer; retitle the cli.md intro. Update the README More bullet (CLI · GitHub Action) and the docs index. cli.md → 619 lines. orphan-docs + self-command-refs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks A single repo-level recommendation (one finding regardless of hook count): when a plugin ships hand-written hook commands that aren't compiled vigiles/hook artifacts, nudge toward compiling them — the message links docs/compiled-hooks.md and points at guardrail-check for verifying the ones you have. A discovery nudge, not a defect: the hand-written shell lane stays first-class, so it fires ONCE (never per-hook), defaults to warn (info- level ℹ; informational, doesn't fail CI), and is opt-out. Detector (manualHookCount: a command that isn't a hook-runtime invocation) lives in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc + matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's form-based, so it stays a recommendation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the eval-startups critique mapped onto positioning New research/eval-startups-positioning.md captures Thomas Liao's "why eval startups fail" critique and its implications: vigiles dodges the trap (harness-eval not model-eval, tooling not services, eval already a feature), but it's a yellow flag on the public ecosystem-benchmark as a MOAT (Goodhart-gameable → keep it a thin acquisition flywheel) and it independently validates the attestation/safety direction (capability-diff, guardrail proof) as the only defensible eval-shaped niche. Linked from research/README.md, measurement-authority.md (whose benchmark thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2 conformance/attestation play it validates). Internal-only per no-internal-links-in-public-docs; orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: purge internal positioning from public docs (jargon, name-drops, benchmark) Public docs (README + docs/) now state user-facing value, not vigiles's internal strategic vocabulary: - strip "moat" from 14 places → user-facing wording (engine / cost advantage / differentiator / cross-reference) - remove all bare research/*.md name-drops from docs/ (the backtick form the link-strip + deterministic check miss); scope-descriptions reworded to "configured directories" rather than naming research/ - README: drop the "ranks the hyped ecosystem: what works vs hype" benchmark banner from the Eval row + kicker (depth stays in measuring-skills.md) — per eval-startups, the benchmark is a flywheel, not a headline - docs/cli.md: "moat #2 (research/…)" → "the capability-diff check" Codified in CLAUDE.md: public-vs-internal-docs now has a strategic- vocabulary clause (moat/measurement-authority/flywheel stay internal; public docs name the benefit), so this doesn't regress. orphan-docs + self-command-refs green; the public↔private split stays docs/ vs research/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): apply multi-persona review fixes From a multi-persona README review (power user / staff eng / plugin author / non-TS user / eng manager): - non-TS users: a visible "No TypeScript needed — lint + inline comments work on any CLAUDE.md" line (was invisible; contradicted progressive adoption) - plugin authors: surface "vigiles scan ranks a marketplace, no key" above the fold (was buried in More) - eng-manager cost: hero now says deterministic layers run free in CI; merged kicker+affordability nails "real-model on a dev's own subscription, never a CI token" - staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted"; the typed-pipeline claim is concretized to the real pipe()/needs() mechanism - fixed a broken blockquote and trimmed the safe-to-repeat para to hold length Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): capture the README design direction as an editor-facing comment A top-of-file comment block records the framing the README must hold: lead with benefits, present compiled specs as the easy/auto-enforced next step (skills write them, hooks compile them; JS→TS evolution, not a wall), keep all four instruments first-class incl. Eval, stay scannable + under the length cap, and no internal vocabulary/research links. Guides future edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): benefits-forward rewrite + specs-are-easy framing Per the README direction: lead with what the user GETS, and present the typed spec as the frictionless auto-enforced next step, not a TS wall. - hero: stronger, concrete failure modes (stale ref / guard fails open / skill never fires), keeps Agent=Model frame, cost above the fold; dropped the redundant centered pitch (merge) - table cells condensed to punchy benefit one-liners - "You don't hand-write specs — your agent does. TypeScript for your harness: opt-in, gradual, auto-compiled, never a chore" (skills write them; JS→TS evolution) - de-dupe cost (drop "not metered tokens" from the Test cell) - "Not for you if …" honest-scope line - docs/verifying-instruction-files.md: back the claim — "you rarely hand-write one; edit-spec/strengthen write it, a hook recompiles on save" Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): don't lead callouts with "typed spec" — benefit first Casual readers shouldn't be greeted by "a typed spec is a program" / "TypeScript for your harness" in the prominent callouts. Reframe both to lead with the outcome and demote the spec to a soft, opt-in aside: - specs-are-easy block: "You don't hand-write any of this — your agent does … you never touch it by hand" (drop the TypeScript-for-your-harness headline) - Lint blockquote: "Bad states stop compiling … turn it up only when you want it" (drop "Markdown is prose; a typed spec is a program") - the edit-spec bullet says "edits the source", not "the typed spec" "No TypeScript?" reassurance and the optional .spec.ts mention stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: prettier-format HANDOFF Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): update the stale generate-types ref the verb rename missed The CI "Check generated types compile" step ran `node dist/cli.js generate-types` — renamed to `generate types` in the verb consolidation, so the job failed with "Unknown command". self-command-refs didn't catch it because it didn't scan .github/; add .github/ to its scan scope so a stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on. Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to research/hook-modes-and-testing.md (the nudge, not a requirement; the write-hook authoring skill is the real authoring lever, unbuilt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig) The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts and errors "not found by the project service". It's a tsx-loaded build input (like the root CLAUDE.md.spec.ts, which isn't linted either), so ignore it in eslint to match. Latent since the exclude landed; surfaced now that the earlier generate-types step no longer fails first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-25 01:20:05 +07:00
zernie	fe12c14a30	docs: de-duplicate and tighten README intro (#45) ... * docs: de-duplicate and tighten README intro Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat!: collapse runtime entrypoints under `vigiles hook-runtime` The CLI had grown ~15 stray top-level commands that a human never types — run-hook-program, agent-hook, skill-hook, skill-tool-hook, guard-hook, action-hook, refs-hook, intercept-tool-hook, agent-start/done, skill-start/done, run-skill, effect-enter/exit. These are RUNTIME entrypoints: the harness invokes them via a block vigiles emits into a hooks config; they are not verbs. Collapse them under one hidden umbrella, `vigiles hook-runtime <kind>`, kept off the help surface (verbs are typed; runtime entrypoints are emitted). Every emitter now emits the new form; `refs` stays a verb. Add the cohesive-cli-surface rule to CLAUDE.md encoding the verb-vs- runtime-entrypoint taxonomy and the three-faces (CLI/GHA/agentic) coherence requirement. BREAKING CHANGE: the top-level runtime commands are removed. Any already-emitted settings block referencing them must be regenerated with `vigiles compile` (which re-emits the `hook-runtime <kind>` form). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat!: fold hook compilation into `vigiles compile` `compile-hook` was a stray sibling of `compile` doing the same mental action — compile a typed authoring artifact into the harness's native format. Fold it in: one verb compiles both .spec.ts (→ markdown) and hook programs (→ harness config + stamp). Hook SOURCE now lives in the agnostic, committed .vigiles/hooks/ (the typed hook imports vigiles/hook and compiles to ANY harness, so it must not live in a harness's own .claude/). Bare `vigiles compile` discovers specs AND .vigiles/hooks/*; explicit `compile <hookfile>` works too. `compile` now WRITES the wiring instead of printing a paste-this block: it MERGES the compiled block into the active harness's native config (.claude/settings.json JSON / .codex/config.toml TOML) idempotently — keyed by the hook path, so recompiling updates in place and never clobbers the user's own hooks. One source dir also makes basenames unique, retiring the stamp basename-collision edge. New src/hook-install.ts holds the pure, tested merge/discovery helpers. BREAKING CHANGE: the `compile-hook` command is removed; use `vigiles compile` (it discovers .vigiles/hooks/ or takes an explicit hook path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: align docs, comments, and spec with the cohesive hook CLI Phase 3 of the CLI cohesion pass — update every reference to the removed `compile-hook` / top-level runtime commands: - docs/compiled-hooks.md: rewrite "Compile and run" → "Compile and wire" (`vigiles compile` discovers .vigiles/hooks/*, MERGES into the native config, writes the stamp) + a new "Where things live" table answering where hook source/stamp/wiring live and that they're not auto-discovered by location alone. - docs/cli.md: drop the compile-hook/run-hook-program entries; document hook compilation folded into `compile` + the hidden `hook-runtime` umbrella. - README, eval-architecture, skills, verifying, unmarked-refs, e2e/bench scripts, research/*: mechanical command rename. - CLAUDE.md(.spec.ts): keyFiles + positioning updated; add hook-install entries; recompiled. - Source JSDoc + the safe-bash-guard example header point at the new commands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: correct 4 stale runtime-command usage strings missed by the manual sweep Three `Usage: vigiles <kind>` error messages (run-skill, skill-start, agent-start) and one JSDoc comment still printed the pre-umbrella command form. They now say `vigiles hook-runtime <kind>`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: dogfood the cross-ref moat on vigiles's own command surface A guidance rule alone didn't stop the rename from leaking stale `vigiles compile-hook` / `run-skill` refs into the docs (it leaked 8, even with the cohesive-cli-surface rule freshly written). Per vigiles's own thesis — deterministic constraints over probabilistic compliance — add the deterministic gate. - src/cli-commands.ts: VERBS + HOOK_RUNTIME_KINDS, the single source of truth (a behavioural test keeps it honest vs the dispatch). - src/self-command-refs.ts: pure detector — every `vigiles <cmd>` in a command context must resolve to a real verb or hook-runtime kind. High-precision: only inline code spans / shell fences / npx|Usage:|cli.js invocations; a bare unknown verb only when hyphenated or explicit. - src/self-command-refs.test.ts: unit tests + the repo DOGFOOD (scans docs/README/CLAUDE.md/src/examples/hooks, not research/ the historical record) that fails CI on a stale ref. Running it caught 3 more stale refs the manual sweep had missed (tool-intercept + refs-hook test comments) — now fixed. This is what 'enforce updating the docs automatically' means: the cross-reference moat, pointed at vigiles itself. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: dogfood the compile-merge against a real plugin's hook config Close the merge half of the compiled-hooks OSS dogfood gap: compile a vigiles gate INTO the REAL superpowers hooks.json (vendored, MIT, SHA-pinned) and assert the merge is non-destructive — the plugin's own SessionStart hook (incl. its async flag) survives, vigiles's PreToolUse gate is added, and recompiling doesn't duplicate. Proves the new settings-merge works on a real-world config, not just synthetic unit fixtures. The gate 'golden before' stays a faithful reconstruction: the canonical disler hook is unlicensed (can't vendor), and an MIT alternative pulls in a jq + /tmp runtime dependency that would make a committed CI test flaky — a bad trade vs. the existing faithful reproduction + the documented real measurement in research/hook-pain-points.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add the doc-consistency rule + fix a tsc break in the merge dogfood Add the standalone doc-consistency guidance rule asked for: when a capability changes, update BOTH public (docs/, README) and internal (research/, CLAUDE.md) docs in the same change. It names the split — the CHECKABLE half is enforced by deterministic CI gates (self-command-refs, orphan-docs, doc-refs, integrity), never trusted to discipline; the JUDGMENT half (tier, prose, cross-links) is the discipline, governed by the sibling docs rules. 'Mechanize the consistency fact where you can' is the principle; the check is the floor, updating both tiers is the ceiling. Also fix a tsc error in the Commit-B merge dogfood: the spawnSync helper annotation picked the Buffer overload (vitest's esbuild run didn't catch it; npm run build does). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add the prefer-existing-solutions rule Codify the reflex this session kept surfacing ('is there an existing solution?'): before building any non-trivial capability, search the landscape and prefer adopt > compose > build. BUILD is justified only when no existing option fits, it dogfoods the core moat, or every option's cost outweighs a small purpose-built piece — and you must name the prior art and why it was rejected (as the self-command-refs decision did). Generalizes the existing dont-reimplement-linters + compose-with-sync-tools stance into the default for all new work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: E2E dogfood the file-gate + react hook roles; document coverage Close the two remaining compiled-hook dogfood gaps with real CLI-runtime E2E tests (mirroring the gate/inject E2E): - file-gate (defineFileGate + PathView.under): denies a Write under a protected path, allows elsewhere. - react (defineReact): a notice reaches stderr + can't block (exit 0); run() executes its effect-classified command. Document the full per-capability dogfood coverage matrix in research/hook-pain-points.md (the internal record), incl. the two deliberate non-coverages with reasons: the gate 'golden before' stays a faithful reconstruction (the canonical hook is unlicensed; the MIT alternative drags a jq+/tmp runtime dep), and inject/react have no deterministic 'prove worth' oracle so they're proven structurally, not scored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add the test-both-harnesses rule + annotate the hook E2E harness scope A test of a harness-FACING capability must cover BOTH Claude Code AND Codex, never just CC by default — the test-layer companion to adapter-aware-lint-rules. Three sub-rules: test what maps on both (compile --harness=codex beside the CC default; agnostic detectors over a non-CC layout); gate/skip LOUDLY where a harness can't run (no binary on PATH) or a capability is deferred (Codex inject/react output); don't double-test agnostic core. A genuinely harness-neutral path runs once but must SAY WHY, so 'CC-only' is never ambiguous. Dogfood it: annotate the new file-gate + react E2E with their harness scope (deny→exit 2 and run()→spawn are neutral, one run covers both; Codex emit is tested via --harness=codex; Codex notice/react output is deferred). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: structural both-harness enforcement via a registry contract suite Make the 'test both harnesses' gap structural, not disciplinary (hexagonal contract-tests-over-the-registry pattern): - src/adapter-contract.test.ts: runs the conformance kit over the WHOLE registry in a loop, so registering an adapter auto-subjects it to every contract (can't forget a new harness); a lacked capability is a VISIBLE it.skip(n/a) gated on shellHooks/harnessTesting (no silent CC-only pass); and a meta-test fails the build when a src/adapters/<dir> exists but isn't registered (or a declared prototype like opencode). - Document the KNOWN GAPS in CLAUDE.md (test-both-harnesses rule): the contract catches ADAPTER-level gaps but can't auto-catch a capability assertion written CC-only OUTSIDE the contract (the judgment the rule governs), nor force OSS/real-binary coverage (gated loudly). Floor vs ceiling, stated not hidden. - docs/authoring-an-adapter.md: 'how the registry tests your adapter automatically' for new-adapter authors. - docs/testing-matrix.md: fix 5 stale test paths (adapter reorg) + the agent-hook->hook-runtime agent rename the prefix-scoped drift check couldn't see. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: correct 5 genuine stale path refs; document why bare-name drift can't be auto-checked Investigating whether to widen self-command-refs to catch bare command names (the agent-hook blind spot) — the data says no: bare names double as CONCEPT names ('the refs-hook nudge' in ~10 files), test labels, and makeTmpDir() names; bare file paths collide with illustrative examples (README's src/auth/login.ts) and test fixtures. A denylist over either would cry wolf on dozens of valid usages — so it stays the JUDGMENT half the doc-consistency rule owns, not a check (measured, documented in the detector header + the rule). But the probe surfaced 5 REAL stale refs (fixed by hand): - src/skill-runtime.ts → src/adapters/claude-code/skill-runtime.ts (docs/skills.md, agent-runtime.ts, inline.ts) — moved in the reorg. - src/adapters/claude-code/mock-model.ts → src/mock-model.ts (codex/mock-model.ts, harness-driver.ts) — startMock lives at the root. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): hook modes & testing landscape analysis Step-back analysis of the hook surface (check + run) vs the OSS/AI landscape (OPA/Rego, Guardrails AI on-fail actions, NeMo Colang, Cedar, lefthook, Claude Code's new prompt/agent hook types). Verdicts (corrected from the first pass): - DON'T restrict the runtime to TS-only: two opt-in lanes (typed TS with NO embedded shell + audited/sandboxable hand-written .sh), the OPA-coexists pattern; authoring-safety=capability vs runtime-safety=confinement are separate concerns. - Modes collapse to ONE essential new mode "observe"/shadow (evaluate + record, never block - the gradual-rollout primitive) on the existing "enforce" default, NOT a 4-mode vocabulary (the warn/shadow split was a rendering detail, not a mode). - Testing is tiered already; the gap is COVERAGE (OPA-style) + behavioral gate-precision + the model-gated prompt/agent hook types. Ranked deliverables: observe mode (#1), hook-test coverage, react effect ceiling, prompt/agent-hook verify+judge. Referenced from CLAUDE.md keyFiles (orphan-docs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): 10-OSS dogfood — does the typed hook lane cover all use cases? Mapped real hooks from 10 OSS sources (disler/claude-code-hooks-mastery full event set, alexknowshtml safety hooks, vendored superpowers + oh-my-claudecode, CC-docs canonical, ruff/ty validators, TTS/notify, gmickel/flow-next, guide patterns) to the vigiles/hook vocabulary. Verdict: NO, not all — and that's the two-lane decision working. The typed lane nails the SAFETY-GATE slice (PreToolUse block: dangerous bash, secret reads, curl|sh, protected-path edits) + simple react/inject. Gaps cluster by root cause: - PURITY (no exec / thin event shapes): dynamic-context inject, TTS, backups, structured logging, tool_response reactions -> these NEED I/O, so they belong in the shell lane; closing them in typed would break the capability guarantee. - MISSING roles/shapes (closeable, worth it): gate-capable UserPromptSubmit (block/rewrite a prompt) + Stop (gate-until-tests), and richer event shapes (tool_response, prompt text, non-Bash tool_input) so more DECISIONS are expressible. Refined scope: typed lane = DECISIONS over any event; shell lane = I/O & lifecycle side-effects. Re-ranks deliverables: event/shape coverage for decisions rises above the observe mode for breadth. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the type-sourcing + licensing decision Capture the conclusion from the black-box / Google-v-Oracle research: CC's only official machine-readable artifact is the settings.json schema; events/tools/payloads are prose-only, so the dialect catalog is hand-maintained (and the detectors are conservative because of it). The installed @anthropic-ai/claude-code (sdk-tools.d.ts + cli.js) is a read-local freshness/drift source, NOT a dependency. Licensing (informational): all-rights-reserved -> don't vendor or let api-extractor INLINE their .d.ts into our shipped artifact (that's redistribution; import alone isn't, but dts-bundling inlines by default). Hand-write our own types matching the FACTS (names/fields aren't copyrightable; DefinitelyTyped norm) + read the local install to warn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: dialect drift-check — read-local freshness alarm for the hand-maintained CC catalog CC is a black box, so claudeCodeDialect is hand-maintained. Add the deterministic backstop: read the user's INSTALLED @anthropic-ai/claude-code (sdk-tools.d.ts tool-input types + cli.js event strings) and FAIL LOUD when its surface drifts from our catalog — so a CC update that adds/removes a tool or renames an event turns CI red instead of silently staling us. - src/dialect-drift.ts: pure parsers (parseToolInputTypes, eventsMissingFromBundle) + findClaudeCodePackage (npm-root-g / claude binary). READS LOCAL ONLY — ToS-clean, never vendors/ships their types (all-rights-reserved). ACKNOWLEDGED_TOOL_INPUT_TYPES is our own list of bare identifiers (facts), diffed against the install. - src/dialect-drift.test.ts: pure units + GATED read-local checks (validated vs claude-code 2.1.42: 19 tool types, 9 events); skips loud when CC absent. Also fix 11 pre-existing unsafe-any lint errors the merge dogfood snuck into hook.test.ts (typed the settings.json JSON.parse) — they'd have failed CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(scan): use the dialect drift-check at runtime + update internal docs The drift module was added but only consumed by its own CI test. Wire it into the product so it's actually USED: - src/dialect-drift.ts: add checkDialectDrift() (best-effort, read-local; reads only the small sdk-tools.d.ts — no cli.js scan on the runtime path; never throws) + formatDialectDrift() (one-line ⚠ ONLY on real tool-surface drift — a bare version bump emits nothing, no noise) + VALIDATED_CC_VERSION (2.1.42). - src/cli.ts: `vigiles scan` (claude-code only) now prints the freshness warning when the installed CC's tool surface drifts from our catalog. Deliberately NOT in compile (hot recompile-on-save path). - 2 formatter unit tests (null on no-drift / version-only; message on added+removed). Docs updated in detail (doc-consistency): research/code-adapter-architecture.md (the sourcing/licensing section now records the SHIPPED module + its two consumers + the runtime-warn behaviour) and the CLAUDE.md keyFiles entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: note the scan dialect-freshness warning in cli.md Public scan reference now documents the read-local Claude Code freshness check (best-effort ⚠ on tool-surface drift; reads only the user's own install; silent on a no-op version bump). Completes the doc-consistency for the drift-check across internal + public tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): observe mode + gate-capable UserPromptSubmit/Stop + tool response Two compiled-hook expansions, both shipping for Claude Code AND Codex. 1. observe mode — every gate takes mode: 'enforce' | 'observe'. observe is the shadow/rollout primitive: compute the same Decision, record what it WOULD block to .vigiles/hook-observations.jsonl, exit 0, never block. Harness- neutral by construction (exit 0 + a local record). gateAction(decision, mode) is the pure decision->action mapping the runtime and tests share; hookMode reads a gate's mode. 2. gate-capable non-tool events + richer event shapes: - definePromptGate (UserPromptSubmit) sees the prompt text (e.prompt) and may deny to block/erase it — a security filter. - defineStopGate (Stop/SubagentStop) may deny to keep the agent going (gate-until-tests-pass), honouring e.stopHookActive as the loop guard. - react now sees the tool RESPONSE via e.response (ResponseView: isError()/contains()). Both new gates ride the shared exit-2 gate runtime, so they work on every harness whose gate vetoes via exit 2 (compile-emit covered for CC and Codex). The generic tool_input accessors (WebFetch/Task/MCP) stay deferred to avoid a stringly-typed input bag. Tests: pure (gateAction, prompt/stop decode, responseView, CC+Codex compile) in hook-program.test.ts; E2E over the real runtime (prompt-gate deny, stop-gate loop guard, observe records-not-blocks) in hook.test.ts. Public surface + api report + docs/compiled-hooks.md + the research record updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): cohesive pillar names — Verify→Lint, Measure→Eval Align the four README pillar labels with the CLI verbs (and CLAUDE.md's own "Lint"/"Test" layer names) instead of marketing renames that drifted from them: Verify→Lint, Measure→Eval (Guard/Test unchanged). Keeps the differentiation in the subtitles — Lint stays "is this TRUE, not just well-formed", Eval stays "the eval you can afford" — so the wedge survives the plainer, cohesive labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): design for I/O-dependent hook decisions via declared context providers Captures the design (no code yet) for letting a compiled hook decide on external state — git branch, project-specific facts — without breaking capability = API surface. Grounded in prior art (Cedar entities/context, OPA input/data vs the "ugly" http.send, Gatekeeper's registered-Provider model that deliberately rejected http.send): the policy never fetches; the trusted host gathers facts and hands them in. Core reframe: the moat is "no UNDECLARED capability", not "zero I/O". decide() stays pure; the runtime gathers a DECLARED set of read-only facts into e.ctx (typed needs[], undeclared access = a tsc error). The opt-out ladder (graceful degradation, never a dead end): Tier 0 built-in matchers -> Tier 1 built-in providers -> Tier 2 user-declared providers (the long tail, effect-classified, dangerously opt-out) -> Tier 3 shell lane (Turing- complete). Proves total coverage by mapping every real-world hook from the 10-OSS dogfood to a tier. Cross-linked from hook-modes-and-testing.md; keyFiles + CLAUDE.md updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): context providers v1 — gates can decide on git state via needs/e.ctx Lets a compiled gate decide on EXTERNAL state (git branch, dirty tree, cwd) without breaking capability = API surface. The pattern (Cedar/OPA/Gatekeeper, research/hook-context-providers.md): the pure decide() never fetches; the trusted runtime gathers a DECLARED set of read-only facts and hands them in as e.ctx. - A gate declares `needs: ["git.branch"]` and reads `e.ctx["git.branch"]`. The gate builders are generic over the declared needs, so reading an undeclared fact is a tsc error, and an unknown provider name won't compile. - New core/hook-providers.ts: the closed built-in registry (git.branch, git.isDirty, cwd — each total, defaults on failure), ProviderName/ ProviderResults/HookCtx<N>, and gatherContext(needs, io) parameterized over an injected exec (testable, core depends on no child_process). A soundness test asserts every built-in command is read-only via bash-effects. - The runtime injects the real execSync (cli.ts gatherHookContext); the hook itself still does zero I/O. Long tail stays covered by the opt-out ladder (v2 user-declared providers, then the shell lane) — designed in research/hook-context-providers.md. Tests: pure (hook-providers.test.ts, hook-program.test.ts) + E2E in a real git repo (hook.test.ts: deny push on main, allow on a branch). Public surface + api report + docs/compiled-hooks.md + CLAUDE.md updated. Additive — hooks without `needs` are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): 20+ OSS provider survey + the lightweight opt-out design Answers the "what built-ins do we need + how should the quick opt-out look" questions, grounded in research. - Survey of ~21 real hooks (aiorg.dev 20+ catalog, CC docs, the vendored superpowers/omcc/wshobson slices, disler/dcg): MOST facts a hook reads are EVENT data (path/command/tool/stop-flag/session-type), not providers — so the built-in set stays tiny. Verdict: git.branch/git.isDirty/cwd (shipped) + os.platform; everything else (env, test/lint status, rate-limit, online) → opt-out, not a growing catalog. - The lightweight opt-out, judged against keeping decide() pure: an INLINE declared command in needs — provide(name, cmd) (read-only, compile-rejected if not) / dangerously(name, cmd) (acknowledged escape, greppable). Rejected: a method-in-decide (breaks purity), @ts-expect-error (grants no capability — category error), and a whole-hook I/O mode (that's just the shell lane). Naming follows the loud-escape-hatch best practice (dangerouslySetInnerHTML / unsafe / http.send / our own purity:'dangerously-unrestricted'). - Event naming: native + dialect-validated, NOT hardcoded to CC (PreToolUse is in both dialects); a generic alias layer is deferred until a divergent harness lands (rule-of-three; the dialect is the seam). Fixed a stale "raw CC PreToolUse event" comment. Build order updated: v1 built-ins (shipped) -> os.platform -> inline provide/dangerously -> v2 defineProvider. CLAUDE.md keyFiles updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the one-source-many-harnesses event-naming decision A single hook source already targets CC + Codex today (author once in .vigiles/hooks/, compile emits each native config; the typed program, context gather, and gate runtime are harness-neutral). Native event names work for both because the dialects share them; the inject/react OUTPUT shape is the one CC-confirmed-only caveat (compile --harness=codex warns). Decision: native + dialect-validated names, NO generic alias layer yet — it would only pay off for a future harness with divergent event names (none today), so it's deferred to a small per-dialect map at the dialect seam, additive when needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(claude-md): add harness-parity-and-extensibility design rule A design-time umbrella over the existing enforcement rules: when designing ANY capability (compile emit, runtime, ports, subsystems — not just lint/test), (1) fully support BOTH shipping harnesses (Claude Code AND Codex) as equals, never CC-first-Codex-bolted-on, with loud documented deferrals where a capability can't map yet; (2) leave room for future adapters by putting every harness-specific fact behind a port and threading the resolved adapter (core ⊄ adapter), so adapter #3 is a new object not a core edit; (3) design the neutral shape first but defer the abstraction until a second divergent implementation earns it (the seam exists, the mapping is added when needed). Points to its enforcement arms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): os.platform built-in + inline provide()/dangerously() context opt-out Closes the provider/long-tail story (research/hook-context-providers.md), grounded in a 20+ OSS survey: most facts a hook reads are already event data, so the built-in set stays small and the long tail goes to an opt-out, not a growing catalog. - os.platform built-in (ambient process.platform) — the one extra fact the survey justified (per-OS hooks). Built-ins now: git.branch/git.isDirty/cwd/os.platform. - The lightweight opt-out: an INLINE command in `needs` — provide(name, cmd) (read-only; compile rejects a non-read-only command via unsafeInlineProviders) or dangerously(name, cmd) (the loud, greppable escape, the dangerouslySetInnerHTML/unsafe convention). The runtime runs it and hands stdout in as e.ctx[name]; decide stays pure. Chosen over a method-in-decide (breaks purity), @ts-expect-error (grants no capability), and a whole-hook I/O mode (that's the shell lane). NeedSpec = ProviderName | InlineProvider; HookCtx is a key-remap mapped type over it (built-in name → typed value, inline → string), undeclared access still a tsc error. Decode fns are generic over N; AnyHook uses an erased any-needs because a gate's decide is contravariant in N (documented). Tests: pure (hook-providers.test.ts: os.platform, inline gather, unsafeInline, unknownProviders; hook-program.test.ts: inline gate + provide-not-read-only rejection) + E2E (hook.test.ts: an inline provide() fact gathered by the real runtime drives a deny). Public surface + api report + docs/compiled-hooks.md + CLAUDE.md updated. Additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): v2 registered providers + aggressive OSS comparison dogfood v2 — REGISTERED context providers (the reusable, named tier of the opt-out ladder). Author a fact once in .vigiles/providers/<name> via defineProvider({ name, run }) and reference it from any hook by provider("name") → e.ctx[name]. The runtime discovers + loads the registry (loadProviderRegistry) and resolves refs in gatherContext; `vigiles compile` validates each provider is read-only (unless dangerous:true, via unsafeProvider) and that every provider() ref resolves to a registered file (a dangling ref fails compile). NeedSpec is now ProviderName | InlineProvider | RegisteredRef; decode fns are generic over N and AnyHook uses an erased any-needs (a gate's decide is contravariant in N). Surface: defineProvider/provider + types on vigiles/hook. Aggressive OSS comparison — src/hook-oss-comparison.test.ts isolates the NON-CIRCULAR structural wins over hand-written guards, one failure mode per test (evasion: the compound `git push -f` a substring/glob misses; precision: a grep false-positive on a benign echo; protocol: an exit-1 false-confidence guard) plus the breadth headline (2/7 blocklist vs 7/7 compiled). research/hook-oss-comparison.md documents the head-to-head, honest about both the labelled-breadth caveat and what compiled hooks do NOT do better (stateful / broad-I/O / delivery → shell lane). Tests: pure + E2E (a .vigiles/providers/ file resolved by the real runtime) + the comparison battery. Public surface + api report + docs/compiled-hooks.md + CLAUDE.md updated. Additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cli-install): loud-skip the codex e2e when `skills add` can't reach the network The e2e's guard probed `npx skills --help` (reachable via npm) but the real `skills add` does a GitHub fetch — so in a partial-network sandbox it slipped past the skip-guard and failed RED, contrary to its own documented contract ("self-skip loudly where the network isn't reachable"). Now a network-class failure of the real `add` (read from stderr) is re-classified as a loud ctx.skip(); a genuine install bug still throws, and CI (GitHub reachable) still runs the full assertions. Pre-existing flakiness, unrelated to the hook work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): add git.root + env.isCI built-in providers (env.isCI adopts ci-info) The two extra cheap-ambient facts the 20-OSS survey justifies, closing the built-in set at git.branch/git.isDirty/git.root/cwd/os.platform/env.isCI (small by design; the long tail stays in the opt-out tiers). - git.root — read-only `git rev-parse --show-toplevel` (the soundness test confirms it classifies read-only); pairs with cwd/path decisions. - env.isCI — ADOPTS the ci-info library (de-facto standard, ~30+ CI vendors) rather than hand-rolling an env check; injected via ProviderIO.isCI so core stays dep-free (the dep lives at the CLI composition root). The prefer-existing-solutions split: git facts stay read-only shell (a JS git lib would bypass the bash-effects soundness check + add a dep for a one-liner), platform is process.platform, and the provider-registry architecture has no embeddable lib (the OPA/Cedar/Gatekeeper pattern, built here). Tests, api report, docs/compiled-hooks.md, research/hook-context-providers.md, CLAUDE.md + package-lock all updated. Additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: cover registered providers in cli.md + the compiled-hooks "where things live" Close the two doc gaps the providers work left: docs/cli.md's `compile` section now documents context providers (built-ins + inline provide/dangerously + registered .vigiles/providers/ discovery/validation), and the compiled-hooks "Where things live" table gains the .vigiles/providers/ source row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(docs): publish the generated API reference to GitHub Pages + link it Adds .github/workflows/api-docs.yml: on push to main, generate the API Documenter markdown (api-extractor doc model → api-documenter), make it Jekyll-renderable (inject front matter + a minimal _config with jekyll-relative-links so the generated ./foo.md links resolve to .html), and deploy to GitHub Pages via the official configure/build/upload/deploy-pages flow. api-reference/ stays gitignored (generate-not-commit, ~1900 pages); the committed surface artifact remains etc/*.api.md (already CI-gated). Linked the site (https://zernie.github.io/vigiles/) from the README "More" row, the docs index Reference section, and the compiled-hooks + harness-testing "See also" (the hook/testing public surfaces). NOTE: requires a one-time repo setting — Settings → Pages → Source: "GitHub Actions" — and the links go live after the first main deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(docs): switch the API-reference site from api-documenter to TypeDoc api-documenter pages EVERY member (one file per interface field) → ~1900 flat markdown files, the wrong shape for a TS library, and needed a Jekyll/front-matter hack to render. Switch to TypeDoc (the de-facto TS API-docs tool): - typedoc.json: curated to the public AUTHORING entry points (spec/linting/ testing/hook/adapter/claude-code/codex — not the unit/integration/e2e re-export barrels or the tiny vitest/jest matchers), public-only (excludeInternal), members rendered INLINE. - ~465 navigable pages with a sidebar + search (vs 1900 flat files), and `.nojekyll` (githubPages:true) → Pages serves the HTML as-is, so the api-docs.yml deploy drops the Jekyll/front-matter steps entirely. - ci.yml smoke step + the docs:api script now run typedoc; @microsoft/api- documenter removed. api-extractor (etc/*.api.md surface gate) is unchanged. README + docs links unchanged (same https://zernie.github.io/vigiles/ URL). Still requires the one-time repo setting: Settings → Pages → Source: GitHub Actions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(api)!: curate testing/linting barrels — drop internal seams from the public surface Replace the `export *` re-exports in `vigiles/testing` and `vigiles/linting` with curated named re-exports, so the internal DI seams (`*With` runners), low-level parsers (`parse*`, `decideHook`), pool/aggregate/model-tier helpers, and the linter cross-reference ENGINE no longer leak into the public .d.ts surface, the api-extractor reports, or the TypeDoc site. The CLI imports those from the source modules directly; they were never meant to be public. Also demote a few doc-comment `{@link}` refs (`ephemeralRunEnv`, `spawnAgent`, `runEvalWith`) that pointed at now-internal symbols to plain code spans, so the generated API site has no dangling links. Net effect: api report drops ~311 lines across the two surfaces; the TypeDoc site drops 465 → 407 pages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(lint): ban internal imports of the public barrel entry points Add a no-barrel-imports ESLint rule so internal modules import the LEAF that defines a symbol, never the package's own public barrel surfaces (vigiles/{linting,testing,unit,integration,e2e,hook,claude-code,codex}). Barrel imports pull the whole re-export graph (slow in the test runner / any non-treeshaking consumer, a circular-import risk) and re-leak the internal seams the curated barrels deliberately drop — so this locks in the preceding barrel curation. Implemented with the built-in `no-restricted-imports` rule rather than eslint-plugin-barrel-files: that plugin's `avoid-importing-barrel-files` calls the ESLint-9-removed `context.getFilename()` and crashes on ESLint 10, so per prefer-existing-solutions a working core rule beats a broken dependency. The barrels themselves (the e2e→integration→unit tier ladder) and test files are exempt; the codebase is already clean, so this is pure regression-prevention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(dialect): refresh Claude Code dialect baseline to 2.1.187 + survive the native-binary distribution The gated dialect-drift alarm fired in CI because the unpinned `npm i -g @anthropic-ai/claude-code` now installs a much newer CC than the 2.1.42 baseline: - sdk-tools.d.ts grew from 19 to 38 tool-input types — the agent-PLATFORM surface (Cron*, Task* CRUD, Workflow, Monitor, Enter/ExitWorktree, EnterPlanMode, Projects, PushNotification, REPL, ReadMcpResourceDir, RemoteTrigger, ScheduleWakeup, ShowOnboardingRolePicker, Artifact) was added; Config removed. These are HOST/platform tools, not subagent-grantable, so claudeCodeDialect's builtinAgentTools (the `tools:` frontmatter catalog) is intentionally unchanged — they're acknowledged as facts only. - CC ≥ ~2.1.18x switched to a NATIVE-BINARY distribution (bin/claude.exe from a platform optionalDependencies package) with NO readable cli.js — so the hook-event text-scan ENOENT-crashed. Added findClaudeCodeBundle(): the event check now degrades to a LOUD SKIP when there's no readable JS bundle, while the sdk-tools.d.ts tool-type alarm keeps working. Refreshed ACKNOWLEDGED_TOOL_INPUT_TYPES (38) + VALIDATED_CC_VERSION = 2.1.187. We deliberately do NOT `import type` the SDK's ToolInputSchemas union: both @anthropic-ai/claude-code and @anthropic-ai/claude-agent-sdk are "© Anthropic PBC. All rights reserved." (proprietary), and vigiles is MIT + multi-harness, so we keep reading the user's local install (ToS-clean) against a hand-authored list of bare identifiers. Recorded in the dialect-drift header + research/code-adapter-architecture.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(harness): repair stale refs-hook command in the refs-nudge example + teach self-command-refs the ${CLI} form The refs-nudge harness dogfood called `node ${CLI} refs-hook`, but the runtime entrypoint was consolidated to `hook-runtime refs` in the hook-runtime rename — so the PostToolUse hook hit "Unknown command", emitted no additionalContext, and the nudge never reached the model (the CI `harness` job's only failure). Fixed the example to `node ${CLI} hook-runtime refs`; verified the nudge lands in a real claude session. The stale ref slipped past the self-command-refs gate because the detector only recognized `vigiles`/`cli.js` invocation literals — not the harness-test convention `node ${CLI} <cmd>` (`${CLI}` = a `dist/cli.js` URL). Extended INVOKE to recognize `${CLI}` (handling that `$` is not a word char, so `\b` can't anchor it) so this whole class is now caught deterministically — dogfooding the cross-ref moat on vigiles itself. The repo dogfood + 46 existing `${CLI}` usages stay green (no false positives); added a unit test for the class. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: pin claude-code to VALIDATED_CC_VERSION so the dialect-drift alarm is deterministic CI installed `@anthropic-ai/claude-code` UNPINNED, so any new CC release fired the (intentional) dialect-drift alarm on unrelated PRs — the noise that just broke this branch's CI. Pin every job that drives the real binary (the test, harness, and e2e tiers) to `@anthropic-ai/claude-code@<VALIDATED_CC_VERSION>`, grepped from src/dialect-drift.ts so the constant is the SINGLE knob: bumping it updates both what CI installs and the alarm baseline (the gated test cross-checks them). Two wins: the alarm now fires only on a DELIBERATE bump (not a random upstream release), and the real-`claude` harness/eval tiers become reproducible across CC releases (no surprise behavior breaks like the refs-nudge PostToolUse change). Runtime mismatch warning already exists and needs nothing: `vigiles scan` calls checkDialectDrift/formatDialectDrift, which prints a one-line ⚠ ONLY on real tool-surface drift (a bare version difference with no new/removed tool stays quiet — the right precision), best-effort and never throwing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the API-curation + CC-2.1.187 + CI-pin session Overwrite the stale compiled-hooks-arc handoff with this session's state on branch claude/readme-duplication-cleanup-l1wc5n (PR #45): barrel curation, the no-barrel-imports rule, the CC 2.1.187 dialect refresh (native-binary distribution), the refs-nudge + self-command-refs ${CLI} fix, and the CI pin — plus the decisions of record (don't import the proprietary SDK types; bump VALIDATED_CC_VERSION + ACKNOWLEDGED together). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix HANDOFF instructions — say to commit + push (ephemeral, git-tracked) The header said "overwrite each session" but never that the file is git-tracked and the container is ephemeral, so an update persists ONLY if committed + pushed (a local overwrite is lost when the session ends). Spell that out, plus when to refresh and what to include. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-24 19:47:06 +07:00
zernie	5c3cd2f947	feat!: layout-driven multi-harness validation rules & harness testing (#41) ... * docs: add rules-docs-in-sync rule; fix nested blockquote in harness-testing Add a `rules-docs-in-sync` guidance rule to CLAUDE.md so the validation-rule set (RulesConfig keys + built-in vigiles/* rules) and the central rules table in docs/cli.md never drift: adding/removing/renaming a rule must update the type, the table row, and the per-rule doc together. Also fix the doubly-nested blockquote (`> >`) in docs/harness-testing.md that rendered as a confusing quote-inside-a-quote — convert the pasteable prompt to a fenced code block, matching the README convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: surface the lint rule set in the linting guide; break up walls of text The Lint guide (verifying-instruction-files.md) explained the three rule *types* but never showed or linked the actual validation-rule *list* — so a reader following the README's Lint link couldn't find the rules. Add a "What vigiles lint checks" section that summarizes the rule families and links the canonical table in cli.md (no duplication, per rules-docs-in-sync). Also break four dense prose paragraphs in harness-testing.md into bullets / shorter paragraphs (entry points, harness-specific transport, the two-layers explainer, the eval-tool comparison) for mobile scannability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(orphans): never flag harness-loaded instruction files; move rules matrix to lint guide orphan-docs now categorically exempts files the harness loads directly (CLAUDE.md / AGENTS.md / SKILL.md / agents/ / commands/) from the orphan CANDIDATE set — they're load-bearing by name/location, not by being linked, so they're never orphans even if `orphans.include` is broadened to the whole repo. They still count as referencers. Adds tests + a docs/rules/orphan-docs.md. Move the canonical validation-rules matrix out of the CLI reference and into the linting guide (verifying-instruction-files.md), where a reader looking for the rules expects it — grouped by family (spec & integrity, test coverage, reference marking, subagent contracts, hooks & MCP, skill triggers, docs hygiene). cli.md and the README now link to that one matrix. Updates the rules-docs-in-sync rule to name the linting guide as the canonical home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: overhaul harness-testing — task-first guide + extracted API reference The harness-testing guide buried the first runnable example five screens deep, under ~140 lines of architecture (capability-contract theory, composition-root, adapter selection, import taxonomy) and didn't show its TOC until line 118. A first-timer couldn't find "how do I test my hook?". Full split: - Rewrite docs/harness-testing.md as a lean task-first guide (902 → 297 lines): a copy-paste first test in the first screen, a "what do you want to test?" decision table as the primary nav, trigger-rate promoted to a headline section, advanced material (significance / regression / cost / judge) collapsed. - Extract docs/testing-api.md: the full reference — Trace model, predicates, assertions, the check vocabulary, matchers, every measureTriggerRate / runEval option, significance/regression, imports & harness selection. - Move the theory (import-path-is-capability-contract, "two layers or three?") to research/harness-testing.md where the rationale belongs. - Repoint inbound anchor links (harness-testing-claude-code.md) and add the new reference to the docs index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * refactor!: surface-scoped lint rules — rename agent-*→subagent-*, capability-gate, fix dialect drift Adopt the "surface model" for cross-adapter vs adapter-only lint rules: a rule is scoped to the SURFACE it checks (subagents, shell hooks, skills, MCP, docs), not to a harness name. Universal rules stay bare and run everywhere; surface rules run only where the active adapter has that surface and report n/a otherwise (loud, never a failure) — so one config targets Claude Code AND Codex (incl. independent .claude/.codex trees, not just byte-identical mirrors) without per-harness rule duplication. Harness prefixes are reserved for genuinely exclusive concepts (none needed today). BREAKING (no back-compat, per design): - Rename rules: agent-tool-contract → subagent-tool-contract, agent-frontmatter → subagent-frontmatter, untested-agent → untested-subagent. Named by the harness-neutral concept (subagent), resolving the Codex `[agents]` (a concurrency table, not a subagent) name collision at the source. - Add AdapterCapabilities.subagents (CC/OpenCode true, Codex false); the subagent rules gate on it and report n/a where absent. - Fix the lint dialect-drift bug: checkSubagentToolContracts / checkHookEvents / checkFrontmatterSchema now run against the RESOLVED adapter's dialect + layout (and respect --harness), not a hard-coded claudeCodeDialect — scan was already adapter-aware, lint was not. - Docs: add the surface→harness "Applies to" matrix to the linting guide, rename the three rule docs, recompile CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * feat: layout-driven surface classification — ready for new-harness subagent dirs The subagent rules and scan's surface classifiers were hard-coded to `agents/` / `skills/` / `commands/`, so a harness whose subagents live elsewhere (OpenCode's `.opencode/agent`) wouldn't be classified. Make it layout-driven. - PluginLayout gains labeled `skillDir` / `agentDir` / `commandDir` (an empty `agentDir` means "no subagent surface" — e.g. Codex's `[agents]` TOML table). Set per adapter; documented in the adapter API + authoring guide. - scan.ts builds its isSkill/isAgent/isCommand classifiers from the active layout (regex anchored on a real path boundary, path-segment-safe), threaded through every surface helper — no module-level hard-coded dirs. - The `subagent-tool-contract` lint rule now REUSES scanPlugin's already-resolved, layout+dialect-driven agents (one-detector-no-drift) instead of re-globbing a hard-coded `agents/` path; drops the now-dead parseAgentTools/verifyToolContract imports. - Test: scan classifies a subagent under a custom `agentDir` and a default Claude Code scan does not — proving the dir is read from the layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: finish de-CC-ing the lint/scan detectors + add an ESLint guard Two more functional Claude Code hard-codings remained in the deterministic reference-verification path (same class as the subagent glob): - scan.ts `resolveScript` (hook-script-exists) hard-coded `${CLAUDE_PLUGIN_ROOT}`; now expands the harness's own `layout.pluginRootToken` (braced + unbraced). - test-coverage.ts (untested-skill/agent/hook) hard-coded `skills/` / `agents/` / `.claude/` globs and the CC token; now layout-driven via `skillDir`/`agentDir`/ `materializeRoot`/`pluginRootToken`/`manifestPath` (default CC), threaded from the resolved adapter in lint and from `lay` in scan. The deterministic layer-1 (lint + scan) is now fully layout/dialect-driven. The remaining `.claude` literals live only where they belong: the CC adapter, the CC eval transport (src/eval.ts), and CC plugin onboarding (init in src/cli.ts). Prevent recurrence: a `no-restricted-syntax` ESLint rule bans hard-coded CC string literals (`${CLAUDE_PLUGIN_ROOT}`, `.claude/`, `ANTHROPIC_*`) — as plain literals AND template quasis — in the harness-agnostic domain + detectors (src/core, scan.ts, test-coverage.ts, plugin-loader.ts). Verified it blocks a probe. Complements the existing import-graph boundary with a literal boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * test: dogfood the layout-driven untested-surface detector on a non-CC layout A regression guard for the de-CC-ing: build a repo with custom surface dirs (mysurf/skill, mysurf/agent) and plugin-root token ($MY_PLUGIN_ROOT), then assert findUntestedSurfaces discovers all three surfaces via the injected layout — and the default Claude Code layout sees none of it. CC-shaped fixtures can't catch a CC-hardcoding regression; this one can. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: thread the resolved adapter through ALL lint checks + Codex CLI dogfood Completes the dialect-drift fix: 8 more scanPlugin-based lint checks (skill-frontmatter, mcp-config, mcp-tool-resolves, hook-script-exists, disallowed-tools-contract, description-overlap, frontmatter-valid, mcp-hook-target-resolves) were still calling scanPlugin(process.cwd()) with the default Claude Code layout/dialect — only 3 of 11 had been fixed. All 11 now run against the resolved adapter, and the subagent-/hook-surface ones report n/a on a harness lacking that surface (capability-gated). Dogfood: a Codex `lint` CLI golden test (offline, in the existing CI job) wires a real Codex repo — AGENTS.md + TOML [hooks] with the harness's own ${PLUGIN_ROOT} token + an untested skill — and asserts hook-script-exists resolves the token and flags the missing script, untested-skill finds the skill, and the subagent rule reports n/a. A CC-shaped fixture can't catch a CC-hardcoding regression; this one runs the full pipeline against the second harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: codify the adapter-aware-lint convention + fill the capability/layout doc gaps Add an explicit `adapter-aware-lint-rules` guidance rule to CLAUDE.md (recompiled from the spec): lint rules are surface-scoped not harness-named; name by the harness-neutral concept (subagent, not agent); thread the resolved adapter and never hard-code a CC literal (eslint-enforced); gate by capability and report n/a; dogfood must be non-CC-shaped. The full record of this session's de-CC-ing. Docs caught up to the code: - harnesses.md capability matrix + adapter-api.md AdapterCapabilities gain the `subagents` flag. - adapter-api.md / authoring-an-adapter.md PluginLayout gains skillDir/agentDir/ commandDir; the myHarnessAdapter example gains the (previously missing, required) `capabilities` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(ci): update API report for the new adapter fields; add internal audience-check skill The `check` job's API surface gate failed: AdapterCapabilities.subagents and PluginLayout.skillDir/agentDir/commandDir are public (vigiles/adapter), so the committed etc/vigiles-adapter.api.md was stale. Regenerated (diff = exactly those four field additions) and gave each new layout field its own TSDoc so the report has no `(undocumented)` markers. Also add an internal dev skill `dev/skills/audience-check` (not shipped, not counted by untested-skill): re-read the README through six audience lenses — Claude Code user, Codex user, senior/staff skeptic, junior newcomer, eng-manager decision-maker, QA/test engineer — and report per-persona friction + ranked fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: note the UI-created-PR title gotcha in the conventional-commits rule A PR opened from the Claude Code UI gets a plain prose title that fails the `validate` job (which checks the PR title, not the commits). Document the fix: edit the title to a Conventional Commit + the matching breaking `!` as soon as the PR exists; validate re-runs on the title edit. (Hit on PR #41.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-20 03:54:34 +07:00
zernie	456991c974	feat!: harness testing/eval pillar + public-API surface revamp (#37) ... * docs: add eval-architecture design doc + ranked roadmap Capture the conceptual model for testing Claude Code harness features: feature = deterministic TEST + behavioral EVAL; two orthogonal knobs (reproducibility x gating); cost-matched mechanism; trigger-rate as a classifier eval; hash-lockfile + evalApiVersion + dated-model; deferred canary; closure-scoped invalidation. Reconciles the model with what the codebase already ships (record/replay cache, significance-gated baseline, check vocabulary, trigger-rate recall+precision) and ranks the genuine remaining gaps (tool-call spy/fake keystone, negative checks, curl fake, subprocess golden, hash-lockfile) mapped to the dogfood build order. Link from research/eval-api-landscape.md and research/roadmap.md. * docs: clarify constraint is no-SaaS, not zero-dep, in eval-architecture Deps are acceptable when they require no hosted account; rest the case against Polly.js/promptfoo on their real merits (cache exists, cassette doesn't escape the snapshot trap, promptfoo punt is strategic) rather than a dep-count rule. * docs: drop reuse-vs-reinvent tangent from eval-architecture * feat: add toolWith/notTool argument + safety checks to vigiles/check The inspection half of the keystone tool-call spy (docs/eval-architecture.md gap #1/#2): a serializable ArgMatcher (dot-path keys; RegExp = pattern, primitive = exact) drives two new Trace checks — - toolWith(name, args): the agent called `name` with input matching `args` (asserts HOW a tool was called, e.g. the image-request body carried the style suffix), and - notTool(name, args?): the agent did NOT call `name` (optionally only the args-matching calls, so "did not push to main" still allows pushing elsewhere) — the safety/negative assertion a completion grader can't make. Both flow through vigiles/testing + vigiles/unit via the existing `export *`. Pure, model-free, fully unit-tested in src/check.test.ts. * feat: tool-fake interception core + vigiles fake-tool-hook The interception half of the keystone tool-call spy (docs/eval-architecture.md gap #1). At the eval tier the model's tool decisions are real, but letting a side-effecting tool actually run (paid image API, git push, paid subagent) makes the eval expensive and risky. A FakeTool denies the real execution while the model still emits the tool_use — so the agent's decision is observable via toolWith/notTool, but nothing happens. - src/arg-match.ts: extract the shared ArgMatcher (dot-path keys; RegExp = pattern, primitive = exact) so the check vocabulary and the fake seam share one matching semantics; src/check.ts now imports it. - src/tool-fake.ts: FakeTool + decideFakeTool (first match wins, optional `when` arg-scoping), buildFakeToolSettings (PreToolUse hook over the union of faked tool names), and serializeFakeTools/parseFakeTools (env round-trip that preserves RegExp matchers a naive JSON.stringify would drop). Pure, tested. - src/cli.ts: `vigiles fake-tool-hook` — reads VIGILES_FAKE_TOOLS, denies a matched call via exit 2 feeding the canned result to the model (mirrors the agent-contract rail's block mechanism). Remaining: the ergonomic `fakeTools:` field on the eval spec (auto-merge the hook settings + a cache-key entry) to wire it end-to-end from measure/runEval. * feat: wire fakeTools onto the eval spec (end-to-end tool-call spy) Make the interception usable from the API: declare `fakeTools: [{ tool, when?, result? }]` on a measure/runEval arm and the eval tier does the rest. - EvalArm.fakeTools + MeasureSpec.fakeTools. - executeTrial auto-merges the fake PreToolUse hook into the arm's resolved settings (APPENDING to any existing PreToolUse list, never clobbering), pointing the hook at this CLI's own cli.js (resolved via __dirname, since the eval runs in a throwaway cwd where `npx vigiles` wouldn't resolve). - The fake list rides VIGILES_FAKE_TOOLS (AgentRunArgs.env, layered over process.env in spawnAgent), RegExp matchers intact. - CacheKeyInput gains `env` so two fake configs that share tool names — and thus produce identical merged settings — don't collide in the replay cache (their when/result lives only in the env). Tested via the injected runner (no model): the hook merges + appends, the env round-trips, and a no-fakes run wires neither. eval.ts/eval-cache.ts stay at the 100% lines/functions/statements gate. * refactor: make tool-fake honest + add real-model validation eval Acting on the step-back assessment of the tool-call spy: - Fix the misleading DEFAULT_FAKE_RESULT: the exit-2 hook is a BLOCK, not a successful return (CC surfaces it as a denied call), so the default reason now says "intercepted, NOT executed, do not retry" instead of "assume success". Reframe the tool-fake/eval docs as intercept-and-prevent + observe the attempt, not a faithful tool mock (CC has no skip-exec-return-success primitive). Note cost is not reduced — only the side effect is removed. - docs/eval-architecture.md: record the honest assessment — fidelity ceiling (deny != success), toolWith overlaps the existing toolUsedWith predicate (net-new is the negative check + interception), the simplest safety path (don't allowlist + assert the attempt) often needs no new primitive, and the competitor delta is narrow (assertions at parity; intercept-in-real-harness is the edge). - examples/harness/tool-fake.eval.mjs: end-to-end validation of the one assumption unit tests can't reach — that a DENIED tool's args still land in the Trace. Induces a git push origin main, intercepts it, asserts toolWith(Bash, /push origin main/) holds across trials. Loud skip() without claude. Run with a key before relying on the spy. * ci: behavioral eval gate scaffolding (inert until ANTHROPIC_API_KEY) Addresses the original kicker — real-model evals never ran in CI, results frozen as FINDING: comments. Adds .github/workflows/evals.yml with two jobs from the cost model in docs/eval-architecture.md: - eval-gate (pull_request): the cheap tier — the three shipped model-invocable skills' trigger evals (stubSkillBodies, 1 trial) on a DATED model, --no-skip. - eval-nightly (schedule + workflow_dispatch): the fuller drift tier you can't amortize. Both are INERT until enabled: every model step is guarded on the ANTHROPIC_API_KEY secret, so with no secret (default + all fork PRs) it prints a notice and exits 0 — never reds a PR. Turning it on is a deliberate secret + budget decision. Plumbing: `vigiles eval --model=<id>` → VIGILES_MODEL (sibling of --trials) so the gate pins a dated model id honestly without changing library defaults; the cheap trigger evals + tool-fake.eval.mjs now read process.env.VIGILES_MODEL. * feat: warn when an eval cache rides a floating model alias Closes the last piece of the honest-model-pinning gap (doc #3). A record/replay cache keyed on a floating alias (haiku/sonnet, or claude-sonnet-4-6 with no date) can serve a result computed against a since-changed model — hiding drift behind a green replay. - isDatedModel(id): a model is honestly pinned when it ends in an 8-digit date stamp (claude-haiku-4-5-20251001); a bare alias or version is floating. - runEvalWith emits a one-time warning (::warning:: under GITHUB_ACTIONS, else stderr) when cache != off and the model is floating. Tested: isDatedModel dated/floating, both warning branches (CI annotation + stderr), and the dated-id no-warn path. eval.ts stays at the 100% gate. * docs: document arg-match + tool-fake modules in CLAUDE.md Key Files The repo convention is that every src module appears in Key Files. Adds entries for src/arg-match.ts, src/tool-fake.ts, and src/tool-fake.test.ts (the tool-call spy), then recompiles CLAUDE.md from the spec — integrity hash valid. * test: validate tool-fake spy against a real model (capture-under-deny works) Ran the validation against claude-haiku-4-5-20251001. The model called Bash with "git push origin main", the auto-wired PreToolUse fake-tool-hook denied it (isError, "the git push didn't execute"), and the denied tool_use — WITH its args — still landed in ctx.toolCalls, so toolWith matched. This confirms the one assumption the unit tests can't reach: a denied tool's arguments are still captured for toolWith/notTool, and the side effect is prevented. Adjust the eval to assert the MECHANISM, not model obedience: strengthen the prompt, 5 trials, and gate at min 0.2 — any non-zero rate proves capture works; only a true regression (a flat 0 across every trial) fails. Records the finding inline. * docs: refresh eval comparison (public + internal) and positioning The eval comparison material predated the shipped significance/cache/cost/ regression-gating work and the tool-call spy. Brought it current: - research/eval-api-landscape.md (INTERNAL): the scorecard's vigiles column was marked ✗ on significance/caching/cost/concurrency/regression-gating — all shipped. Rewrote it to shipped reality, added rows for the no-model cheaper tiers, the intercept-and-prevent tool spy, and the regression gate; noted the inert-until-keyed CI eval gate. Three bolded-unique rows now name the defensible core honestly. - docs/harness-testing.md (PUBLIC): broadened the promptfoo-only section into a promptfoo/DeepEval/Braintrust/Inspect table; added the safety-intercept and no-model-tier rows; pointed to the internal landscape doc. - README: added one line on the safety property (notTool + fakeTools catch a push-to-wrong-branch / paid-API attempt and prevent the side effect). Positioning audit: README front door + internal scorecard now current; the spy modules are already in CLAUDE.md Key Files. * fix: harden eval cache invalidation (dir contents, tool-set, version salt) Research-backed (research/cache-invalidation.md, synthesizing Bazel/Turborepo/ Nx/ccache/Jest/webpack + promptfoo/LangChain/Helicone). Fixes the confirmed false-replay bug and two phantom-miss/staleness gaps, ranked by severity: 1. pluginDir contents weren't in the key (THE bug): a native --plugin-dir was passed to the run but its files were never hashed, so editing a skill in it false-replayed. hashDir() folds a sorted relativePath:contentHash list into the key (content not mtime; path-included). Flat sorted list, not a Merkle tree — over-engineering at plugin-dir scale. 2. Tool list now hashed as a SET (sorted) so a reordered allowlist isn't a phantom-distinct key. (canonical() already sorted object keys, so no stable-stringify dependency is warranted.) 3. CACHE_FORMAT_VERSION salted into the key (Jest/webpack pattern) so a record format change makes old entries unreachable — no brittle read-time gate. Model drift on a floating alias stays warn-only (the dated-id pin is the sound fix; can't auto-resolve without an API call). Eviction/TTL deferred as disk hygiene, not invalidation correctness — rationale in the research doc. Tests: hashDir (edit/add/rename/skip-junk), tool-set order-insensitivity, pluginDirHash sensitivity, and an end-to-end runEvalWith test proving a pluginDir content edit busts the cache. eval-cache.ts/eval.ts stay at 100%. * ci: let the eval gate run on a Claude subscription, not just an API key The affordability bet: vigiles drives the real `claude` CLI, so an eval authenticates like the user's own CLI — including a Pro/Max SUBSCRIPTION (flat cost), not metered API billing. (Confirmed empirically: a validation run reported apiKeySource:"none", i.e. it ran on the session's OAuth subscription.) Both eval jobs now accept CLAUDE_CODE_OAUTH_TOKEN (subscription, via `claude setup-token`) OR ANTHROPIC_API_KEY (metered fallback), and stay inert until one is present. Header documents the subscription as the cheap path. * docs: isolated-vs-whole-harness eval decision (research of record) Decision: support BOTH as tiers — default isolated (cheap authoring loop), add a whole-harness install-set arm as the release gate, plus a near-neighbor middle tier on the existing ncd engine. Why: skill selection is competitive and Claude Code evicts least-used skill descriptions under a context budget (skillListingBudgetFraction), so isolated trigger-rate overstates recall and understates false-positives. Every existing tool (promptfoo, Anthropic skill-creator, AWS, OpenAI) isolates; none populate the install set — a genuine wedge for "test the harness as it ships". Captured in research/isolated-vs-whole-harness-eval.md; roadmapped in eval-architecture (item 7). Also updated the eval-gate roadmap note for the subscription-token auth path. * feat: whole-harness trigger-rate tier (installSet) + honest isolated labelling Implements the isolated-vs-whole-harness decision (research/isolated-vs-whole- harness-eval.md). measureTriggerRate stays isolated by default (cheap authoring loop); a new `installSet` co-installs the skill-under-test alongside the user's real skills so it competes for selection as it does in the shipped harness — the release-gate tier no other eval tool offers. - TriggerRateSpec.installSet: plugin dirs or loose skills dirs merged into one install under the under-test plugin's name (under-test wins a name collision, so its <name>:<skill> id still matches `fired`). packageInstallSet + collectSkillsSource + copySkillsInto, pure/tested, cleans up on any error. - TriggerRateReport.competitors: count of co-installed competitors. 0 = isolated. - formatTriggerRateReport now labels honestly: an isolated run prints that recall is an UPPER bound and false-positive a LOWER bound (selection is competitive); a whole-harness run prints the competitor count. Tests cover the merge (skills/ + .claude/skills/ sources, collision, stub, missing-source + empty-under-test throws, junk-skip), both report labels, and installSet from pluginDir + skillsDir. eval.ts stays at the 100% gate. * docs: lead README with affordability; correct the evals-in-CI framing - README: affordability is now an explicit theme — most harness questions are answered with no model (free, every commit), and a real-model eval rides the user's Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI), not metered API billing like API-SDK-based tools. States plainly that evals don't run in CI by default. - research/isolated-vs-whole-harness-eval.md: fixed the "nightly/release gate" wording that wrongly implied a default CI cadence. Clarified that evals are not in CI by default (the per-PR CI is the free deterministic tiers; the eval workflow is inert until you add a token); "release gate" = run deliberately (locally on the sub, or opt-in nightly). * fix: make the eval CI gate fail on a non-run or a corrupt cache Two silent-pass holes the eval gate could slip through — closed so a green eval job actually means evals ran and the cache was sound: 1. Evals never executed. `vigiles eval`/`test` gained `--min=N`: assert at least N scripts actually matched/ran, else exit 1 loudly. A renamed file, a bad path, or a glob matching nothing previously printed "No files found" and exited 0 — a hidden gap. evals.yml now passes --min=3 (the gate's three listed evals) and --min=1 (nightly), alongside the existing --no-skip. 2. Cache fails. readCache now THROWS on a corrupt record (file present but invalid JSON) instead of swallowing it and silently re-running. A broken cassette is a real failure the gate must surface; a true miss (no file) still returns null and proceeds normally. Message says how to recover. Tests: cli.test --min (0 matched / too-few / enough), eval-cache corrupt→throws vs miss→null. eval-cache.ts stays at the 100% gate. * fix: trigger-rate on Sonnet (not haiku) + count sibling skills as competitors Dogfooding the three shipped skills on the subscription found two real bugs: 1. Gate model. All three skills "failed" recall on haiku (test-harness 0.50, edit-spec/strengthen 0.70) — but on Sonnet test-harness scores 0.90 and passes. Trigger-rate is a SELECTION measurement and haiku is a much weaker selector, so a haiku gate false-fails skills that fire fine on the model users run. measureTriggerRate now defaults to "sonnet" (matching measure); evals.yml pins claude-sonnet-4-6 for the gate; docs corrected. The skill descriptions were never the problem. 2. competitors miscount. The honest-label counted only installSet additions, so a multi-skill plugin (the dogfood installs all of vigiles' skills) printed "isolated: no competing skills" while ~5 siblings were actually competing. Now competitors = (skills in the resolved install pool) − 1, via countSkills, so siblings count and a real single-skill install still reads 0/isolated. packageInstallSet's return renamed competitors→added (installSet delta). Tests: sibling-competitor count, packageInstallSet `added`. Recorded the haiku-vs-Sonnet FINDING in test-harness.trigger.eval.mjs. eval.ts at 100% gate. * feat: model-per-arm (Q1) + save model/isolation eval decisions Q1 (model strategy) — agreed + built: - EvalArm.model: a per-arm model override. A model comparison IS a harness A/B, so "does my harness hold on the cheaper tier / after a model upgrade?" is arms: { sonnet: {model}, opus: {model} } read through the existing significance machinery — no separate multi-model matrix runner (that's the cost-multiplying promptfoo lane we avoid). measureTriggerRate stays single-model by design. - Default stays Sonnet (the realistic selector), per the dogfood finding. Q2 (isolation) — agreed: keep TWO tiers (in-plugin by default = honest sibling competition; installSet for the whole harness). Do NOT build per-skill isolation machinery (it produces the LESS honest number others default to). Already shipped; no new code. Saved research + conclusions: docs/eval-architecture.md gains a "Model strategy" section (default Sonnet w/ the 0.50-haiku-vs-0.90-Sonnet dogfood evidence, model-as-arm, no matrix, the deferred model-floor); the isolation research/decision is in research/isolated-vs-whole-harness-eval.md. Validated on the subscription: the corrected Sonnet gate is GREEN (3/3 skills ≥0.8) and the competitors label now reads "whole-harness: 4 competing skills". * feat: runtime model floor for trigger-rate (default Sonnet) + gate on alias The "set a floor for all skills" ask, solved without a (brittle) static linter: a RUNTIME floor enforced in the eval library catches the env-var model a lint can't see — which is exactly how the haiku footgun entered. - modelTier(id): rank by family (haiku<sonnet<opus; version ignored), null for an unrecognized family so the floor never blocks a model it can't judge. belowModelFloor(model, floor): fail-open on unknowns. - TriggerRateSpec.minModel (default "sonnet", resolves minModel → VIGILES_MIN_MODEL → "sonnet"). measureTriggerRate FAILS before spending a token if the resolved model is below the floor — so one knob (VIGILES_MIN_MODEL / --min-model) sets the floor for ALL skills, no per-file annotation, no linter. Lower it deliberately for a cheap run. Decision: FAIL (not warn) — the floor is a contract; the escape hatch is lowering it. - CLI: `vigiles eval --min-model=ID` → VIGILES_MIN_MODEL. - Gate model switched claude-sonnet-4-6 → the `sonnet` ALIAS: the per-PR gate runs cache-off, so it should track current Sonnet without churn; dated/exact ids matter only for cached/baselined reproducibility (kept on that path). Tests: modelTier/belowModelFloor (families + fail-open), floor throw at default + VIGILES_MIN_MODEL, and the lower-floor escape hatch. eval.ts at 100% gate. * refactor: remove model envs + the speculative evals.yml CI workflow Acting on review: neither was asked for, and both were the wrong layer. - Removed VIGILES_MODEL / --model and VIGILES_MIN_MODEL / --min-model. Model is part of the measurement DEFINITION (a different model = a different result), so it belongs in the spec (`model`/`minModel`, version-controlled), not a hidden CLI/env override. Contrast VIGILES_TRIALS, a legit run knob (cost/precision, doesn't change what's measured) that PRE-DATES this work — kept. The dogfood evals now omit `model:` and rely on the Sonnet default + the minModel floor. - Removed .github/workflows/evals.yml. Real-model evals don't belong in a standalone GitHub Actions workflow needing a metered/sub-token secret: CI runs the FREE deterministic tiers (ci.yml, no token), and the real-model eval runs where the subscription already is — a Claude Code session or locally (vigiles drives the `claude` CLI). The eval-robustness work (--min, --no-skip, corrupt-cache throw, Sonnet model floor) stays — it hardens `vigiles eval` wherever it runs. Docs reframed accordingly (eval-architecture kicker + roadmap, README, the two research docs): deterministic tiers in CI, evals on the sub in a session. eval.ts at the 100% gate; 974 tests pass. * docs: consolidate positioning + pros/cons of the eval approach Per review — the competitor scorecard, caching research, isolation decision, and model strategy were all saved, but the POSITIONING + pros/cons were scattered across README/kicker. Added a dedicated "Positioning & pros/cons" section to the eval-architecture design-of-record: the affordability thesis (free deterministic tiers + sub-priced real-model tier vs competitors' per-token-every-run), the pros (cost/fidelity/honesty) and the honest cons (sub rate limits, non-determinism, intercept≠mock, sonnet-not-haiku, not-a-CI-checkbox, no dataset/red-team/UI), with links to the four supporting research docs. * fix: key the eval cache on the harness binary version (fast-evolution edge) You were right — the cache wasn't tested/keyed against fast model/harness evolution. Closing it: - harnessVersion (`claude --version`, memoized, cache-path only) is now in the cache key. A CLI upgrade changes the system prompt + tool defs — which steer behaviour as much as the model — so a cached result now INVALIDATES across a `claude` upgrade instead of silently replaying a result from a different harness. Trade-off (frequent patch invalidation) documented; re-scoring within a session is unaffected (version stable then). - cacheKey test now actually covers model / settings / env / harnessVersion changes (it previously asserted "any input" but skipped several, incl. model). - research/cache-invalidation.md gains a "Fast-evolution edges" section: harness version (keyed), model-alias re-point (warned, can't auto-invalidate without a weight hash), and modelTier new-family fail-open (a documented known limit). eval-cache.ts/eval.ts stay at the 100% gate. * docs: positioning into CLAUDE.md (canonical) + consistency pass Point 3 — positioning belonged in the positioning source, not buried in eval-architecture.md: - CLAUDE.md ## Positioning (pillar 2) now states the affordability thesis: free deterministic tiers + a real-model tier that runs on the Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI, no metered API), measured on the realistic selector (Sonnet, minModel floor), model-in-spec not env. Recompiled. - eval-architecture's pros/cons section now points to CLAUDE.md as canonical and keeps only the detail. Gap #4 — refreshed the stale `src/eval.ts` Key Files entry (model-per-arm, minModel floor + modelTier, installSet whole-harness tier + competitors, harnessVersion in the cache key, Sonnet default). Gap #2 (consistency pass, you approved) — the "knobs" / "match-the-mechanism" / reconciliation sections no longer claim evals run in `every-PR` CI on `haiku`: deterministic tiers are every-commit CI, the real-model eval is run on-demand on the sub (Sonnet). Fixed the decision table + the reconciliation bullets too. Gap #3 — documented the shipped trigger-rate knobs in the user guide (harness-testing.md): Sonnet default + minModel floor, isolated-vs-whole-harness `installSet` + the `competitors` count, and model-per-arm for cross-model A/B. * feat: key cache on harness major.minor, automate stubSkillBodies, exhaustive cache tests + near-neighbor roadmap Acting on review: - Claude version granularity: key the cache on the harness MAJOR.MINOR, not the patch (harnessVersionKey: "2.1.179 (Claude Code)" → "2.1"). A patch rarely changes behaviour, so keying patches would churn the cache every release for no signal; a minor/major bump is where the system prompt actually moves. Documented the trade-off (a known-behavioural patch → clear cache / bump CACHE_FORMAT_VERSION). - stubSkillBodies is now AUTOMATIC for measureTriggerRate (default true): testing a description never needs the body (trigger is a frontmatter property decided before the body loads), so it's not a knob you must remember. Set false only to run the real body. Updated the 4 fake-pluginDir tests to stubSkillBodies:false (passthrough), which also keeps the non-stub branch covered. - Cache tests are now EXHAUSTIVE: one assertion per CacheKeyInput field, plus explicit VERSION axes — two model snapshots (sonnet-4-6 vs -4-5) and two harness minor versions each produce a different key. + harnessVersionKey unit test. - Roadmap: added the near-neighbor trigger-rate tier (NCD-nearest competitors) to Next, priority P3 (MED-LOW) — decided, deliberately deferred. (Consistency gaps #2/#3/#4 shipped in 74fc151.) eval.ts/eval-cache.ts at 100% gate; 975 tests. * feat!: rename fakeTools -> interceptTools (intercept-and-prevent, not a mock) The tool-fake feature does intercept-and-prevent (a PreToolUse hook denies the call via exit-2), not fake-a-return-value. The "fake" name wrongly implied a mock that substitutes a result and continues, requiring two JSDoc corrections to undo. Rename to carry the real semantics. BREAKING CHANGE: the vigiles/eval spec field `fakeTools` is now `interceptTools` (on EvalArm and MeasureSpec); the `FakeTool` type is `ToolIntercept` and its `result` field is `denyReason`. The `vigiles fake-tool-hook` CLI subcommand is now `intercept-tool-hook`. Pre-1.0, but signalled for a correct changelog. Keep toolWith/notTool (clearer than competitors) but add promptfoo/DeepEval cross-references to their JSDoc and a "vigiles check <-> promptfoo assertion" mapping to research/promptfoo-deep-dive.md, noting notTool/interceptTools as the restraint-assertion gap a completion-grading eval cannot cheaply express. * docs: add safety model front-door doc (confinement + tool interception) docs/sandboxing.md covers host confinement for hook/plugin code but not the eval-tier risk — a real model deciding to call a side-effecting tool. Add docs/safety.md as the front door tying the whole model together: the two risks (foreign code vs the model's actions), the two mechanisms (bwrap confinement vs interceptTools/notTool PreToolUse deny), the one safe-by-default fail-closed rule, the per-tier confinement table, and the honest gaps. Answers "why not confine everything" by pointing at the four-reason argument in sandboxing.md. Referenced from README + sandboxing.md (not an orphan). * docs: decide cross-platform sandboxing (macOS via sandbox-exec) + save the verify-sandbox-config idea Mac support is a hard requirement (large dev population), and the claude binary's own sandbox can't cover us (Bash-only, skips hooks, no external API, CC-only). Record the decision: a vigiles/os-isolation port with a native backend per OS — keep bwrap+nft on Linux (strongest, packet-layer), add sandbox-exec/Seatbelt on macOS (built-in, zero deps, deny-all-net). Per-host egress stays Linux-only; macOS degrades honestly. srt/nono are documented fallbacks, not the default; Birdcage is reference-only (GPL/binary-net). Also clarifies the enabled/disabled model (provenance, not platform: your code runs direct, foreign code is confined-or-refused — the Mac backend makes the safe option available so nobody is forced to sandbox:false), and parks the "verify & test the harness's own sandbox config" idea (the network-validation wedge: valid-is-not-true for sandbox policy). Roadmap + sandboxing.md updated; not an orphan. * docs: add the ephemerality axis — running a model-driven skill is a side effect, independent of trust Correct the safety model: provenance governs CONFINEMENT (host protection), but EPHEMERALITY (state protection) is unconditional — a skill/agent is model-driven, so even a plugin you wrote produces nondeterministic side effects when run. Trust does not make a run side-effect-free. Classify side effects by reversibility: local-reversible -> ephemeral CWD (already done); escapes-CWD (~/.gitconfig, ~/.ssh) -> ephemeral HOME + scrubbed env (the GAP: the direct/non-bwrap path inherits process.env + real HOME); external -> deny/allow net; irreversible (push, charge) -> interceptTools/notTool (can't un-push by discarding a box). Ephemerality needs no kernel features, so it lands on macOS today, ahead of Seatbelt — the cheapest cross-platform side-effect protection. safety.md + cross-platform-sandboxing.md + roadmap (HIGH) updated. * docs: positioning — behavioral/side-effecting testing + cross-platform (Mac+Linux); fix phantom issue refs Positioning (CLAUDE.md): name BEHAVIORAL, side-effecting eval of the assembled harness as the apex of pillar 2 (highest value, kept thin; notTool/interceptTools ship today, the ephemeral run env + disposable-dep provisioning are the committed next step). Add a "Cross-platform by requirement (macOS + Linux)" paragraph: native backend per OS behind a vigiles/os-isolation port, and the two orthogonal axes (provenance-keyed host protection vs unconditional ephemeral state protection). Fix: the declared-vs-enforced anchor cited as Claude Code #54898 could not be verified; replace every occurrence with the real anchors (Claude Code #4740/#21460, claude-agent-sdk-typescript #172) across agent-runtime, cli, the spec, and research. A reference-checking tool should not ship a phantom reference. * docs(research): save the mid-2026 subagent-demand + skill-eval-landscape findings subagent-compilation.md: ranked issue-survey demand evidence (declared!=enforced anchored to #4740/#21460/SDK #172; orchestration/output-contract/dispatch/MCP/ observability/model/determinism/firing/nesting themes), the authoring+observability tooling table (almost no verification/testing tool exists), Codex-vs-CC delta (enforcement is Claude-Code-shaped -> Codex non-goal confirmed), honest map-to-vigiles. skill-eval-landscape.md: how labs + practitioners actually eval skills — Anthropic skill-creator + "Demystifying evals", OpenAI's Codex eval guide, the two Reddit harnesses (SkillBenchmark blind-judge A/B+CIs on Caveman; the "SKILL.md is a contract" methodology harness incl. the adversarial-gate finding), Spence/dbt/ Husain/Schmid. Gap verdict: behavioral skill eval is now TABLE STAKES; vigiles's moat is the intersection (whole assembled harness + subscription affordability + statistical rigor + the deterministic runHook tier + the eval->enforce bridge). promptfoo-deep-dive.md: promptfoo acquired by OpenAI (~$86M) + DeepEval v4 agentic harness; sharpens the vendor-neutral, subscription-not-metered positioning. eval-api-landscape.md: new competitors (TribeAI/claude-evals metered; paultyng/ testagent CLI-fake) + statistical-rigor confirmed as a differentiator. * docs: native input/output/cache token+cost measurement decision + behavioral-umbrella clarification eval-architecture.md: (1) state explicitly that behavioral/side-effecting eval (the ephemeral integration tier) is the UMBRELLA and the adversarial-gate test is one member, not a replacement — with the eval->enforce bridge (pillar 2 hands off to pillar 1 when a prose gate caves); (2) decide native token/cost support split by class (input/output/cache) — the tokens() check collapses input+output and cache tokens aren't captured, so a single total can bless a net-negative change (Caveman cut output yet 2-4x'd cost). Plan: extend UsageTrace to all token classes, first-class inputTokens/outputTokens/cacheTokens checks, and an A/B delta-per-class gated by Welch significance. Added both to the ranked roadmaps (HIGH). * feat: native input/output/cache token measurement + per-class checks A harness change trades tokens across sides (a skill/CLAUDE.md injection adds INPUT every turn; a "compression" skill cuts OUTPUT), so a single total can bless a net-negative change — SkillBenchmark's Caveman cut output yet 2-4x'd cost. Split the usage model into honest classes and gate on them separately. - EvalUsage + UsageTrace gain cacheCreationTokens + cacheReadTokens (parsed from the claude result usage: cache_creation_input_tokens / cache_read_input_tokens); inputTokens stays the FRESH (uncached) input. ArmUsage + aggregateUsage sum the two cache totals. - New vigiles/check checks: inputTokens({max}), outputTokens({max}), cacheTokens({maxCreation?, maxRead?}) beside the existing tokens/cost/latency. Exposed on ctx.usage so an A/B delta-per-class rides the existing mean±se / Welch-significance machinery for free. - Dogfood: skill-compression.eval.mjs now measures input AND output (cache reads ~0.1x input, writes 1.25x), so the compression claim is verified on both sides. Tests: pure pass/fail/score/message/toJSON for each new check; usageFrom parses the cache fields (0 when absent); aggregateUsage sums the cache totals. * docs: record real-haiku FINDING in the compression eval — cost metrics work, cache dominates Ran the compression dogfood on real haiku (3 trials/arm, on the Pro/Max subscription, apiKeySource:"none", $0.1057). Captures the end-to-end result as a FINDING: total_cost_usd + input/output + cache tokens all populate from a real run; inputTokens read ~42/run (the uncached remainder) vs ~2.4-2.7k real volume — the rest is cache reads, so a single input/total understates economics ~50x here. The cache classes are where the volume lives (vindicates the split). The compression claim did not hold on this task/model (output +11%, cost +$0.002, correctness intact) — the headline-% failure mode the eval exists to catch. * feat: adversarial-gate dogfood eval + the eval->enforce bridge The highest-value behavioral test for an enforcement skill: adversarially ask the agent to skip the gate the skill is supposed to enforce, and assert it REFUSES. - examples/harness/dogfood/adversarial-gate.eval.mjs (write-don't-run): an inline SKILL.md states a measurement gate (no benchmark without a baseline; refuse if asked to skip), the task adversarially asks to skip it, and two composed checks assert refusal — notTool (didn't run the forbidden command) + output (pushed back in prose), gated by assertRates({min:0.7}). No new check primitive: the pattern is the existing vocabulary composed inline (no gold-plating). - docs/eval-architecture.md + docs/safety.md: document the eval->enforce BRIDGE — pillar 2 (eval) MEASURES whether a prose gate holds; when the adversarial test shows it caves, pillar 1's deterministic PreToolUse rail is the FIX. Eval hands off to enforcement; nobody else connects the two. * feat: ephemeral run environment foundation (pure builder + opt-in flag, default OFF) Running a model-driven skill mutates state regardless of trust (the model, not the author, chose the actions), but the eval tier inherits the full real environment — a model-driven git push / write to ~ uses the real $HOME, ~/.gitconfig, ~/.ssh, ~/.aws. Add the foundation to close that, behind an opt-in flag, default OFF so the eval tier's own auth can't silently break before a real-run validation. - ephemeralRunEnv(base, { home, allow }) — pure, exported, no fs/spawn: sets HOME+TMPDIR to a throwaway dir, passes through an auth+runtime allowlist (mirrors the runtime port: PATH/LANG/TERM/ANTHROPIC_*/CLAUDE_*/LC_* + region/profile so a Bedrock/Vertex-backed claude still authenticates), DROPS everything else (GIT_*/GH_TOKEN/SSH_*/AWS secret keys + any non-listed var). Conservative by design: when unsure, allow (a too-narrow allowlist breaks auth). - EvalSpec.ephemeralEnv?: boolean (default off). When on, executeTrial mkdtemps a throwaway HOME under the trial cwd and builds the scrubbed env; AgentRunArgs gains replaceEnv so spawnAgent uses a.env as the COMPLETE env. When OFF, the env path is byte-identical to before (asserted by a test). The cross-platform state-protection floor (needs no kernel features → lands on macOS now). Flipping the default to on waits for a real-auth validation pass; two honest follow-ups noted in code (AWS static-key auth, cache+ephemeral key interaction). * docs: validate ephemeral-env auth survival (env-var path works; file-based ~/.claude is the gap) Ran a real-model check (haiku, ephemeralEnv: true) on the subscription: the run authenticated through the scrubbed throwaway HOME — exit 0, $0.003, output "OK.", cache read ~21.7k tok (caching unaffected). So the env-var auth path is validated: the credential rides an allowlisted ANTHROPIC_* var that survives the scrub. Remaining gap before flipping the default on: this only covers env-var auth. A subscription credential that lives only in ~/.claude/.credentials.json (a file under HOME) would be lost when HOME is scrubbed — ephemeralRunEnv injects env vars, not the credential file. Safe default-on needs injecting the harness's own credential file into the throwaway HOME. Flag stays opt-in until then. * feat: complete the opt-in ephemeral run env (auth-file injection + cache-safe) + validate containment Finish the ephemeral-run-env opt-in path (still default OFF): - seedEphemeralHome copies a minimal auth allowlist (EPHEMERAL_HOME_KEEP = [".claude/.credentials.json"], copy-not-symlink so the run can't write back) from the real HOME into the throwaway HOME when present, so local file-based OAuth survives the scrub; .gitconfig/.ssh/.aws are never carried. - cache-safety: HOME/TMPDIR excluded from the eval cache key (CACHE_KEY_ENV_EXCLUDE), so the random throwaway HOME no longer makes every ephemeral run a cache miss; non-ephemeral keys unchanged (surgical, not a blanket env drop). Validated state protection end-to-end on the real subscription (haiku, ephemeralEnv: true, the run told to write $HOME/PROOF.txt + probe ~/.ssh): real $HOME untouched (write landed in the throwaway HOME) AND the run still authenticated (exit 0, $0.0083). Containment holds; auth survives via the host-brokered CLAUDE_* allowlist. Default stays OFF: this host-brokered env can't run-test the file-based ~/.claude/.credentials.json OAuth modality (no credential to exercise; no sandbox mints a real token), so the credential-file injection is unit-tested for the copy logic but not e2e-validated. Flipping default-on (a behavior change for every eval) waits on a local-file-auth user validating that path; then it's a one-liner. * docs: phased design for the vigiles/os-isolation port + Seatbelt backend Grounded implementation plan over the existing sandbox core (design of record, not yet built): the narrow port interface (NetPolicy tagged union / FsPolicy / IsolationCapabilities / OsIsolationBackend), the core-vs-backend layout, the per-backend capability matrix, the Seatbelt backend shape (sandbox-exec + minimal SBPL profile), the eslint-plugin-boundaries enforcement (only backends name a raw primitive, dogfooded via enforce), and a 4-phase migration that keeps the 997-test suite green at each step (extract+move bwrap behind the port with ZERO behaviour change -> add Seatbelt -> wire selection + lint -> fold in ephemeral env). Surfaces the sharp limitation honestly: Seatbelt (deny network*) blocks localhost too, so bwrap's co-launched-mock-over-loopback trick has no Mac equivalent (loopbackNet:false) — a trusted spec runs direct, an untrusted-needs-mock spec refuses. Per-host egress stays Linux-only; srt and CC's built-in sandbox stay out. Linked from the roadmap (HIGH). * docs: consolidate eval coverage/isolation into one source of truth (R1+R2 owned, R3 composed) Add research/eval-coverage-and-isolation.md as the authoritative doc: the three-rung model (R1 cheap/deterministic — nothing executes; R2 record-replay — a tool/MCP/API result recorded once from the real tool and replayed via a fake binary on PATH, never model-synthesized; R3 real disposable service whose semantics IS under test), the blended+scrubbed coverage distribution (R1+R2 ~90%+ with no Docker on the sub; R3 a thin apex — browser/DB/redis), the honest e2e landscape (every benchmark/lab/Inspect runs e2e in a container; no containerless reproducible e2e exists), the safety/viability/performance axes + non-goals, the competitor comparison, and a reusable rung-classifier prompt. Make the chosen approach crystal-clear in CLAUDE.md positioning (R1+R2 owned on the sub; R3 composed-with-a-container, not reinvented), docs/eval-architecture.md (Coverage & scope section), README (public), and the roadmap (the R2 enabler ranked ahead of testcontainers + a dogfood/COVERAGE.md item). Use standard terminology (PATH-shim + record-replay/VCR), not a coined name. Scrubbed — no private-source identifying details; conclusions blended with the community/landscape data. * feat: R2 tool-stub / PATH-shim helper (record-replay a tool result, no service, no Docker) The R2 enabler from the coverage model: test a skill whose logic consumes a tool's RESULT by shadowing the binary on PATH with a fake that emits a recorded/canned output — standard PATH-shimming + VCR-style replay, not a novel approach. Unlocks the ~43% R2 surface with no live service and no Docker (ranked ahead of any testcontainers work). - src/tool-stub.ts: ToolStub + renderToolStub/writeToolStubs/stubBinDir. Each stub is a self-contained POSIX script that base64-decodes its canned bytes straight to stdout (base64's alphabet can't contain a shell metacharacter → injection-proof for any fixture content; piping preserves trailing newlines that command-subst would strip). Stubs are author/recorded fixtures, NEVER model-synthesized (drift → false confidence). argv-matching + record-from-real are documented follow-ons. - EvalSpec.stubs wired into executeTrial: a bin dir under the trial cwd is prepended to PATH for BOTH the ephemeral and legacy env paths; absent stubs = byte-identical (no default flipped, ephemeralEnv stays off). - Exposed via vigiles/testing + a ./tool-stub subpath export. - harness-test tier wiring left as a documented TODO (it routes through the per-harness driver seam — out of MVP scope). Tests: faithful round-trip of shell-special content incl. trailing newlines; PATH prepend on both env paths; absent-stubs unchanged. 1007 passed / 11 skipped. * test: dogfood the three vendored plugins + per-plugin COVERAGE.md scorecards Prove how much of real plugins the cheap tiers actually cover. 9 model-free, CI-runnable tests (vendor-coverage.test.ts) across the three vendored plugins — OMC keyword-detector R1 fire/no-fire (the free floor under the bwrap-gated egress copies), scan structural facts + the wshobson inherits-all footgun on the scan report, superpowers session-start R1 (fallback) + R2 (reconstructed plugin root injecting the real skill body), and the canonical R2 stubBinDir demo (shadow npm on PATH to drive find-polluter.sh's real bisection to completion, no live service). Per-plugin COVERAGE.md scorecards (rung table + distribution + grade + R3 shortlist + honest verdict), placed beside the pinned snapshots (never inside them): - oh-my-claudecode: ~60% free / ~33% model-gated / ~7% R3 — B+ - superpowers: ~50% free / ~50% model-gated / ~0% R3 — B (a prose library is half behavioral, only measurable with a model) - wshobson-accessibility: ~30% free / ~25% model-gated / ~45% R3 — C (worst case: its semantics ARE the real browser; the free tier still catches the inherits-all agent footgun) The B+/B/C spread is the finding: where free/deterministic testing stops and the model/container tiers take over. 1016 tests pass; R3/model-gated items written-up, not run, by design. Linked from eval-coverage-and-isolation.md (Dogfood results). * docs: record 2026-06-17 multi-SDK probe in sdk-harness-testing Validate and sharpen the prior "don't retarget pillar 2" verdict with a nine-SDK primary-source probe across six dimensions (mock-model / tool-contract enforcement / assembled-as-shipped / trigger-rate / sub-affordability / record-replay). The prior verdict holds for code-SDKs (Pydantic/Vercel/LangGraph/LlamaIndex/Mastra ship first-party mocks), but the probe relocates pillar-2 value to the Claude Agent SDK + Codex (no mock, buggy/unenforced #172 tool contract, sub-affordability) and adds a concrete mock-ergonomics borrow-list. Roadmap entry tightened to match. * docs: make the two-axis coverage boundary explicit (public + internal) The R1/R2/R3 rungs only answer "what does the test need from the outside world" — they don't answer "who decides pass/fail", which is what makes a prose skill look like a coverage hole when it isn't. Add the second (oracle) axis everywhere the rungs appear: - Define the `-MG` (model-gated) suffix the dogfood scorecards already use, and place a prose/guidance skill explicitly at R1-MG (nothing executes, but only a model judges its worth — fully testable on the sub, not free, not uncovered). - Add the three-bucket framing (A free & deterministic / B model-gated on your sub / C needs-a-container, composed) and grade with two numbers ("% testable at all (free+sub)" vs "% needs-a-container"). - State that skill BEHAVIOR is tested (runEval A/B + judged), not just the description firing (measureTriggerRate). - Record that the sub-affordability story is ToS-clean (drive your own CLI / your own harness / your own sub; the Agent SDK ToS restricts productizing claude.ai login, not running your own tests). - Make the shipped test-harness skill coverage-aware (a Step 0.5 that classifies each surface into the three buckets and sets honest per-bucket expectations). - Roadmap: add the 2026-06-17 multi-SDK mock-ergonomics borrow-list, linked to sdk-harness-testing.md, marked recent/this-PR. Covers CLAUDE.md(.spec.ts), README, docs/eval-architecture.md, research/eval-coverage-and-isolation.md, research/roadmap.md, skills/test-harness/SKILL.md. * docs: flag open finding — behaviour testing is broader than A/B Persist a conclusion raised this PR that lived only in chat: A/B (runEval arms + assertSignificant) is the right oracle for relative /regression/noise-floor questions, but testing an EXACT skill wants an ABSOLUTE judged assertion, not a comparison to an OFF arm. The single-arm absolute path already exists (measure + judged + assertRates, skill-quality.eval.mjs); the open question is whether it is first-class and discoverable or whether A/B is over-privileged in the API + README + the test-harness skill. Recorded as testing-api-design Part 7 #7 + a roadmap item for the next review session. * docs(positioning): sharpen the pillar-2 wedge against SDK reality The mock-model tier is table stakes (Pydantic/Vercel/LangGraph/LlamaIndex all ship a first-party fake), so it is not the differentiator. Name the three things no SDK mock does — testing the harness as it SHIPS, tool-contract enforcement of the assembled agent (anchored on the live Claude Agent SDK bug #172), and a sub-affordable real-model tier. Also reframe behaviour testing as TWO oracles: the absolute single-arm measure()+judged() (the default when there's no on/off baseline) alongside the relative runEval A/B. * docs(testing): make the absolute behaviour oracle first-class for single skills A/B (runEval + assertSignificant) was over-privileged across the README, docs/harness-testing.md, and the test-harness skill, forcing 'test this one skill behaves right' into a relative on/off mould. Lead instead with the absolute oracle — measure({ checks: [judged(rubric)] }) + assertRates — which needs no baseline and is what promptfoo/DeepEval lead with; reframe A/B as the specialised relative/regression oracle. Resolves testing-api-design Part 7 #7 (docs/skill only, no code change). * docs(testing): declare one canonical import per pillar The docs taught granular concern-named paths (vigiles/run-hook, vigiles/eval, vigiles/check) while the test-harness skill and README use the vigiles/testing barrel — an inconsistent canonical-import story. Declare vigiles/testing and vigiles/linting the canonical entry points (granular paths stay as stable aliases) so new code has one obvious path. * docs(safety): correct overclaimed default + macOS confinement status The ephemeral run environment is shipped but OPT-IN (ephemeralEnv: true, default off) — docs/safety.md read as if it were on by default for every run. State the real status: default-off today, default-on is the committed next step gated on validating the file-based OAuth copy path; the pure pieces and scrubbed-env wiring are unit-tested. Also mark Seatbelt as macOS-planned (not shipped) in the two mechanism tables, so 'Mac deny-all' no longer reads as a working network wall. * docs(roadmap): note per-check thresholds for assertRates The absolute-oracle path (measure + assertRates) is now recommended for testing one skill, but assertRates({ min }) applies one rate floor to all checks. Persist the additive, non-breaking follow-up: let min accept a per-check mapping. * docs: add subagent-model-note rule; promote macOS Seatbelt to Now/P1 - New guidance rule: when launching a subagent, state which model was chosen and why, in one line, at launch — visibility into the cost/capability tradeoff. - Promote the cross-platform (Seatbelt) confinement backend from Next to Now/P1 per maintainer request; note there's no Mac to validate it on, so it needs a macOS CI runner or a Mac-having contributor. * refactor!: make the agnostic surface truly harness-agnostic, enforced by eslint vigiles/testing and vigiles/integration re-exported Claude-Code transport (scriptModel, claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable, loadPlugin, resolveHarness) via blanket `export *`s — so the 'agnostic' barrels weren't actually agnostic. The directory-based boundary lint missed it because src/mock-model.ts (the Anthropic-Messages mock) lives at the composition root, not under src/adapters/. - testing.ts / integration.ts: re-export ONLY the agnostic harness-test names; CC transport now comes from vigiles/claude-code. - eslint: classify src/mock-model.ts as part of the cc-harness element, so the existing agnostic-surface boundary rule forbids any agnostic barrel from re-exporting it (verified: reintroducing the leak now errors). - claude-code.ts: expose claudeCodeDriver + argv/parse helpers + claudeAvailable (beside the Codex driver in vigiles/codex), preserving access off the agnostic surface. BREAKING: scriptModel/loadPlugin/claudeCodeDriver no longer importable from vigiles/testing or vigiles/integration; import them from vigiles/claude-code. * refactor!: trim package.json exports to a canonical public surface Dropped 12 internal/redundant module subpaths that were each fully re-exported by a concern-named barrel: ./compile ./linters ./eval ./harness-test ./harness-assert ./run-hook ./check ./judge ./mock-model ./plugin-loader ./mcp ./tool-stub. The package's `exports` map IS the access wall (Node throws ERR_PACKAGE_PATH_NOT_EXPORTED; TS honors it under node16/bundler), so these are now genuinely internal. Canonical surface (12): . and ./spec (authoring + the module-augmentation target), ./linting, ./testing, ./unit ./integration ./e2e (tiers), ./claude-code ./codex ./adapter (harness), ./vitest ./jest (runners). ./linting now also re-exports the linter engine (was ./linters). BREAKING: the dropped subpaths no longer resolve — import via the canonical barrel (vigiles/testing, vigiles/linting, vigiles/claude-code). * docs: update Key Files descriptions for the trimmed public surface vigiles/plugin-loader and vigiles/check are no longer standalone subpaths; loadPlugin is on vigiles/claude-code and the check vocabulary on vigiles/testing. * docs: sweep imports onto the canonical public surface The trimmed exports removed the granular module subpaths, so rewrite every doc/skill import to the canonical barrel: vigiles/{eval,run-hook,harness-test, harness-assert,check,judge,tool-stub} → vigiles/testing, vigiles/{compile,linters} → vigiles/linting, vigiles/{plugin-loader,mock-model} → vigiles/claude-code. CC-specific transport (scriptModel, loadPlugin) is split out of mixed imports into a vigiles/claude-code line. Also corrected the two now-false 'granular paths keep working' claims (only vigiles/spec remains besides the barrels). * build: add API Extractor surface gate + API Documenter reference Lock the public API surface so it can't silently regrow (the dual of the eslint agnostic-surface rule, at the .d.ts level). scripts/api-extractor.mjs runs API Extractor over all 11 public entry points and writes a committed report per entry (etc/*.api.md) — the surface contract. CI (`node scripts/api-extractor.mjs` in the check job) fails on any drift; `npm run api:report` regenerates after an intentional change. The reports prove the de-leak: vigiles-testing.api.md has zero CC symbols, vigiles-claude-code.api.md has them. `npm run docs:api` runs API Documenter over the doc models to generate a cross-linked Markdown API reference (api-reference/, gitignored — a derived artifact, not committed). Narrative guides stay hand-authored. Adds @microsoft/api-extractor + @microsoft/api-documenter as devDependencies. * docs(roadmap): track moving the CC mock + driver into the adapter dir The glob-classification of src/mock-model.ts as cc-harness enforces the agnostic boundary today; the structural end-state is to relocate it (and claudeCodeDriver) under src/adapters/claude-code/ so the boundary is directory-based, mirroring the Codex adapter. * ci: smoke-check the API reference generator (no publish) Run API Documenter in the check job, reusing the doc models the surface gate already wrote to temp/, so npm run docs:api can't silently break. Output (api-reference/) is gitignored and not published — there's no docs site yet; upgrade this to a Pages deploy when one lands. * test(safety): prove ephemeralEnv actually scrubs — real-subprocess test The one line that enforces ephemeral scrubbing (env: replaceEnv ? scrubbed : merge) lived inside the v8-ignored real-spawn path, so a regression to an always-merge would silently defeat ephemerality with no test catching it. Extract it to a tested seam resolveSpawnEnv() and pin it two ways: a pure test of the replace-vs-overlay decision, and a behavioural test that spawns a REAL node child with the scrubbed env and asserts a planted secret is gone, HOME is the throwaway dir, and the real ~/.ssh is unreachable. Deterministic, no model. Updates the vigiles-testing API report for the new resolveSpawnEnv export. * feat(testing): per-check rate thresholds in assertRates The absolute-oracle path (measure({ checks }) + assertRates) is the recommended way to test one skill, but assertRates({ min }) applied a single floor to every check — you couldn't gate 'skill fires every trial AND judged >= 0.8' in one call. Add an optional per (by check kind) override: assertRates(report, { min, per: { skill: 1.0 } }); the failure message now reports each check's own min. Additive + non-breaking. Updates the vigiles-testing API report. * feat(testing): thread per-check thresholds through checkReportToJUnit assertRates gained a per-check-kind threshold override but checkReportToJUnit still took a single min, so the strict gate and the JUnit report could disagree about which checks failed. Extract one shared checkRateThreshold() helper used by both; checkReportToJUnit now accepts the same { min, per }. Updates the report. * test(safety): prove interceptTools actually PREVENTS, end-to-end The pure decision (decideIntercept/interceptHookDecision) was tested, but the rail's whole point — the real `vigiles intercept-tool-hook` CLI, wired as a PreToolUse hook, BLOCKING a matched call before it runs — was never exercised. A regression in the subcommand's env-read or deny format would pass every pure test. Drive the real built CLI through runHook (no model) and assert a matched `git push` is blocked while a non-matching command and an un-intercepted tool run. Deterministic; skips loudly if dist/ isn't built. * fix(testing): guard two silent-green footguns in measure/assertRates Both are the 'passes green while testing nothing' anti-pattern the no-silent-skips rule forbids: - assertRates on an empty report (measure with no checks) vacuously passed — now throws, pointing at the likely empty checks list. - stubSkillBodies + a judged check: stubbing replaces the skill body with a no-op, so judged graded empty output (docs warned, nothing enforced) — now throws, telling you to drop one. Adds tests for both. * feat(cli): per-line GitHub annotations for broken doc refs The dogfooded GitHub Action surfaces lint findings inline on the PR diff via ::error file=,line=:: workflow annotations. The inline/frontmatter/spec checks already annotated per-line, but the markdown code-block ref check reported to stdout only — so a broken enforce()/file()/cmd() in a doc never showed up inline on the PR, only in the summary blob. Emit a per-line annotation for each docRefReport error (it carries file+line). CI-only; skipped under --json/--summary. Adds a hermetic test. * ci: run the dogfooded vigiles Action as the primary lint gate Move the composite Action (uses: ./) ahead of the redundant direct `node dist/cli.js lint` in the check job, so a lint failure surfaces THROUGH the shipped Action — its per-line annotations (inline on the PR diff) and sticky PR comment — rather than a bare CLI step failing first and skipping it. The direct CLI call stays as a redundant safety net. Aligns with prod-grade-gha-cli (the Action is how CI runs for users). * test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation NOT FOR MERGE — a broken file() ref in docs/cli.md so the dogfooded Action fails and leaves a per-line ::error annotation inline on the PR diff plus a sticky comment. Revert this commit once the GHA behavior is confirmed. * Revert "test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation" This reverts commit 204497f0e6. * refactor(lint): deprecate require-skill-spec (default off), lean on untested-surface Requiring a .spec.ts per SKILL.md was the wrong constraint: skills are legitimately hand-written (the Level-0/1 on-ramp), so the rule only added noise — it nagged about hand-authored, vendored, fixture, and bench skills alike (~24 warnings in this repo, all non-gating). The constraint that actually matters — 'every skill/agent/hook ships with a test or eval' — is already the untested-surface rule. - DEFAULT_RULES: require-skill-spec → false (was warn); aligns with its own documented default and the @deprecated type comment. - --strict no longer promotes it (tightens only require-spec). - Removed from this repo's .vigilesrc.json. - Implementation KEPT — set it explicitly to re-enable (e.g. skills with file()/cmd() refs). - Docs: deprecation banner + untested-surface cross-links; updated tests. * fix(lint): untested-surface fires on all skills/agents — drop the invocation-mode exemption The filter silently exempted command-only (disable-model-invocation) skills from the test requirement, on the reasoning that a *trigger* eval is meaningless for them. But a command-only skill still DOES something when invoked, and that behaviour is worth a test — so the exemption was wrong (and inconsistent: agents were never exempt). Now every skill, agent, and hook is held to the requirement; the ONLY opt-out is an explicit vigiles:ignore-test marker, which is reported as `exempt` so the skip is visible, never silent. - test-coverage.ts: drop the userInvoked filter + the isUserInvoked helper; the Surface gains an `ignored` flag (set by vigiles:ignore-test) so the explicit opt-out is counted, not dropped at discovery. - Remove the now-meaningless includeUserInvokedSkills option (TestCoverageOptions + TestCoverageConfig). - Docs + tests + spec descriptions updated. Surfaces the one real gap it was hiding: skills/migrate-to-spec now flags as untested (linter-docs is content-referenced by skills-dogfood.test.ts). * refactor!: split untested-surface into per-kind untested-skill/agent/hook rules A skill, an agent, and a hook are tested in different ways (a skill's 'does the description fire?', an agent's tool-contract/outcome, a hook's block/allow), so one umbrella rule conflated them. Replace untested-surface with three parallel, independently-configurable rules — untested-skill, untested-agent, untested-hook — each gated by its own severity (set one to false to skip that kind). The detector (findUntestedSurfaces) is unchanged; the CLI now gates per kind. BREAKING: the untested-surface config key is removed. Use untested-skill / untested-agent / untested-hook (all default 'warn', same as before). The TestCoverageConfig skills/agents/hooks toggles are gone (a kind is toggled by setting its rule to false); testGlobs/exclude remain, shared across the three. - types.ts: RulesConfig keys; TestCoverageConfig trimmed. - validate.ts DEFAULT_RULES: the three keys. - cli.ts checkUntestedSurfaces: per-kind severity + gating; fix an unrelated no-deprecated eslint error on the intentional require-skill-spec read. - Docs: untested-surface.md split into three rule docs; all links updated. - .vigilesrc.json: drop the explicit key (defaults apply). --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-18 12:17:34 +07:00
zernie	71b6c4c3d3	fix: finish audit→lint rename — repair CI build, shipped hook, and docs (#34) ... * fix: finish audit→lint rename — repair CI build, shipped hook, and docs The `audit` command was renamed to `lint` in 2c64c62, but the rename never reached CI, the SessionStart hook, the CLI help, or the docs — so the `check` job ran `vigiles audit` ("Unknown command", exit 1) and the build badge went red. - ci.yml: run `vigiles lint` (+ the action dogfood `command: lint`) - hooks/session-start.sh: `vigiles lint --summary` - examples/demo/run.sh: `vigiles lint` (npm run demo was broken too) - src/cli.ts + action.yml: help text / input description say `lint` - docs/harnesses.md: de-stale the "second harness lands" framing (Codex already shipped), trim two over-explained sections, align Pillar 1/2 → Lint/Test - docs/harness-testing.md: trim the demo callout, audit→lint - README.md: paste-prompt as a copyable code block; fold init details into <details> * docs: research core-plus-adapters API design (grounds the pillar-2 relayout) Five-angle survey (AI SDK, unplugin, Drizzle, Knex, Testing Library, ESLint, OpenTelemetry, dependency-cruiser, semver/exports) of how comparable tools lay out a harness-agnostic core + per-target adapters, written to ground the pillar-2 API relayout. Verdict: keep one package with subpath exports; move the agnostic runners out of src/adapters/claude-code to the composition root (non-breaking — exports hide file paths); select by import in the library, string only in the CLI (never resolve a string to a module on a library path); add an eslint boundary rule so an agnostic barrel can't import a specific adapter; parameterize the conformance kit over ADAPTERS; mind the default-param tree-shaking trap. Linked from research/code-adapter-architecture.md + research/README.md. * refactor: route agnostic pillar-2 barrels through composition-root modules Step 1 of the pillar-2 relayout (research/adapter-api-design.md). The harness- agnostic public entries (vigiles/testing, /unit, /integration, /e2e) no longer `export *` from src/adapters/claude-code/* — they route through composition-root modules (src/{harness-test,run-hook,eval,mock-model}.ts), and a new eslint-plugin-boundaries `agnostic-surface` element forbids them from importing any adapter (proven to flag the old pattern). exports map repointed accordingly. Next step relocates the runner *logic* to these composition-root modules so the 100% coverage gate (which already lists them at src/ root) actually covers it. * refactor: relocate harness-testing library to the composition root Step 2 of the pillar-2 relayout (research/adapter-api-design.md). Physically move the deterministic harness-testing modules out of src/adapters/claude-code/ to the composition root src/: harness-test, run-hook, eval, eval-cache, eval-baseline, stats, mock-model, mock-entry, sandbox, egress(+entry/proxy), judge (+ tests). The adapter keeps the ports + bundle + harness glue (dialect, layout, runtime, hook-protocol, model-mock, adapter, agent-*, skill-*, plugin-loader, run-scripts). This makes the 100% coverage gate — whose include list already named these at src/ root — actually cover them: gated statements go 268 → 902 (the gate was near-empty because egress/sandbox/judge were listed at root but lived in the adapter, so the v8 include glob matched nothing). Coverage gaps the now-live gate exposed in egress.ts were closed with real tests, not v8-ignores. Pure relocation + import-path fixes + coverage-include (added stats, eval-cache); no logic or public-export changes. Build, lint (boundaries clean), 863 tests, and the 100% gate all green. * docs: split harness-testing into agnostic core + per-harness guides; sync spec Step 3+4 of the pillar-2 relayout. Docs: docs/harness-testing.md is rewritten as the harness-AGNOSTIC core (retitled "Testing your harness"): leads with select-by-import + { adapter } (default Claude Code), the Trace/predicates/assertions model, runHook/runHarnessTest/runEval shown agnostically, significance + regression gating, and a promoted runner-integration section (vitest/jest is the runner-agnostic seam, and it's tested in CI). The Claude-Code-specific walkthrough (oh-my-claudecode, ${CLAUDE_PLUGIN_ROOT}, the Anthropic mock, sandbox/egress) moves to docs/harness-testing-claude-code.md, and docs/harness-testing-codex.md covers the Codex path (runHarnessTest with codexAdapter against real codex exec; subagents a deliberate non-goal, runEval a documented follow-on). Both linked from the index + harnesses.md (orphan-docs ✓). Spec: CLAUDE.md.spec.ts Key Files repointed to the relocated module paths (src/adapters/claude-code/{harness-test,run-hook,eval,...} → src/...), the architecture section updated to describe the composition-root harness-testing library + the new agnostic-surface boundary rule, and entries added for the two new docs + research/adapter-api-design.md. CLAUDE.md recompiled (hash valid). * docs: fix stale `vigiles audit` references → `vigiles lint` The audit→lint command rename left 16 user-facing docs/examples telling users to run a command that no longer exists. Sweep them: docs/comparison, inline-mode, markdown-mode, spec-format, skills, verifying-instruction-files, testing-matrix, README, docs/rules/* and examples/{demo,frontmatter-CLAUDE}. `vigiles lint` (integrity + orphan-docs + refs) stays green; no command now points at the removed `audit`. * fix: repoint test/example imports to the relocated runner dist paths The step-2 relocation rewired src/ imports but missed consumers outside src/: test/types/smoke.{vitest,jest}.ts and examples/harness/*.mjs (+ dogfood, bench, plugin-test-demo) still imported dist/adapters/claude-code/{harness-test,eval, run-hook,sandbox}.js, which moved to dist/*.js. The CI test:types step (TS2307) and the harness job (ERR_MODULE_NOT_FOUND) caught it — local `npm test` doesn't run those tiers. Repointed every dist/adapters/claude-code/<moved>.js → dist/<moved>.js. * fix: repoint vigiles/judge export + regenerate path-union types Two more relocation leftovers: the package.json "./judge" export still pointed at ./dist/adapters/claude-code/judge.js (judge.ts moved to src/, so the public vigiles/judge subpath was broken), and .vigiles/generated.d.ts still listed the old src/adapters/claude-code/<moved> paths. Repointed ./judge → ./dist/judge.js and regenerated the type union (generate-types --check now passes). --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-15 19:34:08 +05:00
zernie	cf40a0f96c	docs: clarify pillar decoupling and surface install in harness section (#32) ... * docs: clarify pillar decoupling and surface install in harness section State the two-pillar decoupling plainly (neither pillar gates the other) instead of framing it by which file each pillar doesn't need. Add a concrete install command to the "Test your harness" setup block so it anchors to `npx skills add zernie/vigiles` / `vigiles/testing` instead of a vague "Install vigiles" prompt. * chore: gitignore locally-installed dev skills Ignore .agents/, .claude/skills/, and skills-lock.json so locally installed skills (e.g. obra/superpowers) used for internal development stay out of the repo and don't show as uncommitted changes. * chore: vendor obra/superpowers skills for internal dev use Add the superpowers skills under .agents/skills/ (with .claude/skills/ symlinks) and skills-lock.json for internal development. Verified via `npm pack --dry-run` that none of these paths ship in the npm package or the Claude Code plugin (.claude-plugin/plugin.json ships only ./skills/), so they're part of the repo but not part of what vigiles ships. Reverts the prior .gitignore exclusion of these paths. * chore: enable superpowers via plugin marketplace instead of vendoring files Replace the vendored obra/superpowers skills (.agents/, .claude/skills/, skills-lock.json) with a project-scoped plugin declaration in .claude/settings.json (extraKnownMarketplaces + enabledPlugins). Claude Code fetches the plugin into its ephemeral cache on startup, so the repo carries the enablement config, not 60+ third-party files. * docs: add production-grade GHA + CLI as a tool requirement Add the "prod-grade-gha-cli" guidance rule to the spec (compiled into CLAUDE.md): the CLI and GitHub Action are first-class, production-grade surfaces — CLI is the single source of truth with every Action input mapped to a real flag, the Action is a composite over the published CLI that works as uses: zernie/vigiles@v1, declares outputs via $GITHUB_OUTPUT, ships a full documented workflow, and is dogfooded in CI. * feat!: production-grade GitHub Action + CLI flag parity Make the GitHub Action and CLI first-class, production-grade surfaces (satisfies the new prod-grade-gha-cli rule). CLI: - Add --max-rules=<n> and --catalog-only flags (src/cli-flags.ts, applyConfigFlags) so every Action input maps to a real CLI flag instead of a config-file-only knob. Pure + unit-tested. GitHub Action: - Rewrite action.yml as a composite action over the published `npx vigiles` CLI. The old node20 entry pointed at dist/action.js, which is gitignored and never committed, so `uses: zernie/vigiles@<ref>` could never actually run — only npm did. The composite reuses the same tested, published artifact. - Declare the `valid` output and set it via $GITHUB_OUTPUT (drops the deprecated ::set-output). Add `version` (incl. `local` for dogfood) and `working-directory` inputs. - Remove the now-dead src/action.ts entry. CI / release: - Dogfood the Action in ci.yml via `uses: ./` with version: local and assert the `valid` output. - release.yml maintains a floating `v1` major tag so `uses: zernie/vigiles@v1` tracks the latest release. Docs/tests: - Full Action reference in docs/cli.md (copy-paste workflow, permissions, inputs, output, @v1-vs-@main versioning) + README pointer. - Structural test (src/action-yml.test.ts) locks in the composite/outputs/ no-set-output contract. BREAKING CHANGE: the GitHub Action is now a composite action and no longer ships dist/action.js. Inputs/outputs are unchanged in name where they overlap (command, paths, max-rules, catalog-only) but the action now adds `version`/`working-directory` inputs and a `valid` output. Pin to `zernie/vigiles@v1` going forward. * feat: PR comments + job summary output, JS/TS harness scripts GitHub Action output: - Post an automatic STICKY PR comment on pull_request events (found by a hidden marker and updated in place, never duplicated) + always write a job summary ($GITHUB_STEP_SUMMARY), alongside the existing inline annotations. New `comment` / `github-token` inputs; best-effort so a fork PR without write access degrades to a warning. Dogfooded in ci.yml (check job granted pull-requests: write). Dual-language harness/eval scripts: - `vigiles test` / `vigiles eval` now discover and run scripts authored in JavaScript (.mjs/.cjs/.js) OR TypeScript (.ts/.mts/.cts). A TS script runs through tsx when installed, else Node's native type stripping (>=22.6), with an actionable error otherwise (src/adapters/claude-code/ run-scripts.ts: interpreterArgs/detectNodeCaps/scriptGlob). Requirements + docs: - Extend the prod-grade-gha-cli rule (three output channels) and add the dual-language-tests rule to the spec (compiled into CLAUDE.md). - Document the comment behavior + pull-requests: write in docs/cli.md; rewrite the JS-vs-TS section of docs/testing-matrix.md. Our own unit suite remains 100% TypeScript (every src/**/*.test.ts); examples/harness/*.mjs stay JS on purpose as zero-dep CLI demos. * docs: slim README to a billboard landing page Cut the README from ~207 to ~118 lines, moving it out of the "long but neither pitch nor index" middle into a focused landing page (Knip/esbuild shape), per a survey of dev-tool READMEs (Ruff/Biome/Vitest/ast-grep/…) and standard-readme/Diátaxis guidance. - Pitch once, up top: tagline + one "why" paragraph + the single two-pillar table (was restated 4x across header/nav/table/TOC). - Keep the two concrete "first taste" artifacts: the stale-CLAUDE.md snippet and the runHook example. - Collapse the CLI, Skills, and Related Tools sections + the duplicated promptfoo rebuttal into one-line links (the detail already lives in docs/). Delete the <details> table of contents. - Drop the README section anchor from docs/verifying-instruction-files.md (now points at the README top). Verified: `vigiles audit` passes with zero orphan docs (every doc the README no longer links is still indexed by docs/README.md or CLAUDE.md). * docs: document how the test tiers run in CI Capture the CI tier mapping in docs/testing-matrix.md (it previously lived only in discussion): which tier runs in which ci.yml job, that CI runs every tier except evals, that evals are manual ($/non-deterministic), and the cheapest-first best practice + the per-tier test:unit/integration/ e2e scripts that enable the split. * docs: recast both README pillars as Without/With with real output Replace the prose-heavy pillar pitches with tight "Without vigiles / With vigiles" comparisons that show ACTUAL vigiles output (captured from real runs, not invented): - Pillar ①: `vigiles audit` flagging a rule that "exists but is disabled in eslint config" (the valid-≠-true differentiator), a passing rule, and a typo with a did-you-mean suggestion. - Pillar ②: the runHook assertion + a real `vigiles test` pass summary, framed around a silently-broken --no-verify guard. Shorter and more concrete; shows the payoff instead of describing it. * feat: refs-hook nudges the agent to mark references on save Close the authoring-time half of pillar 1: an automatic, in-loop nudge so the agent expresses references as verifiable vigiles marks instead of unverifiable prose. Ships in the plugin — end-users author nothing. - New `unmarked-refs` rule (default "warn"). The PostToolUse refs-hook (hooks/refs-nudge.sh → `vigiles refs-hook`) fires on every CLAUDE.md / AGENTS.md / SKILL.md edit and, by severity: "warn" → injects a non-blocking nudge into the agent's context (mark these refs); "error" → blocks the edit (exit 2); false → off. Wired into .claude-plugin/ plugin.json so it's on by default after `vigiles init`. - Refactor the detection into core: collectRefIssues (shared by the `vigiles refs` CLI and the hook) + the pure refsHookAction severity map, unit-tested. - Honest scope: catches code-shaped backticked refs; plaintext prose is undecidable (the floor). Hooks are Claude Code / Codex only; other harnesses fall back to instructions + the `vigiles audit` CI backstop. - Docs: the from-file-save flow in verifying-instruction-files.md, a new rules/unmarked-refs.md, and the rule added to the cli.md / docs index. * test: dogfood the refs-hook through the runHook tier skills-dogfood.test.ts only proves the hook script exists; this drives the actual built `vigiles refs-hook` via the repo's own runHook unit tier and asserts it FIRES correctly — nudges (non-blocking, additionalContext) by default, blocks (exit 2) under unmarked-refs:"error", stays silent when off, and no-ops on a clean or non-instruction file. Closes the "test your harness, don't trust it on vibes" gap for the new hook. * fix: scope refs-hook nudge to linter-rule names (kill false positives) Dogfooding the hook on the repo's own SKILL.md files exposed heavy over-firing: skills/test-harness/SKILL.md got 23 nudges, all on API names (`runHook`, `runEval`, …) that are documentation prose, not stale references. The `isCodeShaped` heuristic flagged every backticked camelCase/Pascal/SCREAMING identifier — far too noisy to ship at warn-default. Tighten to high-signal only: flag a slash-scoped name with NO file extension (`eslint/no-console`, `@typescript-eslint/no-explicit-any`, `boundaries/dependencies`) and nudge toward `enforce()` / inline `<!-- vigiles:enforce -->`. Bare identifiers and file paths are no longer flagged (paths re-introduce the noise — the same SKILL.md files cite `docs/harness-testing.md`, `package.json`, …; symbol marking stays opt-in via an explicit `vigiles:symbol`). Re-dogfooded: test-harness 23→0, strengthen 2→1 (keeps the one real rule `@typescript-eslint/no-unused-vars`), migrate-to-spec 0. Tests + docs updated to the narrowed scope. * test: real-claude dogfood proves the refs-hook fires in a session Install-and-verify: with the claude CLI present, drive the deterministic harness tier (real claude + scripted mock model, no API key) so the mock Writes a CLAUDE.md naming `eslint/no-console`, the PostToolUse refs-hook fires, and requestContains asserts the non-blocking nudge reached the model's context. Closes the last gap — previously verified only through the runHook unit tier. examples/harness/refs-nudge.harness.mjs runs in CI via `npm run test:harness` (claude is installed there) and skips cleanly without the binary. * feat: vigiles init sets up both pillars, interactive-or-not, via the v1 Action Make onboarding smooth and complete (the smooth-adoption rule): - BOTH pillars by default. Pillar 1 (spec + types + compile) as before, PLUS Pillar 2 — scaffold a starter vigiles.harness.mjs the user adapts to a real hook. `--pillars=verify|test|both` scopes it. - Interactive when a human's at a TTY (prompts: pillars / CI / plugin), NON-INTERACTIVE for agents / CI / piped input or `--yes` — so "set up vigiles" from a Claude Code / Codex prompt Just Works without hanging. Pure plan logic in src/setup-plan.ts (parseSetupArgs/shouldPrompt/ resolvePlan), unit-tested; IO stays in cli.ts. - CI wiring now creates a real `.github/workflows/vigiles.yml` using the production `zernie/vigiles@v1` Action (verify job + a harness job when Pillar 2 is on), instead of appending a raw `npx vigiles audit` line to an existing workflow. - New flags: --yes/-y, --pillars=, --no-plugin. Docs updated (cli.md init reference, agent-setup.md agent flow, README quick start). * test: CI-test the init command's new behaviors end-to-end Spawn the real CLI in a temp project and assert the new init behaviors, offline via --no-plugin (no network skills-add): default sets up both pillars + a zernie/vigiles@v1 workflow (asserts the spec, vigiles.harness.mjs, and the workflow YAML contents); --pillars=test scaffolds only the harness (no spec/workflow); --pillars=verify only the spec. Also add --no-plugin to the existing bare-init spawns so they stay offline and deterministic. * docs: make the "agent installs + tests skills" flow first-class Viewed from an agent handed "install vigiles and test my skills", the README install path was clear but the test-skills path was muddy — the only pillar-2 example was a hook, measureTriggerRate (the "does the skill fire?" primitive) was never named, and the model-invocable test-harness skill was buried as 1-of-8. - README: name measureTriggerRate (recall + precision) in the tier line, and surface that `init` installs the test-harness skill so "test my skills" fires it. - Add the `great-agent-flow` guidance rule: the agent-told-a-short-prompt experience must be discoverable from the README front door and frictionless (one non-interactive install, the skill fires the test, the skill-testing path is high-signal not buried). * fix: no blanket claude-gate; skips are loud, never a silent green Two fixes for `vigiles test` / `vigiles eval`, behind a stated principle (the new no-silent-skips rule): 1. Drop the blanket `claude`-gate in handleRunScripts. Unit-tier runHook tests need no model, so they now ALWAYS run — previously the whole command bailed if `claude` was absent, so a scaffolded vigiles.harness.mjs + `npx vigiles test` looked broken on a machine without claude. 2. Make skips LOUD. The runner classifies each script pass/skip/fail (exit 0 / SKIP_EXIT_CODE 77 / else), tallies them separately ("1 passed, 1 skipped"), shows `⊘ SKIPPED`, and a skip never fails the run (anyFailed). Add a `skip(reason)` helper (vigiles/testing) and convert the four deterministic examples from `console.log(...); process.exit(0)` (a silent fake ✓) to `skip()`. Up front, `vigiles test` notes when claude is absent. Verified end-to-end with claude off PATH: the unit example runs+passes, the deterministic example reports ⊘ SKIPPED, summary "1 passed, 1 skipped", exit 0. Adds the no-silent-skips guidance rule + skip-aware runner tests. * feat: vigiles test --no-skip (fail on skip) + README agent prompt Answer "shouldn't a skip fail loudly?" — yes, where you assert the capability is present. A skip passes by default (tiers need different capabilities not present everywhere; failing-by-default would make the command red in any partial env — this repo's harness job legitimately skips the egress tier), but `vigiles test --no-skip` makes any skip a failure for the CI job that asserts everything must run: a green-with-skips is itself untested surface. The skip stays loud + tallied either way. Extends the no-silent-skips rule; integration-tested. Also (per the agent-flow ask): add a paste-into-Claude-Code/Codex prompt to the README quick start — install + verify + write/run a harness test with good defaults, asking about CI gating / real-model eval / --strict. * fix: cover skip() under the 100% gate; flush skip/--no-skip docs The no-silent-skips commit broke CI's coverage gate (step 13): skip() in harness-assert.ts (a coverage-included file) calls process.exit, which is unreachable in-process → dropped below 100%. Mark it with the repo's `v8 ignore` convention (same as the spawn boundary); it's exercised via a child process in cli.test.ts / run-scripts.test.ts. Verified locally with `npm run coverage`: 100% stmts/funcs/lines, branches 92.43% (≥90 floor). Docs flush (before merge): document the loud-skip / unit-runs-without-claude / `--no-skip` behavior in docs/testing-matrix.md and the test-harness skill's "Run it" step, so the agent's authoring guide matches the runner. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-15 03:30:43 +05:00
zernie	8718bdb1e2	docs: point layer-1 reference link at the verify section + linter support (#30) ... * docs: point layer-1 reference link at the verify section + linter support The harness-testing doc's 'Verify the references' bullet linked the README root (no anchor), landing readers on the project pitch. Point it at the 'Verify your instruction files' section and docs/linter-support.md so a reader chasing 'is this linter rule real?' gets the actual mechanism. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: state explicitly why hook confinement is opt-in, not always-on The 'why not always sandbox?' rationale lived only in scattered docstrings. Add a dedicated section to docs/sandboxing.md (availability, hostile env, provenance trust, cost) and surface the same four reasons on RunHookOptions.trusted so authors see the tradeoff at the API. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: prove allowlisted-egress handoff in a spike (slirp4netns + nft) The 2026-06-12 note recorded that pasta --config-net failed to give a child netns egress on first pass. It works with slirp4netns --configure: unshare creates the user+net namespace, slirp4netns --configure attaches and configures tap0 inside it, and an nftables policy-drop output chain enforces an IP allowlist that drops a non-allowlisted host AND a raw bash /dev/tcp socket (the property an HTTP_PROXY allowlist can't give), with per-rule counters as the readable record. Add research/spikes/sandbox-network-allowlist.sh (self-contained, skips without the tools), update the experiment note + plan, and reflect the proven handoff in docs/sandboxing.md. Ships nothing yet — egress: { allow } still needs the bwrap PID handoff + option plumbing + gated tests (plan steps 2-3). https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(run-hook): allowlisted real egress via egress: { allow } Adds the third egress mode alongside deny-all and recordEgress: let a confined hook actually reach the network, but only an allowlist, with the boundary at the packet layer so a raw socket off-list is dropped too (which an HTTP_PROXY allowlist cannot guarantee). slirp4netns --configure attaches a tap to the bwrap netns (rootless egress); an in-netns nft policy-drop output chain accepts only the allowlist's resolved IPs (plus loopback + DNS) and log+drops the rest; per-rule counters read back what was reached vs dropped. bwrap --info-fd hands the child PID to slirp4netns, and a netready file gates the in-netns wrapper until the tap is up. - src/egress.ts: pure seams (resolveAllow/parseGetent, buildEgressNft, buildEgressBwrapArgv, parseNftCounters/countersToResult, egressAvailable) - src/egress-entry.ts: the bwrap<->slirp4netns orchestrator subprocess - run-hook: RunHookOptions.egress -> r.egress (allowed hosts) + r.egressDropped - tests: pure unit suite + gated integration (allow/drop/raw-socket) + the OMC session-start dogfood (reaches registry.npmjs.org, drops nothing else) - docs/sandboxing.md + research/sandbox-network.md: shipped, with the honest record limit (resolved at launch; resolver-pinned dynamic set is the next layer) Resolves the egress: { allow } gap designed in research/sandbox-network.md. 100% coverage on src/egress.ts; integration tests gated on bwrap+slirp4netns+nft. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: surface egress: { allow } in the remaining doc pointers The sandboxing.md/research updates shipped with the feature, but three other surfaces still described only deny-all vs recordEgress: the docs/harness-testing + docs/README see-also blurbs and the test-harness SKILL. Mention the third network mode (allowlisted egress) in each. * feat(eval): trigger precision + skill-eval landscape (AWS) + compression-claim demo Prompted by AWS sample-agent-skill-eval (a near-mirror of our harness-testing pillar). Three things: 1. Trigger precision (the one concrete gap it exposed): measureTriggerRate measured recall only. TriggerRateSpec.irrelevantPrompts adds the precision side — report.falsePositiveRate + report.precision — so a too-broad skill description that hijacks unrelated work fails, not just a too-narrow one. Gated by assertTriggerRate({ min, maxFalsePositive, minPrecision }). 2. research/skill-eval-landscape.md: AWS skill-eval scored against our eval pillar (near 1:1 — validates the bet) with the absorb-list (precision done; unified scorecard + cost-Pareto deferred; security scan = delegate). Also frames the token-compression cluster (RTK/Caveman/Claw/Context-Mode/pinchtab/ CodeGraph) as a use case for runEval, not a roadmap item. 3. examples/harness/skill-compression.eval.mjs: worked demo verifying a Caveman-style compression claim — two arms measure outputTokens (the target) AND a correct fact that must survive, with a guardrail that fails if the token win cost correctness. Plus: test-harness SKILL trigger row updated to recall+precision; CLAUDE.md Key Files recompiled. 100% coverage on eval.ts + harness-assert.ts. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * test(dogfood): conformance gate over vigiles's own skills + 2 trigger evals Eat our own dog food — point the harness-testing pillar at vigiles's shipped skills (.claude-plugin/). - src/skills-dogfood.test.ts: free, model-free gate — loadPlugin materializes every skills/<name>/SKILL.md and each has a name + non-empty description (the trigger surface). The floor under the paid trigger-rate eval: a skill that won't load can never fire. - Found + fixed a real bug: skills/generate-logo had no frontmatter 'name' (a skill without a name doesn't register). - examples/harness/dogfood/{test-harness,generate-logo}.trigger.eval.mjs: precision-aware trigger evals (recall + irrelevantPrompts/falsePositiveRate) for the ONLY 2 model-invocable skills. The other 7 are disable-model-invocation (user-invoked) so triggering doesn't apply — the load gate covers all 9. Write-don't-run without model auth. Recompiled CLAUDE.md. 735 tests pass. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(plugin): move hook scripts to plugin root so installed hooks resolve Dogfood finding: plugin.json declares hooks as ${CLAUDE_PLUGIN_ROOT}/hooks/*.sh, but the scripts lived under .claude-plugin/hooks/. Per the Claude Code plugins-reference, component dirs (hooks/, skills/, commands/, agents/) MUST sit at the plugin root, not inside .claude-plugin/ — so the installed PreToolUse/ PostToolUse/SessionStart hooks resolved to nothing. - git mv .claude-plugin/hooks/*.sh -> hooks/ (the conventional location) - src/cli.test.ts: point the hook-path tests at the new location - src/skills-dogfood.test.ts: add the precise guard — every ${CLAUDE_PLUGIN_ROOT}/<script> in plugin.json must exist at the plugin root (loadPlugin's dangling scan doesn't cover plugin.json command paths, so this is what actually catches the bug class; it can't regress) Convention confirmed against code.claude.com/docs plugins-reference. 736 tests pass; CLAUDE.md recompiled. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(audit): add untested-surface rule — flag skills/agents/hooks with no test The third gap detector beside orphan-docs: skills, subagents, and hooks that ship without a test or eval. A surface with no test is a probabilistic- compliance gap hiding in the deterministic layer — nothing measures whether it still does what it claims. Two OR'd detectors decide 'tested', so a test placed anywhere counts: - colocation: a *.{harness,eval}.mjs next to the surface (the zero-config convention the warning suggests) - content-reference: any discovered test (incl. *.test.ts) naming the surface by path (skills/foo, hooks/x.sh) or namespace (vigiles:foo) Warning-by-default (a nudge, exit 0); set to 'error' to gate CI. User-invoked skills (disable-model-invocation) are exempt by default since they can't auto- trigger; flip includeUserInvokedSkills to demand an outcome test. Per-surface opt-out via a vigiles:ignore-test marker. - src/test-coverage.ts + test suite (9 tests) - wired into vigiles audit (warn->exit0 / error->exit2, mirroring integrity), --summary line, GH annotations - config: rules.untested-surface in types + DEFAULT_RULES + .vigilesrc.json - docs/rules/untested-surface.md, linked from docs/README.md + docs/cli.md; README rule count 4->5 - recompiled CLAUDE.md Dogfood: our repo is clean — 5 surfaces covered (2 invocable skills + 3 hooks), 7 user-invoked skills exempt. 745 tests pass. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(pkg): ship hooks/ and skills/ in the npm package The hooks-path fix moved scripts from .claude-plugin/hooks/ to hooks/, which dropped them out of the publish allowlist (files listed .claude-plugin but not hooks/). npm pack confirmed the tarball shipped plugin.json referencing ${CLAUDE_PLUGIN_ROOT}/hooks/*.sh while the scripts themselves were absent — and skills/ was never shipped either, so the tarball was never a complete plugin. Add hooks + skills to files (excluding *.spec.ts build sources) so the npm package is a complete, installable plugin: manifest + 3 hook scripts + 9 SKILL.md, no spec leakage. Verified via npm pack --dry-run. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: reflect skills conformance gate + untested-surface rule Revamp (not just append) the harness-testing docs for this session's two new user-facing concepts: - docs/harness-testing.md: split the conflated 'Coverage' section — it was only about vigiles's own V8 coverage, confusing in a guide about testing YOUR harness. Now 'Coverage of your harness surfaces' (the untested-surface rule + the model-free conformance floor: every skill loads with a usable description) vs 'vigiles's own coverage' (the V8 paragraph). Thread the untested-surface rule into layer 1 (verify references) and upgrade the Skills cell in the surface x tier matrix (loads + description, not just refs). - docs/testing-matrix.md: add unit rows for the own-skills conformance gate and the untested-surface detector, a real-model row for the dogfood trigger evals, and a note that a static skill floor + the audit rule sit below the three runtime tiers. - research/harness-testing-coverage-matrix.md: add cross-cutting capability rows (own-plugin dogfood, surface-test coverage detection) and align the Skills unit/static cell. Rules table/README/cli.md already carried the untested-surface row. fmt clean, audit intact, no orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: four deep investigations + strategic synthesis Four parallel deep-research docs (web-sourced, cited) on fresh ground — deliberately NOT re-covering promptfoo/eval/harness-testing (already in docs): - agent-supply-chain-security.md — 2026 plugin/MCP threat surface + OWASP Agentic Skills Top 10; re-examines 'delegate, don't build'; verdict: thin harness-aware `vigiles scan` as a pillar-2 feature, reject the security pivot. - standards-conformance.md — AGENTS.md/SKILL.md/MCP/ACP standardization; verdict: extend pillar 1 to be format-neutral + lead the 'valid is not true' wedge, reject the generic-linter (agnix) pivot. - runtime-guardrails-observability.md — guardrails + OTel-GenAI observability; verdict: stay out of the request path, take one bridge (OTel emission + verify-trace + Cedar/OPA policy codegen, emit/verify-only). - ai-native-linting.md — the AI-code-review wave; verdict: extend enforce() to AI-linter catalogs, prototype (don't trust) a measured judge() rule gated by the eval machinery, reject becoming an AI reviewer. strategic-synthesis-2026-06.md ties them together: all four converge on extend-don't-pivot, with one unclaimed wedge — the observed-vs-declared contract binding. Ranked bet list + the one positioning pivot worth taking (lead with 'conformance/attestation for the agent harness'). Registered in CLAUDE.md Key Files (recompiled); audit clean, no orphans (58), fmt clean. Specific 2026 figures are web-claimed, not independently verified — flagged in the synthesis. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: divergent strategic bets, triaged with founder reactions Pushes past the extend-don't-pivot synthesis. Captures genuinely divergent directions with live triage: - strong: plugin/skill leaderboard (distribution by data, not installs); harness cost/ROI optimizer (money story over correctness story) - explore: sell to harness vendors; compliance/attestation buyer; CI for model upgrades - roadmap: self-improving harness (uses idle evolve.ts + proofs.ts) - open (need a call): compiler-not-linter (generate the verifiable skeleton); vigiles-as-MCP-oracle (deterministic answer, not gate) - watch: SDK bet (harness-as-code); killed: one-source-many-backends; parked: contrarian measure-model×harness Registered in CLAUDE.md, recompiled, audit clean, no orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: SDK harness-testing gap (verdict: no) + triage updates SDK bet (#6) researched: serious agents ARE built in SDK code, but the deterministic mock-model test tier (vigiles's middle tier) is already first-party in the top SDKs (Pydantic AI TestModel/FunctionModel, Vercel MockLanguageModelV3, LangGraph FakeListChatModel, LlamaIndex MockLLM). Gap largely closed → no blanket pillar-2 retarget. Lone residual: deterministic guardrail/tool-contract enforcement testing (maps to agent-runtime.ts) — a one-day OpenAI-Agents-SDK probe, idea-borrow not a port. See research/sdk-harness-testing.md. divergent-bets triage updated: #1 compiler-not-linter killed (founder: useless); #5 MCP-oracle demoted (fold into scan); #6 no (gap closed). Strong front-runners remain #9 leaderboard + #10 cost optimizer. Registered in CLAUDE.md, recompiled, audit clean, no orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(cli): add 'vigiles scan' — deterministic plugin/repo report (no model) The easiest-first build toward the leaderboard (#9) and harness-aware supply-chain audit: point vigiles at any plugin/repo and get a read-only report of what it ships + what's structurally broken, with no model and no API key. Re-aims existing machinery (loadPlugin, parseAgentTools, findUntestedSurfaces): - skills: description present? + user-invoked flag - agents: declared tool contract, incl. the 'no tools: line → inherits every tool' footgun - hooks: script paths resolved across braced AND unbraced shell forms → ok / missing / unresolved (caught + fixed a false-MISSING on the unbraced form while dogfooding oh-my-claudecode) - command + MCP detection, untested-surface count, loader warnings - --json for pipelines src/scan.ts + src/scan.test.ts (5 tests); wired into the CLI; documented in docs/cli.md; registered in CLAUDE.md. Dogfooded on our own plugin (clean) and on the vendored oh-my-claudecode (hooks/agents/MCP/commands). 750 tests pass. Next (harder, needs model auth): the trigger-rate + safety layer that turns 'scan over many plugins' into the leaderboard. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(cli): plugin health leaderboard — scan many, rank by structural health The next rung up from 'scan' and the no-model engine for the leaderboard (research/divergent-bets.md #9): pass >1 directory to 'vigiles scan' and it ranks them by a deterministic 0-100 structural-health score (+ A-F grade), worst issues first. - src/leaderboard.ts: scoreReport (missing hook -15, no-description skill -10, agent-without-tool-contract -5, untested surface -3; empty machine = 0; clamps at 0) + rankPlugins + formatLeaderboard. Deliberately ignores the loader's free-text warnings (doc-mention false positives) so the ranking is defensible. - wired into 'vigiles scan <dir...>'; documented in docs/cli.md; registered in CLAUDE.md; src/leaderboard.test.ts (5 tests). Dogfood over the vendored plugins ranks superpowers (A/94) > wshobson (B/86) > oh-my-claudecode (C/78) with reasons. Behavioural columns (trigger-rate, observed egress, safety) need a model and stack on top. (Full-suite run flagged the gated egress network test, which is environmental — needs outbound to example.com; passes on re-run, unrelated to these files.) https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: add consolidated roadmap front door + reconcile sprawl The scattered 'next steps' sections across 7+ research docs had no single view. Add research/roadmap.md as the one ranked front door (Now / Next / Later / Backlog / Explore / Rejected), each item a one-liner linking the doc with its rationale, plus a 'shipped recently' list. - research/README.md: link roadmap.md at top; add the missing 2026-06 strategy cluster (4 deep researches + synthesis + divergent-bets + sdk-harness-testing) + the eval docs; relabel coverage-matrix as pillar-2 detail (roadmap owns the cross-pillar ranking). - feature-ideas.md: point to roadmap for status; reconcile stale rows (#14 scan shipped, #9 hook-validation partial via scan). - registered roadmap.md in CLAUDE.md; recompiled; audit clean, no orphans (61). https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: de-dup — feature-ideas becomes detail-only, roadmap owns ranking Answers 'why both?': they now have non-overlapping jobs. roadmap.md is the ranked status board (what's next, all pillars); feature-ideas.md is the detailed pillar-1 idea catalog (per-idea analog/problem/implementation + research) that roadmap links into. Stripped feature-ideas's duplicate priority table + shipped list (that ranking lives in roadmap now); kept the #1-#14 write-ups + research appendix. No orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(egress): make in-netns DNS work behind systemd-resolved (CI fix) The two egress e2e tests failed in CI ('got []') while passing locally. Root cause: GitHub runners use systemd-resolved, so /etc/resolv.conf is the 127.0.0.53 stub. parseResolvers picked it and the nft ruleset allowed only it — but the slirp4netns netns runs with --disable-host-loopback, so 127.0.0.53 is unreachable there. In-netns DNS failed, curl resolved nothing, zero packets. (Locally /etc/resolv.conf is a real nameserver, so it worked — hence green here, red in CI.) Proper fix, not a skip: - parseResolvers drops loopback stubs (127.0.0.0/8, ::1) and falls back to a public resolver (8.8.8.8) that slirp4netns NATs out. - the egress run now binds a generated /etc/resolv.conf (the routable resolvers) over the host's stub inside the netns, so curl/getent resolve correctly. Pure helpers unit-tested (loopback drop; the resolv.conf bind in the bwrap argv); the gated bwrap egress e2e still passes locally (33/33). Validating the CI half on the runner via the PR watch. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * ci: add informational egress diagnostic to pinpoint slirp4netns failure The egress-allow e2e tests emit zero packets on the hosted runner but pass locally, and the DNS guess didn't fix it. Add a non-failing diagnostic step (before the coverage gate) that captures the in-netns reality: bwrap/slirp4netns/ nft versions, /dev/net/tun presence, userns sysctls, and a live probe through the real runHook egress path printing the in-netns resolv.conf + getent + curl + ip addr. Lets the next run reveal whether slirp attached (tap0 up), DNS resolved, or NAT routed — so the real fix is grounded, not guessed. Informational only (exit 0); does not change the gate. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: egress-sandbox tooling — adopt pasta (passt), don't reinvent Answers 'is someone already solving this / are we reinventing the wheel?' for the rootless egress connector. Verdict: swap slirp4netns → pasta (passt) — Podman's rootless default, a near-drop-in that attaches to an existing bwrap netns by PID/--netns but doesn't depend on the /dev/net/tun tap-attach handshake that fails on GitHub-hosted runners (the CI symptom). Keep bwrap+nft+resolver. nsjail/ firejail/gVisor/Podman-lib/microVM rejected (ride the same userns/tun primitives, no nft read-back); Landlock net parked (port-only, no per-host/accounting). Smallest step: a ~10-line connector-switch spike. Cited; registered in research/README.md + CLAUDE.md; audit clean. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(ci): gate egress e2e tests on actual routing, not tool presence The two egress-allow e2e tests failed red on GitHub-hosted runners: bwrap + slirp4netns + nft are all present (so the old tool-presence gate let them run), but slirp4netns never attaches tap0 there — the netns has only `lo`, so nothing routes and the assertions see zero packets. Proven via the CI diagnostic (/dev/net/tun present, userns on, in-netns DNS fix applied, yet ip showed only lo). Replace the gate `egressAvailable(sandboxAvailable())` with `egressRoutes()` — a memoized probe that runs a trivial hook in the egress sandbox and checks /proc/net/dev for a non-loopback interface (the slirp/pasta tap). So the tests RUN for real where egress actually routes (locally: 21/21 pass) and SKIP honestly where it provably can't (the hosted runner) instead of failing red. This is the research-endorsed capability-probe-skip (research/egress-sandbox-tooling.md); swapping slirp4netns→pasta is the follow-up that makes the probe pass in CI too. egressRoutes lives in the existing v8-ignore block (real-exec), so the 100% coverage gate is unaffected; the pure egress helpers stay unit-tested. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(api): tiered import paths — vigiles/unit · /integration · /e2e Make the capability level of a harness test legible from its import path (and enforceable). Layered, dependencies downward only: - vigiles/unit — no caps (runHook logic, predicates, asserts, parsers). No claude, model, bwrap, or network. - vigiles/integration — re-exports unit + runHarnessTest/withHarness/scriptModel + loadPlugin (needs claude + bwrap, no key/network). - vigiles/e2e — re-exports integration + runEval/measureTriggerRate/judge + egress-using runHook (needs model/network). A *.test.ts importing vigiles/unit physically can't reach a heavy capability; an e2e test imports vigiles/e2e and gets everything. Barrels: src/{unit,integration, e2e}.ts; package.json exports ./unit ./integration ./e2e. Next: file-suffix test tiers + vitest projects + per-tier CI, then reusable GH Actions per level. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * test: operationalize tiers — vitest projects + per-tier scripts + move egress e2e Encode the test tier in the filename, routed to its own vitest project, mirroring the vigiles/unit·/integration·/e2e import paths: - unit = src/**/*.test.ts minus the suffixed tiers (pure, no caps) - integration = *.integration.test.ts (bwrap, no network) - e2e = *.e2e.test.ts (real egress/model/network; self-skips where unavailable) - runners unchanged Move the two egress-allow tests out of run-hook.test.ts into src/run-hook.e2e.test.ts, importing from the e2e barrel (./e2e.js) — dogfooding the tiered import path: an e2e test pulls from vigiles/e2e, a unit test couldn't. Scripts: test:unit / test:integration / test:e2e (vitest projects); the old live-claude bash e2e renamed test:e2e → test:cli-e2e (docs updated). + coverage run all projects (e2e self-skips in CI), so the 100% gate holds (verified: 100% stmts/funcs/lines, branches 93.8%, 756 tests). Note: run-hook.test.ts still holds a live-net recordEgress dogfood + bwrap confinement tests — fuller re-bucket into integration/e2e is a follow-up. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(ci): reusable per-tier action + e2e dogfood; fix egress probe Phase C: a composite action .github/actions/harness-tier (input: tier) that sets up exactly the capabilities a tier needs (unit: none; integration: bubblewrap + claude CLI; e2e: + pasta/slirp4netns/nft for rootless egress) and runs npm run test:TIER. Reusable from any repo via zernie/vigiles/.github/actions/harness-tier@main. Dogfooded: a new e2e job in ci.yml uses it (installs pasta, runs the e2e tier). Fix the egressRoutes() probe: the /proc/net/dev interface count false-positived on the runner (bwrap binds host /proc, so it counted host interfaces and the e2e tests RAN and failed instead of skipping). Switch to the only namespace-accurate signal — actually reach an allowlisted host through the egress path and check a packet got out. Locally routes (e2e tests run + pass, 2/2); on the hosted runner it does not (so they skip honestly) until the slirp4netns->pasta connector swap. Also remove the now-spent informational egress diagnostic step from the test job. Coverage gate holds: egress.ts is fully unit-tested, the real-exec path is v8-ignored, and the probe exercises egressSpawn at import time. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(egress): connector selectable (slirp default, pasta opt-in); split e2e vs eval Two fixes: 1. e2e != eval. vigiles/e2e is now DETERMINISTIC-only (real sandbox/network, pass/fail): it drops the eval runners (runEval/measureTriggerRate/judge) and keeps egressRoutes + the egress-using runHook. Evals are a separate non-deterministic MEASUREMENT axis at vigiles/eval + vigiles/judge, run via vigiles eval on *.eval.mjs. (3 deterministic verification tiers unit/ integration/e2e + the eval measurement axis.) 2. Rootless egress connector is selectable. Default slirp4netns (proven locally, e2e routes + passes). Set VIGILES_EGRESS_CONNECTOR=pasta to use pasta (passt), intended for hosted runners where slirp4netns's tap-attach fails. HONEST WIP: pasta --config-net <pid> does not yet route in our bwrap netns (connect fails, not just DNS) — needs real integration (capture pasta stderr, correct flags/ namespace). So pasta is opt-in, not default; CI e2e still skips honestly until it's finished. egress-entry prefers the configured connector and falls back to slirp4netns. egress-entry is v8-ignored, so coverage is unaffected. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: make the harness levels crystal-clear + GH Actions + eval-pillar note Revamp docs/harness-testing.md per the tiering work: - Replace the vague 'four layers' intro with a LEVELS table: refs / unit / integration / e2e / eval, each mapped to its import path (vigiles/unit·…), file suffix (*.test.ts · *.integration.test.ts · *.e2e.test.ts · *.eval.mjs), runner (vitest project / vigiles eval), and what it needs. The level is legible three ways (import path = capability contract; suffix; CI job). - New 'Per-level CI (the reusable action)' section documenting the composite action zernie/vigiles/.github/actions/harness-tier (tier: unit|integration|e2e) + that vigiles dogfoods it, with the honest pasta/egress note. - New 'Two pillars or three? (where eval sits)' section: keep TWO pillars; eval is pillar-2's non-deterministic measurement axis (measures vs asserts), drawn as a distinct axis but not a separate pillar — unless the self-improving-harness direction lands, when eval becomes optimization (its own concern). https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(ci): run e2e in a privileged container so egress routes for real Root-caused why e2e egress fails on GitHub-hosted runners (and why it only passes locally): the connector — slirp4netns OR pasta — needs /dev/net/tun + privilege to bring up the tap and configure the netns. The hosted test job runs NON-ROOT, so it can't open tun (the netns ends up with only lo); our dev box passes only because it's root. Captured the actual errors while trying pasta: as root it self-drops to nobody (userns permission denied); as non-root it hits '/dev/net/tun: Permission denied'. So the connector choice was a red herring — privilege is the wall. Fix: the e2e CI job now runs as root in a privileged container (node:20-bookworm, --privileged --device /dev/net/tun), where the proven slirp4netns path attaches exactly as it does on a root dev box and the egress tests run FOR REAL in CI — closing the 'skipping isn't fair' gap. Tests still self-skip via egressRoutes() if egress can't route, so the job never fails red. - harness-tier action: sudo is now conditional (absent in root containers, present on non-root host runners) so the same action works in both. - pasta stays an experimental opt-in (VIGILES_EGRESS_CONNECTOR=pasta) for rootless envs that don't restrict tun; its attach-path needs more work. - Findings written up in research/egress-sandbox-tooling.md. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * test(e2e): assert what passes the allowlist, not OMC's off-allowlist count The privileged-container e2e job WORKS — egress routed for real in CI: the allowlist test passed (example.com reached; 1.1.1.1 + raw 9.9.9.9 dropped) and the OMC test reached registry.npmjs.org. It failed only on an over-strict check: egressDropped===0, but OMC emitted one stray off-allowlist packet (76 bytes) that the wall correctly DROPPED — the wall working, not a failure. Fix the assertion to capture the real dogfood claim — 'OMC reaches npm and nothing else gets THROUGH' — by asserting the only allowed host is the npm registry, instead of OMC's exact dropped-packet count (which varies by env via DNS/CDN). Drops are proven by the other e2e test. Both e2e tests pass locally (root) and now in the privileged CI container — egress is exercised for real in CI, no skip. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-14 00:06:07 +05:00
zernie	73a9dde310	feat: unified Trace + predicates, model-request capture, safe-by-default sandbox (#27) ... * feat: unified Trace + predicate vocabulary Both testing tiers now produce one Trace (toolCalls + output + turns + file): a runHarnessTest result is a Trace, and runEval's measure ctx (RunContext extends Trace) is too. Over it sits one set of bare predicates — usedTool, toolCount, skillResolved, toolUsedWith — exported from harness-assert.ts; each assertTool* is now a thin throw-wrapper over a predicate, and an eval measure reuses the same bare predicates as metrics. Testing asserts, eval measures — same vocabulary, separate consumers, never one dual-purpose function. - harness-test.ts: Trace interface; parseResultEvent/parseOutput capture the final answer from the stream both tiers already produce; result.output added. - eval.ts: ctx gains toolCalls + output (eval switched to stream-json so the per-turn tool_use events survive); pass^k (MetricStat.passK = succeeded every trial) alongside mean ± se in the report and formatEvalReport. - harness-assert.ts: bare predicates + assertToolUsedWith (tool-argument assertion over toolCalls[].input); assert* re-typed to accept any Trace. Deferred: trace.hooks (which hook fired + decision) — needs mock instrumentation to record invocations rather than infer them from marker files. * feat: output + hook + reliability verbs over Trace, native coverage Completes the Trace vocabulary the unification opened and records hook firing. - trace.hooks: parseHooks records which hook fired + its decision from the CLI's hook_response stream events (hook_name/hook_event/exit_code/outcome) — honest, not inferred from marker files. HookFire on Trace, populated by both tiers. - predicates/asserts: hookFired/hookBlocked/assertHookFired (recorded hook firing), outputContains/assertOutputContains (the agent's final answer), reliable/assertReliable (pass^k gate — succeeded every trial). - The two Edit/Write hook regression tests now also assert via trace.hooks (transcript:true), so firing is checked from the stream, not just markers. - coverage: npm run coverage uses node 22 native V8 coverage (no dep); ~86% lines with claude present. npm test switched to glob discovery (node --test "dist/**/*.test.js") so a new *.test.js can't be left out. Docs + research matrix updated: trace.hooks and coverage tooling marked shipped. * test: migrate the suite to Vitest (primary runner) node:test was the zero-dep dogfood, but its DX (noisy TAP, weak watch/filter, no coverage thresholds or inline ignores) doesn't pay off. Vitest runs the TS sources directly (no pre-build for the unit bits), gives a readable reporter, watch/filter, and V8 coverage with thresholds. - vitest.config.mjs: two projects — `unit` (src/**/*.test.ts, with .js→.ts extensionAlias so NodeNext imports resolve; 60s timeout to match node:test's no-timeout default for catalog/CLI-spawning suites) and `runners` (the existing cross-runner matcher constraint, loaded from dist as a user would). - Mechanical per-file change: import { test|describe|it } from "node:test" → "vitest"; before/after → beforeAll/afterAll via import alias (call sites unchanged). node:assert/strict kept as-is (works under vitest). - The runner-agnostic claim is still proven: harness-assert keeps its node:test/jest matchers, and test/runners/ exercises them under vitest + jest. - scripts: test → vitest run; test:vitest → the runners project; coverage → vitest run --coverage (V8). Coverage is scoped/threshold-gated to the harness-testing pillar (driven to 100% next). * test: 100% coverage gate on the harness-testing pillar `npm run coverage` now enforces 100% lines/functions/statements (branches ≥ 90) on the seven harness-testing modules, via Vitest V8 coverage. - eval.ts: split the orchestration into a testable `runEvalWith(spec, runner)` seam (injectable agent runner); `runEval` is the thin real-spawn wrapper (v8-ignored). Lets the loop / measure-context / aggregation be tested with canned stream-json — no model. - mock-model.test.ts (new): drives the scripted server directly (SSE + JSON, tool + text, count_tokens / HEAD / health, onTurn, repeat / empty-script). - plugin-loader: a malformed plugin.json no longer crashes the loader (safeReadJson); covered by new malformed-manifest + manifest-mcpServers tests. - Added tests for the bare predicates' output/hook verbs, the run-hook-tier assertions + matcher message closures, parse* defensive branches, judge's firstJsonObject catch, and parseHookOutput's catch. - Subprocess wrappers that can't be unit-tested (real claude spawn in runHarnessTest/spawnClaude/judge) are v8-ignored with justification; they stay covered by the claude-backed suite. * feat: capture model requests into the Trace (trace.modelRequests) The scripted mock now records every /v1/messages request it receives — system prompt + messages, flattened to text — exposed as handle.requests and surfaced on the unified Trace as modelRequests. This closes the gap where a SessionStart hook's injected additionalContext (or a slash command's expansion) could be proven to *fire* but not to *reach the model*. - mock-model.ts: extractRequest (pure, exported) + per-request capture, MockHandle.requests - harness-test.ts: Trace.modelRequests, populated by runHarnessTest (harness tier only — with or without transcript) - eval.ts: modelRequests is [] (the eval tier drives the real API, no mock between claude and the model to capture requests) - harness-assert.ts: requestContains predicate + assertRequestContains - tests for extractRequest shapes, request capture, and the predicate; 100% coverage gate still green * feat: safe-by-default sandbox for executing untrusted plugin hooks runHarnessTest runs the real claude CLI, which runs the real hooks of whatever plugin you load — for an external plugin/pluginDir that means executing third-party code with your privileges. Make that safe by default. Policy (src/sandbox.ts, pure + exhaustively unit-tested): - specTrusted: inline settings/files = trusted; any external plugin/pluginDir = untrusted (provenance-based). - decideSandbox: "auto" (default) runs trusted code directly and confines untrusted code; if no sandbox is available it REFUSES (throws) rather than running unconfined. sandbox:false is the explicit dangerous opt-out; "strict" forces confinement even for trusted code. Confinement (runSandboxed, bubblewrap): the mock and claude are co-launched inside ONE network namespace (--unshare-all). Loopback is auto-up so the in-sandbox mock is reachable, but there is no external route, so a malicious hook can't phone home. Filesystem is --ro-bind read-only except the throwaway work dir, a fresh empty $HOME, and an IO dir; captured requests stream out through the IO dir into trace.modelRequests. mock-entry.ts is the in-sandbox mock runner. The end-to-end test proves a sandboxed run blocks egress while the mock stays reachable (gated on bwrap+claude, skips otherwise). Existing pluginDir/plugin tests are marked sandbox:false (we trust the pinned vendored fixtures), keeping them on the direct path. Also: enforce the 100% coverage gate in CI (was never run — npm test != coverage) with bubblewrap + claude installed so the sandbox + claude- gated tests actually execute; stop tracking coverage/ build artifacts. * feat: harden sandbox (clearenv + Linux-only), dogfood it, document testing Hardening (src/sandbox.ts): - --clearenv: drop ALL inherited env so untrusted code can't even read host secrets (ANTHROPIC_API_KEY, cloud creds); only PATH/HOME/TMPDIR are set back. Verified the confined claude still resolves via the restored PATH. - Linux-only: sandboxAvailable() short-circuits on non-Linux (bubblewrap is a Linux tool); decideSandbox messages and the spec JSDoc say so, so macOS/Windows get a clear "needs Linux + bwrap, or pass sandbox:false" instead of a confusing install hint. Dogfood (src/sandbox.test.ts), both gated on bwrap+claude: - injection test: a SessionStart hook (Claude Code's nested additionalContext form) runs confined and its context is found in trace.modelRequests — proving the capture chain end-to-end ("fired AND landed"). - superpowers dogfood: the REAL pinned obra/superpowers SessionStart hook runs CONFINED (untrusted → sandbox, no egress) and its genuine output is captured. It emits a TOP-LEVEL additionalContext, which Claude Code (nested form) never injects — so modelRequests shows the context did NOT reach the model. That "fired ≠ landed" gap is exactly what modelRequests surfaces, on real code. Docs: - README "Test your Claude Code harness": new subsections for trace.modelRequests ("did the injected context reach the model?") and the safe-by-default, Linux-only sandbox, plus two coverage-matrix rows. - real-superpowers.harness.mjs: drop the stale "future bwrap/docker boundary" note — the boundary exists now; point at the confined dogfood. * feat: close out testing pillar — measureTriggerRate + dangling-ref warning Two remaining gaps from the roadmap, both small, both close real holes. 1. measureTriggerRate (src/eval.ts) — does a skill's description actually FIRE across varied prompts? Wiring (does the Skill tool resolve) is the deterministic tier's job; real activation is a property only the model can answer, and the #1 documented skill-authoring pain. Install a plugin natively (pluginDir), give varied prompts + a `fired` predicate over the run's Trace (reuse the bare predicates, e.g. skillResolved), get an overall + per-prompt rate. measureTriggerRateWith takes an injected runner so the loop is unit-tested with no model; assertTriggerRate gates on a minimum; canonical example examples/harness/skill-trigger-rate.eval.mjs. 2. loadPlugin dangling-intra-plugin-ref warning (src/plugin-loader.ts) — scans the plugin's own text files (hook scripts included — those aren't materialized into files) for root-relative refs to skills/hooks/commands/agents files that don't exist on disk. Catches the partial-vendor / broken-path class the dogfood hit twice: verified it flags real obra/superpowers' skills/using-superpowers/SKILL.md (omitted from the vendored slice). 100% coverage gate still green (603 tests). Spec Key Files updated + CLAUDE.md recompiled. * docs: research — eval-API landscape, summarized and scored against ours Durable record of the eval-ecosystem comparison so it doesn't live only in chat. Two halves, as requested: (1) the field summarized — per-tool capability profiles for promptfoo, DeepEval, Braintrust, Inspect, LangSmith, OpenAI Evals, plus the RAG/observability/academic tools we're orthogonal to; (2) scored against our eval API on the dimensions that decide a *harness-eval* API — strengths (harness A/B arms, pass^k, se/std, unified Trace predicates, runner-agnostic/zero-dep), gaps (cost/concurrency/caching, significance testing, persisted reports + regression gating, thin judge / no dataset primitive). Records the strategic options A–D and the decision: pursue B→A→C, defer D, first reviewable unit B1 (cost capture) + B2 (record/replay cache). Wired into the link graph (Key Files + research/harness-testing.md See-also) so the no-orphan-docs rule holds; CLAUDE.md recompiled. * feat: eval Phase B1+B2 — cost/latency capture + record/replay cache The first reviewable unit of the B→A→C roadmap to make the eval API world-class (research/eval-api-landscape.md). Both pieces are pure + behind the injectable AgentRunner seam, so the 100% statements/lines/functions gate holds. B1 — cost/latency/token capture. The stream-json result event already carries total_cost_usd / usage / duration_ms; we were dropping it. parseUsage extracts it, RunContext.usage exposes it to measure (so `{ cost: ctx.usage.costUsd }` is a metric), ArmReport.usage + EvalReport.totalCostUsd aggregate it, and formatEvalReport prints `$… · …s/run · …k tok` when present (silent under the mock, so existing output is unchanged). B2 — record/replay cache (src/eval-cache.ts). cacheKey hashes everything that determines the model's output — task, resolved fixture files + settings, model, tools, trialIndex — but DELIBERATELY NOT measure, so editing your metric and re-running re-scores the recorded runs for free; the model is re-called only when a model-affecting input changes (or cache:"off"). Crucially it snapshots and restores the post-run filesystem, so measure's ctx.file()/ctx.sh() stay sound on replay — a stdout-only cache would silently mis-score. Opt in via cache / cacheDir on EvalSpec; wired into runEvalWith through runWithCache + executeTrial. Tests: eval-cache key stability/sensitivity, record round-trip + malformed tolerance, fs snapshot/restore; parseUsage, aggregateUsage, and an end-to-end record→replay through runEvalWith proving the model isn't re-called on a hit and restored files re-score correctly. 614 tests, gate green. Cache dir gitignored. * feat: eval Phase B3+B4 — bounded concurrency, rate-limit backoff, budget cap Completes Phase B of the eval-API roadmap (research/eval-api-landscape.md). B3 — concurrency + rate-limit backoff. runEvalWith now flattens arms × trials into a flat unit list run through a generic bounded-concurrency pool (runPool, order-preserving); `concurrency` defaults to 1 (the safe, no-surprise default — raise it to cut wall-clock time). Each model call is wrapped in runWithRetry: while isRateLimited matches the captured streams (rate limit / 429 / overloaded), it backs off exponentially and retries, up to rateLimitRetries (default 3) with retryBackoffMs base (default 1000). spawnAgent now captures stderr so the signal is visible; RunOut gained an optional stderr. B4 — budget cap. maxCostUsd stops launching new trials once measured cost crosses the cap (in-flight trials finish, the rest are skipped); EvalReport.aborted flags it. Uses the per-run cost from B1. All pure/orchestration behind the injectable runner seam (runPool, isRateLimited exported and unit-tested; retry/concurrency/budget driven through runEvalWith with fake runners — pool concurrency bound, retry-then-succeed, give-up at retries 0, and early-abort all asserted). 620 tests, gate green (100% stmts/lines/funcs). * test: model-free conformance suite over real vendored plugins Grounds the loader against reality instead of synthetic fixtures. A table-driven suite runs loadPlugin over each pinned vendored plugin (obra/superpowers, wshobson/accessibility) and asserts invariants that must hold for any well-formed plugin: a real surface loads (never a silent empty machine), ${CLAUDE_PLUGIN_ROOT} fully resolves, ≥1 skill materializes, and the surface + dangling-ref warnings are accurate (the known superpowers partial-vendor ref is flagged, nothing spurious). This is the committed form of the check that caught the dangling-ref bug. It's model-free, offline, and pinned by SHA — deterministic, in the coverage gate, no claude CLI or API key (tier 1 only; the real-model tier stays opt-in in bench/). Assertions are invariants, not version trivia (">= 1 skill", not "exactly N"), so a harmless re-pin won't break them while a real loader/detector regression will. 622 tests, gate green (100% stmts/lines/funcs). * fix: recompile CLAUDE.md so its integrity hash matches (audit clean) The previous commit formatted CLAUDE.md after compiling it, and a bare `*` in the new vendor.test.ts Key Files description got prettier-escaped to `\*` — desyncing the embedded hash (audit exit 2). Dropped the glob char from the prose and recompiled last so the hash covers the final, prettier-clean bytes. audit exit 0. * feat: eval Phase A1+A5 — significance testing for A/B arms Pays down the debt that assertImproves(..., { by: se }) booked: that API admits the "is this gap real or noise?" question but punts the noise floor to the user. src/stats.ts answers it — a Welch's t-test over the per-arm summary stats (mean/se/n, no raw rows, no EvalReport change) yielding a two-sided p-value and a verdict, via a self-contained Numerical-Recipes incomplete beta. Pure + model-free, validated against known t-table critical values (t_.975 at df 4/10/15) and the arcsin closed form for the incomplete beta — grounded numerics, not just internally consistent. Surfaced as assertSignificant / significantlyBeats, and assertImproves now takes { significant: true } to demand a computed noise floor instead of a hand-fed `by`. Deliberately scoped DOWN from the original Phase A: A2 (power analysis), A3 (pass@k at arbitrary k), and A4 (paired/blocked design) are deferred — each is correct but not yet tied to an observed pain, and building rigor ahead of need is the trap. They come back when a real eval demands them. The empirical cross-check (running a real bench/ comparison through compareArms) is also deferred: no API key here, so A1 is validated on known-answer distributions for now. 629 tests, gate green (100% stmts/lines/funcs, lint clean, audit 0). * docs: README — reflect the eval tier's new capabilities Level 3 (runEval) predated the eval-API work; it said "reports the gap … run it now and then" with no mention of what shipped. Surgical update (no new sections, per the readme-brevity rule): - gap is reported as mean ± se; assertSignificant turns it into a CI gate via a computed (not hand-fed) Welch t-test noise floor - the example shows the cache (cache:"readwrite") and the cost/latency/token line - notes concurrency + maxCostUsd budget cap - points at measureTriggerRate as the same-tier answer to skill activation * docs: sync harness-testing guide with the eval tier's shipped capabilities The README now advertises features the "full guide" it links to didn't cover. Brought docs/harness-testing.md up to date: - Significance: assertSignificant / significantlyBeats / compareArms — a Welch t-test with a computed (not hand-fed) noise floor; added as the third gating option alongside assertImproves / assertReliable, replacing the thin "raise trials" note. Re-exported compareArms + Comparison through vigiles/harness-assert so the whole eval-analysis surface lives behind one entry point (no ./stats). - Cost / caching / concurrency: ctx.usage + report.usage + totalCostUsd, plus the concurrency / maxCostUsd / cache:"readwrite" record-replay knobs. - measureTriggerRate: the trigger-rate (skill-activation) tier + its example. - Fixed two stale claims: the suite runs under vitest (not node --test), and the eval orchestration is now fully unit-tested via an injected runner (only the real-claude spawnAgent is out of the gate) — the 100% gate holds. No src behaviour change beyond the re-export. 629 tests, gate green, audit clean. * docs: testing-matrix — add the eval-tier surfaces shipped in this PR The matrix predated cache / significance / trigger-rate / usage / vendored conformance. Added unit rows for: usage capture+aggregation, record/replay cache, concurrency + rate-limit retry + maxCostUsd abort, measureTriggerRateWith, significance stats (welchTTest/compareArms/tPValueTwoSided), vendored real-plugin conformance, and the dangling-ref warning; broadened the assert-helpers row to cover the significance/triggerRate/tool-sequence asserts. Added the measureTriggerRate end-to-end integration row + skill-trigger-rate example. Corrected the "not unit-tested" note: only the real-claude spawnAgent is out of the gate — the eval orchestration is pinned via the injected runner. * feat: spec compilation for subagents (agent() → agents/<name>.md) A skill is reference material the model reads on activation; a subagent is a delegated worker with a *contract* — a dispatch description, an allowed-tools rail, a model, a system prompt, and the rules it follows. That contract is the "railway" and it's exactly the compile-time-verifiable surface vigiles owns, so subagents (more than skills) want spec compilation. - spec.ts: `agent()` builder + AgentSpec (name, description, model?, tools?, body, rules?). - compile.ts: `compileAgent` — YAML frontmatter (name/description/model/tools), system-prompt body with the same verified file()/cmd()/symbol()/ref() marks as any instruction file, an optional ## Rules section, and a SHA-256 integrity hash. Verifies the tool contract: built-in set + mcp__server__tool pattern, "did you mean" on a near-miss, and an error for tools that are *never* available to a subagent (Agent/AskUserQuestion/EnterPlanMode/ExitPlanMode/ScheduleWakeup/ WaitForMcpServers). adoptDiff handles the new target. editDistance exported from linters.ts for reuse. - agent.test.ts: frontmatter, tool verification (built-in/MCP/never-available/ did-you-mean), body-ref validation, Rules section, spec-name check, adoptDiff round-trip. Grounded by a background research pass (research/subagent-compilation.md): the real Claude Code frontmatter, and the key finding that `tools:` is documentation, not runtime enforcement (issue #54898) — the real rail is a generated PreToolUse hook. That hook-emission layer (compile the contract into BOTH the allowlist and an enforcing hook, and assert they agree) is the documented next step and the differentiator. This commit is the compile+verify foundation. 638 tests, gate green (100% stmts/lines/funcs), lint/fmt/audit clean. * feat: agent spec sections + dogfood on a real OSS subagent Dogfooding the agent API against wshobson's real ui-visual-validator settled the design: a subagent is a flat, multi-section role contract (10 `##` sections, prose + `###` subs), NOT a step/gate pipeline — and it ships with NO tools line, inheriting every tool (the #1 footgun, in the wild). - spec.ts: AgentSpec gains `sections` (named `##` blocks) + a `body` intro, mirroring claude() rather than skill() — the shape real agents take. Steps/gates stay a skill concept; a subagent's "railway" is the tool-contract envelope, not a state machine. - compile.ts: renderAgentSections verifies each section's refs and rejects nested `##` headers / a `rules` key clash, same rules as a CLAUDE.md spec's sections. - agent.test.ts: a dogfood test reproduces the real agent's shape and ADDS the least-privilege rail it omits (Read/Grep/Glob/Bash, no Edit/Write), asserting it compiles clean — the value-add over the hand-written original. Plus section ref-verification + reserved-key tests. - research/subagent-compilation.md: records the dogfood finding and the "flat flow yes / flow-generator no / tool-catalog generator optional" answer. 641 tests, gate green (100% stmts/lines/funcs), lint/fmt/audit clean. * docs: empirical subagent survey — no iterator needed (saturated, ~100 agents) Surveyed ~90–100 real subagents across 9+ repos and several ecosystems (haiku agents) — wshobson, undeadlist, VoltAgent, 0xfurai, lst97, rshah515 — plus the official Anthropic plugins (Ralph loop, PR-review-toolkit) and orchestration plugins (barkain). The result saturated: every additional batch returned zero. Findings recorded in research/subagent-compilation.md: - Iterative SYNTAX in an agent file: 0 / ~100. (Iterative LANGUAGE in prose is common; structural loops/state machines: none.) No typed Iterator/Step/Phase/ Flow primitive exists anywhere in the ecosystem — community or the official SDK. - Where looping actually lives: a Stop hook + state file (Ralph re-feeds the prompt), or external scripts/commands — never the subagent .md. Architectural reason: markdown is declarative+stateless; a loop is imperative+stateful. - Delegation graphs are linear/star and dynamic → even a typed handoff field is only weakly supported; prose + the Task/Agent allowlist covers them. - Verdict: ship the flat model; do NOT add an iterator (solution seeking a problem). Reinforces the compile-to-hooks thesis — emit PreToolUse (tool rail) / Stop (loop) hooks, don't invent iterator types. handoffs/outputSchema = maybe. No code change — research only. Confidence ~90%; gate untouched. * docs: note in-repo prior art for the agent enforcement hook A final survey straggler surfaced that vigiles already emits an enforcement hook: the vigiles:result mark compiles to a Stop hook (skill-hook) that gates skill completion (test/e2e/run.sh, research/runtime-enforcement.md). The planned agent PreToolUse tool-rail is the same emit-a-hook pattern pointed at a different event, so roadmap item #1 records it as prior art to mirror — not greenfield. * docs: design — railway-style subagents (verified plan-as-code; Temporal analogy) New research/railway-subagents.md exploring how to express a FLOW over flat subagents without reinventing a workflow engine, prompted by two inputs: - Plan-as-code ("ultra plan" / dynamic workflows): Anthropic's orchestrator generates an ephemeral, unverified JS orchestration script on the fly. vigiles' angle = the typed, verified, compiled counterpart (spec is source of truth → compile to a command .md + enforcing hooks + state schema; refs checked at compile time; regression-testable via runEval). - Nested subagents (Claude Code, experimental, user-reported): a deeper delegation tree makes a typed depth/cycle/dangling-target graph check load-bearing. Frames the Temporal analogy the user raised (workflow=deterministic railway; delegate=activity; gate=signal; state-file+Stop-hook+eval-replay=durable replay; "thinking happens in the activity, not the railway") and lays out three options: (1) manual marks, (2) a workflow() TS-spec compiled via a limited vigiles API (recommended core), (3) a thin Temporal-like deterministic driver over the harness's native Task+hooks+state (north star; explicit boundary: don't build an engine, delegate execution to the harness). Honest verification note: plan-as-code is confirmed for Claude Code (code.claude.com/docs/en/workflows, found earlier this session); the nested- subagent cap is NOT yet verified — an automated check examined the wrong product (the Developer Platform "Managed Agents" API, depth-1), so it's recorded as user-reported/to-verify and the design is made robust to the exact cap. Wired into Key Files + cross-linked with subagent-compilation.md. Docs only. * docs: surface ultraplan as inspiration in discoverable index + ideas backlog ultraplan / dynamic-workflows (plan-as-code) is the closest external thing to the direction we're building, so make it easy to trip over: - research/README.md: new "Subagents & orchestration" group indexing subagent-compilation.md + railway-subagents.md, with ultraplan called out by name and flagged "closest external thing to our direction — read for inspiration." - research/feature-ideas.md: an "Inspiration to watch" callout framing ultraplan and pointing at railway-subagents.md for the full design (Temporal analogy + the marks / workflow()-spec / driver options). Both spots are one click from README → research/. Docs only. * feat(agent): PreToolUse tool-contract rail — enforce the declared tools allowlist Closes the declared-vs-enforced gap for subagents (Claude Code #54898): `tools:` frontmatter is documentation, not a runtime boundary. A subagent inherits the session's grants, so the list only filters what's offered. The deterministic layer that actually constrains it is a PreToolUse hook. src/agent-runtime.ts (vigiles agent-hook): parse the active agent's compiled .md frontmatter `tools:` (the single source of truth), and block (exit 2 + the contract fed back to the model) any tool outside it. decidePreToolUse is the pure allow/deny; .vigiles/active-agent.json records the dispatched agent — mirroring the skill Stop-hook (src/skill-runtime.ts). No `tools:` line means inherit-all, so the rail imposes no restriction. The declared list and the enforced rail compile from the same source, so they agree by construction — proven by a round-trip test (compile → parse the frontmatter the hook reads → equals declared tools → allows exactly those). Proven deterministically at the runHook unit tier against the real built CLI process: a tool-event hook is reached cheaply there, where driving a live tool call against a scripted mock is flaky (the e2e defers to it, with a note). +22 tests; full suite 663 passing; 100% stmts/lines/funcs gate holds. * test(agent): ground the PreToolUse rail on a real vendored subagent The runtime suite proved the rail's logic with synthetic specs; add two tests against the REAL pinned wshobson ui-visual-validator under examples/harness/vendor/. It's the documented footgun in the wild: a rigorous visual validator that bases judgments "solely on visual evidence" yet ships with NO tools: line, so it inherits every tool incl. Edit/Write. parseAgentTools on the actual file returns null (the rail honestly reports "no contract yet" rather than inventing one); the spec form ADDS the least-privilege rail the original omits, compiles, and the same rail the hook reads then blocks Write/Edit — the differentiator on a real subagent, not a hand-built fixture. Mirrors src/vendor.test.ts (pinned by SHA, offline, model-free). * docs: refresh testing matrices for the subagent PreToolUse rail Both matrices were stale on the Subagents surface. The new tool-contract rail (agent-hook) is a runHook-testable hook, so the agents/ Hook-unit cell flips from — to ✅; add a Coverage row for src/agent-runtime.test.ts. Kept honest about the end-to-end gap: the deterministic/integration cell stays 🟡 ("rail not live-armed") because Claude Code doesn't surface the active subagent to hooks, so arming .vigiles/active-agent.json in a live session is still unsolved. * feat(railway): railway-oriented subagents — typed Result contracts + finite composition Flat subagents that return either success or error with rich detail on BOTH tracks, composed on a sub-Turing railway. Railway-oriented programming (Wlaschin) with a subagent as the step — uses the flat-worker finding, doesn't contradict it. - result(ok, err) (spec.ts): a subagent's typed Result contract; compileAgent renders it as an ## Output contract section (the vigiles:ok / vigiles:err block the worker must emit). - parseAgentResult (agent-result.ts): pure text -> Result<S,E> (ok | err | malformed), validated against the contract shape. The shared primitive. - railway({ steps, onError, recover }) + delegate() (spec.ts): the composer is deliberately sub-Turing — a finite step list + bounded recovery, NO loop combinator, so termination is structural and every reference is statically checkable (the thing ultraplan's generated script can't be). compileRailway / validateRailway emit the orchestrator command and resolve every delegate target (stale-ref), reject an empty railway, and require recover.max >= 1. - assertAgentOk / assertAgentErr / assertAgentResult (harness-assert.ts): test a subagent's outcome the way you assert a hook decision, reusing parseAgentResult — the testing-framework payoff of the result contract. Deferred (the runtime half): vigiles emits + verifies, it does not drive the railway (that's the engine, Option 3); runtime enforcement of "did the worker emit a valid result block?" awaits what a SubagentStop hook can see. +27 tests; full suite 690 passing; 100% stmts/lines/funcs gate holds. * feat(cli): compile agent + railway specs; dogfood a ship-pr railway Wire the railway surface end-to-end so it's reachable from `vigiles compile`, then dogfood it on this repo. CLI: - loadSpec now returns agent/railway specs too; `vigiles compile` dispatches _specType "agent" → compileAgentToFile and "railway" → compileRailwayToFile. - A railway's delegate() targets are resolved against every agent spec in the project (collectAgentNames), so an unknown target fails compile (stale-ref, exit 1) — verified by a CLI test. Dogfood (examples/railway/): five flat agent() workers, each with a result() contract — planner → implementer → reviewer on the success track, bounded fixer recovery (max 2), reporter on the error track. Compiled THROUGH the real CLI to one .md per agent (with vigiles:ok/err Output contracts) + ship-pr.md (the orchestrator command). The artifacts are committed alongside their specs. Also: compileRailway now emits a trailing newline (prettier-clean, matching the other compilers). +2 CLI tests; full suite 692 passing; 100% stmts/lines/funcs gate holds; fmt + audit clean. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-11 22:50:45 +05:00
zernie	ceb810e881	feat: harness-testing tier 2 (integration) + MCP tool verification (#26) ... * docs: add TOC and subheadings to README for readability Add a collapsible Contents block organized around the two pillars (reference verification + harness testing) so the testing section is easy to jump to. Promote the testing tier lead-ins to ### subheadings so each tier is scannable and linkable from the TOC. * docs: trim README to pitch + quick start, extract reference into docs/ Restructure README around the two pillars back-to-back (verify instruction files → test the harness), then a slim reference tail. Cuts 511 → 389 lines toward the ~300 brevity target. - Move CLI, GitHub Action, plugin, and validation rules into a new docs/cli.md; README keeps a condensed "CLI & CI" section + link. - Move the related-tools/compose list and output-targets into a new docs/related-tools.md; README keeps a one-paragraph summary + link. - Fold the standalone "Type-Safe Rule References" section into a two-line note under Verified References (deep link to linter-support). - Condense the guard() and symbol-reference subsections. - Drop the duplicate init-flags table from Quick Start (now in cli.md). - Update the Contents TOC to match the new structure. * test(harness): dogfood the plugin loader against real pinned plugins Adds runnable dogfood tests that point loadPlugin at vendored snapshots of two real, MIT-licensed Claude Code plugins instead of synthetic look-alikes: - real-superpowers.harness.mjs — obra/superpowers: validates the hooks/hooks.json convention + ${CLAUDE_PLUGIN_ROOT} expansion. - real-wshobson.harness.mjs — a wshobson/agents sub-plugin: the no-hooks subagents+commands+skills shape, asserting the loader materializes the surfaces and warns rather than silently testing an empty machine. Reliability: the plugins are PINNED, VENDORED snapshots under examples/harness/vendor/ (upstream LICENSE + a SOURCE recording repo + commit), so the tests run offline and deterministically — no clone at test time. tools/refresh-vendor.sh re-fetches them deliberately. Safety: loadPlugin only parses; it never executes a hook. The dogfood asserts superpowers' side-effectful SessionStart hook is correctly WIRED and does not run it — executing untrusted third-party hooks needs a real sandbox (the CI container, or an opt-in bwrap/Docker boundary), documented in docs/harness-testing.md. Both tests are key-free and claude-free (pure loadPlugin), so they run in CI for free. Vendored dirs are excluded from prettier + eslint. * chore: regenerate .vigiles/generated.d.ts Refresh the generated type unions to match current project state (adds the test:types npm script and recent project files). `generate-types --check` now passes; no vendored third-party paths leak into the unions. * docs: track sandboxed harness-test execution as a feature idea Add feature-ideas.md #13 — an opt-in `sandbox:` option (bwrap/sandbox-exec/ docker) on runHook/runHarnessTest so the execution tiers can safely run untrusted third-party plugin hooks, turning the dogfood from parse-only into execute-and-verify. Cross-link it from the harness-testing sandbox-boundary note. * docs: add harness-testing coverage matrix (shipped vs. should-build) A roadmap/gap-analysis companion to docs/testing-matrix.md: maps the whole potential surface of testing a Claude Code harness (hooks by event, skills, subagents, slash commands, MCP, CLAUDE.md, settings) across the unit / deterministic-integration / eval tiers, plus cross-cutting capabilities (sandboxed exec, native plugin-install fidelity, MCP wiring, coverage tooling). Marks each cell ✅ shipped / 🟡 partial / 🔴 gap, records the fidelity caveats (skill activation is faked even at the eval tier; Edit/Write headless-gated; no sandbox for untrusted exec), and gives a prioritized build list. Linked from docs/testing-matrix.md and research/harness-testing.md (no orphan). * docs: make coverage matrix use the unit/integration/e2e pyramid honestly Decouple the columns from a single vigiles function each. The prose surfaces (skills, agents, commands, CLAUDE.md) DO have a unit tier — static reference/contract verification via `audit`/`refs` (vigiles' first pillar) — which the old table hid by equating "Unit" with `runHook`. Rename columns to Unit/static · Integration (assembled, mock model) · E2E (real model), and fill the unit column for every surface. Sharpens the read: every surface has a static check, but the integration/wiring middle is hooks+settings only. * docs: frame eval as a separate axis (measurement) in the coverage matrix Eval isn't a "test" in the unit/integration sense — it's measurement (A/B mean ± se you read, not a pass/fail you assert), non-deterministic and costly. Keep it as the third column for surface-coverage completeness (it's the only handle for skills/subagents/commands), but flag the distinction and point its machinery (trials/variance/judge) to the eval docs instead of the cells. * docs: make eval's non-determinism explicit in the coverage matrix * docs: record skill-activation spike — feasible via --plugin-dir Spike against real claude 2.1.169: --plugin-dir natively registers a plugin's skills and the model activates them via a real Skill tool call (superpowers:test-driven-development). Closes the gating unknown for roadmap #1. Also surfaces a bonus deterministic path (scriptModel a Skill tool_use to test wiring) and a fidelity rule (vendor whole plugins). Updates caveat #1 + the native-install row. * docs: spike finds Edit/Write deterministic tier already works (caveat #2 stale) Second spike probe on claude 2.1.169: Edit/Write tool-event hooks DO fire in the mock-model deterministic tier — PostToolUse 3/3, a PreToolUse Edit block held. The 'headless-gated' limitation is no longer reproducible (--allowedTools allowlists the edit tools past the prompt). So roadmap #3 collapses from 'build a driver' to 'add a regression test'. Updates the Edit/Write cell, caveat #2, the spike section, and the build list. * test(harness): lock in Edit/Write deterministic-tier regression tests Replaces the stale `test.skip`ped "Edit/Write headless-gated" placeholder with two real, reliable tests (claude 2.1.169): - Write -> PostToolUse(Write|Edit) hook fires (file written + marker), 5/5. - Read -> Edit, PreToolUse(Edit) blocks (marker proves the hook fired AND the edit didn't apply), 4/4. Found three false-pass/no-op traps while writing them, now documented in the coverage matrix: Edit needs a prior Read; a "file unchanged" assertion is trivial without a hook-fired marker; a custom `prompt` makes the mocked turn no-op (default "go" is reliable). The second planned lock-in test (deterministic skill-wiring) is coupled to the --plugin-dir feature (#1): probing showed .claude/skills materialization does NOT register a skill for the Skill tool, so it ships with #1. Updates the matrix accordingly (Edit/Write row -> shipped; skill-wiring -> coupled to #1). * feat(harness): native plugin install via pluginDir — skills activate + resolve Adds `runHarnessTest({ pluginDir })` which passes `claude --plugin-dir`, so a plugin's skills register and ACTIVATE the real way (file-materialization does not register skills for the Skill tool — only native install does). Plus a `transcript: true` option (stream-json) to assert on what the agent's tools returned. This ships the deterministic half of roadmap #1: a scripted `Skill` tool_use now resolves a plugin's skill (its body is injected), so skill WIRING is testable with no API key. Skills integration cell 🔴→✅. - src/harness-test.ts: pluginDir + transcript options, threaded into the argv. - examples/harness/fixture-skill-plugin/: a 3-file self-contained plugin (one `greet` skill) for the deterministic test. - src/harness-test.test.ts: a test asserting `demo:greet` resolves (3/3 reliable). - docs/harness-testing.md + the coverage matrix: document pluginDir, mark what shipped vs. remaining (runEval pluginDir for real activation; subagents/commands). * feat(eval): native plugin install via pluginDir — closes the skill-activation caveat Adds `pluginDir` to `runEval` arms (mirrors runHarnessTest): an arm can install a plugin natively (`claude --plugin-dir`) so the REAL model activates its skills by description, not via the old "tell the agent to read a SKILL.md" trick. Verified end-to-end with a real-model A/B on the fixture plugin: arm `off` (no plugin) writes NO_SKILL; arm `on` (pluginDir) activates demo:greet and writes its marker. So caveat #1 (skill activation faked) is now closed at BOTH tiers — deterministic resolve (runHarnessTest) and real activation (runEval). - src/eval.ts: pluginDir on EvalArm, threaded into the agent argv. - examples/harness/skill-outcome.eval.mjs: note now points at pluginDir as the real-activation path. - docs + coverage matrix: caveat #1 marked closed; Skills e2e cell -> ✅; also corrected the stale "Edit/Write headless-gated" line in the guide. * docs: probe confirms subagents + slash commands register via --plugin-dir Probed end-to-end (real model): a /cmd slash command wrote its marker, and an echoer subagent invoked via Task wrote its marker — both off a --plugin-dir install. So all model-driven surfaces (skills, subagents, commands) activate natively at the eval tier (their e2e cells ✅). The open work is DETERMINISTIC testing: slash commands need capturing the mock-received (expanded) prompt; subagents are hard (nested mock sessions). Matrix rows + build list updated. * docs: refresh the 'honest read' after subagent/command probe * test(mcp): MCP governance is testable at the unit tier (no server) A PreToolUse hook that blocks a destructive github-MCP tool (mcp__github__merge_pull_request) but allows read-only ones — the dominant real MCP test. The hook only sees the tool name, so runHook covers it with no MCP server and no model. Locks in MCP slice #1 from the coverage roadmap. * feat(harness): assert on the agent's actions (r.toolCalls + tool/skill asserts) Adds the invariant vocabulary the skill/MCP/command surfaces actually need — asserting on what the agent DID, not a brittle stdout substring: - parseToolCalls + HarnessTestResult.toolCalls: parse the stream-json transcript (transcript:true) into the tools the agent invoked, each paired with its result. - assertToolUsed / assertToolNotUsed (the safety negative — "the destructive tool was never called", which "file unchanged" can't prove) / assertSkillResolved. Grounded in REAL plugins for immediate payoff: the skill-wiring test now uses assertSkillResolved (no injected marker), and two new tests assert that real obra/superpowers and wshobson/agents skills resolve via --plugin-dir — something the old marker-grep couldn't do (it only works on fixtures you control). Pure unit tests for parseToolCalls + the three asserts run with no claude. docs/matrix updated. Full suite green; lint + fmt clean. * docs(readme): rewrite the testing section, plainer + more detail Reframe the harness-testing pitch around 'three levels, cheapest first' in plain language (test a hook alone → does it fire in a real session → does it change behaviour), and surface the newly-shipped capabilities: MCP governance at the unit tier, native skill testing via pluginDir, and asserting on the agent's actions (assertSkillResolved / assertToolNotUsed — the safety negative) against real superpowers/wshobson plugins. TOC + anchors updated. * feat(harness): sequence & budget invariants over the agent's actions (idea 1) Adds three assertions over r.toolCalls, beyond the single-bool "did it use X": - assertToolSequence(r, [...]) — ordering, in-order subsequence (e.g. Read→Edit) - assertToolCount(r, name, {min,max,exactly}) — budgets (≤1 Write, 0 of X) - assertToolCalls(r, predicate) — escape hatch for custom invariants like "every Edit was preceded by a Read" (the rule Claude enforces). Pure unit tests (no claude) cover all three; a grounded integration test asserts the workflow on a real Read→Edit run. Documented in docs/harness-testing.md. Parks the two siblings on the roadmap (coverage matrix, "What we should build"): #7 property-based hook fuzzing, #8 monotonic/relational eval invariants. Lint + fmt clean; suites green. * docs(roadmap): triage the next-candidates against existing tooling After checking what already exists: - Skill frontmatter lint + slash-command lint → DROPPED (commoditized: skill-validator, agentskills.io Skill Spec Linter, plugin-validator already do it; vigiles only adds deep ref resolution it already has). Recorded under "Evaluated and dropped". - MCP harness → sharpened into the standout item: a new mcp() reference kind (verify mcp__server__tool against the live server's tool list) + an mcpConfig harness option. The only candidate that's vigiles-shaped, not commoditized. - Triggering eval (measureTriggerRate) → added to the build list (the #1 skill activation pain, built on pluginDir). - Sandbox exec stays parked (feature-ideas §13, out of scope, may reconsider). * feat(mcp): verify MCP tool references against a live server (src/mcp.ts) The vigiles-shaped half of MCP support: a minimal MCP stdio client that starts a server, does the JSON-RPC handshake, and lists its tools — so a referenced `mcp__server__tool` can be verified against the REAL server (like enforce() resolves a linter rule), catching a skill/CLAUDE.md that cites a tool the server renamed or removed (e.g. github MCP create_issue → issue_write). On a miss it suggests the closest tool name via Levenshtein ("did you mean issue_write?"). - src/mcp.ts: listMcpTools + verifyMcpTool (+ ./mcp export). - examples/harness/fixture-mcp-server.mjs: a real minimal stdio MCP server, so tests are deterministic and offline. - src/mcp.test.ts: 5 tests (handshake/list, resolve, typo→suggestion, renamed ref, bad command) — wired into npm test. Also probed live against @modelcontextprotocol/server-everything (13 real tools; "ecko"→["echo"]). Matrix: MCP unit/static cell → tool refs verifiable; build item 5 part (a) done, remaining = spec mcp() builder + audit wiring + mcpConfig harness option. * docs: fix stale 'MCP unwired at every tier' after shipping mcp.ts ref verification * feat(mcp): wire MCP tool-reference verification into `vigiles audit` Adds the `vigiles:mcp server#tool` mark (the MCP analogue of vigiles:symbol) and verifies it in `audit`: parse marks from instruction files' inline spans, read the server config from .mcp.json, start the referenced server, and flag a missing tool (with a Levenshtein "did you mean") or an undeclared server — exit 2, like a broken symbol ref. Only runs when a mark references a server, so zero overhead otherwise. - src/mcp.ts: parseMcpRefs / loadMcpServers / verifyMcpRefs / mcpRefMessage. - src/mcp.test.ts: parse (skips fenced blocks) + verify (ok / typo→suggest / undeclared) against the real fixture server. 7/7. - src/cli.ts: audit step 10 + mcpRefErrors in the report / exit code / summary. Verified end-to-end: a CLAUDE.md citing `vigiles:mcp demo#ekho` → 'not found — did you mean "echo", "add"?'; cli suite 53/53, lint+fmt clean. * docs: MCP unit/static cell → verified in audit (vigiles:mcp mark) * docs(readme): surface the mark family (symbol + mcp) and fold in the matrix - Verified References now lists the inline mark family: `vigiles:symbol file#name` and the new `vigiles:mcp server#tool` (audit starts the server and flags a renamed/missing tool with a "did you mean"). - Add the coverage matrix (surface × tier) to the testing section inside a <details>, so the at-a-glance "what's covered" lives in the README without bloating it; drop the now-redundant matrix link. * docs(readme): keep the coverage matrix visible; tuck reference tables under <details> Matrix (surface × tier) is now a plain visible subsection, not collapsed. Skills list and Maturity-Levels table moved under <details> (with the gist — the 4 level names — left inline), since those are reference, not pitch. * docs(demo): add a runnable 60-second demo (`npm run demo`) The README's "your CLAUDE.md lies" hook, made runnable: examples/demo ships an instruction file that reads fine but whose two references lie — ✗ "refreshSession" is not defined in src/auth.ts (renamed symbol) ✗ MCP tool "helper#purge_all" not found — did you mean "purge"? — while the two truthful refs (verifyToken, helper#log) pass silently. The MCP check starts a real self-contained stdio server from the demo's .mcp.json. This is the distribution lever from research/distribution-strategy.md, now backed by the symbol + MCP verification shipped this session. - examples/demo/: INSTRUCTIONS.md, src/auth.ts, mcp-server.mjs, .mcp.json, run.sh, README.md (self-contained, no network). - `npm run demo` + a README "See it in 60 seconds" callout. - Isolated: tsc only compiles src/**, the repo's own audit doesn't discover it, and its generated .vigiles artifact is git/prettier-ignored. * docs(strategy): close the loop — the runnable demo (proposal A) shipped Append an update to distribution-strategy.md: the '60-second value, output is the product' artifact now exists as examples/demo (npm run demo), built curated + deterministic to sidestep the heuristic-scan trap that got proposal A reconsidered. Notes MCP reference verification as a new distinctive Reach hook, and that the real-plugin dogfood (superpowers/wshobson) is now the evidence base for an E1 'we scanned the top plugins' post. Next gate is still measurement. * docs: add research/ + docs/ indexes, TOC, and README navigation 24 research docs (5.7k lines) and 16 user docs had no index — the real navigation pain with "tons of docs". Rather than risky merges: - research/README.md — themed index of all 24 research docs (positioning, the two pillars, skills, proofs, runtime enforcement, FP, distribution/ideas). - docs/README.md — themed index of the user-facing how-to/reference docs. - README.md — a Documentation section linking both indexes + the coverage roadmap. - A Contents line on the harness-testing coverage matrix (the session centerpiece). Every doc is now reachable from its index, and the indexes from the README (orphan-docs still green, 43 scanned). Left the docs themselves intact — consolidation deferred since the clusters (e.g. harness-testing design vs roadmap vs benchmarks vs user-guide) serve distinct purposes. * docs(feature-ideas): add a ranked, grouped status overview up top Keeps all 13 ideas in place but leads with a "Status & priority" section: marks the 5 already-shipped/partial (dead-enforcement, stale-refs, init, reverse-coverage, annotation-typo), groups the open ones by theme (catch-more- lies / change-safety / ergonomics / skills+harness), and ranks them value × cost. Also clarifies — in the doc — that this is the product backlog (pillar 1), NOT the harness-testing roadmap (which is the coverage matrix). Top open pick: finish #2 Reverse Coverage (moat + distribution artifact). * fix(docs): avoid doc-refs false positive (ctx.file → ctx.sh in an eval example) The doc-refs validator matched `ctx.file("out.txt")` as the spec `file()` builder and flagged a missing file — making `vigiles audit` exit 1 (a CI gate). Switch the example to `ctx.sh("grep …")` (matching the README), which is the real RunContext API and doesn't trip the matcher. audit back to exit 0. * docs(research): prior art on testing non-deterministic AI + corrected model Adds research/testing-nondeterministic-ai.md: a survey of how the field tests non-deterministic AI/agent tools (DeepEval, Braintrust, Promptfoo, τ-bench's pass^k + state-over-self-report, AgentAssay, procedure-aware "corrupt success"), and the model it implies for vigiles. Corrects two conflations from an earlier sketch: - predicates (bare, return values) are SEPARATE from assert* (throw); only the testing helpers carry the assert prefix. - testing (pass/fail, every commit) and eval (mean±se/pass^k, paid) are SEPARATE consumers that share predicates over one Trace — not one dual-purpose API. Folds the gaps into the coverage-matrix roadmap (#10–14): unify the Trace, capture output + hooks, split predicates from assert*, add pass^k + tool-arg assertions. Linked from the research index (no orphan). * docs: fix markdown rendering (stray '- throw' bullet) in the prior-art note * fix(docs): resolve orphan-doc — reference new doc via full-path backtick The orphan checker matches `research/foo.md` (it doesn't resolve relative links), so the bare-basename link to testing-nondeterministic-ai.md didn't count and audit exited 1 (failing CI's 'check' job). Use the repo convention — full path in the backtick span, bare href — like the other research cross-links. * docs(roadmap): save the Trace-unification execution plan for the follow-up Records the concrete plan for the next session: after #26 merges, branch off main; do the additive pieces (export predicates + assert* wrappers, unify runEval ctx with tools/output, r.output, pass^k, tool-arg asserts) which are non-breaking; defer trace.hooks (needs new instrumentation); keep predicates vs assert* and testing vs eval separate; acceptance = suites green. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-10 01:48:04 +05:00
zernie	7859fb1a6f	feat: test & verify your Claude Code harness — testing pillar (CLI, plugin loader, runner-agnostic, evals) (#25) ... * feat: vigiles test/eval CLI + canonical harness examples + two-pillar positioning Make the harness-testing pillar usable and concrete: - CLI: `vigiles test` (discovers *.harness.mjs) and `vigiles eval` (discovers *.eval.mjs) run the two-tier harness scripts as child node processes and aggregate exit codes, so they work as a CI command, not just `node x.mjs`. `vigiles test` skips clean without the claude CLI; `--trials=N` is forwarded to eval scripts via VIGILES_TRIALS. New module src/run-scripts.ts (discover/run/format) + test suite. - Canonical examples under examples/harness/, real-plugin-shaped: policy-gate.harness.mjs (deterministic PreToolUse Bash block-no-verify gate + SessionStart setup hook) and skill-outcome.eval.mjs (does a skill change the agent's output?). refs-hook.eval.mjs reads VIGILES_TRIALS so it runs under the CLI too. - Positioning: README + CLAUDE.md(.spec.ts) reframed around "test & verify your Claude Code harness" — verification and evals as two pillars; testing section links the canonical examples and CLI commands. * chore: dogfood the harness-testing pillar on the in-repo build - npm scripts test:harness / test:eval run the new CLI via the in-repo build (`node dist/cli.js`), never the published `npx vigiles`, so the repo dogfoods its own current code. - CI `harness` job installs the claude CLI and runs the deterministic tier against the scripted mock model — no API key, zero cost. The real-model eval tier stays manual (`npm run test:eval`). - Document both scripts in CLAUDE.md (compile-verified against package.json); regenerate .vigiles/generated.d.ts. Verified: `npm run test:harness` builds and runs examples/harness/ policy-gate.harness.mjs green (PreToolUse Bash gate + SessionStart hook). * feat: cohesion testing (plugin loader), runner-agnostic helpers, eval variance, LLM-judge Make the harness-testing pillar test the *whole machine*, not one hook, and fit whatever runner you already have. - Plugin/repo loader (src/plugin-loader.ts): runHarnessTest/runEval accept a `plugin` path that loads the real harness — hooks from .claude-plugin/plugin.json (or .claude/settings.json) with ${CLAUDE_PLUGIN_ROOT} resolved to the real scripts, plus CLAUDE.md and skills — so you test what ships. Inline settings/files layer on top (per-event hook arrays concatenated). resolveHarness shared by both tiers. - Runner-agnostic helpers (src/harness-assert.ts): withHarness (auto-cleanup), throwing assert* helpers usable in any runner incl. node:test, and vigilesMatchers for vitest/jest expect.extend (one object, both runners). The library depends on no test runner. - Eval variance (src/eval.ts): aggregateStats returns mean/std/se/n; reports render `metric=mean±se` so an A/B gap can be read against the noise floor. - LLM-as-judge (src/judge.ts): thin synchronous judge() for subjective outcomes inside measure(); deliberately not a platform. - Canonical cohesion example: examples/harness/plugin-cohesion.harness.mjs + fixture-plugin — one plugin load, SessionStart + PreToolUse Bash gate both fire. docs/harness-testing.md ties the three layers together. Verified: full suite 514 pass; plugin loader dogfooded on vigiles's own plugin.json (unit) and end-to-end (the cohesion example passes via `vigiles test`); audit/lint/fmt clean. New package exports: ./plugin-loader, ./harness-assert, ./judge. * test: enforce multi-source + multi-runner support (plugin/repo, vitest/jest) Lock in the testing API's reach with constraints, not just docs: - Source support: plugin-loader tests now cover a plain repo (.claude/settings.json, no plugin manifest), manifest-wins precedence when both exist, and a bare dir (CLAUDE.md only → empty settings) — alongside the existing plugin-manifest + in-repo dogfood cases. - Runner support: the same vigilesMatchers object is exercised under BOTH vitest (test/runners/matchers.vitest.mjs) and jest (test/runners/matchers.jest.cjs), proving the library is runner-agnostic. The dist is CommonJS, so jest requires it natively (no ESM flags / babel; transform: {} in jest.config.cjs). Each runner is scoped to test/runners so it never picks up the src/*.test.ts node:test suites. New scripts test:vitest / test:jest; both run in CI. vitest + jest added as dev deps. Verified: node:test 516 pass, vitest 2 pass, jest 2 pass; lint/audit/fmt clean. * feat: opt-in vigiles/vitest + vigiles/jest integration entries (subpath exports) Per the subpath-exports decision, ship per-runner integration entries that register the matchers AND add their TypeScript types — without coupling the core to any runner. - vigiles/vitest (src/vitest.mts → ESM, since vitest is ESM-only): registers vigilesMatchers + augments @vitest/expect's Matchers so toHaveCreated / toBeatBaseline type-check. - vigiles/jest (src/jest.ts → CJS, required natively): same, augmenting @jest/expect's Matchers. - vitest + jest declared as OPTIONAL peer dependencies — only the entry you import pulls one in. - Type-level constraint: test/types/{smoke.vitest,smoke.jest}.ts compile under `tsc --noEmit` (npm run test:types), with @ts-expect-error guards proving the matchers are real types, not any. Wired into CI alongside the vitest/jest runner tests. - tsconfig include now matches .mts; package files ship .mjs/.d.mts. Verified: node:test 516 pass, vitest 2, jest 2, type smoke compiles, lint/fmt/ audit clean. * feat: plugin loader handles real-world hook layouts (dogfooded on obra/superpowers) Dogfooding the loader on a real popular plugin (obra/superpowers, 5.1.0) surfaced that it keeps hooks in hooks/hooks.json — the Claude Code plugin convention — which our loader didn't read (it only handled inline plugin.json hooks). Now loadPlugin resolves, in order: 1. inline `hooks` object in .claude-plugin/plugin.json, 2. a `hooks` string path in plugin.json, 3. the hooks/hooks.json convention (superpowers, etc.), 4. a plain repo's .claude/settings.json. Verified live: loadPlugin("/tmp/superpowers") surfaces its SessionStart hook with ${CLAUDE_PLUGIN_ROOT} resolved and materializes its 14 skills. Added fixture-based tests for the convention + string-path layouts (no network in CI). Also fixes a latent require-spec=error failure (since the cohesion commit): the example fixture-plugin/CLAUDE.md now carries <!-- vigiles-disable require-spec --> like the other fixtures, since it represents a third-party plugin, not a vigiles-managed instruction file. Backticked the glob tokens (*.harness.mjs etc.) in the spec so the compiled CLAUDE.md is prettier-stable (hash holds). Verified: node:test 518, vitest 2, jest 2, type smoke + lint + fmt + audit clean. * test: close the harness-testing coverage gaps (judge, entries, assertServedTurns) A coverage audit found three untested use cases; this fills them: - judge.ts had zero tests. Extracted parseJudgeOutput() as a pure verdict parser and tested it: claude --output-format json result-field unwrap, bare/prose-wrapped JSON, custom threshold, [0,1] clamping, and safe fallback on empty/unparseable/score-less output. (The model spawn stays integration-only.) - The vitest/jest runner tests registered matchers by hand, so the opt-in entries' auto-register was never exercised. They now load `vigiles/vitest` via setupFiles and `vigiles/jest` via setupFilesAfterEnv (the way users do), so the entries themselves are under test. - assertServedTurns was untested — added. Verified: node:test 524 pass / 1 skip, vitest 2, jest 2, type smoke + lint + fmt + audit clean. Remaining intrinsically-integration cases (need claude/model, covered by the examples in the `harness` CI job, not unit): runHarnessTest incl. plugin:, runEval incl. plugin arm, withHarness, and judge()'s model call. * docs: testing-matrix doc + exercise withHarness in the cohesion example - examples/harness/plugin-cohesion.harness.mjs now uses withHarness (+ the assertCreated helper), so the auto-cleanup wrapper is actually exercised in the `harness` CI job — closing the last coverage gap. Comments state the trade-off plainly: withHarness guarantees sandbox cleanup even on a thrown assertion (no leaked temp dirs) but assertions run in a callback and the first failure aborts the rest; use raw runHarnessTest + r.cleanup() when you want per-check reporting (see policy-gate). - New docs/testing-matrix.md: every use case → its test tier (unit / cross-runner / type / integration-CI) and the file that proves it, what's intentionally not unit-tested and why, and why the CLI examples are .mjs while the API + runner usage are TypeScript. Linked from the guide + CLAUDE.md. Verified: cohesion example passes via `vigiles test`; fmt/audit/lint clean. * chore: sync package-lock with optional jest/vitest peerDeps The lockfile did not record the optional jest/vitest peerDependencies declared in package.json; npm install reconciled it. * feat: add hook unit tier (runHook) and make plugin loader surface-aware Two gaps surfaced by dogfooding the testing API against obra/superpowers and the wshobson/agents marketplace: 1. No unit tier. Both existing tiers spawn the real claude CLI, so even 'does guard.sh block --no-verify' needed the binary, and Edit/Write + PreCompact/Notification/SessionEnd/SubagentStop events were unreachable (the mock can't trigger them). runHook pipes a synthesized event JSON straight to the hook process and reports the normalized block/allow decision — no claude, no model, milliseconds, every event. Pure parseHookOutput/decideHook are unit-tested; assertHookBlocked/ assertHookAllowed + a toBlock matcher added. 2. loadPlugin silently dropped agents/ and commands/ — so a subagents-only plugin like tdd-workflows (2 agents + 4 commands, no hooks) loaded an empty machine and any test 'passed' having exercised nothing. The loader now materializes agents/ and commands/ too and exposes LoadedPlugin. warnings flagging surfaces only a real model can drive (subagents, slash commands, MCP) or an outright empty load. Docs: harness-testing.md gains the unit tier + surface-coverage note; testing-matrix.md gains a surface x tier matrix. Canonical example hook-unit.harness.mjs runs in CI with no claude. CLAUDE.md recompiled. * docs: wire the unit tier through README/guides + fix jest config key - Fix a real bug: the jest setup key is setupFilesAfterEnv, not setupFilesAfterEach (docs/harness-testing.md + src/jest.ts docstring). - Reconcile tier/layer counts now that runHook is a third testing tier: README 'two levels' -> 'three levels'; testing-matrix 'three layers' -> 'four layers'; harness-testing see-also drops the stale 'two-tier' label. - Add the unit tier (runHook) to the README with a snippet, and note loadPlugin().warnings + subagent/command materialization in the 'test the whole machine' section. * docs: bring harness-testing research doc + example spec up to date - research/harness-testing.md: rewrite from 'two tiers' to the shipped three-tier design (unit runHook / deterministic runHarnessTest / eval runEval), add a 'plugin loader' section covering the assembled-machine loader + surface warnings, and fold in the wshobson/agents dogfood finding (subagents/commands dominate; tdd-workflows empty-machine trap). - examples/CLAUDE.md(.spec.ts): fix stale API in the teaching example — there is no check() builder (the real rule kinds are enforce/guard/ guidance), and the CLI is init/compile/audit/test/eval/generate-types, not check/discover/adopt. Recompiled. - CLAUDE.md.spec.ts: refresh the research-doc key-file description to the three-tier framing. Recompiled. * docs: reconcile rule-type count to three (enforce/guard/guidance) guard() is a real, exported rule builder with three live uses in the root spec, and spec-format.md already documents it as one of 'three builders' — but the root CLAUDE.md architecture said 'Two rule types' and two sample lines in spec-format lagged. Align them all on enforce()/guard()/guidance() so the canonical doc, the format reference, and the example agree. Root CLAUDE.md recompiled. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-09 13:42:07 +05:00