mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
v26.2.0
44 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
195c1e14b7 |
feat: launch blockers, a hook-state test API, and the site's missing proof (#210)
Ten launch blockers, all sharing one cause: a first reader gets a false claim or an embarrassing failure in their first ten minutes. Fixes: ruff enabled-state was never read on a real repo; compile was unreachable without node_modules/vigiles (every Python/Rust/Go repo); a hooks-only plugin graded F/0 while its own report listed inlineHooks: 2; skill-resource findings printed a body-relative line number; the README quoted a caveman number its own FINDINGS.md had invalidated; STABILITY.md claimed v21 against 26.0.1; the Action's floating v1 tag was never created; a react's notice() reached nobody. Features: experimental_hookState, so a stateful compiled hook can be tested by backdating the fact it throttles on; the site now leads with the proof (compiled hooks, the disaster battery, 2/7) and its OG card is rendered rather than hand-drawn, with a generator that fails loud if the page fetches anything off file:. Also: this repository's first compiled hook, which is what exposed the notice-delivery defect; a floor under the harness CI job, which was green on zero files; and a smoke test of the PUBLISHED package, since every gate until now ran dist/. |
||
|
zernie
|
7dacd8f402 |
docs: finish the #34692 correction, and give every site the same narrow scope (#196)
The subagent-bypass fix was recorded in one place and left stale in four others, so the docs understated the product while contradicting themselves. Corrected — these stated the old limit as CURRENT: - CLAUDE.md keyFiles for hook-program.ts, guardrail-check.ts and guards.ts, each contradicting the already-corrected header of the very file it describes. - docs/compiled-hooks.md's cons summary listed "delivery (#34692)" as a live limit eight lines above the section explaining it was fixed. Scope added where the correction had none. src/subagent-delivery.test.ts measures headless `claude -p` only; interactive is unmeasured and depth-2 nesting does not occur there. STABILITY.md, docs/cli.md, src/hook.ts, src/core/hook-program.ts and src/guardrail-check.ts said "fixed as of 2.1.241" without that limit — one overclaim replacing another. Unchanged because it still holds: a model can route around a tool entirely (#45427 / #32376), so a gate is a strong default, never an unbypassable wall. Every dated record of what was once true stays. Not refactored, flagged only: agent-runtime.ts reconstructs the active subagent from a dispatch-time stack, while the event now carries `agent_type` directly — a simplification opportunity in a rail already marked experimental and parked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
d6b1ad06aa |
fix!: exclude reaches every pass, and the guard stops missing re-spelled commands (#194)
`.vigilesrc.json#exclude` was honoured by 3 of 11 discovery walks, and the walks that did honour it disagreed on the dialect: glob's string `ignore` treated bare `bench` and `bench/` as matching nothing, while the minimatch helper accepted them. One `ExcludeSet` is now parsed where `loadConfig()` runs and taken as a REQUIRED parameter by every in-scope walk in cli.ts — the shape that let `findSpecs` go a year without it was optional-with-default, which compiles silently. Closes #192. The disaster battery now generates its own inputs: `experimental_alternateSpellings` (renamed from `equivalentDisasters`) re-spells hook test cases so the shell executes them identically — nine families, 7 seeds to 136 spellings. Five of those came from a normalizer fix: `g\it push --force` runs git but was not matched by `runs("git push")`. That fix then had to be scoped, because CI caught the regression it caused: escapes are resolved for the HEAD and FLAG-shaped operands (what the command IS) and never for a path operand (where it WRITES) — the same bytes are a resolved escape to `sh` and a real Windows UNC path on Windows, and fail-closed keeps the path. Both directions pinned by mutation. Also: `spec-refs` and `duplicate-rules` had a meta and a doc but no row in the canonical rules matrix, and nothing compared either against it. A test now asserts every rule has a matrix ROW (matched on the table row, not a prose mention). BREAKING CHANGE: `findSpecs` changed signature; `orphans` gained a `repoExclude` option; `equivalentDisasters` was renamed to `experimental_alternateSpellings` and now takes a required argument. Any repo with an `exclude` key sees it applied to every pass, so an excluded spec is no longer compiled and an excluded harness/eval script is no longer discovered. |
||
|
zernie
|
85c54e7e78 |
fix(lint)!: make the lint report complete, tierable and machine-readable (#187)
Closes #181, closes #182, closes #183, closes #184, closes #185, and the residual half of #173. #185 — every per-surface check read ONE root, so a monorepo with `skills/` plus `plugins/*/skills/` scored only the first: 4 skills over budget, 2 reported, exit 0. The default now NAMES the bundles it did not score; `--bundles=all` scores them in one pass with one exit code. Root-only stays the default so a vendored third-party corpus is not scored as if it were yours. #181 — `orphan-docs` had a documented severity nothing read, and the duplicate-rules check had no id at all, so neither could be tiered. Both are real rules now and the exit code respects their severity. #183 — the log and `--json` disagreed (21 lines against 88 findings) because some checks print per finding and others per group. One computed total, printed and carried in `--json`. #182 — `--json-out=<file>` writes the JSON while stdout stays human-readable, so a CI job stops scanning twice. #184 — `vigiles test` resolves a harness's bare `vigiles` import from the CLI's own install, so `npx vigiles@<version> test` needs nothing in the project. NODE_PATH was measured, not assumed: Node ignores it for ESM. #173 residual — `lint` verified the integrity HASH, which says nothing about whether the things a file NAMES still exist, so an artifact committed while its refs were live stayed green after the target was deleted. The new `spec-refs` rule re-derives them. It immediately found four dead references in this repo's own `examples/CLAUDE.md.spec.ts`, left behind by the hexagonal refactor. BREAKING CHANGE: `orphan-docs` and `duplicate-rules` now default to "warn" — a repo relying on either to fail CI must set "error" explicitly. Registering them surfaced that both are heuristic proxies, and this repo's calibration rule is that a heuristic never defaults to error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
94af23a5b7 |
feat(hook,cli)!: close five adopter-reported issues — halted-turn guards, dead-ref artifacts, centralized tests (#186)
Closes #173, #174, #175, #176 and #170. #174 — decideHook read exit 2, `decision` and `permissionDecision` but not the harness's halt-the-turn field, so `assertBlocksDisasters` reported a guard that stopped every disaster as blocking none. The field is read from the PORT, not hard-coded: `continue` is documented for Claude Code and unverified for Codex. The issue's adjacent suggestion was deliberately NOT implemented — adding the halt to `hook-block-ineffective` as a block-attempt signal would fire on hooks that WORK, since a halt is not event-scoped. It reads as a suppressor instead, fixing a real false alarm. A mutation implementing the original suggestion literally is pinned by a test that fails on it. #173 — compile printed its errors, exited 1, and wrote a hash-valid CLAUDE.md anyway, so `lint` went green over refs it had just called dead. Nothing is written on a failed compile now. #176 — eight small defects, including `audit --out` outside the repo appending unreachable `../../..` entries to .gitignore (a documented read-only report editing a tracked file), an unknown command exiting 1 against a documented 2, and STABILITY.md promising the removed `scan` verb. #175 + #176.2 — three missing capabilities, each opt-in and additive: `{surface}` in testGlobs, `--single`, and `rulesDir` so `.claude/rules/**` stops being invisible. #170 — one enumeration of export forms written against the grammar, both naming checks driven off it, and `check-internal-tag.mjs` asking the TypeScript compiler instead of a regex. It immediately found the issue's open case, 13 tagged types the regex had never seen, and a live gap in the ESLint rule. BREAKING CHANGE: `HookRunResult` gains a required `haltsTurn`, so code that CONSTRUCTS one (a test fake) must set it. Reading a result is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
681edf7673 |
fix(hook): the subagent-delivery caveat is stale, and now a test says so (#172)
* fix(hook): the subagent-delivery caveat is stale, and now a test says so Claude Code #34692 — a PreToolUse hook not firing for a subagent's tool calls — was closed not-planned and quoted across these docs for months as a standing limit on what a gate can promise. It is fixed. Measured against a stock @anthropic-ai/claude-code@2.1.241 installed from the registry, not the container's binary: a subagent's own Bash reaches the hook and an exit-2 deny stops it, with the parent's identical command in the same run as the control and the marker files on disk as ground truth. The event also carries agent_type, naming which subagent made the call. The claim is about someone else's product, so prose cannot keep it honest — a doc cannot notice that the platform moved. src/subagent-delivery.test.ts pins both directions and goes red on a regression; removing the deny from its hook fails it on its own assertion. What did NOT change is stated at every site that was edited: a model can still route around a tool entirely, so a gate remains a strong default and is never unbypassable. The measurement's scope is stated too — headless only, with interactive sessions unmeasured and depth-2 nesting absent there. Gates run, in order: vitest (3282 passed), lint (0 errors), prettier --check, docs:check, internal:check, api:check, build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: regenerate the committed project-file types for the new test CI's generated-types:committed gate compares a fresh `generate types` run against the committed file, and adding a source file changes that list. Caught by CI rather than locally because I ran the individual commands instead of `npm run check`, which is the aggregate CI actually runs — the same shortcut this repo has already written down as a recurring miss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7c97e4ed7d |
refactor(spec)!: one experimental root per feature (#171)
The prefix answers "is this name stable?" and cannot answer "is the name my stable symbol depends on stable?". That gap shipped: railway(), delegate() and result() carried stable names while being meaningless without the experimental builder, and the lint rule cannot see it — it compares tag and name within one declaration, not the dependency between two. So the subagent vocabulary now hangs off one marked root: result, railway, delegate, pipe, pipeStep, needs, start and andThen become members of the builder. Eight names leave the top level. Two further gains the prefix alone could not reach: collision-prone words stop owning package-level names, and the warning survives destructuring, since the binding site still names the root — unlike a namespace object or an import subpath, which mark the import line. Object.assign rather than a TypeScript namespace: namespaces are an error in this repo's lint config, and one merges only with function declarations while the skill builder is a const. Types stay top-level, for the same reason the prefix rule excludes them. No deprecation window: nothing outside this repository imports these, and a window subtracts nothing. The container-rollback argument for a window does not reach them — they take no part in loading hooks. Found along the way, and worth more than the refactor: two example specs had not compiled since the previous rename, because the examples directory was in no tsconfig at all. Both fixed, and the class is closed — examples now type-check. Gates run, in order: vitest (3281 passed), lint (0 errors), prettier --check, api:check, build, exports:check, internal:check, docs:check, and tsc over the types project now that it includes examples. Claude-Session: https://claude.ai/code/session_017aUJEoaEQ3DxtvoSVWAWqe Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
6532dbf48d |
fix: five defects found dogfooding vigiles on a real consumer repo, plus CI job bounds (#162)
Five defects found dogfooding vigiles on a real consumer repo (zernie/mine — 43 skills,
48 harness files), each measured rather than reasoned about:
1. `exclude` never reached the markdown-refs pass. Measured: 9 broken refs, 8 of them
inside a directory explicitly listed in `exclude`, all 9 still reported. After: 9 → 0.
The consequence was larger than the cause — with no way to reach exit 0 the consumer
set `continue-on-error: true`, and a lint whose exit code is discarded gates nothing.
2. Refs were found by TEXT, per line. Four of six probe inputs classified wrongly
(`ctx.file("OUT")`, `obj.cmd(...)`, a comment, a string), and because the regex ran
per line, any call split across lines was invisible — which had hidden two genuinely
broken refs in this repo's own docs. Now parsed: a ref is a call whose callee is a
bare identifier with a string-literal first argument. `typescript` was already a
runtime dependency. Net: 21 refs → 25, all valid.
3. The script runner was a strict queue. Measured on 48 harness files: 270s → 77s
(3.5x), byte-identical results. The default follows `entry` rather than a new flag:
`test` fans out (free and deterministic by construction — mock model, no API key),
`eval` stays at 1 (real model quota). `stdio: "inherit"` was the actual blocker.
4. Agent path attribution was keyed by name, so with `agents/foo.md` and
`.claude/agents/foo.md` a lethal-trifecta finding was displayed against the wrong
file. Ambiguous names now yield no path instead of an arbitrary one.
5. `compile` emitted invalid YAML. A description containing `: ` produced frontmatter
where `skillContract()` reports `malformed: true, declared: []` — the blessed path
producing the exact defect `frontmatter-valid` exists to report, and punishing anyone
who fixed it by hand with a hash mismatch. Fixed by round-tripping through the same
loader the linter uses, so already-safe values stay byte-identical and no integrity
hash moves.
Plus: all 10 jobs across 6 workflows were unbounded and now carry `timeout-minutes`
sized to measured durations; docs updated in the pages that own each defect.
Gates: vitest 3186 passed - lint 0 errors - prettier clean - api:check no drift - build.
CI: all 7 jobs green.
|
||
|
zernie
|
ad413b0ac5 |
fix(scan): the fix-scan work, merged onto current main (#161)
* fix(scan): two ways a surface that is there was invisible to audit
A marketplace listing suppressed the audit of the directory it sat in, and
`agents/` was read one level deep though the harness reads it recursively.
Both end the same way: vigiles reported nothing, at exit 0, about surfaces
that were on disk — and silence is this tool's success signal, so neither
failure could be seen from the outside.
MARKETPLACE (`audit`, cli.ts). A `marketplace.json` whose members are all
external expanded to zero targets, and a branch printed "nothing to scan"
and returned BEFORE looking at the directory. But that file is a statement
about OTHER directories: a repo may list external members and also be a
plugin. Measured on nyldn/claude-octopus @57cfb9b0 (3,979 stars): 59 skills,
50 subagents, 40 hooks, 54 commands — 203 surfaces — suppressed, exit 0.
The branch is deleted; the `targets` fallback that already existed does the
work. Proof the suppression is gone: deleting the marketplace.json now
yields the byte-identical report, so the file no longer changes the outcome.
A genuinely curated marketplace reaches the existing `score.empty` branch,
which carries the "clone a member plugin" advice and exits 2 — this repo's
own rule that 2 means "I could not do what you asked". `--json` still emits
`kind:"marketplace"` there, for exactly the case it describes.
AGENTS RECURSION (core/layout.ts + its three readers). The vendor documents
`agents/` as scanned recursively, with the subfolder forming the identifier
(`agents/review/security.md` -> `review:security`). vigiles matched
`agents/[^/]+\.md` — in THREE places that each spelled the rule themselves:
the scan classifier and both coverage discoverers. Measured on
rsmdt/the-startup @88d447c7: 16 agent files, 2 read, and the plugin was
still graded B (80/100) over that 12.5%. The rule now has one home,
AGENT_FILE_LEAF_RE, quoted by all three, so the subagent count and the
untested count can no longer be computed over different sets of files.
Agent names become the documented scoped identifier. That is not cosmetic:
a basename cannot be unique once a dir is read recursively, and the
delegation graph keys agents BY NAME, so `agents/a/review.md` and
`agents/b/review.md` would have swallowed each other. Top-level agents keep
their bare filename, which is why no existing report moves.
Two defects the recursion itself introduced, both closed: a `SKILL.md` under
`agents/` matched both classifiers and was counted twice (the harness loads
it as an agent, so it is excluded from skills — the mirror of the existing
skills/<x>/agents/ rule); and the on-disk discoverer normalized separators
for the pattern but not for the name, which on Windows would have silently
fallen back to a basename.
Folded in: a nonexistent path now says it does not exist rather than "no
surface was found there" — one existsSync, its own test.
GRADES MOVE, in the harsher direction, on 2 of the 32 dogfood repos (the
other 30 are byte-identical):
nyldn/claude-octopus suppressed, no grade -> D (60/100), 203 surfaces
rsmdt/the-startup member `team`: B (80) -> D (60), 2 -> 16 agents,
and 2 -> 14 malformed-frontmatter findings
No exit code moved anywhere in the corpus. bench/leaderboard/RESULTS.md is
unaffected: both its pinned repos have zero nested agents (measured).
Tests: 10 new cases across scan/parity/layout/e2e, each proven by mutation
with the patch asserted to have landed. The parity fixture is load-bearing —
reverting only the browser-side discoverer survives the entire pre-existing
suite and dies only on the new nested-agent parity case.
Gates: vitest 2991 passed - lint 0 errors - prettier clean - api:check no
drift - build ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
The same two bytes as on main. Both sides of a merge must be text before git
will do a three-way merge at all — normalising only one side leaves the blob
pair binary.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b9302db4e6 |
feat!: one instability marker — the experimental_ name prefix, enforced (#147)
* docs: генераторный режим убран из публичной доки и припаркован `skill()` помечен экспериментальным. Из `docs/skills.md` убраны §3 (генератор) и секция тестов `vigiles/skill-test`, которая на нём стоит: `genSkill`/`act`/`checkpoint`/`finish`/`runSkill` не достижимы ни из одного сабпаса пакета — то есть снаружи репозитория такой скилл не написать. Ровно эти два фантомных сабпаса и заставляли держать в шапке рукописный баннер; баннер снят вместе с ними. Замерено при этом, и записано в §Status как причины экспериментального статуса: - `vigiles compile` генераторы ПОДДЕРЖИВАЕТ (`cli.ts` ловит `genSkill(`, прогнано end-to-end) — прежняя строчка §Status «wiring it into `vigiles compile` … is the next integration step» устарела; - скомпилированный `SKILL.md` ни разу не был показан загружающимся как скилл: `vigiles:sha256:`-шапка отодвигает YAML-фронтматтер с первой строки, и читатель, привязанный к `^---`, не находит фронтматтера вовсе. Все работающие `SKILL.md`, включая 13 своих, начинаются с `---`; шапку несут только два файла в `examples/`, которые никто не грузит; - принятие всё-или-ничего: `renderSkillSections` собирает документ целиком в фиксированном порядке, пути «моя проза + один проверяемый гейт» нет. Паркинг помечен в шапках трёх модулей, чтобы его не подняли молча. Публично экспортировать нечего: замер показал ноль достижимых символов. `scriptModel` НЕ паркуется — он публичен из `vigiles/claude-code` независимо от генератора. Заодно — дефект гейта из #146, найденный этой же правкой: он переиспользовал маркер `vigiles:ignore-file`, который принадлежит резолверу ссылок. Из-за этого единственный документ с двумя доказуемо битыми импортами был единственным, которого гейт не видел (91 блок сканировался, этот — нет). У гейта теперь свой маркер `vigiles:ignore-imports`; на дореформенном тексте он выдаёт ровно те два TS2307, ради которых писался баннер, на текущем корпусе молчит (92 блока, 0 находок). Тест на разделение маркеров прогнан мутацией: возврат общего маркера роняет оба ассерта. Гейты: vitest 2928 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · check-doc-imports 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: skill() → experimental_skill(), и правило, которое это держит BREAKING: `skill()` переименован в `experimental_skill()` на `vigiles/spec`, `vigiles`, `vigiles/linting` и `vigiles/claude-code`. Пользователей нет, мажор бампаем. Почему: дока открывалась словами «`skill()` is experimental», а имя на экспорте говорило «стабильный». Проза против механики — ровно тот дефект, который vigiles и ищет, у vigiles. Соглашение о префиксе существовало (`experimental_emitTool`, `experimental_parseEmitted`, `experimental_withServices`), нигде не было записано и ничем не проверялось. Побочно рейм снимает КОЛЛИЗИЮ, которой не должно было быть: `skill()` экспортировался из ВОСЬМИ сабпасов и был ДВУМЯ разными функциями — билдером спеки (`vigiles/spec`) и `Check<Trace>` по трассе (`vigiles/testing`, принимает id-строку, отвечает «сработал ли скилл»). Одно слово, два понятия, различимые только дверью импорта. Префикс несёт только билдер; `Check` не тронут (`vigiles-testing.api.md` без изменений). Правило: `scripts/check-experimental-naming.mjs` + шаг CI + `npm run experimental:check`. Публичное объявление с TSDoc-тегом `@experimental` обязано начинаться с `experimental_`. Ни одного нового маркера: тег уже используется в `experimental-emit.ts` и `services.ts`, публичность берётся из закоммиченных `api-surface/*.api.md`, которые `api:check` и так держит честными. Типы вне области: на дореформенной поверхности префикс несли только функции, а `ServiceSpec`/`ServiceHandle`/`ContainerRuntime` — нет; с типами гейт открылся бы 6 косметическими переименованиями против 1 настоящей находки и был бы заглушен в тот же день. Настоящая находка одна и закрыта: `makeDockerRuntime` → `experimental_makeDockerRuntime`. 8 тестов гейта, обе половины у каждого. Один нашёл баг в самом гейте: `\s+\S` в маркере-исключении удовлетворялся переводом строки и звёздочкой продолжения JSDoc, то есть голый маркер без причины проходил. Поправка к предыдущему коммиту: §Status утверждал, что sha256-шапка мешает загрузке скилла. Замер это ОПРОВЕРГАЕТ для соседней поверхности — в `reviewer-ab.eval.mjs` живой Claude Code загрузил скомпилированный `agents/code-reviewer.md` С шапкой через `--plugin-dir` и отдиспатчил в него, 100% прогонов на реальном sonnet (2026-06-20). Формулировка сужена до «непроверено», а не «сломано». Гейты: vitest 2936 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал устаревший импорт в docs/railway-subagents.md — правка внесена) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * docs: STABILITY.md объявлял стабильным то, что переименовано и помечено экспериментальным Найдено субагентом при разборе неймспейс-идеи, подтверждено мной. `STABILITY.md` — документ под заголовком «What's stable — depend on it» — перечислял `skill` среди стабильных билдеров `vigiles/spec`. После переименования такого символа нет вообще, а то, во что он переименован, в предыдущем коммите объявлено экспериментальным. То есть контракт стабильности обещал стабильность несуществующему имени. Третий экземпляр «проза против механики» в этой же области за два дня, и мой гейт его НЕ ловит: он сверяет TSDoc-тег с именем, а не документ с API. Заодно записано то, что вскрылось при проверке: - `input()` и `step()` используются ТОЛЬКО скилл-спеками, но экспортируются без префикса и без тега — то есть словарь экспериментальной фичи выглядит стабильным. Слепая зона гейта: он энфорсит «тег ⇒ имя», но не «на всём, что должно, стоит тег». - Маркеров нестабильности в проекте ТРИ и правила выбора между ними нет: `@internal` (39 объявлений), `@experimental` (18), префикс `experimental_` (8 публичных символов). Секция больше не утверждает, что маркер один. - У `pipe` тег `@internal` стоит только на ПЕРВОЙ перегрузке, поэтому остальные в api-репорте читаются как `@public (undocumented)`. Гейты: prettier ✓ · docs:check 0 · experimental:check 0 · lint 0 errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: один маркер нестабильности вместо трёх — префикс имени BREAKING: `pipe` `pipeStep` `needs` `start` `andThen` `effect` → `experimental_*` на `vigiles` и `vigiles/linting`. Пользователей нет. ## Выбрано **Экспортируем и не стабильно — имя начинается с `experimental_`.** Тег `@experimental` объявляет это один раз, `experimental:check` роняет CI, если у помеченного публичного символа нет префикса — тег и имя разъехаться не могут. `@internal` перестал означать «нестабильно»: он отвечает на ДРУГОЙ вопрос — входит ли символ в API вообще — и верен только на том, что мы не экспортируем. Почему имя, а не тег: тег не виден там, где решают. Имя видно в каждом вызове, дифе, ревью и автокомплите. Ровно так протёк `skill()` — стабильное имя при доке «`skill()` is experimental». ## Исправлено всё Было три маркера без правила выбора: `@internal` 39 объявлений, `@experimental` 18, префикс 8 публичных символов. Найдено 6 противоречий — символы с `@internal`, которые ЭКСПОРТИРУЮТСЯ; все шесть переименованы, теги переведены на `@experimental`. ## Два дефекта в гейте, отгруженном час назад 1. 🔴 Тег читался только если стоял на строке ОДИН (`@experimental\s*$`). Реальные теги несут прозу — `@internal Experimental typed-composition surface — …` у `pipe`, два в `services.ts`. Замер: **2 из 8** `@experimental` и **31 из 39** `@internal` объявлений были гейту не видны. Строгий паттерн выглядел безопаснее и был наоборот. 2. Между JSDoc и объявлением может стоять комментарий (`eslint-disable` перед перегрузками `pipe`) — привязка к следующему символу его теряла. После починки гейт видит 14 объявлений вместо 6. ## Записано `STABILITY.md` — секция «The one rule» + список; `CLAUDE.md.spec.ts` → перекомпилированный `CLAUDE.md` (правило №51). Плюс честно записанная дыра: гейт энфорсит «помечено ⇒ названо», но НЕ «всё, что должно быть помечено, помечено» — живой пример `input()`/`step()`, словарь только скилл-спеков, без тега и префикса. ## Третий раз за сессию Фикстура-спека в шаблонной строке (`generate-harness.test.ts:505`) не видна ни tsc, ни грепу по импорту — ловится только прогоном тестов. До этого то же в `cli.test.ts` (дважды). 5 новых тестов гейта (13 всего), включая закрепление обоих дефектов выше. Гейты: vitest 2941 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал `docs/cli.md:646`) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
e782c8b0bf |
fix(coverage,hook-runtime)!: answer "is this tested?" by execution, and stop the paid tier being reachable by accident (#133)
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since
|
||
|
zernie
|
a87d0cf2bc |
fix(audit,testing): three dogfood defects — a heading, a salvage, and a check count (#130)
A bundled script named in a markdown heading was invisible to skill-resource-resolves, because the use-directive gate reads prose and a heading is not prose. An ATX heading now counts; the widening stops there, measured at 1 finding before and after over 28 SKILL.md files. Malformed frontmatter was regex-salvaged into the tool contract the lethal-trifecta and Safety ring read, so a unit whose allowed-tools a strict loader rejects scored as the narrow contract its author meant. A salvage may now only make the verdict worse, never better — the vendored corpus forced that nuance by going red when the first pass refused salvages outright. vigiles test reported a clean exit with zero assertions as a pass. vigiles/testing now counts checks per tier and the runner reports a fourth, non-fatal state. Silence stays a pass: undefined means nobody counted, 0 means counted and found nothing. No new CLI verb or flag. Every mechanism is deterministic. Direction is strictly more conservative: nothing can now pass that could not before. |
||
|
zernie
|
41169e379a |
fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129)
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state `Tested` collapsed deterministic harnesses and real-model evals into one number, and its finding read "N surfaces with no vigiles test/eval" — one sentence for two prescriptions three orders of magnitude apart in cost. A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on every push, and asks whether a gate still catches what it claims. An eval (`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one question a deterministic read cannot: does this skill fire at all? Folded together, a repo with complete deterministic coverage and no evals scored identically to a repo with neither. - test-coverage: one discovery pass, partitioned by suffix into a harness tier and an eval tier; the union `covered`/`untested` fields are unchanged. Mirrored in the browser-safe twin. - scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and additive; a producer predating the split falls back to the union). - audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both advisory. `Evaluated` has THREE states — a number, a measured `0`, and `not measured` when the executing checks were skipped and no eval exists. Scoring 0 there would report the absence of a check as its result, which is the pattern this tool exists to name elsewhere. In that state the ring names the command that would measure it. - cli: threads the existing execute-skip signal into the ring. - report/HTML/JSON: both rings render; the inventory carries both counts. Grade weighting is unchanged — both rings are advisory and neither enters `reportDeductions`. Audited on this repo before and after: A (100/100) both times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * docs(site): the audit grades six categories, not five Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * fix(lint): a clean union must not read as 'firing verified' Every surface can have a deterministic harness while nothing has ever measured that a skill fires. The gate is unchanged (still the union), but the ✓ now says so instead of implying the question was asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * test(site): regenerate the browser-parity fixture for the Evaluated ring The audit now emits a sixth category, so the Node-generated expected report and the in-browser one differ until the fixture is regenerated — which is what the test's own header prescribes for an intentional engine change. Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the Evaluated ring and its two inventory counts and nothing else: grade and overall are untouched, which is the check that the split did not move the weighting. * fix(coverage): `Tested` measured a STRING, not a test DEFECT. A surface counted as covered when its path or namespace appeared as a SUBSTRING anywhere in any discovered test file (`content-reference`, in src/test-coverage.ts). Two halves, and the second was worse. REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an existing harness file: // probe: skills/argument-arc moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the line put it back. So `Tested` was trivially gameable and the gaming was indistinguishable from real work. The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely asserts over all 21 of its pipeline skills, but builds each path at runtime (`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING. Generality was PENALISED: 21 trivial files each naming their own subject would have scored better than the one harness that actually tests all 21. This is the exact substitution vigiles names in other people's repos — presence of a surface taken for presence of the property, the way `noExplicitAny` in a config is taken for the rule being enforced. The tool was committing it in its own scoring. FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the disk detector and its in-browser twin cannot drift on what decides a grade): 1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring the existing `vigiles:ignore-test` marker convention on the other side. A runtime-path harness can now say what it covers; an accidental mention cannot. Declaration beats inference — the same move as declaring `allowed-tools` instead of inferring capability. 2. PROVENANCE. Every coverage decision now carries HOW it was reached — declared / colocated / mention — and the counts are printed by `formatUntestedReport`, by `formatScanReport`, and carried in `AuditReport.inventory.coverageEvidence` (additive; schema version unchanged). A repo whose coverage rests entirely on substring matches can finally see that about itself; "28 covered" and "28 names that happen to appear in a file" used to print identically. 3. The bare substring detector is KEPT, deliberately, and said so in the code: it is the zero-config path that makes an ordinary `foo.test.ts` naming `skills/foo` count without anyone learning a marker, and dropping it would flip honestly-covered surfaces to "untested" overnight. What it no longer reads is COMMENTS. A path in a comment is prose about a test; a path in executable code is the test reaching for the surface. Prose isn't policy, including ours. It is also now visibly the weakest tier in the report, and when it is the ONLY evidence the report says so in words. Grade weighting is untouched. TESTS prove both halves, each watched failing first: with the fix reverted, "a surface named ONLY in a comment is NOT covered", "a runtime-path harness that DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail (3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE still counts, and a `"https://..."` string literal survives comment-stripping. Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously recommended "add the skill's path in a comment", which was advice to exploit the defect. * fix(site): key the browser-test prebundle to dist, so a stale cache can't lie DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako divergence by comparing an in-browser AuditReport against one generated in Node. `site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on `@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite invalidates on config/lockfile changes, not on the mtimes of an aliased path outside node_modules. So the browser side can compute against a PREVIOUS engine while Node computes against the current one. REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately). Accidentally: an engine change added a report field; the parity test failed showing the field present on the Node side and absent in the browser. I read that aloud as a genuine engine divergence and started hunting a pako bug that does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass. Deliberately, to prove the fix: plant an extra `inventory` field in src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the browser suite with a warm cache. - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1` on one side only — a field-level diff that reads exactly like a real divergence and says nothing about the cache. - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` -> `.vite-engine-a52a27e0137e`. Reverting the plant returns the fixture to its committed bytes. FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/` (`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`), sweeping the directories keyed to other fingerprints so they don't pile up. WHY a key and not the alternatives. Comparing the cache's mtime against `dist/` and forcing a re-bundle, or clearing `.vite` in the test script, both work and both leave the bad state REACHABLE — a comparison someone can get wrong, or a script step bypassed by running `vitest` directly. Keying the directory removes the state instead: a different `dist/` is a different path, so "prebundle built from another engine" cannot be reached, and an unchanged `dist/` still gets a warm cache. By construction rather than by check — the argument this project makes everywhere else about removing a capability rather than policing its use. The test's header instruction ("pako and node:zlib disagree — do NOT paper over it") was correct for one of the two states and actively misleading for the other; it now says which state is no longer reachable and why, and the assertion carries the same note in its failure message. * fix(site): make the parity-fixture generator emit prettier-clean JSON DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with `JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it keeps short arrays exploded across three lines where prettier collapses them. CI gates on `prettier --check .`, so every regeneration of the fixture committed a file that reddened CI later and far from the cause. It just did: branch head |
||
|
zernie
|
7be092ce19 |
refactor(linters): LinterAdapter port + JVM/Go doc & test parity (#116)
LinterAdapter registry (Record<BuiltinLinter, LinterAdapter>) makes a missing linter a tsc error; generate-types iterates the registry; conformance test set-matches registry vs BUILTIN_LINTERS + docs + site chip list. CI installs detekt/ktlint/checkstyle/golangci-lint so their gated tests run (no silent skips). Docs/site catalogs->linters parity. add-a-linter contributor skill. Report-view teaser cards lightened + linter footnote de-staled. Stage 4 (dispatch-map collapse) tracked as a follow-up. |
||
|
zernie
|
3017f70ef5 |
feat: JVM/Go linters + audit/init/compile hardening + adoption UX (#114)
Closes #107, #109, #110, #111, #113. Vlad's 5 issues (skill allowed-tools round-trip, detekt/ktlint/checkstyle/golangci-lint catalogs, 2 scan false-positives, non-JS harness guide, testGlobs docs), a 14-bug dogfood batch, adoption/demo UX, and prevention refactors (one markdown-it fence oracle replacing 5 regex toggles; one js-yaml detekt-config parser replacing 3). LinterAdapter port added to the roadmap as P0. |
||
|
zernie
|
f91617ac27 |
fix(site): verb-map alignment + gate the site e2e (incl. mobile overflow) in CI (#105)
Inline the verb-map answer (sm:col-start-3 on the model note fixes the grid cell collision that bumped it to a middle-indented second row). Wire the site Playwright e2e into the CI site job + a 390px mobile-overflow gate. Add commands-and-how-they-relate.md (audit/lint/test/eval/init model; init-vs-audit for measuring skills) and link it. Roadmap: backend with rate-limited LLM for the full audit demo; interim demo-scope caveat. landing-site skill gains the responsive-grid cell-collision lesson. |
||
|
zernie
|
fd5000af21 |
feat: adoption direction v2 — rule-enforcer rename, Ruff+Clippy routing, segmenter precision (#75)
Rename compiler/ -> @vigiles/rule-enforcer; add Ruff (Python) + Clippy (Rust) routing; segmenter precision (colon-leadin + determiner-description rejects, 818->789); rule-enforceability.md + breadth benchmark + 5 enforcement homes + ranked enforcement-tier backlog; verify-docs-findable skill; adoption direction locked (audit-first). |
||
|
zernie
|
d1f4886a06 |
feat: full ESLint/Pylint parity in the audit rule map (catalog + Python synthesis)
Pylint dynamic catalog + enabled-state (section-aware --list-msgs / --list-msgs-enabled; symbol or numeric code); mergeCatalogs unions ESLint+Pylint with collision handling at the routing lookup (conservative bare-id merge, unique numeric code keeps its own hit). Python custom-rule synthesis in @vigiles/compiler via an astgrep-py engine (ast-grep JSON objects, one gate + injected executors). Two-tier detection (confident / possible / skipped-with-reason). Real-OSS dogfood run in CI (routing + the real pylint binary). Audit DX: auto-gitignore report artifacts (POSIX-normalized), --out, --no-open; consent resolved before routing so a first interactive audit enriches its own run. Design-of-record in research/rule-compiler-design.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
zernie
|
bf9815f0b4 |
feat(audit)!: prose→lint-rule routing map (ESLint + Pylint) + dogfood CI-enforcement + repo-structure pass
Audit rule-routing map (ESLint full + Pylint basics; dynamic catalog, S0/S1 markers, construct/docstring patterns) with the four lanes sharpened (enforceable / hook / custom-rule / judgment-call) and the pr-to-lint-rule gated synthesis skill. Dogfood CI-enforced + indexed (compiler gate asserts verdicts; bench corpus guards; research/dogfood-corpus.md + a vendoring-policy rule; per-adapter asymmetry documented). Repo-structure cleanup: dev/ -> .claude/skills, schemas/ parked under src/, fixtures/ -> test/fixtures, etc/ -> api-surface/, scripts/tools split, root CLAUDE.md Layout map. BREAKING CHANGE: the audit --json schema bumps schemaVersion 1 -> 2 (the mechanism enum for an unrouted rule changed "compile" -> "synthesize"). |
||
|
zernie
|
5ba3516957 |
feat: audit rule-compile tier + verdict-led report revamp (#69)
* research: spec for the audit rule-compile tier (design of record)
Locks the design after two Fable red-team passes + founder refinements. The
audit opt-in tier compiles a repo's CLAUDE.md/AGENTS.md prose rules via the
sibling agent-rules-compiler engine (compile-rules skill + gate.js runGate),
runs the gate-kept ones, and surfaces real violations + per-rule copy-command
buttons. Two surfaces: a deterministic foreign-safe teaser (no model) and a
model-on-sub own-repo tier reusing the existing decideExecute consent. Hard
invariants: difficulty is a display not the auto gate (the gate is the final
trust filter); hook-surface rules never auto-build (false safety = the flagship
2/7-vs-7/7 finding); the report never spends; one engine, two entry points.
Spec only; implementation to follow.
* audit(rule-inventory): FP-safe deterministic detector for documented-but-unconfigured rules
The free/foreign-safe surface of the audit rule-compile tier (spec:
research/audit-rule-compile-tier.md). Pure function: given a repo's
CLAUDE.md/AGENTS.md text + lint-config text, find prose rules that map to an
off-the-shelf lint rule and whether that rule is already in the config —
the one-line-config-fix nudge.
No model, no config execution (textual grep only — avoids the eslint-config
RCE path). High precision by construction: only rule-name/code-token-shaped
keywords, matched as whole tokens; every bare-word trigger the FP measurement
flagged (token/secret/await/!/aria/silently/…) is dropped. Measured: the raw
keyword matcher fired 107x across 4 real CLAUDE.md files (~all garbage); this
detector returns 0 on that prose and fires correctly on real rule mentions.
6 tests incl. the FP guard, embedded-token guard, regex-metachar escaping,
and in-config vs not-in-config.
Next: wire rulesInventory into AuditReport + the report section + the opt-in
model tier (compile/gate/run) per the spec.
* audit(rule-inventory): multi-linter shape (ESLint-first data) + spec build-status
Per founder: the detector was ESLint-only. Refactor the SHAPE to be
linter-keyed (LinterName eslint|ruff|clippy|pylint|rubocop|stylelint) with an
optional {linters} filter — the matcher was already linter-agnostic, so
adding another linter is additive DATA (rule-name entries), not a refactor.
Data stays ESLint-first + honestly labelled; the opt-in tier will resolve
enablement via the existing multi-linter checkLinterRule engine (which execs
config) rather than reinventing it. 8 tests (adds linter-tag + filter cases).
Internal spec updated with build status; public docs held until the feature
renders (no vaporware).
* audit(rule-inventory): wire rulesInventory onto AuditReport (additive, schema unchanged)
Mirror the adoptable/observations pattern: optional rulesInventory? on AuditReport
+ BuildAuditReportOptions, spread through the pure builder. The CLI will compute it
(read instruction + config text -> buildRuleInventory) and pass it in, keeping the
builder fs-free. Additive within schemaVersion 1. 18 audit-report + rule-inventory
tests green; build + eslint clean.
Next: CLI populates it (the fs read) + the report section renders it (report/src/
schema.ts + a section component, Adopt.tsx copy-command pattern).
* audit(rule-inventory): CLI populates rulesInventory (first user-visible output)
The audit command now computes the deterministic rule-inventory teaser: reads the
repo's instruction file + lint-config TEXT (best-effort, fs-only, NEVER executes
config — safe on any repo) and maps documented intents to off-the-shelf rules via
buildRuleInventory, passing it into buildAuditReport. Verified e2e: audit --json on a
repo whose CLAUDE.md documents console.log/no-explicit-any/eslint-disable emits
rulesInventory with in-config (no-console found in eslint.config.js) vs not-in-config
nudges, and correctly ignores a prose token-savings mention (FP-safe live).
cli.test.ts 109/109, all rule-inventory/audit-report/audit-html tests green. (The 6
unit failures are pre-existing env-only linter-CLI tests — rubocop/pylint absent
locally — untouched by this change.)
Next: render it in the HTML report (report/src schema + section component).
* audit(rule-inventory): render the Rule inventory section in the HTML report
Report app now renders rulesInventory: report/src/schema.ts gains the type +
field; a RuleInventory component shows each documented intent -> off-the-shelf
rule with an 'enforced' (green ✓, in config) vs 'documented, not enforced'
(amber ●) state and a copy-config-fix button (the Adopt.tsx static-copy
pattern); Report.tsx renders it as a section. Also add the scoped
@typescript-eslint/* keyword forms so real projects (which cite scoped rule
names) match. Dogfooded: a rule-documenting repo renders 7 items (2 enforced,
5 nudges) with the semantic rules correctly omitted; report bundle rebuilt.
* audit(rule-inventory): copy button emits an agent prompt, not a raw config line
Per founder: the raw JSON snippet is clumsy (where do you paste it?). vigiles
users live in Claude Code / Codex, so the actionable affordance is an
instruction the AGENT executes — 'enable the ESLint rule X (error) in the lint
config so the documented rule is enforced'. Button now copies that
paste-ready prompt; label 'copy agent prompt'.
* audit(rule-inventory): precision-harden after big-OSS dogfood (17/18 FPs → 0)
Dogfooding the rule-inventory teaser on large real ESLint repos exposed
17-of-18 false positives. Three root causes, three fixes:
- Root-only config read missed nested configs. collectLintConfigText now
walks subdirs (depth ≤2, skips node_modules/dist/.git/…, caps 60 files),
textual only — never resolves/executes a config (the RCE path stays shut).
- oxlint + biome configs weren't in the read set though they use ESLint's
rule names; added. RULE_INVENTORY_CONFIG_FILES is now a Set.
- Only the harness-native instruction file was read; rules are often in
AGENTS.md under a claude-code harness. Now reads CLAUDE.md + AGENTS.md +
the native file.
Detector precision: base ↔ @typescript-eslint/ variants now count as the
same rule in config (variantsOf/ruleInConfig), and a `recommended`-preset
extend downgrades an unnamed preset rule to a new "preset-maybe" state
instead of a false "unenforced" alarm (7 intents tagged inRecommended).
Report renders three states — enforced / likely-enforced-preset (muted, no
nudge) / documented-not-enforced (the actionable copy-prompt).
Verified: build green, self-lint pass, 21 tests green (+5 new), and
cloudflare/agents re-dogfooded from 3 FPs to all in-config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* docs: document the rule inventory (audit) + dogfood result
- what-vigiles-catches.md: new shipped row 16 — a prose rule that an
off-the-shelf lint rule already enforces but which was never turned on.
- cli.md: describe the rule-inventory section of the audit report — the
three states (enforced / likely-enforced-preset / documented-not-enforced),
the text-only (never-executed) config read, whole-token matching, the
20-repo dogfood that drove FPs to ~0, and the multi-linter shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit(rule-inventory): suppress false nudge on documented opt-outs
Dogfooding pmndrs/react-spring surfaced a wrong-advice FP: its CLAUDE.md
documents `@typescript-eslint/no-explicit-any` as "off intentionally", and
the inventory nudged to enable it. A rule the author deliberately turns off,
and which really isn't in config, is CONSISTENT — not a gap.
Add isDocumentedOptOut: a deterministic negation window (off / disabled /
not enforced / not enabled / turned off) tested on the context either side
of the matched token — never the token itself, so `disable` can't self-match
inside `eslint-disable`, and the full word `disabled` (not `disable`) avoids
a neighbouring `eslint-disable` leaking its cue into an adjacent rule's
window. When it fires on a not-in-config mention, the item is dropped.
Verified: 13 tests green (+2), react-spring re-dogfooded — the false
no-explicit-any nudge is gone, only the real in-config no-console remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit(rule-inventory): add two off-the-shelf mappings from the real corpus
Cross-referencing the hand-verified rule-adherence corpus (real repos:
motion, mapbox, palantir, …) against INTENT_MAP surfaced two documented
intents that DO have an off-the-shelf ESLint rule but weren't mapped:
- "no default exports" → import/no-default-export
- "no TODO/FIXME" → no-warning-comments
The same cross-reference confirms the deterministic tier's ceiling: of the
corpus's 13 real rules, ~4 route to an off-the-shelf rule, ~5 CONTENT rules
are synthesis residue (object-spread ban, optional-chaining ban, British
spelling, package-manager), and all 4 ACTION rules (no-AI-coauthor,
don't-edit-generated, DCO sign-off, never-push-main) are hook territory —
which is the case for the opt-in compiler + hook tiers, not more INTENT_MAP.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: fold agent-rules-compiler into the monorepo as @vigiles/compiler
Moves the rule-compiler engine (former standalone zernie/agent-rules-compiler)
into the vigiles monorepo as a self-contained opt-in sub-package, mirroring
report/'s package-boundary pattern. Decision + rationale: migratsiya/49.
- compiler/ = @vigiles/compiler (CommonJS, own eslint/ts deps). The two-stage
trust gate runs unchanged in the new home: `cd compiler && npm run gate`
reproduces kept=3 / kept-ungraded=3 / abstain-selftest=1 / abstain-gold=1,
including the R10 silent-leak catch (the paper result).
- Isolation preserved: root tsc is src/** only, root eslint is src/ only, so
neither touches compiler/. Added compiler/ to .prettierignore as its own
package boundary (own toolchain; also holds machine/agent-generated rule
code + gold-set data + exact-byte run artifacts). node_modules stays ignored.
- README documents the pipeline (classify → reuse → synthesize → gate → enforce)
and that this tier loads ONLY behind the opt-in consent, keeping the default
audit/lint surface deterministic and model-free.
Root gates verified green: prettier --check ., build, self-lint (512 files, all
refs valid).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: add extract-existing — resolve a repo's real enabled lint rules
The REUSE tier needs to know what a repo ALREADY has before it synthesizes.
extract-existing runs the target repo's own eslint (`eslint --print-config`)
to enumerate enabled rules + severities, reads installed eslint-plugin-* from
package.json, and exposes ruleStatus() (error/warn/off/absent) with
base↔@typescript-eslint variant unification.
EXECUTES the target config (RCE path) → opt-in / own-repo only; the deliberate
exec-based counterpart to the foreign-safe, exec-free src/rule-inventory.ts.
Verified live on vigiles (141 enabled, 13 off, 4 plugins). Pure logic covered
by extract-existing.selfcheck.js (5/5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* research: end-to-end rule-compiler flow design + decisions (suppressions ratchet, harvest-at-scale, compiler-first)
Captures the whole pipeline (harvest → segment → classify+reconcile → gate →
existing-violations → persist → feedback) so it stops being re-derived stage by
stage. Two decisions recorded: existing violations grandfather via ESLint's
native eslint-suppressions.json ratchet (never eslint-disable), and Stage 0
becomes a bulk rule-harvester/taxonomy. Order: fix the compiler first, then run
the harvest over it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: expand the plugin index (REUSE tier) 20 → 129 intents
The reuse catalog now covers the common real-world agent-rule intents across
~18 ESLint sources: core, typescript-eslint, import, unicorn, sonarjs, promise,
n, security, jsx-a11y, react, react-hooks, regexp, eslint-comments, jest,
vitest, testing-library, no-secrets, boundaries. Adds a self-documenting
`plugins` index (source-prefix → package + purpose) and flags 44 entries with
type-aware / heuristic / rename notes.
Correctness over quantity: all 227 unique rule ids were checked as real, current
rules of their plugin (a programmatic existence-check against installed plugins
is a later step; the compiler's enable path + eslint error surface truly-absent
rules as a safety net). Keywords include both prose forms and rule-name/code
tokens so the classifier can match either. The deterministic high-precision
subset stays in src/rule-inventory.ts and is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: add classify — route each rule + reconcile against real config
The Stage-2 module that ties the pieces together: an atomic agent-rule →
{reuse | synthesize | hook | semantic}, and for reuse, reconciled against the
repo's real config via extract-existing (enforced / warn / off-contradiction /
absent-installed / not-installed). Hook detection is action-cue specific (git
push, generated files, sign-off, shell) so it doesn't grab lint rules.
Hardening from testing on the 13-rule hand-verified corpus:
- empty-match guard: a code-operator keyword like "no & or |" is an empty
regex alternation that matched EVERY rule and collapsed all classification
onto no-bitwise; keywordRe now falls back to escaped-literal for any keyword
whose regex matches "".
- added `as any` / `as unknown as` / `<any>` / `: any` to the no-explicit-any
intent — the article's own centerpiece was missed because `\bany\b type`
needs "any type" adjacent and the backtick form breaks it.
- documented the polarity limitation (banning `??` keyword-matches
prefer-nullish-coalescing) — the deterministic pass routes; the opt-in model
pass disambiguates before any enable.
Corpus distribution: reuse 4 / synthesize 5 / hook 4 / semantic 1 — matches the
tier-coverage thesis (a minority route off-the-shelf; the rest need synthesis
or hooks). 8/8 self-checks pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* research: link compiler-end-to-end-flow from the index (fix orphan-doc lint)
The design doc was orphaned (referenced by no other .md), tripping vigiles's
own orphan-docs lint (exit 2). Linked it from research/README.md next to the
audit-rule-compile-tier spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* lint: exclude compiler/** from self-lint (package boundary, like bench/ + startup/)
The @vigiles/compiler fold brought demo fixtures (compiler/demo-project/CLAUDE.md,
demo eslint configs) into scope, tripping require-instructions-spec + rule-coverage
(exit 2). compiler/ is a self-contained package already excluded from tsc/eslint/
prettier; excluding it from the harness self-lint too. self-lint now PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* readme: temporarily cut the zernie.com blog link (double-blind, paper #1)
The footer 'companion to Feedback Loop Is All You Need' link is a repo -> blog
deanonymization path while AgenticDev paper #1 is under blind review — and the
reviewer-offer email links this repo. Removed with a restore-note for after
notification (2026-08-21). Fable-flagged leak.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* report: fold 'Create specs' to the bottom (de-emphasize the spec-adoption nag)
Moved the 'N surfaces not yet spec-managed' block from mid-report (right after
Fixes) to a collapsed <details> at the very bottom. Surfacing it high read as
homework/nag to a first-time reader; spec-adoption is optional hardening, not a
failure. Collapsed by default, labeled 'Optional'. (Founder feedback.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit revamp: Rules two-zone hero + deterministic CLAUDE.md segmenter
Part of the comprehensive audit revamp (Fable-designed).
- report RuleInventory: rebuilt as the "Rules — prose → enforced" hero. A
summary line + "deterministic · no model" badge, a 100%-stacked distribution
bar, rules GROUPED BY STATE (contradiction + one-line-away open on top;
enforced/preset folded to a count), and a dashed "Compile — the full picture"
CTA that DESCRIBES the opt-in tier (reuse/synthesize/hook/prose) without
fabricating result rows. Adds the `contradiction` configState to the schema
(detector wiring next).
- src/segment.ts: deterministic Tier-A segmenter — splits an instruction file
into atomic candidate rules with provenance (exactQuote + line span) via a
3-cue imperative gate (form/context/shape → high/medium/reject), precision-
first (under-splits, never splits "never X except Y"). No model, no deps.
14/14 tests green. Feeds the upcoming State-B heuristic routing preview.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit revamp: wire the re-scoring verdict engine + config-contradiction rule state
- computeVerdict re-scores the report with each finding removed to derive real
pointsIfFixed per recommendation + a verdict sentence (fixes-to-next-grade),
now assembled into every AuditReport (schema + report mirror + parity test).
- rule-inventory gains a 'contradiction' config-state: a rule the harness
documents as a norm but the config explicitly sets to off/0 (docs vs config
disagree), distinct from merely absent. ruleSetOff is textual-only, no exec.
- research index: add audit-rule-compile-tier + compiler-end-to-end-flow docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit report: verdict-led redesign (grade hero, ranked fixes, scannable strip)
- New hero: the re-scored verdict sentence is the headline, grade+score a
band-bordered badge, dir shown as a plugin basename, trust line beneath.
document.title becomes 'vigiles — B (84) · my-plugin'.
- Fixes are ranked by real pointsIfFixed with a '+N pts' badge and a copy-
agent-prompt button; the left border encodes IMPACT (never green).
- Five heavy category rings collapse into a compact bar-per-category strip.
- 'What it ships' demoted from six stat cards to one meta line; spec-adoption
stays folded at the bottom. Rules section retitled 'Your rules → enforced'.
- band.ts gains a literal BORDER map (the JIT never emitted the string-built
'border-warn'); drop the footer AlertTriangle (green/neutral discipline).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* readme: showcase the rules-compile capability + re-render the hero screenshot
- New Proof 3 ('a rule you wrote that nothing enforces') pays off the tagline:
maps prose rules to the lint rule that enforces them — on, one line away, or
silently turned off — and points at the opt-in compile tier. Secrets-exfil
becomes Proof 4 (still the closing bite).
- 'What it caught' now describes the verdict-led header + the 'your rules →
enforced' section instead of just the five rings.
- Re-rendered vigiles-audit.png from the new verdict-led report (dark), and
updated its alt text + the DIRECTION note's screenshot + spine guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit report: apply Fable review — honesty + hierarchy fixes
- Rules headline no longer rounds 'preset-maybe' up to 'enforced' (the tool's
whole pitch is that config lies about rules): counts + colours enforced,
preset, one-line, and contradiction separately, with a 4th preset bar segment.
- Contradiction row ('the one people screenshot') gains a copy-agent-prompt
(a decision prompt, since the rule is present-but-off) and reads 'set to off
in your config'; one-line rows relabelled 'enable in 1 line' to stop colliding
with the verdict's point-scoring 'fixes'.
- New 'Review — no auto-fix' section surfaces Safety findings (the lethal-
trifecta exfil path) as cards — the report's severest finding was previously
only 11px category-strip text with no card (it maps to no recommendation).
- Category findings de-printf'd at the render layer ('1 tool(s)' → '1 tool');
scorer strings untouched. Compile CTA drops 'blind-gated' internal vocab.
- Sample fixture: broken refs are now file/dir (not a lint rule) so Adoptability
no longer visually collides with the Rules hero; re-rendered the screenshot.
- README Proof 3 reframed as representative reader's-repo output (not a claimed
public-plugin catch under the 'real scans' banner) with reconciled numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit: deterministic State-B rule routing + scorer-level pluralization
State-B routing preview (src/rule-routing.ts):
- Segments the instruction file (segment.ts) into atomic rules and routes each
one deterministically — reuse (off-the-shelf rule -> config line), hook (action
rule a linter can't see -> hook), semantic (judgment call -> prose), or unrouted
(no deterministic route -> the opt-in compile tier decides). Honest by
construction: never claims 'synthesizable' — that's the model tier's job.
- Precision-first: routes only high-confidence segments, so a doc-heavy file
(a keyFiles index of "path — desc" bullets) doesn't flood 'unrouted'.
- Wired into the audit report (schema + CLI + report mirror). The report's
Compile CTA now shows REAL per-category counts + example rules instead of a
generic pitch — grounds the upsell in the reader's own rules.
Scorer-level pluralization:
- audit-score's finding labels resolve the terse "(s)" placeholder against the
count ("1 tool" / "3 tools") at render, via a pure pluralizeLabel; mirrored in
leaderboard.ts and the verdict sentence (audit-verdict.ts). The report's
render-layer humanizer is removed — the logic lives at the scorer now.
Report polish (Fable #4 follow-through):
- 'Adoptability' section renamed 'Broken references' with coherent copy (it reads
the repo, it doesn't describe a hypothetical). Sample fixture pluralized.
- Re-rendered the hero screenshot with the routing breakdown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* fix: type new Array<number> in segment.ts (no-unsafe-assignment lint error)
The CI 'test' job runs the repo's stricter ESLint; new Array(n) is typed any[],
tripping @typescript-eslint/no-unsafe-assignment. Annotate the element type.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
18d7cfcae4 |
docs: honest versioning, README scannability, and docs restructure (#67)
Fix the version claim (v12, not 0.x; the 0.0.0-semantically-released placeholder was the source), reframe the eval section around cost measurement, trim the README for scannability (Diátaxis link buckets, tighter reconciliation + proofs), reorganize the docs index into Guides/Reference/Explanation and add six missing docs, and consolidate overlapping docs (related-tools→comparison, inline-mode→markdown-mode, agent-setup+agent-workflows). Relocate the eval-architecture design-of-record ADR from docs/ to research/ per the doc-tiers rule, and fix the library entry points (vigiles/linting is the compiler surface) plus the Codex setup path. |
||
|
zernie
|
c442f4b523 |
feat: faithful caveman ecosystem benchmark + --plugin-dir/skillsDir install support (#61)
Land the completed ecosystem-benchmark work that was stranded on the gate-release branch: the real caveman plugin install (pluginDir + SessionStart hook) with the codified bare-SKILL.md warning, the 7-task corpus, the analysis scripts, and the canonical 2026-07-07 runs (7 tasks × 5 trials, $10.41; compress input test) plus corrected FINDINGS. This is the data behind the published token-savings article and makes `npx vigiles eval` reproduce the real thing. |
||
|
zernie
|
81be2fd07e |
feat: capability-diff PR comment + local observe ledger (#55)
* docs(vault): harness-quality tech camps + telemetry/default-install decision Two-camps tech map (runtime/observability vs pre-deploy simulation vs app-frameworks), the field direction, and the live fork: local-first deterministic measurement over hosted runtime telemetry. Cross-linked from the vault index and funded-adjacency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(vault): add pseudocode DX + fundability/exit thesis to tech-direction Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): tech direction — four-instrument loop + local observability ledger The architecture decision of record for the next build phase: typed spec as declared ground truth; verify/gate/measure/observe as facets of one loop; the deterministic-vs-behavioral precision principle; the local agent-readable runs.jsonl flight-recorder ledger; promote-prose; per-surface map. Monetization kept out (vault only). Indexed + recompiled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add Direction section to root CLAUDE.md positioning Elevate the next-phase direction to the north-star doc (read first every session): the four-instrument loop (verify/gate/measure/observe) with the typed spec as declared ground truth, the deterministic-vs-behavioral precision principle, the local agent-readable runs.jsonl flight recorder, promote-prose, and the per-surface map. Tech only; monetization stays in the vault. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): local agent-readable flight-recorder ledger (runs.jsonl) The connective layer of the four-instrument loop: a versioned, append-only, tolerant-to-torn-lines JSONL ledger every instrument (verify/gate/measure/observe) appends to, read off one schema by audit, by the agent debugging its own harness, and later by an aggregation surface. Harness-agnostic; lives at the composition root (not core, not an adapter). Best-effort append never breaks a session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for direction + observe-ledger session * feat(observe): emit compiled-hook + agent-rail decisions to the ledger First emit points into .vigiles/runs.jsonl: the compiled-hook gate (emitGate — enforce deny/ask + observe would-be) and the subagent tool-contract rail (agentHookCommand deny). The legacy hook-observations.jsonl still written for back-compat; the unified ledger is additive. E2E asserts the observe-mode run populates runs.jsonl with a decision-shaped hook record. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): render the flight recorder in vigiles audit formatLedgerSummary (pure, tested) — total + counts by kind + recent high-signal denials (blocked gates, out-of-contract tool calls) — and audit prints it (terminal, non-json) off the local ledger. Empty/skipped until something is recorded. JSON AuditReport unchanged (a versioned schema bump is the deliberate follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: gitignore the observability ledger (.vigiles/runs.jsonl) Runtime flight-recorder artifact — local record, never committed (sibling of active-skill.json + eval-cache/). * feat(observe): emit skill-fire + trigger-rate metrics to the ledger skill-start records a SkillObservation{fired:true} (name = the skill dir); a completed measureTriggerRate records recall (+ precision when measured) as eval records. Extends ledger coverage across the behavioral surfaces alongside the deterministic gate/rail emits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(skills): add model-invocable debug-my-harness skill Reads the .vigiles/runs.jsonl flight recorder to diagnose why the harness misbehaved — skill fires/collisions, hook decisions (blocked vs wrongly-allowed), subagent contract violations, trigger-rate drift — and recommends a fix, handing off to strengthen/edit-spec/test-harness. The agent-readable payoff of the observe layer. Ships a dogfood trigger eval (recall + precision); loads clean (skills-dogfood). Skill enumerations + keyFiles synced (3→4 model-invocable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): emit capability-diff to the ledger A computed capability-diff (--capability-diff) now records a capability-diff record (added/removed tools + widened verdict). Completes the ledger's emit coverage — all five record kinds (hook/agent/skill/eval/capability-diff) now have live emitters. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF — observe layer built (ledger + 5 emits + audit render + skill) * feat(observe): carry the flight-recorder summary in the AuditReport JSON + HTML summarizeObservations (pure, shares denial logic with the terminal formatter) → an additive/optional observations field on AuditReport (no schema bump, per the adoptability/adoptable precedent). The CLI reads the ledger once, feeding both the JSON (the product boundary — audit --json + the HTML) and the terminal render. New report app: schema mirror + a presentational Observations component + a Flight recorder section. Covered by observe + audit-report tests; end-to-end --json + HTML verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF — AuditReport JSON/HTML ledger integration done * docs(readme): verify-first front door (catch the silent breakage) Retune the tagline + hook to lead with the verify moat (references are real, not just well-formed) over the prior test-first framing; add a coding-scoped subhead naming CLAUDE.md/skills/hooks + Claude Code & Codex. audit stays the proof-first hero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): lock verify-first tagline in the editor-guide (prevent drift) * docs: encode research-depth discipline (vault) + link directionality (doc-tiers) Vault operating rule 6: competitive/strategy research saved in full detail, depth-tagged (deep/medium/thin), coverage-mapped, cross-linked, indexed — shallow never masquerades as deep. Root doc-tiers: link directionality invariant — links point outward only (vault may cite research; research/public must never cite the vault). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): demote the 7-catalog breadth from the headline Lead the reference-verification pitch with the harness-native cross-ref (a referenced tool/hook/file/skill that doesn't resolve). The 7-linter-catalog breadth becomes a supporting mention, not the headline number. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: delete hook-spec spike and demote scaffold-test to skill-internal Trim the launch surface of two dead-weight items flagged by a strategy review: - Delete the src/core/hook-spec.ts + hook-spec.test.ts spike — imported nowhere and superseded by the shipped compiled hooks (src/core/hook-program.ts). Drops its two keyFiles entries and recompiles CLAUDE.md. - Demote the scaffold-test VERB to skill-internal: remove it from VERBS + its dispatch/handler in the CLI (the engine src/scaffold-test.ts + its test stay, used by the test-harness skill). Prune the now-orphaned helpers/imports, the scan-cli e2e block for the removed verb, and the scaffold-demo example, and update the docs (cli.md, spec-format.md, STABILITY.md, CLAUDE.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF with observe layer + verify-first pivot state Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: format harness-observability-direction.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(action): capability-diff PR comment (blast-radius diff vs PR base) Fold the agent capability-surface diff into the sticky PR comment: on a pull_request the Action materializes the PR base with `git archive` (no worktree/smudge), runs `audit --capability-diff`, and appends a short section only when the surface changed. `fail-on-widen` gates a widening. Dogfooded in this repo's own CI (fetch-depth: 0 + capability-diff: true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: salvage deep agent-eval-testing competitor teardown (partial workflow) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — capability-diff shipped + teardown salvaged Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(vault): post-mortem the teardown throttle+interrupt; log the 2x2 re-run Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: fold in the 4 re-run teardown clusters (all 5 now deep, 41 companies) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: HANDOFF — teardown re-run complete (5 clusters, 41 companies) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: add raw structured teardown appendix (41 companies, durable source) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: add the Opus cross-corpus strategic synthesis (conformance-kernel thesis) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: fold in Fable contrarian pass + Opus reconciliation (registry-first bet) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: record the distro thesis (leading direction) + index it * docs: fix doc-tiers — repo is PUBLIC, strategy is vault-only (stop leaks) The doc-tiering rules wrongly treated research/ + CLAUDE.md as a 'private record' holding the moat/positioning narrative. The repo is open-source, so those tiers are public. Corrected: research/CLAUDE.md hold the TECHNICAL record (how the tool works, public-safe); all strategy/moat/monetization/pivot theses live ONLY in the git-crypt vault. Adds the competitor-playbook test. * docs: de-strategize + refresh HANDOFF (it's public — strategy → vault) * docs: move 6 strategic docs to the git-crypt vault (repo is public) measurement-authority, divergent-bets, distribution-strategy, typed-spec-moat, strategic-synthesis-2026-06, eval-startups-positioning move from research/ (public) to startup/ (encrypted). Inbound references neutralized (no vault inward-links), index/keyFiles entries removed, both specs recompiled, vault index updated. * fix: encrypt the 6 migrated vault docs (git mv bypassed git-crypt) * docs: record the git-mv-into-vault-doesn't-encrypt gotcha + fix rule 3 (all public) * docs: pivot-decision (vault) + roadmap reshape + finish moved-doc ref cleanup Adds startup/pivot-decision-2026-07.md (decision + validation gate). Adds two pivot-agnostic technical items to research/roadmap.md 'Now' (step-enforcement A/B, observe→promote) + de-strategizes its top direction banner. Neutralizes the relative-path refs to the 6 vault-moved docs the migration missed (README + plugin-structural fully clean; roadmap partial — 37 scattered refs flagged as follow-up). * docs: HANDOFF — migration done (targeted-6) + flag roadmap ref-sweep follow-up * vault: save the pivot exploration log (all options considered + why) * docs: finish roadmap de-strategization (neutralize moved-doc refs + gut GTM section) * fix(cli): write capability-diff observation to the audited root's ledger appendObservation defaulted to process.cwd(), so `audit ./after --capability-diff=./before` from a parent dir recorded the blast-radius event into the caller's .vigiles/runs.jsonl instead of ./after's — where the audit reads it back via readObservations(root). Pass root. (Codex review #55.) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
3f33925722 |
feat: skill-description-budget rule + document-the-why + audit/opt-in docs (#54)
* docs: explain why compiler-grade guarantees are opt-in The "opt-in, like TS's strict" line appeared in three public docs but only asserted the analogy, never the reason. Add a dedicated FAQ entry that gives the actual answer — opt-in means gradual not permissive (the default already gates real breakage), compiler-grade proofs need a typed program to check (tsc can't verify inert markdown), and gradual adoption is the TS-strict bet — and link the README and linting-guide one-liners to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: codify auto-vs-nudge enforcement model in the spec skills Apply the init enforcement model at edit time across the three spec skills: auto-apply the con-free wins, nudge (with the tradeoff) for anything that costs config/plugins/CI-failure, never silently escalate into strict gating. - strengthen: name the principle explicitly (cost, not strictness, is the dividing line; Tier 1 auto, Tiers 2-4 are the user's call) - edit-spec: add "do what's asked, don't silently escalate enforcement" - adopt-spec: replace the prose blockquote with explicit, scannable adoption rules (faithful / non-destructive / don't-escalate / reversible / ask-before-writing / lighter-touch-exists) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: explain what audit runs (read vs measure) + selection-collision Close the audit→behavioral→consent documentation gap exposed while dogfooding a skills-heavy repo: a clean-A audit "found little" because a plain audit is a deterministic READ — the behavioral checks (does each skill fire? do two collide?) run only under the one-time audit.measure consent. - faq.md: new "What does audit actually run — and why did it find nothing?" entry stating the read-vs-run model and the headless fallback - for-plugin-authors.md: turn the bare "selection-collision matrix" name-drop into a real explanation (diagonal = recall, off-diagonal = a sibling hijacked the prompt; body-stubbed, so it measures selection cheaply) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add document-the-why rule + doc-command coverage check The "docs suck" failure repeated twice this session (the opt-in why; the audit→behavioral→consent story), and no existing docs rule owns it: the polish, placement, and sync rules govern how well / where / consistency, but none requires the REASONING behind a user-facing decision be documented at all. - new `document-the-why` guidance rule: every user-facing decision/concept gets a plain-language doc home that answers why/what (name-dropping ≠ documenting); the test is "could a user answer it from the docs alone?" - new deterministic FLOOR (the inverse of self-command-refs): every public CLI verb must be mentioned under docs/ — src/doc-command-coverage.ts, dogfooded in its test (catches a verb shipped undocumented). The concept-coverage half stays judgment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: add skill-description-budget lint rule (warn-tier trigger proxy) A model-invocable skill is selected on its description, and the selector weighs the opening most — a bloated description buries the trigger signal and hurts recall + precision. New deterministic heuristic proxy (generous 500-char budget), the free sibling of description-overlap. Wired end-to-end: detector (src/core/skill-description-budget.ts) + rule-meta (heuristic-behavioral → warn ceiling) + DEFAULT_RULES + NUDGE group (never gates) + scan report + lint application + docs/rules + the rules matrix + tests. NOTE on haretrail: it does NOT fire there — the earlier "634/781-char" figures were BYTE counts; in characters (Cyrillic = 2 bytes/char) the longest skill description is 463 chars, under budget. The rule is correctly calibrated FP-safe; forcing a hit would mean crying wolf. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for haretrail-dogfood session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * test: add skill-description-budget to validate.test DEFAULT_RULES literal The new rule landed in DEFAULT_RULES but validate.test.ts asserts the full rules object against a hardcoded literal — add the key so the deep-equal passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: prettier-format HANDOFF.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * refactor: remove the cross-language description scan flag (refuted) The deterministic "non-Latin description → cross-language trigger risk" finding (unexpectedScript / descriptionScript in scan.ts) is removed: the Script type, scriptCounts, unexpectedScript, the ScanSkill.descriptionScript field, its report line + summary, its tests, the docs mention, and the config-expected-script notion. Why: it fires on 100% of non-Latin descriptions, but a non-English description is a legitimate audience choice, not a defect — so it scolds correct work. And the one time we measured it (plugin-behavioral-findings.md Finding 3a, native codex), the risk was REFUTED — the model handles RU-description/EN-prompt selection fine. A warn that reads as a defect on correct authoring cries wolf (lint-rule-calibration). Reasoning saved in research/plugin-behavioral-findings.md; the one-detector-no-drift teaching example reassigned to description-overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: eval cost transparency — surface tokens, API-equivalent $, metered warning Make what a real-model run SPENT impossible to miss (vigiles's affordability pitch is "runs on your subscription, not a metered API" — so every run should say so). - new src/eval-cost.ts: the pure engine — cost snapshots (per-run / per-arm / whole-report), metered-vs-subscription detection (ANTHROPIC_API_KEY set → metered), a session tally, and the formatted block (tokens + API-equivalent $, a LOUD warning + actionable fix when metered, ✅ when on the sub). Fully unit-tested, no model. - runEval now emits the cost block after a run (stderr; report carries usage). - test-harness skill: MUST relay the spend to the user after any eval it runs. - measuring-skills docs: "what a run reports + the metered-API warning" section. HONEST SCOPE: no "% of your subscription" — Anthropic doesn't expose plan quota, so a percentage would be invented; tokens + API-equivalent $ + billed-to is the complete honest picture. FOLLOW-UP: measure/measureArms/measureTriggerRate reports don't carry usage yet (only EvalReport does) — threading a usage field through those model-gated run loops + the eval-lock is the next commit so the skill's trigger-rate path emits too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * fix: brace void-returning arrows in eval-cost (eslint no-confusing-void-expression) Missed running eslint on the new file (only ran fmt:check). The console.error sink and the test beforeEach arrows returned a void expression in shorthand form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add surface-architecture-decisions rule Surface non-trivial architectural decisions (file placement, core vs adapter vs port, extend-vs-create, cross-cutting changes) to chat in a scannable block with the boundary reasoning — the structure analog of subagent-model-note — so the user can catch a wrong placement before the diff, given the hexagonal + multi-adapter constraints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: thread usage into TriggerRateReport so measureTriggerRate emits cost The skill's main path (measureTriggerRate) now surfaces spend too. Per-run usage comes from the parser (makeContext(out, cfg.parse).usage) — harness-neutral, since both the Claude and Codex parsers fill ParsedModelRun.usage (Codex reports $0 on the sub, real tokens). runTriggerTrial returns its run's usage; runTriggerSet collects them; the report aggregates via the existing aggregateUsage (relevant + irrelevant sets both counted); the wrapper emits costFromArm(report.usage). No new port/driver method (the parser already provides usage) and eval-lock-safe (additive field; the lock hashes inputs, not the report). FOLLOW-UP: measure/measureArms (CheckReport) still don't carry usage — same threading applies, next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: fold parse-don't-validate + illegal-states-irrepresentable into surface-architecture-decisions Frame them from this rule's angle (surfacing): a parse-boundary and an illegal-state-forbidding type ARE architectural decisions — name them when you make them (where the boundary sits; what invalid state the type forbids). Cross-refs ts-essentials for the mechanical how, rather than duplicating it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: update vigiles-testing API surface for TriggerRateReport.usage api-extractor tracks the public surface; adding the usage field is intended (additive — consumers read the report, don't construct it). npm run build doesn't run api-extractor, so this was missed locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for the eval-cost / rules session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: capture haretrail-driven audit + eval ideas (research note) Save the session's un-persisted ideas before context is lost: new deterministic cross-ref checks (openai.yaml / trigger-manifest / env-var contract) and new eval helpers (the selection-collision confusion matrix as a first-class primitive, manifest-driven fixtures + auto-negatives, description-ablation). Indexed in research/README.md (Skills) + cross-linked to plugin-behavioral-findings / measurement-authority / audit-wow-ideas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * style: prettier-format the haretrail-eval-ideas note + README Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add status/topic frontmatter + agent-facing index to research corpus Migrate all 114 research/*.md docs to carry `status:` (active/shipped/idea/ superseded/rejected) and `topic:` frontmatter, so the corpus is sliceable deterministically (`grep -l 'status: shipped' research/*.md`). Add research/CLAUDE.md.spec.ts — a scoped claude() spec whose keyFiles map every research doc to a one-line, status-tagged summary (grouped by topic). Compiles to research/CLAUDE.md, the agent-facing index a session loads as directory memory for research/ without opening 114 files. Mirrors src/CLAUDE.md.spec.ts (the code map); this is the research map. Bidirectional sync is enforced, not by hand: - the compiler verifies every keyFiles path EXISTS (a deleted/renamed doc fails `vigiles compile`) - src/research-index.ts + its dogfood test assert the reverse — every research/*.md (except README.md) appears in the index research/README.md stays the human, thematic index and now points at the machine index + the frontmatter query convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for research corpus index work * feat: thread eval cost into measure/measureArms CheckReport The scored-check paths (`measure`, `measureArms`) were the last eval entry points that didn't report spend — only `runEval` and `measureTriggerRate` did. Close the gap the same way: add `usage: ArmUsage` to `CheckReport` (sourced from the arm's already-aggregated usage), and emit the cost summary in the v8-ignored real-CLI wrappers. `measureArms` sums both arms (an A/B run pays for both). So every real-model eval path now prints tokens + API-equivalent $ + the metered-API warning. Wiring tests drive an injected runner (no model) and assert usage aggregates across trials and per arm. API surface + docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: promote selection-collision to an assertable primitive (measureSelectionMatrix) The cross-skill selection-collision matrix — "when I ask for skill i's job, does ONLY skill i fire?" — already existed but was reachable only from the CLI under the audit consent. Promote it to a first-class, assertable eval primitive so a plugin author can gate collisions in a test: - `measureSelectionMatrix(dir, opts)` — zero-setup: prompts auto-derived from each skill's description (or supply your own). Thin wrapper over the existing `measurePluginSelection`; `measureSelectionMatrixWith` is the injectable core. - `assertNoCollision(report, { maxOffDiagonal, maxPluginCollision })` — the throwing gate. Defaults to demanding ZERO collision; throws on a green-that-tested-nothing (unavailable / zero runs), never a silent pass. Both are exported on `vigiles/claude-code`, NOT the agnostic `vigiles/testing`: selection-collision reads which skill the selector chose, which only Claude Code surfaces (Codex has no skill-selection event), and `scan-behavioral` is Codex-adapter-coupled — so the CC surface is the correct home. Adds a canonical example eval, unit tests (auto-derive + all assert branches, no model), the plugin-author doc's CI path, and API/keyFiles updates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — cost threading + selection-matrix landed * docs: tidy HANDOFF formatting + refresh stale gotchas * docs: mark selection-collision as Claude-Code-only, consistently Make the CC-only surface unmistakable across the docs, one convention: - testing-api.md: a dedicated section "Selection-collision matrix (Claude Code only)" — the "(Claude Code only)" heading + a 🔵 callout naming the `vigiles/claude-code` import — plus a Contents entry and an Imports-section bullet cataloguing which measurements are CC-only vs agnostic. - harness-testing-claude-code.md: a pointer from the Tier-3 trigger section (a CC user's natural landing spot for a CC-only measurement). Complements the existing inline note in for-plugin-authors.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
15c0607a5b |
feat!: eval lock (CI staleness gate) + full Codex parity for hooks & inject (#53)
* feat!: eval lock (CI staleness gate) + per-adapter harness versionKey Real-model evals run on your subscription, so they only run locally — never in CI. The eval LOCK closes the resulting gap: a committed integrity stamp that lets CI verify the committed numbers still match the current inputs WITHOUT a model call (distinct from the gitignored, local-speed eval cache). - `vigiles eval --update` (local): records each NAMED eval's report to a committed `.vigiles/eval-locks/<name>.lock.json` + prints the per-number delta. - `vigiles eval --check` (CI): recompute the input hash, fail "stale" on a mismatch — binary-free, deterministic. Green no-op until the first lock is committed (smooth adoption). - The clean split: the lock stores only the model's observed behavior; the script's own assertions re-run live against the replayed report, so a threshold-only edit is a valid replay while an input change is stale. - Honest scope: the harness version is provenance, NOT a hash input (CI's pinned claude ≠ a dev's local claude would false-trip), and there is no nightly run — model/harness drift is a local `--update` concern. Also extracts the harness version reduction to a per-adapter port method (`HarnessRuntime.versionKey`): Claude Code keys major.minor (~quarterly), Codex opts out with "" (its minor is patch-cadence, ~weekly). This fixes the local cache churning weekly for Codex and is where the cadence decision belongs. Whole-package coherence: the GHA gains `command: eval-check`, `vigiles init` scaffolds the eval-check CI job, and this repo dogfoods it. Wired at both eval seams (runEvalWith + measureTriggerRateWith), 100%-covered, docs updated (internal research + public guides). BREAKING CHANGE: `HarnessRuntime` (exported on `vigiles/adapter`) gains a required `versionKey(raw: string): string` method. Third-party adapters must add it; return "" to opt out of version partitioning. First-party adapters (claude-code, codex) are updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * feat: eval-lock agent awareness (skill + nudge hook) + cohesive-delivery rule Close the loop on the eval lock so the AGENT knows it exists and maintains it — without the invasive option of editing the user's CLAUDE.md. - test-harness skill documents the lock lifecycle (--update/--check, commit the lock, re-run after editing a skill). - A PostToolUse nudge hook (hooks/eval-lock-nudge.sh → hook-runtime eval-lock-nudge): after editing a SKILL.md / *.eval.* with committed locks present, inject a NON-BLOCKING reminder to re-run `vigiles eval --update`. Self-gated (silent until a lock is committed), never blocks (the gate is CI's `eval --check`). Wired into the vigiles plugin; runHook + unit + .sh tests. - NO init edit to the user's CLAUDE.md — verified that's not how plugins deliver context (no plugin-level instructions file exists; the channel is a hook/skill). Docs + dedup: - research/agent-context-delivery.md captures the VERIFIED CC/Codex context-delivery mechanics ONCE (SessionStart additionalContext is the plugin channel; CLAUDE.md is the user's file; skill descriptions always-in-context; Codex AGENTS.md + hooks but inject deferred) so we stop re-researching it. - Per-adapter guides (harness-testing-claude-code/-codex.md) explain how the reminder reaches the agent per harness (CC injects; Codex inject is a documented follow-on), with the agnostic guide linking DOWN to them (no duplication). Cohesive-view rule: - New CLAUDE.md `cohesive-feature-delivery` rule — a definition-of-done CHECKLIST (CLI × GHA × init × agent-awareness × both adapters × per-adapter deduped docs × tests) so a feature's INTEGRATION across surfaces is checked as one thing, generalizing cohesive-cli-surface + harness-parity + great-agent-flow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * test: CLI e2e for the eval-lock invalidation flow (update → check → stale) Closes the gap flagged in review: the joined CLI flow wasn't tested as one thing. A real *.eval.cjs fixture (model-free — drives runEvalWith from dist with a fake runner) run through the built CLI proves the whole loop: `eval --update` writes a committed lock, `eval --check` with the same inputs replays green (no model), and an input change makes `eval --check` go red ("stale"). Binary-free, no subscription — exactly how it runs in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * docs: scannable eval-lock docs + prose-clarity & easy-adoption rules Make the eval-lock docs readable at a glance and frame upkeep as agent-driven, not a manual chore. Docs (scannable rewrite — ASCII flow diagram, comparison tables, ✅/❌/⚠️ signposts, short sentences, no walls of text): - docs/harness-testing.md — the lock how-to now leads with "you mostly won't touch this by hand; the test-harness skill records it + a hook reminds you," then a diagram + tables. - docs/harness-testing-claude-code.md / -codex.md — the per-harness nudge delivery as tables; the Codex inject caveat is one clear ⚠️ row. - docs/eval-architecture.md — the "what's stale vs a free replay" split as ✅/❌ bullets. - skills/test-harness/SKILL.md — Step 5 now tells the agent to run `eval --update` + commit the lock as part of its job (so it IS done for you). - README — the Quick-start reassurance leads with "Adoption is smooth: one command, then your agent does the rest," names skills AND hooks, no chores. Two new CLAUDE.md rules: - prose-clarity — public prose must be clear + scannable (lead with the point, one idea per sentence, define/drop jargon, tables/bullets/ASCII/Mermaid over walls, emoji as signposts, one bold per point). Prefer ASCII/Mermaid; never commit generated raster images into docs. - lead-with-easy-adoption — docs (README first screen especially) must make it clear up front that adoption is low-friction because skills + hooks do the work; frame every feature's upkeep as agent/skill/hook-driven, not homework. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * docs: prose-clarity sweep across 13 user-facing guides Apply the new prose-clarity rule to the prose-heavy public guides: lead with the point, one idea per sentence, tables/bullets/ASCII over walls of text, ✅/❌/⚠️/ℹ️ signposts, one bold phrase per point, jargon glossed or dropped. Net −578 lines — shorter and far more scannable, with meaning unchanged. Swept: for-plugin-authors, what-vigiles-catches, measuring-skills, skills, harnesses, comparison, faq, sandboxing, safety, agent-setup, agent-workflows, verifying-instruction-files, compiled-hooks. The onboarding pair leads with the low-friction "one command, the agent does the rest" framing. Reference docs (cli, testing-api, spec-format, …) and the per-rule docs were deliberately left alone — they're already table-shaped, and churning them is high-risk / low-value. Verified: no heading text changed (anchors intact), no `research/` links added to public docs, the validation-rules matrix is complete (28 rule links → 28), safety claims retained, command refs resolve (self-command-refs), no orphan docs, fmt clean. Produced by parallel rewrite agents under strict preserve-anchors/links/ code constraints, then verified + spot-read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * feat!: encode hook inject support into HookProtocol.injectableEvents Codex inject was documented as "deferred" but the official Codex hooks docs confirm `additionalContext` is honored on SessionStart / UserPromptSubmit / PreToolUse / PostToolUse / SubagentStart — the same shape as Claude Code. The gap existed because inject support was a prose deferral, not a tested contract: nothing in the adapter conformance kit asserted a shellHooks harness could actually deliver an inject hook. Encode it as a required port field so the gap can't recur silently: - HookProtocol gains `injectableEvents: readonly string[]` — the events that honor `additionalContext` injection, per-adapter. - claudeCodeHookProtocol: [SessionStart, UserPromptSubmit, PostToolUse]; codexHookProtocol: + [PreToolUse, SubagentStart]. - Conformance rejects a shellHooks adapter with an empty list; the adapter-contract suite asserts every shellHooks adapter declares the events vigiles' shipped hooks use (PostToolUse, SessionStart) — the test that would have caught the original miss. - `compile` warns on an inject hook only when its event isn't injectable on the target harness (react output stays CC-confirmed-only). - Correct every "Codex inject deferred" reference across docs, research, and CLAUDE.md; document injectableEvents in the adapter API + authoring guides. BREAKING CHANGE: HookProtocol (exported on vigiles/adapter) gains a required `injectableEvents` field — a third-party adapter must declare it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for the eval-lock branch + inject-port encoding Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: fan out hook install + the CI workflow to every declared harness A repo can enable BOTH harnesses (`harness: ["claude-code","codex"]`), and instruction files already mirror to both — but `compile` installed a hook into only ONE harness (`installHooks` ignored `config.harness` and auto-detected a single adapter), and the scaffolded CI workflow hardcoded the Claude Code binary. So an inject/gate hook that works on both could land in `.claude/settings.json` while silently missing from `.codex/config.toml` — the "works on CC, absent on Codex" failure the harness-parity rule exists to prevent. - `resolveHarnessAdapters` (adapter-registry.ts) returns the FULL declared set: `--harness=` flag → that one; else config `harness` list → all of them; else auto-detect → the detected one (de-duped by name). - `installHooks` loops over it, merging the SAME compiled hook into every enabled harness's native config (JSON / TOML), with per-harness warnings. - The `init` CI workflow is harness-aware: the deterministic harness job installs `@anthropic-ai/claude-code` and/or `@openai/codex` per declared harness (`harnessTestBinaries`). Skills compile byte-identical across harnesses and subagents are CC-only by design, so hooks + the CI binary were the real fan-out gaps. Tests: resolveHarnessAdapters units (fan-out / flag-singular / detect / alias-dedup), a both-harness hook-install e2e (the same hook in both configs), and harness-aware workflow e2es (codex-only, both). Documented as §4 in research/multi-harness-compile.md + the compiled-hooks guide. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: wire vigiles's nudge hooks into .codex/config.toml during init Closes the last Codex-parity gap in agent-awareness: Claude Code gets the eval-lock + refs nudge hooks from the global marketplace plugin, but Codex has no global plugin store, so it got skills only — the proactive nudges never reached a Codex user (the "works on CC, silently absent on Codex" failure). Codex config is repo-committed (the idiomatic place), so `init` now wires the two PostToolUse nudges into `.codex/config.toml`: - `codexPluginHooks` (pure) declares them as DIRECT `npx vigiles hook-runtime …` commands — no plugin root, no vendored bash script. The runtime entrypoints read the event JSON on stdin and emit the `additionalContext` shape Codex honors on PostToolUse (confirmed + encoded in HookProtocol.injectableEvents). - `applyCodexPluginHooks` (pure, unit-tested) merges them idempotently: re-runs replace in place, and the user's own Codex hooks + every other config key are preserved. `wireCodexHooks` in cli.ts does only the IO. - Safety: only an INTENTIONAL exit 2 blocks an edit (the refs nudge under `unmarked-refs: error`); an npx-resolution failure exits non-2, so a missing dep never blocks. Loud, documented deferral (no-silent-skips): the SessionStart lint summary (plain stdout, unconfirmed on Codex) and the compile-on-edit / pre-edit guards (filename-gated bash, no harness-neutral entrypoint yet) stay manual on Codex. Also: the remaining pre-release Codex item — running the eval tier against the real `codex` binary once — is added to the launch-readiness P0 list in the roadmap. Tests: codexPluginHooks shape + applyCodexPluginHooks (fresh / idempotent / preserves user config); the planPluginInstall codex notes/vendors updated. Verified end-to-end that `init --harness=codex` writes the config and a pre-existing `.codex/config.toml` (model + a user hook) is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: update codex install e2e for the new vendors:true (repo hooks) The codex install plan now writes nudge hooks into .codex/config.toml, so `vendors` flipped to true. Updated the unit assertion last commit but missed this e2e (run by both the `e2e` and `test` CI jobs). The skills-are-global checks below are unchanged and still stand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: cover the non-Actions lock message branch + address Codex review The `test` job's coverage gate failed: `emitLockMessage`'s console.warn/ console.log branches (the non-GITHUB_ACTIONS path) are unreachable in CI, where GITHUB_ACTIONS is globally set. Added a wiring test that DELETES the var to exercise both branches regardless of environment. Also addresses two Codex review findings on the eval-lock work: - P1 (soundness): `evalArmsInputs` omitted `spec.stubs` from the hash, though tool stubs are written onto PATH and are model-facing input — so a changed canned CLI output after `--update` left the same `inputsHash` and `--check` replayed stale results. Now folds a canonical (name-sorted) view of the stubs into the hash; a new wiring test proves a stub change → stale. - P2 (collision): two eval names normalizing to the same slug (`foo/bar` vs `foo bar` → `foo-bar.lock.json`) could let `--check` replay the WRONG eval's report. `readLock` now rejects a lock whose stored `name` differs from the requested one, degrading a collision to a safe miss (→ stale/re-run), never wrong data. Unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: two more eval-lock soundness gaps from Codex review - An unnamed eval in `--check` mode called `produce()` (ran the model), breaking the no-model-in-CI contract: in CI there's no subscription, so it would hit the model or fail for missing auth. Now `check` mode throws a clear, actionable error for an unnamed eval (never reaching the model); `off`/`update` keep producing (the eval just isn't gated locally). - `spec.ephemeralEnv` toggles the trial's run environment (scrubbed env + throwaway HOME vs the inherited process env) — a model-facing input that was omitted from the lock hash, so a report recorded under one mode could be replayed under the other with the same inputsHash. Now folded into the hashed inputs. Both proven by new wiring tests (unnamed-check throws without calling the runner; an ephemeralEnv toggle → stale). Coverage holds at 100% lines/ statements/functions (verified with GITHUB_ACTIONS=true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fold the harness identity into the trigger-rate lock hash Codex review (P2): a trigger-rate eval recorded with one harness driver and later switched via `measureTriggerRate(spec, { evalDriver: codexEvalDriver })` hashed identically (prompts/model/plugin unchanged), so `--check` could replay a Claude-recorded report for a Codex run — defeating the gate for the Codex eval path this PR adds. `EvalDriver` gains an optional `harness` identity (defaults to "claude-code", so existing single-harness locks are unaffected); `claudeEvalDriver` and `codexEvalDriver` declare theirs, and `measureTriggerRate`/`...With` thread it into the hashed trigger inputs. A new wiring test proves a harness switch → stale. API surface regenerated (additive `harness?` on EvalDriver). This closes the input-hash bug class on the trigger-rate seam (the runEval seam's stubs + ephemeralEnv were fixed in the prior two commits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — PR #53 merge loop + the 5 review fixes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: three more Codex-review findings (matchers + plugin-root path hash) - Codex nudge hooks used Claude tool names (`Edit|Write`) in their matcher, but Codex's edit tool is `apply_patch` — so the eval-lock + refs nudges wired into `.codex/config.toml` would never fire. Matcher → `^apply_patch$`. - The compiled-hook FAN-OUT to Codex regex-wraps the raw matcher without translating tool names, so a CC-authored `tools("Edit","Write")` gate emits `^(Edit|Write)$` into Codex config and silently never fires though compile reports success. vigiles has no cross-dialect tool-name map yet, so install now WARNS LOUDLY when a tool-matched hook is fanned to a non-CC harness (no silent non-firing success); a real mapping is the follow-up. - For eval arms using `plugin`, `resolveHarness` expands `${PLUGIN_ROOT}` to the checkout's ABSOLUTE path before the settings are hashed, so a lock recorded at `/home/dev/...` was falsely STALE when `--check` recomputed it at `/home/runner/...` in CI. `stripPluginRoot` normalizes the prefix back to a token → location-independent hash. New wiring test: the same plugin at two different abs paths replays (not stale). Coverage holds at 100% lines/statements/functions (verified GITHUB_ACTIONS=true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: guard stripPluginRoot against a hookless plugin's undefined settings A plugin that ships only instructions/skills (no hooks) resolves to settings:undefined; JSON.stringify(undefined) is undefined, so the prior .split() threw before the eval could run. Guard the non-string case (pass it through). Regression I introduced in the plugin-root normalization last commit; new test covers a hookless-plugin arm (update + check, no throw). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: add a lock-hash COMPLETENESS guard for both seams The Codex review kept finding the same class of bug: a model-facing input omitted from the lock hash, so `--check` replays stale results (stubs, ephemeralEnv, driver-harness, plugin-root all slipped through). Our tests proved the MECHANISM (record→replay, task-change→stale) but never asserted the input set was COMPLETE. This adds a structural guard: for each input that steers the run, mutate ONLY it and assert the lock inputsHash CHANGES — one parametrized test per seam (runEval's evalArmsInputs + trigger-rate's triggerInputs). A new input added to EvalSpec/TriggerRateSpec without hashing it now fails a row HERE in CI, instead of a reviewer finding it later. Covers task/model/tools/fixture/ arm.files/arm.settings/arm.model/interceptTools/stubs/ephemeralEnv and prompts/irrelevantPrompts/model/harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
1767bd3990 |
feat!: audit Safety ring + seven deterministic harness checks, README to 5/5
Adds a Safety ring + seven deterministic, FP-safe harness checks to `vigiles audit` (lethal-trifecta, delegation-trifecta, skill-resource-resolves, skill-missing-fence, plugin-dir-layout, hook-block-ineffective, hook-matcher), each wired into both audit and lint from one shared detector. Declares every rule's decidability bucket in a RuleMeta registry, documents the enforcement model, and lands the README at 5/5. BREAKING CHANGE: the HarnessDialect field `blockingHookEvents` is renamed to `noEffectHookEvents` (plus a new `permissionDecisionHookEvents`), and the public surface gains the RuleMeta registry exports. Adapters that referenced `blockingHookEvents` must use `noEffectHookEvents`. |
||
|
zernie
|
ff542b7c4b |
refactor!: vigiles audit — Lighthouse-style harness report (rename scan→audit)
Rename the read-only scan verb to `vigiles audit` — a local, zero-config Lighthouse-style report scoring a harness across four deterministic rings (Truthfulness/Triggering/Structure/Tested) with each finding's fix inline + a shareable HTML report and a versioned AuditReport JSON.
audit is a local report, not a CI step (CI uses `vigiles lint`). The executing checks (live MCP resolution + trigger-rate + the adoption preview) run only behind one interactive consent; there is no execution flag. The safety battery is not an audit ring — it lives in the vigiles/testing API.
Adds the adoption-gateway preview ("what would vigiles catch in your repo?"): the model drafts a spec from the instruction file, the cross-reference engine verifies every drafted ref deterministically.
BREAKING CHANGE: the `scan` verb is renamed to `audit` (no alias); the `--check-hooks`/`--verify-mcp`/`--trigger`/`--fix-plan`/`--explain`/`--deep`/`--measure`/`--fast`/`--no-measure` flags are removed.
|
||
|
zernie
|
e8fa3b6b78 |
refactor!: pre-release polish — surface freeze, spec-first defaults, auto-adopt (#48)
* chore: refresh HANDOFF — PR #47 merged, surface-freeze still open Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: freeze pre-release public surface + collapse markdown ladder Pre-release surface freeze (research/pre-release-focus.md): - Mark the experimental typed-composition cluster @internal in vigiles/spec (pipe/pipeStep/needs/start/andThen + Supplies/Handoff/KnownAgentName/ Pipeline/PipeStep/NeedsContract) and effect()/EffectRegion. They stay exported at runtime (no removal) but are excluded from the published API docs and flagged unstable in the committed surface reports, so a later change to them is not a breaking change to a documented-stable API. - Note evolve.ts as @internal/research-stage (already off every public entry). - Markdown cut: drop the "Level 0/1/2" ladder framing across public docs; collapse to two on-ramps (plain markdown via inline enforce comments -> typed spec). Frontmatter mode is KEPT, still works, and its full docs + the old ladder are PARKED via an HTML comment in docs/markdown-mode.md (matching the README's parked-section convention) — nothing deleted, trivially un-parked post-launch. - Ship STABILITY.md (0.x: CLI stable; library API evolving; experimental surfaces marked) + a one-line README pointer. - Regenerate etc/*.api.md to reflect the @internal tags (api:check green). BREAKING CHANGE: public symbols in vigiles/spec are re-tagged @internal (typed-composition combinators, effect()). They remain importable but are no longer part of the frozen, supported surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: surface the author-time-vs-runtime wedge + citeable authorities - README hero: state the differentiator positively — vigiles proves the harness "before the agent ever runs" (author-time / pre-run), ending on verify/test/measure (no negative opener, per the README direction rules). - pre-release-focus.md Positioning lock: fold in the canonical one-line wedge (author-time vs runtime, public-safe phrasing, no competitor names) and the citeable harness-engineering authorities (Karpathy "automate what you can verify"; OpenAI "the harness is hard"; arXiv AHE structure-beats- prose = enforce > guidance) for the launch article. The AHE prediction-loop note already lives on the roadmap A2 optimizer item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the README hero with the concrete pain (untested harness) Reframe the hero to go straight to the developer pain: you installed plugins and wrote skills but don't know if they work — a library with no tests. Replaces the "100x coder / 1x verifier" lead (kept as a backup hook in the positioning lock) with the concrete plugins/skills situation + silent-fail triad, ending on the verify/test/measure benefit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the guide openings with the concrete pain, not the mechanics Apply the README's pain-first hook to the subdoc openings: - verifying-instruction-files.md: open with the CLAUDE.md-drifts pain instead of "vigiles's linting layer ... checks every reference". - for-plugin-authors.md: add the installer pain hook (do your skills fire for the people who install it?) above the audience-routing intro. - harness-testing.md: sharpen the lead to the concrete "does it actually block? does your skill fire?" question. Each keeps its README uplink + "what this doc is" line (docs-quality). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: codify the pain-first principle + fix a stale doc link - README direction comment: add rule 1c — lead with the concrete pain in the reader's own situation (the specific silent failure + the "library with no tests" anchor), for the hero AND every subdoc opening; reconcile with 1b (a pain about the reader's situation is a hook, not an apology). - Fix a stale link: src/vendor.test.ts -> src/adapters/claude-code/vendor.test.ts (moved in the adapter relayout). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — surface freeze + launch polish shipped on branch Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs(positioning): not a linter — spec-first + eject, don't fight on structure-lint Lock the decision (Positioning lock + roadmap launch-readiness): vigiles is NOT a markdown/structure linter and won't contest that crowded, commoditized lane. The bold differentiated play is author-time typed specs the agent writes + evals you can afford. Markdown demotes from on-ramp to floor / eject-target. Names the enabling build — a one-command `vigiles eject` (strip the integrity header -> clean owned markdown) so "always ejectable" is true and the two spec objections (TypeScript friction, lock-in) dissolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: add `vigiles eject` + reframe spec-first (managed, but ejectable) The escape hatch that makes "adopt a typed spec, you can always eject" true, and the positioning shift that leans on it. - `vigiles eject [file]`: the inverse of compile. Strips the vigiles:sha256 integrity header -> plain hand-owned markdown, removes the managing spec (--keep-spec to keep it), and adds a `vigiles-disable require-spec` marker so lint stays quiet. Pure helpers (parseIntegrityHeader/ejectMarkdown in core/integrity.ts) + handler + dispatch + help + docs/cli.md. New verb in cli-commands.ts (additive). Unit + e2e tests. - Positioning reframe: don't contest the structure-lint crown. README hero sub-line + spec mentions lead with "your agent writes the spec, and you can eject anytime"; markdown demoted to floor/eject-target (still the zero-TS on-ramp). README direction rule 1c codifies the pain-first hook. - adopt-spec skill: faithful-by-default + eject note; dropped the Level 0/1 ladder pitch; fixed stale commands (check->lint, generate-schema->generate schema, Action @main command:check -> @v1 command:lint). - Closed the scan gap: skills/ now in the self-command-refs dogfood, which caught two rotted `vigiles generate-types` refs (now `generate types`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(scan): nudge toward the real-model trigger tier; agent-safe by default When a plain `scan` finds model-invocable skills AND a model is reachable (ANTHROPIC_API_KEY, or an authenticated Claude Code session via CLAUDECODE / CLAUDE_CODE_ENTRYPOINT), surface the real-model `--trigger` tier that measures whether those skills actually FIRE: - A human at a TTY is OFFERED setup (scaffold a trigger-prompts.json, or run an existing one — the real-model run is always an explicit yes). - An agent / CI run (non-TTY / --json / --no-interactive / --yes) gets a one-line, non-blocking hint and never a prompt (great-agent-flow). Pure core (hasModelAccess / decideTriggerSuggestion / formatTriggerHint / scaffoldTriggerPrompts) + unit tests; CLI wiring + env-gated e2e in scan-cli.test.ts; docs in docs/cli.md. Also records (HANDOFF) that real-model tiers ARE runnable in Claude Code web: the claude CLI is authenticated via session OAuth (no API key), proven by `claude -p` and a live `scan --trigger` run (test-harness recall 100%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: --strict gates broken surfaces (CI red); fix `lint <dir>` crash The actual "edit a skill/subagent/hook badly → CI goes red" payoff. - `vigiles init --strict` now promotes the HIGH-PRECISION, FP-safe structural rules to `error` (not just require-spec): subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap. So a typo'd tool, a dead hook script, a misconfigured MCP server, or two colliding skill descriptions FAIL `vigiles lint` (exit 2). Never clobbers a user-set severity; leaves the lenient/recommendation rules (frontmatter-valid, skill-frontmatter, prefer-compiled-hooks, untested-*) at warn. Honest limit documented: CC itself loads a name-less / broken-YAML skill, so vigiles gates the subagent/hook/MCP defects + skill collisions that genuinely break, not skill content that still works. Proven: clean subagent exit 0, typo'd tool exit 2. - FIX: `vigiles lint .` (a directory arg) crashed with EISDIR (it read `.` as a file). findInstructionFiles now expands a directory arg to the instruction files inside it; a file arg and a non-existent arg pass through unchanged. Tests: setup-plan (strict gate set), cli (lint-dir no-crash + strict exit-2 regression). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: init gates broken surfaces BY DEFAULT (no --strict needed) Fold the structural gating into a plain `init` — no confusing "strict mode" to remember. A default `init` now writes the 9 HIGH-PRECISION, FP-safe structural rules as `error` (subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap), so a broken subagent / hook / MCP ref or a skill collision FAILS `vigiles lint` out of the box — while a well-formed plugin stays green (never cries wolf). `--strict` is reframed to the WORKFLOW-FORCING tier a clean repo can still fail because the work isn't done yet: require-spec, untested-skill/subagent/hook, frontmatter-valid, skill-frontmatter. Opt-in by design (progressive adoption), so a first CI run isn't red just for not having written a spec yet. Never clobbers a severity the user already set. vigiles dogfoods it — the 9 gate rules added to its own .vigilesrc.json (self-lint stays green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(init): interactive init offers the workflow tier (recommended, opt-out) Per the chosen default model: structural gating is always on; the WORKFLOW tier (require specs + a test per surface) is the recommended default a human opts OUT of, never forced on a silent run. - Interactive `init` now asks "Also enforce specs + a test per surface (recommended)? [Y/n]" (default yes) → sets strict. SetupAnswers/applyAnswers carry it; resolvePlan honors it. - A bare NON-interactive `init` (agent/CI, no one to ask) stays structural-only unless `--strict` — so an automated setup never silently turns an existing repo's CI red just for not having written a spec yet. - README agentic-install prompt + docs/cli.md updated: the agent asks before enforcing the workflow tier, structural gating is the safe default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — spec-first defaults + eject + scan nudge; PR is the next step Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * test(init): make the interactive setup Q&A unit-testable (inject ask) The interactive init prompts were the one untested-and-fragile bit of the setup flow (readline can't run in CI's non-TTY env, so the questions/defaults/ answer-mapping had zero coverage). Extract the Q&A into a pure collectSetupAnswers(ask) with an injected AskFn; promptSetup is now just the readline IO shell. Unit-tested with a fake ask: all-defaults, lint/test pillar selection, and declining CI/plugin/strict — deterministic, no TTY, no model. Now the init flow is covered end to end: pure plan logic + the interactive Q&A (unit) + non-interactive init (e2e in cli.test.ts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: install-enforcement model — rule groups (not presets) + rationale Capture the reasoning, not just the outcome, before implementing. - New `install-enforcement-model` rule in CLAUDE.md (via the spec): rules are organized into GROUPS by confidence (structural / workflow / nudge), NOT a relaxed/standard/strict preset MENU — the linter best practice (Clippy/Ruff/ Biome; "presets are a poor emulation of rule groups"). default = structural at error; --strict adds workflow; --report-only is an orthogonal severity dial. Installation is EXPLICIT ABOUT TRADEOFFS (one source reused across the CLI prompt / agent AskUserQuestion / summary). - Reconcile smooth-adoption + progressive-adoption to the new stance: "permissive = doesn't force specs/TS + doesn't cry wolf, NOT ignores breakage." - research/install-enforcement-dx.md: the full design record — the linter landscape (verified), why we rejected a preset menu, the group→rule mapping, the agentic-install DX (no prior art), and the philosophy reconciliation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: rename require-spec → require-instructions-spec; rule groups; init auto-adopt Rename + narrow the rule, name the install rule-groups, and make `init` faithfully adopt existing instruction files so the rule is satisfied by construction. - require-spec → require-instructions-spec, NARROWED: only a `.spec.ts` satisfies it — inline `vigiles:enforce` / `vigiles:` frontmatter no longer do. Disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. - Name the rule groups in src/setup-plan.ts (STRUCTURAL_RULES / WORKFLOW_RULES / NUDGE_RULES) and add `--report-only` (writes the whole gate at warn). - prefer-compiled-hooks defaults off; require-skill-spec un-deprecated (the consistent require-<surface>-spec parallel, default off). - New src/core/adopt.ts: adoptMarkdown/adoptToSpec faithfully convert an existing CLAUDE.md/AGENTS.md into a claude() spec (heading → verbatim section, no rule inferred, always compiles). Wired into init()/setupPillar1; round-trip tested (byte-identical below the integrity header). - Sweep stale Level-0/1/2 markdown-mode comments; docs + CLAUDE.md recompiled. BREAKING CHANGE: the `require-spec` lint rule and config key are renamed to `require-instructions-spec` and narrowed — only a `.spec.ts` satisfies it (inline comments / `vigiles:` frontmatter no longer do), and the per-file disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. No automatic alias; update `.vigilesrc.json` and any disable markers. * fix(init,eject): honor interactive strict; don't delete a shared spec on eject Addresses two Codex review findings on #48 + the fmt:check CI failure. - setup(): write `.vigilesrc.json` and the summary from the RESOLVED plan (`plan.strict`), not the raw `--strict` flag — so an interactive "yes" to the workflow tier is actually persisted instead of silently dropped. - eject(): only delete the header's `compiled from` spec when it EXCLUSIVELY belongs to the file being ejected (`<file>.spec.ts`). A mirrored secondary target (e.g. AGENTS.md compiled from CLAUDE.md.spec.ts) names the shared source; deleting it would break the primary's recompile. Keep it + warn instead. Regression test added. - Format HANDOFF.md (the fmt:check failure). * fix(init): non-destructive adopt, deferred compile, test-only & strict config gating Install-DX hardening from PR #48 review + a self-audit across user/project types. - Auto-adopt is now NON-DESTRUCTIVE: `init` writes the spec but never overwrites your CLAUDE.md/AGENTS.md. Run `vigiles compile` to opt the file into spec management (byte-faithful; review the diff; `eject` reverses it). - Defer the compile when `vigiles` isn't resolvable yet (a fresh repo hasn't run `npm install`) — print the next step instead of a "failed to load" error. - Test-only setup (`init --test` / `--no-lint`) records the harness but writes NO lint rules — honors the positive-flag contract (Codex review). - setup() writes config from the RESOLVED plan, so an interactive "yes" to the workflow tier persists (was dropped). - Next steps reordered (install → compile → strengthen → test); `--report-only` in `init --help`; README + docs surface non-destructive adopt. New e2e tests. * fix(adopt): don't drop intro text when a literal ## Overview already exists Codex review (P2): a file with intro prose before the first heading AND an explicit `## Overview` section produced two sections keyed "Overview" — building the sections object overwrote the synthesized intro, silently dropping it (and a later `compile` would remove it from the file). Extract a shared `allocKey` dedup helper and use it for the synthesized Overview too, so it becomes "Overview (2)" on collision instead of clobbering. Regression test added. * fix(eject): preserve frontmatter on skill/agent eject; keep shared multi-target specs Two Codex review P2s on the eject feature. - ejectMarkdown no longer prepends the require-instructions-spec disable marker before a body that leads with YAML frontmatter (a compiled SKILL.md / subagent): doing so displaced the frontmatter and lost the surface's name/description/tools. The marker is instruction-file-only, so a frontmatter-led body ejects verbatim. - eject deletes the header's spec only when NO other compiled output still references it (a directory scan), instead of a basename heuristic. A multi-target spec (`target: ["CLAUDE.md","AGENTS.md"]`) or mirror now keeps its spec until the LAST output is ejected — ejecting either the primary or a secondary no longer orphans the others. The trailing marker hint prints only when a marker was added. Tests: frontmatter-preserved eject (unit) + eject primary/secondary/last-output of a multi-target spec (e2e). Docs note the two safety behaviors. * fix(eject,scan): path-safety on spec deletion; gate scan prompt on stdin TTY Two Codex review P2s. - eject: the `compiled from <path>` header is untrusted — a hand-edited/forged header naming `package.json` or a `../` path would make rmSync delete an unrelated file. Only remove a `.spec.ts` that resolves INSIDE the project; otherwise keep it and warn. Regression test added. - scan --trigger nudge: prompt only when BOTH stdout AND stdin are a terminal — `askOnce` reads stdin, so a TTY stdout with piped/redirected stdin (agents, pipelines) must take the non-blocking hint path instead of blocking on a read. * docs(readme): add FAQ + docs/faq.md, tighten install prompt, lift adoption line 5-star onboarding pass (1/N). - New docs/faq.md: the objection-handling answers (why a typed spec & is it optional, do I write TS, does init touch my files, CC+Codex, cost/key, non-JS repo, safe to run, vs promptfoo). - README: compact FAQ section linking it; install prompt rewritten to plain English (names adoption, tells the agent to finish: install + compile, no flag jargon); the non-destructive adoption reassurance lifted above the fold. * fix(init): honest guidance for non-JS repos (no package.json) 5-star onboarding pass (2/N). A Python/Rust/Go repo with a CLAUDE.md but no package.json can't resolve the vigiles npm package, so the typed-spec compile path isn't available there. Previously init told them to run `npm install` (which wouldn't help). Now it points at the real paths: - the deferred-compile notice and the adopted next-step both detect a missing package.json and recommend `npx vigiles lint` (verifies with zero install) plus `npm init -y && npm i -D vigiles` to enable specs. (Dropped the post-init lint "aha" idea — a freshly-adopted file is guidance-only with no marks to verify, so it would show nothing until `/strengthen` runs.) Test: init on a no-package.json repo guides to lint, not a bare npm install. * docs: add scripts/demo.sh — self-contained asciinema demo of the Lint layer 5-star onboarding pass (3/N). A zero-dependency, reproducible script for recording an asciinema cast / GIF: it builds a throwaway repo whose CLAUDE.md points (via inline marks) at a file that moved and a script that was renamed, then runs `vigiles lint` to show the ✗/✓. One command to record: asciinema rec --command "bash scripts/demo.sh" Tested end-to-end (real ✗ "File not found" / "Script not found" output). Consistency spot-check: public docs carry no stale adopt/compile language and all README-linked guides open with a README uplink. * docs(readme): final 5-star pass — tighten spec blurb, wire demo GIF slot 5-star onboarding pass (4/4). - Tighten the above-fold spec/adopt/eject paragraph (the new FAQ now carries the "do I write TS / will it touch my files" detail), and point new readers to it. - Wire a DEMO GIF slot at the prime "see it work" spot with the exact asciinema/agg recording command (scripts/demo.sh) + the embed snippet. README rendered content ≈152 non-blank lines (well within the ~200 cap); all internal doc links resolve; command refs pass the self-command-refs dogfood. * docs(handoff): record the 5-star onboarding-polish pass * docs(readme): add the demo GIF — vigiles lint catching stale references Animated terminal of `vigiles lint` flagging a moved file + a renamed script, at the README's prime "see it work" spot. The output lines are verbatim from the real CLI (nothing fabricated). Generated headlessly with Pillow — no recorder needed — so it's reproducible: `python3 scripts/make-demo-gif.py` (generator committed); scripts/demo.sh remains for a live asciinema recording. * docs(rules): sync rule docs with this session's default changes doc-per-rule drift caught auditing the session: - prefer-compiled-hooks.md: default is now OFF (was documented "warn") — a recommendation that shouldn't fire unasked. - require-skill-spec.md: drop the "deprecated" framing — it's the consistent require-<surface>-spec parallel (kept, default off), matching the code + matrix. * fix(eject): scan the whole project tree for shared-spec references, matched by resolved path The shared-spec guard scanned only the ejected file's directory, so a multi-target spec emitting a secondary output in a subdirectory (target: ["CLAUDE.md", "docs/AGENTS.md"]) was missed — eject would delete a spec still feeding another compiled file. Walk the whole project tree (skipping node_modules/.git/dist/build/coverage/.next/out) and match on the resolved spec path, not the basename, so two unrelated CLAUDE.md.spec.ts in different packages don't false-positive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
d8c639f8da |
refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46)
* docs: sharpen README lint/guard copy - Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff, Clippy, and four more)"; Cedar stays in the subdoc - Drop the plugin-audit paragraph from the Lint section (secondary use case, still reachable from More) - Rewrite the Guard intro to lead with the concrete "fail open" failure and name what vigiles emits (exit code, JSON, AST-backed matcher) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface the $0/subscription cost story in the first paragraph The "no metered tokens, runs on your Claude subscription" wedge was buried in the last line of the Eval section. Pull a crisp version into the opening paragraph (above the fold); the Eval section keeps the detailed promptfoo/DeepEval contrast as proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface agent-writes-the-harness flow, reorder More, drop internal links - Quick start now leads with "then your agent writes the harness for you": the three model-invocable skills (test-harness/strengthen/ edit-spec) and the plain-English ask each one answers - Reorder More so CLI sits above the leaderboard (whose link points into the CLI docs); trim the now-redundant Skills bullet to a pointer - Stop linking internal research/ docs from the README (the moat link and the Research index) per public-vs-internal-docs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(compile): auto-refresh an existing harness.gen.ts compile now keeps a whole-harness registry in sync as a side effect, so the user never hand-runs generate-harness. Gated on the file already existing (compile maintains a registry you opted into, never imposes one) and cheap (parses specs, no linter spawn — unlike generate-types/ generate-schema, which spawn every linter and stay on the config-change guard). A duplicate agent name fails the compile. Docs: note the auto-refresh in the compile section and mark the generate-* artifacts as dev-toolchain output rarely run by hand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule Directory-scoped guidance loaded when working in src/: adding a new CLI verb or hook-runtime subcommand must clear a high bar (genuinely new mental action, no overlap with an existing command, siblings collapse to one verb + subcommand, runtime entrypoints stay hidden). Complements the root cohesive-cli-surface rule, placed next to the code that defines the command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec + integrity); dogfoods vigiles compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the "(generated)" annotation from the API reference link Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule The nested src/CLAUDE.md.spec.ts (added last commit) compiles under rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so both load via the tsx fallback (as the root always has). Also adds the no-internal-links-in-public-docs guidance rule (public docs never link research/*.md; keep the fact in research, don't link it), reconciles the old "link to research" line in public-vs-internal-docs, and files the deterministic lint-rule version as a P1 roadmap item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: separate public/internal indexes + surface eval safety - docs/README.md no longer links into internal research/ (public docs don't point at the private record) - research/README.md is explicitly labelled the internal record - README Eval section gains a "Safe to repeat" note (sandbox + ephemeral dir + egress + interceptTools, past-the-sandbox case flagged) linking docs/safety.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: stop public docs linking internal research/ + add safety FAQ Strip every research/*.md link from docs/*.md (the no-internal-links-in- public-docs rule) — ~60 links across 16 guides plus the docs/ index; prose kept readable, no orphaned research doc (all still cited from CLAUDE.md keyFiles / research/README.md, orphan-docs green). Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox question honestly: network via egress allowlist, a real DB/redis/browser via a disposable container you stand up (turnkey provisioning is a committed next step), irreversible externals via interceptTools, and the no-Docker deterministic/record-replay tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind> De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands rule, dogfooded). All breaking: - `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral column folded into scan; trigger-rate + collision matrix unchanged) - `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic WHY, a sibling of --fix-plan on the same ScanReport) - `refs <file>` → folded into `lint` (the unmarked-refs check is already a lint rule; the PostToolUse hook keeps its shared collectRefIssues entry) - `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand; the three dev-toolchain generators grouped, not N hyphenated siblings) scan reuses handleMeasure/handleExplain directly behind --trigger/--explain. Docs, help text, the regen guard, tests, and the self-command-refs dogfood all updated together (cohesive-cli-surface); 1588 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: split the GitHub Action reference out of cli.md into its own doc cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action section into docs/github-action.md (quick start, inputs, output channels, versioning) and leave a short pointer; retitle the cli.md intro. Update the README More bullet (CLI · GitHub Action) and the docs index. cli.md → 619 lines. orphan-docs + self-command-refs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks A single repo-level recommendation (one finding regardless of hook count): when a plugin ships hand-written hook commands that aren't compiled vigiles/hook artifacts, nudge toward compiling them — the message links docs/compiled-hooks.md and points at guardrail-check for verifying the ones you have. A discovery nudge, not a defect: the hand-written shell lane stays first-class, so it fires ONCE (never per-hook), defaults to warn (info- level ℹ; informational, doesn't fail CI), and is opt-out. Detector (manualHookCount: a command that isn't a hook-runtime invocation) lives in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc + matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's form-based, so it stays a recommendation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the eval-startups critique mapped onto positioning New research/eval-startups-positioning.md captures Thomas Liao's "why eval startups fail" critique and its implications: vigiles dodges the trap (harness-eval not model-eval, tooling not services, eval already a feature), but it's a yellow flag on the public ecosystem-benchmark as a MOAT (Goodhart-gameable → keep it a thin acquisition flywheel) and it independently validates the attestation/safety direction (capability-diff, guardrail proof) as the only defensible eval-shaped niche. Linked from research/README.md, measurement-authority.md (whose benchmark thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2 conformance/attestation play it validates). Internal-only per no-internal-links-in-public-docs; orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: purge internal positioning from public docs (jargon, name-drops, benchmark) Public docs (README + docs/) now state user-facing value, not vigiles's internal strategic vocabulary: - strip "moat" from 14 places → user-facing wording (engine / cost advantage / differentiator / cross-reference) - remove all bare research/*.md name-drops from docs/ (the backtick form the link-strip + deterministic check miss); scope-descriptions reworded to "configured directories" rather than naming research/ - README: drop the "ranks the hyped ecosystem: what works vs hype" benchmark banner from the Eval row + kicker (depth stays in measuring-skills.md) — per eval-startups, the benchmark is a flywheel, not a headline - docs/cli.md: "moat #2 (research/…)" → "the capability-diff check" Codified in CLAUDE.md: public-vs-internal-docs now has a strategic- vocabulary clause (moat/measurement-authority/flywheel stay internal; public docs name the benefit), so this doesn't regress. orphan-docs + self-command-refs green; the public↔private split stays docs/ vs research/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): apply multi-persona review fixes From a multi-persona README review (power user / staff eng / plugin author / non-TS user / eng manager): - non-TS users: a visible "No TypeScript needed — lint + inline comments work on any CLAUDE.md" line (was invisible; contradicted progressive adoption) - plugin authors: surface "vigiles scan ranks a marketplace, no key" above the fold (was buried in More) - eng-manager cost: hero now says deterministic layers run free in CI; merged kicker+affordability nails "real-model on a dev's own subscription, never a CI token" - staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted"; the typed-pipeline claim is concretized to the real pipe()/needs() mechanism - fixed a broken blockquote and trimmed the safe-to-repeat para to hold length Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): capture the README design direction as an editor-facing comment A top-of-file comment block records the framing the README must hold: lead with benefits, present compiled specs as the easy/auto-enforced next step (skills write them, hooks compile them; JS→TS evolution, not a wall), keep all four instruments first-class incl. Eval, stay scannable + under the length cap, and no internal vocabulary/research links. Guides future edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): benefits-forward rewrite + specs-are-easy framing Per the README direction: lead with what the user GETS, and present the typed spec as the frictionless auto-enforced next step, not a TS wall. - hero: stronger, concrete failure modes (stale ref / guard fails open / skill never fires), keeps Agent=Model frame, cost above the fold; dropped the redundant centered pitch (merge) - table cells condensed to punchy benefit one-liners - "You don't hand-write specs — your agent does. TypeScript for your harness: opt-in, gradual, auto-compiled, never a chore" (skills write them; JS→TS evolution) - de-dupe cost (drop "not metered tokens" from the Test cell) - "Not for you if …" honest-scope line - docs/verifying-instruction-files.md: back the claim — "you rarely hand-write one; edit-spec/strengthen write it, a hook recompiles on save" Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): don't lead callouts with "typed spec" — benefit first Casual readers shouldn't be greeted by "a typed spec is a program" / "TypeScript for your harness" in the prominent callouts. Reframe both to lead with the outcome and demote the spec to a soft, opt-in aside: - specs-are-easy block: "You don't hand-write any of this — your agent does … you never touch it by hand" (drop the TypeScript-for-your-harness headline) - Lint blockquote: "Bad states stop compiling … turn it up only when you want it" (drop "Markdown is prose; a typed spec is a program") - the edit-spec bullet says "edits the source", not "the typed spec" "No TypeScript?" reassurance and the optional .spec.ts mention stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: prettier-format HANDOFF Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): update the stale generate-types ref the verb rename missed The CI "Check generated types compile" step ran `node dist/cli.js generate-types` — renamed to `generate types` in the verb consolidation, so the job failed with "Unknown command". self-command-refs didn't catch it because it didn't scan .github/; add .github/ to its scan scope so a stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on. Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to research/hook-modes-and-testing.md (the nudge, not a requirement; the write-hook authoring skill is the real authoring lever, unbuilt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig) The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts and errors "not found by the project service". It's a tsx-loaded build input (like the root CLAUDE.md.spec.ts, which isn't linted either), so ignore it in eslint to match. Latent since the exclude landed; surfaced now that the earlier generate-types step no longer fails first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
fe12c14a30 |
docs: de-duplicate and tighten README intro (#45)
* docs: de-duplicate and tighten README intro Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat!: collapse runtime entrypoints under `vigiles hook-runtime` The CLI had grown ~15 stray top-level commands that a human never types — run-hook-program, agent-hook, skill-hook, skill-tool-hook, guard-hook, action-hook, refs-hook, intercept-tool-hook, agent-start/done, skill-start/done, run-skill, effect-enter/exit. These are RUNTIME entrypoints: the harness invokes them via a block vigiles emits into a hooks config; they are not verbs. Collapse them under one hidden umbrella, `vigiles hook-runtime <kind>`, kept off the help surface (verbs are typed; runtime entrypoints are emitted). Every emitter now emits the new form; `refs` stays a verb. Add the cohesive-cli-surface rule to CLAUDE.md encoding the verb-vs- runtime-entrypoint taxonomy and the three-faces (CLI/GHA/agentic) coherence requirement. BREAKING CHANGE: the top-level runtime commands are removed. Any already-emitted settings block referencing them must be regenerated with `vigiles compile` (which re-emits the `hook-runtime <kind>` form). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat!: fold hook compilation into `vigiles compile` `compile-hook` was a stray sibling of `compile` doing the same mental action — compile a typed authoring artifact into the harness's native format. Fold it in: one verb compiles both .spec.ts (→ markdown) and hook programs (→ harness config + stamp). Hook SOURCE now lives in the agnostic, committed .vigiles/hooks/ (the typed hook imports vigiles/hook and compiles to ANY harness, so it must not live in a harness's own .claude/). Bare `vigiles compile` discovers specs AND .vigiles/hooks/*; explicit `compile <hookfile>` works too. `compile` now WRITES the wiring instead of printing a paste-this block: it MERGES the compiled block into the active harness's native config (.claude/settings.json JSON / .codex/config.toml TOML) idempotently — keyed by the hook path, so recompiling updates in place and never clobbers the user's own hooks. One source dir also makes basenames unique, retiring the stamp basename-collision edge. New src/hook-install.ts holds the pure, tested merge/discovery helpers. BREAKING CHANGE: the `compile-hook` command is removed; use `vigiles compile` (it discovers .vigiles/hooks/ or takes an explicit hook path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: align docs, comments, and spec with the cohesive hook CLI Phase 3 of the CLI cohesion pass — update every reference to the removed `compile-hook` / top-level runtime commands: - docs/compiled-hooks.md: rewrite "Compile and run" → "Compile and wire" (`vigiles compile` discovers .vigiles/hooks/*, MERGES into the native config, writes the stamp) + a new "Where things live" table answering where hook source/stamp/wiring live and that they're not auto-discovered by location alone. - docs/cli.md: drop the compile-hook/run-hook-program entries; document hook compilation folded into `compile` + the hidden `hook-runtime` umbrella. - README, eval-architecture, skills, verifying, unmarked-refs, e2e/bench scripts, research/*: mechanical command rename. - CLAUDE.md(.spec.ts): keyFiles + positioning updated; add hook-install entries; recompiled. - Source JSDoc + the safe-bash-guard example header point at the new commands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: correct 4 stale runtime-command usage strings missed by the manual sweep Three `Usage: vigiles <kind>` error messages (run-skill, skill-start, agent-start) and one JSDoc comment still printed the pre-umbrella command form. They now say `vigiles hook-runtime <kind>`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: dogfood the cross-ref moat on vigiles's own command surface A guidance rule alone didn't stop the rename from leaking stale `vigiles compile-hook` / `run-skill` refs into the docs (it leaked 8, even with the cohesive-cli-surface rule freshly written). Per vigiles's own thesis — deterministic constraints over probabilistic compliance — add the deterministic gate. - src/cli-commands.ts: VERBS + HOOK_RUNTIME_KINDS, the single source of truth (a behavioural test keeps it honest vs the dispatch). - src/self-command-refs.ts: pure detector — every `vigiles <cmd>` in a command context must resolve to a real verb or hook-runtime kind. High-precision: only inline code spans / shell fences / npx|Usage:|cli.js invocations; a bare unknown verb only when hyphenated or explicit. - src/self-command-refs.test.ts: unit tests + the repo DOGFOOD (scans docs/README/CLAUDE.md/src/examples/hooks, not research/ the historical record) that fails CI on a stale ref. Running it caught 3 more stale refs the manual sweep had missed (tool-intercept + refs-hook test comments) — now fixed. This is what 'enforce updating the docs automatically' means: the cross-reference moat, pointed at vigiles itself. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: dogfood the compile-merge against a real plugin's hook config Close the merge half of the compiled-hooks OSS dogfood gap: compile a vigiles gate INTO the REAL superpowers hooks.json (vendored, MIT, SHA-pinned) and assert the merge is non-destructive — the plugin's own SessionStart hook (incl. its async flag) survives, vigiles's PreToolUse gate is added, and recompiling doesn't duplicate. Proves the new settings-merge works on a real-world config, not just synthetic unit fixtures. The gate 'golden before' stays a faithful reconstruction: the canonical disler hook is unlicensed (can't vendor), and an MIT alternative pulls in a jq + /tmp runtime dependency that would make a committed CI test flaky — a bad trade vs. the existing faithful reproduction + the documented real measurement in research/hook-pain-points.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add the doc-consistency rule + fix a tsc break in the merge dogfood Add the standalone doc-consistency guidance rule asked for: when a capability changes, update BOTH public (docs/, README) and internal (research/, CLAUDE.md) docs in the same change. It names the split — the CHECKABLE half is enforced by deterministic CI gates (self-command-refs, orphan-docs, doc-refs, integrity), never trusted to discipline; the JUDGMENT half (tier, prose, cross-links) is the discipline, governed by the sibling docs rules. 'Mechanize the consistency fact where you can' is the principle; the check is the floor, updating both tiers is the ceiling. Also fix a tsc error in the Commit-B merge dogfood: the spawnSync helper annotation picked the Buffer overload (vitest's esbuild run didn't catch it; npm run build does). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add the prefer-existing-solutions rule Codify the reflex this session kept surfacing ('is there an existing solution?'): before building any non-trivial capability, search the landscape and prefer adopt > compose > build. BUILD is justified only when no existing option fits, it dogfoods the core moat, or every option's cost outweighs a small purpose-built piece — and you must name the prior art and why it was rejected (as the self-command-refs decision did). Generalizes the existing dont-reimplement-linters + compose-with-sync-tools stance into the default for all new work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: E2E dogfood the file-gate + react hook roles; document coverage Close the two remaining compiled-hook dogfood gaps with real CLI-runtime E2E tests (mirroring the gate/inject E2E): - file-gate (defineFileGate + PathView.under): denies a Write under a protected path, allows elsewhere. - react (defineReact): a notice reaches stderr + can't block (exit 0); run() executes its effect-classified command. Document the full per-capability dogfood coverage matrix in research/hook-pain-points.md (the internal record), incl. the two deliberate non-coverages with reasons: the gate 'golden before' stays a faithful reconstruction (the canonical hook is unlicensed; the MIT alternative drags a jq+/tmp runtime dep), and inject/react have no deterministic 'prove worth' oracle so they're proven structurally, not scored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add the test-both-harnesses rule + annotate the hook E2E harness scope A test of a harness-FACING capability must cover BOTH Claude Code AND Codex, never just CC by default — the test-layer companion to adapter-aware-lint-rules. Three sub-rules: test what maps on both (compile --harness=codex beside the CC default; agnostic detectors over a non-CC layout); gate/skip LOUDLY where a harness can't run (no binary on PATH) or a capability is deferred (Codex inject/react output); don't double-test agnostic core. A genuinely harness-neutral path runs once but must SAY WHY, so 'CC-only' is never ambiguous. Dogfood it: annotate the new file-gate + react E2E with their harness scope (deny→exit 2 and run()→spawn are neutral, one run covers both; Codex emit is tested via --harness=codex; Codex notice/react output is deferred). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: structural both-harness enforcement via a registry contract suite Make the 'test both harnesses' gap structural, not disciplinary (hexagonal contract-tests-over-the-registry pattern): - src/adapter-contract.test.ts: runs the conformance kit over the WHOLE registry in a loop, so registering an adapter auto-subjects it to every contract (can't forget a new harness); a lacked capability is a VISIBLE it.skip(n/a) gated on shellHooks/harnessTesting (no silent CC-only pass); and a meta-test fails the build when a src/adapters/<dir> exists but isn't registered (or a declared prototype like opencode). - Document the KNOWN GAPS in CLAUDE.md (test-both-harnesses rule): the contract catches ADAPTER-level gaps but can't auto-catch a capability assertion written CC-only OUTSIDE the contract (the judgment the rule governs), nor force OSS/real-binary coverage (gated loudly). Floor vs ceiling, stated not hidden. - docs/authoring-an-adapter.md: 'how the registry tests your adapter automatically' for new-adapter authors. - docs/testing-matrix.md: fix 5 stale test paths (adapter reorg) + the agent-hook->hook-runtime agent rename the prefix-scoped drift check couldn't see. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: correct 5 genuine stale path refs; document why bare-name drift can't be auto-checked Investigating whether to widen self-command-refs to catch bare command names (the agent-hook blind spot) — the data says no: bare names double as CONCEPT names ('the refs-hook nudge' in ~10 files), test labels, and makeTmpDir() names; bare file paths collide with illustrative examples (README's src/auth/login.ts) and test fixtures. A denylist over either would cry wolf on dozens of valid usages — so it stays the JUDGMENT half the doc-consistency rule owns, not a check (measured, documented in the detector header + the rule). But the probe surfaced 5 REAL stale refs (fixed by hand): - src/skill-runtime.ts → src/adapters/claude-code/skill-runtime.ts (docs/skills.md, agent-runtime.ts, inline.ts) — moved in the reorg. - src/adapters/claude-code/mock-model.ts → src/mock-model.ts (codex/mock-model.ts, harness-driver.ts) — startMock lives at the root. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): hook modes & testing landscape analysis Step-back analysis of the hook surface (check + run) vs the OSS/AI landscape (OPA/Rego, Guardrails AI on-fail actions, NeMo Colang, Cedar, lefthook, Claude Code's new prompt/agent hook types). Verdicts (corrected from the first pass): - DON'T restrict the runtime to TS-only: two opt-in lanes (typed TS with NO embedded shell + audited/sandboxable hand-written .sh), the OPA-coexists pattern; authoring-safety=capability vs runtime-safety=confinement are separate concerns. - Modes collapse to ONE essential new mode "observe"/shadow (evaluate + record, never block - the gradual-rollout primitive) on the existing "enforce" default, NOT a 4-mode vocabulary (the warn/shadow split was a rendering detail, not a mode). - Testing is tiered already; the gap is COVERAGE (OPA-style) + behavioral gate-precision + the model-gated prompt/agent hook types. Ranked deliverables: observe mode (#1), hook-test coverage, react effect ceiling, prompt/agent-hook verify+judge. Referenced from CLAUDE.md keyFiles (orphan-docs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): 10-OSS dogfood — does the typed hook lane cover all use cases? Mapped real hooks from 10 OSS sources (disler/claude-code-hooks-mastery full event set, alexknowshtml safety hooks, vendored superpowers + oh-my-claudecode, CC-docs canonical, ruff/ty validators, TTS/notify, gmickel/flow-next, guide patterns) to the vigiles/hook vocabulary. Verdict: NO, not all — and that's the two-lane decision working. The typed lane nails the SAFETY-GATE slice (PreToolUse block: dangerous bash, secret reads, curl|sh, protected-path edits) + simple react/inject. Gaps cluster by root cause: - PURITY (no exec / thin event shapes): dynamic-context inject, TTS, backups, structured logging, tool_response reactions -> these NEED I/O, so they belong in the shell lane; closing them in typed would break the capability guarantee. - MISSING roles/shapes (closeable, worth it): gate-capable UserPromptSubmit (block/rewrite a prompt) + Stop (gate-until-tests), and richer event shapes (tool_response, prompt text, non-Bash tool_input) so more DECISIONS are expressible. Refined scope: typed lane = DECISIONS over any event; shell lane = I/O & lifecycle side-effects. Re-ranks deliverables: event/shape coverage for decisions rises above the observe mode for breadth. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the type-sourcing + licensing decision Capture the conclusion from the black-box / Google-v-Oracle research: CC's only official machine-readable artifact is the settings.json schema; events/tools/payloads are prose-only, so the dialect catalog is hand-maintained (and the detectors are conservative because of it). The installed @anthropic-ai/claude-code (sdk-tools.d.ts + cli.js) is a read-local freshness/drift source, NOT a dependency. Licensing (informational): all-rights-reserved -> don't vendor or let api-extractor INLINE their .d.ts into our shipped artifact (that's redistribution; import alone isn't, but dts-bundling inlines by default). Hand-write our own types matching the FACTS (names/fields aren't copyrightable; DefinitelyTyped norm) + read the local install to warn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: dialect drift-check — read-local freshness alarm for the hand-maintained CC catalog CC is a black box, so claudeCodeDialect is hand-maintained. Add the deterministic backstop: read the user's INSTALLED @anthropic-ai/claude-code (sdk-tools.d.ts tool-input types + cli.js event strings) and FAIL LOUD when its surface drifts from our catalog — so a CC update that adds/removes a tool or renames an event turns CI red instead of silently staling us. - src/dialect-drift.ts: pure parsers (parseToolInputTypes, eventsMissingFromBundle) + findClaudeCodePackage (npm-root-g / claude binary). READS LOCAL ONLY — ToS-clean, never vendors/ships their types (all-rights-reserved). ACKNOWLEDGED_TOOL_INPUT_TYPES is our own list of bare identifiers (facts), diffed against the install. - src/dialect-drift.test.ts: pure units + GATED read-local checks (validated vs claude-code 2.1.42: 19 tool types, 9 events); skips loud when CC absent. Also fix 11 pre-existing unsafe-any lint errors the merge dogfood snuck into hook.test.ts (typed the settings.json JSON.parse) — they'd have failed CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(scan): use the dialect drift-check at runtime + update internal docs The drift module was added but only consumed by its own CI test. Wire it into the product so it's actually USED: - src/dialect-drift.ts: add checkDialectDrift() (best-effort, read-local; reads only the small sdk-tools.d.ts — no cli.js scan on the runtime path; never throws) + formatDialectDrift() (one-line ⚠ ONLY on real tool-surface drift — a bare version bump emits nothing, no noise) + VALIDATED_CC_VERSION (2.1.42). - src/cli.ts: `vigiles scan` (claude-code only) now prints the freshness warning when the installed CC's tool surface drifts from our catalog. Deliberately NOT in compile (hot recompile-on-save path). - 2 formatter unit tests (null on no-drift / version-only; message on added+removed). Docs updated in detail (doc-consistency): research/code-adapter-architecture.md (the sourcing/licensing section now records the SHIPPED module + its two consumers + the runtime-warn behaviour) and the CLAUDE.md keyFiles entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: note the scan dialect-freshness warning in cli.md Public scan reference now documents the read-local Claude Code freshness check (best-effort ⚠ on tool-surface drift; reads only the user's own install; silent on a no-op version bump). Completes the doc-consistency for the drift-check across internal + public tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): observe mode + gate-capable UserPromptSubmit/Stop + tool response Two compiled-hook expansions, both shipping for Claude Code AND Codex. 1. observe mode — every gate takes mode: 'enforce' | 'observe'. observe is the shadow/rollout primitive: compute the same Decision, record what it WOULD block to .vigiles/hook-observations.jsonl, exit 0, never block. Harness- neutral by construction (exit 0 + a local record). gateAction(decision, mode) is the pure decision->action mapping the runtime and tests share; hookMode reads a gate's mode. 2. gate-capable non-tool events + richer event shapes: - definePromptGate (UserPromptSubmit) sees the prompt text (e.prompt) and may deny to block/erase it — a security filter. - defineStopGate (Stop/SubagentStop) may deny to keep the agent going (gate-until-tests-pass), honouring e.stopHookActive as the loop guard. - react now sees the tool RESPONSE via e.response (ResponseView: isError()/contains()). Both new gates ride the shared exit-2 gate runtime, so they work on every harness whose gate vetoes via exit 2 (compile-emit covered for CC and Codex). The generic tool_input accessors (WebFetch/Task/MCP) stay deferred to avoid a stringly-typed input bag. Tests: pure (gateAction, prompt/stop decode, responseView, CC+Codex compile) in hook-program.test.ts; E2E over the real runtime (prompt-gate deny, stop-gate loop guard, observe records-not-blocks) in hook.test.ts. Public surface + api report + docs/compiled-hooks.md + the research record updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): cohesive pillar names — Verify→Lint, Measure→Eval Align the four README pillar labels with the CLI verbs (and CLAUDE.md's own "Lint"/"Test" layer names) instead of marketing renames that drifted from them: Verify→Lint, Measure→Eval (Guard/Test unchanged). Keeps the differentiation in the subtitles — Lint stays "is this TRUE, not just well-formed", Eval stays "the eval you can afford" — so the wedge survives the plainer, cohesive labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): design for I/O-dependent hook decisions via declared context providers Captures the design (no code yet) for letting a compiled hook decide on external state — git branch, project-specific facts — without breaking capability = API surface. Grounded in prior art (Cedar entities/context, OPA input/data vs the "ugly" http.send, Gatekeeper's registered-Provider model that deliberately rejected http.send): the policy never fetches; the trusted host gathers facts and hands them in. Core reframe: the moat is "no UNDECLARED capability", not "zero I/O". decide() stays pure; the runtime gathers a DECLARED set of read-only facts into e.ctx (typed needs[], undeclared access = a tsc error). The opt-out ladder (graceful degradation, never a dead end): Tier 0 built-in matchers -> Tier 1 built-in providers -> Tier 2 user-declared providers (the long tail, effect-classified, dangerously opt-out) -> Tier 3 shell lane (Turing- complete). Proves total coverage by mapping every real-world hook from the 10-OSS dogfood to a tier. Cross-linked from hook-modes-and-testing.md; keyFiles + CLAUDE.md updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): context providers v1 — gates can decide on git state via needs/e.ctx Lets a compiled gate decide on EXTERNAL state (git branch, dirty tree, cwd) without breaking capability = API surface. The pattern (Cedar/OPA/Gatekeeper, research/hook-context-providers.md): the pure decide() never fetches; the trusted runtime gathers a DECLARED set of read-only facts and hands them in as e.ctx. - A gate declares `needs: ["git.branch"]` and reads `e.ctx["git.branch"]`. The gate builders are generic over the declared needs, so reading an undeclared fact is a tsc error, and an unknown provider name won't compile. - New core/hook-providers.ts: the closed built-in registry (git.branch, git.isDirty, cwd — each total, defaults on failure), ProviderName/ ProviderResults/HookCtx<N>, and gatherContext(needs, io) parameterized over an injected exec (testable, core depends on no child_process). A soundness test asserts every built-in command is read-only via bash-effects. - The runtime injects the real execSync (cli.ts gatherHookContext); the hook itself still does zero I/O. Long tail stays covered by the opt-out ladder (v2 user-declared providers, then the shell lane) — designed in research/hook-context-providers.md. Tests: pure (hook-providers.test.ts, hook-program.test.ts) + E2E in a real git repo (hook.test.ts: deny push on main, allow on a branch). Public surface + api report + docs/compiled-hooks.md + CLAUDE.md updated. Additive — hooks without `needs` are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): 20+ OSS provider survey + the lightweight opt-out design Answers the "what built-ins do we need + how should the quick opt-out look" questions, grounded in research. - Survey of ~21 real hooks (aiorg.dev 20+ catalog, CC docs, the vendored superpowers/omcc/wshobson slices, disler/dcg): MOST facts a hook reads are EVENT data (path/command/tool/stop-flag/session-type), not providers — so the built-in set stays tiny. Verdict: git.branch/git.isDirty/cwd (shipped) + os.platform; everything else (env, test/lint status, rate-limit, online) → opt-out, not a growing catalog. - The lightweight opt-out, judged against keeping decide() pure: an INLINE declared command in needs — provide(name, cmd) (read-only, compile-rejected if not) / dangerously(name, cmd) (acknowledged escape, greppable). Rejected: a method-in-decide (breaks purity), @ts-expect-error (grants no capability — category error), and a whole-hook I/O mode (that's just the shell lane). Naming follows the loud-escape-hatch best practice (dangerouslySetInnerHTML / unsafe / http.send / our own purity:'dangerously-unrestricted'). - Event naming: native + dialect-validated, NOT hardcoded to CC (PreToolUse is in both dialects); a generic alias layer is deferred until a divergent harness lands (rule-of-three; the dialect is the seam). Fixed a stale "raw CC PreToolUse event" comment. Build order updated: v1 built-ins (shipped) -> os.platform -> inline provide/dangerously -> v2 defineProvider. CLAUDE.md keyFiles updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the one-source-many-harnesses event-naming decision A single hook source already targets CC + Codex today (author once in .vigiles/hooks/, compile emits each native config; the typed program, context gather, and gate runtime are harness-neutral). Native event names work for both because the dialects share them; the inject/react OUTPUT shape is the one CC-confirmed-only caveat (compile --harness=codex warns). Decision: native + dialect-validated names, NO generic alias layer yet — it would only pay off for a future harness with divergent event names (none today), so it's deferred to a small per-dialect map at the dialect seam, additive when needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(claude-md): add harness-parity-and-extensibility design rule A design-time umbrella over the existing enforcement rules: when designing ANY capability (compile emit, runtime, ports, subsystems — not just lint/test), (1) fully support BOTH shipping harnesses (Claude Code AND Codex) as equals, never CC-first-Codex-bolted-on, with loud documented deferrals where a capability can't map yet; (2) leave room for future adapters by putting every harness-specific fact behind a port and threading the resolved adapter (core ⊄ adapter), so adapter #3 is a new object not a core edit; (3) design the neutral shape first but defer the abstraction until a second divergent implementation earns it (the seam exists, the mapping is added when needed). Points to its enforcement arms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): os.platform built-in + inline provide()/dangerously() context opt-out Closes the provider/long-tail story (research/hook-context-providers.md), grounded in a 20+ OSS survey: most facts a hook reads are already event data, so the built-in set stays small and the long tail goes to an opt-out, not a growing catalog. - os.platform built-in (ambient process.platform) — the one extra fact the survey justified (per-OS hooks). Built-ins now: git.branch/git.isDirty/cwd/os.platform. - The lightweight opt-out: an INLINE command in `needs` — provide(name, cmd) (read-only; compile rejects a non-read-only command via unsafeInlineProviders) or dangerously(name, cmd) (the loud, greppable escape, the dangerouslySetInnerHTML/unsafe convention). The runtime runs it and hands stdout in as e.ctx[name]; decide stays pure. Chosen over a method-in-decide (breaks purity), @ts-expect-error (grants no capability), and a whole-hook I/O mode (that's the shell lane). NeedSpec = ProviderName | InlineProvider; HookCtx is a key-remap mapped type over it (built-in name → typed value, inline → string), undeclared access still a tsc error. Decode fns are generic over N; AnyHook uses an erased any-needs because a gate's decide is contravariant in N (documented). Tests: pure (hook-providers.test.ts: os.platform, inline gather, unsafeInline, unknownProviders; hook-program.test.ts: inline gate + provide-not-read-only rejection) + E2E (hook.test.ts: an inline provide() fact gathered by the real runtime drives a deny). Public surface + api report + docs/compiled-hooks.md + CLAUDE.md updated. Additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): v2 registered providers + aggressive OSS comparison dogfood v2 — REGISTERED context providers (the reusable, named tier of the opt-out ladder). Author a fact once in .vigiles/providers/<name> via defineProvider({ name, run }) and reference it from any hook by provider("name") → e.ctx[name]. The runtime discovers + loads the registry (loadProviderRegistry) and resolves refs in gatherContext; `vigiles compile` validates each provider is read-only (unless dangerous:true, via unsafeProvider) and that every provider() ref resolves to a registered file (a dangling ref fails compile). NeedSpec is now ProviderName | InlineProvider | RegisteredRef; decode fns are generic over N and AnyHook uses an erased any-needs (a gate's decide is contravariant in N). Surface: defineProvider/provider + types on vigiles/hook. Aggressive OSS comparison — src/hook-oss-comparison.test.ts isolates the NON-CIRCULAR structural wins over hand-written guards, one failure mode per test (evasion: the compound `git push -f` a substring/glob misses; precision: a grep false-positive on a benign echo; protocol: an exit-1 false-confidence guard) plus the breadth headline (2/7 blocklist vs 7/7 compiled). research/hook-oss-comparison.md documents the head-to-head, honest about both the labelled-breadth caveat and what compiled hooks do NOT do better (stateful / broad-I/O / delivery → shell lane). Tests: pure + E2E (a .vigiles/providers/ file resolved by the real runtime) + the comparison battery. Public surface + api report + docs/compiled-hooks.md + CLAUDE.md updated. Additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cli-install): loud-skip the codex e2e when `skills add` can't reach the network The e2e's guard probed `npx skills --help` (reachable via npm) but the real `skills add` does a GitHub fetch — so in a partial-network sandbox it slipped past the skip-guard and failed RED, contrary to its own documented contract ("self-skip loudly where the network isn't reachable"). Now a network-class failure of the real `add` (read from stderr) is re-classified as a loud ctx.skip(); a genuine install bug still throws, and CI (GitHub reachable) still runs the full assertions. Pre-existing flakiness, unrelated to the hook work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(hook): add git.root + env.isCI built-in providers (env.isCI adopts ci-info) The two extra cheap-ambient facts the 20-OSS survey justifies, closing the built-in set at git.branch/git.isDirty/git.root/cwd/os.platform/env.isCI (small by design; the long tail stays in the opt-out tiers). - git.root — read-only `git rev-parse --show-toplevel` (the soundness test confirms it classifies read-only); pairs with cwd/path decisions. - env.isCI — ADOPTS the ci-info library (de-facto standard, ~30+ CI vendors) rather than hand-rolling an env check; injected via ProviderIO.isCI so core stays dep-free (the dep lives at the CLI composition root). The prefer-existing-solutions split: git facts stay read-only shell (a JS git lib would bypass the bash-effects soundness check + add a dep for a one-liner), platform is process.platform, and the provider-registry architecture has no embeddable lib (the OPA/Cedar/Gatekeeper pattern, built here). Tests, api report, docs/compiled-hooks.md, research/hook-context-providers.md, CLAUDE.md + package-lock all updated. Additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: cover registered providers in cli.md + the compiled-hooks "where things live" Close the two doc gaps the providers work left: docs/cli.md's `compile` section now documents context providers (built-ins + inline provide/dangerously + registered .vigiles/providers/ discovery/validation), and the compiled-hooks "Where things live" table gains the .vigiles/providers/ source row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(docs): publish the generated API reference to GitHub Pages + link it Adds .github/workflows/api-docs.yml: on push to main, generate the API Documenter markdown (api-extractor doc model → api-documenter), make it Jekyll-renderable (inject front matter + a minimal _config with jekyll-relative-links so the generated ./foo.md links resolve to .html), and deploy to GitHub Pages via the official configure/build/upload/deploy-pages flow. api-reference/ stays gitignored (generate-not-commit, ~1900 pages); the committed surface artifact remains etc/*.api.md (already CI-gated). Linked the site (https://zernie.github.io/vigiles/) from the README "More" row, the docs index Reference section, and the compiled-hooks + harness-testing "See also" (the hook/testing public surfaces). NOTE: requires a one-time repo setting — Settings → Pages → Source: "GitHub Actions" — and the links go live after the first main deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(docs): switch the API-reference site from api-documenter to TypeDoc api-documenter pages EVERY member (one file per interface field) → ~1900 flat markdown files, the wrong shape for a TS library, and needed a Jekyll/front-matter hack to render. Switch to TypeDoc (the de-facto TS API-docs tool): - typedoc.json: curated to the public AUTHORING entry points (spec/linting/ testing/hook/adapter/claude-code/codex — not the unit/integration/e2e re-export barrels or the tiny vitest/jest matchers), public-only (excludeInternal), members rendered INLINE. - ~465 navigable pages with a sidebar + search (vs 1900 flat files), and `.nojekyll` (githubPages:true) → Pages serves the HTML as-is, so the api-docs.yml deploy drops the Jekyll/front-matter steps entirely. - ci.yml smoke step + the docs:api script now run typedoc; @microsoft/api- documenter removed. api-extractor (etc/*.api.md surface gate) is unchanged. README + docs links unchanged (same https://zernie.github.io/vigiles/ URL). Still requires the one-time repo setting: Settings → Pages → Source: GitHub Actions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(api)!: curate testing/linting barrels — drop internal seams from the public surface Replace the `export *` re-exports in `vigiles/testing` and `vigiles/linting` with curated named re-exports, so the internal DI seams (`*With` runners), low-level parsers (`parse*`, `decideHook`), pool/aggregate/model-tier helpers, and the linter cross-reference ENGINE no longer leak into the public .d.ts surface, the api-extractor reports, or the TypeDoc site. The CLI imports those from the source modules directly; they were never meant to be public. Also demote a few doc-comment `{@link}` refs (`ephemeralRunEnv`, `spawnAgent`, `runEvalWith`) that pointed at now-internal symbols to plain code spans, so the generated API site has no dangling links. Net effect: api report drops ~311 lines across the two surfaces; the TypeDoc site drops 465 → 407 pages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(lint): ban internal imports of the public barrel entry points Add a no-barrel-imports ESLint rule so internal modules import the LEAF that defines a symbol, never the package's own public barrel surfaces (vigiles/{linting,testing,unit,integration,e2e,hook,claude-code,codex}). Barrel imports pull the whole re-export graph (slow in the test runner / any non-treeshaking consumer, a circular-import risk) and re-leak the internal seams the curated barrels deliberately drop — so this locks in the preceding barrel curation. Implemented with the built-in `no-restricted-imports` rule rather than eslint-plugin-barrel-files: that plugin's `avoid-importing-barrel-files` calls the ESLint-9-removed `context.getFilename()` and crashes on ESLint 10, so per prefer-existing-solutions a working core rule beats a broken dependency. The barrels themselves (the e2e→integration→unit tier ladder) and test files are exempt; the codebase is already clean, so this is pure regression-prevention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(dialect): refresh Claude Code dialect baseline to 2.1.187 + survive the native-binary distribution The gated dialect-drift alarm fired in CI because the unpinned `npm i -g @anthropic-ai/claude-code` now installs a much newer CC than the 2.1.42 baseline: - sdk-tools.d.ts grew from 19 to 38 tool-input types — the agent-PLATFORM surface (Cron*, Task* CRUD, Workflow, Monitor, Enter/ExitWorktree, EnterPlanMode, Projects, PushNotification, REPL, ReadMcpResourceDir, RemoteTrigger, ScheduleWakeup, ShowOnboardingRolePicker, Artifact) was added; Config removed. These are HOST/platform tools, not subagent-grantable, so claudeCodeDialect's builtinAgentTools (the `tools:` frontmatter catalog) is intentionally unchanged — they're acknowledged as facts only. - CC ≥ ~2.1.18x switched to a NATIVE-BINARY distribution (bin/claude.exe from a platform optionalDependencies package) with NO readable cli.js — so the hook-event text-scan ENOENT-crashed. Added findClaudeCodeBundle(): the event check now degrades to a LOUD SKIP when there's no readable JS bundle, while the sdk-tools.d.ts tool-type alarm keeps working. Refreshed ACKNOWLEDGED_TOOL_INPUT_TYPES (38) + VALIDATED_CC_VERSION = 2.1.187. We deliberately do NOT `import type` the SDK's ToolInputSchemas union: both @anthropic-ai/claude-code and @anthropic-ai/claude-agent-sdk are "© Anthropic PBC. All rights reserved." (proprietary), and vigiles is MIT + multi-harness, so we keep reading the user's local install (ToS-clean) against a hand-authored list of bare identifiers. Recorded in the dialect-drift header + research/code-adapter-architecture.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(harness): repair stale refs-hook command in the refs-nudge example + teach self-command-refs the ${CLI} form The refs-nudge harness dogfood called `node ${CLI} refs-hook`, but the runtime entrypoint was consolidated to `hook-runtime refs` in the hook-runtime rename — so the PostToolUse hook hit "Unknown command", emitted no additionalContext, and the nudge never reached the model (the CI `harness` job's only failure). Fixed the example to `node ${CLI} hook-runtime refs`; verified the nudge lands in a real claude session. The stale ref slipped past the self-command-refs gate because the detector only recognized `vigiles`/`cli.js` invocation literals — not the harness-test convention `node ${CLI} <cmd>` (`${CLI}` = a `dist/cli.js` URL). Extended INVOKE to recognize `${CLI}` (handling that `$` is not a word char, so `\b` can't anchor it) so this whole class is now caught deterministically — dogfooding the cross-ref moat on vigiles itself. The repo dogfood + 46 existing `${CLI}` usages stay green (no false positives); added a unit test for the class. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: pin claude-code to VALIDATED_CC_VERSION so the dialect-drift alarm is deterministic CI installed `@anthropic-ai/claude-code` UNPINNED, so any new CC release fired the (intentional) dialect-drift alarm on unrelated PRs — the noise that just broke this branch's CI. Pin every job that drives the real binary (the test, harness, and e2e tiers) to `@anthropic-ai/claude-code@<VALIDATED_CC_VERSION>`, grepped from src/dialect-drift.ts so the constant is the SINGLE knob: bumping it updates both what CI installs and the alarm baseline (the gated test cross-checks them). Two wins: the alarm now fires only on a DELIBERATE bump (not a random upstream release), and the real-`claude` harness/eval tiers become reproducible across CC releases (no surprise behavior breaks like the refs-nudge PostToolUse change). Runtime mismatch warning already exists and needs nothing: `vigiles scan` calls checkDialectDrift/formatDialectDrift, which prints a one-line ⚠ ONLY on real tool-surface drift (a bare version difference with no new/removed tool stays quiet — the right precision), best-effort and never throwing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the API-curation + CC-2.1.187 + CI-pin session Overwrite the stale compiled-hooks-arc handoff with this session's state on branch claude/readme-duplication-cleanup-l1wc5n (PR #45): barrel curation, the no-barrel-imports rule, the CC 2.1.187 dialect refresh (native-binary distribution), the refs-nudge + self-command-refs ${CLI} fix, and the CI pin — plus the decisions of record (don't import the proprietary SDK types; bump VALIDATED_CC_VERSION + ACKNOWLEDGED together). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix HANDOFF instructions — say to commit + push (ephemeral, git-tracked) The header said "overwrite each session" but never that the file is git-tracked and the container is ephemeral, so an update persists ONLY if committed + pushed (a local overwrite is lost when the session ends). Spell that out, plus when to refresh and what to include. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
dc293e52e1 |
chore(bench): add token-efficient to A1 manifest + per-task spread reporting (#44)
* chore(bench): add token-efficient to A1 manifest + per-task spread reporting
Expand the ecosystem-benchmark manifest with a second cleanly-A/B-able
compression entry, drona23/claude-token-efficient (MIT, 5.7k stars,
pinned @0d30a6d) — a single injectable CLAUDE.md whose viral "63%" headline
is a word-count over 4 prompts, while the repo's own token benchmark admits
only ~4-12% output-token reduction (a self-documented claim>>measured gap).
Vendor the real file + its MIT LICENSE; record full provenance and the
follow-on (needs-binary: RTK/CodeGraph/Claw/...) and license-blocked
(Context Mode) cluster classification in SOURCES.md/FINDINGS.md.
Also surface per-task variance in the leaderboard: the mean hid that a skill
can help on one task and hurt on another (the caveman review-doc finding), so
report the output-cut range + the helped/hurt split + a MIXED-direction flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF for A1 manifest-expansion session
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): record A1 haiku pilot — token-efficient debunked, caveman variance flagged
First multi-skill A1 run (haiku, caveman+token-efficient × 2 tasks × 2 trials, $0.29):
- token-efficient: claim 63% vs measured -2% output (output GREW, bill +3%) — DEBUNKED,
worse than the repo's own honest ~4% haiku admission (CLAUDE.md input overhead is
net-negative, as its README warns).
- caveman: +30% output cut here vs -18% on the prior same-config run — a 48-point swing
flagged as a variance/noise signal (raise trials >=5; only the structural <1%
output-share fact is robust).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — A1 pilot ran, token-efficient debunked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(agent-runtime): make the active-agent tracker nesting-safe (depth-aware stack)
The PreToolUse tool-contract rail tracked the active subagent in a single
`.vigiles/active-agent.json` slot. Under CC v2.1.172 depth-5 nested subagents
this allowed a CONTRACT ESCAPE: when an inner subagent returned, SubagentStop
cleared the whole slot while still inside the outer subagent, so the gate then
saw "no active agent" and ALLOWED a tool the outer subagent's contract forbids.
Fix (TLC-certified in AgentWindowStack.tla): replace the slot with a depth-aware
STACK — pushActiveAgent on dispatch, popActiveAgent on SubagentStop (back to the
parent frame, not a full clear), gate on the stack top. The open signal now
recognizes both the `Task` and nested-spawn `Agent` tools. Legacy single-slot
`{ agent }` markers read back-compat as a one-frame stack.
The TLC counterexample Open(writer);Open(writer);Stop;Call(Bash) is now a
regression test (pure + a runHook e2e proving SubagentStop pops to the parent).
Still EXPERIMENTAL / not auto-wired; the effect() sub-region it served stays
dropped — the stack ships for when nested contract enforcement is wanted alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — V1 nesting bug fixed
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record the per-file mixed spec/hand-written adoption model
Capture how vigiles behaves in a repo where some skills/agents/CLAUDE.mds have a
.spec.ts and some don't: per-file handling (compile + integrity + ref verification
for spec'd files; surface lint rules still apply to hand-written ones), the
no-spec-mandate defaults (require-spec off, require-skill-spec deprecated), and
scan's spec-agnostic spec-managed-vs-hand-written labeling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add the spec-value model (what a typed spec buys vs markdown)
Consolidate the recurring "spec vs plain markdown" question into one canonical
internal doc (research/spec-value-model.md): the capability axis (lint/test/eval
run on markdown; only whole-harness type-checking is spec-exclusive), the
two-oracle model (behavioral = spec-agnostic vs deterministic contract/outcome =
spec-enabled), honest leg-grading (result-shape/flow/purity held; effect()
sub-region dropped), the surface ranking, and the settled require-*-spec split +
capability-triggered defaults. Slim the duplicative per-file note in
adoption-strategy.md to a pointer; cross-link from typed-spec-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: cite "Agents Need Control Flow" as third-party validation (+ commoditization caveat)
Record Brian Suh's bsuh.bearblog.dev/agents-need-control-flow/ in the landscape
doc: an independent statement of vigiles's Agent=Model+Harness /
deterministic-constraints thesis, with its three failure modes (babysitting /
post-run audits / accepting on faith) mapping onto our verify+test layers. Framed
honestly as double-edged — external validation of the frame, but confirms the
commoditization signal from the same-day competitive check (the control-flow
thesis is now a named, spreading idea; the edge is substrate + measurement, not
the idea). Cite for positioning, do not treat as a moat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record the competitive reality-check in typed-spec-moat (don't re-inflate)
Same-day competitive sweep found the headline typed-spec "moats" are mostly
claimed/commoditizing: riftmap = PR-time blast-radius diff (moat #2 mechanism for
infra), AgentAuditKit/PolicyLayer = capability-diff + drift for MCP servers, Mastra
= typed workflow handoffs (#5), promptfoo = deterministic structured-output oracle
(#7); the control-flow thesis is now a named idea (bsuh). Add a top caveat + a
"Competitive reality check" section recording per-moat verdicts, the one narrow
unoccupied sliver (first-party harness effect-diff from a typed spec), and the
synthesis: the differentiation is substrate + sub-affordable measurement, not the
capabilities — so don't re-inflate the moat next session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — commoditization finding + consolidate shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: correct the competitive analysis for MARKET (it conflated 3 markets)
The "typed-spec capabilities are commoditizing" read was a market-conflation:
Mastra/promptfoo = app-building (zero CLAUDE.md contact), riftmap = infra,
AgentAuditKit/PolicyLayer = third-party MCP-server security — none are in vigiles's
market. Grounded by reading docs: the ACTUAL in-market competitors (claudelint 114
rules, cclint ×2, claude plugin validate) are pure static surface linters that do NO
cross-referencing, typed specs, test/eval, or capability-diff. So in-market vigiles
uniquely owns those layers; it is NOT differentiated on surface linting. Lead with
cross-referencing (unique in-market, works on plain markdown = no adoption barrier).
Add the market-segmented matrix to landscape-mid-2026.md; correct typed-spec-moat
synthesis + the HANDOFF block. Reviews/adoption pass on claudelint/cclint in flight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add adoption pass — field is wide open, agnix is the most-built rival
Reviews/adoption pass on the real in-market competitors: NO incumbent, near-zero
mindshare. Biggest is agnix (296★, ~414 rules, Rust, 7-harness + LSP) — crowds
vigiles's multi-harness/LSP positioning but does NOT do cross-ref/typed/eval; the
rest are 6–41★ hobby projects; none in awesome-claude-code (47k★); npm downloads
are CI-inflated (agnix Show HN = 1 point). Only entrenched thing is Anthropic's
shallow `claude plugin validate`. Add agnix/AgentLint/SkillCheck to the Market C
matrix + adoption signals; rewrite the read: don't out-rule-count agnix or match its
LSP — lead with cross-referencing ("valid≠true") + the test/eval tier; the real
battle is distribution, not capability (reinforces A1). Update HANDOFF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: capture poach-list, agnix lesson, badge/viral/acquisition GTM, Zod result()
Following the market-segmentation correction: record the useful-but-not-competitor
ideas + adoption plays.
- landscape: the agnix signal (most-built linter, 1 HN pt → linting-without-eval
doesn't pull a community = evidence for the measurement wedge) + a poach list from
the app-building non-competitors (Zod-schema'd contracts, promptfoo assertion vocab,
riftmap distribution model) + business-on-top validated by closed SaaS rivals.
- roadmap §Explore + distribution-strategy: tiered README badge (lint→test→eval funnel
as a growth loop), viral debunk articles (measurement-as-marketing), open-core +
build-on-top + AI-lab-acquisition posture.
- measurement-authority: competitive confirmation of the pivot (agnix → eval is the
puller; no incumbent → distribution is the bottleneck).
- feature-ideas §14b: Zod-schema'd result() (one schema → type + validator + JSON
Schema), poached from Mastra/Pydantic AI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: fix rule-count error (20 not 31) + 11★ honesty; dedup/crosslink the cluster
- Correct the Market-C matrix: vigiles ships ~20 semantic rules (not 31 — that was
the dogfood CLAUDE.md's section count), with an apples-to-oranges caveat (agnix's
414 are micro-checks; vigiles's rules cross-reference) and the real 11★ baseline.
- Reconcile typed-spec-moat's internal contradiction: the top caveat + §7 title said
"commoditizing", overturned by the later market correction — both now say
"capabilities live in adjacent markets; vigiles's own market is wide open".
- Record the PUBLIC plugin leaderboard idea (A1 + leaderboard.ts published as a
claim-vs-measured ranking, head-to-head per category) in roadmap §Explore.
- Cross-link the two matrix hubs (spec-value-model "what a spec buys" ↔ landscape
"who competes").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): record A1 sonnet pass (5 trials) — both compression skills debunked
Sonnet, caveman + token-efficient × {slugify,debounce,review-doc} × 5 trials ($3.43):
- caveman: claim 75% vs measured -18% output (range -55..+8, MIXED), bill +8% — the
-18% is now STABLE (matches the prior sonnet 3x3; haiku +30% was noise). review-doc
(its supposed best case) is the WORST at -55% (output grew 55%).
- token-efficient: claim 63% vs measured -10% output, bill +3% — WORSE than the repo's
own ~12% sonnet admission; review-doc -35%.
- Both: 0 regressions, output ~0.7% of session tokens (the model-agnostic structural
kill). Neither compresses on multi-turn coding; both cost more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — A1 sonnet pass done, numbers publishable
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): audit caveman/token-efficient OWN benchmarks — why they report a win
Read their benchmark code (caveman/benchmarks/run.py, drona23/.../benchmark/run.py +
SUMMARY.md). Record the methodology gap that explains the headline numbers:
- caveman run.py: single-shot one-turn Q&A (no tools/files/multi-turn), output-tokens-
ONLY (ignores the SKILL.md system-prompt input + cache that dominates real sessions),
vanilla "You are a helpful assistant." headline baseline (the concise-baseline harness
lives separately in evals/), and no correctness gate.
- token-efficient: the 63% is a WORD count; the repo's own SUMMARY.md already admits the
token effect is -2%..-11% and "not agentic/coding loops" — vanity headline vs honest
fine-print, and even the fine-print doesn't reproduce on coding.
Also corrected the prior over-stated "baseline discrepancy" point to the fair version
(headline uses vanilla baseline; author didn't hide the concise comparison).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: record caveman-debunk SATURATION — lead with method/leaderboard, not caveman
Reality check (web-confirmed): the caveman critique is ~6mo old and crowded
(Kuba Guzik 14-21%, GrowwStacks, HN top comment, Decrypt, devneeddev/X; author
conceded; token-efficient author self-debunked). We are NOT first on the verdict.
Our novelty is the METHOD (reproducible multi-turn correctness-gated harness +
code audit + the output-GROWS finding + N-skill scale), not the conclusion. Add a
saturation warning to FINDINGS and reframe the roadmap article bullet: method/
leaderboard-first, caveman as a validation row, don't make it the flagship.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record benchmark/leaderboard prior art for the A1 leaderboard
Web sweep of who already benchmarks/ranks Claude Code skills, to position the
public leaderboard: (1) subjective "best skills" listicles abundant (SEO, no
measurement); (2) install/adoption leaderboards exist (quemsah n8n metrics +
published install counts — popularity, not effectiveness); (3) a few one-off
measured benchmarks (caveman saturated; a real superpowers 12-session A/B that
VINDICATES it — 9% cheaper/14% fewer tokens; MindStudio "5 skills 70%" claim-
pushing). Open slice vigiles owns: reproducible + multi-skill + claim-vs-measured
+ correctness-gated + head-to-head + continuous + open-method. Position on the
claim-gap + reproducibility + correctness axes, not "ranking plugins" (install
boards exist) — and honestly (some claims, e.g. superpowers, hold).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — methodology audit, caveman saturation, leaderboard-next
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF competitive block under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): archive canonical eval runs (the data behind FINDINGS survives)
The raw per-run JSON is gitignored scratch under results/; this commits the haiku
+ sonnet canonical runs (the data the published FINDINGS numbers are computed from)
into results-archive/ so the measurement data — the moat — survives a container
reclaim, is auditable/reproducible, and never needs a paid re-run to recover. Adds
a README with provenance + schema + method/reproduce pointers. Curated: only runs
cited in FINDINGS get archived; routine runs stay in results/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: prettier-ignore archived run JSON (machine-written archival data)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: refine cross-ref moat — harness-native is the value, linter-catalog is legacy
The cross-check engine's value is NOT uniform: HARNESS-NATIVE references (tool
contract, MCP server#tool LIVE resolution, hook events, file/script paths,
delegate targets) are the moat — agent-era-specific, fail silently, nobody else
checks them. The 7-linter-catalog integration (ESLint/Ruff/...) is the founding
feature but the WEAKEST leg (linter self-enforces regardless of the prose mention;
heaviest engineering, least harness-specific payoff) — keep it as supporting, don't
add more catalogs. Deepen harness-native instead; first build = live MCP tool
resolution of the real mcp__server__tool refs. Updated landscape (REFINEMENT block),
roadmap §Now, HANDOFF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(mcp): live-verify real mcp__server__tool contract refs (harness-native cross-check)
Bridge the live MCP engine (listMcpTools/closest in mcp.ts, until now wired only to
the explicit `vigiles:mcp` mark) to the agent's REAL `mcp__server__tool` references.
verifyMcpContractTools starts each referenced+declared server once and checks the
TOOL actually exists on it — catching the create_issue→issue_write rename rot that
the static verifyMcpToolServers (server-declared only) misses. Refs to undeclared /
built-in / plugin-namespaced servers are skipped (the static check's job; we have no
config to start them). New mcpToolParts (mcp-tool.ts) extracts {server,tool};
mcpContractToolMessage renders the did-you-mean. 5 tests against the real
fixture-mcp-server.mjs (CI-safe, offline): existing-resolves, renamed→suggestion,
undeclared-skipped, plugin-namespaced/non-MCP-skipped, unreachable-server. Opt-in
(starts processes) — engine first; scan/CLI wiring is the next increment.
This is the harness-native cross-check moat (the valuable leg), not the linter-catalog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — MCP contract-tool resolution shipped; trim to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record first live-MCP dogfood finding — 3 dead tool refs in MadAppGang tester.md
Dogfooded verifyMcpContractTools (the new live MCP tool-resolution) against the real
vendored MadAppGang frontend plugin + the real chrome-devtools-mcp server: tester.md's
tools: lists 27 mcp__chrome-devtools__* tools, 3 of which no longer exist on the live
server — emulate_cpu + emulate_network (consolidated into `emulate`) and
navigate_page_history (removed). 24/27 resolve. Classic MCP rename rot, invisible to
every static linter (the plugin declares no server, so even mcp-tool-resolves is
silent). Recorded in the public-disclosures table + a method/repro note. The
harness-native cross-check moat validated on real OSS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — MCP live-resolution dogfooded, found real MadAppGang bug
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record live-MCP-check scope — high-value but narrow (1/4 vendored plugins)
Dogfooded verifyMcpContractTools on all 4 vendored plugins: only MadAppGang pins
mcp__server__tool refs (the 3 dead-ref finding); superpowers/wshobson reference no
MCP tools; oh-my-claudecode declares a server but no surface pins its tools. So the
live check is sharp but narrow — bites only where a contract hard-lists specific MCP
tool names. Recorded honestly so we don't overstate its breadth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EOF
)
* feat(scan): wire live MCP tool resolution behind opt-in scan --verify-mcp
Expose verifyMcpContractTools through the scan surface: `scan --verify-mcp`
starts each declared MCP server and checks that every mcp__server__tool the
plugin's agents reference actually exists on it — the dynamic check no static
linter can do (catches rename/removal rot like create_issue->issue_write).
Opt-in and side-effecting (spawns servers); returns [] when no servers are
declared. Static verifyMcpToolServers still owns undeclared-server refs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — scan --verify-mcp shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(scan): measure cross-skill selection collision (scan --collisions)
The behavioral confirmation of the deterministic description-overlap rule: run
each model-invocable skill's own prompts against the whole installed plugin and
record WHICH skills fired (whichSkillsFired) — an N×N matrix whose diagonal is
recall and whose off-diagonal mass is one skill HIJACKING a sibling's prompt.
The cross-skill precision question per-skill trigger-rate can't see; the
blast-radius column the leaderboard needs.
- eval.ts: whichSkillsFired + runSkillSelectionTrial (one pass yields the whole
matrix, N× cheaper than per-pair; bodies stubbed so a run stops at selection).
- scan-behavioral.ts: buildSelectionReport (pure fold) + measurePluginSelection
(real claude) + measurePluginSelectionWith (injectable). Claude Code only —
Codex has no discrete skill-selection event.
- CLI: scan <dir> --collisions --prompts=<file.json>.
- Dogfood (on the sub): vigiles's own 3 skills are collision-free; bonus signal
edit-spec under-fired (recall 33%). superpowers slice can't fire (missing
gateway skill). See research/plugin-selection-collision.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — selection-collision matrix shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: verify superpowers null result — hook-primed plugin, harness drops hooks
The superpowers 0% recall is a measurement artifact, root-caused from source:
its skills activate via a SessionStart hook that injects the using-superpowers
gateway priming, but the trigger/collision tier stubs the plugin to skills-only
(packageSkillsDir drops hooks/), so the gateway never runs — compounded by the
vendored slice missing using-superpowers/SKILL.md. Confirmed 0% in --trigger too
(not collision-specific). Records the limitation + a roadmap follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(scan): label hook-primed 0%-recall as a measurement artifact (Layer 1)
A stubbed trigger/collision run rebuilds the plugin to skills-only and drops the
plugin's hooks — so a SessionStart hook that primes skill selection (superpowers'
using-superpowers gateway) is lost, and recall collapses to a misleading 0%.
Layer 1 (honesty before fidelity): when a stubbed run on a SessionStart-hooked
plugin shows EVERY measured skill at recall 0, relabel it "hook-primed — likely a
measurement artifact; re-run against the full plugin install" on both the trigger
and collision columns (hasSessionStartHook + isStubbedHookArtifact). The all-zero
gate is the precision guard — a genuine single-skill miss stays reported as real.
Verified on the live superpowers run + 3 CI fake-driver tests (hooked→labeled,
non-hooked 0%→stays real, trigger relabel). Layer 2 (a --no-stub faithful path +
auto-fallback) is the deferred follow-up in research/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — Layer-1 hook-primed artifact label shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(leaderboard): publishable Markdown table + real names; v0 corpus run
Make the structural-health leaderboard a publishable artifact:
- formatLeaderboardMarkdown — a ranked Markdown table (README/gist/site form),
wired as `scan <dirs...> --md`.
- rankPlugins now labels each plugin by its manifest name (.claude-plugin/
plugin.json) instead of a SHA-pinned dir basename.
bench/leaderboard/ — a reproducible v0 generator (run.mjs + corpus.json) that
clones public plugin repos, finds on-disk plugin roots, ranks them, and writes
RESULTS.md (scores only; no plugin code vendored). First run: 83 plugins from
obra/superpowers + wshobson/agents — grade spread A:16 B:28 C:17 D:9 F:13, with
real outliers (a dead agent tool, large uncontracted/untested collections at F).
The deterministic substrate for the public plugin leaderboard (the adoption
engine). Behavioural columns (trigger-rate, collisions) stack on top later.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — leaderboard v0 shipped, broaden-corpus next
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cli): capability-diff — does this change widen the agent's blast radius? (moat #2)
The bridge bet, v0. `vigiles capability-diff <before> <after>` computes each
version's whole-harness capability lattice (computeHarnessCapabilities — read-only
/ side-effecting / unknown-MCP tools + loosest purity) and diffs them:
- src/core/capability-diff.ts (pure, harness-agnostic): diffCapabilities +
formatCapabilityDiff + isNoOpDiff. WIDENED iff a new side-effecting/unknown tool
or a loosened purity floor; new read-only + removals reported but NOT a widening.
- CLI handler: informational by default (exit 0, a PR comment), --fail-on-widen for
the opt-in CI gate (don't cry wolf — widening is often intended).
6 pure-core tests + 3 CLI e2e (the exit-code contract). v0 reads the tool-bucket
lattice; the richer per-step effect-row + cross-step accumulation (turning "gained
Bash" into "opened an exfil path") is the deferred M1 follow-up in research/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — capability-diff #2 v0 shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cli): consolidate behavioral measurement into a `measure` verb
Bad DX cleanup. scan had accreted ~14 flags by bolting every capability onto it.
Re-grounded on one rule: a verb per JOB/LAYER, flags for modifiers.
- NEW `vigiles measure <dir> --prompts=p.json` — the model-gated behavioral report
(trigger-rate recall+precision AND the selection-collision matrix, one report).
The OSS-testing front door, replacing `scan --trigger` + `scan --collisions`.
- scan goes back to FREE/deterministic: the behavioral flags move to `measure`.
- capability-diff is a FREE check → folded from a standalone command into a scan
flag: `scan <after> --capability-diff=<before>` (was `capability-diff a b`).
- printUsage now lists scan + measure (scan was missing entirely).
- Hand-off text (optimize/score-explainer) + docs/cli.md point at `vigiles measure`.
Net: scan ~14 flags → ~6 (all deterministic); one new justified verb (the missing
paid-measure layer); one needless verb removed. Underlying engine unchanged
(probePluginTriggers / measurePluginSelection / diffCapabilities). Full suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — CLI consolidated into measure verb; lint/scan fold open
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: poach the app-frameworks for a typed CLAUDE.md (web-verified moat synthesis)
research/typed-claude-md-poach.md — consolidates + extends the scattered poach notes
into one moat doc, web-verified (2026-06-22) against Mastra / LangGraph.js / Pydantic
AI / Vercel AI SDK / Inngest / Temporal / Trigger.dev / Effect-TS.
The thesis: they type the agent you BUILD IN CODE and leave instructions a prose hole
(`instructions: string`); vigiles types the prose harness they don't. Verified moat
points: (1) the whole-harness REGISTRY (one tsc over every inter-agent edge) is unique
— nobody else emits a repo-scale typed handoff registry (the routeTree.gen.ts analogy);
(2) cross-agent handoff checking is universally weak/manual even in the typed frameworks
(Pydantic "no auto-derived cross-agent Supplies check"); (3) nobody types the
tool-permission/capability surface at compile time (the effect-row gap). Ranked bets:
effect-row (M1, headline) > Zod-result() > typed context/needs > exhaustive err-track
(poached from Effect-TS, novel).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: the reliability-RUNTIME moat — type the harness's dynamic structure
research/harness-protocol-flow-moat.md (deep FP pass, web-verified 2026-06-22 across
the Codex/CC failure corpus). The existing FP docs type the harness's static capability
SET (effect-row, capability graph, trifecta). This opens the THREE dimensions a set
discards — and reframes them as a category, not a linter:
- ORDER (typestate/session types): a destructive action is UNREACHABLE until its guard
step ran (kills destroy-without-backup: #50027 DROP SCHEMA, terraform destroy, rm -rf).
- FLOW (information-flow/noninterference): untrusted input cannot reach a sink — the
trifecta as a real source→sink path (CVE-2026-21852, Trail of Bits line-jumping).
- REPLAY (linear types/idempotency): exactly-once side effects — the unsolved
duplicate-on-replay hole (ACRFence, 45 issues/12 frameworks; idempotency keys fail
for LLMs). Multiparty session types = deferred 4th axis.
The MOAT framing per the new steer: this is a deterministic RELIABILITY RUNTIME (the
spec→PreToolUse-gate enforcement, surviving compaction OUTSIDE the context window), not
a lint rule — "make the harness measurably more reliable," provable via the eval layer.
Grounded in: prose-doesn't-bind (#32163 "@enforce"), declared≠enforced (SDK #172/#162),
METR <10%@4h, 56% no-recovery, trifecta in 98% of prod agents.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reframe the moat as a reliability RUNTIME (3 axes) + HANDOFF
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guards): prototype typed safe-by-construction harness guards (EXPERIMENTAL)
Dogfooded the real OSS hooks (our pre-edit.sh block-on-compiled, superpowers/OMC
session-start inject, OMC keyword-detector) → the safety-critical shape is the
PreToolUse gate. A hook today is arbitrary shell (the RCE footgun, CVE-2025-59536)
and its safety is undecidable (Rice). So invert it: DECLARE a guard from a closed
vocabulary, GENERATE the hooks block pointing at vigiles's own gate (no user shell).
src/core/guards.ts (not wired to CLI/public API — a sketch):
- guard.block / requireBefore (the ORDER axis: destroy-after-plan) / confine
- decideGuards — the pure runtime decision (reuses arg-match's matchesArgs)
- compileGuards — generates a Claude Code PreToolUse hooks block whose command is
`npx vigiles guard-hook` (vigiles's audited gate), safe-by-construction
- 4 tests: block reproduces pre-edit intent, requireBefore denies destroy-before-plan
+ allows after, confine denies an out-of-allowlist path, compile emits a valid block
See research/harness-protocol-flow-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(roadmap): reliability-runtime moat + ORDER/FLOW/REPLAY (REPLAY=P3), crosslinked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guards): runnable guard-hook gate — session ledger + ORDER enforcement live
Make the EXPERIMENTAL typed-guard prototype run end-to-end. The `vigiles
guard-hook` PreToolUse subcommand reads the live event, loads the declared
guard set (`.vigiles/guards.json`) + the session ledger
(`.vigiles/guard-ledger.json`), runs `decideGuards`, and blocks (exit 2 +
reason) or records the allowed call so a later `requireBefore` sees it.
- guards.ts: serializeGuards/parseGuards (RegExp-safe, mirrors tool-intercept's
env round-trip), loadGuards, readGuardLedger/recordGuardCall (the prior-call
ledger CC doesn't surface to a hook), parseGuardEvent, runGuardHook (allow →
record; deny → record nothing — a blocked call never happened).
- cli.ts: `guard-hook` subcommand mirroring agent-hook.
- Tests: serialize round-trip incl. RegExp, tolerant parse, ledger IO, and a
runHook e2e proving `terraform destroy` is blocked before `terraform plan`
and allowed after — the ORDER axis enforced across hook invocations, live.
The gate's command is vigiles's OWN audited binary, never user shell —
safe-by-construction. Still not on the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): guard-hook gate runs end-to-end (ORDER axis live)
* feat(hooks): spike — typed, effect-classified hook specs (the "type-safe bash" angle)
A spike to test whether the compile step earns its keep on the hook surface —
the least-typed, most-footgun-prone part of a harness. Catches two silent bugs
hand-written shell can't:
1. WRONG-FIELD EXTRACTION (silent no-op): a hook matching `Bash` but reading
`file_path` extracts empty forever and never fires. validateHook makes it an
error; the typed `hookFor` builder makes it a tsc error at edit time (proven
by an @ts-expect-error the build enforces).
2. EFFECT MISDECLARATION ("type-safe bash"): an observe-only hook that runs a
mutating/undecidable command doesn't compile — rides bash-effects.ts, the
deterministic classifier no other plugin tool has.
compileHook refuses (throws) any error-level issue — "an unsafe hook doesn't
compile" — and emits the settings block + correctly-typed field extractions
(never a hand-typed jq string). Pure core, harness-agnostic: the per-tool field
catalog is injected (Codex differs), the CC catalog + typed builder live in the
test. Not wired to the CLI/public API. See research/harness-protocol-flow-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): save the verified hook pain-point corpus + the verify-not-gate killer feature
Five web-research passes (claude-code issues, official docs, dev.to/Medium, HN,
CVEs), all verified. The convergent #1 pain is FALSE CONFIDENCE — a safety hook
looks like a guardrail and silently isn't (exit 1≠2, wrong JSON field,
PostToolUse-can't-block, wrong jq path); RFC #45427 names the thesis and was
closed not-planned. Killer feature: "prove your guardrail actually blocks" via
runHook + decideHook (verify the decision LOGIC, not deliver enforcement) — works
on existing hooks with no spec/compile, sidesteps CC's delivery bugs
(subagent-bypass #34692 etc.). Compile is the optional Level-2 (declared intent →
auto-generated test), not the mechanism. Crosslinked from harness-protocol-flow-moat
+ roadmap (new HIGH item; gate demoted to MED — undercut by the same CC bugs).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(unit): prove your guardrail actually blocks — disaster-event battery over runHook
The #1 verified hook pain is FALSE CONFIDENCE: a safety hook looks like a guard
and silently doesn't block (exit 1≠2, wrong JSON field, wrong jq path) — "three
teams believed they'd blocked force pushes." This is the deterministic answer.
- src/guardrail-check.ts: a curated DISASTER_CATALOG (force-push incl. compound,
reset --hard, rm -rf, --no-verify, ssh-key read, curl|sh) + verifyGuardrail
(feeds each to a hook via runHook, reports blocked), assertBlocksDisasters (the
CI gate — throws naming the misses), formatGuardrailReport (NEUTRAL coverage
map — never judges intent, since not every hook is a bash-safety guard).
- Exposed on vigiles/unit (no model, no key, CI-runnable; works on hand-written
hooks with no spec). api reports regenerated.
- scaffold-test: a hook scaffold now emits the runnable battery (coverage map +
the commented assertBlocksDisasters CI gate keyed by declared category).
- Committed receipt: exit-1 vs exit-2 force-push guards, the engine tells them
apart; intent-declared assert passes for git, fails honestly when a guard
claims fs-safety it doesn't deliver.
Verifies decision LOGIC, not delivery — flags CC's runtime bugs (subagent-bypass
#34692 etc.) but never claims to fix them. See research/hook-pain-points.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): guardrail VERIFY feature shipped; refresh shipped-this-session
* docs(research): record guardrail VERIFY shipped + the neutral-map vs declared-intent design lesson
* docs(research): real-plugin dogfood — disler hook blocks rm/.env but no-ops vs force-push/ssh-read/curl|sh
* feat(spike): hook-as-typed-program probe — testability/safety/eval/portability + stamping
Probe the constrained-API compiled-hook model: a hook is a PURE typed function
(event)=>Decision against a closed `vigiles/hook` vocabulary, vigiles compiles it.
src/core/hook-program.ts + a 5-claim test, all passing in-process (no subprocess,
no model):
1. PURE/TESTABLE — decide() unit-tested directly; the false-confidence protocol
(exit 1≠2, JSON field) is unrepresentable (compiler emits exit-2).
2. AST-BACKED MATCHING — command.runs("git push",{force}) via new bash-effects
`leafCommands` (AST leaf extraction): catches the compound bypass
`cd x && git push -f` the native glob (#30519) misses AND avoids a grep
false-positive on `echo "...git push..."`.
3. COMPILES to a real CC hooks block.
4. CAPABILITY = API SURFACE — checkHookImports rejects any import outside
vigiles/hook (+ eval/Function/dynamic-import); an out-of-API hook does not
compile.
5. STAMPING (the "fix #4" idea) — stampHook/verifyHookStamp make the compiled
artifact tamper-evident (integrity.ts pattern): a hand-edit smuggling
child_process breaks the stamp → runtime refuses it.
Pure core, harness-neutral, not on the public API. Honest limits unchanged
(buy-in, node latency, CC subagent-bypass #34692 — compile fixes authoring not
delivery). See research/hook-pain-points.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record compiled-hooks probe result (5 claims proven incl. stamping)
* feat(spike): widen hook-program probe — path-confine gate + context-injection (vocabulary holds)
Probe 2 tests whether the closed vigiles/hook vocabulary holds across two
genuinely different shapes. It does — and reveals the vocabulary is a small
FAMILY keyed by hook ROLE, whose per-role output type eliminates two more
verified pains by construction:
- A second GATE: defineFileGate over Edit/Write with a PathView.under() matcher
(different tool, field, matcher) — confines writes to src/test, denies escapes.
- A non-gate: defineInject (SessionStart) returns Injection (context text), not a
Decision. runInject emits the RIGHT JSON field (additionalContext) — the author
never picks it, so the wrong-field pain can't occur.
- Correct-by-construction: an inject hook CANNOT return deny() — it's a tsc error
(proven by @ts-expect-error the build enforces). "Block on a SessionStart/
PostToolUse hook" (a documented mistake) becomes a type error, not a silent no-op.
Finding recorded in research/hook-pain-points.md. Still a pure-core spike, not on
the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): record probe 2 — hook vocabulary holds as a role-keyed family
* feat(spike): probe 3 — PostToolUse react hook (bounded side effects complete the family)
The third hook shape: a react hook (PostToolUse) fires after the tool ran, so it
can't block — its job is to DO something (format, recompile, warn). That's where
side effects re-enter (gate + inject are pure). The constrained API keeps them
BOUNDED: a react returns a typed Reaction whose run(cmd) is EFFECT-CLASSIFIED at
construction (bash-effects), so even the side-effecting role stays analyzable —
you can list/diff exactly what every react runs and its effect. And a react still
CANNOT block (no deny in Reaction) → "block on a PostToolUse hook" (a documented
mistake) is a tsc error (build-enforced @ts-expect-error).
Completes the role-keyed family: gate (decide→Decision, pure) / inject
(produce→Injection, pure) / react (react→Reaction, bounded side effects). Each
role's output type makes its wrong-output bugs unrepresentable. Finding recorded
in research/hook-pain-points.md. Pure-core spike, not on the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: probe 3 — hook vocabulary sound across the full role-keyed family (gate/inject/react)
* docs(handoff): condense the hooks narrative under cap
* docs(handoff): center the in-flight compiled-hooks ship arc (steps 1-3) for resume
* feat(hook): ship compiled hooks — author a hook as a pure typed program
Ship the `vigiles/hook` public surface + CLI so a hook is authored as a pure
typed `(event) => Decision` against a CLOSED vocabulary and vigiles compiles
it. This ELIMINATES whole hook bug classes by construction: the author never
writes the exit code / JSON field (the false-confidence class is
unrepresentable), the matcher is AST-backed (catches `cd x && git push -f`
the native glob misses), capability = API surface (an out-of-vocabulary import
does not compile), and the compiled artifact is tamper-evident (a hand-edit
that smuggles a capability breaks the stamp → the runtime refuses it).
- `src/hook.ts` — the `vigiles/hook` barrel (gate/inject/react vocabulary).
- `package.json` exports `./hook`; api-extractor tracks the new surface.
- CLI `compile-hook <file>` (capability-checks, emits the settings block +
stamp sidecar) and `run-hook-program <file>` (the runtime: loads the typed
program, verifies the stamp, dispatches by role — gate→exit 2, inject→
additionalContext, react→classified run).
- Expand CommandView with two principled high-signal matchers: `touches()`
(secret-path reads) and `pipesToShell()` (curl|sh remote code).
- OSS dogfood (src/hook-dogfood.test.ts): a hand-written substring guard (the
widely-copied disler shape) misses 5/7 of the disaster battery; the compiled
hook blocks 7/7 by construction — measured with our own DISASTER_CATALOG.
Honest scope kept in the surface docs: compile/verify fix AUTHORING + LOGIC,
not DELIVERY — CC's subagent-bypass (#34692) still bypasses any PreToolUse
hook, so a gate is a strong default, never an unbypassable wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reposition around harness reliability + the compiled-hooks pillar
Rewrite the README top-line from "lint/test/measure" to "make the harness your
AI agent runs on reliable", with four instruments under it (Verify / Guard /
Test / Measure) — Guard is the new compiled-hooks pillar, not a bolt-on. Add a
README ② Guard section and a full docs/compiled-hooks.md guide that says, plainly
and up front, that compiled hooks make WHOLE CLASSES OF BUGS UNREPRESENTABLE
(false confidence, matcher bypass, capability creep, category mistakes), with a
bug-class table and the OSS dogfood result (2/7 → 7/7).
Cross-link the new guide from the README, the docs index, the CLI reference
(compile-hook / run-hook-program documented), and the sibling verify + test
guides. Internal: add the compiled-hooks positioning paragraph + keyFiles to
CLAUDE.md (via its spec) and recompile.
Honest scope kept everywhere: compile/verify fix the hook's authoring + logic,
not the harness's delivery — a subagent's tool calls still bypass any PreToolUse
hook (#34692), so a gate is a strong default, never an unbypassable wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): design the Codex compiled-hook adapter + record the ship
Add research/compiled-hooks-codex.md: the plan to compile one typed vigiles/hook
program to Codex through the existing ports — no core edits. The key finding is
that the typed program, AST matcher, capability check, stamp, and runtime
decision are all harness-neutral, and Codex's veto is exit-2-identical to Claude
Code's, so the runtime is shared; only the EMIT differs (CC JSON settings vs
Codex TOML [hooks], glob vs anchored-regex matcher), routed through the
PluginLayout.settingsFormat + dialect/HookProtocol ports already in place. Notes
the boundary discipline (ports by injection) and the non-CC-shaped dogfood, and
why OpenCode's in-process hooks are a different emit target.
Record the compiled-hooks ship + the OSS dogfood in hook-pain-points.md and
cross-link from codex-prototype-findings.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): record the compiled-hooks ship + reposition; next = Codex emit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(hook): compile a typed hook to Codex (TOML) as well as Claude Code
Generalize the compiled-hook emit so ONE typed vigiles/hook program compiles to
either harness, through the existing adapter ports — no harness-boundary edits.
The typed program, AST matcher, capability check, stamp, and the gate runtime
are all harness-neutral (Codex vetoes a tool call via exit 2 identically), so
only the emitted settings block differs.
- compileHookProgram(source, hook, gateCommand?) → (source, hook, opts?) where
CompileHookOptions = { gateCommand?, dialect?, hookProtocol?, settingsFormat? },
all optional and defaulting to the Claude Code JSON block (back-compatible).
It validates hook.on against the dialect (verifyHookEvents — a typo'd or
unsupported event won't compile), styles the matcher per
HookProtocol.matcherStyle (CC exact `A|B` vs Codex anchored `^(A|B)$`), and
renders settingsBlock per settingsFormat (CC nested JSON vs Codex TOML
`[[hooks.<event>]]` via @iarna/toml).
- Add HookProtocol.matcherStyle?: "exact" | "regex" (additive); codexHookProtocol
sets "regex".
- compile-hook resolves the target harness (--harness= / detect) and threads the
adapter's dialect/hookProtocol/layout.settingsFormat in; prints the right block
+ target ("add to Codex's config.toml").
Tests (non-CC-shaped, per the adapter-aware-lint discipline): the Codex emit is a
TOML `[[hooks.PreToolUse]]` block with a regex matcher; an unsupported event does
NOT compile; the CC default still emits JSON + exact matcher. Docs: cli.md +
compiled-hooks.md note the multi-harness emit; research/compiled-hooks-codex.md
marked BUILT with the one deferred piece (inject/ask output JSON on Codex). Also
fix 3 pre-existing no-confusing-void-expression lint errors in
guardrail-check.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): Codex compiled-hook emit built (step 4)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(hook): make the Codex inject/react gap LOUD, not silent
The deferral of Codex inject/ask OUTPUT (its field shape is CC-confirmed only)
was correct — guessing a wire format would recreate the silent-failure bug the
feature exists to kill — but it was SILENT: `compile-hook --harness=codex` on an
inject/react hook emitted a clean-looking TOML block whose runtime output is
CC-shaped and may quietly no-op on Codex. Now it warns loudly (⚠) that the
output is unconfirmed for that harness and to use a gate hook meanwhile; a gate
hook (deny→exit 2, cross-harness) compiles with no warning. E2E test covers
both. Docs/research note the loud behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revamp the compiled-hooks docs — "eliminates a class of bugs", link sources, explain the cons
- Lead the bug-class section with "eliminates an entire class of bugs by
construction", and link EVERY class to its verified source: false confidence
(RFC #45427 / #24327), matcher bypass (#30519), capability creep
(CVE-2025-59536), category mistakes (#4362).
- Replace the single delivery-floor caveat with a full "Limitations & trade-offs
(the cons)" section: the delivery floor (#34692 + route-around #45427/#32376),
runtime cost (node spawn per event), buy-in (dep + build step vs inline bash),
the bounded-vocabulary ceiling, "compiling proves the protocol not your policy",
and the unconfirmed Codex inject/ask output.
- Integrate the gate instrument into the reliability frame: README Guard section
uses the "eliminates an entire class of bugs" framing + links to the cons; the
Test guide intro names the four instruments (verify / guard / test / measure)
instead of only a bottom see-also.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(compiled-hooks): clarify the ceiling — capability closure (fundamental) vs matcher coverage (extensible)
The bounded-vocabulary con conflated two surfaces: what a hook can DO (the
deliberate no-fs/net/exec capability closure — the real ceiling and the safety
guarantee) vs what it can SEE (the AST matchers, a soft/extensible limit). Split
them so 'some hooks stay hand-written' reads as the capability rule, not a
weakness of the Bash matcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(hook): in-process testing for compiled hooks — runHookProgram + assertHookDenies/Allows
A compiled hook's decision is a pure function, so it should be testable WITHOUT
spawning the CLI. Add the cheapest test tier:
- `runHookProgram(hook, event)` (core, on `vigiles/hook`) — one in-process
dispatcher over the whole role family: a gate → its Decision, an inject → the
injected context, a react → its classified Reaction, returned as a normalized
discriminated `HookProgramOutcome`. No subprocess, no model.
- `assertHookDenies` / `assertHookAllows` (harness-assert, on `vigiles/unit` +
`vigiles/testing`) — the throwing asserts over a compiled hook program
directly, the sibling of assertHookBlocked/assertHookAllowed (which take a
`runHook` subprocess result).
Tests: runHookProgram dispatches all four roles (hook-program.test.ts);
assertHookDenies/Allows pass/throw with a message naming what it got
(harness-assert.test.ts). Docs: a "Testing a compiled hook" section in
docs/compiled-hooks.md (the three tiers by cost) + a testing-api.md reference
entry; CLAUDE.md keyFiles updated. api-extractor tracks the new surface.
Behavior testing is deliberately NOT added: a hook isn't model-selected and its
output isn't prose, so there's no trigger-rate/judged dimension; "does it fire in
the assembled harness" is the existing runHarnessTest tier.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): record in-process hook testing (step 5)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: link the parked prototype code so it's discoverable later
The unshipped/experimental code wasn't in the Key Files map — a future session
would only find it by grep. Fix the traceability:
- Add CLAUDE.md Key Files entries (via the spec) for the parked prototypes:
`src/core/guards.ts` (+test, WIRED via `vigiles guard-hook` but EXPERIMENTAL —
the GATE-axis bet, superseded in practice by compiled hooks + VERIFY) and
`src/core/hook-spec.ts` (+test, a pure spike imported NOWHERE, superseded by
the shipped compiled hooks). Each says why it's parked + points at
research/harness-protocol-flow-moat.md to resume.
- Add a "parked prototype files" note under the roadmap's guards bet recording
the lineage (hook-spec's typed-effect-classified idea → shipped compiled hooks)
so the durable map carries it too.
No code change; recompiled CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(hook): cover the assert message branches (100% threshold)
The new assertHookDenies/assertHookAllows helpers dropped coverage below the
repo's 100% global threshold (harness-assert.ts:144-145,165 — the describeOutcome
injection/reaction branches + the assertHookAllows fail path were unexercised).
Add asserts that drive each: assertHookAllows throws on a deny, and
assertHookDenies against an inject / react hook fails with a message naming the
role ("an injection" / "run (a reaction)").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(lint): add a config `exclude` so vendored/benchmark fixtures skip require-spec
The dogfood `vigiles lint` (the CI `check` job) failed `require-spec` on
`bench/ecosystem/skills/token-efficient/CLAUDE.md` — a SHA-pinned third-party
file injected verbatim as a benchmark arm (SOURCES.md: "the real CLAUDE.md"). A
per-file disable comment would perturb that verbatim arm + dent its fidelity, so
add a config-level exclude instead: `exclude` globs in .vigilesrc.json drop paths
from `lint` instruction-file discovery (node_modules/dist/.git stay always-out).
Set `"exclude": ["bench/**"]` — benchmark data isn't the repo's harness, so the
repo's own lint shouldn't police it. The vendored file is untouched.
`node dist/cli.js lint` now exits 0. Documented in docs/rules/require-spec.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(eval): cover runSkillSelectionTrial / whichSkillsFired (100% statements)
The `test` job's 100% statement gate failed on eval.ts (99.42%): the
selection-collision primitives runSkillSelectionTrial + whichSkillsFired (from
|
||
|
zernie
|
703d298391 |
feat!: typed-spec moat — typed purity, composition & whole-harness type-checking (#43)
BREAKING CHANGE: effect() in a skill .spec.ts is now a compile error; the skill PreToolUse effect-boundary gate is removed (the per-call purity floor remains). |
||
|
zernie
|
3b81e5042b |
feat: spec-contract hardening + complete spec/railway docs + subagent-trace eval fix (#42)
Spec: context:fork + gated forked-skill output, agent color/disallowedTools, dir()/glob(), didNotWrite(), scaffold-test, section length-guard, migrate->adopt-spec. Docs: docs/spec-format.md completed + docs/railway-subagents.md + research/spec-syntax-and-railway-scope.md decisions. Testing harness: parseSubagents recovers --plugin-dir subagent nested-traces (namespaced subagent_type + the sub's returned result() block). Evals (real-model): caveman debunk hardened on sonnet (-23%/-20%); the does-our-spec-help A/B validated the spec HELPS (parseable outcome 0%->100%, no quality regression). |
||
|
zernie
|
5c3cd2f947 |
feat!: layout-driven multi-harness validation rules & harness testing (#41)
* docs: add rules-docs-in-sync rule; fix nested blockquote in harness-testing Add a `rules-docs-in-sync` guidance rule to CLAUDE.md so the validation-rule set (RulesConfig keys + built-in vigiles/* rules) and the central rules table in docs/cli.md never drift: adding/removing/renaming a rule must update the type, the table row, and the per-rule doc together. Also fix the doubly-nested blockquote (`> >`) in docs/harness-testing.md that rendered as a confusing quote-inside-a-quote — convert the pasteable prompt to a fenced code block, matching the README convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: surface the lint rule set in the linting guide; break up walls of text The Lint guide (verifying-instruction-files.md) explained the three rule *types* but never showed or linked the actual validation-rule *list* — so a reader following the README's Lint link couldn't find the rules. Add a "What vigiles lint checks" section that summarizes the rule families and links the canonical table in cli.md (no duplication, per rules-docs-in-sync). Also break four dense prose paragraphs in harness-testing.md into bullets / shorter paragraphs (entry points, harness-specific transport, the two-layers explainer, the eval-tool comparison) for mobile scannability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(orphans): never flag harness-loaded instruction files; move rules matrix to lint guide orphan-docs now categorically exempts files the harness loads directly (CLAUDE.md / AGENTS.md / SKILL.md / agents/ / commands/) from the orphan CANDIDATE set — they're load-bearing by name/location, not by being linked, so they're never orphans even if `orphans.include` is broadened to the whole repo. They still count as referencers. Adds tests + a docs/rules/orphan-docs.md. Move the canonical validation-rules matrix out of the CLI reference and into the linting guide (verifying-instruction-files.md), where a reader looking for the rules expects it — grouped by family (spec & integrity, test coverage, reference marking, subagent contracts, hooks & MCP, skill triggers, docs hygiene). cli.md and the README now link to that one matrix. Updates the rules-docs-in-sync rule to name the linting guide as the canonical home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: overhaul harness-testing — task-first guide + extracted API reference The harness-testing guide buried the first runnable example five screens deep, under ~140 lines of architecture (capability-contract theory, composition-root, adapter selection, import taxonomy) and didn't show its TOC until line 118. A first-timer couldn't find "how do I test my hook?". Full split: - Rewrite docs/harness-testing.md as a lean task-first guide (902 → 297 lines): a copy-paste first test in the first screen, a "what do you want to test?" decision table as the primary nav, trigger-rate promoted to a headline section, advanced material (significance / regression / cost / judge) collapsed. - Extract docs/testing-api.md: the full reference — Trace model, predicates, assertions, the check vocabulary, matchers, every measureTriggerRate / runEval option, significance/regression, imports & harness selection. - Move the theory (import-path-is-capability-contract, "two layers or three?") to research/harness-testing.md where the rationale belongs. - Repoint inbound anchor links (harness-testing-claude-code.md) and add the new reference to the docs index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * refactor!: surface-scoped lint rules — rename agent-*→subagent-*, capability-gate, fix dialect drift Adopt the "surface model" for cross-adapter vs adapter-only lint rules: a rule is scoped to the SURFACE it checks (subagents, shell hooks, skills, MCP, docs), not to a harness name. Universal rules stay bare and run everywhere; surface rules run only where the active adapter has that surface and report n/a otherwise (loud, never a failure) — so one config targets Claude Code AND Codex (incl. independent .claude/.codex trees, not just byte-identical mirrors) without per-harness rule duplication. Harness prefixes are reserved for genuinely exclusive concepts (none needed today). BREAKING (no back-compat, per design): - Rename rules: agent-tool-contract → subagent-tool-contract, agent-frontmatter → subagent-frontmatter, untested-agent → untested-subagent. Named by the harness-neutral concept (subagent), resolving the Codex `[agents]` (a concurrency table, not a subagent) name collision at the source. - Add AdapterCapabilities.subagents (CC/OpenCode true, Codex false); the subagent rules gate on it and report n/a where absent. - Fix the lint dialect-drift bug: checkSubagentToolContracts / checkHookEvents / checkFrontmatterSchema now run against the RESOLVED adapter's dialect + layout (and respect --harness), not a hard-coded claudeCodeDialect — scan was already adapter-aware, lint was not. - Docs: add the surface→harness "Applies to" matrix to the linting guide, rename the three rule docs, recompile CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * feat: layout-driven surface classification — ready for new-harness subagent dirs The subagent rules and scan's surface classifiers were hard-coded to `agents/` / `skills/` / `commands/`, so a harness whose subagents live elsewhere (OpenCode's `.opencode/agent`) wouldn't be classified. Make it layout-driven. - PluginLayout gains labeled `skillDir` / `agentDir` / `commandDir` (an empty `agentDir` means "no subagent surface" — e.g. Codex's `[agents]` TOML table). Set per adapter; documented in the adapter API + authoring guide. - scan.ts builds its isSkill/isAgent/isCommand classifiers from the active layout (regex anchored on a real path boundary, path-segment-safe), threaded through every surface helper — no module-level hard-coded dirs. - The `subagent-tool-contract` lint rule now REUSES scanPlugin's already-resolved, layout+dialect-driven agents (one-detector-no-drift) instead of re-globbing a hard-coded `agents/` path; drops the now-dead parseAgentTools/verifyToolContract imports. - Test: scan classifies a subagent under a custom `agentDir` and a default Claude Code scan does not — proving the dir is read from the layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: finish de-CC-ing the lint/scan detectors + add an ESLint guard Two more functional Claude Code hard-codings remained in the deterministic reference-verification path (same class as the subagent glob): - scan.ts `resolveScript` (hook-script-exists) hard-coded `${CLAUDE_PLUGIN_ROOT}`; now expands the harness's own `layout.pluginRootToken` (braced + unbraced). - test-coverage.ts (untested-skill/agent/hook) hard-coded `skills/` / `agents/` / `.claude/` globs and the CC token; now layout-driven via `skillDir`/`agentDir`/ `materializeRoot`/`pluginRootToken`/`manifestPath` (default CC), threaded from the resolved adapter in lint and from `lay` in scan. The deterministic layer-1 (lint + scan) is now fully layout/dialect-driven. The remaining `.claude` literals live only where they belong: the CC adapter, the CC eval transport (src/eval.ts), and CC plugin onboarding (init in src/cli.ts). Prevent recurrence: a `no-restricted-syntax` ESLint rule bans hard-coded CC string literals (`${CLAUDE_PLUGIN_ROOT}`, `.claude/`, `ANTHROPIC_*`) — as plain literals AND template quasis — in the harness-agnostic domain + detectors (src/core, scan.ts, test-coverage.ts, plugin-loader.ts). Verified it blocks a probe. Complements the existing import-graph boundary with a literal boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * test: dogfood the layout-driven untested-surface detector on a non-CC layout A regression guard for the de-CC-ing: build a repo with custom surface dirs (mysurf/skill, mysurf/agent) and plugin-root token ($MY_PLUGIN_ROOT), then assert findUntestedSurfaces discovers all three surfaces via the injected layout — and the default Claude Code layout sees none of it. CC-shaped fixtures can't catch a CC-hardcoding regression; this one can. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: thread the resolved adapter through ALL lint checks + Codex CLI dogfood Completes the dialect-drift fix: 8 more scanPlugin-based lint checks (skill-frontmatter, mcp-config, mcp-tool-resolves, hook-script-exists, disallowed-tools-contract, description-overlap, frontmatter-valid, mcp-hook-target-resolves) were still calling scanPlugin(process.cwd()) with the default Claude Code layout/dialect — only 3 of 11 had been fixed. All 11 now run against the resolved adapter, and the subagent-/hook-surface ones report n/a on a harness lacking that surface (capability-gated). Dogfood: a Codex `lint` CLI golden test (offline, in the existing CI job) wires a real Codex repo — AGENTS.md + TOML [hooks] with the harness's own ${PLUGIN_ROOT} token + an untested skill — and asserts hook-script-exists resolves the token and flags the missing script, untested-skill finds the skill, and the subagent rule reports n/a. A CC-shaped fixture can't catch a CC-hardcoding regression; this one runs the full pipeline against the second harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: codify the adapter-aware-lint convention + fill the capability/layout doc gaps Add an explicit `adapter-aware-lint-rules` guidance rule to CLAUDE.md (recompiled from the spec): lint rules are surface-scoped not harness-named; name by the harness-neutral concept (subagent, not agent); thread the resolved adapter and never hard-code a CC literal (eslint-enforced); gate by capability and report n/a; dogfood must be non-CC-shaped. The full record of this session's de-CC-ing. Docs caught up to the code: - harnesses.md capability matrix + adapter-api.md AdapterCapabilities gain the `subagents` flag. - adapter-api.md / authoring-an-adapter.md PluginLayout gains skillDir/agentDir/ commandDir; the myHarnessAdapter example gains the (previously missing, required) `capabilities` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(ci): update API report for the new adapter fields; add internal audience-check skill The `check` job's API surface gate failed: AdapterCapabilities.subagents and PluginLayout.skillDir/agentDir/commandDir are public (vigiles/adapter), so the committed etc/vigiles-adapter.api.md was stale. Regenerated (diff = exactly those four field additions) and gave each new layout field its own TSDoc so the report has no `(undocumented)` markers. Also add an internal dev skill `dev/skills/audience-check` (not shipped, not counted by untested-skill): re-read the README through six audience lenses — Claude Code user, Codex user, senior/staff skeptic, junior newcomer, eng-manager decision-maker, QA/test engineer — and report per-persona friction + ranked fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: note the UI-created-PR title gotcha in the conventional-commits rule A PR opened from the Claude Code UI gets a plain prose title that fails the `validate` job (which checks the PR title, not the commits). Document the fix: edit the title to a Conventional Commit + the matching breaking `!` as soon as the PR exists; validate re-runs on the title edit. (Hit on PR #41.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
1e798a2a4b |
feat: deterministic scan/lint rule suite for agent plugins (the cross-referencing moat) + scanner hardening (#40)
* fix: anchor scan surface regexes so a '-agents' skill dir isn't a phantom agent Pointing `vigiles scan` at popular OSS plugins (obra/superpowers, wshobson/agents marketplace) surfaced a real misclassification bug: the skill/agent/command path matchers were unanchored, so any directory whose NAME merely ends in the keyword was misread. superpowers ships `skills/dispatching-parallel-agents/SKILL.md`, and the `agents/[^/]+\.md$` matcher hit the `-agents/SKILL.md` substring — reporting a phantom agent named "SKILL" with no tool contract. That both corrupts the scan report and unfairly deducts from the leaderboard score (no-contract penalty). Anchor `skills/`, `agents/`, and `commands/` on a real path boundary (start-of-path or `/`). Add regression coverage for `-agents`/`-commands` skill directories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan traverses marketplaces and counts dangling refs as structural issues Two gaps surfaced while pointing `vigiles scan` at popular OSS plugins: 1. Marketplace traversal. `wshobson/agents` ships 80+ plugins under a single `.claude-plugin/marketplace.json`; scanning the repo root found nothing. `expandMarketplace` resolves each member's relative `source` (skipping external git sources and missing dirs), and `vigiles scan <root>` now expands a marketplace and ranks every member through the leaderboard. 2. Dangling intra-plugin references are now a first-class structural finding, not a buried free-text warning. `ScanReport.danglingRefs` exposes the broken-path / partial-vendor refs (obra/superpowers ships three); the report prints them under a "Broken references" ✗ section, the verdict counts them, and the leaderboard deducts -8 each. Previously superpowers reported "✓ no structural issues found" despite three missing files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: kill scan false positives that running against real plugins exposed Validating the marketplace scan against wshobson/agents (80+ plugins) and obra/superpowers showed the two heuristic findings were almost all FALSE POSITIVES — the exact doc-mention trap the leaderboard avoids. Promoting noisy signals to scored "bugs" would discredit the whole pitch, so tighten both detectors to fire only on real defects: - Dangling refs: scan only EXECUTABLE source files (hook/helper scripts, incl. extensionless ones like superpowers' hooks/session-start), never prose. Every hit across the corpus came from markdown — example task text, a literal "wc -w skills/path/SKILL.md" template, "❌ Bad" samples, React `hooks/*.ts` code comments — none real. The true positive (a script that cats a missing plugin file) is preserved. - Skill descriptions: parse YAML block scalars (`description: >` / `>-`). The regex captured only the `>` indicator, mislabeling four richly- described wshobson skills as "can't trigger". Net: the marketplace now reports 0 broken-ref and 0 missing-description false positives; remaining deductions are real (no tool contract, untested). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: log first behavioral findings — superpowers brainstorming under-triggers Ran the model-gated column (measureTriggerRate, real claude on the subscription) against the full obra/superpowers plugin to find bugs the deterministic scan structurally cannot see. Headline: the `brainstorming` skill — structurally green (good description, no scan finding) — fires on only ~20–30% of its own open-ended use-case prompts (recall 3/10 then 2/10 across two runs; precision a perfect 100% on mechanical edits). A diagnostic showed it isn't losing to a competitor: on 8/10 prompts no skill fired at all. The "You MUST use this before any creative work" imperative reads strong but fires weak — exactly the class of bug only a real model reveals. Records method, the over-trigger hypothesis the data refuted, an actionable fix for the author, the egress column we couldn't run here (no bwrap/slirp4netns), and honest n=10/1-trial caveats. Linked from distribution-strategy (E1) and divergent-bets (#9) so it isn't an orphan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empty-context validity + competitor landscape + auto-context idea Capture the methodology analysis from probing brainstorming: - Is empty context valid? Confirmed each trigger run is a cold single prompt (no fixture/history; TriggerRateSpec has no such field). It's the field STANDARD (AWS sample-agent-skill-eval does single relevant/irrelevant queries), faithful for opening-move skills (brainstorming qualifies — finding stands), biased-low for state-dependent ones (a shared blind spot, not vigiles-specific). - Landscape table: AWS (query, same), promptfoo (user-injected vars/hooks), DeepEval (ConversationalTestCase), Inspect/SWE-bench (seeded sandbox for stateful tasks). Nobody auto-reconstructs a skill's trigger state. - New idea — auto-apply known context: (1) add fixture/prior-turn to TriggerRateSpec (parity with measure()/runEval), (2) auto-select a curated preset context from the skill's declared trigger phrases so it's measured in the state it claims to fire on. Preset selection on explicit cues, not prose synthesis. Logged in roadmap.md (Next, P3-MED). The empty-vs-seeded sensitivity probe is noted as deferred (40 serial runs overran the budget here; re-run with concurrency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: roadmap — per-model trigger-rate + context-rot curve (MED) (a) Report trigger-rate per model — already a capability (model-as-arm), make it a standard column. (b) Study how recall rots as the skill roster grows and whether a stronger model rots slower — the concrete form of divergent-bets #11 (measure model × harness). Opt-in matrix, not a default gate; cheap first probe is brainstorming recall at 2 roster sizes × 2 models. Measure, don't claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: trigger-rate gets fixture (context) + concurrency for plugin testing at scale Two enablers for behavioral testing of OSS plugins, both surfaced by this session's probing limits: - `TriggerRateSpec.fixture` — seed files into each run's cwd so a skill is measured in the repo STATE it claims to trigger on. The default empty cwd is faithful for opening-move skills but under-measures state-dependent ones; this closes that gap (parity with measure()/runEval, which already had fixture). - `TriggerRateSpec.concurrency` — run the prompts × trials grid through the existing runPool instead of fully serial. Default 1 (rate-limit polite); raise it to cut wall-clock on a large prompt set or roster sweep. The serial runner is what made the earlier sensitivity probe overrun. Refactors runTriggerSet to flatten the grid into jobs + aggregate per-prompt (order preserved); extracts runTriggerTrial (also drops the function under the complexity ceiling). Unit tests with a fake runner prove fixture lands in the cwd before the agent runs, and that concurrency>1 actually parallelizes while aggregation stays correct. Docs + roadmap updated (step 1 of auto-context: done). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan --trigger behavioral column + record empty-vs-seeded sensitivity (a) scan --trigger — the model-gated behavioral column that stacks on the structural scan (not a separate command). For each model-invocable skill in a plugin it measures recall (+ precision) via measureTriggerRate and prints it as a "Behavioral (trigger-rate)" section. Author-supplied prompts (a JSON map of skill → { prompts, irrelevant }), never model-generated. Degrades honestly to "unavailable" with no claude/auth; a thin prompt set is surfaced per-skill, not a crash. Flags: --prompts= --concurrency= --model= --min-prompts=. New src/scan-behavioral.ts (probePluginTriggers[With] + formatBehavioralReport), keeps scanPlugin pure/sync; spawnAgent exported from eval.ts so the real runner is shared. Unit tests with a fake runner; verified live against superpowers (brainstorming recall 20%, rest unmeasured). (b) Sensitivity recorded — empty 0.00 vs seeded 0.20 (10 runs each, via the new fixture + concurrency). Inconclusive at n=10: empty itself spans 0.0/0.2/0.3 across three samples, so the +20pt delta is within the noise floor. The real lesson — n=10 can't resolve a 20pt effect — is documented. Docs: scan --trigger + marketplace + the -8 dangling-ref weight in cli.md; cross-link from harness-testing.md; sensitivity result in plugin-behavioral-findings.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: behavioral findings 2 & 3 — superpowers recall sweep + haretrail scan Finding 2 — superpowers recall sweep (vigiles scan --trigger across 6 opening-move skills, 10 prompts each): concrete imperative-task skills fire reliably (test-driven-development 100%, writing-skills 100%) while exploratory / orchestration skills barely fire (writing-plans 40%, brainstorming 10%, systematic-debugging 10%, dispatching-parallel-agents 0%). State-dependent skills left unmeasured on purpose (cold prompt = artifact). The 0-10% vs 100% gap on structurally-green skills is the finding. Finding 3 — fleytman/haretrail (Codex/AGENTS.md, 8 skills): structurally clean (no agents/hooks/broken refs). Description-level risks worth a trigger probe: Russian descriptions (cross-language trigger risk on English prompts), a literal {data-repo} placeholder, and overlapping domains (debrief/lessons/postmortem, task/research) = precision risk; good practice in the explicit trigger-phrase lists + postmortem's negative cue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan flags non-Latin skill descriptions as a cross-language trigger risk Operationalizes the haretrail catch as a deterministic, no-model check: each skill description's dominant script is detected via Unicode \p{Script=…} (no dependency), and a description ≥20% non-Latin (Cyrillic/CJK/Arabic/…) is flagged "cross-language trigger risk" — the selector is English-centric, so it may under-fire on English prompts. A RISK flag, not a structural defect (a language-matched audience is fine), reported separately from the verdict and pointing at the model-gated `scan --trigger` to measure the real gap. ScanSkill.descriptionScript carries the detected script; formatScanReport notes it per-skill + a summary line. Latin/English descriptions (and mostly-English with a stray accented word like "résumé") are not flagged. Verified on the real fleytman/haretrail (all 8 Cyrillic skills tripped) and superpowers (0 flagged). Also extracts skillLine() (clears a pre-existing nested-ternary warning). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — one shared detector for lint+scan; behavioral column is the exception Records the convention this session arrived at: a deterministic structural check has ONE home (a single pure detector reused by both `lint` and `scan`, never reimplemented in each, so they can't drift) — untested-surface, dangling-ref, and non-Latin-description detection already work this way. The model-gated behavioral column (scan --trigger / trigger-rate, observed egress) is the deliberate exception: it's LLM-based (auth, tokens, non-deterministic), so it lives only in scan/eval and is NEVER a lint rule. Cheap deterministic detection can be a lint rule; expensive model-gated confirmation cannot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: description-script check is expected-relative (default Latin) + Script union Per review: Latin isn't special, it's a configurable DEFAULT. unexpectedScript(text, expected="Latin") now flags the dominant script only when it DIFFERS from the expected one — so a Cyrillic-targeted pack sets expected="Cyrillic", its Russian descriptions pass, and an English one is flagged instead. Adds a `Script` string union (ts-essentials: literal union over bare string) used by the detector, the SCRIPTS table, and ScanSkill.descriptionScript. unexpectedScript is exported as the single shared detector the future lint rule will reuse (one-detector-no-drift). Reworded scan output + the CLAUDE.md rule from "non-Latin" to "unexpected script". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: eval-tier trace-parser seam (increment 1 of native Codex eval) + docs Native Codex eval (measureTriggerRate/runEval for Codex) needs three things; this lands the env-independent foundation and documents the rest. - Increment 1 (shipped): makeContext's Claude stream-json parsing is extracted as parseClaudeRun and injected as a ModelOutputParser (default = Claude, zero behavior change). measureTriggerRateWith takes one as a 3rd arg, threaded via TriggerRunConfig.parse. A unit test drives a non-Claude line protocol + a custom parser end-to-end and detects firing — the exact slot a Codex `codex exec --json` parser plugs into. - Spike recorded (no live binary here): the Codex driver's parseCodexRun returns toolCalls:[] by design, so the eval tier can't yet detect a Codex tool/skill call; and whether a Codex skill activation is even a discrete `codex exec --json` event is the open question only the binary can settle. Docs (internal + public): research/codex-prototype-findings.md gets the spike + increment plan + env-validation checklist; CLAUDE.md (eval.ts entry) notes the parser seam; docs/harness-testing-codex.md and docs/harnesses.md mark Codex runEval/trigger-rate as in-progress (not over-promised) with the three remaining steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scaffold Codex eval runner + tolerant trace parser (increment 2) src/adapters/codex/eval.ts (exported on vigiles/codex): - parseCodexEvalRun(out): ParsedModelRun — a TOLERANT, multi-shape parser of `codex exec --json` JSONL. No codex binary in the build env, so the line schema is unverified: it probes BOTH plausible Codex event models (older {msg:{type,…}} stream, newer {type:"item.*", item:{…}} thread/item stream) and degrades to empty / output=trimmed-stdout on anything unrecognised. Each extractor (assistantText / toolCall / usage) is isolated so finishing against captured JSONL is a field-name edit. Returns the Claude-shaped ParsedModelRun so it slots straight into the increment-1 ModelOutputParser seam. - codexEvalRunner — spawns real `codex exec --json` (v8-ignored; flags + skill-install wiring best-guess pending the binary). Unit-tested over synthetic fixtures for both shapes + junk tolerance + the output contract (src/adapters/codex/eval.test.ts). Deliberately NOT wired into the public measureTriggerRate({adapter}) dispatch (increment 3) — gated until the schema and the skill-selection question are confirmed against the live binary, so an unvalidated parser can't silently report recall 0. Findings doc + checklist updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: how to authenticate Codex for the eval tier (device-auth for headless) The deterministic tier is keyless; only the eval tier (real model) needs Codex auth. Documents the three flows in order of fit: `codex login --device-auth` (URL + one-time code, the only one that works in a remote/sandbox box — no localhost callback to forward), plain `codex login` (browser, local dev only), and `printenv OPENAI_API_KEY | codex login --with-api-key`. Notes the network- egress requirement, the `npm i -g @openai/codex` install (validated 0.141.0), and the version-sensitivity of the `codex exec --json` schema. Ticks the install/auth box on the env-validation checklist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: confirm Codex eval schema against the live binary + the skill-firing finding Authed real codex (ChatGPT) and captured real `codex exec --json` turns, turning increment 2 from a tolerant guess into a confirmed, live-validated parser. Findings (now documented internal + public): - Schema CONFIRMED (codex-cli 0.139.0): a thread/item JSONL stream — assistant = item.completed/agent_message→text; tool = command_execution→command (+ aggregated_output/exit_code); usage on turn.completed (incl. cached_input_tokens → cacheReadTokens). Count item.completed ONLY (item.started repeats the id). - THE skill question ANSWERED: Codex has NO discrete skill-selection event (no Skill tool). A trigger surfaces as the model READING skills/<name>/SKILL.md via a command_execution — so codexSkillFired() detects that read (best-effort; pair with a judged check). Validated live: a marker skill fired and was detected end-to-end through codexEvalRunner → parseCodexEvalRun → codexSkillFired. - Version regression: pin codex 0.139.0 (0.141.0 broke the keyless mock recipe). - Spawn gotcha: stdin must be /dev/null or codex blocks on stdin. parseCodexEvalRun rewritten to the confirmed schema; fixtures replaced with REAL captures; codexSkillFired added. Docs updated: harness-testing-codex.md (schema + skill-firing sections), harnesses.md footnote, CLAUDE.md (eval.ts clause + new Key Files entries), codex-prototype-findings.md (2026-06-19 live-capture update + checklist ticked). Remaining: increment 3 — {adapter} dispatch into measureTriggerRate/runEval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim public Codex eval docs to high-level (full findings stay internal) The previous commit over-shared internal detail into the public docs (the full codex exec --json JSONL schema, item.started/completed dedup, the version regression specifics, the stdin gotcha). Those belong in research/codex-prototype-findings.md (where they remain). Public docs (harness-testing-codex.md, harnesses.md) now keep only the user-relevant level: runEval-for-Codex is in progress, Codex has no skill-selected event (firing detected via the SKILL.md read), pin 0.139.0 — and link out for the schema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: rule — match doc detail to audience (internal full record vs public link-out) Adds the public-vs-internal-docs guidance rule: research/*.md + CLAUDE.md hold the full record (schemas, parse mechanics, version regressions, spike notes, checklists); public docs (docs/, README) state only what a user needs to act + link out for the rest. Don't paste schemas/internal mechanics/version quirks into a user-facing guide unless a user needs them to use the feature. Codifies the trim just made to the Codex eval docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: haretrail native-Codex behavioral probe — cross-language risk refuted First native-Codex behavioral eval (codexEvalRunner → parseCodexEvalRun → codexSkillFired) on the real fleytman/haretrail skills, RU vs EN prompts: research 3/3 RU & 3/3 EN, summary 3/3 & 3/3, debrief 3/3 & 2/3. The Russian descriptions fire reliably on English prompts too — the deterministic "non-Latin description → cross-language risk" flag is a correct RISK flag that the model-gated measurement CLEARS here, so no "switch to English" fix is warranted. The single debrief/en miss confirms codexSkillFired discriminates. Logged in research (internal) per the audience-detail rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: codexRunError — an errored/rate-limited Codex turn isn't a "skill didn't fire" Dogfooding the haretrail precision probe hit a Codex usage limit, and every run came back all-miss: parseCodexEvalRun left the errored turn an empty trace, so codexSkillFired read it as a clean recall-0 miss — a rate-limit silently corrupting trigger-rate with false negatives. codexRunError(out) now detects the `error` / `turn.failed` event (incl. usage limits) so a caller skips/retries instead of scoring it (the Codex analog of the Claude path's isRateLimited backoff). Increment 3's dispatch must consult it. Test uses the real usage-limit capture. Precision/overlap question remains unmeasured (retry after quota resets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: native-Codex trigger-rate dispatch via { evalDriver } (increment 3) Wire the eval-tier harness dispatch so measureTriggerRate (and the scan --trigger behavioral column) can route through a non-Claude harness, with Codex as the first second-harness driver. - eval.ts: EvalDriver { runner, parse, runError? } bundles a harness's AgentRunner + ModelOutputParser + an optional errored-run detector; claudeEvalDriver (default) + measureTriggerRate({ evalDriver }) select one. A trial whose runError fires is excluded from the denominator (an errored/rate-limited turn isn't a clean miss) and counted as report.errored. - adapters/codex/eval.ts: codexEvalDriver (codexEvalAgentRunner + parseCodexEvalRun + codexRunError) + installCodexSkills (the Codex analog of --plugin-dir: copies skills into <cwd>/.codex/skills). - scan-behavioral.ts: a HarnessProbe (buildProbe(dir, harness)) selects the driver + per-harness "fired" predicate (skillResolved/stubbed for Claude, codexSkillFired for Codex), so scan --trigger --harness=codex routes through the Codex driver. - cli.ts: scan --trigger passes the detected harness through. Built + fake-tested (injected driver, no binary); a live native eval run remains, gated on Codex quota. Docs + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: live-validate the native-Codex trigger dispatch (numbers still quota-blocked) Re-ran the haretrail debrief/lessons/postmortem precision probe through the shipped public path (`vigiles scan --trigger --harness=codex`) now that the `{ evalDriver }` dispatch is wired. The Codex usage limit is still capped, so the recall/precision NUMBERS remain unmeasured — but the run live-validated increment 3's plumbing against the real binary: - the CLI dispatched through the Codex EvalDriver and spawned real `codex exec --json`; - `codexRunError` correctly EXCLUDED every errored trial (report showed `recall 0% (0 runs)` = n=0, not scored as misses) — the errored-run exclusion proven against reality, not just fakes; - the diversity gate fired live (rejected a too-similar irrelevant pair). So the only remaining gap is non-errored trials for actual numbers — a quota wall, not a code gap. Saves the eval-ready cluster prompt set at research/fixtures/haretrail-cluster.json for the retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: command-only / MCP-only plugins are a real surface, not score 0 scoreReport treated any plugin with no skills/agents/hooks as "no loadable plugin surface" (score 0/F) — but a command-only plugin (commands/*.md with a valid plugin.json) or an MCP-only plugin (.mcp.json) is legitimate. Anthropic itself ships command-only plugins (code-review, commit-commands) in its marketplace, and the leaderboard graded both 0/F. Count commands + MCP toward the loadable-surface check so they score on their real structural health (a clean command-only plugin → 100/A); a truly empty directory still scores 0. Found by scanning anthropics/claude-code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: close the two dogfood untested-surface gaps + fix a migrate-to-spec drift vigiles lint flagged two untested surfaces in our OWN plugin. Closing them also surfaced a real skill bug: - hooks/refs-nudge.sh: the sibling refs-hook.test.ts drives the CLI directly, so the SHELL WRAPPER (its not-a-node-project guard + exit-2-only propagation) was untested. Add a runHook unit test that drives the real script, pointing npx at a local node_modules/.bin/vigiles shim so it resolves offline. - skills/migrate-to-spec: a user-invoked prose skill (model-driven, nothing deterministic to test), but it SHOWS agents an exact `import { … } from "vigiles/spec"` template. The template imported `check` and `every`, which vigiles/spec does NOT export — following the skill produced specs that fail to compile. Fixed the template and added a dogfood test asserting every taught import is a real vigiles/spec export (catches skill ⇄ API drift; names the surface so it counts as covered). `vigiles lint` now reports all 9 surfaces tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface `scan` (audit any plugin) in README + Codex/command-only in cli.md - README: add a high-signal "Audit any plugin" bullet — the zero-setup, no-key scan/leaderboard is the strongest adoption hook and wasn't mentioned at all. Kept to one bullet per readme-brevity (links to the cli.md depth). - cli.md: note that command-only / MCP-only plugins score as real surfaces (not 0), and that `scan --trigger --harness=codex` routes the trigger probe through the native Codex driver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: scan reports the instruction file on any cc/codex repo + lint-vs-scan docs scan was plugin-oriented — on a plain repo (just a CLAUDE.md/AGENTS.md, no skills/agents/hooks) it printed an empty inventory and didn't even mention the instruction file, so it looked broken on the most common repo shape. - scan: report the detected instruction file (CLAUDE.md/AGENTS.md) with spec-managed vs hand-written, so it's useful on ANY cc/codex repo. This is a read-only informational fact, NOT the require-spec gate (that stays lint's). - docs/cli.md: add a "Lint vs scan — gate vs report" section with the full capability + target matrices (the answer to "do we need both / does it work on normal & Codex repos"). lint = config-driven gate (refs, 7 catalogs, exit codes); scan = zero-config read-only report (any plugin, leaderboard, + the paid --trigger column). They share one detector for the few common structural checks; everything else is intentionally asymmetric. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: e2e scan over the cc/codex/mixed/marketplace repo matrix Drive the real built CLI (node dist/cli.js scan) across every repo shape, in the free unit tier (deterministic, model-free, offline — like vendor.test.ts). Two fixture sources by design: - DOGFOOD pinned OSS plugins (examples/harness/vendor/*) for the Claude Code surface — real shapes (oh-my-claudecode multi-surface + MCP; the wshobson-accessibility inherits-all agent). - ARTIFICIAL tmp fixtures for shapes we have no vendored example of: a Codex repo (AGENTS.md + TOML [mcp_servers] + skill), a mixed CLAUDE.md+AGENTS.md repo (detection-ambiguity warning + --harness override), a CC instruction-only repo (the new instruction-file line), and a marketplace (leaderboard expansion). Covers the CLI scan WIRING the library-level scan.test.ts can't reach: harness auto-detection, the ambiguity warning, --harness=, the leaderboard branch, --json, and the read-only exit-0 contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan honest on curated marketplaces + dedupe name-aliased plugin dirs Two real bugs surfaced by sweeping ~13 popular plugin marketplaces (~650 plugins) + a variant mix of cc/codex/both repos: 1. A CURATED marketplace — every member an external git/url plugin, nothing vendored in-tree (obra/superpowers-marketplace, anthropics/claude-plugins- community with 2200 entries) — scanned as "nothing was loaded / no structural issues found", indistinguishable from a broken empty repo. Now inspectMarketplace classifies members on-disk vs external and scan reports `Marketplace "X": N plugin(s), all external (…not on disk)` honestly. 2. A marketplace that maps many NAMES onto the same dir (TheBushidoCollective/ han: 338 names → 159 dirs) made the leaderboard double-scan and rank the same plugin repeatedly ("(338 scanned)" with duplicate rows). expandMarket- place now dedupes by resolved path → "(159 scanned)", no dup rows. inspectMarketplace is the new source of truth (MarketplaceInfo: name/onDisk/ external/total); expandMarketplace delegates to it. The sweep also confirmed robustness: no crash across all marketplaces, and correct harness detection + instruction-file reporting across claude-only / codex-only / both-files repos (openai/codex, sst/opencode, wshobson, obra, …). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: eliminate scanner false positives found by sweeping real plugins Sweeping ~13 marketplaces + a variant mix of cc/codex/both repos surfaced that the scanner's "bugs" were mostly FALSE POSITIVES — a scanner that cries wolf on working plugins can't drive adoption. Fixed four classes (all verified against the real plugin that exposed them): 1. Multi-line QUOTED description (`description:\n "…"`, trailofbits/react-pdf) was mislabeled "no description / can't trigger". readField now gathers an indented quoted scalar, not just block scalars. 2. Relative hook path (`./hooks/x.sh`, ananddtyagi/cc-marketplace) was reported MISSING because existsSync was cwd-relative. resolveScript now resolves a relative hook against the PLUGIN ROOT. 3. Existence-guarded hook (`[ ! -f x ] || x`, gmickel/flow-next) — an optional run-if-present — was flagged MISSING. Now treated as a conditional one-liner. 4. A project/runtime path in a script (`$CLAUDE_PROJECT_DIR/.claude/hooks/…`, flow-next) matched mid-path as a plugin ref. danglingRefs/isPluginRooted now only flags PLUGIN-rooted paths (bare or ${PLUGIN_ROOT}/…), skipping literal nested dirs and project/home vars; a genuine plugin-rooted miss still flags. After hardening, the sweep's honest tally over 444 unique plugins: ONE real bug (ananddtyagi/cc-marketplace's claude-dev-infrastructure ships crisis-debugging- advisor + meta-skill-router with NO frontmatter → can't be model-triggered), and the popular plugins are otherwise structurally clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record the structural sweep + public-disclosures / FPs-fixed list New research/plugin-structural-findings.md (sibling of plugin-behavioral- findings.md) logs the deterministic scan sweep of ~13 marketplaces / 444 unique plugins: - Public disclosures: the one genuine external-plugin bug found — ananddtyagi/cc-marketplace's claude-dev-infrastructure ships two skills with no YAML frontmatter (can't be model-triggered). Status: found, not yet reported upstream + disclosure etiquette note. - Scanner false positives found by the sweep and fixed (the dogfood value): command-only 0/F, curated-marketplace "empty", han dedup, multi-line quoted description, relative ./hooks resolution, existence-guarded hook, project- rooted dangling ref — each with the plugin that exposed it + the fix commit. Linked from plugin-behavioral-findings.md and distribution-strategy.md (E1) so it's discoverable and not an orphan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: tool-contract validation — the cross-referencing moat in scan + lint The deterministic scanner described tool contracts but never VERIFIED them. Add the "valid is not true" moat to a subagent's `tools:` rail, as ONE shared detector (src/core/tool-contract.ts) reused by three callers (one-detector-no- drift): compileAgent (strict), scan (read-only), and a new agent-tool-contract lint rule (severity-gated). Calibrated HIGH-PRECISION after sweeping real plugins (the cry-wolf lesson): - flags only never-available tools (curated denylist) + close typos (edit distance ≤2, with did-you-mean) — the two confident cases. - a bare unrecognized tool is NOT flagged: a 280★ plugin uses TaskCreate/TaskGet/… consistently (plugin/MCP-provided), and our catalog can't know every tool — flagging those would be a false-positive flood. Two real bugs this surfaced and fixed along the way: - parseAgentTools didn't handle the inline ARRAY form (`tools: [Read, "Bash"]`) → it returned `[Read` / `Bash]`, which BOTH broke the PreToolUse rail (it would block Read) and tripped the check. Fixed at the source. - builtinAgentTools was incomplete (missing LS/MultiEdit/BashOutput/KillBash) — a latent false "unknown tool" error in compileAgent itself. Catalog fixed. Sweep result after calibration: 5 real never-available dead-contract entries across 444 plugins (wshobson team-lead `Agent`, developer-kit/ananddtyagi `AskUserQuestion`/`ExitPlanMode`), no flood. Leaderboard penalizes a dead tool like a broken ref. New rule defaults to warn (no-op on repos without agents/); doc + README + spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: hook-event validation (#2) — typo'd hook events in scan + lint Cross-reference each hook's event name against the harness event catalog: a hook registered under a name the platform doesn't define silently never fires. Shared detector (src/core/hook-events.ts) → scan + the hook-events lint rule. Caught/fixed two things before it could ship: - dialect.hookEvents was WRONG and unused (PreSession/PostSession don't exist; SessionStart/SessionEnd/Stop/SubagentStop/UserPromptSubmit/PreCompact were missing). Corrected against the events real plugins register. - Validating ALL unknown events flooded FPs: TheBushidoCollective/han ships a custom hook runtime (TeammateIdle, WorktreeRemove, …) — the event set is NOT closed. And ananddtyagi/sugar ships hooks as an ARRAY of {event:"tool-use"} objects (a non-CC format → Object.keys gave indices). So: high-precision (close typos only, via confidentHookEventIssues) + object-keyed shape only. After calibration: 0 on 444 swept plugins (FP-safe), catches PreToolUSe→ PreToolUse at authoring. Default warn (no-op on this repo); doc + README + spec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: frontmatter-schema validation (#3) — missing name/description in scan + lint A SKILL.md needs a `name` to load (CC keys a skill on its frontmatter name); a subagent needs `name` + `description` to register. A missing field is a structurally broken surface that silently never works. Computed in scanPlugin (frontmatterIssuesFor) and consumed by scan + the frontmatter-schema lint rule. Verified real on the sweep: 12 issues across 444 plugins, mostly ananddtyagi/ cc-marketplace — the headline bug (crisis-debugging-advisor / meta-skill-router ship with no frontmatter) is now caught precisely as "missing name", plus subagents (changelog-generator, …) that are pure prose with no frontmatter → never register. Confirmed by hand (changelog-generator.md opens with prose, no --- block). FP-safe (name is required; skill description stays on the skill line to avoid double-counting). Default warn; doc + README + spec + tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: MCP-config validation (#4) — unstartable MCP servers in scan + lint A declared MCP server must say how to reach it: a `command` (stdio) or a `url` (http/sse). An entry with neither can't start, and its tools silently go missing. Shared detector (src/core/mcp-config.ts) → scan + the mcp-config lint rule. FP-safe by construction: command-or-url is an unambiguous requirement, not a catalog that could be incomplete. Reads .mcp.json + manifest mcpServers (JSON CC shape); Codex TOML [mcp_servers] is a documented gap. Found 0 on the 444 swept plugins (they all configure command/url correctly) — survivor bias, as expected; the value is catching a malformed server at authoring time. Default warn; doc + README + spec + detector/scan tests. This completes the deterministic cross-referencing moat across the four agent surfaces: tool contracts (#1), hook events (#2), frontmatter schema (#3), and MCP config (#4) — each a shared detector reused by scan + lint (+ compileAgent for tools), all calibrated high-precision against the real-plugin sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: tool-contract treats `tools: *` as a wildcard (inherits all), not unknown A subagent declaring `tools: "*"` (all tools) was flagged `unknown "*" — did you mean "LS"?` (editDistance("*","LS")=2). A wildcard isn't a tool to validate. Skip "*" like the empty contract. Found on ananddtyagi/cc-marketplace's problem-solver-specialist. One more cry-wolf case closed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: frontmatter-schema is subagents-only — skills need no frontmatter (correction) Verified against the Claude Code docs (prompted by skepticism that "those skills never fire"): a SKILL.md requires NO frontmatter — `name` falls back to the directory name, `description` to the first body paragraph — so a frontmatter-less skill STILL loads and can fire. My earlier claim that ananddtyagi's crisis-debugging-advisor / meta-skill-router "never trigger" was WRONG. A subagent, by contrast, REQUIRES `name` + `description` with no fallback, or it won't register — confirmed by docs/en/sub-agents. So: - frontmatterIssuesFor now checks AGENTS ONLY (skill checks were false positives). - Corrected the disclosures doc: the real ananddtyagi bug is ~9 prose-only SUBAGENTS that never register (changelog-generator, content-creator, …), NOT the skills. Added a loud correction note. - Updated the rule doc (skills explicitly not checked + why), cli.md row, spec. Re-tally after correction: 9 agent-frontmatter bugs in ananddtyagi (real, docs-confirmed), the 2 skills + han's name-less skill correctly drop off. The "verify before you claim" discipline catching my own overclaim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: empirically confirm the ananddtyagi agent finding (real claude CLI) Cloned ananddtyagi/cc-marketplace and drove the real claude CLI to verify the prose-only subagents really are inert (not just doc-inferred): listing subagents omits the prose-only one, dispatching it returns NO_SUCH_AGENT, while a control agent with frontmatter runs. Mirror test confirms a no-frontmatter SKILL still loads (dir-name fallback). Both the agent bug and the earlier skill correction now have ground-truth backing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: save superpowers trigger-probe fixture + session state before probe Persists (compaction-safe): the recall-probe prompt set (research/fixtures/superpowers-trigger-probe.json, 3 skills x 10 prompts) and a durable session-state note in plugin-behavioral-findings.md — the 4-rule moat shipped this session + the probe plan (results to be filled in after the run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: superpowers trigger probe results — systematic-debugging fires 10% Ran the recall probe on the SHIPPED scan --trigger path (claude 2.1.183, 10 prompts/skill on the subscription). Control test-driven-development fires 100% (validates the measurement); systematic-debugging fires only 10% and brainstorming 50%. A structurally-green skill silently never triggering on its own use case — the behavioral bug the deterministic column structurally can't see. Reconfirms the method-vs-situation trigger pattern on the shipped path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: skill-frontmatter rule — recommend explicit skill name+description (warn) Skills require NO frontmatter (name←dir, description←first paragraph), so a missing one isn't a load failure — but relying on fallbacks is fragile (the first paragraph is often a heading → a weak, accidental trigger surface). Add a best-practice NUDGE: scanPlugin.skillMetaIssues (a soft ℹ note, never counted as a structural defect or scored) + the `skill-frontmatter` lint rule. This is the correct form of the earlier "two rules" idea, split by severity/ intent: `frontmatter-schema` = subagents (REQUIRED → won't register, correctness) vs `skill-frontmatter` = skills (RECOMMENDED → reliable trigger surface, quality). Default warn (the skill works without it); set "error" to enforce on your own skills. vigiles's own skills pass. Doc + README + spec + tests. Note (follow-up): the pre-existing scan `hasDescription` "can't trigger" line is the same overclaim class — a skill with no frontmatter description still triggers via the first-paragraph fallback. Left as-is for now (pre-existing, load-bearing on the leaderboard); worth softening separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename frontmatter-schema rule → agent-frontmatter (symmetric pair) With two frontmatter rules now, `frontmatter-schema` (agents) vs `skill-frontmatter` (skills) was lopsided — one generic, one specific. Rename to `agent-frontmatter` so the pair is parallel and self-describing: agent-frontmatter (required → won't register) + skill-frontmatter (recommended → trigger surface). Unreleased on this branch, so no back-compat cost. Config key + doc (git mv to docs/rules/agent-frontmatter.md) + cli.md table + spec + research refs all updated; internal symbol names left as-is (not user-facing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: soften skill 'can't trigger' overclaim — honor the first-paragraph fallback The trigger-surface check flagged any skill without a FRONTMATTER description as 'can't trigger' — but Claude Code falls back to the first body paragraph, so a skill with prose still triggers. Now hasDescription = frontmatter description OR first body paragraph (firstBodyParagraph); only a skill with NEITHER is flagged. The last instance of the frontmatter overclaim class. The explicit-metadata best-practice stays the separate skill-frontmatter rule. (nodesc — name only, body just a heading — is still correctly flagged.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * research: ranked next-deterministic-rule ideas, grounded in the OSS sweep New research/deterministic-rule-ideas.md — the curated backlog of next moat surfaces, each the same 'valid is not true' cross-reference on a new surface, high-precision by design (the session's recurring FP lesson). Tier 1: mcp-tool-resolves (an mcp__server__tool whose server isn't declared), hook-matcher (typo'd tool in a hook matcher), frontmatter-valid (malformed YAML). Tier 2: duplicate-names, hook-shape, and the showpiece description-overlap — a DETERMINISTIC precision-collision proxy reusing the NCD engine (catches a --trigger-class bug with no model), grounded in haretrail's debrief/lessons/ postmortem overlap. Linked from roadmap Now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-tool-resolves rule (the MCP half of the tool moat) Cross-reference an `mcp__server__tool` in a subagent's contract against the plugin's declared `mcpServers` — a server the plugin doesn't declare can't resolve. Completes the tool moat: agent-tool-contract verifies built-in tools but passes any mcp__* token unchecked; this verifies the MCP half. One pure detector (src/core/mcp-tool.ts: verifyMcpToolServers) reused by scan + the mcp-tool-resolves lint rule (one-detector-no-drift). Calibrated high-precision via three guards, each grounded in a real plugin from the mid-2026 sweep: - GATE on a declared set — no .mcp.json → reaches global/project servers, flag nothing (ananddtyagi's mcp__ide__* shape). - ALLOWLIST harness built-ins via a new optional dialect.knownMcpServers (Claude Code's `ide`); optional/additive, non-breaking for adapters. - SKIP the plugin-namespaced mcp__plugin_<plugin>_<server>__ form (han's playwright-mcp — ambiguous join, the plugin's own server). Validated FP-free across the whole sweep (0 hits) and fires correctly on a synthetic undeclared-server plugin. Wired into scan (per-agent mcpToolIssues), lint (warn default), and the leaderboard score. Doc + detector tests + scan tests + CLAUDE.md/cli.md + regenerated API reports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: competitor rule matrix + verified OSS bug PR drafts Save the 3-agent competitor landscape sweep (eval tools, plugin/skill validators, MCP/security scanners) as research/competitor-rule-matrix.md: the rule-by-rule matrix vs agnix / claudelint / cclint / Anthropic's own `claude plugin validate`, the positioning finding (structural rules are converging to table stakes; the moat is semantic cross-reference + harness testing), and the ranked poach backlog. Add research/oss-pr-drafts.md with ready-to-file PR/issue text for two hand-verified third-party bugs the sweep found: subagents listing AskUserQuestion/ExitPlanMode (silently dropped — Anthropic #12890/#18721) and ananddtyagi's prose-only/blank-description subagents (won't register). Record the verified non-bugs (test fixtures, codex source tree) so they aren't re-chased. Cross-linked from plugin-structural-findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add hook-script-exists lint rule Promote scan's missing-hook detection to a lint rule: a hook command referencing a script file that doesn't exist on disk (with ${CLAUDE_PLUGIN_ROOT} resolved) silently never runs. Reuses scanPlugin's hooks status (one-detector-no-drift) — FP-safe by construction (skips unresolved $VARs, existence-guarded one-liners, inline commands). Matches Anthropic's own `claude plugin validate`, making vigiles a superset on the hook surface (poach #1 from the competitor matrix). Wired into lint (warn default) + exit code; doc + types/validate + cli.md table + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add disallowed-tools-contract lint rule (deny-side tool moat) Cross-reference a subagent's `disallowedTools:` block-list against the catalog — the mirror of agent-tool-contract. A close typo there blocks NOTHING (you meant to deny `Bash`, wrote `Bsh`), so the tool stays available silently — worse than no deny-list. New detector disallowedToolIssues in tool-contract.ts, calibrated high-precision: only a close typo of a real built-in flags; a real tool is legitimately blocked, a never-available tool is harmless to list, a bare unknown is likely a plugin tool (no cry-wolf). A check no other validator does (poach from SkillCheck). Generalizes parseAgentTools → parseAgentToolList(md, key) for the disallowedTools parse. Wired into scan (per-agent disallowedToolIssues) + lint (warn) + exit code + leaderboard; doc + detector/scan tests + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: agent-frontmatter also flags invalid model/color values Extend the agent-frontmatter rule from "missing required field" to also catch an invalid VALUE: a model: that's a close typo of an alias (inherit/sonnet/opus/ haiku) silently falls back; a color: typo is ignored. High-precision — close-typo only (≤2 edits), a full/dated model id (claude-sonnet-4-5) is an explicit form left alone, a far-off unknown is suppressed (no cry-wolf). Matches Anthropic's own `claude plugin validate` + cclint (poach #3). New frontmatterValueIssuesFor detector + frontmatterValueIssues on ScanReport, folded into the same agent-frontmatter lint rule + scan Frontmatter section + leaderboard; scan test; agent-frontmatter doc + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add description-overlap lint rule (deterministic precision proxy) The showpiece: two model-invocable skills with near-identical descriptions can't be told apart by the selector, so the wrong one fires (a precision collision). findDescriptionOverlaps (core/description-overlap.ts) reuses proofs.ts ncd to catch a --trigger-class behavioral bug with NO model — bridging the deterministic and behavioral columns, a check no other plugin linter has. Calibrated against the sweep: across 4678 within-plugin description pairs the most-similar legitimately-distinct pair (create-issue/create-pr) sits at NCD 0.25, nothing below it — so the cutoff 0.2 fires only on basically-identical text (FP-free on the whole sweep). User-invoked skills excluded (picked by command, can't collide). Wired into scan (descriptionOverlaps, reported as a precision risk) + lint (warn) + exit code; detector + scan tests; doc + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: mark second-wave rules shipped in the backlog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scan): section headers count entities, not output lines The "Agents (N)" / "Hooks (N)" headers used the rendered line count, so an agent's indented issue lines (and the inline-hooks summary line) inflated the number — "Agents (4)" for one agent with three findings. section() now takes an explicit count; Agents passes agents.length, Hooks passes hooks + inline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: one lenient frontmatter reader (real YAML + regex salvage) Replace the two divergent hand-parsers (scan's readField, agent-runtime's regex) with a single shared reader, core/frontmatter-read.ts: try js-yaml (block scalars, quoted multi-line, flow arrays parse correctly for free), fall back to a regex salvage on malformed input and record malformed:true. readFrontmatter never throws — it audits arbitrary third-party files — and frontmatterScalar / frontmatterList preserve the exact semantics the PreToolUse rail depends on (absent → null = inherits all, present-empty → [] = no tools). Anchored at file start (+ optional BOM + optional leading vigiles integrity comment) so a body --- hr is never read as frontmatter. Behavior-preserving (agent-runtime/scan/vendor/dogfood suites green); js-yaml was already a dependency. core/frontmatter.ts (the Level-1 vigiles: block) is a different concern, untouched. Sets up frontmatter-valid (the malformed flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add frontmatter-valid rule (malformed-YAML frontmatter) The payoff of the lenient-reader refactor: readFrontmatter's `malformed` flag becomes a rule — a skill/agent `---` block that exists but isn't valid YAML, so fields may not parse as intended. Calibrated and shipped CONSERVATIVELY per the verify-before-claiming discipline: a real YAML parser is stricter than some loaders, so a one-line description: with a `: ` colon / <example> block is flagged though it may still load. The sweep showed 7% of blocks flagged, concentrated in 2 repos (ananddtyagi, MadAppGang), 0 in the other 11 — a real authoring smell, but NOT empirically confirmed to fail loading. So: default WARN (not error), scan surfaces it as an informational ℹ note (not a structural defect, no leaderboard penalty), doc states the caveat and says verify before setting error. The unambiguous cases (unclosed bracket, tab, duplicate key) are real regardless. Same detector as scan (malformedFrontmatter); detector reuses the shared reader. Wired into lint + scan note; doc + scan test + types/validate + cli.md + CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: dogfood the scan rules against real vendored plugins (FP-guard + bug fixture) Lock the deterministic rules against reality, offline + model-free, alongside the existing loader dogfood. src/scan-vendor.test.ts asserts golden scan verdicts: - FP-GUARD: oh-my-claudecode / wshobson-accessibility / superpowers must stay CLEAN — every high-precision rule field zero (the regression that catches a rule going noisy; the don't-cry-wolf bet). - TRUE-POSITIVE: a minimal MIT slice vendored BECAUSE it reproduces a real bug — madappgang-frontend@6097ad4 tester.md (a single agent file) — must keep firing agent-tool-contract (AskUserQuestion never-available) AND frontmatter-valid (the one-line description: is invalid YAML). Only the structural file is vendored (not the repo), only from a permissive (MIT) upstream — ananddtyagi has no LICENSE, so its files are NOT committed (its bug stays in research/oss-pr-drafts.md). Provenance + licensing in examples/harness/vendor/SOURCES.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: make the OSS bug drafts filing-ready (recipe + worked diffs) Add a per-repo filing recipe (branch / mechanical edit / conventional commit / gh pr create) plus worked before→after diffs for both verified bugs — the comma-form and array-form AskUserQuestion removal, and the missing-frontmatter prepend. States plainly these can't be filed from a vigiles-scoped session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add PR #40 handoff (resume-here state + next-session prompt) Single consolidated handoff for the branch: what shipped (the deterministic rule suite, scanner hardening, the lenient-reader refactor, the vendor dogfood, the research + OSS PR drafts), the disciplines that must carry forward (verify-before-claim, high-precision, calibrate-against-the-sweep, one-detector-no-drift, check-license-before-vendoring), the ranked open follow-ups (file the OSS PRs, README positioning, the remaining backlog rules), and a paste-ready next-session prompt. Linked from deterministic-rule-ideas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add mcp-hook-target-resolves rule (the moat on the hook surface) Cross-reference a `type: "mcp_tool"` HOOK action against the plugin's declared MCP servers (verified real: CC hooks support command/http/mcp_tool/prompt/agent, and mcp_tool requires server+tool). Flags two silent-no-op cases: an incomplete action missing server/tool (unambiguous, always — like mcp-config), and a server the plugin doesn't declare (gated on a declared mcpServers set + built-ins like `ide` allowlisted, mirroring mcp-tool-resolves). New detector core/mcp-hook.ts (verifyMcpHookTargets), wired into scan (mcpHookIssues) + lint (warn) + exit code + leaderboard; detector/scan tests; doc + types/validate + cli.md + CLAUDE.md. FP-free across the sweep (0 mcp_tool hooks in the wild — advanced feature; value is authoring-time + moat completion). The regex matcher surface is left to a future hook-matcher rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(readme): lead with the cross-reference moat; surface plugin audit Per research/competitor-rule-matrix.md: the structural lints (hook-events, agent-frontmatter, mcp-config) are now table stakes — Anthropic's own `claude plugin validate` ships them — so don't headline them. Elevate what's actually the moat: cross-referencing a declared reference against LIVE tooling. Extend the Lint section to the plugin surface — `vigiles scan` checks a plugin's contracts (subagent tools, mcp__server__tool, mcp_tool hooks, hook events, script paths) actually exist AND resolve, "a superset of Anthropic's `claude plugin validate`" (valid YAML ≠ a tool that's real). Retarget the old "Audit any plugin" More bullet to the leaderboard + --trigger so it doesn't duplicate. Stays within the readme-brevity budget (150 lines). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
2393b155c6 |
fix: repair broken main — 100% coverage gate + CLAUDE.md integrity/format conflict (#38)
* fix: repair broken main — 100% coverage gate + CLAUDE.md integrity/format conflict PR #37 left main red on two CI jobs: - test job: the eval.ts ephemeral-env + interceptTools compose path (the `if (overlay) Object.assign(env, overlay)` line) was unexercised, dropping statement coverage to 99.79% and failing the 100% gate. Add a unit test that drives both features together and asserts the intercept overlay survives the env scrub. - check job: `vigiles compile` emits markdown like `skills/*` but prettier re-escapes it to `skills/\*`, desyncing CLAUDE.md's integrity hash. The compiled CLAUDE.md is a vigiles-owned exact-format artifact (hash-gated), so add it to .prettierignore alongside the other tool-generated files and recompile so the committed bytes match the spec. The spec stays formatted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update session state timestamps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: tighten README — drop "pillar", footnote the Latin, sharpen the sell - Move the "Quis custodiet ipsos custodes?" epigram to a footnote on the name; open with a real one-line pitch instead. - Drop the "pillar" jargon everywhere; lead with the two verbs (Lint / Test). - Lint hook now leads with a renamed file + script (the relatable case), not a disabled eslint rule. - Rename the Test heading "does your harness fire?" → "do its job?" and lay out the full scope as bullets (block / trigger w/ recall+precision / behaviour via evals / safety) — firing is only one of them; evals are now explicit. - State plainly that tests are deterministic (no model, no API key) and evals run on your Claude subscription, never metered API tokens, with links. - Cut ~45 lines of wall-of-text (Test section more than halved). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: make README a scannable front door + add promptfoo eval comparison Rewrite the readme-brevity rule (in CLAUDE.md.spec.ts → recompiled) to demand a WOW-on-first-screen, phone-scannable front door: short (~140 line target), no walls of text, great formatting, links out for detail, and a required promptfoo comparison in the evals section. Fix the README to match: add a compact 3-column cost table contrasting promptfoo/DeepEval (metered API SDK, per-token) against vigiles ($0 extra, runs on your Claude subscription, most tiers need no model at all). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move Quick start above the fold (hook → what → get-it-now → why) Place the slim Quick start right after the what-it-does table, before the Lint and Test deep dives, so a skimmer can try `npx vigiles init` in seconds; the deep-dive sections now read as the "why it's worth keeping". Encode the rule in readme-brevity (CLAUDE.md.spec.ts → recompiled): install path above the fold, verbose setup kept in a <details> so the top stays lean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add docs-quality rule + align lint/test guides with the new README - New `docs-quality` guidance rule (CLAUDE.md.spec.ts → recompiled): the docs/ reference docs are the HOW the README defers to — NOT held to README brevity (depth is their job), but to a doc-appropriate polish bar: open with a one-line "what this is" + an up-link to the README, stay scannable (headings, Contents, tables, runnable code), cross-link siblings + "See also", and stay consistent with the front door (terminology, the promptfoo cost contrast, no stale paths). - Polish the two linked guides accordingly: the testing guide gains the same "README has the pitch; this is the full guide" up-link + a sibling link to the lint guide; the lint guide drops the "Pillar 1" opening (consistent with the README dropping "pillar") and gains the sibling link to the testing guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: add npm repository/homepage/bugs/keywords/license metadata The package is live on npm (vigiles@5.0.0) but package.json lacked discovery metadata: no repository (so npmjs.com had no link back to GitHub), no homepage, bugs, keywords, or license, and a stale description (the old "compile .spec.ts" framing). Add them and update the description to the current "lint & test the harness" positioning so npm search and the npm↔GitHub link are correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scrub "pillar" from the docs and CLAUDE.md spec (use "layer") Following the README, drop the "pillar" jargon from the user-facing docs and the CLAUDE.md positioning/rules — "layer" throughout (the linting layer / the testing layer), matching the front-door voice. Recompiled CLAUDE.md; anchors and cross-links updated consistently. research/ left as-is (internal design vocab). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: regenerate generated.d.ts (sync npm scripts + project files) Regenerated after the package.json metadata change; also picks up pre-existing drift (api:report/api:check/docs:api scripts, vendor-coverage.test.ts) the committed file was missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: remove the deprecated demos (examples/demo + plugin-test-demo) The two demos were deprecated and no longer the surfaced front door. Delete them and every reference: the demo/demo:plugin npm scripts, the doc callout in the harness-testing guide, the examples/harness README block (now points at npx vigiles test/eval), and the .gitignore/.prettierignore lines. Regenerated generated.d.ts (scripts + project files). The roadmap item is updated to "build the ONE polished front-door demo" (the deletion is done; the polished demo + GIF remains as the real next step). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(roadmap): record Sogen evaluated-and-rejected as a sandbox backend Note in the rejected/parked section so it isn't relitigated: Sogen is a syscall-level Windows/Linux binary emulator for malware/DRM research (host isolation "might not be perfect") — wrong workload/guarantee/gap for confining Node+shell. The transferable idea (snapshot + deterministic replay) we already do at the filesystem/trace layer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
456991c974 |
feat!: harness testing/eval pillar + public-API surface revamp (#37)
* docs: add eval-architecture design doc + ranked roadmap
Capture the conceptual model for testing Claude Code harness features:
feature = deterministic TEST + behavioral EVAL; two orthogonal knobs
(reproducibility x gating); cost-matched mechanism; trigger-rate as a
classifier eval; hash-lockfile + evalApiVersion + dated-model; deferred
canary; closure-scoped invalidation. Reconciles the model with what the
codebase already ships (record/replay cache, significance-gated baseline,
check vocabulary, trigger-rate recall+precision) and ranks the genuine
remaining gaps (tool-call spy/fake keystone, negative checks, curl fake,
subprocess golden, hash-lockfile) mapped to the dogfood build order.
Link from research/eval-api-landscape.md and research/roadmap.md.
* docs: clarify constraint is no-SaaS, not zero-dep, in eval-architecture
Deps are acceptable when they require no hosted account; rest the case
against Polly.js/promptfoo on their real merits (cache exists, cassette
doesn't escape the snapshot trap, promptfoo punt is strategic) rather
than a dep-count rule.
* docs: drop reuse-vs-reinvent tangent from eval-architecture
* feat: add toolWith/notTool argument + safety checks to vigiles/check
The inspection half of the keystone tool-call spy (docs/eval-architecture.md
gap #1/#2): a serializable ArgMatcher (dot-path keys; RegExp = pattern,
primitive = exact) drives two new Trace checks —
- toolWith(name, args): the agent called `name` with input matching `args`
(asserts HOW a tool was called, e.g. the image-request body carried the
style suffix), and
- notTool(name, args?): the agent did NOT call `name` (optionally only the
args-matching calls, so "did not push to main" still allows pushing
elsewhere) — the safety/negative assertion a completion grader can't make.
Both flow through vigiles/testing + vigiles/unit via the existing
`export *`. Pure, model-free, fully unit-tested in src/check.test.ts.
* feat: tool-fake interception core + vigiles fake-tool-hook
The interception half of the keystone tool-call spy (docs/eval-architecture.md
gap #1). At the eval tier the model's tool decisions are real, but letting a
side-effecting tool actually run (paid image API, git push, paid subagent)
makes the eval expensive and risky. A FakeTool denies the real execution while
the model still emits the tool_use — so the agent's decision is observable via
toolWith/notTool, but nothing happens.
- src/arg-match.ts: extract the shared ArgMatcher (dot-path keys; RegExp =
pattern, primitive = exact) so the check vocabulary and the fake seam share
one matching semantics; src/check.ts now imports it.
- src/tool-fake.ts: FakeTool + decideFakeTool (first match wins, optional
`when` arg-scoping), buildFakeToolSettings (PreToolUse hook over the union of
faked tool names), and serializeFakeTools/parseFakeTools (env round-trip that
preserves RegExp matchers a naive JSON.stringify would drop). Pure, tested.
- src/cli.ts: `vigiles fake-tool-hook` — reads VIGILES_FAKE_TOOLS, denies a
matched call via exit 2 feeding the canned result to the model (mirrors the
agent-contract rail's block mechanism).
Remaining: the ergonomic `fakeTools:` field on the eval spec (auto-merge the
hook settings + a cache-key entry) to wire it end-to-end from measure/runEval.
* feat: wire fakeTools onto the eval spec (end-to-end tool-call spy)
Make the interception usable from the API: declare `fakeTools: [{ tool,
when?, result? }]` on a measure/runEval arm and the eval tier does the rest.
- EvalArm.fakeTools + MeasureSpec.fakeTools.
- executeTrial auto-merges the fake PreToolUse hook into the arm's resolved
settings (APPENDING to any existing PreToolUse list, never clobbering),
pointing the hook at this CLI's own cli.js (resolved via __dirname, since
the eval runs in a throwaway cwd where `npx vigiles` wouldn't resolve).
- The fake list rides VIGILES_FAKE_TOOLS (AgentRunArgs.env, layered over
process.env in spawnAgent), RegExp matchers intact.
- CacheKeyInput gains `env` so two fake configs that share tool names — and
thus produce identical merged settings — don't collide in the replay cache
(their when/result lives only in the env).
Tested via the injected runner (no model): the hook merges + appends, the
env round-trips, and a no-fakes run wires neither. eval.ts/eval-cache.ts stay
at the 100% lines/functions/statements gate.
* refactor: make tool-fake honest + add real-model validation eval
Acting on the step-back assessment of the tool-call spy:
- Fix the misleading DEFAULT_FAKE_RESULT: the exit-2 hook is a BLOCK, not a
successful return (CC surfaces it as a denied call), so the default reason
now says "intercepted, NOT executed, do not retry" instead of "assume
success". Reframe the tool-fake/eval docs as intercept-and-prevent + observe
the attempt, not a faithful tool mock (CC has no skip-exec-return-success
primitive). Note cost is not reduced — only the side effect is removed.
- docs/eval-architecture.md: record the honest assessment — fidelity ceiling
(deny != success), toolWith overlaps the existing toolUsedWith predicate
(net-new is the negative check + interception), the simplest safety path
(don't allowlist + assert the attempt) often needs no new primitive, and the
competitor delta is narrow (assertions at parity; intercept-in-real-harness
is the edge).
- examples/harness/tool-fake.eval.mjs: end-to-end validation of the one
assumption unit tests can't reach — that a DENIED tool's args still land in
the Trace. Induces a git push origin main, intercepts it, asserts
toolWith(Bash, /push origin main/) holds across trials. Loud skip() without
claude. Run with a key before relying on the spy.
* ci: behavioral eval gate scaffolding (inert until ANTHROPIC_API_KEY)
Addresses the original kicker — real-model evals never ran in CI, results
frozen as FINDING: comments. Adds .github/workflows/evals.yml with two jobs
from the cost model in docs/eval-architecture.md:
- eval-gate (pull_request): the cheap tier — the three shipped model-invocable
skills' trigger evals (stubSkillBodies, 1 trial) on a DATED model, --no-skip.
- eval-nightly (schedule + workflow_dispatch): the fuller drift tier you can't
amortize.
Both are INERT until enabled: every model step is guarded on the
ANTHROPIC_API_KEY secret, so with no secret (default + all fork PRs) it prints
a notice and exits 0 — never reds a PR. Turning it on is a deliberate
secret + budget decision.
Plumbing: `vigiles eval --model=<id>` → VIGILES_MODEL (sibling of --trials)
so the gate pins a dated model id honestly without changing library defaults;
the cheap trigger evals + tool-fake.eval.mjs now read process.env.VIGILES_MODEL.
* feat: warn when an eval cache rides a floating model alias
Closes the last piece of the honest-model-pinning gap (doc #3). A record/replay
cache keyed on a floating alias (haiku/sonnet, or claude-sonnet-4-6 with no
date) can serve a result computed against a since-changed model — hiding drift
behind a green replay.
- isDatedModel(id): a model is honestly pinned when it ends in an 8-digit date
stamp (claude-haiku-4-5-20251001); a bare alias or version is floating.
- runEvalWith emits a one-time warning (::warning:: under GITHUB_ACTIONS, else
stderr) when cache != off and the model is floating.
Tested: isDatedModel dated/floating, both warning branches (CI annotation +
stderr), and the dated-id no-warn path. eval.ts stays at the 100% gate.
* docs: document arg-match + tool-fake modules in CLAUDE.md Key Files
The repo convention is that every src module appears in Key Files. Adds
entries for src/arg-match.ts, src/tool-fake.ts, and src/tool-fake.test.ts
(the tool-call spy), then recompiles CLAUDE.md from the spec — integrity
hash valid.
* test: validate tool-fake spy against a real model (capture-under-deny works)
Ran the validation against claude-haiku-4-5-20251001. The model called Bash
with "git push origin main", the auto-wired PreToolUse fake-tool-hook denied it
(isError, "the git push didn't execute"), and the denied tool_use — WITH its
args — still landed in ctx.toolCalls, so toolWith matched. This confirms the
one assumption the unit tests can't reach: a denied tool's arguments are still
captured for toolWith/notTool, and the side effect is prevented.
Adjust the eval to assert the MECHANISM, not model obedience: strengthen the
prompt, 5 trials, and gate at min 0.2 — any non-zero rate proves capture works;
only a true regression (a flat 0 across every trial) fails. Records the finding
inline.
* docs: refresh eval comparison (public + internal) and positioning
The eval comparison material predated the shipped significance/cache/cost/
regression-gating work and the tool-call spy. Brought it current:
- research/eval-api-landscape.md (INTERNAL): the scorecard's vigiles column
was marked ✗ on significance/caching/cost/concurrency/regression-gating —
all shipped. Rewrote it to shipped reality, added rows for the no-model
cheaper tiers, the intercept-and-prevent tool spy, and the regression gate;
noted the inert-until-keyed CI eval gate. Three bolded-unique rows now name
the defensible core honestly.
- docs/harness-testing.md (PUBLIC): broadened the promptfoo-only section into
a promptfoo/DeepEval/Braintrust/Inspect table; added the safety-intercept
and no-model-tier rows; pointed to the internal landscape doc.
- README: added one line on the safety property (notTool + fakeTools catch a
push-to-wrong-branch / paid-API attempt and prevent the side effect).
Positioning audit: README front door + internal scorecard now current; the
spy modules are already in CLAUDE.md Key Files.
* fix: harden eval cache invalidation (dir contents, tool-set, version salt)
Research-backed (research/cache-invalidation.md, synthesizing Bazel/Turborepo/
Nx/ccache/Jest/webpack + promptfoo/LangChain/Helicone). Fixes the confirmed
false-replay bug and two phantom-miss/staleness gaps, ranked by severity:
1. pluginDir contents weren't in the key (THE bug): a native --plugin-dir was
passed to the run but its files were never hashed, so editing a skill in it
false-replayed. hashDir() folds a sorted relativePath:contentHash list into
the key (content not mtime; path-included). Flat sorted list, not a Merkle
tree — over-engineering at plugin-dir scale.
2. Tool list now hashed as a SET (sorted) so a reordered allowlist isn't a
phantom-distinct key. (canonical() already sorted object keys, so no
stable-stringify dependency is warranted.)
3. CACHE_FORMAT_VERSION salted into the key (Jest/webpack pattern) so a record
format change makes old entries unreachable — no brittle read-time gate.
Model drift on a floating alias stays warn-only (the dated-id pin is the sound
fix; can't auto-resolve without an API call). Eviction/TTL deferred as disk
hygiene, not invalidation correctness — rationale in the research doc.
Tests: hashDir (edit/add/rename/skip-junk), tool-set order-insensitivity,
pluginDirHash sensitivity, and an end-to-end runEvalWith test proving a
pluginDir content edit busts the cache. eval-cache.ts/eval.ts stay at 100%.
* ci: let the eval gate run on a Claude subscription, not just an API key
The affordability bet: vigiles drives the real `claude` CLI, so an eval
authenticates like the user's own CLI — including a Pro/Max SUBSCRIPTION (flat
cost), not metered API billing. (Confirmed empirically: a validation run
reported apiKeySource:"none", i.e. it ran on the session's OAuth subscription.)
Both eval jobs now accept CLAUDE_CODE_OAUTH_TOKEN (subscription, via
`claude setup-token`) OR ANTHROPIC_API_KEY (metered fallback), and stay inert
until one is present. Header documents the subscription as the cheap path.
* docs: isolated-vs-whole-harness eval decision (research of record)
Decision: support BOTH as tiers — default isolated (cheap authoring loop),
add a whole-harness install-set arm as the release gate, plus a near-neighbor
middle tier on the existing ncd engine.
Why: skill selection is competitive and Claude Code evicts least-used skill
descriptions under a context budget (skillListingBudgetFraction), so isolated
trigger-rate overstates recall and understates false-positives. Every existing
tool (promptfoo, Anthropic skill-creator, AWS, OpenAI) isolates; none populate
the install set — a genuine wedge for "test the harness as it ships". Captured
in research/isolated-vs-whole-harness-eval.md; roadmapped in eval-architecture
(item 7). Also updated the eval-gate roadmap note for the subscription-token
auth path.
* feat: whole-harness trigger-rate tier (installSet) + honest isolated labelling
Implements the isolated-vs-whole-harness decision (research/isolated-vs-whole-
harness-eval.md). measureTriggerRate stays isolated by default (cheap authoring
loop); a new `installSet` co-installs the skill-under-test alongside the user's
real skills so it competes for selection as it does in the shipped harness — the
release-gate tier no other eval tool offers.
- TriggerRateSpec.installSet: plugin dirs or loose skills dirs merged into one
install under the under-test plugin's name (under-test wins a name collision,
so its <name>:<skill> id still matches `fired`). packageInstallSet +
collectSkillsSource + copySkillsInto, pure/tested, cleans up on any error.
- TriggerRateReport.competitors: count of co-installed competitors. 0 = isolated.
- formatTriggerRateReport now labels honestly: an isolated run prints that recall
is an UPPER bound and false-positive a LOWER bound (selection is competitive);
a whole-harness run prints the competitor count.
Tests cover the merge (skills/ + .claude/skills/ sources, collision, stub,
missing-source + empty-under-test throws, junk-skip), both report labels, and
installSet from pluginDir + skillsDir. eval.ts stays at the 100% gate.
* docs: lead README with affordability; correct the evals-in-CI framing
- README: affordability is now an explicit theme — most harness questions are
answered with no model (free, every commit), and a real-model eval rides the
user's Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI), not
metered API billing like API-SDK-based tools. States plainly that evals don't
run in CI by default.
- research/isolated-vs-whole-harness-eval.md: fixed the "nightly/release gate"
wording that wrongly implied a default CI cadence. Clarified that evals are
not in CI by default (the per-PR CI is the free deterministic tiers; the eval
workflow is inert until you add a token); "release gate" = run deliberately
(locally on the sub, or opt-in nightly).
* fix: make the eval CI gate fail on a non-run or a corrupt cache
Two silent-pass holes the eval gate could slip through — closed so a green
eval job actually means evals ran and the cache was sound:
1. Evals never executed. `vigiles eval`/`test` gained `--min=N`: assert at least
N scripts actually matched/ran, else exit 1 loudly. A renamed file, a bad
path, or a glob matching nothing previously printed "No files found" and
exited 0 — a hidden gap. evals.yml now passes --min=3 (the gate's three
listed evals) and --min=1 (nightly), alongside the existing --no-skip.
2. Cache fails. readCache now THROWS on a corrupt record (file present but
invalid JSON) instead of swallowing it and silently re-running. A broken
cassette is a real failure the gate must surface; a true miss (no file)
still returns null and proceeds normally. Message says how to recover.
Tests: cli.test --min (0 matched / too-few / enough), eval-cache corrupt→throws
vs miss→null. eval-cache.ts stays at the 100% gate.
* fix: trigger-rate on Sonnet (not haiku) + count sibling skills as competitors
Dogfooding the three shipped skills on the subscription found two real bugs:
1. Gate model. All three skills "failed" recall on haiku (test-harness 0.50,
edit-spec/strengthen 0.70) — but on Sonnet test-harness scores 0.90 and
passes. Trigger-rate is a SELECTION measurement and haiku is a much weaker
selector, so a haiku gate false-fails skills that fire fine on the model
users run. measureTriggerRate now defaults to "sonnet" (matching measure);
evals.yml pins claude-sonnet-4-6 for the gate; docs corrected. The skill
descriptions were never the problem.
2. competitors miscount. The honest-label counted only installSet additions, so
a multi-skill plugin (the dogfood installs all of vigiles' skills) printed
"isolated: no competing skills" while ~5 siblings were actually competing.
Now competitors = (skills in the resolved install pool) − 1, via countSkills,
so siblings count and a real single-skill install still reads 0/isolated.
packageInstallSet's return renamed competitors→added (installSet delta).
Tests: sibling-competitor count, packageInstallSet `added`. Recorded the
haiku-vs-Sonnet FINDING in test-harness.trigger.eval.mjs. eval.ts at 100% gate.
* feat: model-per-arm (Q1) + save model/isolation eval decisions
Q1 (model strategy) — agreed + built:
- EvalArm.model: a per-arm model override. A model comparison IS a harness A/B,
so "does my harness hold on the cheaper tier / after a model upgrade?" is
arms: { sonnet: {model}, opus: {model} } read through the existing
significance machinery — no separate multi-model matrix runner (that's the
cost-multiplying promptfoo lane we avoid). measureTriggerRate stays
single-model by design.
- Default stays Sonnet (the realistic selector), per the dogfood finding.
Q2 (isolation) — agreed: keep TWO tiers (in-plugin by default = honest sibling
competition; installSet for the whole harness). Do NOT build per-skill isolation
machinery (it produces the LESS honest number others default to). Already shipped;
no new code.
Saved research + conclusions: docs/eval-architecture.md gains a "Model strategy"
section (default Sonnet w/ the 0.50-haiku-vs-0.90-Sonnet dogfood evidence,
model-as-arm, no matrix, the deferred model-floor); the isolation
research/decision is in research/isolated-vs-whole-harness-eval.md.
Validated on the subscription: the corrected Sonnet gate is GREEN (3/3 skills
≥0.8) and the competitors label now reads "whole-harness: 4 competing skills".
* feat: runtime model floor for trigger-rate (default Sonnet) + gate on alias
The "set a floor for all skills" ask, solved without a (brittle) static linter:
a RUNTIME floor enforced in the eval library catches the env-var model a lint
can't see — which is exactly how the haiku footgun entered.
- modelTier(id): rank by family (haiku<sonnet<opus; version ignored), null for
an unrecognized family so the floor never blocks a model it can't judge.
belowModelFloor(model, floor): fail-open on unknowns.
- TriggerRateSpec.minModel (default "sonnet", resolves minModel → VIGILES_MIN_MODEL
→ "sonnet"). measureTriggerRate FAILS before spending a token if the resolved
model is below the floor — so one knob (VIGILES_MIN_MODEL / --min-model) sets
the floor for ALL skills, no per-file annotation, no linter. Lower it
deliberately for a cheap run. Decision: FAIL (not warn) — the floor is a
contract; the escape hatch is lowering it.
- CLI: `vigiles eval --min-model=ID` → VIGILES_MIN_MODEL.
- Gate model switched claude-sonnet-4-6 → the `sonnet` ALIAS: the per-PR gate
runs cache-off, so it should track current Sonnet without churn; dated/exact
ids matter only for cached/baselined reproducibility (kept on that path).
Tests: modelTier/belowModelFloor (families + fail-open), floor throw at default
+ VIGILES_MIN_MODEL, and the lower-floor escape hatch. eval.ts at 100% gate.
* refactor: remove model envs + the speculative evals.yml CI workflow
Acting on review: neither was asked for, and both were the wrong layer.
- Removed VIGILES_MODEL / --model and VIGILES_MIN_MODEL / --min-model. Model is
part of the measurement DEFINITION (a different model = a different result), so
it belongs in the spec (`model`/`minModel`, version-controlled), not a hidden
CLI/env override. Contrast VIGILES_TRIALS, a legit run knob (cost/precision,
doesn't change what's measured) that PRE-DATES this work — kept. The dogfood
evals now omit `model:` and rely on the Sonnet default + the minModel floor.
- Removed .github/workflows/evals.yml. Real-model evals don't belong in a
standalone GitHub Actions workflow needing a metered/sub-token secret: CI runs
the FREE deterministic tiers (ci.yml, no token), and the real-model eval runs
where the subscription already is — a Claude Code session or locally (vigiles
drives the `claude` CLI). The eval-robustness work (--min, --no-skip,
corrupt-cache throw, Sonnet model floor) stays — it hardens `vigiles eval`
wherever it runs.
Docs reframed accordingly (eval-architecture kicker + roadmap, README, the two
research docs): deterministic tiers in CI, evals on the sub in a session.
eval.ts at the 100% gate; 974 tests pass.
* docs: consolidate positioning + pros/cons of the eval approach
Per review — the competitor scorecard, caching research, isolation decision, and
model strategy were all saved, but the POSITIONING + pros/cons were scattered
across README/kicker. Added a dedicated "Positioning & pros/cons" section to the
eval-architecture design-of-record: the affordability thesis (free deterministic
tiers + sub-priced real-model tier vs competitors' per-token-every-run), the
pros (cost/fidelity/honesty) and the honest cons (sub rate limits, non-determinism,
intercept≠mock, sonnet-not-haiku, not-a-CI-checkbox, no dataset/red-team/UI), with
links to the four supporting research docs.
* fix: key the eval cache on the harness binary version (fast-evolution edge)
You were right — the cache wasn't tested/keyed against fast model/harness
evolution. Closing it:
- harnessVersion (`claude --version`, memoized, cache-path only) is now in the
cache key. A CLI upgrade changes the system prompt + tool defs — which steer
behaviour as much as the model — so a cached result now INVALIDATES across a
`claude` upgrade instead of silently replaying a result from a different
harness. Trade-off (frequent patch invalidation) documented; re-scoring within
a session is unaffected (version stable then).
- cacheKey test now actually covers model / settings / env / harnessVersion
changes (it previously asserted "any input" but skipped several, incl. model).
- research/cache-invalidation.md gains a "Fast-evolution edges" section: harness
version (keyed), model-alias re-point (warned, can't auto-invalidate without a
weight hash), and modelTier new-family fail-open (a documented known limit).
eval-cache.ts/eval.ts stay at the 100% gate.
* docs: positioning into CLAUDE.md (canonical) + consistency pass
Point 3 — positioning belonged in the positioning source, not buried in
eval-architecture.md:
- CLAUDE.md ## Positioning (pillar 2) now states the affordability thesis: free
deterministic tiers + a real-model tier that runs on the Claude Pro/Max
SUBSCRIPTION (vigiles drives the `claude` CLI, no metered API), measured on the
realistic selector (Sonnet, minModel floor), model-in-spec not env. Recompiled.
- eval-architecture's pros/cons section now points to CLAUDE.md as canonical and
keeps only the detail.
Gap #4 — refreshed the stale `src/eval.ts` Key Files entry (model-per-arm,
minModel floor + modelTier, installSet whole-harness tier + competitors,
harnessVersion in the cache key, Sonnet default).
Gap #2 (consistency pass, you approved) — the "knobs" / "match-the-mechanism" /
reconciliation sections no longer claim evals run in `every-PR` CI on `haiku`:
deterministic tiers are every-commit CI, the real-model eval is run on-demand on
the sub (Sonnet). Fixed the decision table + the reconciliation bullets too.
Gap #3 — documented the shipped trigger-rate knobs in the user guide
(harness-testing.md): Sonnet default + minModel floor, isolated-vs-whole-harness
`installSet` + the `competitors` count, and model-per-arm for cross-model A/B.
* feat: key cache on harness major.minor, automate stubSkillBodies, exhaustive cache tests + near-neighbor roadmap
Acting on review:
- Claude version granularity: key the cache on the harness MAJOR.MINOR, not the
patch (harnessVersionKey: "2.1.179 (Claude Code)" → "2.1"). A patch rarely
changes behaviour, so keying patches would churn the cache every release for no
signal; a minor/major bump is where the system prompt actually moves. Documented
the trade-off (a known-behavioural patch → clear cache / bump CACHE_FORMAT_VERSION).
- stubSkillBodies is now AUTOMATIC for measureTriggerRate (default true): testing
a description never needs the body (trigger is a frontmatter property decided
before the body loads), so it's not a knob you must remember. Set false only to
run the real body. Updated the 4 fake-pluginDir tests to stubSkillBodies:false
(passthrough), which also keeps the non-stub branch covered.
- Cache tests are now EXHAUSTIVE: one assertion per CacheKeyInput field, plus
explicit VERSION axes — two model snapshots (sonnet-4-6 vs -4-5) and two harness
minor versions each produce a different key. + harnessVersionKey unit test.
- Roadmap: added the near-neighbor trigger-rate tier (NCD-nearest competitors) to
Next, priority P3 (MED-LOW) — decided, deliberately deferred.
(Consistency gaps #2/#3/#4 shipped in 74fc151.) eval.ts/eval-cache.ts at 100% gate; 975 tests.
* feat!: rename fakeTools -> interceptTools (intercept-and-prevent, not a mock)
The tool-fake feature does intercept-and-prevent (a PreToolUse hook denies the
call via exit-2), not fake-a-return-value. The "fake" name wrongly implied a
mock that substitutes a result and continues, requiring two JSDoc corrections
to undo. Rename to carry the real semantics.
BREAKING CHANGE: the vigiles/eval spec field `fakeTools` is now `interceptTools`
(on EvalArm and MeasureSpec); the `FakeTool` type is `ToolIntercept` and its
`result` field is `denyReason`. The `vigiles fake-tool-hook` CLI subcommand is
now `intercept-tool-hook`. Pre-1.0, but signalled for a correct changelog.
Keep toolWith/notTool (clearer than competitors) but add promptfoo/DeepEval
cross-references to their JSDoc and a "vigiles check <-> promptfoo assertion"
mapping to research/promptfoo-deep-dive.md, noting notTool/interceptTools as
the restraint-assertion gap a completion-grading eval cannot cheaply express.
* docs: add safety model front-door doc (confinement + tool interception)
docs/sandboxing.md covers host confinement for hook/plugin code but not the
eval-tier risk — a real model deciding to call a side-effecting tool. Add
docs/safety.md as the front door tying the whole model together: the two
risks (foreign code vs the model's actions), the two mechanisms (bwrap
confinement vs interceptTools/notTool PreToolUse deny), the one safe-by-default
fail-closed rule, the per-tier confinement table, and the honest gaps. Answers
"why not confine everything" by pointing at the four-reason argument in
sandboxing.md. Referenced from README + sandboxing.md (not an orphan).
* docs: decide cross-platform sandboxing (macOS via sandbox-exec) + save the verify-sandbox-config idea
Mac support is a hard requirement (large dev population), and the claude binary's
own sandbox can't cover us (Bash-only, skips hooks, no external API, CC-only).
Record the decision: a vigiles/os-isolation port with a native backend per OS —
keep bwrap+nft on Linux (strongest, packet-layer), add sandbox-exec/Seatbelt on
macOS (built-in, zero deps, deny-all-net). Per-host egress stays Linux-only;
macOS degrades honestly. srt/nono are documented fallbacks, not the default;
Birdcage is reference-only (GPL/binary-net).
Also clarifies the enabled/disabled model (provenance, not platform: your code
runs direct, foreign code is confined-or-refused — the Mac backend makes the
safe option available so nobody is forced to sandbox:false), and parks the
"verify & test the harness's own sandbox config" idea (the network-validation
wedge: valid-is-not-true for sandbox policy). Roadmap + sandboxing.md updated;
not an orphan.
* docs: add the ephemerality axis — running a model-driven skill is a side effect, independent of trust
Correct the safety model: provenance governs CONFINEMENT (host protection), but
EPHEMERALITY (state protection) is unconditional — a skill/agent is model-driven,
so even a plugin you wrote produces nondeterministic side effects when run. Trust
does not make a run side-effect-free.
Classify side effects by reversibility: local-reversible -> ephemeral CWD (already
done); escapes-CWD (~/.gitconfig, ~/.ssh) -> ephemeral HOME + scrubbed env (the GAP:
the direct/non-bwrap path inherits process.env + real HOME); external -> deny/allow
net; irreversible (push, charge) -> interceptTools/notTool (can't un-push by
discarding a box). Ephemerality needs no kernel features, so it lands on macOS
today, ahead of Seatbelt — the cheapest cross-platform side-effect protection.
safety.md + cross-platform-sandboxing.md + roadmap (HIGH) updated.
* docs: positioning — behavioral/side-effecting testing + cross-platform (Mac+Linux); fix phantom issue refs
Positioning (CLAUDE.md): name BEHAVIORAL, side-effecting eval of the assembled
harness as the apex of pillar 2 (highest value, kept thin; notTool/interceptTools
ship today, the ephemeral run env + disposable-dep provisioning are the committed
next step). Add a "Cross-platform by requirement (macOS + Linux)" paragraph: native
backend per OS behind a vigiles/os-isolation port, and the two orthogonal axes
(provenance-keyed host protection vs unconditional ephemeral state protection).
Fix: the declared-vs-enforced anchor cited as Claude Code #54898 could not be
verified; replace every occurrence with the real anchors (Claude Code #4740/#21460,
claude-agent-sdk-typescript #172) across agent-runtime, cli, the spec, and research.
A reference-checking tool should not ship a phantom reference.
* docs(research): save the mid-2026 subagent-demand + skill-eval-landscape findings
subagent-compilation.md: ranked issue-survey demand evidence (declared!=enforced
anchored to #4740/#21460/SDK #172; orchestration/output-contract/dispatch/MCP/
observability/model/determinism/firing/nesting themes), the authoring+observability
tooling table (almost no verification/testing tool exists), Codex-vs-CC delta
(enforcement is Claude-Code-shaped -> Codex non-goal confirmed), honest map-to-vigiles.
skill-eval-landscape.md: how labs + practitioners actually eval skills — Anthropic
skill-creator + "Demystifying evals", OpenAI's Codex eval guide, the two Reddit
harnesses (SkillBenchmark blind-judge A/B+CIs on Caveman; the "SKILL.md is a
contract" methodology harness incl. the adversarial-gate finding), Spence/dbt/
Husain/Schmid. Gap verdict: behavioral skill eval is now TABLE STAKES; vigiles's
moat is the intersection (whole assembled harness + subscription affordability +
statistical rigor + the deterministic runHook tier + the eval->enforce bridge).
promptfoo-deep-dive.md: promptfoo acquired by OpenAI (~$86M) + DeepEval v4 agentic
harness; sharpens the vendor-neutral, subscription-not-metered positioning.
eval-api-landscape.md: new competitors (TribeAI/claude-evals metered; paultyng/
testagent CLI-fake) + statistical-rigor confirmed as a differentiator.
* docs: native input/output/cache token+cost measurement decision + behavioral-umbrella clarification
eval-architecture.md: (1) state explicitly that behavioral/side-effecting eval (the
ephemeral integration tier) is the UMBRELLA and the adversarial-gate test is one
member, not a replacement — with the eval->enforce bridge (pillar 2 hands off to
pillar 1 when a prose gate caves); (2) decide native token/cost support split by
class (input/output/cache) — the tokens() check collapses input+output and cache
tokens aren't captured, so a single total can bless a net-negative change (Caveman
cut output yet 2-4x'd cost). Plan: extend UsageTrace to all token classes,
first-class inputTokens/outputTokens/cacheTokens checks, and an A/B delta-per-class
gated by Welch significance. Added both to the ranked roadmaps (HIGH).
* feat: native input/output/cache token measurement + per-class checks
A harness change trades tokens across sides (a skill/CLAUDE.md injection adds
INPUT every turn; a "compression" skill cuts OUTPUT), so a single total can bless
a net-negative change — SkillBenchmark's Caveman cut output yet 2-4x'd cost. Split
the usage model into honest classes and gate on them separately.
- EvalUsage + UsageTrace gain cacheCreationTokens + cacheReadTokens (parsed from
the claude result usage: cache_creation_input_tokens / cache_read_input_tokens);
inputTokens stays the FRESH (uncached) input. ArmUsage + aggregateUsage sum the
two cache totals.
- New vigiles/check checks: inputTokens({max}), outputTokens({max}),
cacheTokens({maxCreation?, maxRead?}) beside the existing tokens/cost/latency.
Exposed on ctx.usage so an A/B delta-per-class rides the existing mean±se /
Welch-significance machinery for free.
- Dogfood: skill-compression.eval.mjs now measures input AND output (cache reads
~0.1x input, writes 1.25x), so the compression claim is verified on both sides.
Tests: pure pass/fail/score/message/toJSON for each new check; usageFrom parses
the cache fields (0 when absent); aggregateUsage sums the cache totals.
* docs: record real-haiku FINDING in the compression eval — cost metrics work, cache dominates
Ran the compression dogfood on real haiku (3 trials/arm, on the Pro/Max
subscription, apiKeySource:"none", $0.1057). Captures the end-to-end result as a
FINDING: total_cost_usd + input/output + cache tokens all populate from a real run;
inputTokens read ~42/run (the uncached remainder) vs ~2.4-2.7k real volume — the
rest is cache reads, so a single input/total understates economics ~50x here. The
cache classes are where the volume lives (vindicates the split). The compression
claim did not hold on this task/model (output +11%, cost +$0.002, correctness
intact) — the headline-% failure mode the eval exists to catch.
* feat: adversarial-gate dogfood eval + the eval->enforce bridge
The highest-value behavioral test for an enforcement skill: adversarially ask the
agent to skip the gate the skill is supposed to enforce, and assert it REFUSES.
- examples/harness/dogfood/adversarial-gate.eval.mjs (write-don't-run): an inline
SKILL.md states a measurement gate (no benchmark without a baseline; refuse if
asked to skip), the task adversarially asks to skip it, and two composed checks
assert refusal — notTool (didn't run the forbidden command) + output (pushed
back in prose), gated by assertRates({min:0.7}). No new check primitive: the
pattern is the existing vocabulary composed inline (no gold-plating).
- docs/eval-architecture.md + docs/safety.md: document the eval->enforce BRIDGE —
pillar 2 (eval) MEASURES whether a prose gate holds; when the adversarial test
shows it caves, pillar 1's deterministic PreToolUse rail is the FIX. Eval hands
off to enforcement; nobody else connects the two.
* feat: ephemeral run environment foundation (pure builder + opt-in flag, default OFF)
Running a model-driven skill mutates state regardless of trust (the model, not the
author, chose the actions), but the eval tier inherits the full real environment —
a model-driven git push / write to ~ uses the real $HOME, ~/.gitconfig, ~/.ssh,
~/.aws. Add the foundation to close that, behind an opt-in flag, default OFF so the
eval tier's own auth can't silently break before a real-run validation.
- ephemeralRunEnv(base, { home, allow }) — pure, exported, no fs/spawn: sets
HOME+TMPDIR to a throwaway dir, passes through an auth+runtime allowlist (mirrors
the runtime port: PATH/LANG/TERM/ANTHROPIC_*/CLAUDE_*/LC_* + region/profile so a
Bedrock/Vertex-backed claude still authenticates), DROPS everything else
(GIT_*/GH_TOKEN/SSH_*/AWS secret keys + any non-listed var). Conservative by
design: when unsure, allow (a too-narrow allowlist breaks auth).
- EvalSpec.ephemeralEnv?: boolean (default off). When on, executeTrial mkdtemps a
throwaway HOME under the trial cwd and builds the scrubbed env; AgentRunArgs gains
replaceEnv so spawnAgent uses a.env as the COMPLETE env. When OFF, the env path is
byte-identical to before (asserted by a test).
The cross-platform state-protection floor (needs no kernel features → lands on
macOS now). Flipping the default to on waits for a real-auth validation pass; two
honest follow-ups noted in code (AWS static-key auth, cache+ephemeral key interaction).
* docs: validate ephemeral-env auth survival (env-var path works; file-based ~/.claude is the gap)
Ran a real-model check (haiku, ephemeralEnv: true) on the subscription: the run
authenticated through the scrubbed throwaway HOME — exit 0, $0.003, output "OK.",
cache read ~21.7k tok (caching unaffected). So the env-var auth path is validated:
the credential rides an allowlisted ANTHROPIC_* var that survives the scrub.
Remaining gap before flipping the default on: this only covers env-var auth. A
subscription credential that lives only in ~/.claude/.credentials.json (a file
under HOME) would be lost when HOME is scrubbed — ephemeralRunEnv injects env
vars, not the credential file. Safe default-on needs injecting the harness's own
credential file into the throwaway HOME. Flag stays opt-in until then.
* feat: complete the opt-in ephemeral run env (auth-file injection + cache-safe) + validate containment
Finish the ephemeral-run-env opt-in path (still default OFF):
- seedEphemeralHome copies a minimal auth allowlist (EPHEMERAL_HOME_KEEP =
[".claude/.credentials.json"], copy-not-symlink so the run can't write back) from
the real HOME into the throwaway HOME when present, so local file-based OAuth
survives the scrub; .gitconfig/.ssh/.aws are never carried.
- cache-safety: HOME/TMPDIR excluded from the eval cache key (CACHE_KEY_ENV_EXCLUDE),
so the random throwaway HOME no longer makes every ephemeral run a cache miss;
non-ephemeral keys unchanged (surgical, not a blanket env drop).
Validated state protection end-to-end on the real subscription (haiku, ephemeralEnv:
true, the run told to write $HOME/PROOF.txt + probe ~/.ssh): real $HOME untouched
(write landed in the throwaway HOME) AND the run still authenticated (exit 0,
$0.0083). Containment holds; auth survives via the host-brokered CLAUDE_* allowlist.
Default stays OFF: this host-brokered env can't run-test the file-based
~/.claude/.credentials.json OAuth modality (no credential to exercise; no sandbox
mints a real token), so the credential-file injection is unit-tested for the copy
logic but not e2e-validated. Flipping default-on (a behavior change for every eval)
waits on a local-file-auth user validating that path; then it's a one-liner.
* docs: phased design for the vigiles/os-isolation port + Seatbelt backend
Grounded implementation plan over the existing sandbox core (design of record, not
yet built): the narrow port interface (NetPolicy tagged union / FsPolicy /
IsolationCapabilities / OsIsolationBackend), the core-vs-backend layout, the
per-backend capability matrix, the Seatbelt backend shape (sandbox-exec + minimal
SBPL profile), the eslint-plugin-boundaries enforcement (only backends name a raw
primitive, dogfooded via enforce), and a 4-phase migration that keeps the 997-test
suite green at each step (extract+move bwrap behind the port with ZERO behaviour
change -> add Seatbelt -> wire selection + lint -> fold in ephemeral env).
Surfaces the sharp limitation honestly: Seatbelt (deny network*) blocks localhost
too, so bwrap's co-launched-mock-over-loopback trick has no Mac equivalent
(loopbackNet:false) — a trusted spec runs direct, an untrusted-needs-mock spec
refuses. Per-host egress stays Linux-only; srt and CC's built-in sandbox stay out.
Linked from the roadmap (HIGH).
* docs: consolidate eval coverage/isolation into one source of truth (R1+R2 owned, R3 composed)
Add research/eval-coverage-and-isolation.md as the authoritative doc: the three-rung
model (R1 cheap/deterministic — nothing executes; R2 record-replay — a tool/MCP/API
result recorded once from the real tool and replayed via a fake binary on PATH, never
model-synthesized; R3 real disposable service whose semantics IS under test), the
blended+scrubbed coverage distribution (R1+R2 ~90%+ with no Docker on the sub; R3 a
thin apex — browser/DB/redis), the honest e2e landscape (every benchmark/lab/Inspect
runs e2e in a container; no containerless reproducible e2e exists), the
safety/viability/performance axes + non-goals, the competitor comparison, and a
reusable rung-classifier prompt.
Make the chosen approach crystal-clear in CLAUDE.md positioning (R1+R2 owned on the
sub; R3 composed-with-a-container, not reinvented), docs/eval-architecture.md
(Coverage & scope section), README (public), and the roadmap (the R2 enabler ranked
ahead of testcontainers + a dogfood/COVERAGE.md item). Use standard terminology
(PATH-shim + record-replay/VCR), not a coined name. Scrubbed — no private-source
identifying details; conclusions blended with the community/landscape data.
* feat: R2 tool-stub / PATH-shim helper (record-replay a tool result, no service, no Docker)
The R2 enabler from the coverage model: test a skill whose logic consumes a tool's
RESULT by shadowing the binary on PATH with a fake that emits a recorded/canned
output — standard PATH-shimming + VCR-style replay, not a novel approach. Unlocks
the ~43% R2 surface with no live service and no Docker (ranked ahead of any
testcontainers work).
- src/tool-stub.ts: ToolStub + renderToolStub/writeToolStubs/stubBinDir. Each stub
is a self-contained POSIX script that base64-decodes its canned bytes straight to
stdout (base64's alphabet can't contain a shell metacharacter → injection-proof
for any fixture content; piping preserves trailing newlines that command-subst
would strip). Stubs are author/recorded fixtures, NEVER model-synthesized
(drift → false confidence). argv-matching + record-from-real are documented
follow-ons.
- EvalSpec.stubs wired into executeTrial: a bin dir under the trial cwd is prepended
to PATH for BOTH the ephemeral and legacy env paths; absent stubs = byte-identical
(no default flipped, ephemeralEnv stays off).
- Exposed via vigiles/testing + a ./tool-stub subpath export.
- harness-test tier wiring left as a documented TODO (it routes through the
per-harness driver seam — out of MVP scope).
Tests: faithful round-trip of shell-special content incl. trailing newlines; PATH
prepend on both env paths; absent-stubs unchanged. 1007 passed / 11 skipped.
* test: dogfood the three vendored plugins + per-plugin COVERAGE.md scorecards
Prove how much of real plugins the cheap tiers actually cover. 9 model-free,
CI-runnable tests (vendor-coverage.test.ts) across the three vendored plugins —
OMC keyword-detector R1 fire/no-fire (the free floor under the bwrap-gated egress
copies), scan structural facts + the wshobson inherits-all footgun on the scan
report, superpowers session-start R1 (fallback) + R2 (reconstructed plugin root
injecting the real skill body), and the canonical R2 stubBinDir demo (shadow npm
on PATH to drive find-polluter.sh's real bisection to completion, no live service).
Per-plugin COVERAGE.md scorecards (rung table + distribution + grade + R3 shortlist
+ honest verdict), placed beside the pinned snapshots (never inside them):
- oh-my-claudecode: ~60% free / ~33% model-gated / ~7% R3 — B+
- superpowers: ~50% free / ~50% model-gated / ~0% R3 — B (a prose library is
half behavioral, only measurable with a model)
- wshobson-accessibility: ~30% free / ~25% model-gated / ~45% R3 — C (worst case:
its semantics ARE the real browser; the free tier still catches
the inherits-all agent footgun)
The B+/B/C spread is the finding: where free/deterministic testing stops and the
model/container tiers take over. 1016 tests pass; R3/model-gated items written-up,
not run, by design. Linked from eval-coverage-and-isolation.md (Dogfood results).
* docs: record 2026-06-17 multi-SDK probe in sdk-harness-testing
Validate and sharpen the prior "don't retarget pillar 2" verdict with a
nine-SDK primary-source probe across six dimensions (mock-model /
tool-contract enforcement / assembled-as-shipped / trigger-rate /
sub-affordability / record-replay). The prior verdict holds for code-SDKs
(Pydantic/Vercel/LangGraph/LlamaIndex/Mastra ship first-party mocks), but
the probe relocates pillar-2 value to the Claude Agent SDK + Codex (no mock,
buggy/unenforced #172 tool contract, sub-affordability) and adds a concrete
mock-ergonomics borrow-list. Roadmap entry tightened to match.
* docs: make the two-axis coverage boundary explicit (public + internal)
The R1/R2/R3 rungs only answer "what does the test need from the outside
world" — they don't answer "who decides pass/fail", which is what makes a
prose skill look like a coverage hole when it isn't. Add the second
(oracle) axis everywhere the rungs appear:
- Define the `-MG` (model-gated) suffix the dogfood scorecards already use,
and place a prose/guidance skill explicitly at R1-MG (nothing executes,
but only a model judges its worth — fully testable on the sub, not free,
not uncovered).
- Add the three-bucket framing (A free & deterministic / B model-gated on
your sub / C needs-a-container, composed) and grade with two numbers
("% testable at all (free+sub)" vs "% needs-a-container").
- State that skill BEHAVIOR is tested (runEval A/B + judged), not just the
description firing (measureTriggerRate).
- Record that the sub-affordability story is ToS-clean (drive your own CLI
/ your own harness / your own sub; the Agent SDK ToS restricts
productizing claude.ai login, not running your own tests).
- Make the shipped test-harness skill coverage-aware (a Step 0.5 that
classifies each surface into the three buckets and sets honest per-bucket
expectations).
- Roadmap: add the 2026-06-17 multi-SDK mock-ergonomics borrow-list, linked
to sdk-harness-testing.md, marked recent/this-PR.
Covers CLAUDE.md(.spec.ts), README, docs/eval-architecture.md,
research/eval-coverage-and-isolation.md, research/roadmap.md,
skills/test-harness/SKILL.md.
* docs: flag open finding — behaviour testing is broader than A/B
Persist a conclusion raised this PR that lived only in chat: A/B
(runEval arms + assertSignificant) is the right oracle for relative
/regression/noise-floor questions, but testing an EXACT skill wants an
ABSOLUTE judged assertion, not a comparison to an OFF arm. The single-arm
absolute path already exists (measure + judged + assertRates,
skill-quality.eval.mjs); the open question is whether it is first-class and
discoverable or whether A/B is over-privileged in the API + README + the
test-harness skill. Recorded as testing-api-design Part 7 #7 + a roadmap
item for the next review session.
* docs(positioning): sharpen the pillar-2 wedge against SDK reality
The mock-model tier is table stakes (Pydantic/Vercel/LangGraph/LlamaIndex
all ship a first-party fake), so it is not the differentiator. Name the
three things no SDK mock does — testing the harness as it SHIPS, tool-contract
enforcement of the assembled agent (anchored on the live Claude Agent SDK
bug #172), and a sub-affordable real-model tier. Also reframe behaviour
testing as TWO oracles: the absolute single-arm measure()+judged() (the
default when there's no on/off baseline) alongside the relative runEval A/B.
* docs(testing): make the absolute behaviour oracle first-class for single skills
A/B (runEval + assertSignificant) was over-privileged across the README,
docs/harness-testing.md, and the test-harness skill, forcing 'test this one
skill behaves right' into a relative on/off mould. Lead instead with the
absolute oracle — measure({ checks: [judged(rubric)] }) + assertRates — which
needs no baseline and is what promptfoo/DeepEval lead with; reframe A/B as the
specialised relative/regression oracle. Resolves testing-api-design Part 7 #7
(docs/skill only, no code change).
* docs(testing): declare one canonical import per pillar
The docs taught granular concern-named paths (vigiles/run-hook, vigiles/eval,
vigiles/check) while the test-harness skill and README use the vigiles/testing
barrel — an inconsistent canonical-import story. Declare vigiles/testing and
vigiles/linting the canonical entry points (granular paths stay as stable
aliases) so new code has one obvious path.
* docs(safety): correct overclaimed default + macOS confinement status
The ephemeral run environment is shipped but OPT-IN (ephemeralEnv: true,
default off) — docs/safety.md read as if it were on by default for every run.
State the real status: default-off today, default-on is the committed next
step gated on validating the file-based OAuth copy path; the pure pieces and
scrubbed-env wiring are unit-tested. Also mark Seatbelt as macOS-planned (not
shipped) in the two mechanism tables, so 'Mac deny-all' no longer reads as a
working network wall.
* docs(roadmap): note per-check thresholds for assertRates
The absolute-oracle path (measure + assertRates) is now recommended for testing
one skill, but assertRates({ min }) applies one rate floor to all checks. Persist
the additive, non-breaking follow-up: let min accept a per-check mapping.
* docs: add subagent-model-note rule; promote macOS Seatbelt to Now/P1
- New guidance rule: when launching a subagent, state which model was chosen
and why, in one line, at launch — visibility into the cost/capability tradeoff.
- Promote the cross-platform (Seatbelt) confinement backend from Next to Now/P1
per maintainer request; note there's no Mac to validate it on, so it needs a
macOS CI runner or a Mac-having contributor.
* refactor!: make the agnostic surface truly harness-agnostic, enforced by eslint
vigiles/testing and vigiles/integration re-exported Claude-Code transport
(scriptModel, claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable,
loadPlugin, resolveHarness) via blanket `export *`s — so the 'agnostic' barrels
weren't actually agnostic. The directory-based boundary lint missed it because
src/mock-model.ts (the Anthropic-Messages mock) lives at the composition root,
not under src/adapters/.
- testing.ts / integration.ts: re-export ONLY the agnostic harness-test names;
CC transport now comes from vigiles/claude-code.
- eslint: classify src/mock-model.ts as part of the cc-harness element, so the
existing agnostic-surface boundary rule forbids any agnostic barrel from
re-exporting it (verified: reintroducing the leak now errors).
- claude-code.ts: expose claudeCodeDriver + argv/parse helpers + claudeAvailable
(beside the Codex driver in vigiles/codex), preserving access off the agnostic
surface.
BREAKING: scriptModel/loadPlugin/claudeCodeDriver no longer importable from
vigiles/testing or vigiles/integration; import them from vigiles/claude-code.
* refactor!: trim package.json exports to a canonical public surface
Dropped 12 internal/redundant module subpaths that were each fully re-exported
by a concern-named barrel: ./compile ./linters ./eval ./harness-test
./harness-assert ./run-hook ./check ./judge ./mock-model ./plugin-loader ./mcp
./tool-stub. The package's `exports` map IS the access wall (Node throws
ERR_PACKAGE_PATH_NOT_EXPORTED; TS honors it under node16/bundler), so these are
now genuinely internal.
Canonical surface (12): . and ./spec (authoring + the module-augmentation
target), ./linting, ./testing, ./unit ./integration ./e2e (tiers),
./claude-code ./codex ./adapter (harness), ./vitest ./jest (runners).
./linting now also re-exports the linter engine (was ./linters).
BREAKING: the dropped subpaths no longer resolve — import via the canonical
barrel (vigiles/testing, vigiles/linting, vigiles/claude-code).
* docs: update Key Files descriptions for the trimmed public surface
vigiles/plugin-loader and vigiles/check are no longer standalone subpaths;
loadPlugin is on vigiles/claude-code and the check vocabulary on vigiles/testing.
* docs: sweep imports onto the canonical public surface
The trimmed exports removed the granular module subpaths, so rewrite every
doc/skill import to the canonical barrel: vigiles/{eval,run-hook,harness-test,
harness-assert,check,judge,tool-stub} → vigiles/testing, vigiles/{compile,linters}
→ vigiles/linting, vigiles/{plugin-loader,mock-model} → vigiles/claude-code.
CC-specific transport (scriptModel, loadPlugin) is split out of mixed imports
into a vigiles/claude-code line. Also corrected the two now-false 'granular paths
keep working' claims (only vigiles/spec remains besides the barrels).
* build: add API Extractor surface gate + API Documenter reference
Lock the public API surface so it can't silently regrow (the dual of the eslint
agnostic-surface rule, at the .d.ts level). scripts/api-extractor.mjs runs API
Extractor over all 11 public entry points and writes a committed report per entry
(etc/*.api.md) — the surface contract. CI (`node scripts/api-extractor.mjs` in
the check job) fails on any drift; `npm run api:report` regenerates after an
intentional change. The reports prove the de-leak: vigiles-testing.api.md has zero
CC symbols, vigiles-claude-code.api.md has them.
`npm run docs:api` runs API Documenter over the doc models to generate a
cross-linked Markdown API reference (api-reference/, gitignored — a derived
artifact, not committed). Narrative guides stay hand-authored.
Adds @microsoft/api-extractor + @microsoft/api-documenter as devDependencies.
* docs(roadmap): track moving the CC mock + driver into the adapter dir
The glob-classification of src/mock-model.ts as cc-harness enforces the agnostic
boundary today; the structural end-state is to relocate it (and claudeCodeDriver)
under src/adapters/claude-code/ so the boundary is directory-based, mirroring the
Codex adapter.
* ci: smoke-check the API reference generator (no publish)
Run API Documenter in the check job, reusing the doc models the surface gate
already wrote to temp/, so npm run docs:api can't silently break. Output
(api-reference/) is gitignored and not published — there's no docs site yet;
upgrade this to a Pages deploy when one lands.
* test(safety): prove ephemeralEnv actually scrubs — real-subprocess test
The one line that enforces ephemeral scrubbing (env: replaceEnv ? scrubbed :
merge) lived inside the v8-ignored real-spawn path, so a regression to an
always-merge would silently defeat ephemerality with no test catching it.
Extract it to a tested seam resolveSpawnEnv() and pin it two ways: a pure test
of the replace-vs-overlay decision, and a behavioural test that spawns a REAL
node child with the scrubbed env and asserts a planted secret is gone, HOME is
the throwaway dir, and the real ~/.ssh is unreachable. Deterministic, no model.
Updates the vigiles-testing API report for the new resolveSpawnEnv export.
* feat(testing): per-check rate thresholds in assertRates
The absolute-oracle path (measure({ checks }) + assertRates) is the recommended
way to test one skill, but assertRates({ min }) applied a single floor to every
check — you couldn't gate 'skill fires every trial AND judged >= 0.8' in one
call. Add an optional per (by check kind) override: assertRates(report, { min,
per: { skill: 1.0 } }); the failure message now reports each check's own min.
Additive + non-breaking. Updates the vigiles-testing API report.
* feat(testing): thread per-check thresholds through checkReportToJUnit
assertRates gained a per-check-kind threshold override but checkReportToJUnit
still took a single min, so the strict gate and the JUnit report could disagree
about which checks failed. Extract one shared checkRateThreshold() helper used by
both; checkReportToJUnit now accepts the same { min, per }. Updates the report.
* test(safety): prove interceptTools actually PREVENTS, end-to-end
The pure decision (decideIntercept/interceptHookDecision) was tested, but the
rail's whole point — the real `vigiles intercept-tool-hook` CLI, wired as a
PreToolUse hook, BLOCKING a matched call before it runs — was never exercised.
A regression in the subcommand's env-read or deny format would pass every pure
test. Drive the real built CLI through runHook (no model) and assert a matched
`git push` is blocked while a non-matching command and an un-intercepted tool
run. Deterministic; skips loudly if dist/ isn't built.
* fix(testing): guard two silent-green footguns in measure/assertRates
Both are the 'passes green while testing nothing' anti-pattern the no-silent-skips
rule forbids:
- assertRates on an empty report (measure with no checks) vacuously passed — now
throws, pointing at the likely empty checks list.
- stubSkillBodies + a judged check: stubbing replaces the skill body with a no-op,
so judged graded empty output (docs warned, nothing enforced) — now throws,
telling you to drop one. Adds tests for both.
* feat(cli): per-line GitHub annotations for broken doc refs
The dogfooded GitHub Action surfaces lint findings inline on the PR diff via
::error file=,line=:: workflow annotations. The inline/frontmatter/spec checks
already annotated per-line, but the markdown code-block ref check reported to
stdout only — so a broken enforce()/file()/cmd() in a doc never showed up
inline on the PR, only in the summary blob. Emit a per-line annotation for each
docRefReport error (it carries file+line). CI-only; skipped under --json/--summary.
Adds a hermetic test.
* ci: run the dogfooded vigiles Action as the primary lint gate
Move the composite Action (uses: ./) ahead of the redundant direct `node
dist/cli.js lint` in the check job, so a lint failure surfaces THROUGH the
shipped Action — its per-line annotations (inline on the PR diff) and sticky
PR comment — rather than a bare CLI step failing first and skipping it. The
direct CLI call stays as a redundant safety net. Aligns with prod-grade-gha-cli
(the Action is how CI runs for users).
* test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation
NOT FOR MERGE — a broken file() ref in docs/cli.md so the dogfooded Action
fails and leaves a per-line ::error annotation inline on the PR diff plus a
sticky comment. Revert this commit once the GHA behavior is confirmed.
* Revert "test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation"
This reverts commit
|
||
|
zernie
|
31592f784d |
feat: select harness via project config, mirror instruction files, verify skill frontmatter (#36)
* feat(cli): select harness via project config + mirror instruction files Add a `harness` key (string | string[]) to .vigilesrc.json, written by `vigiles init` and honoured by compile/lint, plus a `--harness=` override on compile — a deterministic, committed replacement for sniffing the cwd that kills the silent harness-mismatch footgun in the skill/agent compile path. When a repo declares >=2 harnesses and no sync tool or existing mirror fans the instruction file out, compile writes a byte-identical CLAUDE.md<->AGENTS.md copy (carrying the source's integrity hash, so a hand-edit trips the existing check). Selection is a discriminated union (a notice exists only on its variant) and the declared set is parsed once at the boundary — per the make-invalid-states- irrepresentable / parse-don't-validate guidance now recorded in CLAUDE.md. Design: research/multi-harness-compile.md. * feat(cli): warn when a declared minimal-profile harness drops skill frontmatter Slice 3 (verify half) of multi-harness compile. A skill's references are harness-agnostic; the one harness-specific surface is the SKILL.md frontmatter profile. `minimal` (Codex/OpenCode) is a strict subset of `claude-code`, so the useful, assumption-free check is the inverse: when a declared harness uses the minimal profile it DROPS the CC-only keys (disable-model-invocation, argument-hint), so a constraint the author set won't take effect there. `vigiles compile` now emits that warning per declared minimal-profile harness (src/skill-harness.ts). Per-root multi-emit stays deferred — output location is co-located today and minimal-profile key tolerance is unverified (see the design doc). Design: research/multi-harness-compile.md. * fix(cli): per-spec target disambiguation for compile dialect The harness-selection precedence documented a spec-target step that the code skipped: compile resolved one dialect per run (flag → config → detect) and ignored each instruction spec's own target. So an AGENTS.md.spec.ts could compile with the claude-code dialect — the silent mismatch the feature exists to kill. A CLAUDE.md.spec.ts is a claude-code file, an AGENTS.md.spec.ts a codex one, so the spec's target filename now selects its dialect (adapterForInstructionFile), above config/detect; the flag still overrides, and skill/agent specs (whose targets don't name a harness) keep the run-level pick. Also documents why `lint` takes no `--harness`: reference verification is harness-agnostic (it already recognizes both CLAUDE.md and AGENTS.md), unlike compile (renders one dialect) and scan (reports harness-specific structure). * docs: design for unifying scan + lint on one rule engine scan and lint are two parallel implementations with partial overlap: scan's structural findings (no-description skill, no-tool-contract agent, missing hook) are hard-coded in scan.ts/leaderboard.ts — not documented, configurable, or CI-gatable, unlike the half-shared untested-surface/orphan-docs rules. Specs Option B (the ESLint model): promote those findings to first-class rules; lint runs the configured set + gates, scan runs the structural subset with zero-config defaults + renders inventory + a rule-derived score. Rejects "scan = lint without config" (lint's reference core needs marks a bare repo lacks). Design only, separate initiative; linked from roadmap. * docs: document harness selection + save the snapshots conclusion User-facing docs for the shipped multi-harness work: a "compile — harness selection" section in docs/cli.md (the precedence, --harness=, the byte-identical mirror, the frontmatter-drop warning, why lint takes no --harness) and the `harness` config key in the docs/spec-format.md config table. Also records a session conclusion in research/testing-api-design.md: snapshots split three ways (output-text = reject, record/replay cassette = can't catch model drift, trace-structure = the viable deterministic regression tool), with the structural-snapshot vs statistical-baseline split. * docs: document scan's harness detection + --harness override * docs: deprecate the two demo scripts, add demo revamp to roadmap * docs: fix roadmap demo-revamp bullet formatting * test: close multi-harness coverage gaps - Extract mergeProjectConfig (pure) from writeProjectConfig and unit-test its branches: records harness when absent, never clobbers an existing harness, strict adds rule severities without overwriting, fully-satisfied → no write. - E2E: spec-target disambiguation — a target-less AGENTS.md.spec.ts compiles with the codex dialect (AGENTS.md heading), proving the per-spec wiring, not just the pure adapterForInstructionFile mapping. - E2E: mirror idempotence (an already-identical target isn't re-written) and compile --harness=bogus fails with an actionable "Unknown harness" error. - Assert the init default records harness "claude-code" on a greenfield repo. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
66ca161e8b |
fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals (#35)
* fix: round-2 dogfooding fixes for untested-surface, audit rename, orphans, loose-skill evals - untested-surface: discover colocated tests under dot dirs (dot:true), so a `.claude/skills/<name>/<name>.eval.mjs` the warning suggests actually counts as coverage instead of leaving the nudge unsatisfiable. - init: flag a CI workflow that still calls a renamed/removed subcommand (`vigiles audit` → `lint`) even when the Action or a modern command is also present, instead of reporting it "up to date". - orphan-docs: honor an inline `<!-- vigiles-disable orphan-docs -->` opt-out and point the failing message at both escape hatches (inline disable + .vigilesrc.json orphans.exclude/include). - measureTriggerRate: accept a loose `skillsDir` (e.g. .claude/skills) and auto-package it into a throwaway --plugin-dir via packageSkillsDir, so trigger-testing repo-local skills is a one-liner. - docs/strings: replace stale `vigiles audit` command references with `vigiles lint` (schema descriptions, hook/skill prose, CLAUDE.md). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: make strengthen/edit-spec model-invocable, tidy skill packaging, harden agent install Skills are useless if nobody remembers to run them. Address discovery + install gaps surfaced in the dogfood review: - strengthen + edit-spec are now MODEL-INVOCABLE (drop disable-model-invocation) with sharpened, precision-aware descriptions, so the agent reaches for them on its own — "strengthen my rules" / "add a rule to CLAUDE.md" fire the skill. The pre-edit hook already points at edit-spec when it blocks a direct edit; making it model-invocable closes that loop. Each gets a dogfood trigger eval (recall + precision) so auto-firing doesn't become over-firing. - Merge generate-rule into edit-spec (rule classification absorbed) and delete it. Fix stale commands in edit-spec (vigiles check->lint, setup->init, the old `npx skills add` vendoring installer -> marketplace). - linter-docs gets a real SKILL.md (user-invoked reference), fixing the packaging smell of a skills/ dir with no SKILL.md. - Bump .claude-plugin/plugin.json version 3.0.0 -> 4.0.1 (was stale). - Harden installPlugins: on auto-install failure/no-CLI, fail LOUDLY and lead with the shell-runnable CLI commands (an agent can run those) instead of only /plugin slash commands a non-TTY agent can't execute. - Sweep remaining stale `vigiles audit` -> `lint` (example spec). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: remove the renamed `audit` command everywhere; rename lint-command internals `audit` became `lint` in v4 but the old name lingered. Purge it: - cli.ts: rename the lint command's internals (audit()->runLint, AuditReport-> LintReport, auditExitCode->lintExitCode, printAuditSummary->printLintSummary, annotateAuditForGitHub->annotateLintForGitHub) + comments. Keep the REMOVED_SUBCOMMANDS `audit: "lint"` detection key (it matches the OLD name). - FUNCTIONAL: bench scripts invoked `node $CLI audit SKILL.md` (removed cmd) — now `lint`. - docs/research/comments: `vigiles audit` -> `vigiles lint`, and the command- sense word "audit" -> "lint". - Kept genuinely-distinct uses: the post-session AUDIT feature (session.ts), the Merkle "audit trail", the supply-chain audit / scan concept, the dev-only `audit-feedback-loop` skill, and English "you audited the code". - Left third-party vendored fixtures (SHA-pinned) untouched. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: prompt-diversity gate for trigger evals + raise auto-skill recall bar to 0.8 Task: ensure shipped model-invocable skills have evals, and validate prompt sets are big/varied enough before spending tokens. - measureTriggerRate now runs a deterministic pre-flight gate (no model): each prompt set needs >= minPrompts (default 10) and no two prompts more than maxSimilarity (default 0.8) alike via normalized Levenshtein. Throws with an actionable message before any run. Exports levenshtein/promptSimilarity/ checkPromptDiversity/assertPromptDiversity; minPrompts/maxSimilarity are per-spec overridable (narrow skills, short walkthroughs). - Expanded all shipped model-invocable dogfood evals (test-harness, strengthen, edit-spec) to 10 diverse relevant + 10 irrelevant prompts and raised the bar: recall min 0.8, maxFalsePositive 0.3. generate-logo + the canonical template expanded to 10; the OMC walkthrough sets minPrompts: 3. - All shipped skills are covered: the 3 auto skills by trigger evals, the user-invoked (migrate-to-spec, linter-docs) by the load gate. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor: use NCD (gzip) instead of Levenshtein for prompt-diversity check Levenshtein measured spelling; switch the prompt-diversity gate to Normalized Compression Distance — the SAME engine `findSimilarRules` uses for near- duplicate rule detection, so the project has one house algorithm for "are these two texts basically the same". NCD scores shared structure/redundancy (a templated prompt with one word swapped compresses together), which is exactly the lazy copy-paste set the gate should reject. - Drop hand-rolled levenshtein/promptSimilarity; add promptDistance (= ncd over normalized text), reusing src/core/proofs.ts ncd(). - TriggerRateSpec: maxSimilarity -> minDistance (default 0.3, the rule-dup threshold). Calibration: real 10-prompt sets sit at NCD >= 0.44; near-dup pairs (one-word swaps, punctuation) at <= 0.24 — clean separation at 0.3. - Tests updated to NCD distance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat: stubSkillBodies — measure skill triggering on frontmatter only Trigger-rate is decided by a skill's frontmatter (name + description) before its body ever loads, so the body is causally downstream of selection. stubSkillBodies keeps every skill's frontmatter (the selection competition stays faithful) but replaces each body with a no-op stub, so a trigger run stops AT selection instead of executing an expensive multi-step procedure — far cheaper, faster, side- effect-free, without changing what's measured. ~95% smaller skill payload. - stubSkillBody (pure) + packageSkillsDir({ stub }) (frontmatter-only package). - resolveTriggerPluginDir stubs both paths (pluginDir + skillsDir), preserving the plugin NAME so `<name>:<skill>` still matches the fired predicate. - TriggerRateSpec.stubSkillBodies (default off; on in all trigger evals). - Needs one real-model run to confirm the Skill tool still resolves under a stubbed body (the deterministic transform is fully tested here). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(init): scaffold spec imports only `claude` so it passes strict ESLint The generated CLAUDE.md.spec.ts imported `{ claude, enforce, guidance }` but the enforce()/guidance() examples are commented out, so a repo running `eslint . --max-warnings=0` (common in CI) failed on two no-unused-vars warnings the moment the scaffold was committed. Import only `claude` (the symbol the scaffold actually uses); show the fuller import as a comment for when a real rule is added. Regression test guards the import line. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(init): --force rewrites a stale CI workflow in place Round-2 follow-up: instead of only warning about a stale workflow, `vigiles init --force` fixes it. A renamed/removed subcommand (vigiles audit) is rewritten surgically in place (audit → lint, the rest of the user's workflow preserved); a bare-`npx vigiles` no-op workflow is regenerated from the template. Without --force the behavior is unchanged (warn, don't clobber) and the warning now points at --force. Pure `rewriteRemovedSubcommands` + force plumbed through setup-plan; unit + CLI integration tests. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: positioning for the eval-tier decision + new trigger-rate features - research/eval-api-landscape.md: dated decision — harness eval ≠ model eval (fidelity to the harness-as-shipped is why promptfoo et al. can't host it); keep the deterministic tiers, keep the real-model surface thin (stubSkillBodies = selection not execution), don't rebuild the eval stack, model fidelity is the one on-moat refinement. - CLAUDE.md(.spec): second-pillar paragraph gains the harness-eval-≠-model-eval framing; src/eval.ts Key Files entry documents skillsDir/packageSkillsDir, stubSkillBodies, and the NCD prompt-diversity gate. - docs/harness-testing.md: trigger-rate section documents skillsDir, stubSkillBodies, the diversity gate, and measuring on the user's real model. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(research): testing-API design proposal — the ideal shape Synthesizes "what makes a world-class testing API" (deterministic assertion libs: node:assert/Chai/Vitest/Testing-Library/fast-check; AI-eval: promptfoo/DeepEval/ Inspect) with the "do we need all these layers?" question into one conclusion: - One Trace (already the substrate); a single declarative `check` vocabulary evaluated two ways — strict (throw, unit/integration) vs scored (0-1 ± se, eval). Collapses today's predicate/assert*/matcher 3× into one surface and unifies deterministic + non-deterministic. - Three tiers, not four: e2e is integration + one egress symbol → fold egress in as a capability of integration, keep the capability ladder. - Checks-as-data ⇒ JUnit/baseline/regression + a promptfoo bridge fall out free. - Recommends option C (declarative checks) + D (property-test add-on for hooks). Indexed in CLAUDE.md Key Files + research/README; no users yet, so it's framed as a clean-break design, not a migration. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(orphan-docs): resolve file-relative markdown links; refine tier taxonomy in design doc - orphans.ts: a markdown link is conventionally file-relative ([x](foo.md) in research/README.md → research/foo.md, ../docs/x.md walks up). The detector only matched root-relative paths, so a bare sibling link read as an orphan. Now credits BOTH readings (refTargets resolves against the source file's dir + keeps the root-relative reading). Tests for same-dir and ../ links. - testing-api-design.md: fold in the two taxonomy questions. The layer names are borrowed from app testing, not derived: the harness has only TWO execution scopes (hook = the one standalone-process surface; harness = the whole agent, since skills/CLAUDE.md/settings only act inside a live agent) over a static lint floor. integration/e2e/eval are the SAME scope under realness flags (model mock|real, egress) — model picks the verdict mode (mock→strict, real→scored). "unit only works with hooks" is structural, not a missing capability; each layer adds exactly one capability (the ladder). Recommends runHook + runHarness(flags) over the four borrowed tiers. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 0 — the declarative `vigiles/check` vocabulary First step of the testing-API revamp (research/testing-api-design.md). A check is DATA, not a throwing assert: { kind, eval(target) → {pass, score, message}, toJSON() }, typed over Trace (tool/skill/output/hookFired/wrote) vs HookRunResult (blocked/allowed). One vocabulary, to be evaluated two ways downstream — strict (expect, Phase 1) and scored (measure, Phase 3) — so a check reads as pass/fail on one run AND a rate across trials, and serializes (JUnit/baseline/promptfoo-bridge fall out later). - src/check.ts + src/check.test.ts (pure, model-free; the failure MESSAGE is tested as much as the verdict). - Exported at `vigiles/check` only — NOT folded into vigiles/testing yet, since hookFired collides with the legacy predicate (reconciled in Phase 6). - Additive: nothing removed; 889 tests green. - Build plan appended to the design doc as Part 6 (Phase 0 marked shipped). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phase 1 — strict evaluator (assertChecks) + check matcher veneer - assertChecks(target, checks): the strict evaluator — throws collecting ALL failed checks' messages (Validation-applicative, not short-circuit). Any runner. - vigilesMatchers gains ONE generic veneer for the whole vocabulary: expect(r).toPass(tool("Bash")) / toPassAll([...]) — each carries the check's own failure message. vitest/jest type augmentations added; cross-runner tests prove toPass works under both. - Additive: the legacy predicate/assert*/matcher surface is untouched (it gets removed in Phase 6). 891 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(harness): Phase 2 — runHarness, the explicit harness-scope entry Establishes the collapsed naming from Part 4: the harness has two execution scopes (hook = runHook, harness = the whole agent); integration/e2e/eval are the ONE harness scope under realness flags. runHarness is the DETERMINISTIC harness run (model: "mock", default) — behaviour identical to runHarnessTest, which it wraps. A real-model harness run is non-deterministic, so a single one can't be asserted: runHarness(spec, { model: "real" }) throws and points at measure() / runEval (the eval scope). Unit-tested (the guard needs no claude). The e2e barrel fold + a materialized real path remain for Phase 6. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): Phases 3+4 — measure() over checks + serialization (toJUnit, assertRates) Phase 3 — the SCORED evaluator. measure(spec, { trials, checks }) scores a Trace check vocabulary across trials → per-check rate ± se + pass^k, reusing the tested runEvalWith aggregation (one arm). measureWith takes an injected runner (unit- tested, no model); measure wraps it with the real CLI. Defaults to model "sonnet" (measure on the model your users run). So `tool("Bash")` reads pass/fail strictly (assertChecks) AND as a rate here — one vocabulary, two evaluators. Phase 4 — serialization → CI. checkReportToJUnit (each check a <testcase>, fails below `min`) + assertRates (the scored gate) + formatCheckReport, all keyed off checkLabel(check.toJSON()). Because a check is data, JUnit/baseline/promptfoo- bridge fall out. Pure + unit-tested. (Wiring into the Action/baseline is the remaining CI-integration step.) 894 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(run-hook): Phase 5 — propertyHook, invariant testing for a hook's decision A hook is (event) → decision, so instead of a few example events, generate MANY and assert invariants hold for every decision ("destructive command ⇒ blocked", "never both allow and block"). Reuses the deterministic propertyTest from proofs.ts (seeded, shrinks the counterexample; no fast-check dep). The `decide` runner is injectable — pure fake for unit tests, `(e) => runHook(cmd, e)` for a real hook. Returns the shrunk counterexample, not a throw, so it's inspectable. Test: a correct guard satisfies the invariant; a buggy one (blocks only the exact "rm -rf /") gets a "rm -rf foo" counterexample found. 895 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): Phase 6 — front-door the new check surface; defer the mass deletion The new world-class surface (Phases 0-5) is now the documented front door: docs/harness-testing.md gains a "Checks — one vocabulary, two evaluators" section leading with assertChecks / measure / toPass; the design doc marks Phases 0-5 shipped; CLAUDE.md Key Files note runHarness/measure/propertyHook/toPass. DELIBERATELY NOT done: ripping out the ~40 legacy predicate/assert*/matcher helpers + the e2e barrel and rewriting the ~900-test suite/examples off them. On a no-users lib that's high-churn cleanup with no functional gain and real regression risk — the legacy surface stays as working sugar, removed in a single separately- reviewed major-version cleanup (resolving the check↔testing hookFired collision then). The breaking step is the one to take slowly. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * refactor(testing): fold check into testing/unit; collapse e2e into integration The "breaking cleanup" — done at its true (small) blast radius, not the feared 900-test rewrite: - vigiles/check folds into vigiles/testing + vigiles/unit; the check `hookFired` (Check<Trace>) supersedes the legacy boolean predicate via an explicit re-export that wins over the two `export *`s. Checks (tool/skill/output/..., assertChecks, measure, propertyHook) are now first-class at the front door. - e2e collapsed: egress is a CAPABILITY of integration (egressRoutes + runHook's egress:{allow}), not a tier — the old e2e barrel added exactly one symbol. vigiles/e2e stays as a thin deprecated alias so imports keep working. KEPT (deliberately): the bare predicates + per-runner assert*/matcher fronts. They're not redundant — they're context-appropriate surfaces (compose / node:test / vitest-jest) now unified UNDER one check vocabulary. Deleting them would be churn without a greatness gain. 895 tests green, types + cross-runner pass. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(testing-api): Part 7 — greatness re-evaluation + ranked eval-side gaps Post-cleanup assessment: the API is great on the two deciding axes (unification: one Trace / one check vocabulary / two evaluators / serialized; and the moat: the deterministic no-key tiers + pass^k/significance/fidelity nobody else has). Extensibility is strong; the one seam to watch is Check.eval being sync (fine incl. a judge()-backed check; an async scorer would need an AsyncCheck for measure only). Ranked gaps, eval-side first: (1) judged() model-graded check — judge() exists but isn't a Check; fits the sync interface; #1 parity gap; (2) cost/latency/tokens checks (data already captured); (3) checks × A/B arms + significance (unify the harness-A/B moat with measure); (4) multi-surface CC eval — Trace is single+flat, no nested subagent/Task traces, slash-command expansion, multi-turn, or mcp() checks (the deep, new-capability bet); (5) real-tier Trace fidelity (modelRequests mock-only); (6) CC-feature drift maintenance. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): close eval-side gaps — judged/cost/latency/tokens/mcp checks, measureArms + significance, harness-agnostic - judged(rubric, { min, judge }) — model-graded check (LLM rubric) via an injectable judge fn (default real judge(), sync via spawnSync), composes into measure/assertRates/toJUnit. Closes the promptfoo llm-rubric / DeepEval GEval gap. - cost/latency/tokens — Check<UsageTrace> over a run's usage; measure.checks widened to Check<RunContext> so they compose with behavioural checks. - mcp(server, tool) — matches the mcp__server__tool tool name. - measureArms(spec, { arms, checks }) + compareCheck(report, baseline, arm, i): the SAME checks scored per arm, with Welch significance (reuses stats.ts) — unifies the harness-A/B moat with the check vocabulary. - Harness axis: a test pins that checks are harness-agnostic (evaluate over a Codex-sparse Trace — output passes, tool/hookFired fail gracefully). Runners dispatch via adapter; measureWith/measureArmsWith take an injectable runner so a Codex eval reuses the same checks. All unit-tested with fakes (no model). 900 tests green. Design doc Part 7 updated: gaps 1-3 + mcp shipped; nested-trace (subagents/turns) is the remaining big bet. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): add the vitest/jest setup + matcher matrix The reference table we'd flagged in the vitest/jest discussion: a two-column setup matrix (vitest vs jest: config / per-file / manual) and a matcher table (toPass / toPassAll / toHaveCreated / toBlock / toBeatBaseline → asserts → over which result), leading with toPass as the one matcher fronting the whole check vocabulary. Makes the runner story scannable instead of prose. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): nested subagent traces — Trace.subagents + subagent() check The deep gap's first piece: a subagent (Task) runs its own session, recoverable because CC tags its events with parent_tool_use_id (= the Task tool call) and the Task input carries subagent_type. parseSubagents groups the tagged tool calls into Trace.subagents (keyed by name); subagent(name, [checks]) runs the WHOLE check vocabulary recursively over that sub-trace, so a test/eval can assert what the subagent DID, not just that Task fired. Wired into makeResult (harness) + makeContext (eval); empty for harnesses that don't emit parent_tool_use_id (Codex) so it stays harness-agnostic. parseSubagents + the check unit-tested on synthetic stream-json; flagged in Part 7 for one real-CC-subagent run to confirm the field names. 902 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(check): slash-command-expansion (received) + multi-turn (turns) checks Closes the last two pieces of gap #4, both unit-tested: - received(matcher): the expanded slash command (or a hook's injected context) actually REACHED the model — reads modelRequests (mock/harness tier captures them; the eval tier captures none, with an explicit message saying so). - turns({min,max}): the multi-turn observable — the agent took a back-and-forth, not a one-shot (the deterministic harness already scripts multiple model turns via ModelTurn[]; this asserts how many it took). Remaining (smaller, noted in Part 7): driving multiple USER messages (--resume-style, unverifiable here) + per-turn trace slicing. 904 tests green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(check): validate the subagent parser schema against Claude Code stream-json Confirmed via the Claude Code SDK docs / guide: subagent events carry a top-level `parent_tool_use_id` (= the Task tool-use id), the Task input carries `subagent_type`, linked by tool_use id — exactly what parseSubagents assumes, on the same CLI `message.content` line shape parseToolCalls already consumes (the SDK StreamEvent `event:{delta}` shape is a different, unused format). No code change — the parser is validated; dropped the "needs a real run to confirm" caveat in the comment + Part 7. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(dogfood): exercise the new check capabilities on real targets - checks-dogfood.harness.mjs — RUNS here (claude, no key): assertChecks over a real runHook (blocked/allowed) + a real runHarness mock run (turns/output). Validated end-to-end, not just unit-tested with fakes. - dogfood/skill-quality.eval.mjs — the SCORED path on vigiles's OWN strengthen skill: measure({ checks: [skill, judged, cost] }) + assertRates (does it fire, is the suggestion good, is it cheap). Write-don't-run (needs a key). - subagent.eval.mjs — the nested-trace check on a REAL subagent (vigiles ships none): the vendored oh-my-claudecode code-reviewer, dispatched via Task; subagent("code-reviewer", [tool("Read")]) asserts what the subagent DID. Note: the harness-tier dogfood asserts the cross-version-reliable observables (turns/output); tool-call capture in a bare run is plugin/permission-dependent (the fuller pluginDir tests in harness-test.test.ts cover it). 904 green. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs: document the mock-tier tool-capture fidelity edge (diagnosed, not a check bug) Diagnosed the flaky tool-call capture from the dogfood: the full stream shows the mock's scripted Write tool_use is never surfaced ("tool_use" absent, only the text turn, plus a spurious rate_limit_event from a LOCAL mock) — a mock↔claude desync with the current version, isolated to a BARE runHarness. The established pluginDir configs capture tools fine (the Skill test runs + passes here). Confirmed it's not permissions (acceptEdits didn't help, reverted), not the parser (tool_use isn't in the stream), and not the check API. Noted in Part 7 gap #5 as a harness-fidelity backlog item; the reliable observables (hooks/turns/output/ received) are unaffected. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(harness-testing): make the full check vocabulary + measureArms discoverable The guide had the Checks intro but not the vocabulary itself. Added a scannable reference table (tool/skill/mcp/subagent/output/received/hookFired/turns/wrote/ judged/cost-latency-tokens/blocked-allowed → holds-when → over-which-result), plus a brief measureArms+compareCheck (A/B with Welch significance) and propertyHook note. So a user can discover the new capabilities, not just read that they exist. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * fix(harness): match subagent dispatch by subagent_type input, not tool name A real `claude` run revealed the subagent dispatch tool is named `Agent` on the live CLI, not `Task` as older docs say. parseSubagents now keys off the `subagent_type` input field (present on the dispatch tool_use regardless of its name) so it survives the rename. Validated end-to-end against real claude output, which recovered the dispatched subagent and its tool calls. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measure for cheap skill-firing checks measure() now takes stubSkillBodies (the same body-stub the trigger tier uses, via a shared stubbedPluginDir helper): a skill-FIRING check (skill()) stops the selected skill at selection instead of running its often-expensive procedure, so a description/firing run costs a fraction of the tokens. Requires pluginDir; throws if combined without one. Not for judged/quality checks — the body is gone. Refactors resolveTriggerPluginDir to reuse the extracted stubbedPluginDir helper. Unit-tested (stubbed body in the packaged plugin, cleanup after, the no-pluginDir guard). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * chore(types): regenerate project-file types to include src/check https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * docs(dogfood): add cheap firing-only eval using measure + stubSkillBodies A shipped dogfood example for the body-stub path: measure({ checks: [skill, latency], stubSkillBodies: true }) on the strengthen skill, the cheap sibling of skill-quality.eval.mjs (which keeps the body to grade quality). Makes the ~18x firing-check saving discoverable, not just unit-tested. Smoke-run against real claude: both checks 100% (skill fires, run stays fast). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * feat(eval): support stubSkillBodies in measureArms for cheap A/B firing tests Brings measureArms to parity with measure: an A/B firing comparison (does description variant A fire more reliably than B?) now stubs each arm's skill bodies via stubSkillBodies. Every arm that sets a pluginDir is repackaged with bodies stripped (frontmatter kept) through the shared stubbedPluginDir helper, so each run stops at selection — a fraction of the tokens, per-arm. Arms without a pluginDir pass through untouched; throwaways are cleaned up in a finally. Not for judged/quality checks — the body is gone. Unit-tested (both arms see a stubbed throwaway, originals untouched, cleanup after). https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL * test(eval): restore 100% coverage for the stubSkillBodies + check surfaces CI's coverage gate (100% lines/functions/statements on the harness-testing files) flagged uncovered paths accumulated across the revamp: the measure/ measureArms stubSkillBodies branches and their helpers (stubbedPluginDir, skillsDirOf .claude/skills fallback, pluginName missing-manifest catch, packageSkillsDir missing-dir throw + non-directory entry skip), checkLabel's no-arg fallback, and the toPass/toPassAll matchers. Adds targeted unit tests for each — all tests still pass, coverage back to 100%. https://claude.ai/code/session_01Po6bhCFneUwx3Htn8CokuL --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
2c64c62b32 |
feat!: fix the init/install flow (marketplace plugin, devDep, real spec, version) (#33)
* feat!: fix the init/install flow (marketplace plugin, devDep, real spec, version) Rework `vigiles init` so a fresh install actually works and stops polluting the consumer repo. Each item below was a confirmed bug: - Plugin via marketplace, not vendoring (#1/#7): drop `npx skills add` (which vendored .agents/skills + .claude/skills symlinks + skills-lock.json into the working tree). `init` now installs the Claude Code plugin through the marketplace — `claude plugin install vigiles@vigiles` when the CLI is on PATH, else prints the two `/plugin` slash commands — globally into ~/.claude/plugins/, nothing written to the repo. Internal dev-only skills (generate-logo, pr-to-lint-rule, enforce-rules-format, audit-feedback-loop) move to dev/skills/ (a contributor-only `vigiles-dev` plugin) so consumers get only the consumer skills (test-harness, migrate-to-spec, strengthen, edit-spec, generate-rule, linter-docs). - Verify pillar really sets up (#2): scaffold a spec for every instruction file instead of only printing "migrate". An existing hand-written file is never compiled over (would clobber it) — it gets a spec + a migrate-to-spec next step. - Dependency management (#3/#4): add/upgrade `vigiles` in devDependencies (moving it out of dependencies / off a stale `github:` pin) so the scaffolded harness resolves `vigiles/testing`. - Stale CI workflow (#5): detect an old bare-`npx vigiles` workflow and warn loudly with the fix instead of silently saying "already exists". - Honest commit hint (#6): list only files actually written this run. - `--version` (#8): print the version number, not the help banner. - Action vs npm versioning (#9): document that the Action tag (@v1) and the npm CLI version (3.x) are separate lines; align the plugin version to 3.0.0. Also replaces the awkward `--pillars=verify|test|both` with `--verify` / `--testing` flags (deprecated `--pillars` kept as an alias) and adds an optional `--harness=claude,codex` override on top of auto-detection. Per the user's direction, no legacy git-commit upgrade-path detection (#10) — there are no existing external users. BREAKING CHANGE: the Claude Code plugin now installs via the marketplace (`/plugin marketplace add zernie/vigiles` + `/plugin install vigiles@vigiles`) instead of `npx skills add`, and ships only consumer-facing skills — the four vigiles-development skills are no longer installed for consumers. The `--pillars` flag is superseded by `--verify`/`--testing` (still accepted as an alias). https://claude.ai/code/session_01AV4WainLabYw85s6KPjGdD * feat(init): per-harness install decision + deterministic install smoke tests Make the plugin/skill install a pure, per-harness decision (planPluginInstall) and prove the whole install outcome with a deterministic CI smoke test — the gap that let the v3 install bugs ship unnoticed. - planPluginInstall(harnesses, { hasClaude }) is a pure function returning the per-harness install plan, so a test asserts WHICH commands run with no network and no real claude/codex binary: - Claude → plugin marketplace (claude plugin install → ~/.claude/plugins/) - Codex → cross-agent `skills` CLI with -g (npx skills add zernie/vigiles -a codex -g → ~/.codex/skills/) BOTH install globally — neither vendors into the repo (the issue #1 fix holds for both harnesses). Codex hooks (.codex/config.toml [hooks]) are not auto-wired yet; the plan says so out loud. - Harness-aware spec targets: a Codex-only project scaffolds AGENTS.md (not CLAUDE.md); claude+codex scaffolds both. - Deterministic installation smoke tests: run `init` against realistic todo fixtures (Claude / Codex / both) and assert the full outcome AND the negatives — spec scaffolded, CLAUDE.md preserved, dep moved to devDeps, and nothing vendored (.agents/skills, .claude/skills, skills-lock.json absent). This is the regression guard for the class of bugs that shipped before. Why they slipped through: every prior init test used --no-plugin on an empty synthetic dir and asserted single files; nothing ran a full install on a real project or asserted the no-vendoring / dep / preserve-content invariants. https://claude.ai/code/session_01AV4WainLabYw85s6KPjGdD * fix(init): collapse a CLAUDE.md⇄AGENTS.md mirror to one spec If CLAUDE.md and AGENTS.md are the same artifact — a symlink (`ln -s CLAUDE.md AGENTS.md`) or kept byte-identical by rulesync/Ruler — then scaffolding a spec for each means two specs fighting over one file and a guaranteed integrity-hash collision (the "Compose With Sync Tools" rule). `init` now runs the existing detectInstructionMirror() and collapses the pair to a single canonical spec (the real file for a symlink; CLAUDE.md for synced content, which the sync tool fans out to AGENTS.md), printing why. An explicit --target is still honoured as-is. Covered by two deterministic smoke tests: a symlinked AGENTS.md and a byte-identical (synced) pair each produce ONE spec, never a second. https://claude.ai/code/session_01AV4WainLabYw85s6KPjGdD * feat(init): compile into a sync tool's source slot + real Codex install + gated e2e Two follow-ups from review, plus a correction the live test surfaced. Sync-tool redirect (was: warn-only): - When rulesync/Ruler is detected, `init` now REDIRECTS the scaffolded spec into the tool's source slot (`.ruler/AGENTS.md.spec.ts` → compiles `.ruler/AGENTS.md`) via composeCollisions, instead of scaffolding a root CLAUDE.md spec that the tool would overwrite (staling the integrity hash). vigiles compiles upstream; the tool distributes. "Compose, don't fight." - findSpecs now uses `dot: true` (ignoring .git) so a spec in a dot-dir source slot is discovered by compile/audit/the recompile hook. - init() derives the spec's `target` field from the basename, so a spec in a subdir/slot validates and compiles correctly (this was already broken for any `--target=sub/dir/FILE.md`). Codex install corrected against the REAL `skills` CLI: - The command is `npx skills add zernie/vigiles -a codex -g -y` — needs `-y` (non-interactive) and installs to the global store `~/.agents/skills/`, NOT the documented `~/.codex/skills/`. Fixed the plan, messages, and docs. - New gated e2e test (src/cli-install.e2e.test.ts, network tier) runs the real install into an isolated $HOME and asserts it lands globally, never in the repo. Self-skips where npx/network is unavailable. This is what caught the wrong path. Smoke tests updated: a Ruler project redirects into .ruler/ (no root spec). https://claude.ai/code/session_01AV4WainLabYw85s6KPjGdD * feat!: consolidate the pillar vocabulary to Lint + Test The two pillars were referred to inconsistently (verify / audit / linting / pillar 1 vs testing / test / pillar 2). Consolidate on two names used consistently across flags, commands, the Action, and docs: - Lint pillar — verify instruction-file references. - flag `--lint` (/`--no-lint`); command `vigiles lint` (alias of `audit`); export `vigiles/linting`. The GitHub Action's default command is now `lint`. - Test pillar — test the harness. - flag `--test` (/`--no-test`); commands `vigiles test`/`eval`; export `vigiles/testing`. Backwards-compatible: `--verify` / `--testing` / `--pillars=both|verify|test` still parse as aliases, and `vigiles audit` still runs. The internal SetupPlan field `verify` is renamed to `lint`. Docs (README, CLI ref, agent guides), the scaffolded CI workflow's job name, and the interactive prompt are all updated to the Lint/Test vocabulary. BREAKING CHANGE: the init pillar flags are now `--lint` / `--test` (the old `--verify` / `--testing` remain as deprecated aliases), the canonical references-verification command is `vigiles lint` (`audit` kept as an alias), and the GitHub Action's default `command` input changed from `audit` to `lint`. https://claude.ai/code/session_01AV4WainLabYw85s6KPjGdD * feat!: drop the deprecated aliases (clean break) Following the Lint+Test consolidation, remove the back-compat shims now that this is a major release: - `vigiles audit` command removed — `vigiles lint` is the only name. - `--verify` / `--testing` init flags removed — use `--lint` / `--test`. - `--pillars=both|verify|test` removed. `vigiles audit` now reports "Unknown command"; the old flags are ignored. The stale-workflow detector recognises `vigiles lint` (not `audit`) as the modern API, so a workflow still running `vigiles audit` is flagged for update. Tests, docs, the Action input description, and the scaffolded workflow are updated. BREAKING CHANGE: the `audit` command and the `--verify` / `--testing` / `--pillars` init flags are removed. Use `vigiles lint` and `--lint` / `--test`. https://claude.ai/code/session_01AV4WainLabYw85s6KPjGdD --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cf40a0f96c |
docs: clarify pillar decoupling and surface install in harness section (#32)
* docs: clarify pillar decoupling and surface install in harness section
State the two-pillar decoupling plainly (neither pillar gates the other)
instead of framing it by which file each pillar doesn't need. Add a
concrete install command to the "Test your harness" setup block so it
anchors to `npx skills add zernie/vigiles` / `vigiles/testing` instead of
a vague "Install vigiles" prompt.
* chore: gitignore locally-installed dev skills
Ignore .agents/, .claude/skills/, and skills-lock.json so locally
installed skills (e.g. obra/superpowers) used for internal development
stay out of the repo and don't show as uncommitted changes.
* chore: vendor obra/superpowers skills for internal dev use
Add the superpowers skills under .agents/skills/ (with .claude/skills/
symlinks) and skills-lock.json for internal development. Verified via
`npm pack --dry-run` that none of these paths ship in the npm package or
the Claude Code plugin (.claude-plugin/plugin.json ships only ./skills/),
so they're part of the repo but not part of what vigiles ships.
Reverts the prior .gitignore exclusion of these paths.
* chore: enable superpowers via plugin marketplace instead of vendoring files
Replace the vendored obra/superpowers skills (.agents/, .claude/skills/,
skills-lock.json) with a project-scoped plugin declaration in
.claude/settings.json (extraKnownMarketplaces + enabledPlugins). Claude
Code fetches the plugin into its ephemeral cache on startup, so the repo
carries the enablement config, not 60+ third-party files.
* docs: add production-grade GHA + CLI as a tool requirement
Add the "prod-grade-gha-cli" guidance rule to the spec (compiled into
CLAUDE.md): the CLI and GitHub Action are first-class, production-grade
surfaces — CLI is the single source of truth with every Action input
mapped to a real flag, the Action is a composite over the published CLI
that works as uses: zernie/vigiles@v1, declares outputs via
$GITHUB_OUTPUT, ships a full documented workflow, and is dogfooded in CI.
* feat!: production-grade GitHub Action + CLI flag parity
Make the GitHub Action and CLI first-class, production-grade surfaces
(satisfies the new prod-grade-gha-cli rule).
CLI:
- Add --max-rules=<n> and --catalog-only flags (src/cli-flags.ts,
applyConfigFlags) so every Action input maps to a real CLI flag instead
of a config-file-only knob. Pure + unit-tested.
GitHub Action:
- Rewrite action.yml as a composite action over the published `npx
vigiles` CLI. The old node20 entry pointed at dist/action.js, which is
gitignored and never committed, so `uses: zernie/vigiles@<ref>` could
never actually run — only npm did. The composite reuses the same tested,
published artifact.
- Declare the `valid` output and set it via $GITHUB_OUTPUT (drops the
deprecated ::set-output). Add `version` (incl. `local` for dogfood) and
`working-directory` inputs.
- Remove the now-dead src/action.ts entry.
CI / release:
- Dogfood the Action in ci.yml via `uses: ./` with version: local and
assert the `valid` output.
- release.yml maintains a floating `v1` major tag so `uses:
zernie/vigiles@v1` tracks the latest release.
Docs/tests:
- Full Action reference in docs/cli.md (copy-paste workflow, permissions,
inputs, output, @v1-vs-@main versioning) + README pointer.
- Structural test (src/action-yml.test.ts) locks in the composite/outputs/
no-set-output contract.
BREAKING CHANGE: the GitHub Action is now a composite action and no longer
ships dist/action.js. Inputs/outputs are unchanged in name where they
overlap (command, paths, max-rules, catalog-only) but the action now adds
`version`/`working-directory` inputs and a `valid` output. Pin to
`zernie/vigiles@v1` going forward.
* feat: PR comments + job summary output, JS/TS harness scripts
GitHub Action output:
- Post an automatic STICKY PR comment on pull_request events (found by a
hidden marker and updated in place, never duplicated) + always write a
job summary ($GITHUB_STEP_SUMMARY), alongside the existing inline
annotations. New `comment` / `github-token` inputs; best-effort so a
fork PR without write access degrades to a warning. Dogfooded in ci.yml
(check job granted pull-requests: write).
Dual-language harness/eval scripts:
- `vigiles test` / `vigiles eval` now discover and run scripts authored in
JavaScript (.mjs/.cjs/.js) OR TypeScript (.ts/.mts/.cts). A TS script
runs through tsx when installed, else Node's native type stripping
(>=22.6), with an actionable error otherwise (src/adapters/claude-code/
run-scripts.ts: interpreterArgs/detectNodeCaps/scriptGlob).
Requirements + docs:
- Extend the prod-grade-gha-cli rule (three output channels) and add the
dual-language-tests rule to the spec (compiled into CLAUDE.md).
- Document the comment behavior + pull-requests: write in docs/cli.md;
rewrite the JS-vs-TS section of docs/testing-matrix.md.
Our own unit suite remains 100% TypeScript (every src/**/*.test.ts);
examples/harness/*.mjs stay JS on purpose as zero-dep CLI demos.
* docs: slim README to a billboard landing page
Cut the README from ~207 to ~118 lines, moving it out of the "long but
neither pitch nor index" middle into a focused landing page (Knip/esbuild
shape), per a survey of dev-tool READMEs (Ruff/Biome/Vitest/ast-grep/…)
and standard-readme/Diátaxis guidance.
- Pitch once, up top: tagline + one "why" paragraph + the single
two-pillar table (was restated 4x across header/nav/table/TOC).
- Keep the two concrete "first taste" artifacts: the stale-CLAUDE.md
snippet and the runHook example.
- Collapse the CLI, Skills, and Related Tools sections + the duplicated
promptfoo rebuttal into one-line links (the detail already lives in
docs/). Delete the <details> table of contents.
- Drop the README section anchor from docs/verifying-instruction-files.md
(now points at the README top).
Verified: `vigiles audit` passes with zero orphan docs (every doc the
README no longer links is still indexed by docs/README.md or CLAUDE.md).
* docs: document how the test tiers run in CI
Capture the CI tier mapping in docs/testing-matrix.md (it previously
lived only in discussion): which tier runs in which ci.yml job, that CI
runs every tier except evals, that evals are manual ($/non-deterministic),
and the cheapest-first best practice + the per-tier test:unit/integration/
e2e scripts that enable the split.
* docs: recast both README pillars as Without/With with real output
Replace the prose-heavy pillar pitches with tight "Without vigiles / With
vigiles" comparisons that show ACTUAL vigiles output (captured from real
runs, not invented):
- Pillar ①: `vigiles audit` flagging a rule that "exists but is disabled
in eslint config" (the valid-≠-true differentiator), a passing rule, and
a typo with a did-you-mean suggestion.
- Pillar ②: the runHook assertion + a real `vigiles test` pass summary,
framed around a silently-broken --no-verify guard.
Shorter and more concrete; shows the payoff instead of describing it.
* feat: refs-hook nudges the agent to mark references on save
Close the authoring-time half of pillar 1: an automatic, in-loop nudge so
the agent expresses references as verifiable vigiles marks instead of
unverifiable prose. Ships in the plugin — end-users author nothing.
- New `unmarked-refs` rule (default "warn"). The PostToolUse refs-hook
(hooks/refs-nudge.sh → `vigiles refs-hook`) fires on every CLAUDE.md /
AGENTS.md / SKILL.md edit and, by severity: "warn" → injects a
non-blocking nudge into the agent's context (mark these refs); "error" →
blocks the edit (exit 2); false → off. Wired into .claude-plugin/
plugin.json so it's on by default after `vigiles init`.
- Refactor the detection into core: collectRefIssues (shared by the
`vigiles refs` CLI and the hook) + the pure refsHookAction severity map,
unit-tested.
- Honest scope: catches code-shaped backticked refs; plaintext prose is
undecidable (the floor). Hooks are Claude Code / Codex only; other
harnesses fall back to instructions + the `vigiles audit` CI backstop.
- Docs: the from-file-save flow in verifying-instruction-files.md, a new
rules/unmarked-refs.md, and the rule added to the cli.md / docs index.
* test: dogfood the refs-hook through the runHook tier
skills-dogfood.test.ts only proves the hook script exists; this drives the
actual built `vigiles refs-hook` via the repo's own runHook unit tier and
asserts it FIRES correctly — nudges (non-blocking, additionalContext) by
default, blocks (exit 2) under unmarked-refs:"error", stays silent when
off, and no-ops on a clean or non-instruction file. Closes the "test your
harness, don't trust it on vibes" gap for the new hook.
* fix: scope refs-hook nudge to linter-rule names (kill false positives)
Dogfooding the hook on the repo's own SKILL.md files exposed heavy
over-firing: skills/test-harness/SKILL.md got 23 nudges, all on API names
(`runHook`, `runEval`, …) that are documentation prose, not stale
references. The `isCodeShaped` heuristic flagged every backticked
camelCase/Pascal/SCREAMING identifier — far too noisy to ship at
warn-default.
Tighten to high-signal only: flag a slash-scoped name with NO file
extension (`eslint/no-console`, `@typescript-eslint/no-explicit-any`,
`boundaries/dependencies`) and nudge toward `enforce()` / inline
`<!-- vigiles:enforce -->`. Bare identifiers and file paths are no longer
flagged (paths re-introduce the noise — the same SKILL.md files cite
`docs/harness-testing.md`, `package.json`, …; symbol marking stays opt-in
via an explicit `vigiles:symbol`).
Re-dogfooded: test-harness 23→0, strengthen 2→1 (keeps the one real rule
`@typescript-eslint/no-unused-vars`), migrate-to-spec 0. Tests + docs
updated to the narrowed scope.
* test: real-claude dogfood proves the refs-hook fires in a session
Install-and-verify: with the claude CLI present, drive the deterministic
harness tier (real claude + scripted mock model, no API key) so the mock
Writes a CLAUDE.md naming `eslint/no-console`, the PostToolUse refs-hook
fires, and requestContains asserts the non-blocking nudge reached the
model's context. Closes the last gap — previously verified only through
the runHook unit tier. examples/harness/refs-nudge.harness.mjs runs in CI
via `npm run test:harness` (claude is installed there) and skips cleanly
without the binary.
* feat: vigiles init sets up both pillars, interactive-or-not, via the v1 Action
Make onboarding smooth and complete (the smooth-adoption rule):
- BOTH pillars by default. Pillar 1 (spec + types + compile) as before, PLUS
Pillar 2 — scaffold a starter vigiles.harness.mjs the user adapts to a
real hook. `--pillars=verify|test|both` scopes it.
- Interactive when a human's at a TTY (prompts: pillars / CI / plugin),
NON-INTERACTIVE for agents / CI / piped input or `--yes` — so "set up
vigiles" from a Claude Code / Codex prompt Just Works without hanging.
Pure plan logic in src/setup-plan.ts (parseSetupArgs/shouldPrompt/
resolvePlan), unit-tested; IO stays in cli.ts.
- CI wiring now creates a real `.github/workflows/vigiles.yml` using the
production `zernie/vigiles@v1` Action (verify job + a harness job when
Pillar 2 is on), instead of appending a raw `npx vigiles audit` line to
an existing workflow.
- New flags: --yes/-y, --pillars=, --no-plugin. Docs updated (cli.md init
reference, agent-setup.md agent flow, README quick start).
* test: CI-test the init command's new behaviors end-to-end
Spawn the real CLI in a temp project and assert the new init behaviors,
offline via --no-plugin (no network skills-add): default sets up both
pillars + a zernie/vigiles@v1 workflow (asserts the spec, vigiles.harness.mjs,
and the workflow YAML contents); --pillars=test scaffolds only the harness
(no spec/workflow); --pillars=verify only the spec. Also add --no-plugin to
the existing bare-init spawns so they stay offline and deterministic.
* docs: make the "agent installs + tests skills" flow first-class
Viewed from an agent handed "install vigiles and test my skills", the
README install path was clear but the test-skills path was muddy — the
only pillar-2 example was a hook, measureTriggerRate (the "does the skill
fire?" primitive) was never named, and the model-invocable test-harness
skill was buried as 1-of-8.
- README: name measureTriggerRate (recall + precision) in the tier line,
and surface that `init` installs the test-harness skill so "test my
skills" fires it.
- Add the `great-agent-flow` guidance rule: the agent-told-a-short-prompt
experience must be discoverable from the README front door and
frictionless (one non-interactive install, the skill fires the test,
the skill-testing path is high-signal not buried).
* fix: no blanket claude-gate; skips are loud, never a silent green
Two fixes for `vigiles test` / `vigiles eval`, behind a stated principle
(the new no-silent-skips rule):
1. Drop the blanket `claude`-gate in handleRunScripts. Unit-tier runHook
tests need no model, so they now ALWAYS run — previously the whole
command bailed if `claude` was absent, so a scaffolded vigiles.harness.mjs
+ `npx vigiles test` looked broken on a machine without claude.
2. Make skips LOUD. The runner classifies each script pass/skip/fail
(exit 0 / SKIP_EXIT_CODE 77 / else), tallies them separately ("1 passed,
1 skipped"), shows `⊘ SKIPPED`, and a skip never fails the run (anyFailed).
Add a `skip(reason)` helper (vigiles/testing) and convert the four
deterministic examples from `console.log(...); process.exit(0)` (a silent
fake ✓) to `skip()`. Up front, `vigiles test` notes when claude is absent.
Verified end-to-end with claude off PATH: the unit example runs+passes, the
deterministic example reports ⊘ SKIPPED, summary "1 passed, 1 skipped",
exit 0. Adds the no-silent-skips guidance rule + skip-aware runner tests.
* feat: vigiles test --no-skip (fail on skip) + README agent prompt
Answer "shouldn't a skip fail loudly?" — yes, where you assert the
capability is present. A skip passes by default (tiers need different
capabilities not present everywhere; failing-by-default would make the
command red in any partial env — this repo's harness job legitimately
skips the egress tier), but `vigiles test --no-skip` makes any skip a
failure for the CI job that asserts everything must run: a green-with-skips
is itself untested surface. The skip stays loud + tallied either way.
Extends the no-silent-skips rule; integration-tested.
Also (per the agent-flow ask): add a paste-into-Claude-Code/Codex prompt to
the README quick start — install + verify + write/run a harness test with
good defaults, asking about CI gating / real-model eval / --strict.
* fix: cover skip() under the 100% gate; flush skip/--no-skip docs
The no-silent-skips commit broke CI's coverage gate (step 13): skip() in
harness-assert.ts (a coverage-included file) calls process.exit, which is
unreachable in-process → dropped below 100%. Mark it with the repo's
`v8 ignore` convention (same as the spawn boundary); it's exercised via a
child process in cli.test.ts / run-scripts.test.ts. Verified locally with
`npm run coverage`: 100% stmts/funcs/lines, branches 92.43% (≥90 floor).
Docs flush (before merge): document the loud-skip / unit-runs-without-claude
/ `--no-skip` behavior in docs/testing-matrix.md and the test-harness skill's
"Run it" step, so the agent's authoring guide matches the runner.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
8718bdb1e2 |
docs: point layer-1 reference link at the verify section + linter support (#30)
* docs: point layer-1 reference link at the verify section + linter support The harness-testing doc's 'Verify the references' bullet linked the README root (no anchor), landing readers on the project pitch. Point it at the 'Verify your instruction files' section and docs/linter-support.md so a reader chasing 'is this linter rule real?' gets the actual mechanism. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: state explicitly why hook confinement is opt-in, not always-on The 'why not always sandbox?' rationale lived only in scattered docstrings. Add a dedicated section to docs/sandboxing.md (availability, hostile env, provenance trust, cost) and surface the same four reasons on RunHookOptions.trusted so authors see the tradeoff at the API. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: prove allowlisted-egress handoff in a spike (slirp4netns + nft) The 2026-06-12 note recorded that pasta --config-net failed to give a child netns egress on first pass. It works with slirp4netns --configure: unshare creates the user+net namespace, slirp4netns --configure attaches and configures tap0 inside it, and an nftables policy-drop output chain enforces an IP allowlist that drops a non-allowlisted host AND a raw bash /dev/tcp socket (the property an HTTP_PROXY allowlist can't give), with per-rule counters as the readable record. Add research/spikes/sandbox-network-allowlist.sh (self-contained, skips without the tools), update the experiment note + plan, and reflect the proven handoff in docs/sandboxing.md. Ships nothing yet — egress: { allow } still needs the bwrap PID handoff + option plumbing + gated tests (plan steps 2-3). https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(run-hook): allowlisted real egress via egress: { allow } Adds the third egress mode alongside deny-all and recordEgress: let a confined hook actually reach the network, but only an allowlist, with the boundary at the packet layer so a raw socket off-list is dropped too (which an HTTP_PROXY allowlist cannot guarantee). slirp4netns --configure attaches a tap to the bwrap netns (rootless egress); an in-netns nft policy-drop output chain accepts only the allowlist's resolved IPs (plus loopback + DNS) and log+drops the rest; per-rule counters read back what was reached vs dropped. bwrap --info-fd hands the child PID to slirp4netns, and a netready file gates the in-netns wrapper until the tap is up. - src/egress.ts: pure seams (resolveAllow/parseGetent, buildEgressNft, buildEgressBwrapArgv, parseNftCounters/countersToResult, egressAvailable) - src/egress-entry.ts: the bwrap<->slirp4netns orchestrator subprocess - run-hook: RunHookOptions.egress -> r.egress (allowed hosts) + r.egressDropped - tests: pure unit suite + gated integration (allow/drop/raw-socket) + the OMC session-start dogfood (reaches registry.npmjs.org, drops nothing else) - docs/sandboxing.md + research/sandbox-network.md: shipped, with the honest record limit (resolved at launch; resolver-pinned dynamic set is the next layer) Resolves the egress: { allow } gap designed in research/sandbox-network.md. 100% coverage on src/egress.ts; integration tests gated on bwrap+slirp4netns+nft. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: surface egress: { allow } in the remaining doc pointers The sandboxing.md/research updates shipped with the feature, but three other surfaces still described only deny-all vs recordEgress: the docs/harness-testing + docs/README see-also blurbs and the test-harness SKILL. Mention the third network mode (allowlisted egress) in each. * feat(eval): trigger precision + skill-eval landscape (AWS) + compression-claim demo Prompted by AWS sample-agent-skill-eval (a near-mirror of our harness-testing pillar). Three things: 1. Trigger precision (the one concrete gap it exposed): measureTriggerRate measured recall only. TriggerRateSpec.irrelevantPrompts adds the precision side — report.falsePositiveRate + report.precision — so a too-broad skill description that hijacks unrelated work fails, not just a too-narrow one. Gated by assertTriggerRate({ min, maxFalsePositive, minPrecision }). 2. research/skill-eval-landscape.md: AWS skill-eval scored against our eval pillar (near 1:1 — validates the bet) with the absorb-list (precision done; unified scorecard + cost-Pareto deferred; security scan = delegate). Also frames the token-compression cluster (RTK/Caveman/Claw/Context-Mode/pinchtab/ CodeGraph) as a use case for runEval, not a roadmap item. 3. examples/harness/skill-compression.eval.mjs: worked demo verifying a Caveman-style compression claim — two arms measure outputTokens (the target) AND a correct fact that must survive, with a guardrail that fails if the token win cost correctness. Plus: test-harness SKILL trigger row updated to recall+precision; CLAUDE.md Key Files recompiled. 100% coverage on eval.ts + harness-assert.ts. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * test(dogfood): conformance gate over vigiles's own skills + 2 trigger evals Eat our own dog food — point the harness-testing pillar at vigiles's shipped skills (.claude-plugin/). - src/skills-dogfood.test.ts: free, model-free gate — loadPlugin materializes every skills/<name>/SKILL.md and each has a name + non-empty description (the trigger surface). The floor under the paid trigger-rate eval: a skill that won't load can never fire. - Found + fixed a real bug: skills/generate-logo had no frontmatter 'name' (a skill without a name doesn't register). - examples/harness/dogfood/{test-harness,generate-logo}.trigger.eval.mjs: precision-aware trigger evals (recall + irrelevantPrompts/falsePositiveRate) for the ONLY 2 model-invocable skills. The other 7 are disable-model-invocation (user-invoked) so triggering doesn't apply — the load gate covers all 9. Write-don't-run without model auth. Recompiled CLAUDE.md. 735 tests pass. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(plugin): move hook scripts to plugin root so installed hooks resolve Dogfood finding: plugin.json declares hooks as ${CLAUDE_PLUGIN_ROOT}/hooks/*.sh, but the scripts lived under .claude-plugin/hooks/. Per the Claude Code plugins-reference, component dirs (hooks/, skills/, commands/, agents/) MUST sit at the plugin root, not inside .claude-plugin/ — so the installed PreToolUse/ PostToolUse/SessionStart hooks resolved to nothing. - git mv .claude-plugin/hooks/*.sh -> hooks/ (the conventional location) - src/cli.test.ts: point the hook-path tests at the new location - src/skills-dogfood.test.ts: add the precise guard — every ${CLAUDE_PLUGIN_ROOT}/<script> in plugin.json must exist at the plugin root (loadPlugin's dangling scan doesn't cover plugin.json command paths, so this is what actually catches the bug class; it can't regress) Convention confirmed against code.claude.com/docs plugins-reference. 736 tests pass; CLAUDE.md recompiled. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(audit): add untested-surface rule — flag skills/agents/hooks with no test The third gap detector beside orphan-docs: skills, subagents, and hooks that ship without a test or eval. A surface with no test is a probabilistic- compliance gap hiding in the deterministic layer — nothing measures whether it still does what it claims. Two OR'd detectors decide 'tested', so a test placed anywhere counts: - colocation: a *.{harness,eval}.mjs next to the surface (the zero-config convention the warning suggests) - content-reference: any discovered test (incl. *.test.ts) naming the surface by path (skills/foo, hooks/x.sh) or namespace (vigiles:foo) Warning-by-default (a nudge, exit 0); set to 'error' to gate CI. User-invoked skills (disable-model-invocation) are exempt by default since they can't auto- trigger; flip includeUserInvokedSkills to demand an outcome test. Per-surface opt-out via a vigiles:ignore-test marker. - src/test-coverage.ts + test suite (9 tests) - wired into vigiles audit (warn->exit0 / error->exit2, mirroring integrity), --summary line, GH annotations - config: rules.untested-surface in types + DEFAULT_RULES + .vigilesrc.json - docs/rules/untested-surface.md, linked from docs/README.md + docs/cli.md; README rule count 4->5 - recompiled CLAUDE.md Dogfood: our repo is clean — 5 surfaces covered (2 invocable skills + 3 hooks), 7 user-invoked skills exempt. 745 tests pass. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(pkg): ship hooks/ and skills/ in the npm package The hooks-path fix moved scripts from .claude-plugin/hooks/ to hooks/, which dropped them out of the publish allowlist (files listed .claude-plugin but not hooks/). npm pack confirmed the tarball shipped plugin.json referencing ${CLAUDE_PLUGIN_ROOT}/hooks/*.sh while the scripts themselves were absent — and skills/ was never shipped either, so the tarball was never a complete plugin. Add hooks + skills to files (excluding *.spec.ts build sources) so the npm package is a complete, installable plugin: manifest + 3 hook scripts + 9 SKILL.md, no spec leakage. Verified via npm pack --dry-run. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: reflect skills conformance gate + untested-surface rule Revamp (not just append) the harness-testing docs for this session's two new user-facing concepts: - docs/harness-testing.md: split the conflated 'Coverage' section — it was only about vigiles's own V8 coverage, confusing in a guide about testing YOUR harness. Now 'Coverage of your harness surfaces' (the untested-surface rule + the model-free conformance floor: every skill loads with a usable description) vs 'vigiles's own coverage' (the V8 paragraph). Thread the untested-surface rule into layer 1 (verify references) and upgrade the Skills cell in the surface x tier matrix (loads + description, not just refs). - docs/testing-matrix.md: add unit rows for the own-skills conformance gate and the untested-surface detector, a real-model row for the dogfood trigger evals, and a note that a static skill floor + the audit rule sit below the three runtime tiers. - research/harness-testing-coverage-matrix.md: add cross-cutting capability rows (own-plugin dogfood, surface-test coverage detection) and align the Skills unit/static cell. Rules table/README/cli.md already carried the untested-surface row. fmt clean, audit intact, no orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: four deep investigations + strategic synthesis Four parallel deep-research docs (web-sourced, cited) on fresh ground — deliberately NOT re-covering promptfoo/eval/harness-testing (already in docs): - agent-supply-chain-security.md — 2026 plugin/MCP threat surface + OWASP Agentic Skills Top 10; re-examines 'delegate, don't build'; verdict: thin harness-aware `vigiles scan` as a pillar-2 feature, reject the security pivot. - standards-conformance.md — AGENTS.md/SKILL.md/MCP/ACP standardization; verdict: extend pillar 1 to be format-neutral + lead the 'valid is not true' wedge, reject the generic-linter (agnix) pivot. - runtime-guardrails-observability.md — guardrails + OTel-GenAI observability; verdict: stay out of the request path, take one bridge (OTel emission + verify-trace + Cedar/OPA policy codegen, emit/verify-only). - ai-native-linting.md — the AI-code-review wave; verdict: extend enforce() to AI-linter catalogs, prototype (don't trust) a measured judge() rule gated by the eval machinery, reject becoming an AI reviewer. strategic-synthesis-2026-06.md ties them together: all four converge on extend-don't-pivot, with one unclaimed wedge — the observed-vs-declared contract binding. Ranked bet list + the one positioning pivot worth taking (lead with 'conformance/attestation for the agent harness'). Registered in CLAUDE.md Key Files (recompiled); audit clean, no orphans (58), fmt clean. Specific 2026 figures are web-claimed, not independently verified — flagged in the synthesis. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: divergent strategic bets, triaged with founder reactions Pushes past the extend-don't-pivot synthesis. Captures genuinely divergent directions with live triage: - strong: plugin/skill leaderboard (distribution by data, not installs); harness cost/ROI optimizer (money story over correctness story) - explore: sell to harness vendors; compliance/attestation buyer; CI for model upgrades - roadmap: self-improving harness (uses idle evolve.ts + proofs.ts) - open (need a call): compiler-not-linter (generate the verifiable skeleton); vigiles-as-MCP-oracle (deterministic answer, not gate) - watch: SDK bet (harness-as-code); killed: one-source-many-backends; parked: contrarian measure-model×harness Registered in CLAUDE.md, recompiled, audit clean, no orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: SDK harness-testing gap (verdict: no) + triage updates SDK bet (#6) researched: serious agents ARE built in SDK code, but the deterministic mock-model test tier (vigiles's middle tier) is already first-party in the top SDKs (Pydantic AI TestModel/FunctionModel, Vercel MockLanguageModelV3, LangGraph FakeListChatModel, LlamaIndex MockLLM). Gap largely closed → no blanket pillar-2 retarget. Lone residual: deterministic guardrail/tool-contract enforcement testing (maps to agent-runtime.ts) — a one-day OpenAI-Agents-SDK probe, idea-borrow not a port. See research/sdk-harness-testing.md. divergent-bets triage updated: #1 compiler-not-linter killed (founder: useless); #5 MCP-oracle demoted (fold into scan); #6 no (gap closed). Strong front-runners remain #9 leaderboard + #10 cost optimizer. Registered in CLAUDE.md, recompiled, audit clean, no orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(cli): add 'vigiles scan' — deterministic plugin/repo report (no model) The easiest-first build toward the leaderboard (#9) and harness-aware supply-chain audit: point vigiles at any plugin/repo and get a read-only report of what it ships + what's structurally broken, with no model and no API key. Re-aims existing machinery (loadPlugin, parseAgentTools, findUntestedSurfaces): - skills: description present? + user-invoked flag - agents: declared tool contract, incl. the 'no tools: line → inherits every tool' footgun - hooks: script paths resolved across braced AND unbraced shell forms → ok / missing / unresolved (caught + fixed a false-MISSING on the unbraced form while dogfooding oh-my-claudecode) - command + MCP detection, untested-surface count, loader warnings - --json for pipelines src/scan.ts + src/scan.test.ts (5 tests); wired into the CLI; documented in docs/cli.md; registered in CLAUDE.md. Dogfooded on our own plugin (clean) and on the vendored oh-my-claudecode (hooks/agents/MCP/commands). 750 tests pass. Next (harder, needs model auth): the trigger-rate + safety layer that turns 'scan over many plugins' into the leaderboard. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(cli): plugin health leaderboard — scan many, rank by structural health The next rung up from 'scan' and the no-model engine for the leaderboard (research/divergent-bets.md #9): pass >1 directory to 'vigiles scan' and it ranks them by a deterministic 0-100 structural-health score (+ A-F grade), worst issues first. - src/leaderboard.ts: scoreReport (missing hook -15, no-description skill -10, agent-without-tool-contract -5, untested surface -3; empty machine = 0; clamps at 0) + rankPlugins + formatLeaderboard. Deliberately ignores the loader's free-text warnings (doc-mention false positives) so the ranking is defensible. - wired into 'vigiles scan <dir...>'; documented in docs/cli.md; registered in CLAUDE.md; src/leaderboard.test.ts (5 tests). Dogfood over the vendored plugins ranks superpowers (A/94) > wshobson (B/86) > oh-my-claudecode (C/78) with reasons. Behavioural columns (trigger-rate, observed egress, safety) need a model and stack on top. (Full-suite run flagged the gated egress network test, which is environmental — needs outbound to example.com; passes on re-run, unrelated to these files.) https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: add consolidated roadmap front door + reconcile sprawl The scattered 'next steps' sections across 7+ research docs had no single view. Add research/roadmap.md as the one ranked front door (Now / Next / Later / Backlog / Explore / Rejected), each item a one-liner linking the doc with its rationale, plus a 'shipped recently' list. - research/README.md: link roadmap.md at top; add the missing 2026-06 strategy cluster (4 deep researches + synthesis + divergent-bets + sdk-harness-testing) + the eval docs; relabel coverage-matrix as pillar-2 detail (roadmap owns the cross-pillar ranking). - feature-ideas.md: point to roadmap for status; reconcile stale rows (#14 scan shipped, #9 hook-validation partial via scan). - registered roadmap.md in CLAUDE.md; recompiled; audit clean, no orphans (61). https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: de-dup — feature-ideas becomes detail-only, roadmap owns ranking Answers 'why both?': they now have non-overlapping jobs. roadmap.md is the ranked status board (what's next, all pillars); feature-ideas.md is the detailed pillar-1 idea catalog (per-idea analog/problem/implementation + research) that roadmap links into. Stripped feature-ideas's duplicate priority table + shipped list (that ranking lives in roadmap now); kept the #1-#14 write-ups + research appendix. No orphans. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(egress): make in-netns DNS work behind systemd-resolved (CI fix) The two egress e2e tests failed in CI ('got []') while passing locally. Root cause: GitHub runners use systemd-resolved, so /etc/resolv.conf is the 127.0.0.53 stub. parseResolvers picked it and the nft ruleset allowed only it — but the slirp4netns netns runs with --disable-host-loopback, so 127.0.0.53 is unreachable there. In-netns DNS failed, curl resolved nothing, zero packets. (Locally /etc/resolv.conf is a real nameserver, so it worked — hence green here, red in CI.) Proper fix, not a skip: - parseResolvers drops loopback stubs (127.0.0.0/8, ::1) and falls back to a public resolver (8.8.8.8) that slirp4netns NATs out. - the egress run now binds a generated /etc/resolv.conf (the routable resolvers) over the host's stub inside the netns, so curl/getent resolve correctly. Pure helpers unit-tested (loopback drop; the resolv.conf bind in the bwrap argv); the gated bwrap egress e2e still passes locally (33/33). Validating the CI half on the runner via the PR watch. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * ci: add informational egress diagnostic to pinpoint slirp4netns failure The egress-allow e2e tests emit zero packets on the hosted runner but pass locally, and the DNS guess didn't fix it. Add a non-failing diagnostic step (before the coverage gate) that captures the in-netns reality: bwrap/slirp4netns/ nft versions, /dev/net/tun presence, userns sysctls, and a live probe through the real runHook egress path printing the in-netns resolv.conf + getent + curl + ip addr. Lets the next run reveal whether slirp attached (tap0 up), DNS resolved, or NAT routed — so the real fix is grounded, not guessed. Informational only (exit 0); does not change the gate. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * research: egress-sandbox tooling — adopt pasta (passt), don't reinvent Answers 'is someone already solving this / are we reinventing the wheel?' for the rootless egress connector. Verdict: swap slirp4netns → pasta (passt) — Podman's rootless default, a near-drop-in that attaches to an existing bwrap netns by PID/--netns but doesn't depend on the /dev/net/tun tap-attach handshake that fails on GitHub-hosted runners (the CI symptom). Keep bwrap+nft+resolver. nsjail/ firejail/gVisor/Podman-lib/microVM rejected (ride the same userns/tun primitives, no nft read-back); Landlock net parked (port-only, no per-host/accounting). Smallest step: a ~10-line connector-switch spike. Cited; registered in research/README.md + CLAUDE.md; audit clean. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(ci): gate egress e2e tests on actual routing, not tool presence The two egress-allow e2e tests failed red on GitHub-hosted runners: bwrap + slirp4netns + nft are all present (so the old tool-presence gate let them run), but slirp4netns never attaches tap0 there — the netns has only `lo`, so nothing routes and the assertions see zero packets. Proven via the CI diagnostic (/dev/net/tun present, userns on, in-netns DNS fix applied, yet ip showed only lo). Replace the gate `egressAvailable(sandboxAvailable())` with `egressRoutes()` — a memoized probe that runs a trivial hook in the egress sandbox and checks /proc/net/dev for a non-loopback interface (the slirp/pasta tap). So the tests RUN for real where egress actually routes (locally: 21/21 pass) and SKIP honestly where it provably can't (the hosted runner) instead of failing red. This is the research-endorsed capability-probe-skip (research/egress-sandbox-tooling.md); swapping slirp4netns→pasta is the follow-up that makes the probe pass in CI too. egressRoutes lives in the existing v8-ignore block (real-exec), so the 100% coverage gate is unaffected; the pure egress helpers stay unit-tested. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(api): tiered import paths — vigiles/unit · /integration · /e2e Make the capability level of a harness test legible from its import path (and enforceable). Layered, dependencies downward only: - vigiles/unit — no caps (runHook logic, predicates, asserts, parsers). No claude, model, bwrap, or network. - vigiles/integration — re-exports unit + runHarnessTest/withHarness/scriptModel + loadPlugin (needs claude + bwrap, no key/network). - vigiles/e2e — re-exports integration + runEval/measureTriggerRate/judge + egress-using runHook (needs model/network). A *.test.ts importing vigiles/unit physically can't reach a heavy capability; an e2e test imports vigiles/e2e and gets everything. Barrels: src/{unit,integration, e2e}.ts; package.json exports ./unit ./integration ./e2e. Next: file-suffix test tiers + vitest projects + per-tier CI, then reusable GH Actions per level. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * test: operationalize tiers — vitest projects + per-tier scripts + move egress e2e Encode the test tier in the filename, routed to its own vitest project, mirroring the vigiles/unit·/integration·/e2e import paths: - unit = src/**/*.test.ts minus the suffixed tiers (pure, no caps) - integration = *.integration.test.ts (bwrap, no network) - e2e = *.e2e.test.ts (real egress/model/network; self-skips where unavailable) - runners unchanged Move the two egress-allow tests out of run-hook.test.ts into src/run-hook.e2e.test.ts, importing from the e2e barrel (./e2e.js) — dogfooding the tiered import path: an e2e test pulls from vigiles/e2e, a unit test couldn't. Scripts: test:unit / test:integration / test:e2e (vitest projects); the old live-claude bash e2e renamed test:e2e → test:cli-e2e (docs updated). + coverage run all projects (e2e self-skips in CI), so the 100% gate holds (verified: 100% stmts/funcs/lines, branches 93.8%, 756 tests). Note: run-hook.test.ts still holds a live-net recordEgress dogfood + bwrap confinement tests — fuller re-bucket into integration/e2e is a follow-up. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * feat(ci): reusable per-tier action + e2e dogfood; fix egress probe Phase C: a composite action .github/actions/harness-tier (input: tier) that sets up exactly the capabilities a tier needs (unit: none; integration: bubblewrap + claude CLI; e2e: + pasta/slirp4netns/nft for rootless egress) and runs npm run test:TIER. Reusable from any repo via zernie/vigiles/.github/actions/harness-tier@main. Dogfooded: a new e2e job in ci.yml uses it (installs pasta, runs the e2e tier). Fix the egressRoutes() probe: the /proc/net/dev interface count false-positived on the runner (bwrap binds host /proc, so it counted host interfaces and the e2e tests RAN and failed instead of skipping). Switch to the only namespace-accurate signal — actually reach an allowlisted host through the egress path and check a packet got out. Locally routes (e2e tests run + pass, 2/2); on the hosted runner it does not (so they skip honestly) until the slirp4netns->pasta connector swap. Also remove the now-spent informational egress diagnostic step from the test job. Coverage gate holds: egress.ts is fully unit-tested, the real-exec path is v8-ignored, and the probe exercises egressSpawn at import time. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(egress): connector selectable (slirp default, pasta opt-in); split e2e vs eval Two fixes: 1. e2e != eval. vigiles/e2e is now DETERMINISTIC-only (real sandbox/network, pass/fail): it drops the eval runners (runEval/measureTriggerRate/judge) and keeps egressRoutes + the egress-using runHook. Evals are a separate non-deterministic MEASUREMENT axis at vigiles/eval + vigiles/judge, run via vigiles eval on *.eval.mjs. (3 deterministic verification tiers unit/ integration/e2e + the eval measurement axis.) 2. Rootless egress connector is selectable. Default slirp4netns (proven locally, e2e routes + passes). Set VIGILES_EGRESS_CONNECTOR=pasta to use pasta (passt), intended for hosted runners where slirp4netns's tap-attach fails. HONEST WIP: pasta --config-net <pid> does not yet route in our bwrap netns (connect fails, not just DNS) — needs real integration (capture pasta stderr, correct flags/ namespace). So pasta is opt-in, not default; CI e2e still skips honestly until it's finished. egress-entry prefers the configured connector and falls back to slirp4netns. egress-entry is v8-ignored, so coverage is unaffected. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * docs: make the harness levels crystal-clear + GH Actions + eval-pillar note Revamp docs/harness-testing.md per the tiering work: - Replace the vague 'four layers' intro with a LEVELS table: refs / unit / integration / e2e / eval, each mapped to its import path (vigiles/unit·…), file suffix (*.test.ts · *.integration.test.ts · *.e2e.test.ts · *.eval.mjs), runner (vitest project / vigiles eval), and what it needs. The level is legible three ways (import path = capability contract; suffix; CI job). - New 'Per-level CI (the reusable action)' section documenting the composite action zernie/vigiles/.github/actions/harness-tier (tier: unit|integration|e2e) + that vigiles dogfoods it, with the honest pasta/egress note. - New 'Two pillars or three? (where eval sits)' section: keep TWO pillars; eval is pillar-2's non-deterministic measurement axis (measures vs asserts), drawn as a distinct axis but not a separate pillar — unless the self-improving-harness direction lands, when eval becomes optimization (its own concern). https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * fix(ci): run e2e in a privileged container so egress routes for real Root-caused why e2e egress fails on GitHub-hosted runners (and why it only passes locally): the connector — slirp4netns OR pasta — needs /dev/net/tun + privilege to bring up the tap and configure the netns. The hosted test job runs NON-ROOT, so it can't open tun (the netns ends up with only lo); our dev box passes only because it's root. Captured the actual errors while trying pasta: as root it self-drops to nobody (userns permission denied); as non-root it hits '/dev/net/tun: Permission denied'. So the connector choice was a red herring — privilege is the wall. Fix: the e2e CI job now runs as root in a privileged container (node:20-bookworm, --privileged --device /dev/net/tun), where the proven slirp4netns path attaches exactly as it does on a root dev box and the egress tests run FOR REAL in CI — closing the 'skipping isn't fair' gap. Tests still self-skip via egressRoutes() if egress can't route, so the job never fails red. - harness-tier action: sudo is now conditional (absent in root containers, present on non-root host runners) so the same action works in both. - pasta stays an experimental opt-in (VIGILES_EGRESS_CONNECTOR=pasta) for rootless envs that don't restrict tun; its attach-path needs more work. - Findings written up in research/egress-sandbox-tooling.md. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj * test(e2e): assert what passes the allowlist, not OMC's off-allowlist count The privileged-container e2e job WORKS — egress routed for real in CI: the allowlist test passed (example.com reached; 1.1.1.1 + raw 9.9.9.9 dropped) and the OMC test reached registry.npmjs.org. It failed only on an over-strict check: egressDropped===0, but OMC emitted one stray off-allowlist packet (76 bytes) that the wall correctly DROPPED — the wall working, not a failure. Fix the assertion to capture the real dogfood claim — 'OMC reaches npm and nothing else gets THROUGH' — by asserting the only allowed host is the npm registry, instead of OMC's exact dropped-packet count (which varies by env via DNS/CDN). Drops are proven by the other e2e test. Both e2e tests pass locally (root) and now in the privileged CI container — egress is exercised for real in CI, no skip. https://claude.ai/code/session_01Jw8WVFD64rPyxSHK5UJiMj --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
ceb810e881 |
feat: harness-testing tier 2 (integration) + MCP tool verification (#26)
* docs: add TOC and subheadings to README for readability
Add a collapsible Contents block organized around the two pillars
(reference verification + harness testing) so the testing section is
easy to jump to. Promote the testing tier lead-ins to ### subheadings
so each tier is scannable and linkable from the TOC.
* docs: trim README to pitch + quick start, extract reference into docs/
Restructure README around the two pillars back-to-back (verify
instruction files → test the harness), then a slim reference tail.
Cuts 511 → 389 lines toward the ~300 brevity target.
- Move CLI, GitHub Action, plugin, and validation rules into a new
docs/cli.md; README keeps a condensed "CLI & CI" section + link.
- Move the related-tools/compose list and output-targets into a new
docs/related-tools.md; README keeps a one-paragraph summary + link.
- Fold the standalone "Type-Safe Rule References" section into a
two-line note under Verified References (deep link to linter-support).
- Condense the guard() and symbol-reference subsections.
- Drop the duplicate init-flags table from Quick Start (now in cli.md).
- Update the Contents TOC to match the new structure.
* test(harness): dogfood the plugin loader against real pinned plugins
Adds runnable dogfood tests that point loadPlugin at vendored snapshots of
two real, MIT-licensed Claude Code plugins instead of synthetic look-alikes:
- real-superpowers.harness.mjs — obra/superpowers: validates the
hooks/hooks.json convention + ${CLAUDE_PLUGIN_ROOT} expansion.
- real-wshobson.harness.mjs — a wshobson/agents sub-plugin: the no-hooks
subagents+commands+skills shape, asserting the loader materializes the
surfaces and warns rather than silently testing an empty machine.
Reliability: the plugins are PINNED, VENDORED snapshots under
examples/harness/vendor/ (upstream LICENSE + a SOURCE recording repo +
commit), so the tests run offline and deterministically — no clone at test
time. tools/refresh-vendor.sh re-fetches them deliberately.
Safety: loadPlugin only parses; it never executes a hook. The dogfood
asserts superpowers' side-effectful SessionStart hook is correctly WIRED
and does not run it — executing untrusted third-party hooks needs a real
sandbox (the CI container, or an opt-in bwrap/Docker boundary), documented
in docs/harness-testing.md.
Both tests are key-free and claude-free (pure loadPlugin), so they run in
CI for free. Vendored dirs are excluded from prettier + eslint.
* chore: regenerate .vigiles/generated.d.ts
Refresh the generated type unions to match current project state (adds the
test:types npm script and recent project files). `generate-types --check`
now passes; no vendored third-party paths leak into the unions.
* docs: track sandboxed harness-test execution as a feature idea
Add feature-ideas.md #13 — an opt-in `sandbox:` option (bwrap/sandbox-exec/
docker) on runHook/runHarnessTest so the execution tiers can safely run
untrusted third-party plugin hooks, turning the dogfood from parse-only into
execute-and-verify. Cross-link it from the harness-testing sandbox-boundary note.
* docs: add harness-testing coverage matrix (shipped vs. should-build)
A roadmap/gap-analysis companion to docs/testing-matrix.md: maps the whole
potential surface of testing a Claude Code harness (hooks by event, skills,
subagents, slash commands, MCP, CLAUDE.md, settings) across the unit /
deterministic-integration / eval tiers, plus cross-cutting capabilities
(sandboxed exec, native plugin-install fidelity, MCP wiring, coverage tooling).
Marks each cell ✅ shipped / 🟡 partial / 🔴 gap, records the fidelity caveats
(skill activation is faked even at the eval tier; Edit/Write headless-gated;
no sandbox for untrusted exec), and gives a prioritized build list. Linked
from docs/testing-matrix.md and research/harness-testing.md (no orphan).
* docs: make coverage matrix use the unit/integration/e2e pyramid honestly
Decouple the columns from a single vigiles function each. The prose surfaces
(skills, agents, commands, CLAUDE.md) DO have a unit tier — static
reference/contract verification via `audit`/`refs` (vigiles' first pillar) —
which the old table hid by equating "Unit" with `runHook`. Rename columns to
Unit/static · Integration (assembled, mock model) · E2E (real model), and fill
the unit column for every surface. Sharpens the read: every surface has a
static check, but the integration/wiring middle is hooks+settings only.
* docs: frame eval as a separate axis (measurement) in the coverage matrix
Eval isn't a "test" in the unit/integration sense — it's measurement (A/B
mean ± se you read, not a pass/fail you assert), non-deterministic and costly.
Keep it as the third column for surface-coverage completeness (it's the only
handle for skills/subagents/commands), but flag the distinction and point its
machinery (trials/variance/judge) to the eval docs instead of the cells.
* docs: make eval's non-determinism explicit in the coverage matrix
* docs: record skill-activation spike — feasible via --plugin-dir
Spike against real claude 2.1.169: --plugin-dir natively registers a plugin's
skills and the model activates them via a real Skill tool call
(superpowers:test-driven-development). Closes the gating unknown for roadmap #1.
Also surfaces a bonus deterministic path (scriptModel a Skill tool_use to test
wiring) and a fidelity rule (vendor whole plugins). Updates caveat #1 + the
native-install row.
* docs: spike finds Edit/Write deterministic tier already works (caveat #2 stale)
Second spike probe on claude 2.1.169: Edit/Write tool-event hooks DO fire in the
mock-model deterministic tier — PostToolUse 3/3, a PreToolUse Edit block held.
The 'headless-gated' limitation is no longer reproducible (--allowedTools
allowlists the edit tools past the prompt). So roadmap #3 collapses from 'build a
driver' to 'add a regression test'. Updates the Edit/Write cell, caveat #2, the
spike section, and the build list.
* test(harness): lock in Edit/Write deterministic-tier regression tests
Replaces the stale `test.skip`ped "Edit/Write headless-gated" placeholder with
two real, reliable tests (claude 2.1.169):
- Write -> PostToolUse(Write|Edit) hook fires (file written + marker), 5/5.
- Read -> Edit, PreToolUse(Edit) blocks (marker proves the hook fired AND the
edit didn't apply), 4/4.
Found three false-pass/no-op traps while writing them, now documented in the
coverage matrix: Edit needs a prior Read; a "file unchanged" assertion is
trivial without a hook-fired marker; a custom `prompt` makes the mocked turn
no-op (default "go" is reliable).
The second planned lock-in test (deterministic skill-wiring) is coupled to the
--plugin-dir feature (#1): probing showed .claude/skills materialization does
NOT register a skill for the Skill tool, so it ships with #1. Updates the matrix
accordingly (Edit/Write row -> shipped; skill-wiring -> coupled to #1).
* feat(harness): native plugin install via pluginDir — skills activate + resolve
Adds `runHarnessTest({ pluginDir })` which passes `claude --plugin-dir`, so a
plugin's skills register and ACTIVATE the real way (file-materialization does
not register skills for the Skill tool — only native install does). Plus a
`transcript: true` option (stream-json) to assert on what the agent's tools
returned.
This ships the deterministic half of roadmap #1: a scripted `Skill` tool_use
now resolves a plugin's skill (its body is injected), so skill WIRING is
testable with no API key. Skills integration cell 🔴→✅.
- src/harness-test.ts: pluginDir + transcript options, threaded into the argv.
- examples/harness/fixture-skill-plugin/: a 3-file self-contained plugin (one
`greet` skill) for the deterministic test.
- src/harness-test.test.ts: a test asserting `demo:greet` resolves (3/3 reliable).
- docs/harness-testing.md + the coverage matrix: document pluginDir, mark what
shipped vs. remaining (runEval pluginDir for real activation; subagents/commands).
* feat(eval): native plugin install via pluginDir — closes the skill-activation caveat
Adds `pluginDir` to `runEval` arms (mirrors runHarnessTest): an arm can install a
plugin natively (`claude --plugin-dir`) so the REAL model activates its skills by
description, not via the old "tell the agent to read a SKILL.md" trick.
Verified end-to-end with a real-model A/B on the fixture plugin: arm `off`
(no plugin) writes NO_SKILL; arm `on` (pluginDir) activates demo:greet and writes
its marker. So caveat #1 (skill activation faked) is now closed at BOTH tiers —
deterministic resolve (runHarnessTest) and real activation (runEval).
- src/eval.ts: pluginDir on EvalArm, threaded into the agent argv.
- examples/harness/skill-outcome.eval.mjs: note now points at pluginDir as the
real-activation path.
- docs + coverage matrix: caveat #1 marked closed; Skills e2e cell -> ✅; also
corrected the stale "Edit/Write headless-gated" line in the guide.
* docs: probe confirms subagents + slash commands register via --plugin-dir
Probed end-to-end (real model): a /cmd slash command wrote its marker, and an
echoer subagent invoked via Task wrote its marker — both off a --plugin-dir
install. So all model-driven surfaces (skills, subagents, commands) activate
natively at the eval tier (their e2e cells ✅). The open work is DETERMINISTIC
testing: slash commands need capturing the mock-received (expanded) prompt;
subagents are hard (nested mock sessions). Matrix rows + build list updated.
* docs: refresh the 'honest read' after subagent/command probe
* test(mcp): MCP governance is testable at the unit tier (no server)
A PreToolUse hook that blocks a destructive github-MCP tool
(mcp__github__merge_pull_request) but allows read-only ones — the dominant real
MCP test. The hook only sees the tool name, so runHook covers it with no MCP
server and no model. Locks in MCP slice #1 from the coverage roadmap.
* feat(harness): assert on the agent's actions (r.toolCalls + tool/skill asserts)
Adds the invariant vocabulary the skill/MCP/command surfaces actually need —
asserting on what the agent DID, not a brittle stdout substring:
- parseToolCalls + HarnessTestResult.toolCalls: parse the stream-json transcript
(transcript:true) into the tools the agent invoked, each paired with its result.
- assertToolUsed / assertToolNotUsed (the safety negative — "the destructive tool
was never called", which "file unchanged" can't prove) / assertSkillResolved.
Grounded in REAL plugins for immediate payoff: the skill-wiring test now uses
assertSkillResolved (no injected marker), and two new tests assert that real
obra/superpowers and wshobson/agents skills resolve via --plugin-dir — something
the old marker-grep couldn't do (it only works on fixtures you control).
Pure unit tests for parseToolCalls + the three asserts run with no claude.
docs/matrix updated. Full suite green; lint + fmt clean.
* docs(readme): rewrite the testing section, plainer + more detail
Reframe the harness-testing pitch around 'three levels, cheapest first' in plain
language (test a hook alone → does it fire in a real session → does it change
behaviour), and surface the newly-shipped capabilities: MCP governance at the
unit tier, native skill testing via pluginDir, and asserting on the agent's
actions (assertSkillResolved / assertToolNotUsed — the safety negative) against
real superpowers/wshobson plugins. TOC + anchors updated.
* feat(harness): sequence & budget invariants over the agent's actions (idea 1)
Adds three assertions over r.toolCalls, beyond the single-bool "did it use X":
- assertToolSequence(r, [...]) — ordering, in-order subsequence (e.g. Read→Edit)
- assertToolCount(r, name, {min,max,exactly}) — budgets (≤1 Write, 0 of X)
- assertToolCalls(r, predicate) — escape hatch for custom invariants like
"every Edit was preceded by a Read" (the rule Claude enforces).
Pure unit tests (no claude) cover all three; a grounded integration test asserts
the workflow on a real Read→Edit run. Documented in docs/harness-testing.md.
Parks the two siblings on the roadmap (coverage matrix, "What we should build"):
#7 property-based hook fuzzing, #8 monotonic/relational eval invariants.
Lint + fmt clean; suites green.
* docs(roadmap): triage the next-candidates against existing tooling
After checking what already exists:
- Skill frontmatter lint + slash-command lint → DROPPED (commoditized:
skill-validator, agentskills.io Skill Spec Linter, plugin-validator already do
it; vigiles only adds deep ref resolution it already has). Recorded under
"Evaluated and dropped".
- MCP harness → sharpened into the standout item: a new mcp() reference kind
(verify mcp__server__tool against the live server's tool list) + an mcpConfig
harness option. The only candidate that's vigiles-shaped, not commoditized.
- Triggering eval (measureTriggerRate) → added to the build list (the #1 skill
activation pain, built on pluginDir).
- Sandbox exec stays parked (feature-ideas §13, out of scope, may reconsider).
* feat(mcp): verify MCP tool references against a live server (src/mcp.ts)
The vigiles-shaped half of MCP support: a minimal MCP stdio client that starts a
server, does the JSON-RPC handshake, and lists its tools — so a referenced
`mcp__server__tool` can be verified against the REAL server (like enforce()
resolves a linter rule), catching a skill/CLAUDE.md that cites a tool the server
renamed or removed (e.g. github MCP create_issue → issue_write). On a miss it
suggests the closest tool name via Levenshtein ("did you mean issue_write?").
- src/mcp.ts: listMcpTools + verifyMcpTool (+ ./mcp export).
- examples/harness/fixture-mcp-server.mjs: a real minimal stdio MCP server, so
tests are deterministic and offline.
- src/mcp.test.ts: 5 tests (handshake/list, resolve, typo→suggestion, renamed
ref, bad command) — wired into npm test. Also probed live against
@modelcontextprotocol/server-everything (13 real tools; "ecko"→["echo"]).
Matrix: MCP unit/static cell → tool refs verifiable; build item 5 part (a) done,
remaining = spec mcp() builder + audit wiring + mcpConfig harness option.
* docs: fix stale 'MCP unwired at every tier' after shipping mcp.ts ref verification
* feat(mcp): wire MCP tool-reference verification into `vigiles audit`
Adds the `vigiles:mcp server#tool` mark (the MCP analogue of vigiles:symbol) and
verifies it in `audit`: parse marks from instruction files' inline spans, read
the server config from .mcp.json, start the referenced server, and flag a
missing tool (with a Levenshtein "did you mean") or an undeclared server —
exit 2, like a broken symbol ref. Only runs when a mark references a server, so
zero overhead otherwise.
- src/mcp.ts: parseMcpRefs / loadMcpServers / verifyMcpRefs / mcpRefMessage.
- src/mcp.test.ts: parse (skips fenced blocks) + verify (ok / typo→suggest /
undeclared) against the real fixture server. 7/7.
- src/cli.ts: audit step 10 + mcpRefErrors in the report / exit code / summary.
Verified end-to-end: a CLAUDE.md citing `vigiles:mcp demo#ekho` →
'not found — did you mean "echo", "add"?'; cli suite 53/53, lint+fmt clean.
* docs: MCP unit/static cell → verified in audit (vigiles:mcp mark)
* docs(readme): surface the mark family (symbol + mcp) and fold in the matrix
- Verified References now lists the inline mark family: `vigiles:symbol file#name`
and the new `vigiles:mcp server#tool` (audit starts the server and flags a
renamed/missing tool with a "did you mean").
- Add the coverage matrix (surface × tier) to the testing section inside a
<details>, so the at-a-glance "what's covered" lives in the README without
bloating it; drop the now-redundant matrix link.
* docs(readme): keep the coverage matrix visible; tuck reference tables under <details>
Matrix (surface × tier) is now a plain visible subsection, not collapsed.
Skills list and Maturity-Levels table moved under <details> (with the gist —
the 4 level names — left inline), since those are reference, not pitch.
* docs(demo): add a runnable 60-second demo (`npm run demo`)
The README's "your CLAUDE.md lies" hook, made runnable: examples/demo ships an
instruction file that reads fine but whose two references lie —
✗ "refreshSession" is not defined in src/auth.ts (renamed symbol)
✗ MCP tool "helper#purge_all" not found — did you mean "purge"?
— while the two truthful refs (verifyToken, helper#log) pass silently. The MCP
check starts a real self-contained stdio server from the demo's .mcp.json. This
is the distribution lever from research/distribution-strategy.md, now backed by
the symbol + MCP verification shipped this session.
- examples/demo/: INSTRUCTIONS.md, src/auth.ts, mcp-server.mjs, .mcp.json,
run.sh, README.md (self-contained, no network).
- `npm run demo` + a README "See it in 60 seconds" callout.
- Isolated: tsc only compiles src/**, the repo's own audit doesn't discover it,
and its generated .vigiles artifact is git/prettier-ignored.
* docs(strategy): close the loop — the runnable demo (proposal A) shipped
Append an update to distribution-strategy.md: the '60-second value, output is
the product' artifact now exists as examples/demo (npm run demo), built curated
+ deterministic to sidestep the heuristic-scan trap that got proposal A
reconsidered. Notes MCP reference verification as a new distinctive Reach hook,
and that the real-plugin dogfood (superpowers/wshobson) is now the evidence base
for an E1 'we scanned the top plugins' post. Next gate is still measurement.
* docs: add research/ + docs/ indexes, TOC, and README navigation
24 research docs (5.7k lines) and 16 user docs had no index — the real
navigation pain with "tons of docs". Rather than risky merges:
- research/README.md — themed index of all 24 research docs (positioning,
the two pillars, skills, proofs, runtime enforcement, FP, distribution/ideas).
- docs/README.md — themed index of the user-facing how-to/reference docs.
- README.md — a Documentation section linking both indexes + the coverage roadmap.
- A Contents line on the harness-testing coverage matrix (the session centerpiece).
Every doc is now reachable from its index, and the indexes from the README
(orphan-docs still green, 43 scanned). Left the docs themselves intact —
consolidation deferred since the clusters (e.g. harness-testing design vs
roadmap vs benchmarks vs user-guide) serve distinct purposes.
* docs(feature-ideas): add a ranked, grouped status overview up top
Keeps all 13 ideas in place but leads with a "Status & priority" section:
marks the 5 already-shipped/partial (dead-enforcement, stale-refs, init,
reverse-coverage, annotation-typo), groups the open ones by theme (catch-more-
lies / change-safety / ergonomics / skills+harness), and ranks them value × cost.
Also clarifies — in the doc — that this is the product backlog (pillar 1), NOT
the harness-testing roadmap (which is the coverage matrix). Top open pick: finish
#2 Reverse Coverage (moat + distribution artifact).
* fix(docs): avoid doc-refs false positive (ctx.file → ctx.sh in an eval example)
The doc-refs validator matched `ctx.file("out.txt")` as the spec `file()`
builder and flagged a missing file — making `vigiles audit` exit 1 (a CI gate).
Switch the example to `ctx.sh("grep …")` (matching the README), which is the
real RunContext API and doesn't trip the matcher. audit back to exit 0.
* docs(research): prior art on testing non-deterministic AI + corrected model
Adds research/testing-nondeterministic-ai.md: a survey of how the field tests
non-deterministic AI/agent tools (DeepEval, Braintrust, Promptfoo, τ-bench's
pass^k + state-over-self-report, AgentAssay, procedure-aware "corrupt success"),
and the model it implies for vigiles. Corrects two conflations from an earlier
sketch:
- predicates (bare, return values) are SEPARATE from assert* (throw); only the
testing helpers carry the assert prefix.
- testing (pass/fail, every commit) and eval (mean±se/pass^k, paid) are SEPARATE
consumers that share predicates over one Trace — not one dual-purpose API.
Folds the gaps into the coverage-matrix roadmap (#10–14): unify the Trace,
capture output + hooks, split predicates from assert*, add pass^k + tool-arg
assertions. Linked from the research index (no orphan).
* docs: fix markdown rendering (stray '- throw' bullet) in the prior-art note
* fix(docs): resolve orphan-doc — reference new doc via full-path backtick
The orphan checker matches `research/foo.md` (it doesn't resolve relative
links), so the bare-basename link to testing-nondeterministic-ai.md didn't count
and audit exited 1 (failing CI's 'check' job). Use the repo convention —
full path in the backtick span, bare href — like the other research cross-links.
* docs(roadmap): save the Trace-unification execution plan for the follow-up
Records the concrete plan for the next session: after #26 merges, branch off
main; do the additive pieces (export predicates + assert* wrappers, unify
runEval ctx with tools/output, r.output, pass^k, tool-arg asserts) which are
non-breaking; defer trace.hooks (needs new instrumentation); keep predicates vs
assert* and testing vs eval separate; acceptance = suites green.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|