mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
fix/hook-runtime-unused-err
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
d6b1ad06aa |
fix!: exclude reaches every pass, and the guard stops missing re-spelled commands (#194)
`.vigilesrc.json#exclude` was honoured by 3 of 11 discovery walks, and the walks that did honour it disagreed on the dialect: glob's string `ignore` treated bare `bench` and `bench/` as matching nothing, while the minimatch helper accepted them. One `ExcludeSet` is now parsed where `loadConfig()` runs and taken as a REQUIRED parameter by every in-scope walk in cli.ts — the shape that let `findSpecs` go a year without it was optional-with-default, which compiles silently. Closes #192. The disaster battery now generates its own inputs: `experimental_alternateSpellings` (renamed from `equivalentDisasters`) re-spells hook test cases so the shell executes them identically — nine families, 7 seeds to 136 spellings. Five of those came from a normalizer fix: `g\it push --force` runs git but was not matched by `runs("git push")`. That fix then had to be scoped, because CI caught the regression it caused: escapes are resolved for the HEAD and FLAG-shaped operands (what the command IS) and never for a path operand (where it WRITES) — the same bytes are a resolved escape to `sh` and a real Windows UNC path on Windows, and fail-closed keeps the path. Both directions pinned by mutation. Also: `spec-refs` and `duplicate-rules` had a meta and a doc but no row in the canonical rules matrix, and nothing compared either against it. A test now asserts every rule has a matrix ROW (matched on the table row, not a prose mention). BREAKING CHANGE: `findSpecs` changed signature; `orphans` gained a `repoExclude` option; `equivalentDisasters` was renamed to `experimental_alternateSpellings` and now takes a required argument. Any repo with an `exclude` key sees it applied to every pass, so an excluded spec is no longer compiled and an excluded harness/eval script is no longer discovered. |
||
|
zernie
|
4f0e6e8037 |
feat(eval): pin --effort as a measurement input, hashed in lock and cache (#191)
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7c97e4ed7d |
refactor(spec)!: one experimental root per feature (#171)
The prefix answers "is this name stable?" and cannot answer "is the name my stable symbol depends on stable?". That gap shipped: railway(), delegate() and result() carried stable names while being meaningless without the experimental builder, and the lint rule cannot see it — it compares tag and name within one declaration, not the dependency between two. So the subagent vocabulary now hangs off one marked root: result, railway, delegate, pipe, pipeStep, needs, start and andThen become members of the builder. Eight names leave the top level. Two further gains the prefix alone could not reach: collision-prone words stop owning package-level names, and the warning survives destructuring, since the binding site still names the root — unlike a namespace object or an import subpath, which mark the import line. Object.assign rather than a TypeScript namespace: namespaces are an error in this repo's lint config, and one merges only with function declarations while the skill builder is a const. Types stay top-level, for the same reason the prefix rule excludes them. No deprecation window: nothing outside this repository imports these, and a window subtracts nothing. The container-rollback argument for a window does not reach them — they take no part in loading hooks. Found along the way, and worth more than the refactor: two example specs had not compiled since the previous rename, because the examples directory was in no tsconfig at all. Both fixed, and the class is closed — examples now type-check. Gates run, in order: vitest (3281 passed), lint (0 errors), prettier --check, api:check, build, exports:check, internal:check, docs:check, and tsc over the types project now that it includes examples. Claude-Session: https://claude.ai/code/session_017aUJEoaEQ3DxtvoSVWAWqe Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
900dc7e06a |
refactor(spec)!: name things for what they are; retire the vigiles/experimental subpath (#169)
Three exported symbols were named for something other than what they are, and
two mechanisms were doing one job.
RENAMES (old names kept one major, @deprecated, removed one major AFTER this one):
claude() → instructionFile() compiles CLAUDE.md AND AGENTS.md, so a
name from one harness was never right
instructions`` → prose`` builds a prose FRAGMENT; the plural read
as "the instruction file", which is what
instructionFile() builds
agent() → experimental_agent() the shape is not settled, and the tag was
never there (git log -S empty)
SkillSpec.result: → postcondition: `result` already meant the subagent output
contract one screen down; the doc comment
had to spend a line disambiguating them
The alias window is not politeness. Measured: the previous rename shipped with
ZERO overlap (18.1.1 old names only, 19.0.0 new only). A container rollback put
the old node_modules under new sources, PreToolUse stopped loading, and a
PreToolUse that does not load denies EVERY Bash command — including the one that
fixes it. One window per major makes that pair unrepresentable.
SUBPATH RETIRED. vigiles/experimental is gone; its members moved to the door
their feature already uses. Of two mechanisms for one promise the subpath was the
weaker: it marks the import line, out of view by the time anyone reads the call.
Measured on the only user-facing example — with the prefix aliased away at
import, the marker survived at 0 of 5 call sites. So the prefix stays, the advice
to alias it away became an explicit prohibition, and the six define* entry points
are DECLARED prefixed rather than gaining it at re-export.
That de-aliasing is what unblocked moving the naming check into ESLint, and the
ORDER was load-bearing: measured before it, dropping the public/internal
exemption produced 14 findings, all from the re-export indirection. Measured
after, across all of src/ with no exemption: 22 tagged declarations, 0
unprefixed. The rule turns on silent.
EXPORT CLEANUP. vigiles/linting export * → curated, 103 → 79 symbols; its header
claimed to be curated with an `export *` one line below, and claimed the builders
were also at the package root (191 exports there, zero matches). ClaudeTool and
HookEvent deleted — zero references, and both had drifted off the dialect they
claimed to mirror; HookEvent listed 5 events, two of which do not exist, against
the 31 the vendor documents.
DOCS. experimental.md deleted and distributed per feature: the skill half into
skills.md §Status, the hook half into a new compiled-hooks.md §Status, and the
emit half — 111 of its 177 lines, a topic doc wearing a stability page's clothes
— into docs/emit-channel.md, which gains the "which channel when" comparison it
never had. The one cross-cutting sentence went to the README.
TWO GATES WERE GREEN AND WRONG, both found by this work:
- api:check reported "verified for 11 entries — no drift" while reading
dist/experimental.d.ts from a PREVIOUS build (tsc does not clean dist/) for a
source file that no longer existed. Now asserted against package.json
exports, with a mutation proving exit 1 on a phantom entry.
- check-export-prefixes caught MY error: the R3 services first landed on
vigiles/eval, where `paid_` means "this call can bill you" — and
startServices takes a ContainerRuntime and calls no model. Naming it
paid_experimental_startServices would have redefined paid_ as "expensive in
some sense". They went to the package root.
SEVENTEEN review findings landed on this branch, all real. Two mechanical
closures came out of them: src/example-imports.test.ts resolves every relative
import under examples/ instead of matching a string, and the naming rule now
ENUMERATES the three export paths it visits rather than describing its coverage
in the abstract. One attempt is recorded as REFUTED: teaching the doc gate to
flag our own exports as free variables measured 66 findings across 93 blocks, so
that gate's original scoping was right.
BREAKING CHANGE: vigiles/experimental is removed. Its exports are unchanged in
name and behaviour and are now reached from `vigiles` (the emit channel, the R3
disposable-service tier). Every renamed symbol keeps a working @deprecated alias
for this major.
|
||
|
zernie
|
4acfd31fc9 |
feat(eval)!: an eval file describes, the runner performs — importing one no longer spends (#159)
* fix(audit): coverage caveats printed by lint never reached the audit report `formatUntestedReport` grew two qualifier lines — a test file still carrying the retired `vigiles:covers` marker, and a run record older than the text it measured. `audit` assembles its own fact block from the SAME UntestedReport and carried neither, so the note that exists to explain a silent migration was itself silent for anyone whose habit is `audit` rather than `lint`. Measured before the fix on a fixture whose only harness carries the marker: lint → "1 test file(s) still carry the retired `vigiles:covers` marker …" audit → "Untested surfaces: 0" and nothing else The caveats now come from one exported builder (`coverageCaveats`) that both renderers read, so a third qualifier reaches both callers or neither — a renderer can no longer carry half the list. `ScanReport.coverageCaveats` is absent rather than `[]` when empty, because the browser twin structurally cannot compute these (no `.vigiles/coverage.json` without a filesystem) and byte-parity between the two engines is asserted field-for-field. Both halves tested at the product level (`node dist/cli.js audit`): it fires on a repo carrying the marker and stays silent on one that does not. Proven by two mutations, each with the patched source grepped first: dropping the render line fails the "fires" assertion; emitting a caveat unconditionally fails the "silent" one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the suffix that stopped counting now says so `*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run collects exactly those names and a skill test spawns a model — so a misnamed harness was being executed, and paid for, by somebody else's runner. The removal was announced to nobody. Its own comment argues "the migration is a rename, and the untested finding prints the exact path", but the path printed is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore told to write the test they wrote, with neither the file nor the reason named. The sibling half of the same migration — the retired `vigiles:covers` marker — got its explanatory line at the time. This is the half that did not. Scoped to surfaces that are otherwise UNCOVERED: that is the only position where the count contradicts what the author can see in the directory. A stray `*.test.*` beside a properly covered surface is somebody's ordinary unit test and none of our business — asserted, because an advisory line that fires on the innocent case gets muted within a day. Nothing about coverage changes: the name still grants nothing. Measured silent on all seven real corpora available here (this repo, six vendored dogfood plugins) and on the `mine` knowledge base; it fires on a planted fixture in both `lint` and `audit`, the latter through the shared caveat list added in the previous commit rather than a second wiring. Three mutations, each with the patched source grepped before the run and each dying on its own assertion: dropping the note from the caveat list · widening the scope from untested to every considered surface · dropping the suffix test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * refactor(scan): compute the caveat list once Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was called twice in the same object literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the retired-suffix note claimed a cost the guard already closes The note said a `*.test.*` harness "spawns a model" under someone else's runner. `refuseUnderForeignRunner` guards every spawn door and refuses under vitest/jest — measured here against the detector: argv of `node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model: this process is running under vitest." What actually happens is that the foreign run FAILS on the file, which is the true and smaller claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it Importing a `*.eval.*` file spent real money. These files did their work in the module body, and in ESM `import` IS execution — so the cheapest imaginable question, "does this file even parse?", launched a paid run against a live model when it was asked with `import()`. That happened, and was paid for. The one guard that exists does not close the door and cannot be made to. Under `node -e 'import(f)'` there is no `process.argv[1]` at all, so `foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays silent; and no process fact distinguishes that from a legitimate runner doing the same import, so an argv-shaped guard would have to refuse the correct invocation to catch the wrong one — the exact failure that got `process.env.VITEST` rejected in `core/foreign-runner.ts`. So the shape changed instead. An eval file default-exports a description: export default defineEval({ measureTriggerRate: { …spec… }, assert: (report) => assertTriggerRate(report, { min: 0.8 }), }); `defineEval` builds a plain value; there is no call in it to run. The five paid runners are reached only by `dist/eval-entry.js`, the program `vigiles eval` now spawns per file — not a new verb, the interpreter the existing verb uses. `defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and that is the point: after this change no eval file imports a runner at all. Three loud edges, because a silent no-op would be the same defect in a new place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares nothing (or two measurements) is reported by the runner instead of exiting green; and a half-migrated file — a description plus a leftover top-level `measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and throws a migration message instead of billing. `node --check` still works and is what the free syntax check should be. Subtractions: every eval file loses its `process.env.VIGILES_TRIALS || process.argv[2]` line, its `console.log(format…Report(report))`, its `if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval` import. The runner does all five, once. BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})` instead of calling a runner at the top level, and `node <file>.eval.mjs` no longer runs an eval. To migrate: move the runner call into a key named after that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` · `measureSelectionMatrix` — declare exactly one), move anything you did with the report into `assert(report)`, and delete the trials line, the `format…` print and the `report.n === 0` check — `vigiles eval` does those. A capability probe that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s directory argument becomes a `pluginDir` field; and `measureTriggerRate(spec, { evalDriver })`'s second argument becomes a top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials. Run with `vigiles eval <file>`; a file left on the old shape fails with a message naming docs/harness-testing.md, never silently. All 19 eval files in this repository were migrated in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture and generated types The eval-definition change rewrote one report hint (`paid_measureTriggerRate` → `export default defineEval({ measureTriggerRate: … })`), which is exactly the divergence the parity test is built to catch. Regenerated per that test's own instruction for an intentional engine change; the diff is that one line — grade and every other field are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
863c1b42ee |
docs: два примера импортируют символ не из того сабпаса — оба падают на копипасте (#145)
Найдено ресёрчем по инструментам doctest и подтверждено вызовом, а не чтением: scriptModel: vigiles/testing → undefined, vigiles/claude-code → function runHookProgram: vigiles/testing → undefined, vigiles/hook → function Оба примера читатель копирует целиком, и оба падают на первой строке. Это не опечатка в прозе, а неработающий код в документации. 🔴 Про `scriptModel` это уже ТРЕТИЙ раз. Тот же неверный импорт дважды записан в потребительской базе знаний (`vigiles/s59.md` §3 и починен в скилле по s61 §4) — с пометкой, что граница намеренная: `scriptModel` живёт в адаптере Claude Code, а не в харнесс-агностичном барреле. Правило было известно, задокументировано, и третий экземпляр всё равно уехал в публичные доки. Ровно тот класс, который проза не удержит: граница верная, документация о ней верная, а конкретная строка импорта — нет. Механическая проверка предложена в `vigiles/ideas/19` (в приватной базе); этот коммит закрывает два известных экземпляра, а не класс. Гейты: npx prettier --check docs/ · npm run lint (0 errors). Правок кода нет. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
10e528e8db |
refactor(testing)!: изъять runMutations — она не про домен харнессов (#143)
* docs(testing): runMutations was in the reference only — the guide never mentioned it Measured after shipping it, not guessed: `runMutations` appeared in exactly ONE file, `testing-api.md`. Zero mentions in `harness-testing.md` (the task-first guide), `testing-matrix.md` and `docs/README.md`. The reference says of itself "the guide has the task-first how-to; this is the reference" — so anyone following the guide would never learn the API exists. And the runner table still opened with "Four runners" while listing five. That is the same defect the API was built to catch, produced by its own docs: prose and content disagreeing, with nothing to notice. - harness-testing.md — a routing row in "What do you want to test?" plus a full section, including the two things the reference did not say at all: WHEN NOT TO reach for it (a one-off script, an already-red test, or wanting a coverage number — mutation counts here say how many cases you wrote, not how good the suite is), and CI, where the honest answer is usually "don't": a run forks a process per case over full files, so it belongs in a hand-run pass after editing the test or the checker, or in its own job if wired at all. - testing-api.md — "Four runners" now says four for the harness plus a fifth that asks about your tests, which is what the table actually shows. - testing-matrix.md — a row for src/mutations.test.ts, naming the properties it covers, so the coverage map matches the suite. - docs/README.md — an entry under the API reference pointing at both. Gates: prettier · lint 0 errors · the 10 mutation tests still pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * revert(testing): withdraw runMutations — it does not belong in this package Shipped in #142 and 15.3.0; withdrawn a day later on the owner's question: «how is this vigiles' responsibility, it's just nodejs?» He is right, and the measurement is what settles it rather than taste. Against the rest of this package: runHook knows the Claude Code hook protocol, measureTriggerRate knows what a skill is, skillContract reads allowed-tools out of frontmatter. This module knows none of that. It replaces a substring, spawns `node`, and reads stdout — lift it into any Node repository and it works unchanged. A newcomer reading the testing docs hits exactly the owner's question, which is the cost this surface was charging. The confirming measurement, and the one I should have taken before merging: NO TEST IN THIS REPOSITORY USED IT. The only references were its own test file, its export line and its docs. The package did not dogfood the API it published. What actually fixed the problem it was extracted for was CONSOLIDATION, not RELOCATION. Ten hand-written copies of one driver had drifted apart in a consumer repository — the no-op guard in 4 of 10, the retry in 1 of 10. One shared copy buys all of that. Moving that copy into a product for harness authors was a second and separate step, and nothing justified it. The engine survives where its callers are, with the reasoning recorded in its header so it is not re-proposed. The shape that WOULD belong here is a different function that does not exist yet: mutate the HOOK — remove one branch of a guard — and require the assembled harness test to notice. Only vigiles knows what a hook is. Removed: src/mutations.ts, src/mutations.test.ts, the export block in src/testing.ts, the api-surface entry, and the docs sections in testing-api.md (from #142) plus the guide, matrix and README entries this branch had added. Gates, in order: npx vitest run (2916 passed, 172 files) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: перезапуск проверки заголовка PR Тип `revert` не входит в список, который принимает линтер заголовков (feat/fix/docs/chore/refactor/test/perf/ci/build). Заголовок переименован в `refactor(testing)!` — восклицательный знак несёт то, что `revert` нёс именем: изъятие опубликованного API ломающее. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b5da1c5604 |
feat(testing): runMutations — prove a test can fail, by breaking what it watches (#142)
A green test says the checker passed. It does not say the test would notice if the check were DELETED: an assertion can be vacuous, a fixture can be wrong in the same direction as the code, and both look exactly like a pass. `runMutations` plants a defect, runs the test that owns it, and requires that test to go red WITH THE MESSAGE THAT NAMES the defect. ## Why this belongs in the package It arrived as ten hand-written copies of one driver in a dogfooding repo (1799 lines), and the copies had drifted. Counted against the committed versions: - the NO-OP guard (a replacement equal to the original leaves a green test that proves nothing) was in 4 of 10; - the RETRY on a non-kill (a flake reported as a survivor sends the next reader to rewrite a working assertion) was in 1 of 10; - the strict rule — killed by its OWN named assertion, not merely by something going red — was in 1 of 10. Each was written AFTER it caught something, in whichever copy caught it, and never travelled. Two copies also named a test file a refactor had deleted, and the runner they used exits 0 on a path matching nothing, so those cases reported SURVIVED on every run for three days. ## Five verdicts, one of which is proof killed · wrong-assertion (a neighbour caught it) · survived · unjudgeable (the test was already red before the run, so "red" says nothing) · not-applied (find matched 0 or many, or the replacement equalled the original). The baseline matters more than it looks: without it, a test that is red ON PURPOSE — an open finding it exists to report — makes every run announce a failed restore about a working one, and makes every case against it look killed. ## What refuses rather than reports An empty case list and a named test file that does not exist both throw BEFORE any file is touched. A runner with nothing to run reports success, and that is the one claim this API exists to prevent. ## Verification vitest 2924 passed · lint 0 errors · prettier · api:check (additive only, +53 lines to vigiles-testing) · build. The new file has 10 tests, each building a real checker and a real test in a temp dir — a mocked runner would be judging a mock. Mutation-tested itself: removing the strict-kill rule, the no-op guard, or the existence check each turns the suite red on its own assertion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
41169e379a |
fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129)
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state `Tested` collapsed deterministic harnesses and real-model evals into one number, and its finding read "N surfaces with no vigiles test/eval" — one sentence for two prescriptions three orders of magnitude apart in cost. A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on every push, and asks whether a gate still catches what it claims. An eval (`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one question a deterministic read cannot: does this skill fire at all? Folded together, a repo with complete deterministic coverage and no evals scored identically to a repo with neither. - test-coverage: one discovery pass, partitioned by suffix into a harness tier and an eval tier; the union `covered`/`untested` fields are unchanged. Mirrored in the browser-safe twin. - scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and additive; a producer predating the split falls back to the union). - audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both advisory. `Evaluated` has THREE states — a number, a measured `0`, and `not measured` when the executing checks were skipped and no eval exists. Scoring 0 there would report the absence of a check as its result, which is the pattern this tool exists to name elsewhere. In that state the ring names the command that would measure it. - cli: threads the existing execute-skip signal into the ring. - report/HTML/JSON: both rings render; the inventory carries both counts. Grade weighting is unchanged — both rings are advisory and neither enters `reportDeductions`. Audited on this repo before and after: A (100/100) both times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * docs(site): the audit grades six categories, not five Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * fix(lint): a clean union must not read as 'firing verified' Every surface can have a deterministic harness while nothing has ever measured that a skill fires. The gate is unchanged (still the union), but the ✓ now says so instead of implying the question was asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * test(site): regenerate the browser-parity fixture for the Evaluated ring The audit now emits a sixth category, so the Node-generated expected report and the in-browser one differ until the fixture is regenerated — which is what the test's own header prescribes for an intentional engine change. Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the Evaluated ring and its two inventory counts and nothing else: grade and overall are untouched, which is the check that the split did not move the weighting. * fix(coverage): `Tested` measured a STRING, not a test DEFECT. A surface counted as covered when its path or namespace appeared as a SUBSTRING anywhere in any discovered test file (`content-reference`, in src/test-coverage.ts). Two halves, and the second was worse. REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an existing harness file: // probe: skills/argument-arc moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the line put it back. So `Tested` was trivially gameable and the gaming was indistinguishable from real work. The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely asserts over all 21 of its pipeline skills, but builds each path at runtime (`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING. Generality was PENALISED: 21 trivial files each naming their own subject would have scored better than the one harness that actually tests all 21. This is the exact substitution vigiles names in other people's repos — presence of a surface taken for presence of the property, the way `noExplicitAny` in a config is taken for the rule being enforced. The tool was committing it in its own scoring. FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the disk detector and its in-browser twin cannot drift on what decides a grade): 1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring the existing `vigiles:ignore-test` marker convention on the other side. A runtime-path harness can now say what it covers; an accidental mention cannot. Declaration beats inference — the same move as declaring `allowed-tools` instead of inferring capability. 2. PROVENANCE. Every coverage decision now carries HOW it was reached — declared / colocated / mention — and the counts are printed by `formatUntestedReport`, by `formatScanReport`, and carried in `AuditReport.inventory.coverageEvidence` (additive; schema version unchanged). A repo whose coverage rests entirely on substring matches can finally see that about itself; "28 covered" and "28 names that happen to appear in a file" used to print identically. 3. The bare substring detector is KEPT, deliberately, and said so in the code: it is the zero-config path that makes an ordinary `foo.test.ts` naming `skills/foo` count without anyone learning a marker, and dropping it would flip honestly-covered surfaces to "untested" overnight. What it no longer reads is COMMENTS. A path in a comment is prose about a test; a path in executable code is the test reaching for the surface. Prose isn't policy, including ours. It is also now visibly the weakest tier in the report, and when it is the ONLY evidence the report says so in words. Grade weighting is untouched. TESTS prove both halves, each watched failing first: with the fix reverted, "a surface named ONLY in a comment is NOT covered", "a runtime-path harness that DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail (3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE still counts, and a `"https://..."` string literal survives comment-stripping. Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously recommended "add the skill's path in a comment", which was advice to exploit the defect. * fix(site): key the browser-test prebundle to dist, so a stale cache can't lie DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako divergence by comparing an in-browser AuditReport against one generated in Node. `site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on `@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite invalidates on config/lockfile changes, not on the mtimes of an aliased path outside node_modules. So the browser side can compute against a PREVIOUS engine while Node computes against the current one. REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately). Accidentally: an engine change added a report field; the parity test failed showing the field present on the Node side and absent in the browser. I read that aloud as a genuine engine divergence and started hunting a pako bug that does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass. Deliberately, to prove the fix: plant an extra `inventory` field in src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the browser suite with a warm cache. - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1` on one side only — a field-level diff that reads exactly like a real divergence and says nothing about the cache. - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` -> `.vite-engine-a52a27e0137e`. Reverting the plant returns the fixture to its committed bytes. FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/` (`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`), sweeping the directories keyed to other fingerprints so they don't pile up. WHY a key and not the alternatives. Comparing the cache's mtime against `dist/` and forcing a re-bundle, or clearing `.vite` in the test script, both work and both leave the bad state REACHABLE — a comparison someone can get wrong, or a script step bypassed by running `vitest` directly. Keying the directory removes the state instead: a different `dist/` is a different path, so "prebundle built from another engine" cannot be reached, and an unchanged `dist/` still gets a warm cache. By construction rather than by check — the argument this project makes everywhere else about removing a capability rather than policing its use. The test's header instruction ("pako and node:zlib disagree — do NOT paper over it") was correct for one of the two states and actively misleading for the other; it now says which state is no longer reachable and why, and the assertion carries the same note in its failure message. * fix(site): make the parity-fixture generator emit prettier-clean JSON DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with `JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it keeps short arrays exploded across three lines where prettier collapses them. CI gates on `prettier --check .`, so every regeneration of the fixture committed a file that reddened CI later and far from the cause. It just did: branch head |
||
|
zernie
|
7b98ae35c4 |
feat(testing): export loadHook + add the react assertions (#123)
The compiled-hook testing story had a hole at each end.
1. NO PUBLIC LOADER. `assertHookDenies(hook, event)` / `assertHookAllows` take
the hook OBJECT, but a `.harness.mjs` test only has its PATH, and the loader
that bridges the two (`loadHookProgram`) lived privately inside `cli.ts`. So
the intended in-process test path was unreachable from the file format
`vigiles test` actually runs, and authors fell back to spawning the runtime
as a subprocess — precisely the plumbing compiled hooks exist to remove.
`loadHook(path)` is now exported from `vigiles/testing` and `vigiles/unit`,
and the CLI runtime calls the SAME function, so a hook that loads in a test
loads identically in production (one loader, no drift). It keeps the
actionable errors: a `.ts` hook on a runtime without type stripping says to
use tsx / Node >= 23.6, and a module with no default export says
`use \`export default defineHook({…})\``.
2. NO ASSERTION FOR A REACT HOOK. A react can't block, so the gate assertions
don't apply, and nothing else did either. That left `notice()` a live trap:
its message goes to STDERR, so a probe built on `execFileSync` (stdout only)
sees nothing and reports a healthy react hook as DEAD — measured against
three real hooks on 2026-08-03.
`assertHookNotices(hook, event, matcher?)` and `assertHookSilent(hook, event)`
read the reaction in-process, so the stream never enters into it. Both name
what they got instead on failure, including when aimed at the wrong role.
Also ships the example the finding asked for:
`examples/harness/compiled-hook-inprocess.harness.mjs` loads a gate and a react
hook BY PATH and asserts over both. It runs under `vigiles test` (so CI proves
the documented pattern works, not just that it parses), and the docs in
compiled-hooks.md / testing-api.md now show this path, with the stderr trap
called out explicitly.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
3f33925722 |
feat: skill-description-budget rule + document-the-why + audit/opt-in docs (#54)
* docs: explain why compiler-grade guarantees are opt-in The "opt-in, like TS's strict" line appeared in three public docs but only asserted the analogy, never the reason. Add a dedicated FAQ entry that gives the actual answer — opt-in means gradual not permissive (the default already gates real breakage), compiler-grade proofs need a typed program to check (tsc can't verify inert markdown), and gradual adoption is the TS-strict bet — and link the README and linting-guide one-liners to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: codify auto-vs-nudge enforcement model in the spec skills Apply the init enforcement model at edit time across the three spec skills: auto-apply the con-free wins, nudge (with the tradeoff) for anything that costs config/plugins/CI-failure, never silently escalate into strict gating. - strengthen: name the principle explicitly (cost, not strictness, is the dividing line; Tier 1 auto, Tiers 2-4 are the user's call) - edit-spec: add "do what's asked, don't silently escalate enforcement" - adopt-spec: replace the prose blockquote with explicit, scannable adoption rules (faithful / non-destructive / don't-escalate / reversible / ask-before-writing / lighter-touch-exists) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: explain what audit runs (read vs measure) + selection-collision Close the audit→behavioral→consent documentation gap exposed while dogfooding a skills-heavy repo: a clean-A audit "found little" because a plain audit is a deterministic READ — the behavioral checks (does each skill fire? do two collide?) run only under the one-time audit.measure consent. - faq.md: new "What does audit actually run — and why did it find nothing?" entry stating the read-vs-run model and the headless fallback - for-plugin-authors.md: turn the bare "selection-collision matrix" name-drop into a real explanation (diagonal = recall, off-diagonal = a sibling hijacked the prompt; body-stubbed, so it measures selection cheaply) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add document-the-why rule + doc-command coverage check The "docs suck" failure repeated twice this session (the opt-in why; the audit→behavioral→consent story), and no existing docs rule owns it: the polish, placement, and sync rules govern how well / where / consistency, but none requires the REASONING behind a user-facing decision be documented at all. - new `document-the-why` guidance rule: every user-facing decision/concept gets a plain-language doc home that answers why/what (name-dropping ≠ documenting); the test is "could a user answer it from the docs alone?" - new deterministic FLOOR (the inverse of self-command-refs): every public CLI verb must be mentioned under docs/ — src/doc-command-coverage.ts, dogfooded in its test (catches a verb shipped undocumented). The concept-coverage half stays judgment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: add skill-description-budget lint rule (warn-tier trigger proxy) A model-invocable skill is selected on its description, and the selector weighs the opening most — a bloated description buries the trigger signal and hurts recall + precision. New deterministic heuristic proxy (generous 500-char budget), the free sibling of description-overlap. Wired end-to-end: detector (src/core/skill-description-budget.ts) + rule-meta (heuristic-behavioral → warn ceiling) + DEFAULT_RULES + NUDGE group (never gates) + scan report + lint application + docs/rules + the rules matrix + tests. NOTE on haretrail: it does NOT fire there — the earlier "634/781-char" figures were BYTE counts; in characters (Cyrillic = 2 bytes/char) the longest skill description is 463 chars, under budget. The rule is correctly calibrated FP-safe; forcing a hit would mean crying wolf. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for haretrail-dogfood session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * test: add skill-description-budget to validate.test DEFAULT_RULES literal The new rule landed in DEFAULT_RULES but validate.test.ts asserts the full rules object against a hardcoded literal — add the key so the deep-equal passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: prettier-format HANDOFF.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * refactor: remove the cross-language description scan flag (refuted) The deterministic "non-Latin description → cross-language trigger risk" finding (unexpectedScript / descriptionScript in scan.ts) is removed: the Script type, scriptCounts, unexpectedScript, the ScanSkill.descriptionScript field, its report line + summary, its tests, the docs mention, and the config-expected-script notion. Why: it fires on 100% of non-Latin descriptions, but a non-English description is a legitimate audience choice, not a defect — so it scolds correct work. And the one time we measured it (plugin-behavioral-findings.md Finding 3a, native codex), the risk was REFUTED — the model handles RU-description/EN-prompt selection fine. A warn that reads as a defect on correct authoring cries wolf (lint-rule-calibration). Reasoning saved in research/plugin-behavioral-findings.md; the one-detector-no-drift teaching example reassigned to description-overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: eval cost transparency — surface tokens, API-equivalent $, metered warning Make what a real-model run SPENT impossible to miss (vigiles's affordability pitch is "runs on your subscription, not a metered API" — so every run should say so). - new src/eval-cost.ts: the pure engine — cost snapshots (per-run / per-arm / whole-report), metered-vs-subscription detection (ANTHROPIC_API_KEY set → metered), a session tally, and the formatted block (tokens + API-equivalent $, a LOUD warning + actionable fix when metered, ✅ when on the sub). Fully unit-tested, no model. - runEval now emits the cost block after a run (stderr; report carries usage). - test-harness skill: MUST relay the spend to the user after any eval it runs. - measuring-skills docs: "what a run reports + the metered-API warning" section. HONEST SCOPE: no "% of your subscription" — Anthropic doesn't expose plan quota, so a percentage would be invented; tokens + API-equivalent $ + billed-to is the complete honest picture. FOLLOW-UP: measure/measureArms/measureTriggerRate reports don't carry usage yet (only EvalReport does) — threading a usage field through those model-gated run loops + the eval-lock is the next commit so the skill's trigger-rate path emits too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * fix: brace void-returning arrows in eval-cost (eslint no-confusing-void-expression) Missed running eslint on the new file (only ran fmt:check). The console.error sink and the test beforeEach arrows returned a void expression in shorthand form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add surface-architecture-decisions rule Surface non-trivial architectural decisions (file placement, core vs adapter vs port, extend-vs-create, cross-cutting changes) to chat in a scannable block with the boundary reasoning — the structure analog of subagent-model-note — so the user can catch a wrong placement before the diff, given the hexagonal + multi-adapter constraints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: thread usage into TriggerRateReport so measureTriggerRate emits cost The skill's main path (measureTriggerRate) now surfaces spend too. Per-run usage comes from the parser (makeContext(out, cfg.parse).usage) — harness-neutral, since both the Claude and Codex parsers fill ParsedModelRun.usage (Codex reports $0 on the sub, real tokens). runTriggerTrial returns its run's usage; runTriggerSet collects them; the report aggregates via the existing aggregateUsage (relevant + irrelevant sets both counted); the wrapper emits costFromArm(report.usage). No new port/driver method (the parser already provides usage) and eval-lock-safe (additive field; the lock hashes inputs, not the report). FOLLOW-UP: measure/measureArms (CheckReport) still don't carry usage — same threading applies, next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: fold parse-don't-validate + illegal-states-irrepresentable into surface-architecture-decisions Frame them from this rule's angle (surfacing): a parse-boundary and an illegal-state-forbidding type ARE architectural decisions — name them when you make them (where the boundary sits; what invalid state the type forbids). Cross-refs ts-essentials for the mechanical how, rather than duplicating it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: update vigiles-testing API surface for TriggerRateReport.usage api-extractor tracks the public surface; adding the usage field is intended (additive — consumers read the report, don't construct it). npm run build doesn't run api-extractor, so this was missed locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for the eval-cost / rules session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: capture haretrail-driven audit + eval ideas (research note) Save the session's un-persisted ideas before context is lost: new deterministic cross-ref checks (openai.yaml / trigger-manifest / env-var contract) and new eval helpers (the selection-collision confusion matrix as a first-class primitive, manifest-driven fixtures + auto-negatives, description-ablation). Indexed in research/README.md (Skills) + cross-linked to plugin-behavioral-findings / measurement-authority / audit-wow-ideas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * style: prettier-format the haretrail-eval-ideas note + README Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add status/topic frontmatter + agent-facing index to research corpus Migrate all 114 research/*.md docs to carry `status:` (active/shipped/idea/ superseded/rejected) and `topic:` frontmatter, so the corpus is sliceable deterministically (`grep -l 'status: shipped' research/*.md`). Add research/CLAUDE.md.spec.ts — a scoped claude() spec whose keyFiles map every research doc to a one-line, status-tagged summary (grouped by topic). Compiles to research/CLAUDE.md, the agent-facing index a session loads as directory memory for research/ without opening 114 files. Mirrors src/CLAUDE.md.spec.ts (the code map); this is the research map. Bidirectional sync is enforced, not by hand: - the compiler verifies every keyFiles path EXISTS (a deleted/renamed doc fails `vigiles compile`) - src/research-index.ts + its dogfood test assert the reverse — every research/*.md (except README.md) appears in the index research/README.md stays the human, thematic index and now points at the machine index + the frontmatter query convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for research corpus index work * feat: thread eval cost into measure/measureArms CheckReport The scored-check paths (`measure`, `measureArms`) were the last eval entry points that didn't report spend — only `runEval` and `measureTriggerRate` did. Close the gap the same way: add `usage: ArmUsage` to `CheckReport` (sourced from the arm's already-aggregated usage), and emit the cost summary in the v8-ignored real-CLI wrappers. `measureArms` sums both arms (an A/B run pays for both). So every real-model eval path now prints tokens + API-equivalent $ + the metered-API warning. Wiring tests drive an injected runner (no model) and assert usage aggregates across trials and per arm. API surface + docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: promote selection-collision to an assertable primitive (measureSelectionMatrix) The cross-skill selection-collision matrix — "when I ask for skill i's job, does ONLY skill i fire?" — already existed but was reachable only from the CLI under the audit consent. Promote it to a first-class, assertable eval primitive so a plugin author can gate collisions in a test: - `measureSelectionMatrix(dir, opts)` — zero-setup: prompts auto-derived from each skill's description (or supply your own). Thin wrapper over the existing `measurePluginSelection`; `measureSelectionMatrixWith` is the injectable core. - `assertNoCollision(report, { maxOffDiagonal, maxPluginCollision })` — the throwing gate. Defaults to demanding ZERO collision; throws on a green-that-tested-nothing (unavailable / zero runs), never a silent pass. Both are exported on `vigiles/claude-code`, NOT the agnostic `vigiles/testing`: selection-collision reads which skill the selector chose, which only Claude Code surfaces (Codex has no skill-selection event), and `scan-behavioral` is Codex-adapter-coupled — so the CC surface is the correct home. Adds a canonical example eval, unit tests (auto-derive + all assert branches, no model), the plugin-author doc's CI path, and API/keyFiles updates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — cost threading + selection-matrix landed * docs: tidy HANDOFF formatting + refresh stale gotchas * docs: mark selection-collision as Claude-Code-only, consistently Make the CC-only surface unmistakable across the docs, one convention: - testing-api.md: a dedicated section "Selection-collision matrix (Claude Code only)" — the "(Claude Code only)" heading + a 🔵 callout naming the `vigiles/claude-code` import — plus a Contents entry and an Imports-section bullet cataloguing which measurements are CC-only vs agnostic. - harness-testing-claude-code.md: a pointer from the Tier-3 trigger section (a CC user's natural landing spot for a CC-only measurement). Complements the existing inline note in for-plugin-authors.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
d8c639f8da |
refactor(cli)!: consolidate the verb surface (13→8) + docs/README overhaul (#46)
* docs: sharpen README lint/guard copy - Replace internal "7 catalogs" jargon with "7 linters (ESLint, Ruff, Clippy, and four more)"; Cedar stays in the subdoc - Drop the plugin-audit paragraph from the Lint section (secondary use case, still reachable from More) - Rewrite the Guard intro to lead with the concrete "fail open" failure and name what vigiles emits (exit code, JSON, AST-backed matcher) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface the $0/subscription cost story in the first paragraph The "no metered tokens, runs on your Claude subscription" wedge was buried in the last line of the Eval section. Pull a crisp version into the opening paragraph (above the fold); the Eval section keeps the detailed promptfoo/DeepEval contrast as proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: surface agent-writes-the-harness flow, reorder More, drop internal links - Quick start now leads with "then your agent writes the harness for you": the three model-invocable skills (test-harness/strengthen/ edit-spec) and the plain-English ask each one answers - Reorder More so CLI sits above the leaderboard (whose link points into the CLI docs); trim the now-redundant Skills bullet to a pointer - Stop linking internal research/ docs from the README (the moat link and the Research index) per public-vs-internal-docs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(compile): auto-refresh an existing harness.gen.ts compile now keeps a whole-harness registry in sync as a side effect, so the user never hand-runs generate-harness. Gated on the file already existing (compile maintains a registry you opted into, never imposes one) and cheap (parses specs, no linter spawn — unlike generate-types/ generate-schema, which spawn every linter and stay on the config-change guard). A duplicate agent name fails the compile. Docs: note the auto-refresh in the compile section and mark the generate-* artifacts as dev-toolchain output rarely run by hand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(src): add nested CLAUDE.md with a high-bar-for-new-commands rule Directory-scoped guidance loaded when working in src/: adding a new CLI verb or hook-runtime subcommand must clear a high bar (genuinely new mental action, no overlap with an existing command, siblings collapse to one verb + subcommand, runtime entrypoints stay hidden). Complements the root cohesive-cli-surface rule, placed next to the code that defines the command surface (cli.ts, cli-commands.ts). Spec-managed (require-spec + integrity); dogfoods vigiles compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the "(generated)" annotation from the API reference link Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop src/CLAUDE.md.spec.ts shadowing the root spec + add docs rule The nested src/CLAUDE.md.spec.ts (added last commit) compiles under rootDir:src to dist/CLAUDE.md.spec.js — the exact path loadSpec resolves the root CLAUDE.md.spec.ts to, so recompiling the root spec loaded the src spec and clobbered CLAUDE.md. Exclude src/*.md.spec.ts from tsc so both load via the tsx fallback (as the root always has). Also adds the no-internal-links-in-public-docs guidance rule (public docs never link research/*.md; keep the fact in research, don't link it), reconciles the old "link to research" line in public-vs-internal-docs, and files the deterministic lint-rule version as a P1 roadmap item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: separate public/internal indexes + surface eval safety - docs/README.md no longer links into internal research/ (public docs don't point at the private record) - research/README.md is explicitly labelled the internal record - README Eval section gains a "Safe to repeat" note (sandbox + ephemeral dir + egress + interceptTools, past-the-sandbox case flagged) linking docs/safety.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: stop public docs linking internal research/ + add safety FAQ Strip every research/*.md link from docs/*.md (the no-internal-links-in- public-docs rule) — ~60 links across 16 guides plus the docs/ index; prose kept readable, no orphaned research doc (all still cited from CLAUDE.md keyFiles / research/README.md, orphan-docs green). Add a FAQ to docs/safety.md answering the side-effects-beyond-the-sandbox question honestly: network via egress allowlist, a real DB/redis/browser via a disposable container you stand up (turnkey provisioning is a committed next step), irreversible externals via interceptTools, and the no-Docker deterministic/record-replay tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(cli)!: consolidate verbs — measure/explain→scan flags, refs→lint, generate-*→generate <kind> De-sprawl the human verb surface from 13 to 8 (the high-bar-for-new-commands rule, dogfooded). All breaking: - `measure <dir>` → `scan <dir> --trigger` (the model-gated behavioral column folded into scan; trigger-rate + collision matrix unchanged) - `explain <dir> [name]` → `scan <dir> --explain [name]` (the deterministic WHY, a sibling of --fix-plan on the same ScanReport) - `refs <file>` → folded into `lint` (the unmarked-refs check is already a lint rule; the PostToolUse hook keeps its shared collectRefIssues entry) - `generate-types|schema|harness` → `generate <kind>` (one verb + subcommand; the three dev-toolchain generators grouped, not N hyphenated siblings) scan reuses handleMeasure/handleExplain directly behind --trigger/--explain. Docs, help text, the regen guard, tests, and the self-command-refs dogfood all updated together (cohesive-cli-surface); 1588 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: split the GitHub Action reference out of cli.md into its own doc cli.md was 718 lines (kitchen sink). Extract the ~108-line GitHub Action section into docs/github-action.md (quick start, inputs, output channels, versioning) and leave a short pointer; retitle the cli.md intro. Update the README More bullet (CLI · GitHub Action) and the docs index. cli.md → 619 lines. orphan-docs + self-command-refs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lint): add prefer-compiled-hooks rule — one nudge toward compiled hooks A single repo-level recommendation (one finding regardless of hook count): when a plugin ships hand-written hook commands that aren't compiled vigiles/hook artifacts, nudge toward compiling them — the message links docs/compiled-hooks.md and points at guardrail-check for verifying the ones you have. A discovery nudge, not a defect: the hand-written shell lane stays first-class, so it fires ONCE (never per-hook), defaults to warn (info- level ℹ; informational, doesn't fail CI), and is opt-out. Detector (manualHookCount: a command that isn't a hook-runtime invocation) lives in scan.ts and is shared by scan + lint (one-detector-no-drift). Doc + matrix rows added (rules-docs-in-sync). Honest caveat in the doc: it's form-based, so it stays a recommendation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): record the eval-startups critique mapped onto positioning New research/eval-startups-positioning.md captures Thomas Liao's "why eval startups fail" critique and its implications: vigiles dodges the trap (harness-eval not model-eval, tooling not services, eval already a feature), but it's a yellow flag on the public ecosystem-benchmark as a MOAT (Goodhart-gameable → keep it a thin acquisition flywheel) and it independently validates the attestation/safety direction (capability-diff, guardrail proof) as the only defensible eval-shaped niche. Linked from research/README.md, measurement-authority.md (whose benchmark thesis it challenges), and strategic-synthesis-2026-06.md (whose Tier-2 conformance/attestation play it validates). Internal-only per no-internal-links-in-public-docs; orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: purge internal positioning from public docs (jargon, name-drops, benchmark) Public docs (README + docs/) now state user-facing value, not vigiles's internal strategic vocabulary: - strip "moat" from 14 places → user-facing wording (engine / cost advantage / differentiator / cross-reference) - remove all bare research/*.md name-drops from docs/ (the backtick form the link-strip + deterministic check miss); scope-descriptions reworded to "configured directories" rather than naming research/ - README: drop the "ranks the hyped ecosystem: what works vs hype" benchmark banner from the Eval row + kicker (depth stays in measuring-skills.md) — per eval-startups, the benchmark is a flywheel, not a headline - docs/cli.md: "moat #2 (research/…)" → "the capability-diff check" Codified in CLAUDE.md: public-vs-internal-docs now has a strategic- vocabulary clause (moat/measurement-authority/flywheel stay internal; public docs name the benefit), so this doesn't regress. orphan-docs + self-command-refs green; the public↔private split stays docs/ vs research/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): apply multi-persona review fixes From a multi-persona README review (power user / staff eng / plugin author / non-TS user / eng manager): - non-TS users: a visible "No TypeScript needed — lint + inline comments work on any CLAUDE.md" line (was invisible; contradicted progressive adoption) - plugin authors: surface "vigiles scan ranks a marketplace, no key" above the fold (was buried in More) - eng-manager cost: hero now says deterministic layers run free in CI; merged kicker+affordability nails "real-model on a dev's own subscription, never a CI token" - staff-eng skepticism: the 2/7→7/7 claim now reads "measured, not asserted"; the typed-pipeline claim is concretized to the real pipe()/needs() mechanism - fixed a broken blockquote and trimmed the safe-to-repeat para to hold length Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): capture the README design direction as an editor-facing comment A top-of-file comment block records the framing the README must hold: lead with benefits, present compiled specs as the easy/auto-enforced next step (skills write them, hooks compile them; JS→TS evolution, not a wall), keep all four instruments first-class incl. Eval, stay scannable + under the length cap, and no internal vocabulary/research links. Guides future edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): benefits-forward rewrite + specs-are-easy framing Per the README direction: lead with what the user GETS, and present the typed spec as the frictionless auto-enforced next step, not a TS wall. - hero: stronger, concrete failure modes (stale ref / guard fails open / skill never fires), keeps Agent=Model frame, cost above the fold; dropped the redundant centered pitch (merge) - table cells condensed to punchy benefit one-liners - "You don't hand-write specs — your agent does. TypeScript for your harness: opt-in, gradual, auto-compiled, never a chore" (skills write them; JS→TS evolution) - de-dupe cost (drop "not metered tokens" from the Test cell) - "Not for you if …" honest-scope line - docs/verifying-instruction-files.md: back the claim — "you rarely hand-write one; edit-spec/strengthen write it, a hook recompiles on save" Eval stays a first-class instrument. fmt + self-command-refs + orphan-docs green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): don't lead callouts with "typed spec" — benefit first Casual readers shouldn't be greeted by "a typed spec is a program" / "TypeScript for your harness" in the prominent callouts. Reframe both to lead with the outcome and demote the spec to a soft, opt-in aside: - specs-are-easy block: "You don't hand-write any of this — your agent does … you never touch it by hand" (drop the TypeScript-for-your-harness headline) - Lint blockquote: "Bad states stop compiling … turn it up only when you want it" (drop "Markdown is prose; a typed spec is a program") - the edit-spec bullet says "edits the source", not "the typed spec" "No TypeScript?" reassurance and the optional .spec.ts mention stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for the CLI-consolidation + docs/README overhaul session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: prettier-format HANDOFF Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): update the stale generate-types ref the verb rename missed The CI "Check generated types compile" step ran `node dist/cli.js generate-types` — renamed to `generate types` in the verb consolidation, so the job failed with "Unknown command". self-command-refs didn't catch it because it didn't scan .github/; add .github/ to its scan scope so a stale `cli.js <cmd>` ref in a workflow fails the dogfood from now on. Also saves the prefer-compiled-hooks-vs-require-hook-spec decision to research/hook-modes-and-testing.md (the nudge, not a requirement; the write-hook authoring skill is the real authoring lever, unbuilt). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lint): eslint-ignore src/*.md.spec.ts (excluded from tsconfig) The dist-collision fix excluded src/*.md.spec.ts from tsconfig, so eslint's type-aware project service can't resolve src/CLAUDE.md.spec.ts and errors "not found by the project service". It's a tsx-loaded build input (like the root CLAUDE.md.spec.ts, which isn't linted either), so ignore it in eslint to match. Latent since the exclude landed; surfaced now that the earlier generate-types step no longer fails first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
dc293e52e1 |
chore(bench): add token-efficient to A1 manifest + per-task spread reporting (#44)
* chore(bench): add token-efficient to A1 manifest + per-task spread reporting
Expand the ecosystem-benchmark manifest with a second cleanly-A/B-able
compression entry, drona23/claude-token-efficient (MIT, 5.7k stars,
pinned @0d30a6d) — a single injectable CLAUDE.md whose viral "63%" headline
is a word-count over 4 prompts, while the repo's own token benchmark admits
only ~4-12% output-token reduction (a self-documented claim>>measured gap).
Vendor the real file + its MIT LICENSE; record full provenance and the
follow-on (needs-binary: RTK/CodeGraph/Claw/...) and license-blocked
(Context Mode) cluster classification in SOURCES.md/FINDINGS.md.
Also surface per-task variance in the leaderboard: the mean hid that a skill
can help on one task and hurt on another (the caveman review-doc finding), so
report the output-cut range + the helped/hurt split + a MIXED-direction flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF for A1 manifest-expansion session
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): record A1 haiku pilot — token-efficient debunked, caveman variance flagged
First multi-skill A1 run (haiku, caveman+token-efficient × 2 tasks × 2 trials, $0.29):
- token-efficient: claim 63% vs measured -2% output (output GREW, bill +3%) — DEBUNKED,
worse than the repo's own honest ~4% haiku admission (CLAUDE.md input overhead is
net-negative, as its README warns).
- caveman: +30% output cut here vs -18% on the prior same-config run — a 48-point swing
flagged as a variance/noise signal (raise trials >=5; only the structural <1%
output-share fact is robust).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — A1 pilot ran, token-efficient debunked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(agent-runtime): make the active-agent tracker nesting-safe (depth-aware stack)
The PreToolUse tool-contract rail tracked the active subagent in a single
`.vigiles/active-agent.json` slot. Under CC v2.1.172 depth-5 nested subagents
this allowed a CONTRACT ESCAPE: when an inner subagent returned, SubagentStop
cleared the whole slot while still inside the outer subagent, so the gate then
saw "no active agent" and ALLOWED a tool the outer subagent's contract forbids.
Fix (TLC-certified in AgentWindowStack.tla): replace the slot with a depth-aware
STACK — pushActiveAgent on dispatch, popActiveAgent on SubagentStop (back to the
parent frame, not a full clear), gate on the stack top. The open signal now
recognizes both the `Task` and nested-spawn `Agent` tools. Legacy single-slot
`{ agent }` markers read back-compat as a one-frame stack.
The TLC counterexample Open(writer);Open(writer);Stop;Call(Bash) is now a
regression test (pure + a runHook e2e proving SubagentStop pops to the parent).
Still EXPERIMENTAL / not auto-wired; the effect() sub-region it served stays
dropped — the stack ships for when nested contract enforcement is wanted alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — V1 nesting bug fixed
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record the per-file mixed spec/hand-written adoption model
Capture how vigiles behaves in a repo where some skills/agents/CLAUDE.mds have a
.spec.ts and some don't: per-file handling (compile + integrity + ref verification
for spec'd files; surface lint rules still apply to hand-written ones), the
no-spec-mandate defaults (require-spec off, require-skill-spec deprecated), and
scan's spec-agnostic spec-managed-vs-hand-written labeling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add the spec-value model (what a typed spec buys vs markdown)
Consolidate the recurring "spec vs plain markdown" question into one canonical
internal doc (research/spec-value-model.md): the capability axis (lint/test/eval
run on markdown; only whole-harness type-checking is spec-exclusive), the
two-oracle model (behavioral = spec-agnostic vs deterministic contract/outcome =
spec-enabled), honest leg-grading (result-shape/flow/purity held; effect()
sub-region dropped), the surface ranking, and the settled require-*-spec split +
capability-triggered defaults. Slim the duplicative per-file note in
adoption-strategy.md to a pointer; cross-link from typed-spec-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: cite "Agents Need Control Flow" as third-party validation (+ commoditization caveat)
Record Brian Suh's bsuh.bearblog.dev/agents-need-control-flow/ in the landscape
doc: an independent statement of vigiles's Agent=Model+Harness /
deterministic-constraints thesis, with its three failure modes (babysitting /
post-run audits / accepting on faith) mapping onto our verify+test layers. Framed
honestly as double-edged — external validation of the frame, but confirms the
commoditization signal from the same-day competitive check (the control-flow
thesis is now a named, spreading idea; the edge is substrate + measurement, not
the idea). Cite for positioning, do not treat as a moat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record the competitive reality-check in typed-spec-moat (don't re-inflate)
Same-day competitive sweep found the headline typed-spec "moats" are mostly
claimed/commoditizing: riftmap = PR-time blast-radius diff (moat #2 mechanism for
infra), AgentAuditKit/PolicyLayer = capability-diff + drift for MCP servers, Mastra
= typed workflow handoffs (#5), promptfoo = deterministic structured-output oracle
(#7); the control-flow thesis is now a named idea (bsuh). Add a top caveat + a
"Competitive reality check" section recording per-moat verdicts, the one narrow
unoccupied sliver (first-party harness effect-diff from a typed spec), and the
synthesis: the differentiation is substrate + sub-affordable measurement, not the
capabilities — so don't re-inflate the moat next session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — commoditization finding + consolidate shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: correct the competitive analysis for MARKET (it conflated 3 markets)
The "typed-spec capabilities are commoditizing" read was a market-conflation:
Mastra/promptfoo = app-building (zero CLAUDE.md contact), riftmap = infra,
AgentAuditKit/PolicyLayer = third-party MCP-server security — none are in vigiles's
market. Grounded by reading docs: the ACTUAL in-market competitors (claudelint 114
rules, cclint ×2, claude plugin validate) are pure static surface linters that do NO
cross-referencing, typed specs, test/eval, or capability-diff. So in-market vigiles
uniquely owns those layers; it is NOT differentiated on surface linting. Lead with
cross-referencing (unique in-market, works on plain markdown = no adoption barrier).
Add the market-segmented matrix to landscape-mid-2026.md; correct typed-spec-moat
synthesis + the HANDOFF block. Reviews/adoption pass on claudelint/cclint in flight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add adoption pass — field is wide open, agnix is the most-built rival
Reviews/adoption pass on the real in-market competitors: NO incumbent, near-zero
mindshare. Biggest is agnix (296★, ~414 rules, Rust, 7-harness + LSP) — crowds
vigiles's multi-harness/LSP positioning but does NOT do cross-ref/typed/eval; the
rest are 6–41★ hobby projects; none in awesome-claude-code (47k★); npm downloads
are CI-inflated (agnix Show HN = 1 point). Only entrenched thing is Anthropic's
shallow `claude plugin validate`. Add agnix/AgentLint/SkillCheck to the Market C
matrix + adoption signals; rewrite the read: don't out-rule-count agnix or match its
LSP — lead with cross-referencing ("valid≠true") + the test/eval tier; the real
battle is distribution, not capability (reinforces A1). Update HANDOFF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: capture poach-list, agnix lesson, badge/viral/acquisition GTM, Zod result()
Following the market-segmentation correction: record the useful-but-not-competitor
ideas + adoption plays.
- landscape: the agnix signal (most-built linter, 1 HN pt → linting-without-eval
doesn't pull a community = evidence for the measurement wedge) + a poach list from
the app-building non-competitors (Zod-schema'd contracts, promptfoo assertion vocab,
riftmap distribution model) + business-on-top validated by closed SaaS rivals.
- roadmap §Explore + distribution-strategy: tiered README badge (lint→test→eval funnel
as a growth loop), viral debunk articles (measurement-as-marketing), open-core +
build-on-top + AI-lab-acquisition posture.
- measurement-authority: competitive confirmation of the pivot (agnix → eval is the
puller; no incumbent → distribution is the bottleneck).
- feature-ideas §14b: Zod-schema'd result() (one schema → type + validator + JSON
Schema), poached from Mastra/Pydantic AI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: fix rule-count error (20 not 31) + 11★ honesty; dedup/crosslink the cluster
- Correct the Market-C matrix: vigiles ships ~20 semantic rules (not 31 — that was
the dogfood CLAUDE.md's section count), with an apples-to-oranges caveat (agnix's
414 are micro-checks; vigiles's rules cross-reference) and the real 11★ baseline.
- Reconcile typed-spec-moat's internal contradiction: the top caveat + §7 title said
"commoditizing", overturned by the later market correction — both now say
"capabilities live in adjacent markets; vigiles's own market is wide open".
- Record the PUBLIC plugin leaderboard idea (A1 + leaderboard.ts published as a
claim-vs-measured ranking, head-to-head per category) in roadmap §Explore.
- Cross-link the two matrix hubs (spec-value-model "what a spec buys" ↔ landscape
"who competes").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): record A1 sonnet pass (5 trials) — both compression skills debunked
Sonnet, caveman + token-efficient × {slugify,debounce,review-doc} × 5 trials ($3.43):
- caveman: claim 75% vs measured -18% output (range -55..+8, MIXED), bill +8% — the
-18% is now STABLE (matches the prior sonnet 3x3; haiku +30% was noise). review-doc
(its supposed best case) is the WORST at -55% (output grew 55%).
- token-efficient: claim 63% vs measured -10% output, bill +3% — WORSE than the repo's
own ~12% sonnet admission; review-doc -35%.
- Both: 0 regressions, output ~0.7% of session tokens (the model-agnostic structural
kill). Neither compresses on multi-turn coding; both cost more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — A1 sonnet pass done, numbers publishable
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): audit caveman/token-efficient OWN benchmarks — why they report a win
Read their benchmark code (caveman/benchmarks/run.py, drona23/.../benchmark/run.py +
SUMMARY.md). Record the methodology gap that explains the headline numbers:
- caveman run.py: single-shot one-turn Q&A (no tools/files/multi-turn), output-tokens-
ONLY (ignores the SKILL.md system-prompt input + cache that dominates real sessions),
vanilla "You are a helpful assistant." headline baseline (the concise-baseline harness
lives separately in evals/), and no correctness gate.
- token-efficient: the 63% is a WORD count; the repo's own SUMMARY.md already admits the
token effect is -2%..-11% and "not agentic/coding loops" — vanity headline vs honest
fine-print, and even the fine-print doesn't reproduce on coding.
Also corrected the prior over-stated "baseline discrepancy" point to the fair version
(headline uses vanilla baseline; author didn't hide the concise comparison).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: record caveman-debunk SATURATION — lead with method/leaderboard, not caveman
Reality check (web-confirmed): the caveman critique is ~6mo old and crowded
(Kuba Guzik 14-21%, GrowwStacks, HN top comment, Decrypt, devneeddev/X; author
conceded; token-efficient author self-debunked). We are NOT first on the verdict.
Our novelty is the METHOD (reproducible multi-turn correctness-gated harness +
code audit + the output-GROWS finding + N-skill scale), not the conclusion. Add a
saturation warning to FINDINGS and reframe the roadmap article bullet: method/
leaderboard-first, caveman as a validation row, don't make it the flagship.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record benchmark/leaderboard prior art for the A1 leaderboard
Web sweep of who already benchmarks/ranks Claude Code skills, to position the
public leaderboard: (1) subjective "best skills" listicles abundant (SEO, no
measurement); (2) install/adoption leaderboards exist (quemsah n8n metrics +
published install counts — popularity, not effectiveness); (3) a few one-off
measured benchmarks (caveman saturated; a real superpowers 12-session A/B that
VINDICATES it — 9% cheaper/14% fewer tokens; MindStudio "5 skills 70%" claim-
pushing). Open slice vigiles owns: reproducible + multi-skill + claim-vs-measured
+ correctness-gated + head-to-head + continuous + open-method. Position on the
claim-gap + reproducibility + correctness axes, not "ranking plugins" (install
boards exist) — and honestly (some claims, e.g. superpowers, hold).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — methodology audit, caveman saturation, leaderboard-next
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF competitive block under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): archive canonical eval runs (the data behind FINDINGS survives)
The raw per-run JSON is gitignored scratch under results/; this commits the haiku
+ sonnet canonical runs (the data the published FINDINGS numbers are computed from)
into results-archive/ so the measurement data — the moat — survives a container
reclaim, is auditable/reproducible, and never needs a paid re-run to recover. Adds
a README with provenance + schema + method/reproduce pointers. Curated: only runs
cited in FINDINGS get archived; routine runs stay in results/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: prettier-ignore archived run JSON (machine-written archival data)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: refine cross-ref moat — harness-native is the value, linter-catalog is legacy
The cross-check engine's value is NOT uniform: HARNESS-NATIVE references (tool
contract, MCP server#tool LIVE resolution, hook events, file/script paths,
delegate targets) are the moat — agent-era-specific, fail silently, nobody else
checks them. The 7-linter-catalog integration (ESLint/Ruff/...) is the founding
feature but the WEAKEST leg (linter self-enforces regardless of the prose mention;
heaviest engineering, least harness-specific payoff) — keep it as supporting, don't
add more catalogs. Deepen harness-native instead; first build = live MCP tool
resolution of the real mcp__server__tool refs. Updated landscape (REFINEMENT block),
roadmap §Now, HANDOFF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(mcp): live-verify real mcp__server__tool contract refs (harness-native cross-check)
Bridge the live MCP engine (listMcpTools/closest in mcp.ts, until now wired only to
the explicit `vigiles:mcp` mark) to the agent's REAL `mcp__server__tool` references.
verifyMcpContractTools starts each referenced+declared server once and checks the
TOOL actually exists on it — catching the create_issue→issue_write rename rot that
the static verifyMcpToolServers (server-declared only) misses. Refs to undeclared /
built-in / plugin-namespaced servers are skipped (the static check's job; we have no
config to start them). New mcpToolParts (mcp-tool.ts) extracts {server,tool};
mcpContractToolMessage renders the did-you-mean. 5 tests against the real
fixture-mcp-server.mjs (CI-safe, offline): existing-resolves, renamed→suggestion,
undeclared-skipped, plugin-namespaced/non-MCP-skipped, unreachable-server. Opt-in
(starts processes) — engine first; scan/CLI wiring is the next increment.
This is the harness-native cross-check moat (the valuable leg), not the linter-catalog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — MCP contract-tool resolution shipped; trim to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record first live-MCP dogfood finding — 3 dead tool refs in MadAppGang tester.md
Dogfooded verifyMcpContractTools (the new live MCP tool-resolution) against the real
vendored MadAppGang frontend plugin + the real chrome-devtools-mcp server: tester.md's
tools: lists 27 mcp__chrome-devtools__* tools, 3 of which no longer exist on the live
server — emulate_cpu + emulate_network (consolidated into `emulate`) and
navigate_page_history (removed). 24/27 resolve. Classic MCP rename rot, invisible to
every static linter (the plugin declares no server, so even mcp-tool-resolves is
silent). Recorded in the public-disclosures table + a method/repro note. The
harness-native cross-check moat validated on real OSS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — MCP live-resolution dogfooded, found real MadAppGang bug
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record live-MCP-check scope — high-value but narrow (1/4 vendored plugins)
Dogfooded verifyMcpContractTools on all 4 vendored plugins: only MadAppGang pins
mcp__server__tool refs (the 3 dead-ref finding); superpowers/wshobson reference no
MCP tools; oh-my-claudecode declares a server but no surface pins its tools. So the
live check is sharp but narrow — bites only where a contract hard-lists specific MCP
tool names. Recorded honestly so we don't overstate its breadth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EOF
)
* feat(scan): wire live MCP tool resolution behind opt-in scan --verify-mcp
Expose verifyMcpContractTools through the scan surface: `scan --verify-mcp`
starts each declared MCP server and checks that every mcp__server__tool the
plugin's agents reference actually exists on it — the dynamic check no static
linter can do (catches rename/removal rot like create_issue->issue_write).
Opt-in and side-effecting (spawns servers); returns [] when no servers are
declared. Static verifyMcpToolServers still owns undeclared-server refs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — scan --verify-mcp shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(scan): measure cross-skill selection collision (scan --collisions)
The behavioral confirmation of the deterministic description-overlap rule: run
each model-invocable skill's own prompts against the whole installed plugin and
record WHICH skills fired (whichSkillsFired) — an N×N matrix whose diagonal is
recall and whose off-diagonal mass is one skill HIJACKING a sibling's prompt.
The cross-skill precision question per-skill trigger-rate can't see; the
blast-radius column the leaderboard needs.
- eval.ts: whichSkillsFired + runSkillSelectionTrial (one pass yields the whole
matrix, N× cheaper than per-pair; bodies stubbed so a run stops at selection).
- scan-behavioral.ts: buildSelectionReport (pure fold) + measurePluginSelection
(real claude) + measurePluginSelectionWith (injectable). Claude Code only —
Codex has no discrete skill-selection event.
- CLI: scan <dir> --collisions --prompts=<file.json>.
- Dogfood (on the sub): vigiles's own 3 skills are collision-free; bonus signal
edit-spec under-fired (recall 33%). superpowers slice can't fire (missing
gateway skill). See research/plugin-selection-collision.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — selection-collision matrix shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: verify superpowers null result — hook-primed plugin, harness drops hooks
The superpowers 0% recall is a measurement artifact, root-caused from source:
its skills activate via a SessionStart hook that injects the using-superpowers
gateway priming, but the trigger/collision tier stubs the plugin to skills-only
(packageSkillsDir drops hooks/), so the gateway never runs — compounded by the
vendored slice missing using-superpowers/SKILL.md. Confirmed 0% in --trigger too
(not collision-specific). Records the limitation + a roadmap follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(scan): label hook-primed 0%-recall as a measurement artifact (Layer 1)
A stubbed trigger/collision run rebuilds the plugin to skills-only and drops the
plugin's hooks — so a SessionStart hook that primes skill selection (superpowers'
using-superpowers gateway) is lost, and recall collapses to a misleading 0%.
Layer 1 (honesty before fidelity): when a stubbed run on a SessionStart-hooked
plugin shows EVERY measured skill at recall 0, relabel it "hook-primed — likely a
measurement artifact; re-run against the full plugin install" on both the trigger
and collision columns (hasSessionStartHook + isStubbedHookArtifact). The all-zero
gate is the precision guard — a genuine single-skill miss stays reported as real.
Verified on the live superpowers run + 3 CI fake-driver tests (hooked→labeled,
non-hooked 0%→stays real, trigger relabel). Layer 2 (a --no-stub faithful path +
auto-fallback) is the deferred follow-up in research/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — Layer-1 hook-primed artifact label shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(leaderboard): publishable Markdown table + real names; v0 corpus run
Make the structural-health leaderboard a publishable artifact:
- formatLeaderboardMarkdown — a ranked Markdown table (README/gist/site form),
wired as `scan <dirs...> --md`.
- rankPlugins now labels each plugin by its manifest name (.claude-plugin/
plugin.json) instead of a SHA-pinned dir basename.
bench/leaderboard/ — a reproducible v0 generator (run.mjs + corpus.json) that
clones public plugin repos, finds on-disk plugin roots, ranks them, and writes
RESULTS.md (scores only; no plugin code vendored). First run: 83 plugins from
obra/superpowers + wshobson/agents — grade spread A:16 B:28 C:17 D:9 F:13, with
real outliers (a dead agent tool, large uncontracted/untested collections at F).
The deterministic substrate for the public plugin leaderboard (the adoption
engine). Behavioural columns (trigger-rate, collisions) stack on top later.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — leaderboard v0 shipped, broaden-corpus next
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cli): capability-diff — does this change widen the agent's blast radius? (moat #2)
The bridge bet, v0. `vigiles capability-diff <before> <after>` computes each
version's whole-harness capability lattice (computeHarnessCapabilities — read-only
/ side-effecting / unknown-MCP tools + loosest purity) and diffs them:
- src/core/capability-diff.ts (pure, harness-agnostic): diffCapabilities +
formatCapabilityDiff + isNoOpDiff. WIDENED iff a new side-effecting/unknown tool
or a loosened purity floor; new read-only + removals reported but NOT a widening.
- CLI handler: informational by default (exit 0, a PR comment), --fail-on-widen for
the opt-in CI gate (don't cry wolf — widening is often intended).
6 pure-core tests + 3 CLI e2e (the exit-code contract). v0 reads the tool-bucket
lattice; the richer per-step effect-row + cross-step accumulation (turning "gained
Bash" into "opened an exfil path") is the deferred M1 follow-up in research/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — capability-diff #2 v0 shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cli): consolidate behavioral measurement into a `measure` verb
Bad DX cleanup. scan had accreted ~14 flags by bolting every capability onto it.
Re-grounded on one rule: a verb per JOB/LAYER, flags for modifiers.
- NEW `vigiles measure <dir> --prompts=p.json` — the model-gated behavioral report
(trigger-rate recall+precision AND the selection-collision matrix, one report).
The OSS-testing front door, replacing `scan --trigger` + `scan --collisions`.
- scan goes back to FREE/deterministic: the behavioral flags move to `measure`.
- capability-diff is a FREE check → folded from a standalone command into a scan
flag: `scan <after> --capability-diff=<before>` (was `capability-diff a b`).
- printUsage now lists scan + measure (scan was missing entirely).
- Hand-off text (optimize/score-explainer) + docs/cli.md point at `vigiles measure`.
Net: scan ~14 flags → ~6 (all deterministic); one new justified verb (the missing
paid-measure layer); one needless verb removed. Underlying engine unchanged
(probePluginTriggers / measurePluginSelection / diffCapabilities). Full suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — CLI consolidated into measure verb; lint/scan fold open
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: poach the app-frameworks for a typed CLAUDE.md (web-verified moat synthesis)
research/typed-claude-md-poach.md — consolidates + extends the scattered poach notes
into one moat doc, web-verified (2026-06-22) against Mastra / LangGraph.js / Pydantic
AI / Vercel AI SDK / Inngest / Temporal / Trigger.dev / Effect-TS.
The thesis: they type the agent you BUILD IN CODE and leave instructions a prose hole
(`instructions: string`); vigiles types the prose harness they don't. Verified moat
points: (1) the whole-harness REGISTRY (one tsc over every inter-agent edge) is unique
— nobody else emits a repo-scale typed handoff registry (the routeTree.gen.ts analogy);
(2) cross-agent handoff checking is universally weak/manual even in the typed frameworks
(Pydantic "no auto-derived cross-agent Supplies check"); (3) nobody types the
tool-permission/capability surface at compile time (the effect-row gap). Ranked bets:
effect-row (M1, headline) > Zod-result() > typed context/needs > exhaustive err-track
(poached from Effect-TS, novel).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: the reliability-RUNTIME moat — type the harness's dynamic structure
research/harness-protocol-flow-moat.md (deep FP pass, web-verified 2026-06-22 across
the Codex/CC failure corpus). The existing FP docs type the harness's static capability
SET (effect-row, capability graph, trifecta). This opens the THREE dimensions a set
discards — and reframes them as a category, not a linter:
- ORDER (typestate/session types): a destructive action is UNREACHABLE until its guard
step ran (kills destroy-without-backup: #50027 DROP SCHEMA, terraform destroy, rm -rf).
- FLOW (information-flow/noninterference): untrusted input cannot reach a sink — the
trifecta as a real source→sink path (CVE-2026-21852, Trail of Bits line-jumping).
- REPLAY (linear types/idempotency): exactly-once side effects — the unsolved
duplicate-on-replay hole (ACRFence, 45 issues/12 frameworks; idempotency keys fail
for LLMs). Multiparty session types = deferred 4th axis.
The MOAT framing per the new steer: this is a deterministic RELIABILITY RUNTIME (the
spec→PreToolUse-gate enforcement, surviving compaction OUTSIDE the context window), not
a lint rule — "make the harness measurably more reliable," provable via the eval layer.
Grounded in: prose-doesn't-bind (#32163 "@enforce"), declared≠enforced (SDK #172/#162),
METR <10%@4h, 56% no-recovery, trifecta in 98% of prod agents.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reframe the moat as a reliability RUNTIME (3 axes) + HANDOFF
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guards): prototype typed safe-by-construction harness guards (EXPERIMENTAL)
Dogfooded the real OSS hooks (our pre-edit.sh block-on-compiled, superpowers/OMC
session-start inject, OMC keyword-detector) → the safety-critical shape is the
PreToolUse gate. A hook today is arbitrary shell (the RCE footgun, CVE-2025-59536)
and its safety is undecidable (Rice). So invert it: DECLARE a guard from a closed
vocabulary, GENERATE the hooks block pointing at vigiles's own gate (no user shell).
src/core/guards.ts (not wired to CLI/public API — a sketch):
- guard.block / requireBefore (the ORDER axis: destroy-after-plan) / confine
- decideGuards — the pure runtime decision (reuses arg-match's matchesArgs)
- compileGuards — generates a Claude Code PreToolUse hooks block whose command is
`npx vigiles guard-hook` (vigiles's audited gate), safe-by-construction
- 4 tests: block reproduces pre-edit intent, requireBefore denies destroy-before-plan
+ allows after, confine denies an out-of-allowlist path, compile emits a valid block
See research/harness-protocol-flow-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(roadmap): reliability-runtime moat + ORDER/FLOW/REPLAY (REPLAY=P3), crosslinked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guards): runnable guard-hook gate — session ledger + ORDER enforcement live
Make the EXPERIMENTAL typed-guard prototype run end-to-end. The `vigiles
guard-hook` PreToolUse subcommand reads the live event, loads the declared
guard set (`.vigiles/guards.json`) + the session ledger
(`.vigiles/guard-ledger.json`), runs `decideGuards`, and blocks (exit 2 +
reason) or records the allowed call so a later `requireBefore` sees it.
- guards.ts: serializeGuards/parseGuards (RegExp-safe, mirrors tool-intercept's
env round-trip), loadGuards, readGuardLedger/recordGuardCall (the prior-call
ledger CC doesn't surface to a hook), parseGuardEvent, runGuardHook (allow →
record; deny → record nothing — a blocked call never happened).
- cli.ts: `guard-hook` subcommand mirroring agent-hook.
- Tests: serialize round-trip incl. RegExp, tolerant parse, ledger IO, and a
runHook e2e proving `terraform destroy` is blocked before `terraform plan`
and allowed after — the ORDER axis enforced across hook invocations, live.
The gate's command is vigiles's OWN audited binary, never user shell —
safe-by-construction. Still not on the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): guard-hook gate runs end-to-end (ORDER axis live)
* feat(hooks): spike — typed, effect-classified hook specs (the "type-safe bash" angle)
A spike to test whether the compile step earns its keep on the hook surface —
the least-typed, most-footgun-prone part of a harness. Catches two silent bugs
hand-written shell can't:
1. WRONG-FIELD EXTRACTION (silent no-op): a hook matching `Bash` but reading
`file_path` extracts empty forever and never fires. validateHook makes it an
error; the typed `hookFor` builder makes it a tsc error at edit time (proven
by an @ts-expect-error the build enforces).
2. EFFECT MISDECLARATION ("type-safe bash"): an observe-only hook that runs a
mutating/undecidable command doesn't compile — rides bash-effects.ts, the
deterministic classifier no other plugin tool has.
compileHook refuses (throws) any error-level issue — "an unsafe hook doesn't
compile" — and emits the settings block + correctly-typed field extractions
(never a hand-typed jq string). Pure core, harness-agnostic: the per-tool field
catalog is injected (Codex differs), the CC catalog + typed builder live in the
test. Not wired to the CLI/public API. See research/harness-protocol-flow-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): save the verified hook pain-point corpus + the verify-not-gate killer feature
Five web-research passes (claude-code issues, official docs, dev.to/Medium, HN,
CVEs), all verified. The convergent #1 pain is FALSE CONFIDENCE — a safety hook
looks like a guardrail and silently isn't (exit 1≠2, wrong JSON field,
PostToolUse-can't-block, wrong jq path); RFC #45427 names the thesis and was
closed not-planned. Killer feature: "prove your guardrail actually blocks" via
runHook + decideHook (verify the decision LOGIC, not deliver enforcement) — works
on existing hooks with no spec/compile, sidesteps CC's delivery bugs
(subagent-bypass #34692 etc.). Compile is the optional Level-2 (declared intent →
auto-generated test), not the mechanism. Crosslinked from harness-protocol-flow-moat
+ roadmap (new HIGH item; gate demoted to MED — undercut by the same CC bugs).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(unit): prove your guardrail actually blocks — disaster-event battery over runHook
The #1 verified hook pain is FALSE CONFIDENCE: a safety hook looks like a guard
and silently doesn't block (exit 1≠2, wrong JSON field, wrong jq path) — "three
teams believed they'd blocked force pushes." This is the deterministic answer.
- src/guardrail-check.ts: a curated DISASTER_CATALOG (force-push incl. compound,
reset --hard, rm -rf, --no-verify, ssh-key read, curl|sh) + verifyGuardrail
(feeds each to a hook via runHook, reports blocked), assertBlocksDisasters (the
CI gate — throws naming the misses), formatGuardrailReport (NEUTRAL coverage
map — never judges intent, since not every hook is a bash-safety guard).
- Exposed on vigiles/unit (no model, no key, CI-runnable; works on hand-written
hooks with no spec). api reports regenerated.
- scaffold-test: a hook scaffold now emits the runnable battery (coverage map +
the commented assertBlocksDisasters CI gate keyed by declared category).
- Committed receipt: exit-1 vs exit-2 force-push guards, the engine tells them
apart; intent-declared assert passes for git, fails honestly when a guard
claims fs-safety it doesn't deliver.
Verifies decision LOGIC, not delivery — flags CC's runtime bugs (subagent-bypass
#34692 etc.) but never claims to fix them. See research/hook-pain-points.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): guardrail VERIFY feature shipped; refresh shipped-this-session
* docs(research): record guardrail VERIFY shipped + the neutral-map vs declared-intent design lesson
* docs(research): real-plugin dogfood — disler hook blocks rm/.env but no-ops vs force-push/ssh-read/curl|sh
* feat(spike): hook-as-typed-program probe — testability/safety/eval/portability + stamping
Probe the constrained-API compiled-hook model: a hook is a PURE typed function
(event)=>Decision against a closed `vigiles/hook` vocabulary, vigiles compiles it.
src/core/hook-program.ts + a 5-claim test, all passing in-process (no subprocess,
no model):
1. PURE/TESTABLE — decide() unit-tested directly; the false-confidence protocol
(exit 1≠2, JSON field) is unrepresentable (compiler emits exit-2).
2. AST-BACKED MATCHING — command.runs("git push",{force}) via new bash-effects
`leafCommands` (AST leaf extraction): catches the compound bypass
`cd x && git push -f` the native glob (#30519) misses AND avoids a grep
false-positive on `echo "...git push..."`.
3. COMPILES to a real CC hooks block.
4. CAPABILITY = API SURFACE — checkHookImports rejects any import outside
vigiles/hook (+ eval/Function/dynamic-import); an out-of-API hook does not
compile.
5. STAMPING (the "fix #4" idea) — stampHook/verifyHookStamp make the compiled
artifact tamper-evident (integrity.ts pattern): a hand-edit smuggling
child_process breaks the stamp → runtime refuses it.
Pure core, harness-neutral, not on the public API. Honest limits unchanged
(buy-in, node latency, CC subagent-bypass #34692 — compile fixes authoring not
delivery). See research/hook-pain-points.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record compiled-hooks probe result (5 claims proven incl. stamping)
* feat(spike): widen hook-program probe — path-confine gate + context-injection (vocabulary holds)
Probe 2 tests whether the closed vigiles/hook vocabulary holds across two
genuinely different shapes. It does — and reveals the vocabulary is a small
FAMILY keyed by hook ROLE, whose per-role output type eliminates two more
verified pains by construction:
- A second GATE: defineFileGate over Edit/Write with a PathView.under() matcher
(different tool, field, matcher) — confines writes to src/test, denies escapes.
- A non-gate: defineInject (SessionStart) returns Injection (context text), not a
Decision. runInject emits the RIGHT JSON field (additionalContext) — the author
never picks it, so the wrong-field pain can't occur.
- Correct-by-construction: an inject hook CANNOT return deny() — it's a tsc error
(proven by @ts-expect-error the build enforces). "Block on a SessionStart/
PostToolUse hook" (a documented mistake) becomes a type error, not a silent no-op.
Finding recorded in research/hook-pain-points.md. Still a pure-core spike, not on
the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): record probe 2 — hook vocabulary holds as a role-keyed family
* feat(spike): probe 3 — PostToolUse react hook (bounded side effects complete the family)
The third hook shape: a react hook (PostToolUse) fires after the tool ran, so it
can't block — its job is to DO something (format, recompile, warn). That's where
side effects re-enter (gate + inject are pure). The constrained API keeps them
BOUNDED: a react returns a typed Reaction whose run(cmd) is EFFECT-CLASSIFIED at
construction (bash-effects), so even the side-effecting role stays analyzable —
you can list/diff exactly what every react runs and its effect. And a react still
CANNOT block (no deny in Reaction) → "block on a PostToolUse hook" (a documented
mistake) is a tsc error (build-enforced @ts-expect-error).
Completes the role-keyed family: gate (decide→Decision, pure) / inject
(produce→Injection, pure) / react (react→Reaction, bounded side effects). Each
role's output type makes its wrong-output bugs unrepresentable. Finding recorded
in research/hook-pain-points.md. Pure-core spike, not on the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: probe 3 — hook vocabulary sound across the full role-keyed family (gate/inject/react)
* docs(handoff): condense the hooks narrative under cap
* docs(handoff): center the in-flight compiled-hooks ship arc (steps 1-3) for resume
* feat(hook): ship compiled hooks — author a hook as a pure typed program
Ship the `vigiles/hook` public surface + CLI so a hook is authored as a pure
typed `(event) => Decision` against a CLOSED vocabulary and vigiles compiles
it. This ELIMINATES whole hook bug classes by construction: the author never
writes the exit code / JSON field (the false-confidence class is
unrepresentable), the matcher is AST-backed (catches `cd x && git push -f`
the native glob misses), capability = API surface (an out-of-vocabulary import
does not compile), and the compiled artifact is tamper-evident (a hand-edit
that smuggles a capability breaks the stamp → the runtime refuses it).
- `src/hook.ts` — the `vigiles/hook` barrel (gate/inject/react vocabulary).
- `package.json` exports `./hook`; api-extractor tracks the new surface.
- CLI `compile-hook <file>` (capability-checks, emits the settings block +
stamp sidecar) and `run-hook-program <file>` (the runtime: loads the typed
program, verifies the stamp, dispatches by role — gate→exit 2, inject→
additionalContext, react→classified run).
- Expand CommandView with two principled high-signal matchers: `touches()`
(secret-path reads) and `pipesToShell()` (curl|sh remote code).
- OSS dogfood (src/hook-dogfood.test.ts): a hand-written substring guard (the
widely-copied disler shape) misses 5/7 of the disaster battery; the compiled
hook blocks 7/7 by construction — measured with our own DISASTER_CATALOG.
Honest scope kept in the surface docs: compile/verify fix AUTHORING + LOGIC,
not DELIVERY — CC's subagent-bypass (#34692) still bypasses any PreToolUse
hook, so a gate is a strong default, never an unbypassable wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reposition around harness reliability + the compiled-hooks pillar
Rewrite the README top-line from "lint/test/measure" to "make the harness your
AI agent runs on reliable", with four instruments under it (Verify / Guard /
Test / Measure) — Guard is the new compiled-hooks pillar, not a bolt-on. Add a
README ② Guard section and a full docs/compiled-hooks.md guide that says, plainly
and up front, that compiled hooks make WHOLE CLASSES OF BUGS UNREPRESENTABLE
(false confidence, matcher bypass, capability creep, category mistakes), with a
bug-class table and the OSS dogfood result (2/7 → 7/7).
Cross-link the new guide from the README, the docs index, the CLI reference
(compile-hook / run-hook-program documented), and the sibling verify + test
guides. Internal: add the compiled-hooks positioning paragraph + keyFiles to
CLAUDE.md (via its spec) and recompile.
Honest scope kept everywhere: compile/verify fix the hook's authoring + logic,
not the harness's delivery — a subagent's tool calls still bypass any PreToolUse
hook (#34692), so a gate is a strong default, never an unbypassable wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): design the Codex compiled-hook adapter + record the ship
Add research/compiled-hooks-codex.md: the plan to compile one typed vigiles/hook
program to Codex through the existing ports — no core edits. The key finding is
that the typed program, AST matcher, capability check, stamp, and runtime
decision are all harness-neutral, and Codex's veto is exit-2-identical to Claude
Code's, so the runtime is shared; only the EMIT differs (CC JSON settings vs
Codex TOML [hooks], glob vs anchored-regex matcher), routed through the
PluginLayout.settingsFormat + dialect/HookProtocol ports already in place. Notes
the boundary discipline (ports by injection) and the non-CC-shaped dogfood, and
why OpenCode's in-process hooks are a different emit target.
Record the compiled-hooks ship + the OSS dogfood in hook-pain-points.md and
cross-link from codex-prototype-findings.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): record the compiled-hooks ship + reposition; next = Codex emit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(hook): compile a typed hook to Codex (TOML) as well as Claude Code
Generalize the compiled-hook emit so ONE typed vigiles/hook program compiles to
either harness, through the existing adapter ports — no harness-boundary edits.
The typed program, AST matcher, capability check, stamp, and the gate runtime
are all harness-neutral (Codex vetoes a tool call via exit 2 identically), so
only the emitted settings block differs.
- compileHookProgram(source, hook, gateCommand?) → (source, hook, opts?) where
CompileHookOptions = { gateCommand?, dialect?, hookProtocol?, settingsFormat? },
all optional and defaulting to the Claude Code JSON block (back-compatible).
It validates hook.on against the dialect (verifyHookEvents — a typo'd or
unsupported event won't compile), styles the matcher per
HookProtocol.matcherStyle (CC exact `A|B` vs Codex anchored `^(A|B)$`), and
renders settingsBlock per settingsFormat (CC nested JSON vs Codex TOML
`[[hooks.<event>]]` via @iarna/toml).
- Add HookProtocol.matcherStyle?: "exact" | "regex" (additive); codexHookProtocol
sets "regex".
- compile-hook resolves the target harness (--harness= / detect) and threads the
adapter's dialect/hookProtocol/layout.settingsFormat in; prints the right block
+ target ("add to Codex's config.toml").
Tests (non-CC-shaped, per the adapter-aware-lint discipline): the Codex emit is a
TOML `[[hooks.PreToolUse]]` block with a regex matcher; an unsupported event does
NOT compile; the CC default still emits JSON + exact matcher. Docs: cli.md +
compiled-hooks.md note the multi-harness emit; research/compiled-hooks-codex.md
marked BUILT with the one deferred piece (inject/ask output JSON on Codex). Also
fix 3 pre-existing no-confusing-void-expression lint errors in
guardrail-check.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): Codex compiled-hook emit built (step 4)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(hook): make the Codex inject/react gap LOUD, not silent
The deferral of Codex inject/ask OUTPUT (its field shape is CC-confirmed only)
was correct — guessing a wire format would recreate the silent-failure bug the
feature exists to kill — but it was SILENT: `compile-hook --harness=codex` on an
inject/react hook emitted a clean-looking TOML block whose runtime output is
CC-shaped and may quietly no-op on Codex. Now it warns loudly (⚠) that the
output is unconfirmed for that harness and to use a gate hook meanwhile; a gate
hook (deny→exit 2, cross-harness) compiles with no warning. E2E test covers
both. Docs/research note the loud behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revamp the compiled-hooks docs — "eliminates a class of bugs", link sources, explain the cons
- Lead the bug-class section with "eliminates an entire class of bugs by
construction", and link EVERY class to its verified source: false confidence
(RFC #45427 / #24327), matcher bypass (#30519), capability creep
(CVE-2025-59536), category mistakes (#4362).
- Replace the single delivery-floor caveat with a full "Limitations & trade-offs
(the cons)" section: the delivery floor (#34692 + route-around #45427/#32376),
runtime cost (node spawn per event), buy-in (dep + build step vs inline bash),
the bounded-vocabulary ceiling, "compiling proves the protocol not your policy",
and the unconfirmed Codex inject/ask output.
- Integrate the gate instrument into the reliability frame: README Guard section
uses the "eliminates an entire class of bugs" framing + links to the cons; the
Test guide intro names the four instruments (verify / guard / test / measure)
instead of only a bottom see-also.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(compiled-hooks): clarify the ceiling — capability closure (fundamental) vs matcher coverage (extensible)
The bounded-vocabulary con conflated two surfaces: what a hook can DO (the
deliberate no-fs/net/exec capability closure — the real ceiling and the safety
guarantee) vs what it can SEE (the AST matchers, a soft/extensible limit). Split
them so 'some hooks stay hand-written' reads as the capability rule, not a
weakness of the Bash matcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(hook): in-process testing for compiled hooks — runHookProgram + assertHookDenies/Allows
A compiled hook's decision is a pure function, so it should be testable WITHOUT
spawning the CLI. Add the cheapest test tier:
- `runHookProgram(hook, event)` (core, on `vigiles/hook`) — one in-process
dispatcher over the whole role family: a gate → its Decision, an inject → the
injected context, a react → its classified Reaction, returned as a normalized
discriminated `HookProgramOutcome`. No subprocess, no model.
- `assertHookDenies` / `assertHookAllows` (harness-assert, on `vigiles/unit` +
`vigiles/testing`) — the throwing asserts over a compiled hook program
directly, the sibling of assertHookBlocked/assertHookAllowed (which take a
`runHook` subprocess result).
Tests: runHookProgram dispatches all four roles (hook-program.test.ts);
assertHookDenies/Allows pass/throw with a message naming what it got
(harness-assert.test.ts). Docs: a "Testing a compiled hook" section in
docs/compiled-hooks.md (the three tiers by cost) + a testing-api.md reference
entry; CLAUDE.md keyFiles updated. api-extractor tracks the new surface.
Behavior testing is deliberately NOT added: a hook isn't model-selected and its
output isn't prose, so there's no trigger-rate/judged dimension; "does it fire in
the assembled harness" is the existing runHarnessTest tier.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): record in-process hook testing (step 5)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: link the parked prototype code so it's discoverable later
The unshipped/experimental code wasn't in the Key Files map — a future session
would only find it by grep. Fix the traceability:
- Add CLAUDE.md Key Files entries (via the spec) for the parked prototypes:
`src/core/guards.ts` (+test, WIRED via `vigiles guard-hook` but EXPERIMENTAL —
the GATE-axis bet, superseded in practice by compiled hooks + VERIFY) and
`src/core/hook-spec.ts` (+test, a pure spike imported NOWHERE, superseded by
the shipped compiled hooks). Each says why it's parked + points at
research/harness-protocol-flow-moat.md to resume.
- Add a "parked prototype files" note under the roadmap's guards bet recording
the lineage (hook-spec's typed-effect-classified idea → shipped compiled hooks)
so the durable map carries it too.
No code change; recompiled CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(hook): cover the assert message branches (100% threshold)
The new assertHookDenies/assertHookAllows helpers dropped coverage below the
repo's 100% global threshold (harness-assert.ts:144-145,165 — the describeOutcome
injection/reaction branches + the assertHookAllows fail path were unexercised).
Add asserts that drive each: assertHookAllows throws on a deny, and
assertHookDenies against an inject / react hook fails with a message naming the
role ("an injection" / "run (a reaction)").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(lint): add a config `exclude` so vendored/benchmark fixtures skip require-spec
The dogfood `vigiles lint` (the CI `check` job) failed `require-spec` on
`bench/ecosystem/skills/token-efficient/CLAUDE.md` — a SHA-pinned third-party
file injected verbatim as a benchmark arm (SOURCES.md: "the real CLAUDE.md"). A
per-file disable comment would perturb that verbatim arm + dent its fidelity, so
add a config-level exclude instead: `exclude` globs in .vigilesrc.json drop paths
from `lint` instruction-file discovery (node_modules/dist/.git stay always-out).
Set `"exclude": ["bench/**"]` — benchmark data isn't the repo's harness, so the
repo's own lint shouldn't police it. The vendored file is untouched.
`node dist/cli.js lint` now exits 0. Documented in docs/rules/require-spec.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(eval): cover runSkillSelectionTrial / whichSkillsFired (100% statements)
The `test` job's 100% statement gate failed on eval.ts (99.42%): the
selection-collision primitives runSkillSelectionTrial + whichSkillsFired (from
|
||
|
zernie
|
3b81e5042b |
feat: spec-contract hardening + complete spec/railway docs + subagent-trace eval fix (#42)
Spec: context:fork + gated forked-skill output, agent color/disallowedTools, dir()/glob(), didNotWrite(), scaffold-test, section length-guard, migrate->adopt-spec. Docs: docs/spec-format.md completed + docs/railway-subagents.md + research/spec-syntax-and-railway-scope.md decisions. Testing harness: parseSubagents recovers --plugin-dir subagent nested-traces (namespaced subagent_type + the sub's returned result() block). Evals (real-model): caveman debunk hardened on sonnet (-23%/-20%); the does-our-spec-help A/B validated the spec HELPS (parseable outcome 0%->100%, no quality regression). |
||
|
zernie
|
5c3cd2f947 |
feat!: layout-driven multi-harness validation rules & harness testing (#41)
* docs: add rules-docs-in-sync rule; fix nested blockquote in harness-testing Add a `rules-docs-in-sync` guidance rule to CLAUDE.md so the validation-rule set (RulesConfig keys + built-in vigiles/* rules) and the central rules table in docs/cli.md never drift: adding/removing/renaming a rule must update the type, the table row, and the per-rule doc together. Also fix the doubly-nested blockquote (`> >`) in docs/harness-testing.md that rendered as a confusing quote-inside-a-quote — convert the pasteable prompt to a fenced code block, matching the README convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: surface the lint rule set in the linting guide; break up walls of text The Lint guide (verifying-instruction-files.md) explained the three rule *types* but never showed or linked the actual validation-rule *list* — so a reader following the README's Lint link couldn't find the rules. Add a "What vigiles lint checks" section that summarizes the rule families and links the canonical table in cli.md (no duplication, per rules-docs-in-sync). Also break four dense prose paragraphs in harness-testing.md into bullets / shorter paragraphs (entry points, harness-specific transport, the two-layers explainer, the eval-tool comparison) for mobile scannability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(orphans): never flag harness-loaded instruction files; move rules matrix to lint guide orphan-docs now categorically exempts files the harness loads directly (CLAUDE.md / AGENTS.md / SKILL.md / agents/ / commands/) from the orphan CANDIDATE set — they're load-bearing by name/location, not by being linked, so they're never orphans even if `orphans.include` is broadened to the whole repo. They still count as referencers. Adds tests + a docs/rules/orphan-docs.md. Move the canonical validation-rules matrix out of the CLI reference and into the linting guide (verifying-instruction-files.md), where a reader looking for the rules expects it — grouped by family (spec & integrity, test coverage, reference marking, subagent contracts, hooks & MCP, skill triggers, docs hygiene). cli.md and the README now link to that one matrix. Updates the rules-docs-in-sync rule to name the linting guide as the canonical home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: overhaul harness-testing — task-first guide + extracted API reference The harness-testing guide buried the first runnable example five screens deep, under ~140 lines of architecture (capability-contract theory, composition-root, adapter selection, import taxonomy) and didn't show its TOC until line 118. A first-timer couldn't find "how do I test my hook?". Full split: - Rewrite docs/harness-testing.md as a lean task-first guide (902 → 297 lines): a copy-paste first test in the first screen, a "what do you want to test?" decision table as the primary nav, trigger-rate promoted to a headline section, advanced material (significance / regression / cost / judge) collapsed. - Extract docs/testing-api.md: the full reference — Trace model, predicates, assertions, the check vocabulary, matchers, every measureTriggerRate / runEval option, significance/regression, imports & harness selection. - Move the theory (import-path-is-capability-contract, "two layers or three?") to research/harness-testing.md where the rationale belongs. - Repoint inbound anchor links (harness-testing-claude-code.md) and add the new reference to the docs index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * refactor!: surface-scoped lint rules — rename agent-*→subagent-*, capability-gate, fix dialect drift Adopt the "surface model" for cross-adapter vs adapter-only lint rules: a rule is scoped to the SURFACE it checks (subagents, shell hooks, skills, MCP, docs), not to a harness name. Universal rules stay bare and run everywhere; surface rules run only where the active adapter has that surface and report n/a otherwise (loud, never a failure) — so one config targets Claude Code AND Codex (incl. independent .claude/.codex trees, not just byte-identical mirrors) without per-harness rule duplication. Harness prefixes are reserved for genuinely exclusive concepts (none needed today). BREAKING (no back-compat, per design): - Rename rules: agent-tool-contract → subagent-tool-contract, agent-frontmatter → subagent-frontmatter, untested-agent → untested-subagent. Named by the harness-neutral concept (subagent), resolving the Codex `[agents]` (a concurrency table, not a subagent) name collision at the source. - Add AdapterCapabilities.subagents (CC/OpenCode true, Codex false); the subagent rules gate on it and report n/a where absent. - Fix the lint dialect-drift bug: checkSubagentToolContracts / checkHookEvents / checkFrontmatterSchema now run against the RESOLVED adapter's dialect + layout (and respect --harness), not a hard-coded claudeCodeDialect — scan was already adapter-aware, lint was not. - Docs: add the surface→harness "Applies to" matrix to the linting guide, rename the three rule docs, recompile CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * feat: layout-driven surface classification — ready for new-harness subagent dirs The subagent rules and scan's surface classifiers were hard-coded to `agents/` / `skills/` / `commands/`, so a harness whose subagents live elsewhere (OpenCode's `.opencode/agent`) wouldn't be classified. Make it layout-driven. - PluginLayout gains labeled `skillDir` / `agentDir` / `commandDir` (an empty `agentDir` means "no subagent surface" — e.g. Codex's `[agents]` TOML table). Set per adapter; documented in the adapter API + authoring guide. - scan.ts builds its isSkill/isAgent/isCommand classifiers from the active layout (regex anchored on a real path boundary, path-segment-safe), threaded through every surface helper — no module-level hard-coded dirs. - The `subagent-tool-contract` lint rule now REUSES scanPlugin's already-resolved, layout+dialect-driven agents (one-detector-no-drift) instead of re-globbing a hard-coded `agents/` path; drops the now-dead parseAgentTools/verifyToolContract imports. - Test: scan classifies a subagent under a custom `agentDir` and a default Claude Code scan does not — proving the dir is read from the layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: finish de-CC-ing the lint/scan detectors + add an ESLint guard Two more functional Claude Code hard-codings remained in the deterministic reference-verification path (same class as the subagent glob): - scan.ts `resolveScript` (hook-script-exists) hard-coded `${CLAUDE_PLUGIN_ROOT}`; now expands the harness's own `layout.pluginRootToken` (braced + unbraced). - test-coverage.ts (untested-skill/agent/hook) hard-coded `skills/` / `agents/` / `.claude/` globs and the CC token; now layout-driven via `skillDir`/`agentDir`/ `materializeRoot`/`pluginRootToken`/`manifestPath` (default CC), threaded from the resolved adapter in lint and from `lay` in scan. The deterministic layer-1 (lint + scan) is now fully layout/dialect-driven. The remaining `.claude` literals live only where they belong: the CC adapter, the CC eval transport (src/eval.ts), and CC plugin onboarding (init in src/cli.ts). Prevent recurrence: a `no-restricted-syntax` ESLint rule bans hard-coded CC string literals (`${CLAUDE_PLUGIN_ROOT}`, `.claude/`, `ANTHROPIC_*`) — as plain literals AND template quasis — in the harness-agnostic domain + detectors (src/core, scan.ts, test-coverage.ts, plugin-loader.ts). Verified it blocks a probe. Complements the existing import-graph boundary with a literal boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * test: dogfood the layout-driven untested-surface detector on a non-CC layout A regression guard for the de-CC-ing: build a repo with custom surface dirs (mysurf/skill, mysurf/agent) and plugin-root token ($MY_PLUGIN_ROOT), then assert findUntestedSurfaces discovers all three surfaces via the injected layout — and the default Claude Code layout sees none of it. CC-shaped fixtures can't catch a CC-hardcoding regression; this one can. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix: thread the resolved adapter through ALL lint checks + Codex CLI dogfood Completes the dialect-drift fix: 8 more scanPlugin-based lint checks (skill-frontmatter, mcp-config, mcp-tool-resolves, hook-script-exists, disallowed-tools-contract, description-overlap, frontmatter-valid, mcp-hook-target-resolves) were still calling scanPlugin(process.cwd()) with the default Claude Code layout/dialect — only 3 of 11 had been fixed. All 11 now run against the resolved adapter, and the subagent-/hook-surface ones report n/a on a harness lacking that surface (capability-gated). Dogfood: a Codex `lint` CLI golden test (offline, in the existing CI job) wires a real Codex repo — AGENTS.md + TOML [hooks] with the harness's own ${PLUGIN_ROOT} token + an untested skill — and asserts hook-script-exists resolves the token and flags the missing script, untested-skill finds the skill, and the subagent rule reports n/a. A CC-shaped fixture can't catch a CC-hardcoding regression; this one runs the full pipeline against the second harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: codify the adapter-aware-lint convention + fill the capability/layout doc gaps Add an explicit `adapter-aware-lint-rules` guidance rule to CLAUDE.md (recompiled from the spec): lint rules are surface-scoped not harness-named; name by the harness-neutral concept (subagent, not agent); thread the resolved adapter and never hard-code a CC literal (eslint-enforced); gate by capability and report n/a; dogfood must be non-CC-shaped. The full record of this session's de-CC-ing. Docs caught up to the code: - harnesses.md capability matrix + adapter-api.md AdapterCapabilities gain the `subagents` flag. - adapter-api.md / authoring-an-adapter.md PluginLayout gains skillDir/agentDir/ commandDir; the myHarnessAdapter example gains the (previously missing, required) `capabilities` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * fix(ci): update API report for the new adapter fields; add internal audience-check skill The `check` job's API surface gate failed: AdapterCapabilities.subagents and PluginLayout.skillDir/agentDir/commandDir are public (vigiles/adapter), so the committed etc/vigiles-adapter.api.md was stale. Regenerated (diff = exactly those four field additions) and gave each new layout field its own TSDoc so the report has no `(undocumented)` markers. Also add an internal dev skill `dev/skills/audience-check` (not shipped, not counted by untested-skill): re-read the README through six audience lenses — Claude Code user, Codex user, senior/staff skeptic, junior newcomer, eng-manager decision-maker, QA/test engineer — and report per-persona friction + ranked fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 * docs: note the UI-created-PR title gotcha in the conventional-commits rule A PR opened from the Claude Code UI gets a plain prose title that fails the `validate` job (which checks the PR title, not the commits). Document the fix: edit the title to a Conventional Commit + the matching breaking `!` as soon as the PR exists; validate re-runs on the title edit. (Hit on PR #41.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ct8VY2VCY4h6kDGVNBQQ4 --------- Co-authored-by: Claude <noreply@anthropic.com> |