mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
v23
86 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
94af23a5b7 |
feat(hook,cli)!: close five adopter-reported issues — halted-turn guards, dead-ref artifacts, centralized tests (#186)
Closes #173, #174, #175, #176 and #170. #174 — decideHook read exit 2, `decision` and `permissionDecision` but not the harness's halt-the-turn field, so `assertBlocksDisasters` reported a guard that stopped every disaster as blocking none. The field is read from the PORT, not hard-coded: `continue` is documented for Claude Code and unverified for Codex. The issue's adjacent suggestion was deliberately NOT implemented — adding the halt to `hook-block-ineffective` as a block-attempt signal would fire on hooks that WORK, since a halt is not event-scoped. It reads as a suppressor instead, fixing a real false alarm. A mutation implementing the original suggestion literally is pinned by a test that fails on it. #173 — compile printed its errors, exited 1, and wrote a hash-valid CLAUDE.md anyway, so `lint` went green over refs it had just called dead. Nothing is written on a failed compile now. #176 — eight small defects, including `audit --out` outside the repo appending unreachable `../../..` entries to .gitignore (a documented read-only report editing a tracked file), an unknown command exiting 1 against a documented 2, and STABILITY.md promising the removed `scan` verb. #175 + #176.2 — three missing capabilities, each opt-in and additive: `{surface}` in testGlobs, `--single`, and `rulesDir` so `.claude/rules/**` stops being invisible. #170 — one enumeration of export forms written against the grammar, both naming checks driven off it, and `check-internal-tag.mjs` asking the TypeScript compiler instead of a regex. It immediately found the issue's open case, 13 tagged types the regex had never seen, and a live gap in the ESLint rule. BREAKING CHANGE: `HookRunResult` gains a required `haltsTurn`, so code that CONSTRUCTS one (a test fake) must set it. Reading a result is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
9b89383cad |
docs(skills): document companion files, and stop restating the detector (#180)
A skill is a directory, and the spec recommends splitting once SKILL.md passes 500 lines — so references/, scripts/ and assets/ companions are the normal shape. docs/skills.md did not mention them. The load-bearing decision is what the section stops doing. The first version restated the detector's rules — which cues suppress a reference, which target shapes are skipped, whether a directive verb is needed. Every restatement was a claim about another component's regex, and review found a wrong one on each pass. The section now says what to DO and sends the reader to rules/skill-resource-resolves.md for what is CHECKED, with that page stating its own lists are illustrative and the constants in src/core/skill-resources.ts are the authority. Reversed the file() advice. An earlier revision told authors to reference a companion with file() instead of a markdown link. That is wrong for any skill that ships: renderFragment emits a FileRef as a visible repo-root-relative path, nothing rebases it at packaging, and the installed skill would carry a working link beside a pointer that resolves nowhere. Guidance is the markdown link — skill-relative, travels with the skill, already linted. The missing piece (a compile-verified reference that stays skill-relative and emits nothing) is named as a gap, not worked around. Corrected two stale Status/pending claims: the integrity header no longer precedes the frontmatter (line 1 vs line 10, measured), and a compiled SKILL.md now has one datapoint as an installed skill — recorded with its limit, since the skill pre-existed the conversion and firing was never measured. The exit criterion says the milestone is unmet. Two product defects documented but not fixed here: resolution is not transitive (a link inside a companion is unchecked, and progressive disclosure produces exactly that chain), and reported line numbers are body-relative because scanSkills strips the frontmatter before counting. |
||
|
zernie
|
7144826614 |
fix(audit): cap the score on a confident breakage, split test-harness, retract an unverified claim (#177)
A confident breakage now caps the health score at 89, one point below the healthy band, so a definitively-broken harness can never render a green dot or grade A. Measured: Structure scored 92 with a green dot while carrying a dead tool contract, because a summed score dilutes one real breakage among clean siblings. confidentBreakages() enumerates what qualifies — decidable AND breakage, never the heuristic rings, since capping on those is how a gate gets switched off. The Safety ring stopped repeating one paragraph per unit: past three, ineffective disallowed-tools fences collapse into a single line that still names every skill. Measured 4,500 -> 810 characters. test-harness split into SKILL.md + references/, 5,534 -> 2,633 tokens loaded on activation, measured with a real tokenizer rather than chars/4. Retracted "no other plugin linter has it" on description-overlap: Cisco's skill-scanner ships --check-overlap. Edited the spec, not the artifact. Blast radius measured: this repo unchanged, clean vendored plugins still A. Gates: vitest 3288 passed, lint, prettier, api:check, build — plus the site browser-parity fixture regenerated for the new expected value. |
||
|
zernie
|
681edf7673 |
fix(hook): the subagent-delivery caveat is stale, and now a test says so (#172)
* fix(hook): the subagent-delivery caveat is stale, and now a test says so Claude Code #34692 — a PreToolUse hook not firing for a subagent's tool calls — was closed not-planned and quoted across these docs for months as a standing limit on what a gate can promise. It is fixed. Measured against a stock @anthropic-ai/claude-code@2.1.241 installed from the registry, not the container's binary: a subagent's own Bash reaches the hook and an exit-2 deny stops it, with the parent's identical command in the same run as the control and the marker files on disk as ground truth. The event also carries agent_type, naming which subagent made the call. The claim is about someone else's product, so prose cannot keep it honest — a doc cannot notice that the platform moved. src/subagent-delivery.test.ts pins both directions and goes red on a regression; removing the deny from its hook fails it on its own assertion. What did NOT change is stated at every site that was edited: a model can still route around a tool entirely, so a gate remains a strong default and is never unbypassable. The measurement's scope is stated too — headless only, with interactive sessions unmeasured and depth-2 nesting absent there. Gates run, in order: vitest (3282 passed), lint (0 errors), prettier --check, docs:check, internal:check, api:check, build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: regenerate the committed project-file types for the new test CI's generated-types:committed gate compares a fresh `generate types` run against the committed file, and adding a source file changes that list. Caught by CI rather than locally because I ran the individual commands instead of `npm run check`, which is the aggregate CI actually runs — the same shortcut this repo has already written down as a recurring miss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7c97e4ed7d |
refactor(spec)!: one experimental root per feature (#171)
The prefix answers "is this name stable?" and cannot answer "is the name my stable symbol depends on stable?". That gap shipped: railway(), delegate() and result() carried stable names while being meaningless without the experimental builder, and the lint rule cannot see it — it compares tag and name within one declaration, not the dependency between two. So the subagent vocabulary now hangs off one marked root: result, railway, delegate, pipe, pipeStep, needs, start and andThen become members of the builder. Eight names leave the top level. Two further gains the prefix alone could not reach: collision-prone words stop owning package-level names, and the warning survives destructuring, since the binding site still names the root — unlike a namespace object or an import subpath, which mark the import line. Object.assign rather than a TypeScript namespace: namespaces are an error in this repo's lint config, and one merges only with function declarations while the skill builder is a const. Types stay top-level, for the same reason the prefix rule excludes them. No deprecation window: nothing outside this repository imports these, and a window subtracts nothing. The container-rollback argument for a window does not reach them — they take no part in loading hooks. Found along the way, and worth more than the refactor: two example specs had not compiled since the previous rename, because the examples directory was in no tsconfig at all. Both fixed, and the class is closed — examples now type-check. Gates run, in order: vitest (3281 passed), lint (0 errors), prettier --check, api:check, build, exports:check, internal:check, docs:check, and tsc over the types project now that it includes examples. Claude-Session: https://claude.ai/code/session_017aUJEoaEQ3DxtvoSVWAWqe Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
900dc7e06a |
refactor(spec)!: name things for what they are; retire the vigiles/experimental subpath (#169)
Three exported symbols were named for something other than what they are, and
two mechanisms were doing one job.
RENAMES (old names kept one major, @deprecated, removed one major AFTER this one):
claude() → instructionFile() compiles CLAUDE.md AND AGENTS.md, so a
name from one harness was never right
instructions`` → prose`` builds a prose FRAGMENT; the plural read
as "the instruction file", which is what
instructionFile() builds
agent() → experimental_agent() the shape is not settled, and the tag was
never there (git log -S empty)
SkillSpec.result: → postcondition: `result` already meant the subagent output
contract one screen down; the doc comment
had to spend a line disambiguating them
The alias window is not politeness. Measured: the previous rename shipped with
ZERO overlap (18.1.1 old names only, 19.0.0 new only). A container rollback put
the old node_modules under new sources, PreToolUse stopped loading, and a
PreToolUse that does not load denies EVERY Bash command — including the one that
fixes it. One window per major makes that pair unrepresentable.
SUBPATH RETIRED. vigiles/experimental is gone; its members moved to the door
their feature already uses. Of two mechanisms for one promise the subpath was the
weaker: it marks the import line, out of view by the time anyone reads the call.
Measured on the only user-facing example — with the prefix aliased away at
import, the marker survived at 0 of 5 call sites. So the prefix stays, the advice
to alias it away became an explicit prohibition, and the six define* entry points
are DECLARED prefixed rather than gaining it at re-export.
That de-aliasing is what unblocked moving the naming check into ESLint, and the
ORDER was load-bearing: measured before it, dropping the public/internal
exemption produced 14 findings, all from the re-export indirection. Measured
after, across all of src/ with no exemption: 22 tagged declarations, 0
unprefixed. The rule turns on silent.
EXPORT CLEANUP. vigiles/linting export * → curated, 103 → 79 symbols; its header
claimed to be curated with an `export *` one line below, and claimed the builders
were also at the package root (191 exports there, zero matches). ClaudeTool and
HookEvent deleted — zero references, and both had drifted off the dialect they
claimed to mirror; HookEvent listed 5 events, two of which do not exist, against
the 31 the vendor documents.
DOCS. experimental.md deleted and distributed per feature: the skill half into
skills.md §Status, the hook half into a new compiled-hooks.md §Status, and the
emit half — 111 of its 177 lines, a topic doc wearing a stability page's clothes
— into docs/emit-channel.md, which gains the "which channel when" comparison it
never had. The one cross-cutting sentence went to the README.
TWO GATES WERE GREEN AND WRONG, both found by this work:
- api:check reported "verified for 11 entries — no drift" while reading
dist/experimental.d.ts from a PREVIOUS build (tsc does not clean dist/) for a
source file that no longer existed. Now asserted against package.json
exports, with a mutation proving exit 1 on a phantom entry.
- check-export-prefixes caught MY error: the R3 services first landed on
vigiles/eval, where `paid_` means "this call can bill you" — and
startServices takes a ContainerRuntime and calls no model. Naming it
paid_experimental_startServices would have redefined paid_ as "expensive in
some sense". They went to the package root.
SEVENTEEN review findings landed on this branch, all real. Two mechanical
closures came out of them: src/example-imports.test.ts resolves every relative
import under examples/ instead of matching a string, and the naming rule now
ENUMERATES the three export paths it visits rather than describing its coverage
in the abstract. One attempt is recorded as REFUTED: teaching the doc gate to
flag our own exports as free variables measured 66 findings across 93 blocks, so
that gate's original scoping was right.
BREAKING CHANGE: vigiles/experimental is removed. Its exports are unchanged in
name and behaviour and are now reached from `vigiles` (the emit channel, the R3
disposable-service tier). Every renamed symbol keeps a working @deprecated alias
for this major.
|
||
|
zernie
|
7b8cdecd5e |
feat!: doc-refs is opt-in, because it measured 0 true positives (#167)
* fix(loader): a surface that exists on disk was never read, and the report named it anyway
The loader read the repo-root `skills/` **or** the project-level
`.claude/skills/` — never both — and materialized whichever it picked under the
SAME canonical `<materializeRoot>/<surface>/…` key. Two real files, one key: the
loser was never opened, and the winner's bytes sat under the loser's name.
The vendor settles which one the harness loads: BOTH, in two namespaces.
> Plugin skills use a `plugin-name:skill-name` namespace, so they can't
> conflict with other levels. For example, `my-plugin/skills/deploy/SKILL.md`
> becomes `/my-plugin:deploy` and loads alongside a `deploy` skill in your
> project's `.claude/skills/`.
> — https://code.claude.com/docs/en/skills § "Where skills live"
So "pick one" was never a tie-break to get right; it was a question with no
answer, asked because the key shape forced one. The fix removes the question:
every scope present is read, and a scope's key prefix is DERIVED FROM THE SCOPE,
so two files can no longer claim one key — there is no ordering, no
"if already taken", and no last-write-wins left to get wrong.
Measured (36 SHA-pinned public repos, 2026-08-18):
* 573 → 493 surface files that exist on disk and are never opened.
The 80 recovered are the shadowed ones: nyldn/claude-octopus 71 → 0,
hyhmrright/brooks-lint 8 → 0, thedotmack/claude-mem 1 → 0.
* The 493 that remain are a DIFFERENT, pre-existing gap, unchanged by this
commit: directory symlinks under `skills/`, which `entryOf` deliberately
does not descend (the ELOOP containment) but Claude Code does follow.
* nyldn/claude-octopus ships each skill twice — 61 under `skills/`, 57 under
`.claude/skills/`, 50 names in both, and ALL 50 pairs differ. One pair is
vendored as a true-positive fixture.
* vigiles' own repo: 6 skills → 19, lethal-trifecta units 4 → 12. Its own
audit was blind to two thirds of its own machine.
* Grades moved on 1 of 36: brooks-lint C(70) → B(82). NO finding was deleted
(6 before, 6 after) and the COUNT of dangerous units rose (6 → 7); the
Safety ring is a ratio, and this moves its denominator from 6 to the real
12. See the report for the full justification.
Also fixed, because reading both scopes caused it: a name no longer identifies a
surface, so `description-overlap` would have printed `skills "dup" and "dup"` —
50 times on claude-octopus. Overlap labels now carry the real path.
SUBTRACTS: the `root` | `user` | `none` variants of `SurfaceSource` (the either/or
is gone, so "we picked the wrong one" is unrepresentable), the duplicated
precedence rules in `scan-files.ts` (the decision now lives once, IO-free), and
one net lint warning in the two touched files (6 → 4).
Gates: npx vitest run · npm run lint · npx prettier --check . · npm run api:check
· npm run build — all clean, in that order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
* chore: regenerate the browser-parity fixture for the two-scope disambiguation
The overlap finding now names each skill by PATH, not by bare name — which is
the point of the two-scope work: with root skills/ and .claude/skills both
present, a bare name does not identify a surface. Intentional engine change,
regenerated per the parity test's own instruction.
* chore: regenerate generated.d.ts for the merged tree
Единственный конфликт мёрджа был одной строкой в СГЕНЕРИРОВАННОМ файле — счётчик файлов
проекта, 388 против 387. Разрешено не выбором стороны, а регенерацией: у слитого дерева
счёт 389, то есть ОБЕ стороны были неверны для него. Выбор любой из них дал бы зелёный
мёрдж с неправдой в артефакте.
* fix: four defects found by dogfooding vigiles on a real consumer repo
Все четыре найдены за один день догфуда на `zernie/mine` (43 скилла, 48 харнесс-файлов),
каждый — с замером, а не с рассуждением.
1. `exclude` не доходил до прохода по markdown-рефам
`findDocRefs({ basePath })` вызывался без `ignore`, хотя поле `exclude` документировано
ровно под этот случай («vendored or benchmark fixtures the repo's own lint shouldn't
police»). Замер: 9 битых рефов, 8 из них внутри каталога, явно перечисленного в
`exclude`, и все 9 репортились. Следствие дороже причины: выйти в 0 стало нечем, поэтому
шаг пометили `continue-on-error: true` — а lint, чей код возврата выбрасывают, не гейтит
ничего, и рядом выросли самодельные CI-шаги, чтобы гейтить вместо него. Один непереданный
аргумент — вся эта цепочка. После починки: 9 → 0.
2. Рефы искались ТЕКСТОМ, а не разбором
`/\b(enforce|file|cmd|ref)\(\s*["']…/` — четыре из шести входов классифицировались
неверно (замер 19.08): `ctx.file("OUT")` (метод чужого объекта — живой ложный
срабатыв в реальной репе), `obj.cmd(...)`, `// cmd(...)` в комментарии,
`'cmd("x")'` внутри строки. `\b` спокойно стоит после точки, а у регулярки вообще нет
понятия комментария и строкового литерала. Разбор делает все четыре НЕВЫРАЗИМЫМИ: AST
отдаёт только вызов, у которого callee — голый идентификатор. `typescript` и так
runtime-зависимость пакета — парсер лежал в коробке всё это время.
3. Раннер скриптов был строгой очередью
`files.forEach(... spawnSync ...)` со `stdio: "inherit"`. Замер на 48 харнессах:
270s → 77s (3.5×), результат ПОБАЙТОВО тот же (84 passed, 1 with 0 checks, 7 failed).
Пул по умолчанию выводится из `entry`, а не из нового флага: `test` не передаёт entry и
разъезжается по ядрам (файлы бесплатны и детерминированы по построению — мок-модель, без
API-ключа), `eval` передаёт и остаётся на 1, потому что параллель там — это одновременные
оплачиваемые вызовы модели. Вывод буферизуется на ребёнка: с `inherit` 48 отчётов
шинкуются друг о друга, и это настоящая цена параллели, а не сам цикл.
4. Путь агента в находке брался по ИМЕНИ
`new Map(agents.map(a => [a.name, a.path]))` оставляет последнюю запись, поэтому при
`agents/foo.md` и `.claude/agents/foo.md` (разные неймспейсы у Claude) находка о первом
репортилась против файла второго. Находка lethal-trifecta, указывающая не на тот файл,
хуже отсутствующей: она отправляет читателя проверять невиновный код. До полной
scope-qualified идентичности (правка крупнее, `finding.name` несёт только голое имя)
неоднозначное имя даёт ПУСТОЙ путь вместо произвольного.
Плюс: все 10 джобов в 6 воркфлоу этого репозитория были без `timeout-minutes` — то есть
зависший джоб шёл до шестичасового потолка GitHub. Бонды ~3-4× ЗАМЕРЕННОЙ длительности
собственных прогонов репы, замер процитирован над каждой строкой.
Тесты: два новых на пул, обе половины. ⚠️ Первая версия фикстуры считала маркеры, которые
только СОЗДАВАЛИСЬ, — то есть «сколько стартовало всего», а это достигает N и при строго
последовательном прогоне; тест проходил по причине, не связанной с параллелью. Исправлено
освобождением слота на выходе. Обе мутации (форсировать serial · распараллелить eval)
валят каждая свой ассерт, каждая с доказательством, что патч лёг.
Гейты: `npm run lint` 0 errors · `npx prettier --check .` чисто · `npm run api:check` без
дрейфа · `npm run build` · `npx vitest run` — падает только пре-существующий
`scripts/release-preset.test.ts` (не находит `@semantic-release/commit-analyzer`, объявленный
в package.json, но не установленный в этом окружении; на CI со свежей установкой зелено).
* docs: record what each of the four defects was, in the pages that own them
comparison.md — что СЧИТАЕТСЯ рефом теперь (вызов, у которого callee — голый идентификатор,
а первый аргумент строковый литерал) и почему три формы больше не рефы: метод чужого объекта,
упоминание в комментарии, строка с текстом вызова. Плюс область: проход читает `**/*.md` и
теперь уважает `exclude`, чего не делал — из-за чего репозиторий с вендоренной чужой
разметкой не мог получить чистый lint вообще.
docs/rules/delegation-trifecta.md — новая секция про два агента с одним именем: находка
теперь показывается БЕЗ пути, а не с произвольно выбранным из двух. Явно сказано, почему
это лучше: неверный путь отправляет читателя проверять невиновный файл.
docs/cli.md — почему `test` гоняет несколько скриптов разом, а `eval` по одному, и что
это следует из яруса, а не из флага. С замером 270s → 77s при побайтово одинаковом
результате, и с ценой: вывод буферизуется, живого стрима больше нет.
Все три — то, что было НАЙДЕНО, а не только что изменилось: через месяц полезно именно это.
* fix(docs): mark two illustrative enforce() blocks the old regex could not see
Разбор нашёл на два рефа больше, чем регулярка, и оба оказались СЛОМАННЫМИ — годами,
незаметно: `enforce("rubocop/Custom/NoDirectDbQuery")` и `enforce("pylint/no-direct-db-query")`
в справочных страницах скилла. Это примеры того, как объявить СВОЁ правило, то есть имена
заведомо несуществующие — им место под `<!-- vigiles:ignore -->`.
🔴 Почему регулярка их не видела, и это отдельный класс: она применялась ПОСТРОЧНО
(`for (const {lineNo, text} of blockLines) text.matchAll(CALL_RE)`), поэтому вызов,
разнесённый на несколько строк — а оба именно такие — не совпадал никогда. То есть у прежнего
детектора были не только ложные срабатывания (метод чужого объекта, комментарий, строка), но и
ложные ПРОПУСКИ на самой обычной форматировке. Разбор снимает оба класса разом: у AST нет
понятия «строка файла».
Итог на этом репозитории: 21 реф → 25 (после двух маркеров), все валидны.
* fix(compile): quote frontmatter scalars YAML would otherwise mis-read
`description: ${spec.description}` подставлялся сырым, поэтому описание с двоеточием и
пробелом — «…ловит ошибки, где замер врёт: неаналоги в выборке» — давало frontmatter,
который НЕ является валидным YAML. Замер 19.08 на двух реальных скиллах: сразу после
`vigiles compile` метод `skillContract()` отдаёт `malformed: true` и `declared: []`. Файл
ВЫГЛЯДИТ объявляющим `allowed-tools`, строгий парсер не читает ничего, скилл молча
наследует все инструменты сессии.
🔴 Это ровно тот дефект, который правило `frontmatter-valid` существует чтобы репортить —
то есть благословлённый путь производил то, что продукт ищет. Хуже: он НАКАЗЫВАЛ за
починку. Человек, закавычивший значение руками, получал на следующем линте расхождение
хеша («manually edited after compilation»), а перекомпиляция молча откатывала правку.
Проверка — КРУГОВОЙ ПРОГОН, а не список опасных символов: эмитим, читаем обратно тем же
загрузчиком, которым пользуется линтер, и квотим, только если вернулось не то, что ушло.
Два следствия этого выбора:
· уже безопасное значение эмитится побайтово как раньше — ни один скомпилированный файл не
меняется, ни один интегрити-хеш не двигается (подтверждено: все 1303 core-теста зелены);
· множество «опасных» входов не надо перечислять и поддерживать — решает сам YAML, поэтому
`#`, `[`, `&`, `*`, ведущий пробел, `yes`, `null` покрыты, не будучи названными.
Применено и к субагентам: описание там пишется так же свободно и ломает frontmatter так же.
Тесты — обе половины, и обе несущие (мутации с доказательством, что патч лёг):
· вернуть сырую подстановку → падает «compile quotes a description YAML would mis-read»
· квотить всегда → падает «leaves an already-safe description bare — no hash churn»
Вторая не менее важна первой: избыточное квотирование переписало бы каждый файл в каждой
репе и сдвинуло каждый хеш, что читается как «vigiles переписал мне весь репозиторий».
* feat!: doc-refs is opt-in, because it measured 0 true positives
The markdown code-block ref pass ran unconditionally and could not be turned
off. Measured across two real repositories on 2026-08-19 — 2 582 markdown
files, 52 builder refs — it produced ZERO true positives, and every error it
had ever raised was false: design prose sketching an API that does not exist
yet (`cmd("npm test")` for a package with no such script), and a third-party
CLAUDE.md captured verbatim as benchmark data.
The cause is structural rather than a threshold miss. A fenced block in prose
is a DRAWING of config, and this pass read it as config. No calibration fixes
that, because the block is not wrong — it is not config.
The consumer repo could only reach a passing lint by listing whole directories
in `exclude`, silencing every other check over a third of the tree; measured
afterwards the pass walked 604 files and found 0 refs — inert, still paying for
the walk. Off by default replaces that blunt instrument with an honest default.
BREAKING CHANGE: the markdown code-block ref pass no longer runs by default.
Enable it with {"rules": {"doc-refs": "error"}} — appropriate where markdown
genuinely is the source (a docs site whose fences are copied into live specs).
At "warn" findings print marked ℹ and the exit code is untouched; at "error"
they gate with exit 2 (they previously contributed exit 1, the soft tier,
because the pass fired unasked).
Known gap recorded for whoever raises the default again: the walker globs
without `dot`, so .claude/** — where deployed skills and agents live — has
never been scanned. The rule has never once judged a real instruction file.
* test(doc-refs): the GitHub annotation follows the configured severity
The inline-annotation test drove the pass with no config, which no longer runs
it. Extended to all three tiers rather than just re-enabled: "error" annotates
::error, "warn" annotates ::warning, and absent annotates nothing at all. The
middle case is the one worth asserting — a ::error rendered next to an exit code
of 0 is how an advisory rule gets read as a gate.
* chore: regenerate generated.d.ts, and stop two docs lines claiming doc-refs is on
CI caught what my five local gates could not: `npm run check` runs 18 stages
and I ran 5 of them. The new test file was missing from the generated project-
file union — 17/18, failing on `generated-types:committed`.
Also two sentences in docs/comparison.md still described the pass as
unconditional: the section's opening claim ("vigiles validates … calls inside
fenced TS/JS code blocks") and the Scope paragraph. Both now say opt-in; the
reader hits the opening line four paragraphs before the new note explaining the
default.
* ci(site): stop the browser install from eating the job
The site job was cancelled at its 15-minute bond twice in a row (runs
32300064395 and 32303343213), both times inside `playwright install --with-deps
chromium`, and neither failure was in the diff under review. The log shows the
cause plainly: `--with-deps` shells out to apt, apt hung on
azure.archive.ubuntu.com (`Ign:` on every line, then twelve minutes of silence),
and the orphan reaped at the end is `npm exec playwright install --with-deps`.
Three changes, one per cause:
- Drop `--with-deps`. ubuntu-latest already ships the libraries Chromium needs;
the flag exists for bare containers. Removing it removes the only part of this
step that touches a distro mirror. If a library really is missing, the browser
fails to LAUNCH in the next step within seconds and names the missing .so —
loud, and far better than a silent 15-minute hang.
- Cache ~/.cache/ms-playwright, keyed on the lockfiles. The step's floor was a
full ~150 MB download on every run, cache or no cache, good day or bad.
- Bound the step at 5 minutes. The job bond cannot tell "this step wedged" from
"the suite is slow", so a wedge cost the whole job; now it costs five minutes
and names the step that did it.
Measured baseline for the job: 3m35s.
* ci: run a job only when the commit can reach its inputs
This workflow had no path filter at all, so every PR paid for all five jobs
whatever it touched. Measured on this very PR (16 files across src/ and docs/,
none under site/): the site job built the engine, downloaded a browser and ran
the demo suite against code the diff could not reach — twice, at ~15 minutes
each, because the browser install wedged.
A workflow-level `paths:` cannot express this. It decides whether to run the
workflow AT ALL and says nothing about individual jobs. So the decision moves
into a `changes` job: one API call, no checkout, ~10s, feeding an `if:` on the
jobs whose inputs are disjoint from what changed.
Two flags, each conservative in the direction that costs minutes rather than
correctness:
root false ONLY when every changed file is under site/ — nothing outside
site/ imports anything inside it. Gates test, harness, e2e.
site false ONLY when every changed file is prose or agent config
(docs/, .claude/, root *.md). Gates site.
`check` stays unconditional: it runs types:site and cli-lint, so both halves of
the repo are among its inputs.
The site flag is a measurement, not an assumption. The demo imports the built
engine through three aliases, so any src/ change can reach it and must run it.
What cannot: grepping the three bundles it actually loads for the lint rule table
(DEFAULT_RULES, frontmatter-valid, doc-refs) returns zero in all three, and its
check pages are generated from site/src/checks/checks.ts rather than from docs/.
Tested in src/ci-path-filter.test.ts, which EXTRACTS the patterns from the
workflow rather than restating them — a test that agrees with its own copy of a
rule proves nothing about the rule that runs. A skipped job and a passed job look
identical in the checks list, so both directions are asserted. Three mutations,
each failing on its own assertion with the patch verified in place:
prose arm widened to any *.md → 'a markdown file NESTED in src'
prose arm widened to include src/ → 'runs everything for a normal src change'
empty-list bail flipped to false → 'an empty diff is not a licence to skip'
* test(ci): a job added later cannot silently escape the path gate
The filter itself was tested; the SHAPE of the workflow was not. In the checks
list a skipped job and a passed job are the same tick, so a job added tomorrow
with no `if:` looks exactly like a correctly-gated one that happened to run —
the filter quietly stops meaning anything and nothing goes red.
Exemptions are named with their reason rather than inferred, so granting one is
a visible edit: `changes` is the classifier itself (gating it on itself is a
cycle), `check` runs types:site and cli-lint so both halves of the repo are its
inputs. Plus a vacuity guard — if every job ends up exempt the assertion can no
longer fail, and an empty list would read as health.
Mutation, patch verified in place: stripping the gate off `site` fails with
"job(s) run on every commit with no path gate: site".
Mirrors the same guard added to mine's paper-gates.harness.mjs, where an ungated
19m42s job was what prompted it.
Gates, in order: npx vitest run (8 in this file) · npm run check 18/18.
* Revert "test(ci): a job added later cannot silently escape the path gate"
This reverts commit
|
||
|
zernie
|
6532dbf48d |
fix: five defects found dogfooding vigiles on a real consumer repo, plus CI job bounds (#162)
Five defects found dogfooding vigiles on a real consumer repo (zernie/mine — 43 skills,
48 harness files), each measured rather than reasoned about:
1. `exclude` never reached the markdown-refs pass. Measured: 9 broken refs, 8 of them
inside a directory explicitly listed in `exclude`, all 9 still reported. After: 9 → 0.
The consequence was larger than the cause — with no way to reach exit 0 the consumer
set `continue-on-error: true`, and a lint whose exit code is discarded gates nothing.
2. Refs were found by TEXT, per line. Four of six probe inputs classified wrongly
(`ctx.file("OUT")`, `obj.cmd(...)`, a comment, a string), and because the regex ran
per line, any call split across lines was invisible — which had hidden two genuinely
broken refs in this repo's own docs. Now parsed: a ref is a call whose callee is a
bare identifier with a string-literal first argument. `typescript` was already a
runtime dependency. Net: 21 refs → 25, all valid.
3. The script runner was a strict queue. Measured on 48 harness files: 270s → 77s
(3.5x), byte-identical results. The default follows `entry` rather than a new flag:
`test` fans out (free and deterministic by construction — mock model, no API key),
`eval` stays at 1 (real model quota). `stdio: "inherit"` was the actual blocker.
4. Agent path attribution was keyed by name, so with `agents/foo.md` and
`.claude/agents/foo.md` a lethal-trifecta finding was displayed against the wrong
file. Ambiguous names now yield no path instead of an arbitrary one.
5. `compile` emitted invalid YAML. A description containing `: ` produced frontmatter
where `skillContract()` reports `malformed: true, declared: []` — the blessed path
producing the exact defect `frontmatter-valid` exists to report, and punishing anyone
who fixed it by hand with a hash mismatch. Fixed by round-tripping through the same
loader the linter uses, so already-safe values stay byte-identical and no integrity
hash moves.
Plus: all 10 jobs across 6 workflows were unbounded and now carry `timeout-minutes`
sized to measured durations; docs updated in the pages that own each defect.
Gates: vitest 3186 passed - lint 0 errors - prettier clean - api:check no drift - build.
CI: all 7 jobs green.
|
||
|
zernie
|
04225f9980 |
feat: an errored emit is not an emission, and a contract field can be an enum (#165)
Two locked design items (vigiles/ideas/23), plus the public docs brought level with
what has actually been measured.
1. A CALL THAT ERRORED IS NOT AN EMISSION. Measured on the shipped build: a
permission-denied call carrying a valid payload returned {"kind":"ok"} — the reader
filtered by name and never looked at ToolCall.isError, a field that was on the type
all along. The server never received that call. This is the exact class of lie the
emit channel exists to remove from the fenced rail, reproduced inside the channel.
Errored calls now get their own branch, because all three collapses lie in a
different direction: counting them reports success for a call nobody received;
dropping them silently says "no tool call", sending the reader to the skill's
instructions when the fault is in permissions; lumping them with "called twice"
gives a true signal under a false name when a model retries a denial. A successful
call alongside a denied one is one successful emission — the contract was met.
Four tests, each naming its collapse. Mutation-checked: removing the split fails
three of the four.
2. ENUM, the one extension to OutputFieldType. A readonly tuple of literals declares
a field whose value must be one of them; it renders as a JSON-Schema enum in the
tool definition and as "CUT" | "MERGE" in the fenced rail, so both deliveries show
a worker the same permitted values. The measurement behind it: a verdict field
declared as `string` held three mutually incomparable invented categories over three
runs, with 3/19 = 16% vocabulary compliance. `string` cannot express "one of these".
Nothing relational follows it — no object[], no tuples. On 14 real payloads, 23
scalar values carried every assertion anyone made while 80,981 characters of prose
carried none, so the body stays prose by decision rather than by omission.
3. DOCS. docs/experimental.md gains the outcome table — every way an emission can fail
to satisfy the contract, with the reason text — which was the one thing a reader
could not find. Measurements updated to the breadth study (15/15 on sonnet; a zero
on haiku that was the skill stopping early, not the channel dropping); the serving
question is marked answered; the prefix-removal conditions now say which of the
three is met. Also states plainly what the channel does NOT check: whether the
emitted result is true. Gating on an emitted verdict was measured and rejected —
self-report disagreed with a judged check in 45-75.8% of runs.
Gates: npx vitest run, npm run lint, npx prettier --check ., npm run api:check,
npm run build — via the check script, 18 of 18. The api-surface diff is the intended
widening of OutputFieldType and EmitFieldSchema, reviewed and committed.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
4f0a01b2f9 |
CLI clarity, an execution-tier finding, and the Action's boundary documented (#163)
Five changes from a measured review of the CLI surface and of a consumer's CI. 1. BEHAVIOUR — a positional target that matches nothing now FAILS. Bare discovery finding nothing still exits 0: an empty repo is a legitimate place to stand, but a path or glob you NAMED that matched nothing is stale, wrong, or never reached. Collapsing the two cost a real repo three days of green CI verifying zero files. Jest and Vitest fail on no-tests-found and make you opt in with --passWithNoTests; pytest exits 5. --min=0 is the opt-out here, so no flag is added. 2. HELP — the four checkers (audit/lint/test/eval) all read as "check my stuff". They are now grouped by READ vs RUN, and each line states its own consequence positively. The success criterion was checkable and is met: "NOT a CI step — use vigiles lint in CI" is deleted, because nothing needs defending. A help line that says what a command ISN'T is a description that failed. Per-verb flag detail moves into the per-verb help; audit's four wrapped lines naming nine flags were most of the felt crowding. Verb count is unchanged at 8 — the smallest of every tool measured (vitest 8 verbs and 164 flags, cargo 48, docker 57, git 166), and none of them thinned their surface by removing verbs. 3. DOCBLOCK — src/cli.ts listed four of the eight commands, having quietly stopped growing. Replaced by a pointer to the single source; a new test asserts every verb appears in the RENDERED help exactly once, because a constant is what drifted. 4. AUDIT — new Tested finding: "N surfaces counted as covered by PLACEMENT only — no run on record ever exercised one". Colocated evidence is a claim about the filesystem and the provenance line already admits it; nobody reads a provenance line. Measured in a consumer: 26 colocated and 0 executed, while its CI skipped every one of those harnesses and reported success. The corpus-wide threshold is a stated limit in the code, not a silence. 5. DOCS + SKILL — the Action reference now explains why there is no test input (the Action never runs npm ci; harnesses import the library), and the test-harness skill gained a six-line CI pointer: an Action exists, init wires it, inputs are documented there. It REFERENCES rather than explains, because copied explanations rot — three measured proofs, one of them inside this product. Gates, in order: vitest, lint, prettier, api:check, build — via the check script, 18 of 18. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
ad413b0ac5 |
fix(scan): the fix-scan work, merged onto current main (#161)
* fix(scan): two ways a surface that is there was invisible to audit
A marketplace listing suppressed the audit of the directory it sat in, and
`agents/` was read one level deep though the harness reads it recursively.
Both end the same way: vigiles reported nothing, at exit 0, about surfaces
that were on disk — and silence is this tool's success signal, so neither
failure could be seen from the outside.
MARKETPLACE (`audit`, cli.ts). A `marketplace.json` whose members are all
external expanded to zero targets, and a branch printed "nothing to scan"
and returned BEFORE looking at the directory. But that file is a statement
about OTHER directories: a repo may list external members and also be a
plugin. Measured on nyldn/claude-octopus @57cfb9b0 (3,979 stars): 59 skills,
50 subagents, 40 hooks, 54 commands — 203 surfaces — suppressed, exit 0.
The branch is deleted; the `targets` fallback that already existed does the
work. Proof the suppression is gone: deleting the marketplace.json now
yields the byte-identical report, so the file no longer changes the outcome.
A genuinely curated marketplace reaches the existing `score.empty` branch,
which carries the "clone a member plugin" advice and exits 2 — this repo's
own rule that 2 means "I could not do what you asked". `--json` still emits
`kind:"marketplace"` there, for exactly the case it describes.
AGENTS RECURSION (core/layout.ts + its three readers). The vendor documents
`agents/` as scanned recursively, with the subfolder forming the identifier
(`agents/review/security.md` -> `review:security`). vigiles matched
`agents/[^/]+\.md` — in THREE places that each spelled the rule themselves:
the scan classifier and both coverage discoverers. Measured on
rsmdt/the-startup @88d447c7: 16 agent files, 2 read, and the plugin was
still graded B (80/100) over that 12.5%. The rule now has one home,
AGENT_FILE_LEAF_RE, quoted by all three, so the subagent count and the
untested count can no longer be computed over different sets of files.
Agent names become the documented scoped identifier. That is not cosmetic:
a basename cannot be unique once a dir is read recursively, and the
delegation graph keys agents BY NAME, so `agents/a/review.md` and
`agents/b/review.md` would have swallowed each other. Top-level agents keep
their bare filename, which is why no existing report moves.
Two defects the recursion itself introduced, both closed: a `SKILL.md` under
`agents/` matched both classifiers and was counted twice (the harness loads
it as an agent, so it is excluded from skills — the mirror of the existing
skills/<x>/agents/ rule); and the on-disk discoverer normalized separators
for the pattern but not for the name, which on Windows would have silently
fallen back to a basename.
Folded in: a nonexistent path now says it does not exist rather than "no
surface was found there" — one existsSync, its own test.
GRADES MOVE, in the harsher direction, on 2 of the 32 dogfood repos (the
other 30 are byte-identical):
nyldn/claude-octopus suppressed, no grade -> D (60/100), 203 surfaces
rsmdt/the-startup member `team`: B (80) -> D (60), 2 -> 16 agents,
and 2 -> 14 malformed-frontmatter findings
No exit code moved anywhere in the corpus. bench/leaderboard/RESULTS.md is
unaffected: both its pinned repos have zero nested agents (measured).
Tests: 10 new cases across scan/parity/layout/e2e, each proven by mutation
with the patch asserted to have landed. The parity fixture is load-bearing —
reverting only the browser-side discoverer survives the entire pre-existing
suite and dies only on the new nested-agent parity case.
Gates: vitest 2991 passed - lint 0 errors - prettier clean - api:check no
drift - build ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
The same two bytes as on main. Both sides of a merge must be text before git
will do a three-way merge at all — normalising only one side leaves the blob
pair binary.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
4acfd31fc9 |
feat(eval)!: an eval file describes, the runner performs — importing one no longer spends (#159)
* fix(audit): coverage caveats printed by lint never reached the audit report `formatUntestedReport` grew two qualifier lines — a test file still carrying the retired `vigiles:covers` marker, and a run record older than the text it measured. `audit` assembles its own fact block from the SAME UntestedReport and carried neither, so the note that exists to explain a silent migration was itself silent for anyone whose habit is `audit` rather than `lint`. Measured before the fix on a fixture whose only harness carries the marker: lint → "1 test file(s) still carry the retired `vigiles:covers` marker …" audit → "Untested surfaces: 0" and nothing else The caveats now come from one exported builder (`coverageCaveats`) that both renderers read, so a third qualifier reaches both callers or neither — a renderer can no longer carry half the list. `ScanReport.coverageCaveats` is absent rather than `[]` when empty, because the browser twin structurally cannot compute these (no `.vigiles/coverage.json` without a filesystem) and byte-parity between the two engines is asserted field-for-field. Both halves tested at the product level (`node dist/cli.js audit`): it fires on a repo carrying the marker and stays silent on one that does not. Proven by two mutations, each with the patched source grepped first: dropping the render line fails the "fires" assertion; emitting a caveat unconditionally fails the "silent" one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the suffix that stopped counting now says so `*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run collects exactly those names and a skill test spawns a model — so a misnamed harness was being executed, and paid for, by somebody else's runner. The removal was announced to nobody. Its own comment argues "the migration is a rename, and the untested finding prints the exact path", but the path printed is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore told to write the test they wrote, with neither the file nor the reason named. The sibling half of the same migration — the retired `vigiles:covers` marker — got its explanatory line at the time. This is the half that did not. Scoped to surfaces that are otherwise UNCOVERED: that is the only position where the count contradicts what the author can see in the directory. A stray `*.test.*` beside a properly covered surface is somebody's ordinary unit test and none of our business — asserted, because an advisory line that fires on the innocent case gets muted within a day. Nothing about coverage changes: the name still grants nothing. Measured silent on all seven real corpora available here (this repo, six vendored dogfood plugins) and on the `mine` knowledge base; it fires on a planted fixture in both `lint` and `audit`, the latter through the shared caveat list added in the previous commit rather than a second wiring. Three mutations, each with the patched source grepped before the run and each dying on its own assertion: dropping the note from the caveat list · widening the scope from untested to every considered surface · dropping the suffix test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * refactor(scan): compute the caveat list once Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was called twice in the same object literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the retired-suffix note claimed a cost the guard already closes The note said a `*.test.*` harness "spawns a model" under someone else's runner. `refuseUnderForeignRunner` guards every spawn door and refuses under vitest/jest — measured here against the detector: argv of `node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model: this process is running under vitest." What actually happens is that the foreign run FAILS on the file, which is the true and smaller claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it Importing a `*.eval.*` file spent real money. These files did their work in the module body, and in ESM `import` IS execution — so the cheapest imaginable question, "does this file even parse?", launched a paid run against a live model when it was asked with `import()`. That happened, and was paid for. The one guard that exists does not close the door and cannot be made to. Under `node -e 'import(f)'` there is no `process.argv[1]` at all, so `foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays silent; and no process fact distinguishes that from a legitimate runner doing the same import, so an argv-shaped guard would have to refuse the correct invocation to catch the wrong one — the exact failure that got `process.env.VITEST` rejected in `core/foreign-runner.ts`. So the shape changed instead. An eval file default-exports a description: export default defineEval({ measureTriggerRate: { …spec… }, assert: (report) => assertTriggerRate(report, { min: 0.8 }), }); `defineEval` builds a plain value; there is no call in it to run. The five paid runners are reached only by `dist/eval-entry.js`, the program `vigiles eval` now spawns per file — not a new verb, the interpreter the existing verb uses. `defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and that is the point: after this change no eval file imports a runner at all. Three loud edges, because a silent no-op would be the same defect in a new place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares nothing (or two measurements) is reported by the runner instead of exiting green; and a half-migrated file — a description plus a leftover top-level `measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and throws a migration message instead of billing. `node --check` still works and is what the free syntax check should be. Subtractions: every eval file loses its `process.env.VIGILES_TRIALS || process.argv[2]` line, its `console.log(format…Report(report))`, its `if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval` import. The runner does all five, once. BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})` instead of calling a runner at the top level, and `node <file>.eval.mjs` no longer runs an eval. To migrate: move the runner call into a key named after that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` · `measureSelectionMatrix` — declare exactly one), move anything you did with the report into `assert(report)`, and delete the trials line, the `format…` print and the `report.n === 0` check — `vigiles eval` does those. A capability probe that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s directory argument becomes a `pluginDir` field; and `measureTriggerRate(spec, { evalDriver })`'s second argument becomes a top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials. Run with `vigiles eval <file>`; a file left on the old shape fails with a message naming docs/harness-testing.md, never silently. All 19 eval files in this repository were migrated in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture and generated types The eval-definition change rewrote one report hint (`paid_measureTriggerRate` → `export default defineEval({ measureTriggerRate: … })`), which is exactly the divergence the parity test is built to catch. Regenerated per that test's own instruction for an intentional engine change; the diff is that one line — grade and every other field are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7afa00f00a |
fix(refs): 17 false "file is missing" accusations, none of them true (#157)
* fix(scan): the reference extractor accused files that were present
Three false-accusation defects found by a 32-repo OSS dogfood sweep, all the
same shape — vigiles telling a maintainer a file they have is missing. For a
tool whose pitch is deterministic precision, a false positive costs more than a
miss: it is the failure mode that gets the tool uninstalled.
Measured on the corpus (91 audit units): 74 reference findings before, 57
after. All 17 removed are verified false; zero true positives lost, zero new
findings anywhere. Eight units change grade, every one upward — rsmdt/the-startup
F(54)→C(70), ayghri/i-have-adhd B(85)→A(100), fcakyon fable-advisor B(82)→A(90),
jjanczur/tyran D(64)→C(72), and Truthfulness 47→100 / 25→100 on two Microsoft
plugins.
One extractor was being run over five grammars. It now reads three, each through
that grammar's parser, and deliberately stops reading the other two:
markdown markdownRefs() in core/markdown.ts — a link's DESTINATION, and code
spans that are NOT inside link text. The old line-scanner reported a
link's LABEL as a missing resource while its destination resolved.
shell commandWords() in core/bash-effects.ts — the words a shell would
resolve, with inline program text (`node -e`, `python -c`) removed
and a nested `sh -c` parsed rather than discarded. The old regex cut
a character run out of a JavaScript payload and reported it as the
hook's script name.
source core/source-refs.ts — the one place a path-token pattern is built,
with both boundaries un-omittable. `js` could claim the head of
`json`; `(?:agents)/` could claim the tail of `claude-agents/`. Both
were live.
NOT READ: frontmatter (already stripped) and full-line comments in every
language, not only shell — both remaining corpus false positives here came out
of JSDoc.
Subtracted: four hand-written extension alternations (two of them duplicated
across the disk detector and its browser twin), a hand-rolled markdown link
regex, a hand-rolled inline-code regex, a whitespace split standing in for shell
words, and `stripShellComments`.
No new CLI verb, no new flag, no public API change (`api:check`: no drift).
Tests: 10 mutations, each proven applied by grep before running and each killed
by its own assertion, across the whole relevant suite. Every new case carries
both halves — fires on the planted defect, silent on the clean shape beside it.
* fix(scan): drop the redundant start-of-content guard in isPluginRooted
`startsAtSeparator` already answers idx <= 0 with true, and the next line
reaches it for idx === 0 (content[-1] is undefined, so never "/"). The guard
was a second copy of a boundary rule this module deliberately does not own —
the same duplication the module comment warns about — and it was the one
statement the 100% gate could not reach.
Also regenerates .vigiles/generated.d.ts for the new source files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
1ebef8320a |
ci: one list of commands instead of two — CI now calls npm run check (#151)
* ci: один список команд вместо двух — CI зовёт `npm run check` Сегодня три PR уехали в `main` красными: список гейтов гонялся по памяти (пять команд), а джоб `check` гонял тринадцать. Упавшая — `tsc -p test/types/tsconfig.json` — живёт в ОТДЕЛЬНОМ tsconfig, которого корневой тайпчек не видит, и три `@ts-expect-error`-ассерта тихо перестали срабатывать. Длинный список, который надо помнить, это не чинит. Чинит то, что список СТАЛ ОДИН и CI его ВЫЗЫВАЕТ: `scripts/check.mjs` + `run: npm run check` вместо тринадцати шагов. Это стандартный паттерн поля (pre-commit / just / mise): определение задач — источник правды, CI его дёргает. Прежняя моя идея «завести гейт на совпадение двух списков» снята: она была подпоркой под дубль, который правильнее убрать. Стадии несущие, а не косметика. Часть команд ПИШЕТ файлы, которые другие ЧИТАЮТ (`generate types` → `.vigiles/generated.d.ts`, `api-extractor` → `api-surface/*`, `build` → `dist/`), поэтому писатели сериализованы, а параллельно идёт только read-only группа. 17/17 за 70 с против ~3 мин последовательно. Имена падающих шагов не потеряны — раннер печатает, что именно упало, и `--list` показывает команды без запуска. 🔴 Два шага `uses: ./` НЕ входят в `npm run check` намеренно: они догфудят собственный composite Action, у которого нет shell-эквивалента. Один из них (`eval-check`) при первой правке был снесён случайно — он стоял между двумя command-шагами, и правка по диапазону его проглотила. Возвращён, причина записана прямо над ним. Раннер сразу поймал два собственных промаха: неотформатированный `check.mjs` и протухший `generated.d.ts` после добавления скрипта `check`. Проверено: `npm run check` 17/17 ✓ · ci.yml парсится. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: словарь скилла уехал на сам билдер — experimental_skill.input/.step BREAKING: `input` и `step` больше не самостоятельные экспорты. Достаются как `experimental_skill.input(…)` / `.step(…)`. Пользователей нет. Зачем. Префикс `experimental_` — привычка, а привычка уже один раз протекла: `skill()` отгрузился со стабильным именем при доке «`skill()` is experimental». На самих `input`/`step` тега и префикса не было вообще, то есть словарь экспериментальной фичи выглядел стабильным. Повесив их на билдер, маркировку делаем СТРУКТУРНОЙ: до `input()` не добраться, не назвав `experimental_skill`. Забрано ровно два символа, и это замер, а не вкус. Перепроверено независимо от вердикта субагента: `input` и `step` встречаются только в скилл-спеках, а `cmd` (5 агентских спек), `file`, `ref`, `result` (8 агентских) общие — их спрятать в неймспейс скилла нельзя. 🔴 По дороге закрыта асимметрия, которую правка иначе бы УХУДШИЛА: дверь `vigiles/claude-code` своих `input`/`step` не экспортировала никогда, и типизированный автор брал их из второй двери. После снятия standalone- экспортов он остался бы без них совсем. Теперь CC-билдер несёт те же хелперы. Форма — `Object.assign`, не `export namespace`: последний запрещён нашим же линтом (`no-namespace: error`). Генерик пережил: в api-репорте `typeof skillSpec & { input; step }`, `ae-forgotten-export` нет. Честная граница записана и в коде, и в доке: `const { input } = experimental_skill` снимает маркер внутри файла. Гарантия у́же — немаркированное имя не пересекает границу пакета. Доки: `docs/skills.md` — пример на алиасе (`experimental_skill as skill`), маркер держится на строке импорта. `docs/experimental.md` — вступление утверждало, что префикс носит всё из `vigiles/experimental`, что после переезда `experimental_skill` в `vigiles/spec` стало неверным; плюс своя секция. `docs/README.md` — доиндексированы `testing-matrix.md` и `non-js-harnesses.md`. `safety.md` намеренно НЕ индексирован: он сам пишет «INTERNAL / WIP — not surfaced in the public docs yet». Гейты: npm run check 17/17 ✓ · vitest 2954 ✓. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b9302db4e6 |
feat!: one instability marker — the experimental_ name prefix, enforced (#147)
* docs: генераторный режим убран из публичной доки и припаркован `skill()` помечен экспериментальным. Из `docs/skills.md` убраны §3 (генератор) и секция тестов `vigiles/skill-test`, которая на нём стоит: `genSkill`/`act`/`checkpoint`/`finish`/`runSkill` не достижимы ни из одного сабпаса пакета — то есть снаружи репозитория такой скилл не написать. Ровно эти два фантомных сабпаса и заставляли держать в шапке рукописный баннер; баннер снят вместе с ними. Замерено при этом, и записано в §Status как причины экспериментального статуса: - `vigiles compile` генераторы ПОДДЕРЖИВАЕТ (`cli.ts` ловит `genSkill(`, прогнано end-to-end) — прежняя строчка §Status «wiring it into `vigiles compile` … is the next integration step» устарела; - скомпилированный `SKILL.md` ни разу не был показан загружающимся как скилл: `vigiles:sha256:`-шапка отодвигает YAML-фронтматтер с первой строки, и читатель, привязанный к `^---`, не находит фронтматтера вовсе. Все работающие `SKILL.md`, включая 13 своих, начинаются с `---`; шапку несут только два файла в `examples/`, которые никто не грузит; - принятие всё-или-ничего: `renderSkillSections` собирает документ целиком в фиксированном порядке, пути «моя проза + один проверяемый гейт» нет. Паркинг помечен в шапках трёх модулей, чтобы его не подняли молча. Публично экспортировать нечего: замер показал ноль достижимых символов. `scriptModel` НЕ паркуется — он публичен из `vigiles/claude-code` независимо от генератора. Заодно — дефект гейта из #146, найденный этой же правкой: он переиспользовал маркер `vigiles:ignore-file`, который принадлежит резолверу ссылок. Из-за этого единственный документ с двумя доказуемо битыми импортами был единственным, которого гейт не видел (91 блок сканировался, этот — нет). У гейта теперь свой маркер `vigiles:ignore-imports`; на дореформенном тексте он выдаёт ровно те два TS2307, ради которых писался баннер, на текущем корпусе молчит (92 блока, 0 находок). Тест на разделение маркеров прогнан мутацией: возврат общего маркера роняет оба ассерта. Гейты: vitest 2928 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · check-doc-imports 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: skill() → experimental_skill(), и правило, которое это держит BREAKING: `skill()` переименован в `experimental_skill()` на `vigiles/spec`, `vigiles`, `vigiles/linting` и `vigiles/claude-code`. Пользователей нет, мажор бампаем. Почему: дока открывалась словами «`skill()` is experimental», а имя на экспорте говорило «стабильный». Проза против механики — ровно тот дефект, который vigiles и ищет, у vigiles. Соглашение о префиксе существовало (`experimental_emitTool`, `experimental_parseEmitted`, `experimental_withServices`), нигде не было записано и ничем не проверялось. Побочно рейм снимает КОЛЛИЗИЮ, которой не должно было быть: `skill()` экспортировался из ВОСЬМИ сабпасов и был ДВУМЯ разными функциями — билдером спеки (`vigiles/spec`) и `Check<Trace>` по трассе (`vigiles/testing`, принимает id-строку, отвечает «сработал ли скилл»). Одно слово, два понятия, различимые только дверью импорта. Префикс несёт только билдер; `Check` не тронут (`vigiles-testing.api.md` без изменений). Правило: `scripts/check-experimental-naming.mjs` + шаг CI + `npm run experimental:check`. Публичное объявление с TSDoc-тегом `@experimental` обязано начинаться с `experimental_`. Ни одного нового маркера: тег уже используется в `experimental-emit.ts` и `services.ts`, публичность берётся из закоммиченных `api-surface/*.api.md`, которые `api:check` и так держит честными. Типы вне области: на дореформенной поверхности префикс несли только функции, а `ServiceSpec`/`ServiceHandle`/`ContainerRuntime` — нет; с типами гейт открылся бы 6 косметическими переименованиями против 1 настоящей находки и был бы заглушен в тот же день. Настоящая находка одна и закрыта: `makeDockerRuntime` → `experimental_makeDockerRuntime`. 8 тестов гейта, обе половины у каждого. Один нашёл баг в самом гейте: `\s+\S` в маркере-исключении удовлетворялся переводом строки и звёздочкой продолжения JSDoc, то есть голый маркер без причины проходил. Поправка к предыдущему коммиту: §Status утверждал, что sha256-шапка мешает загрузке скилла. Замер это ОПРОВЕРГАЕТ для соседней поверхности — в `reviewer-ab.eval.mjs` живой Claude Code загрузил скомпилированный `agents/code-reviewer.md` С шапкой через `--plugin-dir` и отдиспатчил в него, 100% прогонов на реальном sonnet (2026-06-20). Формулировка сужена до «непроверено», а не «сломано». Гейты: vitest 2936 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал устаревший импорт в docs/railway-subagents.md — правка внесена) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * docs: STABILITY.md объявлял стабильным то, что переименовано и помечено экспериментальным Найдено субагентом при разборе неймспейс-идеи, подтверждено мной. `STABILITY.md` — документ под заголовком «What's stable — depend on it» — перечислял `skill` среди стабильных билдеров `vigiles/spec`. После переименования такого символа нет вообще, а то, во что он переименован, в предыдущем коммите объявлено экспериментальным. То есть контракт стабильности обещал стабильность несуществующему имени. Третий экземпляр «проза против механики» в этой же области за два дня, и мой гейт его НЕ ловит: он сверяет TSDoc-тег с именем, а не документ с API. Заодно записано то, что вскрылось при проверке: - `input()` и `step()` используются ТОЛЬКО скилл-спеками, но экспортируются без префикса и без тега — то есть словарь экспериментальной фичи выглядит стабильным. Слепая зона гейта: он энфорсит «тег ⇒ имя», но не «на всём, что должно, стоит тег». - Маркеров нестабильности в проекте ТРИ и правила выбора между ними нет: `@internal` (39 объявлений), `@experimental` (18), префикс `experimental_` (8 публичных символов). Секция больше не утверждает, что маркер один. - У `pipe` тег `@internal` стоит только на ПЕРВОЙ перегрузке, поэтому остальные в api-репорте читаются как `@public (undocumented)`. Гейты: prettier ✓ · docs:check 0 · experimental:check 0 · lint 0 errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: один маркер нестабильности вместо трёх — префикс имени BREAKING: `pipe` `pipeStep` `needs` `start` `andThen` `effect` → `experimental_*` на `vigiles` и `vigiles/linting`. Пользователей нет. ## Выбрано **Экспортируем и не стабильно — имя начинается с `experimental_`.** Тег `@experimental` объявляет это один раз, `experimental:check` роняет CI, если у помеченного публичного символа нет префикса — тег и имя разъехаться не могут. `@internal` перестал означать «нестабильно»: он отвечает на ДРУГОЙ вопрос — входит ли символ в API вообще — и верен только на том, что мы не экспортируем. Почему имя, а не тег: тег не виден там, где решают. Имя видно в каждом вызове, дифе, ревью и автокомплите. Ровно так протёк `skill()` — стабильное имя при доке «`skill()` is experimental». ## Исправлено всё Было три маркера без правила выбора: `@internal` 39 объявлений, `@experimental` 18, префикс 8 публичных символов. Найдено 6 противоречий — символы с `@internal`, которые ЭКСПОРТИРУЮТСЯ; все шесть переименованы, теги переведены на `@experimental`. ## Два дефекта в гейте, отгруженном час назад 1. 🔴 Тег читался только если стоял на строке ОДИН (`@experimental\s*$`). Реальные теги несут прозу — `@internal Experimental typed-composition surface — …` у `pipe`, два в `services.ts`. Замер: **2 из 8** `@experimental` и **31 из 39** `@internal` объявлений были гейту не видны. Строгий паттерн выглядел безопаснее и был наоборот. 2. Между JSDoc и объявлением может стоять комментарий (`eslint-disable` перед перегрузками `pipe`) — привязка к следующему символу его теряла. После починки гейт видит 14 объявлений вместо 6. ## Записано `STABILITY.md` — секция «The one rule» + список; `CLAUDE.md.spec.ts` → перекомпилированный `CLAUDE.md` (правило №51). Плюс честно записанная дыра: гейт энфорсит «помечено ⇒ названо», но НЕ «всё, что должно быть помечено, помечено» — живой пример `input()`/`step()`, словарь только скилл-спеков, без тега и префикса. ## Третий раз за сессию Фикстура-спека в шаблонной строке (`generate-harness.test.ts:505`) не видна ни tsc, ни грепу по импорту — ловится только прогоном тестов. До этого то же в `cli.test.ts` (дважды). 5 новых тестов гейта (13 всего), включая закрепление обоих дефектов выше. Гейты: vitest 2941 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал `docs/cli.md:646`) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
863c1b42ee |
docs: два примера импортируют символ не из того сабпаса — оба падают на копипасте (#145)
Найдено ресёрчем по инструментам doctest и подтверждено вызовом, а не чтением: scriptModel: vigiles/testing → undefined, vigiles/claude-code → function runHookProgram: vigiles/testing → undefined, vigiles/hook → function Оба примера читатель копирует целиком, и оба падают на первой строке. Это не опечатка в прозе, а неработающий код в документации. 🔴 Про `scriptModel` это уже ТРЕТИЙ раз. Тот же неверный импорт дважды записан в потребительской базе знаний (`vigiles/s59.md` §3 и починен в скилле по s61 §4) — с пометкой, что граница намеренная: `scriptModel` живёт в адаптере Claude Code, а не в харнесс-агностичном барреле. Правило было известно, задокументировано, и третий экземпляр всё равно уехал в публичные доки. Ровно тот класс, который проза не удержит: граница верная, документация о ней верная, а конкретная строка импорта — нет. Механическая проверка предложена в `vigiles/ideas/19` (в приватной базе); этот коммит закрывает два известных экземпляра, а не класс. Гейты: npx prettier --check docs/ · npm run lint (0 errors). Правок кода нет. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
a2af365682 |
docs: четыре утверждения о собственной публичной поверхности были ложными (#144)
Все четыре проверены командой, а не чтением, и каждое обещало читателю то, чего пакет не делает. 1. ШЕСТЬ SUBPATH, КОТОРЫЕ ПАКЕТ РЕКЛАМИРУЕТ САМ СЕБЕ И НЕ ОБСЛУЖИВАЕТ. `vigiles/eval`, `vigiles/run-hook`, `vigiles/harness-test`, `vigiles/harness-assert`, `vigiles/skill`, `vigiles/skill-test` — каждый даёт ERR_PACKAGE_PATH_NOT_EXPORTED. Они стояли в комментариях одиннадцати файлов examples/, в докстринге `harness-assert.ts` как готовый пример импорта, и в `e2e.ts`, который сам объясняет депрекейт и по дороге называет несуществующий путь. Заменены на существующие. 2. `src/testing.ts` УТВЕРЖДАЛ, ЧТО ЭТИ ПУТИ «still work». Обещание, что импорт продолжает работать, — ровно то, что читатель не станет перепроверять. 3. `vigiles/unit` ОБЕЩАЕТ «no `claude`, no model, no bubblewrap, no network». Из четырёх верны три: `judged` re-экспортируется отсюда, а его судья по умолчанию — реальный вызов модели (`check.ts`: `opts.judge ?? runJudge(o)`). Тест, импортирующий ТОЛЬКО из этого барреля, может тратить деньги, и единственная строка, по которой читатель заключил бы обратное, — эта. Честная починка — увезти `judged` с барреля, но это правка публичной поверхности, а поверхность сейчас пересматривается отдельно (замеренное предложение режет по СТОИМОСТИ, и там дефект становится невыразимым, а не задокументированным). До тех пор утверждение исправлено, а не оставлено. 4. `docs/skills.md` УЧИТ ИМПОРТИРОВАТЬ ТО, ЧЕГО НЕТ НИГДЕ. `genSkill` / `act` / `checkpoint` / `finish` / `runSkill` не экспортируются НИ ИЗ ОДНОГО subpath — проверено перебором всех. Генераторный API компилируется из исходника командой `vigiles compile`, и точки входа ему не завели. Форма скилла в документе описана верно; строки `import` — намерение, а не рабочий код, и теперь документ говорит это сам. Плюс `site/index.html`: alt-текст og-карточки называл показатель «Truthful refs», которого у продукта нет (`audit-score.ts`: Truthfulness · Triggering · Structure · Safety · Tested). Инструмент, который ловит расхождение прозы с механикой, показывал в собственном превью несуществующее имя своего же показателя. Гейты, поимённо и в порядке: npx vitest run (2926 passed, 173 файла) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
10e528e8db |
refactor(testing)!: изъять runMutations — она не про домен харнессов (#143)
* docs(testing): runMutations was in the reference only — the guide never mentioned it Measured after shipping it, not guessed: `runMutations` appeared in exactly ONE file, `testing-api.md`. Zero mentions in `harness-testing.md` (the task-first guide), `testing-matrix.md` and `docs/README.md`. The reference says of itself "the guide has the task-first how-to; this is the reference" — so anyone following the guide would never learn the API exists. And the runner table still opened with "Four runners" while listing five. That is the same defect the API was built to catch, produced by its own docs: prose and content disagreeing, with nothing to notice. - harness-testing.md — a routing row in "What do you want to test?" plus a full section, including the two things the reference did not say at all: WHEN NOT TO reach for it (a one-off script, an already-red test, or wanting a coverage number — mutation counts here say how many cases you wrote, not how good the suite is), and CI, where the honest answer is usually "don't": a run forks a process per case over full files, so it belongs in a hand-run pass after editing the test or the checker, or in its own job if wired at all. - testing-api.md — "Four runners" now says four for the harness plus a fifth that asks about your tests, which is what the table actually shows. - testing-matrix.md — a row for src/mutations.test.ts, naming the properties it covers, so the coverage map matches the suite. - docs/README.md — an entry under the API reference pointing at both. Gates: prettier · lint 0 errors · the 10 mutation tests still pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * revert(testing): withdraw runMutations — it does not belong in this package Shipped in #142 and 15.3.0; withdrawn a day later on the owner's question: «how is this vigiles' responsibility, it's just nodejs?» He is right, and the measurement is what settles it rather than taste. Against the rest of this package: runHook knows the Claude Code hook protocol, measureTriggerRate knows what a skill is, skillContract reads allowed-tools out of frontmatter. This module knows none of that. It replaces a substring, spawns `node`, and reads stdout — lift it into any Node repository and it works unchanged. A newcomer reading the testing docs hits exactly the owner's question, which is the cost this surface was charging. The confirming measurement, and the one I should have taken before merging: NO TEST IN THIS REPOSITORY USED IT. The only references were its own test file, its export line and its docs. The package did not dogfood the API it published. What actually fixed the problem it was extracted for was CONSOLIDATION, not RELOCATION. Ten hand-written copies of one driver had drifted apart in a consumer repository — the no-op guard in 4 of 10, the retry in 1 of 10. One shared copy buys all of that. Moving that copy into a product for harness authors was a second and separate step, and nothing justified it. The engine survives where its callers are, with the reasoning recorded in its header so it is not re-proposed. The shape that WOULD belong here is a different function that does not exist yet: mutate the HOOK — remove one branch of a guard — and require the assembled harness test to notice. Only vigiles knows what a hook is. Removed: src/mutations.ts, src/mutations.test.ts, the export block in src/testing.ts, the api-surface entry, and the docs sections in testing-api.md (from #142) plus the guide, matrix and README entries this branch had added. Gates, in order: npx vitest run (2916 passed, 172 files) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: перезапуск проверки заголовка PR Тип `revert` не входит в список, который принимает линтер заголовков (feat/fix/docs/chore/refactor/test/perf/ci/build). Заголовок переименован в `refactor(testing)!` — восклицательный знак несёт то, что `revert` нёс именем: изъятие опубликованного API ломающее. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b5da1c5604 |
feat(testing): runMutations — prove a test can fail, by breaking what it watches (#142)
A green test says the checker passed. It does not say the test would notice if the check were DELETED: an assertion can be vacuous, a fixture can be wrong in the same direction as the code, and both look exactly like a pass. `runMutations` plants a defect, runs the test that owns it, and requires that test to go red WITH THE MESSAGE THAT NAMES the defect. ## Why this belongs in the package It arrived as ten hand-written copies of one driver in a dogfooding repo (1799 lines), and the copies had drifted. Counted against the committed versions: - the NO-OP guard (a replacement equal to the original leaves a green test that proves nothing) was in 4 of 10; - the RETRY on a non-kill (a flake reported as a survivor sends the next reader to rewrite a working assertion) was in 1 of 10; - the strict rule — killed by its OWN named assertion, not merely by something going red — was in 1 of 10. Each was written AFTER it caught something, in whichever copy caught it, and never travelled. Two copies also named a test file a refactor had deleted, and the runner they used exits 0 on a path matching nothing, so those cases reported SURVIVED on every run for three days. ## Five verdicts, one of which is proof killed · wrong-assertion (a neighbour caught it) · survived · unjudgeable (the test was already red before the run, so "red" says nothing) · not-applied (find matched 0 or many, or the replacement equalled the original). The baseline matters more than it looks: without it, a test that is red ON PURPOSE — an open finding it exists to report — makes every run announce a failed restore about a working one, and makes every case against it look killed. ## What refuses rather than reports An empty case list and a named test file that does not exist both throw BEFORE any file is touched. A runner with nothing to run reports success, and that is the one claim this API exists to prevent. ## Verification vitest 2924 passed · lint 0 errors · prettier · api:check (additive only, +53 lines to vigiles-testing) · build. The new file has 10 tests, each building a real checker and a real test in a temp dir — a mocked runner would be judging a mock. Mutation-tested itself: removing the strict-kill rule, the no-op guard, or the existence check each turns the suite red on its own assertion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cce1936460 |
docs: экспериментальная поверхность и именованная память (#140)
* wip: branch anchor
* feat(experimental): experimental_emitTool/parseEmitted/assertEmittedOk — a typed result carried by a TOOL CALL
An unforked skill has no call→return boundary, so compile hard-errors
output-without-fork on 31 of our 38 skills. A tool call needs no return
boundary: the skill EMITS mid-conversation and the call lands in
Trace.toolCalls. Same OutputContract, different delivery.
Marked experimental in the API itself — experimental_ prefix on every
runtime export plus a JSDoc block listing, by number, what is unproven
(N=3, no compile-time consumer, transport untested, runtime does not
enforce required) and what would have to be true to drop the prefix.
No new CLI verbs or flags.
shapeError is exported from the CC agent-result parser so both rails
validate a contract with the SAME code — a second copy would drift.
* examples(experimental-emit): the runnable measurement — MCP server + runner
emit-server.mjs serves ONE tool, read from JSON that experimental_emitTool
produced; hand-copying the schema would test the copy, not the API.
run-emit.mjs takes a real skill dir, refuses one with context: fork, swaps
exactly the '## Record the verdict' section for the emit instruction, runs
it through the real CLI and reads the result back from ctx.toolCalls.
Records are appended per trial, not buffered to the end. VIGILES_EMIT_DRY
rehearses the swap for $0. Not named *.eval.mjs so a CI sweep can never
reach a paid run.
* measure: 8 runs of an UNFORKED skill, 8 emits, 8/8 parse — plus a $3 defect in isRateLimited
The measurement. paper-status (no context: fork, so it can carry no output:
contract today) emitted a typed verdict by calling the tool, and the call
lands in Trace.toolCalls next to the Skill activation that spliced the skill
in inline. 8 runs, 8 emits, all ok, none repeated; scored offline for $0 by
score-emits.mjs through the same experimental_parseEmitted a test would use.
Two independent witnesses (MCP server log + CLI transcript) agree 8/8 on
track, count, verdict and report hash.
The run also exposed a real defect: the CLI emits an informational
{"type":"rate_limit_event",...,"status":"allowed"} line on EVERY run, and
isRateLimited's /rate.?limit/i matched it because . matches _. So every eval
trial silently ran retries+1 = 4 times, and maxCostUsd only ever saw the LAST
attempt: 2 trials became 8 model runs and a $0.60 cap was crossed at roughly
$3. Separator is now [ -]? plus an explicit rate_limit_error clause; the
telemetry line is captured verbatim as a fixture and asserted both ways.
Records are redacted (the long free-text report bodies are replaced by
head+length+sha256) and the raw files are gitignored: a run replays a skill
from a private corpus into a public repo.
* measure: run the TODAY-assertion against the same 8 outputs — green 8/8 while its number is NaN 8/8
The write-up asserted that the prose regex would break on real output. Asserting
that is the failure mode the standing rules name, so it is executed instead:
/BLOCKED/ matches 8 of 8 and /(\d+)\s+blockers?/ matches 0 of 8 — the assertion is
green and the count it exists to guard is NaN every time. Only the derived booleans
are published; the model prose describes a private corpus.
(Pushed via the GitHub API: git-over-HTTPS is failing in this container because
/root/.ccr/agent-proxy-ca.crt was truncated to 2 bytes mid-session.)
* docs: раздел про record()/state() + публичная страница экспериментальной поверхности
Обе фичи отгружены без единой строки в docs/. record()/state() приехали в #135,
experimental_emitTool — на этой ветке; ни то, ни другое из публичных доков не читалось.
docs/experimental.md намеренно отдельным файлом, а не абзацем в основных гайдах:
недоказанное не должно стоять в одном ряду с semver-стабильным. На странице —
что померено (N=8, 8/8), что НЕ померено (одна модель, один скилл, нет compile-time
пути, транспорт не проверен), чего оно НЕ покупает (схему рантайм не энфорсит,
Bash не уходит — allowed-tools это пред-одобрение), что должно стать правдой,
чтобы снять префикс, и стабильная альтернатива.
Прогнано: prettier --check docs/ · npm run lint · vitest по experimental-emit ·
vigiles lint (21 ссылка в код-блоках, все валидны).
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
f80a6333f6 |
feat(hook): writeTargets(prefixes) — API перестаёт выбрасывать, какой файл пишется (#138)
`commandView` already builds the list of write targets and collapses it to a boolean on the way out, so every consumer that needed a FILENAME rebuilt it from the raw command string. The shipped paper guard carries a 69-line character scan with its own quote tracking for exactly that, and that scan has three documented silent bypasses (a quoted path, an absolute path, a path eaten by a wrapper). - `writeTargets(prefixes)` returns the matched targets; `prefixes` is required and there is no unfiltered form, because a consumer holding the raw list has to re-implement the prefix matching that produced every one of those bypasses. - `writesTo` becomes `writeTargets(prefixes).length > 0` — one code path, so the boolean and the list cannot drift the way `runs()` and `writesTo` did. - `NormalizedLeaf.chdir`: `stripWrappers` read the `env -C` / `sudo -D` value in order to skip past it and then discarded it, so `env -C migratsiya sed -i papers/x.tex` reported NO write under `migratsiya/papers` — measured false before this change. A leaf's own chdir wrapper now resolves. A redirection is deliberately NOT joined onto it: the shell opens `> out.txt` in its own directory before `env` ever runs. Out of scope and pinned by a characterization assertion: cross-statement cwd (`cd x && …`, `pushd`, subshells). Connectors do not survive leaf extraction and `cd x; cmd` writes into the OLD directory when the `cd` fails — a model with failure semantics, not a field on a leaf. Both halves tested for every new behaviour, and both mutations verified to land before running: returning the unfiltered list fails 6 tests; restoring `writesTo` as its own extraction fails 2, including the dedicated derivation test. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cfa2fc04af |
fix(hook-runtime): resolve hook paths against a project root, so file and command gates stop failing on the spelling the harness actually sends (#134)
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since
|
||
|
zernie
|
e782c8b0bf |
fix(coverage,hook-runtime)!: answer "is this tested?" by execution, and stop the paid tier being reachable by accident (#133)
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since
|
||
|
zernie
|
ee204a99c8 |
fix(audit): validate a hook matcher as a pattern, not as a literal tool name (#132)
The rule shape-checked a matcher as though it had to look like mcp__server__tool, so it reported working regexes as never firing and accepted patterns that fire on nothing. A user obeying it replaced a working matcher with a dead one — the failure the rule exists to prevent, reached by complying with it. Semantics were measured against the real CLI rather than inferred, and the reproduction ships as tools/measure-hook-matcher-semantics.mjs: a matcher with no metacharacter is compared by string equality, one with metacharacters is an unanchored regex. Both contested rows were confirmed live against an MCP server connected under two namings. A literal keeps the shape check. A pattern is compiled and tested against probes covering the server namings that actually occur, including an underscore inside the server segment and a hyphenated uuid, plus segments derived positionally from the matcher itself so a deliberately scoped mcp__memory__.* stays clean. Firing on some names but not others is now its own finding that names the missed probes and never says never fires. An uncompilable regex is its own finding rather than silently valid. The suggestion can no longer recurse: a recovered server segment must be name-shaped, and a property test asserts every suggestion satisfies the rule that produced it. Fixes #131 |
||
|
zernie
|
a87d0cf2bc |
fix(audit,testing): three dogfood defects — a heading, a salvage, and a check count (#130)
A bundled script named in a markdown heading was invisible to skill-resource-resolves, because the use-directive gate reads prose and a heading is not prose. An ATX heading now counts; the widening stops there, measured at 1 finding before and after over 28 SKILL.md files. Malformed frontmatter was regex-salvaged into the tool contract the lethal-trifecta and Safety ring read, so a unit whose allowed-tools a strict loader rejects scored as the narrow contract its author meant. A salvage may now only make the verdict worse, never better — the vendored corpus forced that nuance by going red when the first pass refused salvages outright. vigiles test reported a clean exit with zero assertions as a pass. vigiles/testing now counts checks per tier and the runner reports a fourth, non-fatal state. Silence stays a pass: undefined means nobody counted, 0 means counted and found nothing. No new CLI verb or flag. Every mechanism is deterministic. Direction is strictly more conservative: nothing can now pass that could not before. |
||
|
zernie
|
41169e379a |
fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129)
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state `Tested` collapsed deterministic harnesses and real-model evals into one number, and its finding read "N surfaces with no vigiles test/eval" — one sentence for two prescriptions three orders of magnitude apart in cost. A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on every push, and asks whether a gate still catches what it claims. An eval (`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one question a deterministic read cannot: does this skill fire at all? Folded together, a repo with complete deterministic coverage and no evals scored identically to a repo with neither. - test-coverage: one discovery pass, partitioned by suffix into a harness tier and an eval tier; the union `covered`/`untested` fields are unchanged. Mirrored in the browser-safe twin. - scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and additive; a producer predating the split falls back to the union). - audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both advisory. `Evaluated` has THREE states — a number, a measured `0`, and `not measured` when the executing checks were skipped and no eval exists. Scoring 0 there would report the absence of a check as its result, which is the pattern this tool exists to name elsewhere. In that state the ring names the command that would measure it. - cli: threads the existing execute-skip signal into the ring. - report/HTML/JSON: both rings render; the inventory carries both counts. Grade weighting is unchanged — both rings are advisory and neither enters `reportDeductions`. Audited on this repo before and after: A (100/100) both times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * docs(site): the audit grades six categories, not five Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * fix(lint): a clean union must not read as 'firing verified' Every surface can have a deterministic harness while nothing has ever measured that a skill fires. The gate is unchanged (still the union), but the ✓ now says so instead of implying the question was asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * test(site): regenerate the browser-parity fixture for the Evaluated ring The audit now emits a sixth category, so the Node-generated expected report and the in-browser one differ until the fixture is regenerated — which is what the test's own header prescribes for an intentional engine change. Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the Evaluated ring and its two inventory counts and nothing else: grade and overall are untouched, which is the check that the split did not move the weighting. * fix(coverage): `Tested` measured a STRING, not a test DEFECT. A surface counted as covered when its path or namespace appeared as a SUBSTRING anywhere in any discovered test file (`content-reference`, in src/test-coverage.ts). Two halves, and the second was worse. REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an existing harness file: // probe: skills/argument-arc moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the line put it back. So `Tested` was trivially gameable and the gaming was indistinguishable from real work. The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely asserts over all 21 of its pipeline skills, but builds each path at runtime (`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING. Generality was PENALISED: 21 trivial files each naming their own subject would have scored better than the one harness that actually tests all 21. This is the exact substitution vigiles names in other people's repos — presence of a surface taken for presence of the property, the way `noExplicitAny` in a config is taken for the rule being enforced. The tool was committing it in its own scoring. FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the disk detector and its in-browser twin cannot drift on what decides a grade): 1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring the existing `vigiles:ignore-test` marker convention on the other side. A runtime-path harness can now say what it covers; an accidental mention cannot. Declaration beats inference — the same move as declaring `allowed-tools` instead of inferring capability. 2. PROVENANCE. Every coverage decision now carries HOW it was reached — declared / colocated / mention — and the counts are printed by `formatUntestedReport`, by `formatScanReport`, and carried in `AuditReport.inventory.coverageEvidence` (additive; schema version unchanged). A repo whose coverage rests entirely on substring matches can finally see that about itself; "28 covered" and "28 names that happen to appear in a file" used to print identically. 3. The bare substring detector is KEPT, deliberately, and said so in the code: it is the zero-config path that makes an ordinary `foo.test.ts` naming `skills/foo` count without anyone learning a marker, and dropping it would flip honestly-covered surfaces to "untested" overnight. What it no longer reads is COMMENTS. A path in a comment is prose about a test; a path in executable code is the test reaching for the surface. Prose isn't policy, including ours. It is also now visibly the weakest tier in the report, and when it is the ONLY evidence the report says so in words. Grade weighting is untouched. TESTS prove both halves, each watched failing first: with the fix reverted, "a surface named ONLY in a comment is NOT covered", "a runtime-path harness that DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail (3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE still counts, and a `"https://..."` string literal survives comment-stripping. Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously recommended "add the skill's path in a comment", which was advice to exploit the defect. * fix(site): key the browser-test prebundle to dist, so a stale cache can't lie DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako divergence by comparing an in-browser AuditReport against one generated in Node. `site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on `@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite invalidates on config/lockfile changes, not on the mtimes of an aliased path outside node_modules. So the browser side can compute against a PREVIOUS engine while Node computes against the current one. REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately). Accidentally: an engine change added a report field; the parity test failed showing the field present on the Node side and absent in the browser. I read that aloud as a genuine engine divergence and started hunting a pako bug that does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass. Deliberately, to prove the fix: plant an extra `inventory` field in src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the browser suite with a warm cache. - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1` on one side only — a field-level diff that reads exactly like a real divergence and says nothing about the cache. - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` -> `.vite-engine-a52a27e0137e`. Reverting the plant returns the fixture to its committed bytes. FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/` (`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`), sweeping the directories keyed to other fingerprints so they don't pile up. WHY a key and not the alternatives. Comparing the cache's mtime against `dist/` and forcing a re-bundle, or clearing `.vite` in the test script, both work and both leave the bad state REACHABLE — a comparison someone can get wrong, or a script step bypassed by running `vitest` directly. Keying the directory removes the state instead: a different `dist/` is a different path, so "prebundle built from another engine" cannot be reached, and an unchanged `dist/` still gets a warm cache. By construction rather than by check — the argument this project makes everywhere else about removing a capability rather than policing its use. The test's header instruction ("pako and node:zlib disagree — do NOT paper over it") was correct for one of the two states and actively misleading for the other; it now says which state is no longer reachable and why, and the assertion carries the same note in its failure message. * fix(site): make the parity-fixture generator emit prettier-clean JSON DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with `JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it keeps short arrays exploded across three lines where prettier collapses them. CI gates on `prettier --check .`, so every regeneration of the fixture committed a file that reddened CI later and far from the cause. It just did: branch head |
||
|
zernie
|
86fdb5f141 |
fix(audit): read MCP servers from the layout, not a hard-coded path (#127)
Three defects, plus a cheap bet on the Agent Plugins packaging format. - marketplace.json had drifted a full major behind plugin.json; a CI gate now fails on any future name/version/description/license drift between the manifests. - Both collectMcpServers implementations hard-coded ".mcp.json", bypassing layout.mcpConfigFile and violating adapter-aware-lint-rules. Behaviour is unchanged for both shipping harnesses; the literal is gone. - The report could say "MCP servers: no" while listing an MCP finding. A plugin packaged to the Agent Plugins standard puts its MCP servers in a root mcp.json that no harness layout names, so three MCP rules silently checked nothing on such a plugin. src/core/agent-plugins.ts recognizes the standard by the manifest's $schema, never by filename. Not modelled as a HarnessAdapter: the standard is a packaging format with no tool catalog, hook events or runtime. vigiles also ships as a conformant plugin. Docs are explicit that this is additive — VS Code reads the standard's root manifest; Claude Code and Codex document their own manifest paths. |
||
|
zernie
|
94e9180218 |
fix(hook-runtime): let the repair through so a stale stamp can't wedge the repo (#122)
`verifyStampOrRefuse` makes a compiled hook refuse to run when its source no
longer matches the stamp — correct for tampering. But when that hook is a
PreToolUse Bash gate wired into the same repo, editing the hook source makes
it block EVERY Bash command, including `vigiles compile`, which is the only
command that regenerates the stamp. The repo wedges: you cannot recompile
because the stale hook refuses to let you.
Observed 2026-08-03. The only escape was hand-editing `.claude/settings.json`
to unwire the gate, compile, then rewire.
Fix: exempt the repair action, loudly. `isStampRepairEvent(event, hookFile)`
is true for
- a Bash command invoking `vigiles compile` (AST-backed, so `npx …`,
`pnpm exec …`, `./node_modules/.bin/vigiles …` and a compound command all
count), or
- an edit/write whose target IS the hook's own source file, so a FILE gate
over the repo can also be fixed.
Everything else still fails CLOSED. The same escape covers a hook that no
longer LOADS (a typo mid-edit), which is the same wedge reached a different
way — otherwise the deadlock stays reachable from a normal edit-compile cycle,
which is the property that has to hold.
Why this over failing open or a --force flag: failing open silently disarms
every gate on any edit, and a flag is one more thing to know at exactly the
moment you're stuck. This keeps fail-closed as the default and names the one
door out, in the error message itself.
It also does not weaken what the stamp actually buys. While the stamp is
stale the hook enforces NOTHING (it refuses every call), and anyone able to
rewrite a hook's source can equally rewrite `.claude/settings.json` — the
escape they would otherwise use. The guarantee is that a smuggled capability
can never run SILENTLY; that is unchanged, and the pass-through prints three
lines of stderr saying exactly what it did.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
7b98ae35c4 |
feat(testing): export loadHook + add the react assertions (#123)
The compiled-hook testing story had a hole at each end.
1. NO PUBLIC LOADER. `assertHookDenies(hook, event)` / `assertHookAllows` take
the hook OBJECT, but a `.harness.mjs` test only has its PATH, and the loader
that bridges the two (`loadHookProgram`) lived privately inside `cli.ts`. So
the intended in-process test path was unreachable from the file format
`vigiles test` actually runs, and authors fell back to spawning the runtime
as a subprocess — precisely the plumbing compiled hooks exist to remove.
`loadHook(path)` is now exported from `vigiles/testing` and `vigiles/unit`,
and the CLI runtime calls the SAME function, so a hook that loads in a test
loads identically in production (one loader, no drift). It keeps the
actionable errors: a `.ts` hook on a runtime without type stripping says to
use tsx / Node >= 23.6, and a module with no default export says
`use \`export default defineHook({…})\``.
2. NO ASSERTION FOR A REACT HOOK. A react can't block, so the gate assertions
don't apply, and nothing else did either. That left `notice()` a live trap:
its message goes to STDERR, so a probe built on `execFileSync` (stdout only)
sees nothing and reports a healthy react hook as DEAD — measured against
three real hooks on 2026-08-03.
`assertHookNotices(hook, event, matcher?)` and `assertHookSilent(hook, event)`
read the reaction in-process, so the stream never enters into it. Both name
what they got instead on failure, including when aimed at the wrong role.
Also ships the example the finding asked for:
`examples/harness/compiled-hook-inprocess.harness.mjs` loads a gate and a react
hook BY PATH and asserts over both. It runs under `vigiles test` (so CI proves
the documented pattern works, not just that it parses), and the docs in
compiled-hooks.md / testing-api.md now show this path, with the stderr trap
called out explicitly.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
246be8e9c7 |
feat(hook): add CommandView.writesTo() and capture redirections (#120)
`leafCommandsNormalized("echo x > papers/x/paper.md")` returned
[{head:"echo", argv:["echo","x"], args:["x"], flags:{}, assigns:{}}]
The redirection and its target were DROPPED by the parser — they live on the
enclosing `Stmt`, and the walk only visited `CallExpr` nodes. So the file the
command actually WROTE appeared in no field of any leaf, and `cmd > file` /
`cmd >> file` — the single most common write shape — was invisible to any
compiled gate. A gate written to protect a directory from Bash writes was
silently weaker than the `grep` it replaced.
The only available discriminator was `isSideEffecting()`, which classifies the
WHOLE command line. So `grep -c x notes/S.md 2>/dev/null` classified
side-effecting, `touches` matched, and a plain READ got blocked. Measured
while dogfooding on 2026-08-03: it blocked real work twice within an hour.
Two changes:
1. `NormalizedLeaf.redirects: readonly LeafRedirect[]` — `{op, target, fd,
writes}` per redirection, target quote-unwrapped and $HOME-canonicalized
like every other word (dynamic → null, present but unresolved, never
guessed). `leafCommandsNormalized` now iterates `Stmt`s; every `CallExpr`
in the mvdan AST is a `Stmt.Cmd`, so the same leaves are found.
2. `CommandView.writesTo(prefixes)` — answers the WRITE question precisely:
redirect targets with a write op, plus the argv positions of known
file-writing programs (`sed -i`, `cp`/`mv`/`install` destination, `tee`,
`dd of=`, `truncate`, `shred`). An unlisted head contributes no target, so
it never guesses.
`touches` is unchanged — it still answers "mentioned" — and both doc comments
now say so explicitly, since conflating them is exactly the trap.
Quoting is handled by the PARSER, not a regex, so a write quoted inside
another command does not match: in `echo 'echo y > a/paper.md' > /tmp/note.txt`
the only real target is `/tmp/note.txt`. Pinned by a test.
The mvdan redirect-op codes have no public name mapping, so a test re-derives
the whole `op -> string` + write-classification table from the installed
parser: an upgrade that renumbers a token fails loudly instead of silently
reclassifying a write as a read.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
5f7531e5ce |
fix(audit): grade an inherited lethal trifecta like an explicit one (#119)
* fix(audit): grade an inherited lethal trifecta like an explicit one
The Safety score penalized the remediation it recommends.
A unit whose contract NAMES all three trifecta legs produced a graded -10.
A unit that INHERITS all tools — strictly worse, it holds those three legs
implicitly plus every other capability — produced an ungraded advisory. So
declaring an `allowed-tools` contract, a genuine risk reduction, could only
ever LOWER the score.
Measured while dogfooding a real 35-skill repo (2026-08-03):
before: Safety 70 = 100 - 3x10, counting only the 3 units that already
had explicit contracts; the 31 units inheriting everything
contributed ZERO
after: Safety 0 after adding `allowed-tools` to all 35, which took the
units holding the full trifecta from 35/35 down to 17/35
The tool reported the safer configuration as strictly worse, and was silent
on the unsafe one. Two model defects, not a threshold to tune:
1. NON-MONOTONE — only the explicit case counted.
2. SATURATING — with a flat per-unit weight, 35 and 17 exposed units both
blow past the clamp and score 0, so halving exposure shows no change.
Fix: one shared `trifectaExposure()` counts EVERY unit holding all three
legs, whatever the severity, and charges
`min(W_TRIFECTA x exposed, W_TRIFECTA_MAX x exposed/assessable)` — per-unit
cost capped against the share of the model-invocable surface exposed. Both
the Safety ring and the shared `reportDeductions` read that one number, so
the ring and the headline still agree.
The same two measured states now read Safety 70 -> 85.
Existing calibration anchors are preserved: a single explicit all-three
subagent is still -10, and the `feature-dev` 3-of-3 shape is still -30 -> C.
What changes is that a harness whose whole surface inherits everything now
takes the same capped -30 instead of a free pass.
Deliberately NOT changed: `formatScanReport`'s "structural issue(s)" tally
still counts only `hard` findings. That line is about structural defects,
not the safety axis; promoting every idiomatic inherits-all unit to "broken"
in the CLI summary is a louder, separate call.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(site): the demo sample repo now grades F (2 of 3 units hold the trifecta)
The DemoAudit browser test hard-coded the sample map's grade. With the
inherits-all trifecta graded, the sample repo — 2 of its 3 model-invocable
units inherit all tools — drops D(61) to F(41), matching the regenerated
parity fixture.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(site): the Playwright demo assertion follows the sample repo to F
The in-frame report e2e hard-coded the grade badge alongside the browser
test fixed in the previous commit.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
7be092ce19 |
refactor(linters): LinterAdapter port + JVM/Go doc & test parity (#116)
LinterAdapter registry (Record<BuiltinLinter, LinterAdapter>) makes a missing linter a tsc error; generate-types iterates the registry; conformance test set-matches registry vs BUILTIN_LINTERS + docs + site chip list. CI installs detekt/ktlint/checkstyle/golangci-lint so their gated tests run (no silent skips). Docs/site catalogs->linters parity. add-a-linter contributor skill. Report-view teaser cards lightened + linter footnote de-staled. Stage 4 (dispatch-map collapse) tracked as a follow-up. |
||
|
zernie
|
3017f70ef5 |
feat: JVM/Go linters + audit/init/compile hardening + adoption UX (#114)
Closes #107, #109, #110, #111, #113. Vlad's 5 issues (skill allowed-tools round-trip, detekt/ktlint/checkstyle/golangci-lint catalogs, 2 scan false-positives, non-JS harness guide, testGlobs docs), a 14-bug dogfood batch, adoption/demo UX, and prevention refactors (one markdown-it fence oracle replacing 5 regex toggles; one js-yaml detekt-config parser replacing 3). LinterAdapter port added to the roadmap as P0. |
||
|
zernie
|
f91617ac27 |
fix(site): verb-map alignment + gate the site e2e (incl. mobile overflow) in CI (#105)
Inline the verb-map answer (sm:col-start-3 on the model note fixes the grid cell collision that bumped it to a middle-indented second row). Wire the site Playwright e2e into the CI site job + a 390px mobile-overflow gate. Add commands-and-how-they-relate.md (audit/lint/test/eval/init model; init-vs-audit for measuring skills) and link it. Roadmap: backend with rate-limited LLM for the full audit demo; interim demo-scope caveat. landing-site skill gains the responsive-grid cell-collision lesson. |
||
|
zernie
|
a4c07e8791 |
feat(site): cohesive report overhaul + README front-door slim (#102)
Reveal every finding (with the concrete broken path via an additive brokenReferences field), fix the "structurally clean" verdict below 100, fold the model-gated tease in with an explicit copy button, explain the verbs, add live chip stars, and slim the README into a front door that links to vigiles.sh. Regenerated the browser-parity fixture for the new field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
zernie
|
8f214f2738 |
feat: mark Codex trigger-rate experimental (Codex-only) + redesign landing hero (#79)
Codex has no skill-selection event, so codexSkillFired infers firing from a SKILL.md read (cache → false negative, exploratory read → false positive) — so the trigger-rate is surfaced as EXPERIMENTAL (Codex-only; Claude Code stays supported): an experimental caveat on the driver → report, a stderr warning, a loud formatter line. Deterministic vigiles audit stays full Codex parity. Documented in docs/harness-testing-codex.md + research/harness-capabilities.md. Also ships the landing hero redesign (site/, deploys to vigiles.sh, not in the npm package): the merged audit widget (deeplink + npx fallback), sticky CTA, the audit report on the fold, shared lib/deeplink.ts. |
||
|
zernie
|
bf6fc3a0a7 |
feat: landing site + audit calibration + combined Pages deploy (#76)
Landing site (site/, Vite+React+shadcn) served via a combined GitHub Pages deploy (landing at /, API docs at /api; replaces api-docs.yml + site.yml), with the vigiles.sh CNAME baked into the artifact. Audit calibration: dialect-drift reconciles the freshness alarm with the running claude; skill-resource-resolves stops flagging illustrative prose paths; lethal-trifecta graded at half weight instead of fail-to-F. Plus research/feature-index.md and a screenshot dev skill. |
||
|
zernie
|
fd5000af21 |
feat: adoption direction v2 — rule-enforcer rename, Ruff+Clippy routing, segmenter precision (#75)
Rename compiler/ -> @vigiles/rule-enforcer; add Ruff (Python) + Clippy (Rust) routing; segmenter precision (colon-leadin + determiner-description rejects, 818->789); rule-enforceability.md + breadth benchmark + 5 enforcement homes + ranked enforcement-tier backlog; verify-docs-findable skill; adoption direction locked (audit-first). |
||
|
zernie
|
88b68020be |
refactor(audit): decompose the rule-map pipeline + alpha/experimental pass + docs
Behavior-preserving refactor of the audit rule-map pipeline: segmentInstructions (CC 108) and extractMarkedRules (CC 39) decomposed into pure per-block helpers; the rescue ladder → a module-level RESCUE_SOURCES; the tier split → a pure partitionCandidates; a new src/rule-signals.ts unifies the deontic/imperative lexicon (FORM_HEAD/RULE_PREDICATE/NORM_SIGNAL); LANE_META is the single glyph+label source. segment.ts + rule-routing.ts went ~27 lint warnings → 0; Fable differential-fuzz verified zero output divergence over 3.6k comparisons. Mark the rule map ALPHA/experimental: the HTML report section is demoted below the deterministic sections + badged experimental, the CLI header is tagged [experimental], and the public + internal docs frame it as a preview. research/rule-compiler-design.md gains §8 (scope-freeze + backlog — tuning is infinite, default answer is no unless corpus-breadth or a measured win) and §9 (the OSS-e2e/LLM-in-CI answer: the map is model-free and already CI-dogfooded on real OSS; only synthesis/behavioral tiers are model-gated → on-sub + manual). Re-rendered the README hero screenshot to match the new layout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
zernie
|
d1f4886a06 |
feat: full ESLint/Pylint parity in the audit rule map (catalog + Python synthesis)
Pylint dynamic catalog + enabled-state (section-aware --list-msgs / --list-msgs-enabled; symbol or numeric code); mergeCatalogs unions ESLint+Pylint with collision handling at the routing lookup (conservative bare-id merge, unique numeric code keeps its own hit). Python custom-rule synthesis in @vigiles/compiler via an astgrep-py engine (ast-grep JSON objects, one gate + injected executors). Two-tier detection (confident / possible / skipped-with-reason). Real-OSS dogfood run in CI (routing + the real pylint binary). Audit DX: auto-gitignore report artifacts (POSIX-normalized), --out, --no-open; consent resolved before routing so a first interactive audit enriches its own run. Design-of-record in research/rule-compiler-design.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
zernie
|
bf9815f0b4 |
feat(audit)!: prose→lint-rule routing map (ESLint + Pylint) + dogfood CI-enforcement + repo-structure pass
Audit rule-routing map (ESLint full + Pylint basics; dynamic catalog, S0/S1 markers, construct/docstring patterns) with the four lanes sharpened (enforceable / hook / custom-rule / judgment-call) and the pr-to-lint-rule gated synthesis skill. Dogfood CI-enforced + indexed (compiler gate asserts verdicts; bench corpus guards; research/dogfood-corpus.md + a vendoring-policy rule; per-adapter asymmetry documented). Repo-structure cleanup: dev/ -> .claude/skills, schemas/ parked under src/, fixtures/ -> test/fixtures, etc/ -> api-surface/, scripts/tools split, root CLAUDE.md Layout map. BREAKING CHANGE: the audit --json schema bumps schemaVersion 1 -> 2 (the mechanism enum for an unrouted rule changed "compile" -> "synthesize"). |
||
|
zernie
|
993bfd37a3 |
feat!: make orphan-docs opt-in, scoped to docs/ by default (#70)
- Gate the scan behind an `orphans` block in .vigilesrc.json (presence = opt-in); the CLI prints "not enabled" otherwise and never touches the exit code. Default `include` is docs/ only (dropped research/, which is vigiles-specific). Driven by an OSS sweep: ~100% false positives on nav-managed doc sites, genuine rot ~0. - Reclassify the rule-meta bucket to heuristic-behavioral (the metric is decidable but the "rot" claim is a proxy → warn is the permanent ceiling). - Derive harness surface exemptions from the injected PluginLayout instead of hard-coded CLAUDE.md/agents/ literals in core, anchored to real surface roots so a docs subdir sharing a surface name is still scanned. - Docs: rule-doc rewrite, rules matrix row, spec-format section, a doc-hygiene row in what-vigiles-catches.md, and CLAUDE.md. BREAKING CHANGE: orphan-docs no longer runs by default. A repo that relied on the unconditional scan must declare an `orphans` block in .vigilesrc.json to keep it, and the default scope no longer includes research/. |
||
|
zernie
|
5ba3516957 |
feat: audit rule-compile tier + verdict-led report revamp (#69)
* research: spec for the audit rule-compile tier (design of record)
Locks the design after two Fable red-team passes + founder refinements. The
audit opt-in tier compiles a repo's CLAUDE.md/AGENTS.md prose rules via the
sibling agent-rules-compiler engine (compile-rules skill + gate.js runGate),
runs the gate-kept ones, and surfaces real violations + per-rule copy-command
buttons. Two surfaces: a deterministic foreign-safe teaser (no model) and a
model-on-sub own-repo tier reusing the existing decideExecute consent. Hard
invariants: difficulty is a display not the auto gate (the gate is the final
trust filter); hook-surface rules never auto-build (false safety = the flagship
2/7-vs-7/7 finding); the report never spends; one engine, two entry points.
Spec only; implementation to follow.
* audit(rule-inventory): FP-safe deterministic detector for documented-but-unconfigured rules
The free/foreign-safe surface of the audit rule-compile tier (spec:
research/audit-rule-compile-tier.md). Pure function: given a repo's
CLAUDE.md/AGENTS.md text + lint-config text, find prose rules that map to an
off-the-shelf lint rule and whether that rule is already in the config —
the one-line-config-fix nudge.
No model, no config execution (textual grep only — avoids the eslint-config
RCE path). High precision by construction: only rule-name/code-token-shaped
keywords, matched as whole tokens; every bare-word trigger the FP measurement
flagged (token/secret/await/!/aria/silently/…) is dropped. Measured: the raw
keyword matcher fired 107x across 4 real CLAUDE.md files (~all garbage); this
detector returns 0 on that prose and fires correctly on real rule mentions.
6 tests incl. the FP guard, embedded-token guard, regex-metachar escaping,
and in-config vs not-in-config.
Next: wire rulesInventory into AuditReport + the report section + the opt-in
model tier (compile/gate/run) per the spec.
* audit(rule-inventory): multi-linter shape (ESLint-first data) + spec build-status
Per founder: the detector was ESLint-only. Refactor the SHAPE to be
linter-keyed (LinterName eslint|ruff|clippy|pylint|rubocop|stylelint) with an
optional {linters} filter — the matcher was already linter-agnostic, so
adding another linter is additive DATA (rule-name entries), not a refactor.
Data stays ESLint-first + honestly labelled; the opt-in tier will resolve
enablement via the existing multi-linter checkLinterRule engine (which execs
config) rather than reinventing it. 8 tests (adds linter-tag + filter cases).
Internal spec updated with build status; public docs held until the feature
renders (no vaporware).
* audit(rule-inventory): wire rulesInventory onto AuditReport (additive, schema unchanged)
Mirror the adoptable/observations pattern: optional rulesInventory? on AuditReport
+ BuildAuditReportOptions, spread through the pure builder. The CLI will compute it
(read instruction + config text -> buildRuleInventory) and pass it in, keeping the
builder fs-free. Additive within schemaVersion 1. 18 audit-report + rule-inventory
tests green; build + eslint clean.
Next: CLI populates it (the fs read) + the report section renders it (report/src/
schema.ts + a section component, Adopt.tsx copy-command pattern).
* audit(rule-inventory): CLI populates rulesInventory (first user-visible output)
The audit command now computes the deterministic rule-inventory teaser: reads the
repo's instruction file + lint-config TEXT (best-effort, fs-only, NEVER executes
config — safe on any repo) and maps documented intents to off-the-shelf rules via
buildRuleInventory, passing it into buildAuditReport. Verified e2e: audit --json on a
repo whose CLAUDE.md documents console.log/no-explicit-any/eslint-disable emits
rulesInventory with in-config (no-console found in eslint.config.js) vs not-in-config
nudges, and correctly ignores a prose token-savings mention (FP-safe live).
cli.test.ts 109/109, all rule-inventory/audit-report/audit-html tests green. (The 6
unit failures are pre-existing env-only linter-CLI tests — rubocop/pylint absent
locally — untouched by this change.)
Next: render it in the HTML report (report/src schema + section component).
* audit(rule-inventory): render the Rule inventory section in the HTML report
Report app now renders rulesInventory: report/src/schema.ts gains the type +
field; a RuleInventory component shows each documented intent -> off-the-shelf
rule with an 'enforced' (green ✓, in config) vs 'documented, not enforced'
(amber ●) state and a copy-config-fix button (the Adopt.tsx static-copy
pattern); Report.tsx renders it as a section. Also add the scoped
@typescript-eslint/* keyword forms so real projects (which cite scoped rule
names) match. Dogfooded: a rule-documenting repo renders 7 items (2 enforced,
5 nudges) with the semantic rules correctly omitted; report bundle rebuilt.
* audit(rule-inventory): copy button emits an agent prompt, not a raw config line
Per founder: the raw JSON snippet is clumsy (where do you paste it?). vigiles
users live in Claude Code / Codex, so the actionable affordance is an
instruction the AGENT executes — 'enable the ESLint rule X (error) in the lint
config so the documented rule is enforced'. Button now copies that
paste-ready prompt; label 'copy agent prompt'.
* audit(rule-inventory): precision-harden after big-OSS dogfood (17/18 FPs → 0)
Dogfooding the rule-inventory teaser on large real ESLint repos exposed
17-of-18 false positives. Three root causes, three fixes:
- Root-only config read missed nested configs. collectLintConfigText now
walks subdirs (depth ≤2, skips node_modules/dist/.git/…, caps 60 files),
textual only — never resolves/executes a config (the RCE path stays shut).
- oxlint + biome configs weren't in the read set though they use ESLint's
rule names; added. RULE_INVENTORY_CONFIG_FILES is now a Set.
- Only the harness-native instruction file was read; rules are often in
AGENTS.md under a claude-code harness. Now reads CLAUDE.md + AGENTS.md +
the native file.
Detector precision: base ↔ @typescript-eslint/ variants now count as the
same rule in config (variantsOf/ruleInConfig), and a `recommended`-preset
extend downgrades an unnamed preset rule to a new "preset-maybe" state
instead of a false "unenforced" alarm (7 intents tagged inRecommended).
Report renders three states — enforced / likely-enforced-preset (muted, no
nudge) / documented-not-enforced (the actionable copy-prompt).
Verified: build green, self-lint pass, 21 tests green (+5 new), and
cloudflare/agents re-dogfooded from 3 FPs to all in-config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* docs: document the rule inventory (audit) + dogfood result
- what-vigiles-catches.md: new shipped row 16 — a prose rule that an
off-the-shelf lint rule already enforces but which was never turned on.
- cli.md: describe the rule-inventory section of the audit report — the
three states (enforced / likely-enforced-preset / documented-not-enforced),
the text-only (never-executed) config read, whole-token matching, the
20-repo dogfood that drove FPs to ~0, and the multi-linter shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit(rule-inventory): suppress false nudge on documented opt-outs
Dogfooding pmndrs/react-spring surfaced a wrong-advice FP: its CLAUDE.md
documents `@typescript-eslint/no-explicit-any` as "off intentionally", and
the inventory nudged to enable it. A rule the author deliberately turns off,
and which really isn't in config, is CONSISTENT — not a gap.
Add isDocumentedOptOut: a deterministic negation window (off / disabled /
not enforced / not enabled / turned off) tested on the context either side
of the matched token — never the token itself, so `disable` can't self-match
inside `eslint-disable`, and the full word `disabled` (not `disable`) avoids
a neighbouring `eslint-disable` leaking its cue into an adjacent rule's
window. When it fires on a not-in-config mention, the item is dropped.
Verified: 13 tests green (+2), react-spring re-dogfooded — the false
no-explicit-any nudge is gone, only the real in-config no-console remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit(rule-inventory): add two off-the-shelf mappings from the real corpus
Cross-referencing the hand-verified rule-adherence corpus (real repos:
motion, mapbox, palantir, …) against INTENT_MAP surfaced two documented
intents that DO have an off-the-shelf ESLint rule but weren't mapped:
- "no default exports" → import/no-default-export
- "no TODO/FIXME" → no-warning-comments
The same cross-reference confirms the deterministic tier's ceiling: of the
corpus's 13 real rules, ~4 route to an off-the-shelf rule, ~5 CONTENT rules
are synthesis residue (object-spread ban, optional-chaining ban, British
spelling, package-manager), and all 4 ACTION rules (no-AI-coauthor,
don't-edit-generated, DCO sign-off, never-push-main) are hook territory —
which is the case for the opt-in compiler + hook tiers, not more INTENT_MAP.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: fold agent-rules-compiler into the monorepo as @vigiles/compiler
Moves the rule-compiler engine (former standalone zernie/agent-rules-compiler)
into the vigiles monorepo as a self-contained opt-in sub-package, mirroring
report/'s package-boundary pattern. Decision + rationale: migratsiya/49.
- compiler/ = @vigiles/compiler (CommonJS, own eslint/ts deps). The two-stage
trust gate runs unchanged in the new home: `cd compiler && npm run gate`
reproduces kept=3 / kept-ungraded=3 / abstain-selftest=1 / abstain-gold=1,
including the R10 silent-leak catch (the paper result).
- Isolation preserved: root tsc is src/** only, root eslint is src/ only, so
neither touches compiler/. Added compiler/ to .prettierignore as its own
package boundary (own toolchain; also holds machine/agent-generated rule
code + gold-set data + exact-byte run artifacts). node_modules stays ignored.
- README documents the pipeline (classify → reuse → synthesize → gate → enforce)
and that this tier loads ONLY behind the opt-in consent, keeping the default
audit/lint surface deterministic and model-free.
Root gates verified green: prettier --check ., build, self-lint (512 files, all
refs valid).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: add extract-existing — resolve a repo's real enabled lint rules
The REUSE tier needs to know what a repo ALREADY has before it synthesizes.
extract-existing runs the target repo's own eslint (`eslint --print-config`)
to enumerate enabled rules + severities, reads installed eslint-plugin-* from
package.json, and exposes ruleStatus() (error/warn/off/absent) with
base↔@typescript-eslint variant unification.
EXECUTES the target config (RCE path) → opt-in / own-repo only; the deliberate
exec-based counterpart to the foreign-safe, exec-free src/rule-inventory.ts.
Verified live on vigiles (141 enabled, 13 off, 4 plugins). Pure logic covered
by extract-existing.selfcheck.js (5/5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* research: end-to-end rule-compiler flow design + decisions (suppressions ratchet, harvest-at-scale, compiler-first)
Captures the whole pipeline (harvest → segment → classify+reconcile → gate →
existing-violations → persist → feedback) so it stops being re-derived stage by
stage. Two decisions recorded: existing violations grandfather via ESLint's
native eslint-suppressions.json ratchet (never eslint-disable), and Stage 0
becomes a bulk rule-harvester/taxonomy. Order: fix the compiler first, then run
the harvest over it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: expand the plugin index (REUSE tier) 20 → 129 intents
The reuse catalog now covers the common real-world agent-rule intents across
~18 ESLint sources: core, typescript-eslint, import, unicorn, sonarjs, promise,
n, security, jsx-a11y, react, react-hooks, regexp, eslint-comments, jest,
vitest, testing-library, no-secrets, boundaries. Adds a self-documenting
`plugins` index (source-prefix → package + purpose) and flags 44 entries with
type-aware / heuristic / rename notes.
Correctness over quantity: all 227 unique rule ids were checked as real, current
rules of their plugin (a programmatic existence-check against installed plugins
is a later step; the compiler's enable path + eslint error surface truly-absent
rules as a safety net). Keywords include both prose forms and rule-name/code
tokens so the classifier can match either. The deterministic high-precision
subset stays in src/rule-inventory.ts and is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* compiler: add classify — route each rule + reconcile against real config
The Stage-2 module that ties the pieces together: an atomic agent-rule →
{reuse | synthesize | hook | semantic}, and for reuse, reconciled against the
repo's real config via extract-existing (enforced / warn / off-contradiction /
absent-installed / not-installed). Hook detection is action-cue specific (git
push, generated files, sign-off, shell) so it doesn't grab lint rules.
Hardening from testing on the 13-rule hand-verified corpus:
- empty-match guard: a code-operator keyword like "no & or |" is an empty
regex alternation that matched EVERY rule and collapsed all classification
onto no-bitwise; keywordRe now falls back to escaped-literal for any keyword
whose regex matches "".
- added `as any` / `as unknown as` / `<any>` / `: any` to the no-explicit-any
intent — the article's own centerpiece was missed because `\bany\b type`
needs "any type" adjacent and the backtick form breaks it.
- documented the polarity limitation (banning `??` keyword-matches
prefer-nullish-coalescing) — the deterministic pass routes; the opt-in model
pass disambiguates before any enable.
Corpus distribution: reuse 4 / synthesize 5 / hook 4 / semantic 1 — matches the
tier-coverage thesis (a minority route off-the-shelf; the rest need synthesis
or hooks). 8/8 self-checks pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* research: link compiler-end-to-end-flow from the index (fix orphan-doc lint)
The design doc was orphaned (referenced by no other .md), tripping vigiles's
own orphan-docs lint (exit 2). Linked it from research/README.md next to the
audit-rule-compile-tier spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* lint: exclude compiler/** from self-lint (package boundary, like bench/ + startup/)
The @vigiles/compiler fold brought demo fixtures (compiler/demo-project/CLAUDE.md,
demo eslint configs) into scope, tripping require-instructions-spec + rule-coverage
(exit 2). compiler/ is a self-contained package already excluded from tsc/eslint/
prettier; excluding it from the harness self-lint too. self-lint now PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* readme: temporarily cut the zernie.com blog link (double-blind, paper #1)
The footer 'companion to Feedback Loop Is All You Need' link is a repo -> blog
deanonymization path while AgenticDev paper #1 is under blind review — and the
reviewer-offer email links this repo. Removed with a restore-note for after
notification (2026-08-21). Fable-flagged leak.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* report: fold 'Create specs' to the bottom (de-emphasize the spec-adoption nag)
Moved the 'N surfaces not yet spec-managed' block from mid-report (right after
Fixes) to a collapsed <details> at the very bottom. Surfacing it high read as
homework/nag to a first-time reader; spec-adoption is optional hardening, not a
failure. Collapsed by default, labeled 'Optional'. (Founder feedback.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit revamp: Rules two-zone hero + deterministic CLAUDE.md segmenter
Part of the comprehensive audit revamp (Fable-designed).
- report RuleInventory: rebuilt as the "Rules — prose → enforced" hero. A
summary line + "deterministic · no model" badge, a 100%-stacked distribution
bar, rules GROUPED BY STATE (contradiction + one-line-away open on top;
enforced/preset folded to a count), and a dashed "Compile — the full picture"
CTA that DESCRIBES the opt-in tier (reuse/synthesize/hook/prose) without
fabricating result rows. Adds the `contradiction` configState to the schema
(detector wiring next).
- src/segment.ts: deterministic Tier-A segmenter — splits an instruction file
into atomic candidate rules with provenance (exactQuote + line span) via a
3-cue imperative gate (form/context/shape → high/medium/reject), precision-
first (under-splits, never splits "never X except Y"). No model, no deps.
14/14 tests green. Feeds the upcoming State-B heuristic routing preview.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit revamp: wire the re-scoring verdict engine + config-contradiction rule state
- computeVerdict re-scores the report with each finding removed to derive real
pointsIfFixed per recommendation + a verdict sentence (fixes-to-next-grade),
now assembled into every AuditReport (schema + report mirror + parity test).
- rule-inventory gains a 'contradiction' config-state: a rule the harness
documents as a norm but the config explicitly sets to off/0 (docs vs config
disagree), distinct from merely absent. ruleSetOff is textual-only, no exec.
- research index: add audit-rule-compile-tier + compiler-end-to-end-flow docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit report: verdict-led redesign (grade hero, ranked fixes, scannable strip)
- New hero: the re-scored verdict sentence is the headline, grade+score a
band-bordered badge, dir shown as a plugin basename, trust line beneath.
document.title becomes 'vigiles — B (84) · my-plugin'.
- Fixes are ranked by real pointsIfFixed with a '+N pts' badge and a copy-
agent-prompt button; the left border encodes IMPACT (never green).
- Five heavy category rings collapse into a compact bar-per-category strip.
- 'What it ships' demoted from six stat cards to one meta line; spec-adoption
stays folded at the bottom. Rules section retitled 'Your rules → enforced'.
- band.ts gains a literal BORDER map (the JIT never emitted the string-built
'border-warn'); drop the footer AlertTriangle (green/neutral discipline).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* readme: showcase the rules-compile capability + re-render the hero screenshot
- New Proof 3 ('a rule you wrote that nothing enforces') pays off the tagline:
maps prose rules to the lint rule that enforces them — on, one line away, or
silently turned off — and points at the opt-in compile tier. Secrets-exfil
becomes Proof 4 (still the closing bite).
- 'What it caught' now describes the verdict-led header + the 'your rules →
enforced' section instead of just the five rings.
- Re-rendered vigiles-audit.png from the new verdict-led report (dark), and
updated its alt text + the DIRECTION note's screenshot + spine guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit report: apply Fable review — honesty + hierarchy fixes
- Rules headline no longer rounds 'preset-maybe' up to 'enforced' (the tool's
whole pitch is that config lies about rules): counts + colours enforced,
preset, one-line, and contradiction separately, with a 4th preset bar segment.
- Contradiction row ('the one people screenshot') gains a copy-agent-prompt
(a decision prompt, since the rule is present-but-off) and reads 'set to off
in your config'; one-line rows relabelled 'enable in 1 line' to stop colliding
with the verdict's point-scoring 'fixes'.
- New 'Review — no auto-fix' section surfaces Safety findings (the lethal-
trifecta exfil path) as cards — the report's severest finding was previously
only 11px category-strip text with no card (it maps to no recommendation).
- Category findings de-printf'd at the render layer ('1 tool(s)' → '1 tool');
scorer strings untouched. Compile CTA drops 'blind-gated' internal vocab.
- Sample fixture: broken refs are now file/dir (not a lint rule) so Adoptability
no longer visually collides with the Rules hero; re-rendered the screenshot.
- README Proof 3 reframed as representative reader's-repo output (not a claimed
public-plugin catch under the 'real scans' banner) with reconciled numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* audit: deterministic State-B rule routing + scorer-level pluralization
State-B routing preview (src/rule-routing.ts):
- Segments the instruction file (segment.ts) into atomic rules and routes each
one deterministically — reuse (off-the-shelf rule -> config line), hook (action
rule a linter can't see -> hook), semantic (judgment call -> prose), or unrouted
(no deterministic route -> the opt-in compile tier decides). Honest by
construction: never claims 'synthesizable' — that's the model tier's job.
- Precision-first: routes only high-confidence segments, so a doc-heavy file
(a keyFiles index of "path — desc" bullets) doesn't flood 'unrouted'.
- Wired into the audit report (schema + CLI + report mirror). The report's
Compile CTA now shows REAL per-category counts + example rules instead of a
generic pitch — grounds the upsell in the reader's own rules.
Scorer-level pluralization:
- audit-score's finding labels resolve the terse "(s)" placeholder against the
count ("1 tool" / "3 tools") at render, via a pure pluralizeLabel; mirrored in
leaderboard.ts and the verdict sentence (audit-verdict.ts). The report's
render-layer humanizer is removed — the logic lives at the scorer now.
Report polish (Fable #4 follow-through):
- 'Adoptability' section renamed 'Broken references' with coherent copy (it reads
the repo, it doesn't describe a hypothetical). Sample fixture pluralized.
- Re-rendered the hero screenshot with the routing breakdown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vN4N2WCoaxzvoDVVGMynP
* fix: type new Array<number> in segment.ts (no-unsafe-assignment lint error)
The CI 'test' job runs the repo's stricter ESLint; new Array(n) is typed any[],
tripping @typescript-eslint/no-unsafe-assignment. Annotate the element type.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
18d7cfcae4 |
docs: honest versioning, README scannability, and docs restructure (#67)
Fix the version claim (v12, not 0.x; the 0.0.0-semantically-released placeholder was the source), reframe the eval section around cost measurement, trim the README for scannability (Diátaxis link buckets, tighter reconciliation + proofs), reorganize the docs index into Guides/Reference/Explanation and add six missing docs, and consolidate overlapping docs (related-tools→comparison, inline-mode→markdown-mode, agent-setup+agent-workflows). Relocate the eval-architecture design-of-record ADR from docs/ to research/ per the doc-tiers rule, and fix the library entry points (vigiles/linting is the compiler surface) plus the Codex setup path. |
||
|
zernie
|
bd44a35127 |
feat(scan): support non-plugin skills repos + FP-safe bundled-resource refs
Adoption fixes from a skills-monorepo field report. - Recognize three repo shapes (published plugin / bare skills library / plain .claude user repo) via a parsed SurfaceSource union; project .claude only. - lint scopes to an explicit directory arg (bare lint unchanged); shared refs resolve against the linted repo's own root. - skill-resources skips globs/placeholders and ~/ paths; opt-in sharedDirs. - Lethal-trifecta advisory collapsed to one line per unit; docs/skills-monorepo.md. |
||
|
zernie
|
467f65d430 |
feat(experimental): R3 disposable-service eval tier + promptfoo migration guide
Add the experimental R3 disposable-service eval tier (vigiles/experimental) for real side-effect skill testing on the subscription, plus a promptfoo migration guide. README eval section leads with the measured cost case. |
||
|
zernie
|
c442f4b523 |
feat: faithful caveman ecosystem benchmark + --plugin-dir/skillsDir install support (#61)
Land the completed ecosystem-benchmark work that was stranded on the gate-release branch: the real caveman plugin install (pluginDir + SessionStart hook) with the codified bare-SKILL.md warning, the 7-task corpus, the analysis scripts, and the canonical 2026-07-07 runs (7 tasks × 5 trials, $10.41; compress input test) plus corrected FINDINGS. This is the data behind the published token-savings article and makes `npx vigiles eval` reproduce the real thing. |
||
|
zernie
|
81be2fd07e |
feat: capability-diff PR comment + local observe ledger (#55)
* docs(vault): harness-quality tech camps + telemetry/default-install decision Two-camps tech map (runtime/observability vs pre-deploy simulation vs app-frameworks), the field direction, and the live fork: local-first deterministic measurement over hosted runtime telemetry. Cross-linked from the vault index and funded-adjacency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(vault): add pseudocode DX + fundability/exit thesis to tech-direction Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): tech direction — four-instrument loop + local observability ledger The architecture decision of record for the next build phase: typed spec as declared ground truth; verify/gate/measure/observe as facets of one loop; the deterministic-vs-behavioral precision principle; the local agent-readable runs.jsonl flight-recorder ledger; promote-prose; per-surface map. Monetization kept out (vault only). Indexed + recompiled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add Direction section to root CLAUDE.md positioning Elevate the next-phase direction to the north-star doc (read first every session): the four-instrument loop (verify/gate/measure/observe) with the typed spec as declared ground truth, the deterministic-vs-behavioral precision principle, the local agent-readable runs.jsonl flight recorder, promote-prose, and the per-surface map. Tech only; monetization stays in the vault. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): local agent-readable flight-recorder ledger (runs.jsonl) The connective layer of the four-instrument loop: a versioned, append-only, tolerant-to-torn-lines JSONL ledger every instrument (verify/gate/measure/observe) appends to, read off one schema by audit, by the agent debugging its own harness, and later by an aggregation surface. Harness-agnostic; lives at the composition root (not core, not an adapter). Best-effort append never breaks a session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for direction + observe-ledger session * feat(observe): emit compiled-hook + agent-rail decisions to the ledger First emit points into .vigiles/runs.jsonl: the compiled-hook gate (emitGate — enforce deny/ask + observe would-be) and the subagent tool-contract rail (agentHookCommand deny). The legacy hook-observations.jsonl still written for back-compat; the unified ledger is additive. E2E asserts the observe-mode run populates runs.jsonl with a decision-shaped hook record. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): render the flight recorder in vigiles audit formatLedgerSummary (pure, tested) — total + counts by kind + recent high-signal denials (blocked gates, out-of-contract tool calls) — and audit prints it (terminal, non-json) off the local ledger. Empty/skipped until something is recorded. JSON AuditReport unchanged (a versioned schema bump is the deliberate follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: gitignore the observability ledger (.vigiles/runs.jsonl) Runtime flight-recorder artifact — local record, never committed (sibling of active-skill.json + eval-cache/). * feat(observe): emit skill-fire + trigger-rate metrics to the ledger skill-start records a SkillObservation{fired:true} (name = the skill dir); a completed measureTriggerRate records recall (+ precision when measured) as eval records. Extends ledger coverage across the behavioral surfaces alongside the deterministic gate/rail emits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(skills): add model-invocable debug-my-harness skill Reads the .vigiles/runs.jsonl flight recorder to diagnose why the harness misbehaved — skill fires/collisions, hook decisions (blocked vs wrongly-allowed), subagent contract violations, trigger-rate drift — and recommends a fix, handing off to strengthen/edit-spec/test-harness. The agent-readable payoff of the observe layer. Ships a dogfood trigger eval (recall + precision); loads clean (skills-dogfood). Skill enumerations + keyFiles synced (3→4 model-invocable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): emit capability-diff to the ledger A computed capability-diff (--capability-diff) now records a capability-diff record (added/removed tools + widened verdict). Completes the ledger's emit coverage — all five record kinds (hook/agent/skill/eval/capability-diff) now have live emitters. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF — observe layer built (ledger + 5 emits + audit render + skill) * feat(observe): carry the flight-recorder summary in the AuditReport JSON + HTML summarizeObservations (pure, shares denial logic with the terminal formatter) → an additive/optional observations field on AuditReport (no schema bump, per the adoptability/adoptable precedent). The CLI reads the ledger once, feeding both the JSON (the product boundary — audit --json + the HTML) and the terminal render. New report app: schema mirror + a presentational Observations component + a Flight recorder section. Covered by observe + audit-report tests; end-to-end --json + HTML verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF — AuditReport JSON/HTML ledger integration done * docs(readme): verify-first front door (catch the silent breakage) Retune the tagline + hook to lead with the verify moat (references are real, not just well-formed) over the prior test-first framing; add a coding-scoped subhead naming CLAUDE.md/skills/hooks + Claude Code & Codex. audit stays the proof-first hero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): lock verify-first tagline in the editor-guide (prevent drift) * docs: encode research-depth discipline (vault) + link directionality (doc-tiers) Vault operating rule 6: competitive/strategy research saved in full detail, depth-tagged (deep/medium/thin), coverage-mapped, cross-linked, indexed — shallow never masquerades as deep. Root doc-tiers: link directionality invariant — links point outward only (vault may cite research; research/public must never cite the vault). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): demote the 7-catalog breadth from the headline Lead the reference-verification pitch with the harness-native cross-ref (a referenced tool/hook/file/skill that doesn't resolve). The 7-linter-catalog breadth becomes a supporting mention, not the headline number. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: delete hook-spec spike and demote scaffold-test to skill-internal Trim the launch surface of two dead-weight items flagged by a strategy review: - Delete the src/core/hook-spec.ts + hook-spec.test.ts spike — imported nowhere and superseded by the shipped compiled hooks (src/core/hook-program.ts). Drops its two keyFiles entries and recompiles CLAUDE.md. - Demote the scaffold-test VERB to skill-internal: remove it from VERBS + its dispatch/handler in the CLI (the engine src/scaffold-test.ts + its test stay, used by the test-harness skill). Prune the now-orphaned helpers/imports, the scan-cli e2e block for the removed verb, and the scaffold-demo example, and update the docs (cli.md, spec-format.md, STABILITY.md, CLAUDE.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF with observe layer + verify-first pivot state Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: format harness-observability-direction.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(action): capability-diff PR comment (blast-radius diff vs PR base) Fold the agent capability-surface diff into the sticky PR comment: on a pull_request the Action materializes the PR base with `git archive` (no worktree/smudge), runs `audit --capability-diff`, and appends a short section only when the surface changed. `fail-on-widen` gates a widening. Dogfooded in this repo's own CI (fetch-depth: 0 + capability-diff: true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: salvage deep agent-eval-testing competitor teardown (partial workflow) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — capability-diff shipped + teardown salvaged Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(vault): post-mortem the teardown throttle+interrupt; log the 2x2 re-run Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: fold in the 4 re-run teardown clusters (all 5 now deep, 41 companies) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: HANDOFF — teardown re-run complete (5 clusters, 41 companies) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: add raw structured teardown appendix (41 companies, durable source) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: add the Opus cross-corpus strategic synthesis (conformance-kernel thesis) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: fold in Fable contrarian pass + Opus reconciliation (registry-first bet) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: record the distro thesis (leading direction) + index it * docs: fix doc-tiers — repo is PUBLIC, strategy is vault-only (stop leaks) The doc-tiering rules wrongly treated research/ + CLAUDE.md as a 'private record' holding the moat/positioning narrative. The repo is open-source, so those tiers are public. Corrected: research/CLAUDE.md hold the TECHNICAL record (how the tool works, public-safe); all strategy/moat/monetization/pivot theses live ONLY in the git-crypt vault. Adds the competitor-playbook test. * docs: de-strategize + refresh HANDOFF (it's public — strategy → vault) * docs: move 6 strategic docs to the git-crypt vault (repo is public) measurement-authority, divergent-bets, distribution-strategy, typed-spec-moat, strategic-synthesis-2026-06, eval-startups-positioning move from research/ (public) to startup/ (encrypted). Inbound references neutralized (no vault inward-links), index/keyFiles entries removed, both specs recompiled, vault index updated. * fix: encrypt the 6 migrated vault docs (git mv bypassed git-crypt) * docs: record the git-mv-into-vault-doesn't-encrypt gotcha + fix rule 3 (all public) * docs: pivot-decision (vault) + roadmap reshape + finish moved-doc ref cleanup Adds startup/pivot-decision-2026-07.md (decision + validation gate). Adds two pivot-agnostic technical items to research/roadmap.md 'Now' (step-enforcement A/B, observe→promote) + de-strategizes its top direction banner. Neutralizes the relative-path refs to the 6 vault-moved docs the migration missed (README + plugin-structural fully clean; roadmap partial — 37 scattered refs flagged as follow-up). * docs: HANDOFF — migration done (targeted-6) + flag roadmap ref-sweep follow-up * vault: save the pivot exploration log (all options considered + why) * docs: finish roadmap de-strategization (neutralize moved-doc refs + gut GTM section) * fix(cli): write capability-diff observation to the audited root's ledger appendObservation defaulted to process.cwd(), so `audit ./after --capability-diff=./before` from a parent dir recorded the blast-radius event into the caller's .vigiles/runs.jsonl instead of ./after's — where the audit reads it back via readObservations(root). Pass root. (Codex review #55.) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
3f33925722 |
feat: skill-description-budget rule + document-the-why + audit/opt-in docs (#54)
* docs: explain why compiler-grade guarantees are opt-in The "opt-in, like TS's strict" line appeared in three public docs but only asserted the analogy, never the reason. Add a dedicated FAQ entry that gives the actual answer — opt-in means gradual not permissive (the default already gates real breakage), compiler-grade proofs need a typed program to check (tsc can't verify inert markdown), and gradual adoption is the TS-strict bet — and link the README and linting-guide one-liners to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: codify auto-vs-nudge enforcement model in the spec skills Apply the init enforcement model at edit time across the three spec skills: auto-apply the con-free wins, nudge (with the tradeoff) for anything that costs config/plugins/CI-failure, never silently escalate into strict gating. - strengthen: name the principle explicitly (cost, not strictness, is the dividing line; Tier 1 auto, Tiers 2-4 are the user's call) - edit-spec: add "do what's asked, don't silently escalate enforcement" - adopt-spec: replace the prose blockquote with explicit, scannable adoption rules (faithful / non-destructive / don't-escalate / reversible / ask-before-writing / lighter-touch-exists) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: explain what audit runs (read vs measure) + selection-collision Close the audit→behavioral→consent documentation gap exposed while dogfooding a skills-heavy repo: a clean-A audit "found little" because a plain audit is a deterministic READ — the behavioral checks (does each skill fire? do two collide?) run only under the one-time audit.measure consent. - faq.md: new "What does audit actually run — and why did it find nothing?" entry stating the read-vs-run model and the headless fallback - for-plugin-authors.md: turn the bare "selection-collision matrix" name-drop into a real explanation (diagonal = recall, off-diagonal = a sibling hijacked the prompt; body-stubbed, so it measures selection cheaply) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add document-the-why rule + doc-command coverage check The "docs suck" failure repeated twice this session (the opt-in why; the audit→behavioral→consent story), and no existing docs rule owns it: the polish, placement, and sync rules govern how well / where / consistency, but none requires the REASONING behind a user-facing decision be documented at all. - new `document-the-why` guidance rule: every user-facing decision/concept gets a plain-language doc home that answers why/what (name-dropping ≠ documenting); the test is "could a user answer it from the docs alone?" - new deterministic FLOOR (the inverse of self-command-refs): every public CLI verb must be mentioned under docs/ — src/doc-command-coverage.ts, dogfooded in its test (catches a verb shipped undocumented). The concept-coverage half stays judgment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: add skill-description-budget lint rule (warn-tier trigger proxy) A model-invocable skill is selected on its description, and the selector weighs the opening most — a bloated description buries the trigger signal and hurts recall + precision. New deterministic heuristic proxy (generous 500-char budget), the free sibling of description-overlap. Wired end-to-end: detector (src/core/skill-description-budget.ts) + rule-meta (heuristic-behavioral → warn ceiling) + DEFAULT_RULES + NUDGE group (never gates) + scan report + lint application + docs/rules + the rules matrix + tests. NOTE on haretrail: it does NOT fire there — the earlier "634/781-char" figures were BYTE counts; in characters (Cyrillic = 2 bytes/char) the longest skill description is 463 chars, under budget. The rule is correctly calibrated FP-safe; forcing a hit would mean crying wolf. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for haretrail-dogfood session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * test: add skill-description-budget to validate.test DEFAULT_RULES literal The new rule landed in DEFAULT_RULES but validate.test.ts asserts the full rules object against a hardcoded literal — add the key so the deep-equal passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: prettier-format HANDOFF.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * refactor: remove the cross-language description scan flag (refuted) The deterministic "non-Latin description → cross-language trigger risk" finding (unexpectedScript / descriptionScript in scan.ts) is removed: the Script type, scriptCounts, unexpectedScript, the ScanSkill.descriptionScript field, its report line + summary, its tests, the docs mention, and the config-expected-script notion. Why: it fires on 100% of non-Latin descriptions, but a non-English description is a legitimate audience choice, not a defect — so it scolds correct work. And the one time we measured it (plugin-behavioral-findings.md Finding 3a, native codex), the risk was REFUTED — the model handles RU-description/EN-prompt selection fine. A warn that reads as a defect on correct authoring cries wolf (lint-rule-calibration). Reasoning saved in research/plugin-behavioral-findings.md; the one-detector-no-drift teaching example reassigned to description-overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: eval cost transparency — surface tokens, API-equivalent $, metered warning Make what a real-model run SPENT impossible to miss (vigiles's affordability pitch is "runs on your subscription, not a metered API" — so every run should say so). - new src/eval-cost.ts: the pure engine — cost snapshots (per-run / per-arm / whole-report), metered-vs-subscription detection (ANTHROPIC_API_KEY set → metered), a session tally, and the formatted block (tokens + API-equivalent $, a LOUD warning + actionable fix when metered, ✅ when on the sub). Fully unit-tested, no model. - runEval now emits the cost block after a run (stderr; report carries usage). - test-harness skill: MUST relay the spend to the user after any eval it runs. - measuring-skills docs: "what a run reports + the metered-API warning" section. HONEST SCOPE: no "% of your subscription" — Anthropic doesn't expose plan quota, so a percentage would be invented; tokens + API-equivalent $ + billed-to is the complete honest picture. FOLLOW-UP: measure/measureArms/measureTriggerRate reports don't carry usage yet (only EvalReport does) — threading a usage field through those model-gated run loops + the eval-lock is the next commit so the skill's trigger-rate path emits too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * fix: brace void-returning arrows in eval-cost (eslint no-confusing-void-expression) Missed running eslint on the new file (only ran fmt:check). The console.error sink and the test beforeEach arrows returned a void expression in shorthand form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add surface-architecture-decisions rule Surface non-trivial architectural decisions (file placement, core vs adapter vs port, extend-vs-create, cross-cutting changes) to chat in a scannable block with the boundary reasoning — the structure analog of subagent-model-note — so the user can catch a wrong placement before the diff, given the hexagonal + multi-adapter constraints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * feat: thread usage into TriggerRateReport so measureTriggerRate emits cost The skill's main path (measureTriggerRate) now surfaces spend too. Per-run usage comes from the parser (makeContext(out, cfg.parse).usage) — harness-neutral, since both the Claude and Codex parsers fill ParsedModelRun.usage (Codex reports $0 on the sub, real tokens). runTriggerTrial returns its run's usage; runTriggerSet collects them; the report aggregates via the existing aggregateUsage (relevant + irrelevant sets both counted); the wrapper emits costFromArm(report.usage). No new port/driver method (the parser already provides usage) and eval-lock-safe (additive field; the lock hashes inputs, not the report). FOLLOW-UP: measure/measureArms (CheckReport) still don't carry usage — same threading applies, next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: fold parse-don't-validate + illegal-states-irrepresentable into surface-architecture-decisions Frame them from this rule's angle (surfacing): a parse-boundary and an illegal-state-forbidding type ARE architectural decisions — name them when you make them (where the boundary sits; what invalid state the type forbids). Cross-refs ts-essentials for the mechanical how, rather than duplicating it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: update vigiles-testing API surface for TriggerRateReport.usage api-extractor tracks the public surface; adding the usage field is intended (additive — consumers read the report, don't construct it). npm run build doesn't run api-extractor, so this was missed locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * chore: refresh HANDOFF for the eval-cost / rules session Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: capture haretrail-driven audit + eval ideas (research note) Save the session's un-persisted ideas before context is lost: new deterministic cross-ref checks (openai.yaml / trigger-manifest / env-var contract) and new eval helpers (the selection-collision confusion matrix as a first-class primitive, manifest-driven fixtures + auto-negatives, description-ablation). Indexed in research/README.md (Skills) + cross-linked to plugin-behavioral-findings / measurement-authority / audit-wow-ideas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * style: prettier-format the haretrail-eval-ideas note + README Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GqWj8R3k4KccckmW68KCK * docs: add status/topic frontmatter + agent-facing index to research corpus Migrate all 114 research/*.md docs to carry `status:` (active/shipped/idea/ superseded/rejected) and `topic:` frontmatter, so the corpus is sliceable deterministically (`grep -l 'status: shipped' research/*.md`). Add research/CLAUDE.md.spec.ts — a scoped claude() spec whose keyFiles map every research doc to a one-line, status-tagged summary (grouped by topic). Compiles to research/CLAUDE.md, the agent-facing index a session loads as directory memory for research/ without opening 114 files. Mirrors src/CLAUDE.md.spec.ts (the code map); this is the research map. Bidirectional sync is enforced, not by hand: - the compiler verifies every keyFiles path EXISTS (a deleted/renamed doc fails `vigiles compile`) - src/research-index.ts + its dogfood test assert the reverse — every research/*.md (except README.md) appears in the index research/README.md stays the human, thematic index and now points at the machine index + the frontmatter query convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for research corpus index work * feat: thread eval cost into measure/measureArms CheckReport The scored-check paths (`measure`, `measureArms`) were the last eval entry points that didn't report spend — only `runEval` and `measureTriggerRate` did. Close the gap the same way: add `usage: ArmUsage` to `CheckReport` (sourced from the arm's already-aggregated usage), and emit the cost summary in the v8-ignored real-CLI wrappers. `measureArms` sums both arms (an A/B run pays for both). So every real-model eval path now prints tokens + API-equivalent $ + the metered-API warning. Wiring tests drive an injected runner (no model) and assert usage aggregates across trials and per arm. API surface + docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: promote selection-collision to an assertable primitive (measureSelectionMatrix) The cross-skill selection-collision matrix — "when I ask for skill i's job, does ONLY skill i fire?" — already existed but was reachable only from the CLI under the audit consent. Promote it to a first-class, assertable eval primitive so a plugin author can gate collisions in a test: - `measureSelectionMatrix(dir, opts)` — zero-setup: prompts auto-derived from each skill's description (or supply your own). Thin wrapper over the existing `measurePluginSelection`; `measureSelectionMatrixWith` is the injectable core. - `assertNoCollision(report, { maxOffDiagonal, maxPluginCollision })` — the throwing gate. Defaults to demanding ZERO collision; throws on a green-that-tested-nothing (unavailable / zero runs), never a silent pass. Both are exported on `vigiles/claude-code`, NOT the agnostic `vigiles/testing`: selection-collision reads which skill the selector chose, which only Claude Code surfaces (Codex has no skill-selection event), and `scan-behavioral` is Codex-adapter-coupled — so the CC surface is the correct home. Adds a canonical example eval, unit tests (auto-derive + all assert branches, no model), the plugin-author doc's CI path, and API/keyFiles updates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — cost threading + selection-matrix landed * docs: tidy HANDOFF formatting + refresh stale gotchas * docs: mark selection-collision as Claude-Code-only, consistently Make the CC-only surface unmistakable across the docs, one convention: - testing-api.md: a dedicated section "Selection-collision matrix (Claude Code only)" — the "(Claude Code only)" heading + a 🔵 callout naming the `vigiles/claude-code` import — plus a Contents entry and an Imports-section bullet cataloguing which measurements are CC-only vs agnostic. - harness-testing-claude-code.md: a pointer from the Tier-3 trigger section (a CC user's natural landing spot for a CC-only measurement). Complements the existing inline note in for-plugin-authors.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |