mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
v24
116 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
85c54e7e78 |
fix(lint)!: make the lint report complete, tierable and machine-readable (#187)
Closes #181, closes #182, closes #183, closes #184, closes #185, and the residual half of #173. #185 — every per-surface check read ONE root, so a monorepo with `skills/` plus `plugins/*/skills/` scored only the first: 4 skills over budget, 2 reported, exit 0. The default now NAMES the bundles it did not score; `--bundles=all` scores them in one pass with one exit code. Root-only stays the default so a vendored third-party corpus is not scored as if it were yours. #181 — `orphan-docs` had a documented severity nothing read, and the duplicate-rules check had no id at all, so neither could be tiered. Both are real rules now and the exit code respects their severity. #183 — the log and `--json` disagreed (21 lines against 88 findings) because some checks print per finding and others per group. One computed total, printed and carried in `--json`. #182 — `--json-out=<file>` writes the JSON while stdout stays human-readable, so a CI job stops scanning twice. #184 — `vigiles test` resolves a harness's bare `vigiles` import from the CLI's own install, so `npx vigiles@<version> test` needs nothing in the project. NODE_PATH was measured, not assumed: Node ignores it for ESM. #173 residual — `lint` verified the integrity HASH, which says nothing about whether the things a file NAMES still exist, so an artifact committed while its refs were live stayed green after the target was deleted. The new `spec-refs` rule re-derives them. It immediately found four dead references in this repo's own `examples/CLAUDE.md.spec.ts`, left behind by the hexagonal refactor. BREAKING CHANGE: `orphan-docs` and `duplicate-rules` now default to "warn" — a repo relying on either to fail CI must set "error" explicitly. Registering them surfaced that both are heuristic proxies, and this repo's calibration rule is that a heuristic never defaults to error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
94af23a5b7 |
feat(hook,cli)!: close five adopter-reported issues — halted-turn guards, dead-ref artifacts, centralized tests (#186)
Closes #173, #174, #175, #176 and #170. #174 — decideHook read exit 2, `decision` and `permissionDecision` but not the harness's halt-the-turn field, so `assertBlocksDisasters` reported a guard that stopped every disaster as blocking none. The field is read from the PORT, not hard-coded: `continue` is documented for Claude Code and unverified for Codex. The issue's adjacent suggestion was deliberately NOT implemented — adding the halt to `hook-block-ineffective` as a block-attempt signal would fire on hooks that WORK, since a halt is not event-scoped. It reads as a suppressor instead, fixing a real false alarm. A mutation implementing the original suggestion literally is pinned by a test that fails on it. #173 — compile printed its errors, exited 1, and wrote a hash-valid CLAUDE.md anyway, so `lint` went green over refs it had just called dead. Nothing is written on a failed compile now. #176 — eight small defects, including `audit --out` outside the repo appending unreachable `../../..` entries to .gitignore (a documented read-only report editing a tracked file), an unknown command exiting 1 against a documented 2, and STABILITY.md promising the removed `scan` verb. #175 + #176.2 — three missing capabilities, each opt-in and additive: `{surface}` in testGlobs, `--single`, and `rulesDir` so `.claude/rules/**` stops being invisible. #170 — one enumeration of export forms written against the grammar, both naming checks driven off it, and `check-internal-tag.mjs` asking the TypeScript compiler instead of a regex. It immediately found the issue's open case, 13 tagged types the regex had never seen, and a live gap in the ESLint rule. BREAKING CHANGE: `HookRunResult` gains a required `haltsTurn`, so code that CONSTRUCTS one (a test fake) must set it. Reading a result is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
1df264ce6b |
feat(cli)!: load specs in a vigiles-owned spec host, not a two-loader guess (#178)
Replaces the native-import/npx-tsx pair with ONE loader: a child process that registers vigiles' own module hooks and streams results as NDJSON. The measured bug: a consuming repo without tsx installed, where `npx tsx` went to the registry — >60s against a 15s budget, so all 50 specs failed at once with advice to run `npm run build`, a step that does not exist in a consumer install. Why a redesign and not more guards: the seam between two loaders cannot be made correct. The fallthrough guard had to answer "did the module body already run?" and Node does not expose that bit — ERR_MODULE_NOT_FOUND and SyntaxError each occur both before and during evaluation. The native path also had no time bound, and an in-flight evaluation cannot be cancelled. Three facts made it cheap: typescript is already a runtime dependency and already used at runtime; specs already round-trip through JSON (the old tsx path did exactly that); and the .js -> .ts rewrite is a rule to define, not inherit. Fixes at once: no npx and no network; a stalled spec is killed and NAMED; double evaluation is impossible by construction; resolution no longer varies by Node version or by which loader won; the Windows quoting bug dies with `npx -e`. Nine review findings closed — six by fixes, three by removing the seam. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014LVXMw2PJtPPtV8BoTANAd |
||
|
zernie
|
7144826614 |
fix(audit): cap the score on a confident breakage, split test-harness, retract an unverified claim (#177)
A confident breakage now caps the health score at 89, one point below the healthy band, so a definitively-broken harness can never render a green dot or grade A. Measured: Structure scored 92 with a green dot while carrying a dead tool contract, because a summed score dilutes one real breakage among clean siblings. confidentBreakages() enumerates what qualifies — decidable AND breakage, never the heuristic rings, since capping on those is how a gate gets switched off. The Safety ring stopped repeating one paragraph per unit: past three, ineffective disallowed-tools fences collapse into a single line that still names every skill. Measured 4,500 -> 810 characters. test-harness split into SKILL.md + references/, 5,534 -> 2,633 tokens loaded on activation, measured with a real tokenizer rather than chars/4. Retracted "no other plugin linter has it" on description-overlap: Cisco's skill-scanner ships --check-overlap. Edited the spec, not the artifact. Blast radius measured: this repo unchanged, clean vendored plugins still A. Gates: vitest 3288 passed, lint, prettier, api:check, build — plus the site browser-parity fixture regenerated for the new expected value. |
||
|
zernie
|
681edf7673 |
fix(hook): the subagent-delivery caveat is stale, and now a test says so (#172)
* fix(hook): the subagent-delivery caveat is stale, and now a test says so Claude Code #34692 — a PreToolUse hook not firing for a subagent's tool calls — was closed not-planned and quoted across these docs for months as a standing limit on what a gate can promise. It is fixed. Measured against a stock @anthropic-ai/claude-code@2.1.241 installed from the registry, not the container's binary: a subagent's own Bash reaches the hook and an exit-2 deny stops it, with the parent's identical command in the same run as the control and the marker files on disk as ground truth. The event also carries agent_type, naming which subagent made the call. The claim is about someone else's product, so prose cannot keep it honest — a doc cannot notice that the platform moved. src/subagent-delivery.test.ts pins both directions and goes red on a regression; removing the deny from its hook fails it on its own assertion. What did NOT change is stated at every site that was edited: a model can still route around a tool entirely, so a gate remains a strong default and is never unbypassable. The measurement's scope is stated too — headless only, with interactive sessions unmeasured and depth-2 nesting absent there. Gates run, in order: vitest (3282 passed), lint (0 errors), prettier --check, docs:check, internal:check, api:check, build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: regenerate the committed project-file types for the new test CI's generated-types:committed gate compares a fresh `generate types` run against the committed file, and adding a source file changes that list. Caught by CI rather than locally because I ran the individual commands instead of `npm run check`, which is the aggregate CI actually runs — the same shortcut this repo has already written down as a recurring miss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7c97e4ed7d |
refactor(spec)!: one experimental root per feature (#171)
The prefix answers "is this name stable?" and cannot answer "is the name my stable symbol depends on stable?". That gap shipped: railway(), delegate() and result() carried stable names while being meaningless without the experimental builder, and the lint rule cannot see it — it compares tag and name within one declaration, not the dependency between two. So the subagent vocabulary now hangs off one marked root: result, railway, delegate, pipe, pipeStep, needs, start and andThen become members of the builder. Eight names leave the top level. Two further gains the prefix alone could not reach: collision-prone words stop owning package-level names, and the warning survives destructuring, since the binding site still names the root — unlike a namespace object or an import subpath, which mark the import line. Object.assign rather than a TypeScript namespace: namespaces are an error in this repo's lint config, and one merges only with function declarations while the skill builder is a const. Types stay top-level, for the same reason the prefix rule excludes them. No deprecation window: nothing outside this repository imports these, and a window subtracts nothing. The container-rollback argument for a window does not reach them — they take no part in loading hooks. Found along the way, and worth more than the refactor: two example specs had not compiled since the previous rename, because the examples directory was in no tsconfig at all. Both fixed, and the class is closed — examples now type-check. Gates run, in order: vitest (3281 passed), lint (0 errors), prettier --check, api:check, build, exports:check, internal:check, docs:check, and tsc over the types project now that it includes examples. Claude-Session: https://claude.ai/code/session_017aUJEoaEQ3DxtvoSVWAWqe Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
900dc7e06a |
refactor(spec)!: name things for what they are; retire the vigiles/experimental subpath (#169)
Three exported symbols were named for something other than what they are, and
two mechanisms were doing one job.
RENAMES (old names kept one major, @deprecated, removed one major AFTER this one):
claude() → instructionFile() compiles CLAUDE.md AND AGENTS.md, so a
name from one harness was never right
instructions`` → prose`` builds a prose FRAGMENT; the plural read
as "the instruction file", which is what
instructionFile() builds
agent() → experimental_agent() the shape is not settled, and the tag was
never there (git log -S empty)
SkillSpec.result: → postcondition: `result` already meant the subagent output
contract one screen down; the doc comment
had to spend a line disambiguating them
The alias window is not politeness. Measured: the previous rename shipped with
ZERO overlap (18.1.1 old names only, 19.0.0 new only). A container rollback put
the old node_modules under new sources, PreToolUse stopped loading, and a
PreToolUse that does not load denies EVERY Bash command — including the one that
fixes it. One window per major makes that pair unrepresentable.
SUBPATH RETIRED. vigiles/experimental is gone; its members moved to the door
their feature already uses. Of two mechanisms for one promise the subpath was the
weaker: it marks the import line, out of view by the time anyone reads the call.
Measured on the only user-facing example — with the prefix aliased away at
import, the marker survived at 0 of 5 call sites. So the prefix stays, the advice
to alias it away became an explicit prohibition, and the six define* entry points
are DECLARED prefixed rather than gaining it at re-export.
That de-aliasing is what unblocked moving the naming check into ESLint, and the
ORDER was load-bearing: measured before it, dropping the public/internal
exemption produced 14 findings, all from the re-export indirection. Measured
after, across all of src/ with no exemption: 22 tagged declarations, 0
unprefixed. The rule turns on silent.
EXPORT CLEANUP. vigiles/linting export * → curated, 103 → 79 symbols; its header
claimed to be curated with an `export *` one line below, and claimed the builders
were also at the package root (191 exports there, zero matches). ClaudeTool and
HookEvent deleted — zero references, and both had drifted off the dialect they
claimed to mirror; HookEvent listed 5 events, two of which do not exist, against
the 31 the vendor documents.
DOCS. experimental.md deleted and distributed per feature: the skill half into
skills.md §Status, the hook half into a new compiled-hooks.md §Status, and the
emit half — 111 of its 177 lines, a topic doc wearing a stability page's clothes
— into docs/emit-channel.md, which gains the "which channel when" comparison it
never had. The one cross-cutting sentence went to the README.
TWO GATES WERE GREEN AND WRONG, both found by this work:
- api:check reported "verified for 11 entries — no drift" while reading
dist/experimental.d.ts from a PREVIOUS build (tsc does not clean dist/) for a
source file that no longer existed. Now asserted against package.json
exports, with a mutation proving exit 1 on a phantom entry.
- check-export-prefixes caught MY error: the R3 services first landed on
vigiles/eval, where `paid_` means "this call can bill you" — and
startServices takes a ContainerRuntime and calls no model. Naming it
paid_experimental_startServices would have redefined paid_ as "expensive in
some sense". They went to the package root.
SEVENTEEN review findings landed on this branch, all real. Two mechanical
closures came out of them: src/example-imports.test.ts resolves every relative
import under examples/ instead of matching a string, and the naming rule now
ENUMERATES the three export paths it visits rather than describing its coverage
in the abstract. One attempt is recorded as REFUTED: teaching the doc gate to
flag our own exports as free variables measured 66 findings across 93 blocks, so
that gate's original scoping was right.
BREAKING CHANGE: vigiles/experimental is removed. Its exports are unchanged in
name and behaviour and are now reached from `vigiles` (the emit channel, the R3
disposable-service tier). Every renamed symbol keeps a working @deprecated alias
for this major.
|
||
|
zernie
|
ec455214c7 |
fix(hook-install): recompiling appended a duplicate beside a hand-wired hook (#168)
Measured in a consumer repo 2026-08-21: `vigiles compile` on twelve hooks
produced twelve DUPLICATE wirings, because `managesHook` could not see that the
existing entry pointed at the same file.
The existing entries were wired the way Claude Code's own docs recommend:
node "$CLAUDE_PROJECT_DIR/node_modules/vigiles/dist/cli.js" \
hook-runtime run-program "$CLAUDE_PROJECT_DIR/.claude/hooks/x.hook.ts"
`managesHook` splits the command on whitespace and canonicalises each token, so
it compared the literal `"$CLAUDE_PROJECT_DIR/.claude/hooks/x.hook.ts"` —
quotes included — against `.claude/hooks/x.hook.ts`. Two independent reasons it
could never match, and they compound:
- QUOTES are not an exotic spelling. A path containing a space MUST be quoted,
so the quoted form is the correct one and was unmatchable.
- `$CLAUDE_PROJECT_DIR` IS the project root — exactly what `normalizeHookRef`
resolves relative paths against — so the two spellings name the same file by
definition, not by guess.
The duplicate is the worse of the two copies: it spells the path relative to the
cwd, so it exits 2 the moment the agent runs from a subdirectory. The healthy
copy beside it keeps working, which is why this went a full day unnoticed —
the harness looked fine until someone `cd`'d.
Only `$CLAUDE_PROJECT_DIR` is stripped. Any other variable names an unknown
location (a sibling checkout, a plugin dir), and treating it as the root would
DELETE a hook pointing somewhere else.
Mutation-proven, each with evidence the patch landed:
| mutation | fails on |
| --- | --- |
| stop stripping quotes | the hand-wired replace case |
| stop stripping $CLAUDE_PROJECT_DIR | the hand-wired replace case |
| strip ANY $VAR/ | "a hook under a DIFFERENT variable" |
⚠️ Two mutations I ran first were WORTHLESS and are recorded so nobody repeats
them: a `command.includes(ref)` over-match passed green (a full-path ref makes
`includes` nearly as strict as `===`, so the mutation changes no behaviour), and
a hand-written regex mutation produced a syntax error, which reads as "no tests"
rather than as a failure. A green mutation is a finding about the TEST, not a
verdict about the code.
Gates, in order: `npm run check` 18/18 · `npx vitest run`.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
7b8cdecd5e |
feat!: doc-refs is opt-in, because it measured 0 true positives (#167)
* fix(loader): a surface that exists on disk was never read, and the report named it anyway
The loader read the repo-root `skills/` **or** the project-level
`.claude/skills/` — never both — and materialized whichever it picked under the
SAME canonical `<materializeRoot>/<surface>/…` key. Two real files, one key: the
loser was never opened, and the winner's bytes sat under the loser's name.
The vendor settles which one the harness loads: BOTH, in two namespaces.
> Plugin skills use a `plugin-name:skill-name` namespace, so they can't
> conflict with other levels. For example, `my-plugin/skills/deploy/SKILL.md`
> becomes `/my-plugin:deploy` and loads alongside a `deploy` skill in your
> project's `.claude/skills/`.
> — https://code.claude.com/docs/en/skills § "Where skills live"
So "pick one" was never a tie-break to get right; it was a question with no
answer, asked because the key shape forced one. The fix removes the question:
every scope present is read, and a scope's key prefix is DERIVED FROM THE SCOPE,
so two files can no longer claim one key — there is no ordering, no
"if already taken", and no last-write-wins left to get wrong.
Measured (36 SHA-pinned public repos, 2026-08-18):
* 573 → 493 surface files that exist on disk and are never opened.
The 80 recovered are the shadowed ones: nyldn/claude-octopus 71 → 0,
hyhmrright/brooks-lint 8 → 0, thedotmack/claude-mem 1 → 0.
* The 493 that remain are a DIFFERENT, pre-existing gap, unchanged by this
commit: directory symlinks under `skills/`, which `entryOf` deliberately
does not descend (the ELOOP containment) but Claude Code does follow.
* nyldn/claude-octopus ships each skill twice — 61 under `skills/`, 57 under
`.claude/skills/`, 50 names in both, and ALL 50 pairs differ. One pair is
vendored as a true-positive fixture.
* vigiles' own repo: 6 skills → 19, lethal-trifecta units 4 → 12. Its own
audit was blind to two thirds of its own machine.
* Grades moved on 1 of 36: brooks-lint C(70) → B(82). NO finding was deleted
(6 before, 6 after) and the COUNT of dangerous units rose (6 → 7); the
Safety ring is a ratio, and this moves its denominator from 6 to the real
12. See the report for the full justification.
Also fixed, because reading both scopes caused it: a name no longer identifies a
surface, so `description-overlap` would have printed `skills "dup" and "dup"` —
50 times on claude-octopus. Overlap labels now carry the real path.
SUBTRACTS: the `root` | `user` | `none` variants of `SurfaceSource` (the either/or
is gone, so "we picked the wrong one" is unrepresentable), the duplicated
precedence rules in `scan-files.ts` (the decision now lives once, IO-free), and
one net lint warning in the two touched files (6 → 4).
Gates: npx vitest run · npm run lint · npx prettier --check . · npm run api:check
· npm run build — all clean, in that order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
* chore: regenerate the browser-parity fixture for the two-scope disambiguation
The overlap finding now names each skill by PATH, not by bare name — which is
the point of the two-scope work: with root skills/ and .claude/skills both
present, a bare name does not identify a surface. Intentional engine change,
regenerated per the parity test's own instruction.
* chore: regenerate generated.d.ts for the merged tree
Единственный конфликт мёрджа был одной строкой в СГЕНЕРИРОВАННОМ файле — счётчик файлов
проекта, 388 против 387. Разрешено не выбором стороны, а регенерацией: у слитого дерева
счёт 389, то есть ОБЕ стороны были неверны для него. Выбор любой из них дал бы зелёный
мёрдж с неправдой в артефакте.
* fix: four defects found by dogfooding vigiles on a real consumer repo
Все четыре найдены за один день догфуда на `zernie/mine` (43 скилла, 48 харнесс-файлов),
каждый — с замером, а не с рассуждением.
1. `exclude` не доходил до прохода по markdown-рефам
`findDocRefs({ basePath })` вызывался без `ignore`, хотя поле `exclude` документировано
ровно под этот случай («vendored or benchmark fixtures the repo's own lint shouldn't
police»). Замер: 9 битых рефов, 8 из них внутри каталога, явно перечисленного в
`exclude`, и все 9 репортились. Следствие дороже причины: выйти в 0 стало нечем, поэтому
шаг пометили `continue-on-error: true` — а lint, чей код возврата выбрасывают, не гейтит
ничего, и рядом выросли самодельные CI-шаги, чтобы гейтить вместо него. Один непереданный
аргумент — вся эта цепочка. После починки: 9 → 0.
2. Рефы искались ТЕКСТОМ, а не разбором
`/\b(enforce|file|cmd|ref)\(\s*["']…/` — четыре из шести входов классифицировались
неверно (замер 19.08): `ctx.file("OUT")` (метод чужого объекта — живой ложный
срабатыв в реальной репе), `obj.cmd(...)`, `// cmd(...)` в комментарии,
`'cmd("x")'` внутри строки. `\b` спокойно стоит после точки, а у регулярки вообще нет
понятия комментария и строкового литерала. Разбор делает все четыре НЕВЫРАЗИМЫМИ: AST
отдаёт только вызов, у которого callee — голый идентификатор. `typescript` и так
runtime-зависимость пакета — парсер лежал в коробке всё это время.
3. Раннер скриптов был строгой очередью
`files.forEach(... spawnSync ...)` со `stdio: "inherit"`. Замер на 48 харнессах:
270s → 77s (3.5×), результат ПОБАЙТОВО тот же (84 passed, 1 with 0 checks, 7 failed).
Пул по умолчанию выводится из `entry`, а не из нового флага: `test` не передаёт entry и
разъезжается по ядрам (файлы бесплатны и детерминированы по построению — мок-модель, без
API-ключа), `eval` передаёт и остаётся на 1, потому что параллель там — это одновременные
оплачиваемые вызовы модели. Вывод буферизуется на ребёнка: с `inherit` 48 отчётов
шинкуются друг о друга, и это настоящая цена параллели, а не сам цикл.
4. Путь агента в находке брался по ИМЕНИ
`new Map(agents.map(a => [a.name, a.path]))` оставляет последнюю запись, поэтому при
`agents/foo.md` и `.claude/agents/foo.md` (разные неймспейсы у Claude) находка о первом
репортилась против файла второго. Находка lethal-trifecta, указывающая не на тот файл,
хуже отсутствующей: она отправляет читателя проверять невиновный код. До полной
scope-qualified идентичности (правка крупнее, `finding.name` несёт только голое имя)
неоднозначное имя даёт ПУСТОЙ путь вместо произвольного.
Плюс: все 10 джобов в 6 воркфлоу этого репозитория были без `timeout-minutes` — то есть
зависший джоб шёл до шестичасового потолка GitHub. Бонды ~3-4× ЗАМЕРЕННОЙ длительности
собственных прогонов репы, замер процитирован над каждой строкой.
Тесты: два новых на пул, обе половины. ⚠️ Первая версия фикстуры считала маркеры, которые
только СОЗДАВАЛИСЬ, — то есть «сколько стартовало всего», а это достигает N и при строго
последовательном прогоне; тест проходил по причине, не связанной с параллелью. Исправлено
освобождением слота на выходе. Обе мутации (форсировать serial · распараллелить eval)
валят каждая свой ассерт, каждая с доказательством, что патч лёг.
Гейты: `npm run lint` 0 errors · `npx prettier --check .` чисто · `npm run api:check` без
дрейфа · `npm run build` · `npx vitest run` — падает только пре-существующий
`scripts/release-preset.test.ts` (не находит `@semantic-release/commit-analyzer`, объявленный
в package.json, но не установленный в этом окружении; на CI со свежей установкой зелено).
* docs: record what each of the four defects was, in the pages that own them
comparison.md — что СЧИТАЕТСЯ рефом теперь (вызов, у которого callee — голый идентификатор,
а первый аргумент строковый литерал) и почему три формы больше не рефы: метод чужого объекта,
упоминание в комментарии, строка с текстом вызова. Плюс область: проход читает `**/*.md` и
теперь уважает `exclude`, чего не делал — из-за чего репозиторий с вендоренной чужой
разметкой не мог получить чистый lint вообще.
docs/rules/delegation-trifecta.md — новая секция про два агента с одним именем: находка
теперь показывается БЕЗ пути, а не с произвольно выбранным из двух. Явно сказано, почему
это лучше: неверный путь отправляет читателя проверять невиновный файл.
docs/cli.md — почему `test` гоняет несколько скриптов разом, а `eval` по одному, и что
это следует из яруса, а не из флага. С замером 270s → 77s при побайтово одинаковом
результате, и с ценой: вывод буферизуется, живого стрима больше нет.
Все три — то, что было НАЙДЕНО, а не только что изменилось: через месяц полезно именно это.
* fix(docs): mark two illustrative enforce() blocks the old regex could not see
Разбор нашёл на два рефа больше, чем регулярка, и оба оказались СЛОМАННЫМИ — годами,
незаметно: `enforce("rubocop/Custom/NoDirectDbQuery")` и `enforce("pylint/no-direct-db-query")`
в справочных страницах скилла. Это примеры того, как объявить СВОЁ правило, то есть имена
заведомо несуществующие — им место под `<!-- vigiles:ignore -->`.
🔴 Почему регулярка их не видела, и это отдельный класс: она применялась ПОСТРОЧНО
(`for (const {lineNo, text} of blockLines) text.matchAll(CALL_RE)`), поэтому вызов,
разнесённый на несколько строк — а оба именно такие — не совпадал никогда. То есть у прежнего
детектора были не только ложные срабатывания (метод чужого объекта, комментарий, строка), но и
ложные ПРОПУСКИ на самой обычной форматировке. Разбор снимает оба класса разом: у AST нет
понятия «строка файла».
Итог на этом репозитории: 21 реф → 25 (после двух маркеров), все валидны.
* fix(compile): quote frontmatter scalars YAML would otherwise mis-read
`description: ${spec.description}` подставлялся сырым, поэтому описание с двоеточием и
пробелом — «…ловит ошибки, где замер врёт: неаналоги в выборке» — давало frontmatter,
который НЕ является валидным YAML. Замер 19.08 на двух реальных скиллах: сразу после
`vigiles compile` метод `skillContract()` отдаёт `malformed: true` и `declared: []`. Файл
ВЫГЛЯДИТ объявляющим `allowed-tools`, строгий парсер не читает ничего, скилл молча
наследует все инструменты сессии.
🔴 Это ровно тот дефект, который правило `frontmatter-valid` существует чтобы репортить —
то есть благословлённый путь производил то, что продукт ищет. Хуже: он НАКАЗЫВАЛ за
починку. Человек, закавычивший значение руками, получал на следующем линте расхождение
хеша («manually edited after compilation»), а перекомпиляция молча откатывала правку.
Проверка — КРУГОВОЙ ПРОГОН, а не список опасных символов: эмитим, читаем обратно тем же
загрузчиком, которым пользуется линтер, и квотим, только если вернулось не то, что ушло.
Два следствия этого выбора:
· уже безопасное значение эмитится побайтово как раньше — ни один скомпилированный файл не
меняется, ни один интегрити-хеш не двигается (подтверждено: все 1303 core-теста зелены);
· множество «опасных» входов не надо перечислять и поддерживать — решает сам YAML, поэтому
`#`, `[`, `&`, `*`, ведущий пробел, `yes`, `null` покрыты, не будучи названными.
Применено и к субагентам: описание там пишется так же свободно и ломает frontmatter так же.
Тесты — обе половины, и обе несущие (мутации с доказательством, что патч лёг):
· вернуть сырую подстановку → падает «compile quotes a description YAML would mis-read»
· квотить всегда → падает «leaves an already-safe description bare — no hash churn»
Вторая не менее важна первой: избыточное квотирование переписало бы каждый файл в каждой
репе и сдвинуло каждый хеш, что читается как «vigiles переписал мне весь репозиторий».
* feat!: doc-refs is opt-in, because it measured 0 true positives
The markdown code-block ref pass ran unconditionally and could not be turned
off. Measured across two real repositories on 2026-08-19 — 2 582 markdown
files, 52 builder refs — it produced ZERO true positives, and every error it
had ever raised was false: design prose sketching an API that does not exist
yet (`cmd("npm test")` for a package with no such script), and a third-party
CLAUDE.md captured verbatim as benchmark data.
The cause is structural rather than a threshold miss. A fenced block in prose
is a DRAWING of config, and this pass read it as config. No calibration fixes
that, because the block is not wrong — it is not config.
The consumer repo could only reach a passing lint by listing whole directories
in `exclude`, silencing every other check over a third of the tree; measured
afterwards the pass walked 604 files and found 0 refs — inert, still paying for
the walk. Off by default replaces that blunt instrument with an honest default.
BREAKING CHANGE: the markdown code-block ref pass no longer runs by default.
Enable it with {"rules": {"doc-refs": "error"}} — appropriate where markdown
genuinely is the source (a docs site whose fences are copied into live specs).
At "warn" findings print marked ℹ and the exit code is untouched; at "error"
they gate with exit 2 (they previously contributed exit 1, the soft tier,
because the pass fired unasked).
Known gap recorded for whoever raises the default again: the walker globs
without `dot`, so .claude/** — where deployed skills and agents live — has
never been scanned. The rule has never once judged a real instruction file.
* test(doc-refs): the GitHub annotation follows the configured severity
The inline-annotation test drove the pass with no config, which no longer runs
it. Extended to all three tiers rather than just re-enabled: "error" annotates
::error, "warn" annotates ::warning, and absent annotates nothing at all. The
middle case is the one worth asserting — a ::error rendered next to an exit code
of 0 is how an advisory rule gets read as a gate.
* chore: regenerate generated.d.ts, and stop two docs lines claiming doc-refs is on
CI caught what my five local gates could not: `npm run check` runs 18 stages
and I ran 5 of them. The new test file was missing from the generated project-
file union — 17/18, failing on `generated-types:committed`.
Also two sentences in docs/comparison.md still described the pass as
unconditional: the section's opening claim ("vigiles validates … calls inside
fenced TS/JS code blocks") and the Scope paragraph. Both now say opt-in; the
reader hits the opening line four paragraphs before the new note explaining the
default.
* ci(site): stop the browser install from eating the job
The site job was cancelled at its 15-minute bond twice in a row (runs
32300064395 and 32303343213), both times inside `playwright install --with-deps
chromium`, and neither failure was in the diff under review. The log shows the
cause plainly: `--with-deps` shells out to apt, apt hung on
azure.archive.ubuntu.com (`Ign:` on every line, then twelve minutes of silence),
and the orphan reaped at the end is `npm exec playwright install --with-deps`.
Three changes, one per cause:
- Drop `--with-deps`. ubuntu-latest already ships the libraries Chromium needs;
the flag exists for bare containers. Removing it removes the only part of this
step that touches a distro mirror. If a library really is missing, the browser
fails to LAUNCH in the next step within seconds and names the missing .so —
loud, and far better than a silent 15-minute hang.
- Cache ~/.cache/ms-playwright, keyed on the lockfiles. The step's floor was a
full ~150 MB download on every run, cache or no cache, good day or bad.
- Bound the step at 5 minutes. The job bond cannot tell "this step wedged" from
"the suite is slow", so a wedge cost the whole job; now it costs five minutes
and names the step that did it.
Measured baseline for the job: 3m35s.
* ci: run a job only when the commit can reach its inputs
This workflow had no path filter at all, so every PR paid for all five jobs
whatever it touched. Measured on this very PR (16 files across src/ and docs/,
none under site/): the site job built the engine, downloaded a browser and ran
the demo suite against code the diff could not reach — twice, at ~15 minutes
each, because the browser install wedged.
A workflow-level `paths:` cannot express this. It decides whether to run the
workflow AT ALL and says nothing about individual jobs. So the decision moves
into a `changes` job: one API call, no checkout, ~10s, feeding an `if:` on the
jobs whose inputs are disjoint from what changed.
Two flags, each conservative in the direction that costs minutes rather than
correctness:
root false ONLY when every changed file is under site/ — nothing outside
site/ imports anything inside it. Gates test, harness, e2e.
site false ONLY when every changed file is prose or agent config
(docs/, .claude/, root *.md). Gates site.
`check` stays unconditional: it runs types:site and cli-lint, so both halves of
the repo are among its inputs.
The site flag is a measurement, not an assumption. The demo imports the built
engine through three aliases, so any src/ change can reach it and must run it.
What cannot: grepping the three bundles it actually loads for the lint rule table
(DEFAULT_RULES, frontmatter-valid, doc-refs) returns zero in all three, and its
check pages are generated from site/src/checks/checks.ts rather than from docs/.
Tested in src/ci-path-filter.test.ts, which EXTRACTS the patterns from the
workflow rather than restating them — a test that agrees with its own copy of a
rule proves nothing about the rule that runs. A skipped job and a passed job look
identical in the checks list, so both directions are asserted. Three mutations,
each failing on its own assertion with the patch verified in place:
prose arm widened to any *.md → 'a markdown file NESTED in src'
prose arm widened to include src/ → 'runs everything for a normal src change'
empty-list bail flipped to false → 'an empty diff is not a licence to skip'
* test(ci): a job added later cannot silently escape the path gate
The filter itself was tested; the SHAPE of the workflow was not. In the checks
list a skipped job and a passed job are the same tick, so a job added tomorrow
with no `if:` looks exactly like a correctly-gated one that happened to run —
the filter quietly stops meaning anything and nothing goes red.
Exemptions are named with their reason rather than inferred, so granting one is
a visible edit: `changes` is the classifier itself (gating it on itself is a
cycle), `check` runs types:site and cli-lint so both halves of the repo are its
inputs. Plus a vacuity guard — if every job ends up exempt the assertion can no
longer fail, and an empty list would read as health.
Mutation, patch verified in place: stripping the gate off `site` fails with
"job(s) run on every commit with no path gate: site".
Mirrors the same guard added to mine's paper-gates.harness.mjs, where an ungated
19m42s job was what prompted it.
Gates, in order: npx vitest run (8 in this file) · npm run check 18/18.
* Revert "test(ci): a job added later cannot silently escape the path gate"
This reverts commit
|
||
|
zernie
|
6532dbf48d |
fix: five defects found dogfooding vigiles on a real consumer repo, plus CI job bounds (#162)
Five defects found dogfooding vigiles on a real consumer repo (zernie/mine — 43 skills,
48 harness files), each measured rather than reasoned about:
1. `exclude` never reached the markdown-refs pass. Measured: 9 broken refs, 8 of them
inside a directory explicitly listed in `exclude`, all 9 still reported. After: 9 → 0.
The consequence was larger than the cause — with no way to reach exit 0 the consumer
set `continue-on-error: true`, and a lint whose exit code is discarded gates nothing.
2. Refs were found by TEXT, per line. Four of six probe inputs classified wrongly
(`ctx.file("OUT")`, `obj.cmd(...)`, a comment, a string), and because the regex ran
per line, any call split across lines was invisible — which had hidden two genuinely
broken refs in this repo's own docs. Now parsed: a ref is a call whose callee is a
bare identifier with a string-literal first argument. `typescript` was already a
runtime dependency. Net: 21 refs → 25, all valid.
3. The script runner was a strict queue. Measured on 48 harness files: 270s → 77s
(3.5x), byte-identical results. The default follows `entry` rather than a new flag:
`test` fans out (free and deterministic by construction — mock model, no API key),
`eval` stays at 1 (real model quota). `stdio: "inherit"` was the actual blocker.
4. Agent path attribution was keyed by name, so with `agents/foo.md` and
`.claude/agents/foo.md` a lethal-trifecta finding was displayed against the wrong
file. Ambiguous names now yield no path instead of an arbitrary one.
5. `compile` emitted invalid YAML. A description containing `: ` produced frontmatter
where `skillContract()` reports `malformed: true, declared: []` — the blessed path
producing the exact defect `frontmatter-valid` exists to report, and punishing anyone
who fixed it by hand with a hash mismatch. Fixed by round-tripping through the same
loader the linter uses, so already-safe values stay byte-identical and no integrity
hash moves.
Plus: all 10 jobs across 6 workflows were unbounded and now carry `timeout-minutes`
sized to measured durations; docs updated in the pages that own each defect.
Gates: vitest 3186 passed - lint 0 errors - prettier clean - api:check no drift - build.
CI: all 7 jobs green.
|
||
|
zernie
|
04225f9980 |
feat: an errored emit is not an emission, and a contract field can be an enum (#165)
Two locked design items (vigiles/ideas/23), plus the public docs brought level with
what has actually been measured.
1. A CALL THAT ERRORED IS NOT AN EMISSION. Measured on the shipped build: a
permission-denied call carrying a valid payload returned {"kind":"ok"} — the reader
filtered by name and never looked at ToolCall.isError, a field that was on the type
all along. The server never received that call. This is the exact class of lie the
emit channel exists to remove from the fenced rail, reproduced inside the channel.
Errored calls now get their own branch, because all three collapses lie in a
different direction: counting them reports success for a call nobody received;
dropping them silently says "no tool call", sending the reader to the skill's
instructions when the fault is in permissions; lumping them with "called twice"
gives a true signal under a false name when a model retries a denial. A successful
call alongside a denied one is one successful emission — the contract was met.
Four tests, each naming its collapse. Mutation-checked: removing the split fails
three of the four.
2. ENUM, the one extension to OutputFieldType. A readonly tuple of literals declares
a field whose value must be one of them; it renders as a JSON-Schema enum in the
tool definition and as "CUT" | "MERGE" in the fenced rail, so both deliveries show
a worker the same permitted values. The measurement behind it: a verdict field
declared as `string` held three mutually incomparable invented categories over three
runs, with 3/19 = 16% vocabulary compliance. `string` cannot express "one of these".
Nothing relational follows it — no object[], no tuples. On 14 real payloads, 23
scalar values carried every assertion anyone made while 80,981 characters of prose
carried none, so the body stays prose by decision rather than by omission.
3. DOCS. docs/experimental.md gains the outcome table — every way an emission can fail
to satisfy the contract, with the reason text — which was the one thing a reader
could not find. Measurements updated to the breadth study (15/15 on sonnet; a zero
on haiku that was the skill stopping early, not the channel dropping); the serving
question is marked answered; the prefix-removal conditions now say which of the
three is met. Also states plainly what the channel does NOT check: whether the
emitted result is true. Gating on an emitted verdict was measured and rejected —
self-report disagreed with a judged check in 45-75.8% of runs.
Gates: npx vitest run, npm run lint, npx prettier --check ., npm run api:check,
npm run build — via the check script, 18 of 18. The api-surface diff is the intended
widening of OutputFieldType and EmitFieldSchema, reviewed and committed.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
4f0a01b2f9 |
CLI clarity, an execution-tier finding, and the Action's boundary documented (#163)
Five changes from a measured review of the CLI surface and of a consumer's CI. 1. BEHAVIOUR — a positional target that matches nothing now FAILS. Bare discovery finding nothing still exits 0: an empty repo is a legitimate place to stand, but a path or glob you NAMED that matched nothing is stale, wrong, or never reached. Collapsing the two cost a real repo three days of green CI verifying zero files. Jest and Vitest fail on no-tests-found and make you opt in with --passWithNoTests; pytest exits 5. --min=0 is the opt-out here, so no flag is added. 2. HELP — the four checkers (audit/lint/test/eval) all read as "check my stuff". They are now grouped by READ vs RUN, and each line states its own consequence positively. The success criterion was checkable and is met: "NOT a CI step — use vigiles lint in CI" is deleted, because nothing needs defending. A help line that says what a command ISN'T is a description that failed. Per-verb flag detail moves into the per-verb help; audit's four wrapped lines naming nine flags were most of the felt crowding. Verb count is unchanged at 8 — the smallest of every tool measured (vitest 8 verbs and 164 flags, cargo 48, docker 57, git 166), and none of them thinned their surface by removing verbs. 3. DOCBLOCK — src/cli.ts listed four of the eight commands, having quietly stopped growing. Replaced by a pointer to the single source; a new test asserts every verb appears in the RENDERED help exactly once, because a constant is what drifted. 4. AUDIT — new Tested finding: "N surfaces counted as covered by PLACEMENT only — no run on record ever exercised one". Colocated evidence is a claim about the filesystem and the provenance line already admits it; nobody reads a provenance line. Measured in a consumer: 26 colocated and 0 executed, while its CI skipped every one of those harnesses and reported success. The corpus-wide threshold is a stated limit in the code, not a silence. 5. DOCS + SKILL — the Action reference now explains why there is no test input (the Action never runs npm ci; harnesses import the library), and the test-harness skill gained a six-line CI pointer: an Action exists, init wires it, inputs are documented there. It REFERENCES rather than explains, because copied explanations rot — three measured proofs, one of them inside this product. Gates, in order: vitest, lint, prettier, api:check, build — via the check script, 18 of 18. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
ad413b0ac5 |
fix(scan): the fix-scan work, merged onto current main (#161)
* fix(scan): two ways a surface that is there was invisible to audit
A marketplace listing suppressed the audit of the directory it sat in, and
`agents/` was read one level deep though the harness reads it recursively.
Both end the same way: vigiles reported nothing, at exit 0, about surfaces
that were on disk — and silence is this tool's success signal, so neither
failure could be seen from the outside.
MARKETPLACE (`audit`, cli.ts). A `marketplace.json` whose members are all
external expanded to zero targets, and a branch printed "nothing to scan"
and returned BEFORE looking at the directory. But that file is a statement
about OTHER directories: a repo may list external members and also be a
plugin. Measured on nyldn/claude-octopus @57cfb9b0 (3,979 stars): 59 skills,
50 subagents, 40 hooks, 54 commands — 203 surfaces — suppressed, exit 0.
The branch is deleted; the `targets` fallback that already existed does the
work. Proof the suppression is gone: deleting the marketplace.json now
yields the byte-identical report, so the file no longer changes the outcome.
A genuinely curated marketplace reaches the existing `score.empty` branch,
which carries the "clone a member plugin" advice and exits 2 — this repo's
own rule that 2 means "I could not do what you asked". `--json` still emits
`kind:"marketplace"` there, for exactly the case it describes.
AGENTS RECURSION (core/layout.ts + its three readers). The vendor documents
`agents/` as scanned recursively, with the subfolder forming the identifier
(`agents/review/security.md` -> `review:security`). vigiles matched
`agents/[^/]+\.md` — in THREE places that each spelled the rule themselves:
the scan classifier and both coverage discoverers. Measured on
rsmdt/the-startup @88d447c7: 16 agent files, 2 read, and the plugin was
still graded B (80/100) over that 12.5%. The rule now has one home,
AGENT_FILE_LEAF_RE, quoted by all three, so the subagent count and the
untested count can no longer be computed over different sets of files.
Agent names become the documented scoped identifier. That is not cosmetic:
a basename cannot be unique once a dir is read recursively, and the
delegation graph keys agents BY NAME, so `agents/a/review.md` and
`agents/b/review.md` would have swallowed each other. Top-level agents keep
their bare filename, which is why no existing report moves.
Two defects the recursion itself introduced, both closed: a `SKILL.md` under
`agents/` matched both classifiers and was counted twice (the harness loads
it as an agent, so it is excluded from skills — the mirror of the existing
skills/<x>/agents/ rule); and the on-disk discoverer normalized separators
for the pattern but not for the name, which on Windows would have silently
fallen back to a basename.
Folded in: a nonexistent path now says it does not exist rather than "no
surface was found there" — one existsSync, its own test.
GRADES MOVE, in the harsher direction, on 2 of the 32 dogfood repos (the
other 30 are byte-identical):
nyldn/claude-octopus suppressed, no grade -> D (60/100), 203 surfaces
rsmdt/the-startup member `team`: B (80) -> D (60), 2 -> 16 agents,
and 2 -> 14 malformed-frontmatter findings
No exit code moved anywhere in the corpus. bench/leaderboard/RESULTS.md is
unaffected: both its pinned repos have zero nested agents (measured).
Tests: 10 new cases across scan/parity/layout/e2e, each proven by mutation
with the patch asserted to have landed. The parity fixture is load-bearing —
reverting only the browser-side discoverer survives the entire pre-existing
suite and dies only on the new nested-agent parity case.
Gates: vitest 2991 passed - lint 0 errors - prettier clean - api:check no
drift - build ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
The same two bytes as on main. Both sides of a merge must be text before git
will do a three-way merge at all — normalising only one side leaves the blob
pair binary.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
34a4f7a556 |
fix: no raw NUL bytes in source, and a gate that keeps it that way (#160)
src/scan-files.test.ts carried two literal NUL bytes inside a template literal where the escape was meant. Git decides "binary" per blob by looking for a NUL, and a binary blob gets no line diff and NO THREE-WAY MERGE — so every branch touching the file conflicted, unresolvably, because there were no hunks to pick. Measured: with the bytes escaped, `main <- fix-scan` and `main <- fix-shadow` each merge with ZERO conflicts. They had been parked for a day as "conflicting with each other"; they never were. The escape is the form the rest of the codebase already uses (skill-contract, scan-core, coverage-artifact, coverage-probe — eight sites), and the string it produces is byte-identical, so scan-files.test.ts still passes its 21 tests. Adds scripts/check-text-sources.mjs as an 18th `check` gate. It refuses to pass on an empty scan, since a scan finding nothing is indistinguishable from a scan looking in the wrong place. Proven by mutation: planting one raw NUL fails it with the file, the count and the offset. Not a .gitattributes marker: that restores diff and merge while leaving the byte in the source, hiding the defect instead of removing it. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
4acfd31fc9 |
feat(eval)!: an eval file describes, the runner performs — importing one no longer spends (#159)
* fix(audit): coverage caveats printed by lint never reached the audit report `formatUntestedReport` grew two qualifier lines — a test file still carrying the retired `vigiles:covers` marker, and a run record older than the text it measured. `audit` assembles its own fact block from the SAME UntestedReport and carried neither, so the note that exists to explain a silent migration was itself silent for anyone whose habit is `audit` rather than `lint`. Measured before the fix on a fixture whose only harness carries the marker: lint → "1 test file(s) still carry the retired `vigiles:covers` marker …" audit → "Untested surfaces: 0" and nothing else The caveats now come from one exported builder (`coverageCaveats`) that both renderers read, so a third qualifier reaches both callers or neither — a renderer can no longer carry half the list. `ScanReport.coverageCaveats` is absent rather than `[]` when empty, because the browser twin structurally cannot compute these (no `.vigiles/coverage.json` without a filesystem) and byte-parity between the two engines is asserted field-for-field. Both halves tested at the product level (`node dist/cli.js audit`): it fires on a repo carrying the marker and stays silent on one that does not. Proven by two mutations, each with the patched source grepped first: dropping the render line fails the "fires" assertion; emitting a caveat unconditionally fails the "silent" one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the suffix that stopped counting now says so `*.test.*` left DEFAULT_TEST_GLOBS in 15.x because a default vitest/jest run collects exactly those names and a skill test spawns a model — so a misnamed harness was being executed, and paid for, by somebody else's runner. The removal was announced to nobody. Its own comment argues "the migration is a rename, and the untested finding prints the exact path", but the path printed is the SURFACE's, together with a suggestion to add `<surface>.eval.mjs`. An author with `foo.test.mjs` already lying beside `foo/SKILL.md` is therefore told to write the test they wrote, with neither the file nor the reason named. The sibling half of the same migration — the retired `vigiles:covers` marker — got its explanatory line at the time. This is the half that did not. Scoped to surfaces that are otherwise UNCOVERED: that is the only position where the count contradicts what the author can see in the directory. A stray `*.test.*` beside a properly covered surface is somebody's ordinary unit test and none of our business — asserted, because an advisory line that fires on the innocent case gets muted within a day. Nothing about coverage changes: the name still grants nothing. Measured silent on all seven real corpora available here (this repo, six vendored dogfood plugins) and on the `mine` knowledge base; it fires on a planted fixture in both `lint` and `audit`, the latter through the shared caveat list added in the previous commit rather than a second wiring. Three mutations, each with the patched source grepped before the run and each dying on its own assertion: dropping the note from the caveat list · widening the scope from untested to every considered surface · dropping the suffix test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * refactor(scan): compute the caveat list once Cosmetic follow-up to the two commits above: `coverageCaveats(coverage)` was called twice in the same object literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(coverage): the retired-suffix note claimed a cost the guard already closes The note said a `*.test.*` harness "spawns a model" under someone else's runner. `refuseUnderForeignRunner` guards every spawn door and refuses under vitest/jest — measured here against the detector: argv of `node_modules/vitest/vitest.mjs` yields "vigiles refused to spawn a model: this process is running under vitest." What actually happens is that the foreign run FAILS on the file, which is the true and smaller claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(eval)!: an eval file DESCRIBES its eval; `vigiles eval` runs it Importing a `*.eval.*` file spent real money. These files did their work in the module body, and in ESM `import` IS execution — so the cheapest imaginable question, "does this file even parse?", launched a paid run against a live model when it was asked with `import()`. That happened, and was paid for. The one guard that exists does not close the door and cannot be made to. Under `node -e 'import(f)'` there is no `process.argv[1]` at all, so `foreignRunner(undefined)` is `null` and `refuseUnderForeignRunner` stays silent; and no process fact distinguishes that from a legitimate runner doing the same import, so an argv-shaped guard would have to refuse the correct invocation to catch the wrong one — the exact failure that got `process.env.VITEST` rejected in `core/foreign-runner.ts`. So the shape changed instead. An eval file default-exports a description: export default defineEval({ measureTriggerRate: { …spec… }, assert: (report) => assertTriggerRate(report, { min: 0.8 }), }); `defineEval` builds a plain value; there is no call in it to run. The five paid runners are reached only by `dist/eval-entry.js`, the program `vigiles eval` now spawns per file — not a new verb, the interpreter the existing verb uses. `defineEval` is exported from the FREE root barrel, not `vigiles/eval`, and that is the point: after this change no eval file imports a runner at all. Three loud edges, because a silent no-op would be the same defect in a new place: `node x.eval.mjs` refuses and names `vigiles eval`; a file that declares nothing (or two measurements) is reported by the runner instead of exiting green; and a half-migrated file — a description plus a leftover top-level `measure(…)` — hits `refuseDuringEvalLoad` during the runner's own import and throws a migration message instead of billing. `node --check` still works and is what the free syntax check should be. Subtractions: every eval file loses its `process.env.VIGILES_TRIALS || process.argv[2]` line, its `console.log(format…Report(report))`, its `if (report.n === 0) throw`, its top-level `await`, and its `vigiles/eval` import. The runner does all five, once. BREAKING CHANGE: `*.eval.*` files must now default-export `defineEval({…})` instead of calling a runner at the top level, and `node <file>.eval.mjs` no longer runs an eval. To migrate: move the runner call into a key named after that runner (`runEval` · `measure` · `measureArms` · `measureTriggerRate` · `measureSelectionMatrix` — declare exactly one), move anything you did with the report into `assert(report)`, and delete the trials line, the `format…` print and the `report.n === 0` check — `vigiles eval` does those. A capability probe that sat at the top of the file becomes `skipIf()`; `measureSelectionMatrix`'s directory argument becomes a `pluginDir` field; and `measureTriggerRate(spec, { evalDriver })`'s second argument becomes a top-level `evalDriver` key. `vigiles eval --trials=N` still overrides trials. Run with `vigiles eval <file>`; a file left on the old shape fails with a message naming docs/harness-testing.md, never silently. All 19 eval files in this repository were migrated in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture and generated types The eval-definition change rewrote one report hint (`paid_measureTriggerRate` → `export default defineEval({ measureTriggerRate: … })`), which is exactly the divergence the parity test is built to catch. Regenerated per that test's own instruction for an intentional engine change; the diff is that one line — grade and every other field are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
effed2afcc |
fix(vocab): the validator knew 9 of 31 hook events, and the other 21 passed by failing open (#158)
* fix(dialect): vocabulary is a status per name, not membership in two lists Three checks answered "does this string look like something I recognise?" instead of "is this string valid?", and decided what to SAY about an unrecognised name by its EDIT DISTANCE to a hand-kept list. Measured against the vendor docs (claude-code 2.1.233, read 2026-08-17): - `Setup`, a documented hook event, was reported as "a hook here never fires" and told to become `Stop` — a different event with different semantics. It was caught only because `setup` sits 2 edits from `stop`. - The catalog held 9 of the vendor's 31 events. The other 22 were not "accepted"; 21 were unrecognised and >2 away from anything, so nothing was said. `SubagentStart` and `PostCompact` sat at 3 — one character from becoming the next `Setup`. - `Agent` was on the never-available denylist while its own deprecated alias `Task` sat in the built-in catalog. The vendor renamed it the other way in 2.1.63 and ships `tools: Agent, Read, Bash` as a worked example, so vigiles told delegating subagents to drop the one tool they exist to use. - `MultiEdit`, `BashOutput`, `KillBash`, `LS` appear zero times across six vendor doc pages. They were in the catalog; four real tools were not. So a valid name got an accusation or silence by spelling luck. The fix is not a longer array: `classify` returns one of four verdicts — available / withheld / conditional / unrecognised — and no near-match is in scope where the report-or-skip decision is made. `unrecognised` is the third option between silence and accusation: an advisory that names vigiles's own capture as the possibly-stale party, surfaced and never scored. The ≤2 threshold was not merely loose, it was set at the width of the collision: over 465 event pairs and 741 tool pairs, ZERO are 1 edit apart and exactly one pair in each is 2 apart — one of them being `Setup`/`Stop`. So distance 1 is scored as a typo and distance 2 stays advisory, and a test pins that measurement so a future catalog entry landing 1 edit from an existing name fails here rather than in someone's repo. Subtracts `confidentToolIssues` / `confidentHookEventIssues`: severity now travels on the issue, so scan, lint and compileAgent cannot disagree about which findings count. `builtinAgentTools` / `neverAvailableTools` / `sideEffectingTools` become projections of one authored vocabulary, and the conformance kit rejects a dialect that contradicts itself — the invariant whose absence let `Agent` sit in two lists while dialect-drift read it out of the vendor's own sdk-tools.d.ts every run. Also fixes an unrelated defect found on the way: the dialect-drift hook-event test never applied the stale-leftover reconciliation its sibling applies, so it only ever ran against whatever old package was lying around. Grades on faithful reductions that reproduce the recorded before-state exactly: SethGammon/Citadel F(57) -> C(73), thedotmack/claude-mem F(55) -> C(70). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(site): keep the browser engine's graph node-free; retire LS in the type test collectVocabularyNotes lived in scan.ts and scan-files.ts imported it as a VALUE, which pulled the node-only graph into the browser bundle: dist/scan-files.js → dist/scan.js → core/mcp.js → core/refs.js → core/symbols.js → @ast-grep/napi (a native .node binding esbuild has no loader for) Measured by walking the built CJS require graph, and confirmed by mutation: restoring the scan.js import makes the chain above reappear, removing it makes @ast-grep/napi unreachable. The functions are pure data transforms and now sit in scan-core.ts, the module whose stated job is to be node-free. test/types/purity.ts still declared LS, which left the catalog on 2026-08-17. It is now a negative case instead: a retired name has to FAIL at tsc time, since its absence from a list proves nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore: regenerate the browser-parity fixture for the vocabulary message The one changed line is the finding this PR exists to fix: the old rationale dumped the whole catalog at the reader — including MultiEdit, BashOutput, KillBash and LS, which are not Claude Code tools — instead of saying what is wrong with the name it was given. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7afa00f00a |
fix(refs): 17 false "file is missing" accusations, none of them true (#157)
* fix(scan): the reference extractor accused files that were present
Three false-accusation defects found by a 32-repo OSS dogfood sweep, all the
same shape — vigiles telling a maintainer a file they have is missing. For a
tool whose pitch is deterministic precision, a false positive costs more than a
miss: it is the failure mode that gets the tool uninstalled.
Measured on the corpus (91 audit units): 74 reference findings before, 57
after. All 17 removed are verified false; zero true positives lost, zero new
findings anywhere. Eight units change grade, every one upward — rsmdt/the-startup
F(54)→C(70), ayghri/i-have-adhd B(85)→A(100), fcakyon fable-advisor B(82)→A(90),
jjanczur/tyran D(64)→C(72), and Truthfulness 47→100 / 25→100 on two Microsoft
plugins.
One extractor was being run over five grammars. It now reads three, each through
that grammar's parser, and deliberately stops reading the other two:
markdown markdownRefs() in core/markdown.ts — a link's DESTINATION, and code
spans that are NOT inside link text. The old line-scanner reported a
link's LABEL as a missing resource while its destination resolved.
shell commandWords() in core/bash-effects.ts — the words a shell would
resolve, with inline program text (`node -e`, `python -c`) removed
and a nested `sh -c` parsed rather than discarded. The old regex cut
a character run out of a JavaScript payload and reported it as the
hook's script name.
source core/source-refs.ts — the one place a path-token pattern is built,
with both boundaries un-omittable. `js` could claim the head of
`json`; `(?:agents)/` could claim the tail of `claude-agents/`. Both
were live.
NOT READ: frontmatter (already stripped) and full-line comments in every
language, not only shell — both remaining corpus false positives here came out
of JSDoc.
Subtracted: four hand-written extension alternations (two of them duplicated
across the disk detector and its browser twin), a hand-rolled markdown link
regex, a hand-rolled inline-code regex, a whitespace split standing in for shell
words, and `stripShellComments`.
No new CLI verb, no new flag, no public API change (`api:check`: no drift).
Tests: 10 mutations, each proven applied by grep before running and each killed
by its own assertion, across the whole relevant suite. Every new case carries
both halves — fires on the planted defect, silent on the clean shape beside it.
* fix(scan): drop the redundant start-of-content guard in isPluginRooted
`startsAtSeparator` already answers idx <= 0 with true, and the next line
reaches it for idx === 0 (content[-1] is undefined, so never "/"). The guard
was a second copy of a boundary rule this module deliberately does not own —
the same duplication the module comment warns about — and it was the one
statement the 100% gate could not reach.
Also regenerates .vigiles/generated.d.ts for the new source files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
41b4da49a2 |
fix(coverage): the runner executed colocated tests and discarded the result (#156)
* fix(coverage): a colocated test that RAN is execution evidence, not a directory listing
Coverage answered "tested?" with two tiers: a recorded run (`executed`) and a
file sitting beside the surface (`colocated`). The second printed its own
caveat — "this says the file EXISTS, not that it ran" — and on a 52-surface
consumer repo it carried 23 of the 27 covered surfaces. Those 23 are ordinary
`*.harness.mjs` that `vigiles test` runs on every push. The run happened, the
runner watched it happen, and the metric described it as `ls` output, because a
harness asserting through `node:assert` reports a CHECK COUNT and no probe —
and `runsFromResults` discarded any run whose probe list was empty.
Measured on that corpus, before → after:
27 covered = 4 MEASURED BY A RUN + 23 colocated
27 covered = 27 MEASURED BY A RUN + 0 colocated
Coverage TOTAL is unchanged and untested stays 30: nothing gains or loses
credit, the evidence behind it moves. Zero attributions were downgraded.
The bar is a REPORTED CHECK, not exit zero. `touch <skill>/<skill>.harness.mjs`
— zero bytes — already drops the untested count by one, and `vigiles test` then
runs it and prints a tick, because an empty script exits 0. Attributing on
"ran and exited 0" would promote that file to "MEASURED BY A RUN": the same
emptiness wearing the stronger label. An empty script cannot report a check, so
it keeps its colocated credit and gains nothing — `statusFor`'s existing
undefined/0 distinction reused, not a new one.
`colocated` is a fourth RunAttribution and deliberately NOT a ProbeOrigin.
Origins are the wire vocabulary `parseCheckReport` reads out of a file the
CHILD writes; putting it there would let a harness hand-write
`{"how":"colocated"}` and mint coverage by declaring it — `vigiles:covers`
returning through the back door. Off `ProbeOrigin`, a script cannot spell it.
Colocation is the FALLBACK attribution, not an additional one. Emitting both
downgraded records: `mergeRuns` dedupes on (surface, tier, script) WITHOUT
`how`, so the second attribution replaces the first — measured, it turned
`.claude/hooks/paper-lint.mjs` from `command` into `colocated` and took three
`fired` activations with it.
SUBTRACTS a mirror: `isColocated` was written twice, and the copy's own
docstring said "Mirror of test-coverage.ts `isColocated`". One body
(`isColocatedTest`), three callers — disk detector, browser twin, run-record
builder. Colocation and execution were answering the same question with two
different pieces of code, which is why a script vigiles had just RUN could not
be attributed by the rule already willing to credit it for existing.
Also corrects a stale claim in coverage-probe.ts's header: it advertised the
in-process tier as recording nothing, which `harness-assert.ts` closed in
August. A header that advertises a gap somebody already closed sends the next
reader to build a second recorder.
Tests: 12 new, each with both halves (fires on a planted defect, silent on the
clean corpus), proven by 8 mutations each grepped for application first —
including one that came back green and exposed that the artifact's `how`
validation was pinned by nothing at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: regenerate generated.d.ts for the new colocated-execution test
The CI gate generated-types:committed compares the checked-in file against a
fresh generation; adding a source file without regenerating leaves them apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
51d439200a |
feat(spec): a skill could not declare the one key that actually removes a tool (#155)
* fix(compile): integrity header displaced a skill's frontmatter, so the harness lost its contract `addHash` prepended `<!-- vigiles:sha256:… -->` to every compiled file. For a SKILL.md that pushes the YAML frontmatter out of first position, and a reader anchored at `^---` then finds NO frontmatter at all — no name, no description, no allowed-tools. 🔴 THE INVARIANT WAS ALREADY WRITTEN DOWN, IN THE MODULE THE FIX LANDS IN. `ejectMarkdown` in integrity.ts has documented since it was written that a compiled SKILL.md "begins with YAML frontmatter (`---`) that MUST stay in first position — prepending an HTML comment there would push the frontmatter out of the lead block and the harness would lose the skill's name/description/tools". It honours that. `addHash`, two modules over, did the opposite to the same files. MEASURED, not reasoned. Five real skills from a live corpus (145–256 lines) were translated to minimal specs and compiled. The header was the ENTIRE delta: body byte-identical section order untouched diff 5 added lines, 1–2 removed — identical on all five regardless of size frontmatter original: found by a `^---` parser · compiled: NOT FOUND So one misplaced line was the whole reason a compiled skill could not be adopted. Everything else people feared about adoption (reordered sections, a generated `## Arguments`) turned out to be opt-in and absent unless `input()` is used. THE REAL DEFECT WAS DUPLICATION, and that is what this removes. Four sites independently encoded "the header is the first line": compile.ts HASH_RE, integrity.ts HASH_LINE_RE, coverage.ts, and cli.ts `targetHasHash`. Position now lives in ONE place — `findIntegrityHeader` / `placeIntegrityHeader` in integrity.ts — and the other three ask it. `targetHasHash` was the dangerous one: it decides whether a file is ours and safe to overwrite. Reading only the first line, it would have called every compiled skill hand-written prose and refused to update it. BACKWARD COMPATIBLE BY CONSTRUCTION. Every SKILL.md compiled before today carries the header first; such a file has no leading frontmatter (the header displaced it), so the same function finds it at the top. Without this, upgrading would report every previously-compiled file as hand-edited. Pinned by a test. Tests: 6 new cases in integrity.test.ts — placement with and without frontmatter, round-trip, tamper detection, the OLD layout, and plain markdown. Mutation-proved: reverting `placeIntegrityHeader` to prepend fails exactly the placement case (1 failed / 14 passed) and restoring passes 15/15. ⚠️ One defect was caught by these tests BEFORE shipping, not by review: the header is written a blank line below the closing `---`, and the finder anchored at position 0 of the body missed it. The file looked correct and `checkIntegrity` failed on it. The `\s*` in HASH_LINE_IN_BODY_RE is what makes it hold, and it is commented as load-bearing. Gates, in order: npx vitest run · npm run lint · npx prettier --check . · npm run api:check · npm run build — via `npm run check`, 17/17 in 77s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(spec): a spec value that was an object shipped as "[object Object]" with no error Second defect from the same 2026-08-17 measurement. Calling `input({ name, description })` — the object form a reader reasonably guesses — compiled with NO error and wrote this into the SKILL.md that would have been shipped: argument-hint: <[object Object]> - `$1` **[object Object]** — undefined WHY TYPES DID NOT CATCH IT, which is the part worth knowing. `input(name: string, hint: string)` is typed correctly. But `vigiles compile` loads a `.spec.ts` through tsx in a child process: transpiled, types erased, never checked. vigiles's OWN specs are cross-checked by a separate `tsc --noEmit` over a generated registry — and that safety net does not travel to the people compiling their own specs. So for every user, the type is documentation. Two guards, one concept each. 1. `input()` refuses a non-string name or hint and names the real signature. "expected string" does not tell the caller what to write instead, so the message carries the call shape. 2. `addHash` refuses any compiled body containing "[object Object]". This is the LAST GATE before a compiled file is written — all four compile paths return through it — so one check covers the whole family (file/cmd/ref/dir/glob/symbol/guidance are all unguarded the same way, and this catches them without eight new checks). Hard error, not a warning: measured zero occurrences of that string across every `.md` in this repository, so there are no false positives to trade against, and a file carrying it is broken in a way its author cannot see by reading the spec. Tests: 5 new cases. Mutation-proved, both with proof the patch landed — neutralising the `[object Object]` check fails exactly its case (1 failed / 19 passed); neutralising the `input()` check fails exactly its two (2 failed / 18 passed); restoring passes 20/20. Gates, in order: npx vitest run · npm run lint · npx prettier --check . · npm run api:check · npm run build — via `npm run check`, 17/17 in 76s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(adopt): the header move missed a caller, and a round-trip grew a stamp each pass Three tests went red on the placement change. Two were stale assertions; the third was a real defect in the fix itself, and only enumerating every caller found it. `splitFrontmatterBody` in adopt.ts consumes the leading frontmatter with a regex that tolerates an HTML comment BEFORE the block — precisely the old placement. Once the stamp moved below the frontmatter it landed inside what that function calls "the body", so adopt → compile → re-adopt carried the stamp into the spec's body, and every further pass added another one. Fixed by delegating to findIntegrityHeader, which is the one site that knows where the header can be — the reason it exists, and this was the caller the move missed. Both of its callers are covered by the change. The two stale assertions are rewritten rather than relocated. They anchored `/^<!-- vigiles:sha256/`, which is the OLD contract; simply moving the anchor would have restored a test that could not see the defect either way, because `name: reviewer` appears in the file under both placements. They now assert through readFrontmatter — a reader must FIND the block and read name and description out of it — which is the property the whole change is about. railway.test.ts is untouched on purpose: its fixture has no frontmatter, so the stamp is legitimately first there, and it now covers the other half of the contract. Also corrects a comment in frontmatter-read.ts that described the pre-move layout as current. Its optional-leading-comment branch is backward compatibility for files compiled before today, not what the compiler emits. Gates, in order: npx vitest run (2964 passed, 0 failures; scripts/release-preset fails to import @semantic-release/commit-analyzer, which is not a dependency — same local-only gap noted in #153) · npm run lint (0 errors, 206 pre-existing warnings) · npx prettier --check . (clean) · npm run api:check (11 entries, no drift) · npm run build (ok). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat(spec): a skill could not declare the ONE key that actually removes a tool `SkillSpec` had `tools` and no fence. `AgentSpec` had both. So adopting a skill into a spec meant LOSING the ability to declare `disallowed-tools:` — and that is the only key measured to take a tool away: `allowed-tools:` is a pre-approval, it waives the prompt for what it lists and removes nothing from the pool. The result was that `audit` reported every compiled skill as holding all three lethal-trifecta legs, and the spec offered no way to answer it. The tool asked for a fence its own authoring surface could not express. 🔴 The key is HYPHENATED for skills. A subagent's is `disallowedTools:` (camelCase) and a different parser reads it, so emitting the agent spelling on a skill produces a key nothing looks at — inert, in the direction that reads as protection. That is the same class as the #107 defect two lines up, where `tools:` on a skill silently lost the contract, so the test asserts the exact bytes rather than a loose match. Writer and reader verified end to end, not assumed: `parseAgentToolList` was run against the actual compiled output and returns ["Bash","WebFetch"] for the emitted flow style, and null when the key is absent (the "no contract" signal the rail honors). A fence that compiles but does not parse would have been worse than none. Entries are typo-checked like a subagent's: "Wrte" removes nothing while the file reads as though Write were blocked, so it is an error, not a fence. Both new tests proved by mutation, each dying on its own assertion: emit `disallowedTools:` instead of `disallowed-tools:` → the key test fails disable the validation block → the typo test fails Gates in order: npx vitest run (2966 passed; scripts/release-preset fails to import @semantic-release/commit-analyzer, not a dependency — the known local-only gap) · npm run lint (0 errors, 209 pre-existing warnings) · npx prettier --check . (clean) · npm run api:check (11 entries, no drift after api:report — the surface diff is exactly the one new field in two reports) · npm run build (ok). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
4fef3779fa |
fix(compile): integrity header displaced a skill's frontmatter — the harness lost its contract (#154)
The header was prepended to EVERY compiled file, which for a SKILL.md pushes the YAML frontmatter out of first position — a reader anchored at `^---` then finds no frontmatter at all: no name, no description, no allowed-tools. The invariant was already written down in the very module this lands in; `addHash`, two modules over, did exactly the prohibited thing to exactly those files. Measured on five real skills (145-256 lines): the header was the ENTIRE delta. Body byte-identical, section order untouched, 5 added / 1-2 removed on all five regardless of size. So one misplaced line was the whole reason a compiled skill could not be adopted. The real defect was duplication — four sites independently encoded "the header is the first line". They now share one resolver in core/integrity.ts, which is the only place that knows where the header can be, and it reads both placements so files compiled before today still verify. Enumerating the callers found a fifth site the move had missed: splitFrontmatterBody in adopt.ts consumed a comment BEFORE the frontmatter only, so adopt -> compile -> re-adopt carried the stamp into the spec's body and grew another one each pass. Two stale assertions were rewritten, not relocated: they anchored the old contract, and simply moving the anchor would have restored a test blind to the defect either way, since `name: reviewer` appears under both placements. They now assert through readFrontmatter — a reader must FIND the block and read the fields out of it. Also: a spec value that was an object shipped as "[object Object]" with no error. Merged with `describe` red on an unrelated GitHub GraphQL 503; every substantive check (test, check, harness, site, e2e, validate) is green. |
||
|
zernie
|
1ebef8320a |
ci: one list of commands instead of two — CI now calls npm run check (#151)
* ci: один список команд вместо двух — CI зовёт `npm run check` Сегодня три PR уехали в `main` красными: список гейтов гонялся по памяти (пять команд), а джоб `check` гонял тринадцать. Упавшая — `tsc -p test/types/tsconfig.json` — живёт в ОТДЕЛЬНОМ tsconfig, которого корневой тайпчек не видит, и три `@ts-expect-error`-ассерта тихо перестали срабатывать. Длинный список, который надо помнить, это не чинит. Чинит то, что список СТАЛ ОДИН и CI его ВЫЗЫВАЕТ: `scripts/check.mjs` + `run: npm run check` вместо тринадцати шагов. Это стандартный паттерн поля (pre-commit / just / mise): определение задач — источник правды, CI его дёргает. Прежняя моя идея «завести гейт на совпадение двух списков» снята: она была подпоркой под дубль, который правильнее убрать. Стадии несущие, а не косметика. Часть команд ПИШЕТ файлы, которые другие ЧИТАЮТ (`generate types` → `.vigiles/generated.d.ts`, `api-extractor` → `api-surface/*`, `build` → `dist/`), поэтому писатели сериализованы, а параллельно идёт только read-only группа. 17/17 за 70 с против ~3 мин последовательно. Имена падающих шагов не потеряны — раннер печатает, что именно упало, и `--list` показывает команды без запуска. 🔴 Два шага `uses: ./` НЕ входят в `npm run check` намеренно: они догфудят собственный composite Action, у которого нет shell-эквивалента. Один из них (`eval-check`) при первой правке был снесён случайно — он стоял между двумя command-шагами, и правка по диапазону его проглотила. Возвращён, причина записана прямо над ним. Раннер сразу поймал два собственных промаха: неотформатированный `check.mjs` и протухший `generated.d.ts` после добавления скрипта `check`. Проверено: `npm run check` 17/17 ✓ · ci.yml парсится. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: словарь скилла уехал на сам билдер — experimental_skill.input/.step BREAKING: `input` и `step` больше не самостоятельные экспорты. Достаются как `experimental_skill.input(…)` / `.step(…)`. Пользователей нет. Зачем. Префикс `experimental_` — привычка, а привычка уже один раз протекла: `skill()` отгрузился со стабильным именем при доке «`skill()` is experimental». На самих `input`/`step` тега и префикса не было вообще, то есть словарь экспериментальной фичи выглядел стабильным. Повесив их на билдер, маркировку делаем СТРУКТУРНОЙ: до `input()` не добраться, не назвав `experimental_skill`. Забрано ровно два символа, и это замер, а не вкус. Перепроверено независимо от вердикта субагента: `input` и `step` встречаются только в скилл-спеках, а `cmd` (5 агентских спек), `file`, `ref`, `result` (8 агентских) общие — их спрятать в неймспейс скилла нельзя. 🔴 По дороге закрыта асимметрия, которую правка иначе бы УХУДШИЛА: дверь `vigiles/claude-code` своих `input`/`step` не экспортировала никогда, и типизированный автор брал их из второй двери. После снятия standalone- экспортов он остался бы без них совсем. Теперь CC-билдер несёт те же хелперы. Форма — `Object.assign`, не `export namespace`: последний запрещён нашим же линтом (`no-namespace: error`). Генерик пережил: в api-репорте `typeof skillSpec & { input; step }`, `ae-forgotten-export` нет. Честная граница записана и в коде, и в доке: `const { input } = experimental_skill` снимает маркер внутри файла. Гарантия у́же — немаркированное имя не пересекает границу пакета. Доки: `docs/skills.md` — пример на алиасе (`experimental_skill as skill`), маркер держится на строке импорта. `docs/experimental.md` — вступление утверждало, что префикс носит всё из `vigiles/experimental`, что после переезда `experimental_skill` в `vigiles/spec` стало неверным; плюс своя секция. `docs/README.md` — доиндексированы `testing-matrix.md` и `non-js-harnesses.md`. `safety.md` намеренно НЕ индексирован: он сам пишет «INTERNAL / WIP — not surfaced in the public docs yet». Гейты: npm run check 17/17 ✓ · vitest 2954 ✓. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b9302db4e6 |
feat!: one instability marker — the experimental_ name prefix, enforced (#147)
* docs: генераторный режим убран из публичной доки и припаркован `skill()` помечен экспериментальным. Из `docs/skills.md` убраны §3 (генератор) и секция тестов `vigiles/skill-test`, которая на нём стоит: `genSkill`/`act`/`checkpoint`/`finish`/`runSkill` не достижимы ни из одного сабпаса пакета — то есть снаружи репозитория такой скилл не написать. Ровно эти два фантомных сабпаса и заставляли держать в шапке рукописный баннер; баннер снят вместе с ними. Замерено при этом, и записано в §Status как причины экспериментального статуса: - `vigiles compile` генераторы ПОДДЕРЖИВАЕТ (`cli.ts` ловит `genSkill(`, прогнано end-to-end) — прежняя строчка §Status «wiring it into `vigiles compile` … is the next integration step» устарела; - скомпилированный `SKILL.md` ни разу не был показан загружающимся как скилл: `vigiles:sha256:`-шапка отодвигает YAML-фронтматтер с первой строки, и читатель, привязанный к `^---`, не находит фронтматтера вовсе. Все работающие `SKILL.md`, включая 13 своих, начинаются с `---`; шапку несут только два файла в `examples/`, которые никто не грузит; - принятие всё-или-ничего: `renderSkillSections` собирает документ целиком в фиксированном порядке, пути «моя проза + один проверяемый гейт» нет. Паркинг помечен в шапках трёх модулей, чтобы его не подняли молча. Публично экспортировать нечего: замер показал ноль достижимых символов. `scriptModel` НЕ паркуется — он публичен из `vigiles/claude-code` независимо от генератора. Заодно — дефект гейта из #146, найденный этой же правкой: он переиспользовал маркер `vigiles:ignore-file`, который принадлежит резолверу ссылок. Из-за этого единственный документ с двумя доказуемо битыми импортами был единственным, которого гейт не видел (91 блок сканировался, этот — нет). У гейта теперь свой маркер `vigiles:ignore-imports`; на дореформенном тексте он выдаёт ровно те два TS2307, ради которых писался баннер, на текущем корпусе молчит (92 блока, 0 находок). Тест на разделение маркеров прогнан мутацией: возврат общего маркера роняет оба ассерта. Гейты: vitest 2928 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · check-doc-imports 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: skill() → experimental_skill(), и правило, которое это держит BREAKING: `skill()` переименован в `experimental_skill()` на `vigiles/spec`, `vigiles`, `vigiles/linting` и `vigiles/claude-code`. Пользователей нет, мажор бампаем. Почему: дока открывалась словами «`skill()` is experimental», а имя на экспорте говорило «стабильный». Проза против механики — ровно тот дефект, который vigiles и ищет, у vigiles. Соглашение о префиксе существовало (`experimental_emitTool`, `experimental_parseEmitted`, `experimental_withServices`), нигде не было записано и ничем не проверялось. Побочно рейм снимает КОЛЛИЗИЮ, которой не должно было быть: `skill()` экспортировался из ВОСЬМИ сабпасов и был ДВУМЯ разными функциями — билдером спеки (`vigiles/spec`) и `Check<Trace>` по трассе (`vigiles/testing`, принимает id-строку, отвечает «сработал ли скилл»). Одно слово, два понятия, различимые только дверью импорта. Префикс несёт только билдер; `Check` не тронут (`vigiles-testing.api.md` без изменений). Правило: `scripts/check-experimental-naming.mjs` + шаг CI + `npm run experimental:check`. Публичное объявление с TSDoc-тегом `@experimental` обязано начинаться с `experimental_`. Ни одного нового маркера: тег уже используется в `experimental-emit.ts` и `services.ts`, публичность берётся из закоммиченных `api-surface/*.api.md`, которые `api:check` и так держит честными. Типы вне области: на дореформенной поверхности префикс несли только функции, а `ServiceSpec`/`ServiceHandle`/`ContainerRuntime` — нет; с типами гейт открылся бы 6 косметическими переименованиями против 1 настоящей находки и был бы заглушен в тот же день. Настоящая находка одна и закрыта: `makeDockerRuntime` → `experimental_makeDockerRuntime`. 8 тестов гейта, обе половины у каждого. Один нашёл баг в самом гейте: `\s+\S` в маркере-исключении удовлетворялся переводом строки и звёздочкой продолжения JSDoc, то есть голый маркер без причины проходил. Поправка к предыдущему коммиту: §Status утверждал, что sha256-шапка мешает загрузке скилла. Замер это ОПРОВЕРГАЕТ для соседней поверхности — в `reviewer-ab.eval.mjs` живой Claude Code загрузил скомпилированный `agents/code-reviewer.md` С шапкой через `--plugin-dir` и отдиспатчил в него, 100% прогонов на реальном sonnet (2026-06-20). Формулировка сужена до «непроверено», а не «сломано». Гейты: vitest 2936 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал устаревший импорт в docs/railway-subagents.md — правка внесена) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * docs: STABILITY.md объявлял стабильным то, что переименовано и помечено экспериментальным Найдено субагентом при разборе неймспейс-идеи, подтверждено мной. `STABILITY.md` — документ под заголовком «What's stable — depend on it» — перечислял `skill` среди стабильных билдеров `vigiles/spec`. После переименования такого символа нет вообще, а то, во что он переименован, в предыдущем коммите объявлено экспериментальным. То есть контракт стабильности обещал стабильность несуществующему имени. Третий экземпляр «проза против механики» в этой же области за два дня, и мой гейт его НЕ ловит: он сверяет TSDoc-тег с именем, а не документ с API. Заодно записано то, что вскрылось при проверке: - `input()` и `step()` используются ТОЛЬКО скилл-спеками, но экспортируются без префикса и без тега — то есть словарь экспериментальной фичи выглядит стабильным. Слепая зона гейта: он энфорсит «тег ⇒ имя», но не «на всём, что должно, стоит тег». - Маркеров нестабильности в проекте ТРИ и правила выбора между ними нет: `@internal` (39 объявлений), `@experimental` (18), префикс `experimental_` (8 публичных символов). Секция больше не утверждает, что маркер один. - У `pipe` тег `@internal` стоит только на ПЕРВОЙ перегрузке, поэтому остальные в api-репорте читаются как `@public (undocumented)`. Гейты: prettier ✓ · docs:check 0 · experimental:check 0 · lint 0 errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: один маркер нестабильности вместо трёх — префикс имени BREAKING: `pipe` `pipeStep` `needs` `start` `andThen` `effect` → `experimental_*` на `vigiles` и `vigiles/linting`. Пользователей нет. ## Выбрано **Экспортируем и не стабильно — имя начинается с `experimental_`.** Тег `@experimental` объявляет это один раз, `experimental:check` роняет CI, если у помеченного публичного символа нет префикса — тег и имя разъехаться не могут. `@internal` перестал означать «нестабильно»: он отвечает на ДРУГОЙ вопрос — входит ли символ в API вообще — и верен только на том, что мы не экспортируем. Почему имя, а не тег: тег не виден там, где решают. Имя видно в каждом вызове, дифе, ревью и автокомплите. Ровно так протёк `skill()` — стабильное имя при доке «`skill()` is experimental». ## Исправлено всё Было три маркера без правила выбора: `@internal` 39 объявлений, `@experimental` 18, префикс 8 публичных символов. Найдено 6 противоречий — символы с `@internal`, которые ЭКСПОРТИРУЮТСЯ; все шесть переименованы, теги переведены на `@experimental`. ## Два дефекта в гейте, отгруженном час назад 1. 🔴 Тег читался только если стоял на строке ОДИН (`@experimental\s*$`). Реальные теги несут прозу — `@internal Experimental typed-composition surface — …` у `pipe`, два в `services.ts`. Замер: **2 из 8** `@experimental` и **31 из 39** `@internal` объявлений были гейту не видны. Строгий паттерн выглядел безопаснее и был наоборот. 2. Между JSDoc и объявлением может стоять комментарий (`eslint-disable` перед перегрузками `pipe`) — привязка к следующему символу его теряла. После починки гейт видит 14 объявлений вместо 6. ## Записано `STABILITY.md` — секция «The one rule» + список; `CLAUDE.md.spec.ts` → перекомпилированный `CLAUDE.md` (правило №51). Плюс честно записанная дыра: гейт энфорсит «помечено ⇒ названо», но НЕ «всё, что должно быть помечено, помечено» — живой пример `input()`/`step()`, словарь только скилл-спеков, без тега и префикса. ## Третий раз за сессию Фикстура-спека в шаблонной строке (`generate-harness.test.ts:505`) не видна ни tsc, ни грепу по импорту — ловится только прогоном тестов. До этого то же в `cli.test.ts` (дважды). 5 новых тестов гейта (13 всего), включая закрепление обоих дефектов выше. Гейты: vitest 2941 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал `docs/cli.md:646`) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
a2af365682 |
docs: четыре утверждения о собственной публичной поверхности были ложными (#144)
Все четыре проверены командой, а не чтением, и каждое обещало читателю то, чего пакет не делает. 1. ШЕСТЬ SUBPATH, КОТОРЫЕ ПАКЕТ РЕКЛАМИРУЕТ САМ СЕБЕ И НЕ ОБСЛУЖИВАЕТ. `vigiles/eval`, `vigiles/run-hook`, `vigiles/harness-test`, `vigiles/harness-assert`, `vigiles/skill`, `vigiles/skill-test` — каждый даёт ERR_PACKAGE_PATH_NOT_EXPORTED. Они стояли в комментариях одиннадцати файлов examples/, в докстринге `harness-assert.ts` как готовый пример импорта, и в `e2e.ts`, который сам объясняет депрекейт и по дороге называет несуществующий путь. Заменены на существующие. 2. `src/testing.ts` УТВЕРЖДАЛ, ЧТО ЭТИ ПУТИ «still work». Обещание, что импорт продолжает работать, — ровно то, что читатель не станет перепроверять. 3. `vigiles/unit` ОБЕЩАЕТ «no `claude`, no model, no bubblewrap, no network». Из четырёх верны три: `judged` re-экспортируется отсюда, а его судья по умолчанию — реальный вызов модели (`check.ts`: `opts.judge ?? runJudge(o)`). Тест, импортирующий ТОЛЬКО из этого барреля, может тратить деньги, и единственная строка, по которой читатель заключил бы обратное, — эта. Честная починка — увезти `judged` с барреля, но это правка публичной поверхности, а поверхность сейчас пересматривается отдельно (замеренное предложение режет по СТОИМОСТИ, и там дефект становится невыразимым, а не задокументированным). До тех пор утверждение исправлено, а не оставлено. 4. `docs/skills.md` УЧИТ ИМПОРТИРОВАТЬ ТО, ЧЕГО НЕТ НИГДЕ. `genSkill` / `act` / `checkpoint` / `finish` / `runSkill` не экспортируются НИ ИЗ ОДНОГО subpath — проверено перебором всех. Генераторный API компилируется из исходника командой `vigiles compile`, и точки входа ему не завели. Форма скилла в документе описана верно; строки `import` — намерение, а не рабочий код, и теперь документ говорит это сам. Плюс `site/index.html`: alt-текст og-карточки называл показатель «Truthful refs», которого у продукта нет (`audit-score.ts`: Truthfulness · Triggering · Structure · Safety · Tested). Инструмент, который ловит расхождение прозы с механикой, показывал в собственном превью несуществующее имя своего же показателя. Гейты, поимённо и в порядке: npx vitest run (2926 passed, 173 файла) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
10e528e8db |
refactor(testing)!: изъять runMutations — она не про домен харнессов (#143)
* docs(testing): runMutations was in the reference only — the guide never mentioned it Measured after shipping it, not guessed: `runMutations` appeared in exactly ONE file, `testing-api.md`. Zero mentions in `harness-testing.md` (the task-first guide), `testing-matrix.md` and `docs/README.md`. The reference says of itself "the guide has the task-first how-to; this is the reference" — so anyone following the guide would never learn the API exists. And the runner table still opened with "Four runners" while listing five. That is the same defect the API was built to catch, produced by its own docs: prose and content disagreeing, with nothing to notice. - harness-testing.md — a routing row in "What do you want to test?" plus a full section, including the two things the reference did not say at all: WHEN NOT TO reach for it (a one-off script, an already-red test, or wanting a coverage number — mutation counts here say how many cases you wrote, not how good the suite is), and CI, where the honest answer is usually "don't": a run forks a process per case over full files, so it belongs in a hand-run pass after editing the test or the checker, or in its own job if wired at all. - testing-api.md — "Four runners" now says four for the harness plus a fifth that asks about your tests, which is what the table actually shows. - testing-matrix.md — a row for src/mutations.test.ts, naming the properties it covers, so the coverage map matches the suite. - docs/README.md — an entry under the API reference pointing at both. Gates: prettier · lint 0 errors · the 10 mutation tests still pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * revert(testing): withdraw runMutations — it does not belong in this package Shipped in #142 and 15.3.0; withdrawn a day later on the owner's question: «how is this vigiles' responsibility, it's just nodejs?» He is right, and the measurement is what settles it rather than taste. Against the rest of this package: runHook knows the Claude Code hook protocol, measureTriggerRate knows what a skill is, skillContract reads allowed-tools out of frontmatter. This module knows none of that. It replaces a substring, spawns `node`, and reads stdout — lift it into any Node repository and it works unchanged. A newcomer reading the testing docs hits exactly the owner's question, which is the cost this surface was charging. The confirming measurement, and the one I should have taken before merging: NO TEST IN THIS REPOSITORY USED IT. The only references were its own test file, its export line and its docs. The package did not dogfood the API it published. What actually fixed the problem it was extracted for was CONSOLIDATION, not RELOCATION. Ten hand-written copies of one driver had drifted apart in a consumer repository — the no-op guard in 4 of 10, the retry in 1 of 10. One shared copy buys all of that. Moving that copy into a product for harness authors was a second and separate step, and nothing justified it. The engine survives where its callers are, with the reasoning recorded in its header so it is not re-proposed. The shape that WOULD belong here is a different function that does not exist yet: mutate the HOOK — remove one branch of a guard — and require the assembled harness test to notice. Only vigiles knows what a hook is. Removed: src/mutations.ts, src/mutations.test.ts, the export block in src/testing.ts, the api-surface entry, and the docs sections in testing-api.md (from #142) plus the guide, matrix and README entries this branch had added. Gates, in order: npx vitest run (2916 passed, 172 files) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: перезапуск проверки заголовка PR Тип `revert` не входит в список, который принимает линтер заголовков (feat/fix/docs/chore/refactor/test/perf/ci/build). Заголовок переименован в `refactor(testing)!` — восклицательный знак несёт то, что `revert` нёс именем: изъятие опубликованного API ломающее. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b5da1c5604 |
feat(testing): runMutations — prove a test can fail, by breaking what it watches (#142)
A green test says the checker passed. It does not say the test would notice if the check were DELETED: an assertion can be vacuous, a fixture can be wrong in the same direction as the code, and both look exactly like a pass. `runMutations` plants a defect, runs the test that owns it, and requires that test to go red WITH THE MESSAGE THAT NAMES the defect. ## Why this belongs in the package It arrived as ten hand-written copies of one driver in a dogfooding repo (1799 lines), and the copies had drifted. Counted against the committed versions: - the NO-OP guard (a replacement equal to the original leaves a green test that proves nothing) was in 4 of 10; - the RETRY on a non-kill (a flake reported as a survivor sends the next reader to rewrite a working assertion) was in 1 of 10; - the strict rule — killed by its OWN named assertion, not merely by something going red — was in 1 of 10. Each was written AFTER it caught something, in whichever copy caught it, and never travelled. Two copies also named a test file a refactor had deleted, and the runner they used exits 0 on a path matching nothing, so those cases reported SURVIVED on every run for three days. ## Five verdicts, one of which is proof killed · wrong-assertion (a neighbour caught it) · survived · unjudgeable (the test was already red before the run, so "red" says nothing) · not-applied (find matched 0 or many, or the replacement equalled the original). The baseline matters more than it looks: without it, a test that is red ON PURPOSE — an open finding it exists to report — makes every run announce a failed restore about a working one, and makes every case against it look killed. ## What refuses rather than reports An empty case list and a named test file that does not exist both throw BEFORE any file is touched. A runner with nothing to run reports success, and that is the one claim this API exists to prevent. ## Verification vitest 2924 passed · lint 0 errors · prettier · api:check (additive only, +53 lines to vigiles-testing) · build. The new file has 10 tests, each building a real checker and a real test in a temp dir — a mocked runner would be judging a mock. Mutation-tested itself: removing the strict-kill rule, the no-op guard, or the existence check each turns the suite red on its own assertion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
7a2f2c8def |
fix(experimental-emit): unsupported field type throws; the example finds a numbered heading (#141)
Two defects found by dogfooding this surface across five real pipeline skills. 1. `fieldSchema` was a switch over four literals with NO default, so a fifth type became `undefined` — and every later step read that as a field. `"actions" in properties` is TRUE for an undefined value, so nothing noticed; JSON.stringify then DROPPED the key while `required` kept it and `additionalProperties: false` forbade it. The served schema demanded a property it also banned, silently, while `.instruction` still told the model to send it. Unreachable from TypeScript, reachable from JavaScript — and the one shipped example of this API is `.mjs`, which is why it needed a runtime test. Verified by mutation: removing the new `default:` arm turns the regression test red on its own assertion. 2. The example matched its target section with a substring, so a numbered heading was a different section to it. Measured on the corpus: 11 of 15 eligible recorders reached, 4 refused for formatting alone (`## 6. Record the verdict`). Now the heading is read as a HEADING through markdown-it — a string search has no notion of heading level and cannot tell a heading from the same words inside a fence. 15 of 15 after. Two defects in that fix, both caught by running it over the real corpus rather than an invented example: the separator class needed whitespace (`Step 8 — record` has a space before the dash), and markdown-it reads YAML frontmatter as a SETEXT h2, so every skill produced a phantom heading whose text is the description — a description containing these words would have spliced the emit into the frontmatter. Gates, in order: npx vitest run (2915 passed) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift). Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cce1936460 |
docs: экспериментальная поверхность и именованная память (#140)
* wip: branch anchor
* feat(experimental): experimental_emitTool/parseEmitted/assertEmittedOk — a typed result carried by a TOOL CALL
An unforked skill has no call→return boundary, so compile hard-errors
output-without-fork on 31 of our 38 skills. A tool call needs no return
boundary: the skill EMITS mid-conversation and the call lands in
Trace.toolCalls. Same OutputContract, different delivery.
Marked experimental in the API itself — experimental_ prefix on every
runtime export plus a JSDoc block listing, by number, what is unproven
(N=3, no compile-time consumer, transport untested, runtime does not
enforce required) and what would have to be true to drop the prefix.
No new CLI verbs or flags.
shapeError is exported from the CC agent-result parser so both rails
validate a contract with the SAME code — a second copy would drift.
* examples(experimental-emit): the runnable measurement — MCP server + runner
emit-server.mjs serves ONE tool, read from JSON that experimental_emitTool
produced; hand-copying the schema would test the copy, not the API.
run-emit.mjs takes a real skill dir, refuses one with context: fork, swaps
exactly the '## Record the verdict' section for the emit instruction, runs
it through the real CLI and reads the result back from ctx.toolCalls.
Records are appended per trial, not buffered to the end. VIGILES_EMIT_DRY
rehearses the swap for $0. Not named *.eval.mjs so a CI sweep can never
reach a paid run.
* measure: 8 runs of an UNFORKED skill, 8 emits, 8/8 parse — plus a $3 defect in isRateLimited
The measurement. paper-status (no context: fork, so it can carry no output:
contract today) emitted a typed verdict by calling the tool, and the call
lands in Trace.toolCalls next to the Skill activation that spliced the skill
in inline. 8 runs, 8 emits, all ok, none repeated; scored offline for $0 by
score-emits.mjs through the same experimental_parseEmitted a test would use.
Two independent witnesses (MCP server log + CLI transcript) agree 8/8 on
track, count, verdict and report hash.
The run also exposed a real defect: the CLI emits an informational
{"type":"rate_limit_event",...,"status":"allowed"} line on EVERY run, and
isRateLimited's /rate.?limit/i matched it because . matches _. So every eval
trial silently ran retries+1 = 4 times, and maxCostUsd only ever saw the LAST
attempt: 2 trials became 8 model runs and a $0.60 cap was crossed at roughly
$3. Separator is now [ -]? plus an explicit rate_limit_error clause; the
telemetry line is captured verbatim as a fixture and asserted both ways.
Records are redacted (the long free-text report bodies are replaced by
head+length+sha256) and the raw files are gitignored: a run replays a skill
from a private corpus into a public repo.
* measure: run the TODAY-assertion against the same 8 outputs — green 8/8 while its number is NaN 8/8
The write-up asserted that the prose regex would break on real output. Asserting
that is the failure mode the standing rules name, so it is executed instead:
/BLOCKED/ matches 8 of 8 and /(\d+)\s+blockers?/ matches 0 of 8 — the assertion is
green and the count it exists to guard is NaN every time. Only the derived booleans
are published; the model prose describes a private corpus.
(Pushed via the GitHub API: git-over-HTTPS is failing in this container because
/root/.ccr/agent-proxy-ca.crt was truncated to 2 bytes mid-session.)
* docs: раздел про record()/state() + публичная страница экспериментальной поверхности
Обе фичи отгружены без единой строки в docs/. record()/state() приехали в #135,
experimental_emitTool — на этой ветке; ни то, ни другое из публичных доков не читалось.
docs/experimental.md намеренно отдельным файлом, а не абзацем в основных гайдах:
недоказанное не должно стоять в одном ряду с semver-стабильным. На странице —
что померено (N=8, 8/8), что НЕ померено (одна модель, один скилл, нет compile-time
пути, транспорт не проверен), чего оно НЕ покупает (схему рантайм не энфорсит,
Bash не уходит — allowed-tools это пред-одобрение), что должно стать правдой,
чтобы снять префикс, и стабильная альтернатива.
Прогнано: prettier --check docs/ · npm run lint · vitest по experimental-emit ·
vigiles lint (21 ссылка в код-блоках, все валидны).
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
82d89ae4de |
fix(agent-result): закрывающая ограда обязана начинать свою строку (#139)
* spike: closing fence must own its line, so a payload can carry a code block Measured 2026-08-13 (N=28 real sonnet runs): a result() contract whose string field is asked to hold a code snippet made every such answer "malformed". The model's JSON was VALID (newlines correctly escaped) but the old pattern's non-greedy body matched up to the first closing fence it found, which sat inside the JSON string. Fences in the PROSE before the block were never the problem. Not a feature; a spike branch. Raw records + the eval that produced them live in the private KB under vigiles/repro/output-contract-2026-08-13/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * spike: correct the measured-numbers comment (5/28, exact correspondence) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
f80a6333f6 |
feat(hook): writeTargets(prefixes) — API перестаёт выбрасывать, какой файл пишется (#138)
`commandView` already builds the list of write targets and collapses it to a boolean on the way out, so every consumer that needed a FILENAME rebuilt it from the raw command string. The shipped paper guard carries a 69-line character scan with its own quote tracking for exactly that, and that scan has three documented silent bypasses (a quoted path, an absolute path, a path eaten by a wrapper). - `writeTargets(prefixes)` returns the matched targets; `prefixes` is required and there is no unfiltered form, because a consumer holding the raw list has to re-implement the prefix matching that produced every one of those bypasses. - `writesTo` becomes `writeTargets(prefixes).length > 0` — one code path, so the boolean and the list cannot drift the way `runs()` and `writesTo` did. - `NormalizedLeaf.chdir`: `stripWrappers` read the `env -C` / `sudo -D` value in order to skip past it and then discarded it, so `env -C migratsiya sed -i papers/x.tex` reported NO write under `migratsiya/papers` — measured false before this change. A leaf's own chdir wrapper now resolves. A redirection is deliberately NOT joined onto it: the shell opens `> out.txt` in its own directory before `env` ever runs. Out of scope and pinned by a characterization assertion: cross-statement cwd (`cd x && …`, `pushd`, subshells). Connectors do not survive leaf extraction and `cd x; cmd` writes into the OLD directory when the `cd` fails — a model with failure semantics, not a field on a leaf. Both halves tested for every new behaviour, and both mutations verified to land before running: returning the unfiltered list fails 6 tests; restoring `writesTo` as its own extraction fails 2, including the dedicated derivation test. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
d6801511d5 |
fix(core): .. зажимается на корне Windows, а не съедает его (#137)
`resolveRef` collapses dot segments itself (core stays dependency-free, no node:path) with an unconditional `out.pop()`. That pops through the ROOT when the root lives inside the segment list — which is exactly the Windows case: C:/../repo/src/x.ts -> repo/src/x.ts (the DRIVE LETTER eaten) //server/share/../../repo/x -> //repo/x (share, then server, eaten) Real filesystems clamp: `C:/..` is `C:/`, `/..` is `/`. The damage is silent — a path naming the repo's own file came back relative-looking, stopped resolving against `C:/repo`, and an allowlist gate/react hook simply stopped recognising it. `isStampRepairEvent` refused the write that unwedges the repo. POSIX was already correct BY ACCIDENT, not by design: its leader `/` is held OUTSIDE `out`, so popping an empty array is already the clamp. Named in a comment so nobody "simplifies" it back. The floor comes from WINDOWS_ROOT — this file's single answer to "what is a Windows root", where both forms and the `//?/` spelling are already enumerated. Consistent with the two decisions the surrounding comments record: a `//` leader is the ROOT's answer (a POSIX stutter is not a share, round 37), while a drive letter names its filesystem BY ITSELF, as `isAtOrUnder` already says of a base. That second half is load-bearing rather than theoretical: `absoluteSpelling` resolves a ROOTLESS absolute path against the literal "/", so a floor gated on the root would have left `C:/../repo/x` losing its drive at that sibling call site — the one a fix written only where the defect was found would have missed. The `windows-fs?` predicate the UNC leader used inline is now shared, so the two questions cannot disagree about the same root. Tests, both halves each: fires on the defect (drive kept, UNC clamped at the share, rootless drive, repair door) and quiet on what already worked (ordinary `a/b/../c` still collapses under every root, POSIX `/..` unchanged, `//repo` + POSIX root still collapses, UNC without `..` untouched). MUTATION (proven landed by grep before each run, whole suite not one file): - restore the unconditional `out.pop()` -> 4 failures, all mine, e.g. "`..` clamps at a DRIVE root — the drive letter is not poppable", false !== true at hook-program.test.ts:1862. - drop the root gate so the floor is read off the operand -> 1 failure, "whether a `//` leader is a SHARE is the ROOT's answer, not the path's". No pre-existing test depended on the old behaviour. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
8c34e2096c |
fix(sidecar): классифицировать записи обхода через entryOf, а не голый statSync (#136)
* fix(sidecar): classify walk entries through entryOf, not raw statSync
`walk` in src/core/sidecar.ts decided "is this a directory" with a bare
`statSync(fullPath).isDirectory()`. `statSync` FOLLOWS symlinks, so a link
inside `.vigiles/` pointing at a directory outside it was descended into,
and a cycle re-entered the same tree lap after lap.
This is the THIRD instance of the class in this repo (after the loader's
`readTree` and the deleted surface enumerator), so the fix is the canonical
one: `entryOf` from src/fs-walk.ts, which refuses a symlinked DIRECTORY,
still reads a symlinked FILE (it cannot recurse), returns "skip" for
anything unreadable — exactly what the removed try/continue did — and
terminates by construction rather than a visited-set. No new helper, no
bookkeeping, no change to entryOf.
Measured, not assumed. Reverting the call site (grep-verified that the
mutation landed before running) fails both new tests:
does not descend into a directory symlink inside .vigiles/
AssertionError: + 'linked/Foreign.md'
reports a manifest once under a symlink cycle, not once per lap
AssertionError: + 'loop/Real.md', 'loop/loop/Real.md', … (41 entries)
The cycle result corrected the write-up: unlike the loader's uncaught walk,
this one never threw. Its own `catch` swallowed the ELOOP, so the defect
showed up as ONE manifest audited 41 times rather than as a crash — quieter,
not milder. The assertion is on the list of targets for that reason; an
"it didn't throw" check stays green under the defect and proves nothing.
Quiet halves included, since a fix that skipped every symlink would pass the
firing half alone and silently shrink the audit: nested sidecars
(`.vigiles/.github/copilot-instructions.md.inputs.json`) are still found, and
a symlink to a manifest FILE is still read.
Gates, in order: npx vitest run (2884 passed, 25 skipped) · npm run lint
(0 errors) · npx prettier --check . (clean) · npm run api:check (no drift) ·
npm run build (ok).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* test(sidecar): pin the symlink policy at the walk — both halves per shape
Tests half of the previous commit, split only because the local `git push`
could not run (the sandbox's proxy CA file was truncated to 2 bytes
mid-session, so TLS to github.com failed and the branch had to be published
through the API). The resulting tree is identical to the local commit's.
FIRES on the defect (both fail when the raw statSync is put back):
does not descend into a directory symlink inside .vigiles/
AssertionError: + 'linked/Foreign.md'
reports a manifest once under a symlink cycle, not once per lap
AssertionError: + 'loop/Real.md', 'loop/loop/Real.md', … (41 entries)
QUIET on the clean case, so a fix that skipped every symlink cannot pass:
still finds nested sidecars and follows a symlinked manifest FILE
(.vigiles/.github/copilot-instructions.md.inputs.json is still found,
and a symlink to a manifest FILE is still read)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
zernie
|
7af6f21e43 |
feat(hook): runtime-owned named state — a hook can remember a fact without reaching for the filesystem (#135)
* feat(hook): runtime-owned named state — record() + state(), and the two routing defects it exposed A compiled hook could read declared context but could not remember anything, so every advisory hook in the dogfood repo stayed hand-written shell. A hook now names a fact, the runtime stores it, and any hook in the same directory declares that name in needs and reads it back with its age. Deliberately NO throttle: field — throttling is this primitive used twice, and two measurements rejected the field. See the design note in core/hook-state.ts. Also fixes two routing defects the dogfood exposed, both measured against the real runtime: a react's tool filter disagreed with the matcher the compiler emits (a family matcher was silently dropped), and a react on a tool-less event (Stop) could never fire at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * fix(hook): an empty tool list must match nothing, not the empty tool name Found by a mutation that was meant to disable tool-less reacts and did not: with no tools the joined pattern is ^()$, which matches the empty string, and a tool-less event's tool name IS the empty string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore(hook): api surface report, lint and prettier for runtime-owned state Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * chore(hook): regenerate the API surface after the rebase onto main --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
cfa2fc04af |
fix(hook-runtime): resolve hook paths against a project root, so file and command gates stop failing on the spelling the harness actually sends (#134)
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since
|
||
|
zernie
|
e782c8b0bf |
fix(coverage,hook-runtime)!: answer "is this tested?" by execution, and stop the paid tier being reachable by accident (#133)
* feat(test-coverage): deliver untested-skill as an edit-time nudge
The `untested-skill` rule was already stated correctly, but it only ran
inside `vigiles lint` — which someone has to remember to type. A rule that
fires only when invoked by hand is prose, not policy.
`skillTestNudge` reuses the SAME detector (`findUntestedSurfaces`) at the
moment a skill/agent surface is edited, and is wired into the existing
PostToolUse entry rather than adding a fourth hook or a new CLI verb.
Two things it deliberately does:
- It distinguishes "no test at all" from "has a harness but was never
EVALUATED". An edit to a SKILL.md is usually an edit to the description,
i.e. to the trigger surface, and a deterministic harness structurally
cannot tell you whether a description still fires.
- It hands off to the `test-harness` skill instead of saying "write a test".
An agent already knows it needs a test; what it does not know is with
what. A nudge that restates the obligation and withholds the vocabulary
is the failure this replaces.
This also corrects a backwards gate: the existing lock nudge is self-gating
on a committed lock, so a repo that already tests got reminded while a repo
that had never written a test heard nothing at all.
* fix(test-harness): name the observation vocabulary, and fix a broken example
Measured on the four questions that actually missed in live work (recorded in
examples/harness/dogfood/test-harness.gaps.trigger.eval.mjs), the skill fired
25% of the time — against 90% on the prompts it was authored with.
Two matched prompt sets, Russian and English, scored identically (25/25),
prompt for prompt, which rules out a language effect: the gap was conceptual.
The description enumerated SURFACES ("a hook fires or blocks", "a skill
triggers", "injected context lands") and never named the OBSERVATION
vocabulary — which tools a run called, whether it stayed inside its declared
allowed-tools, what it wrote, how to intercept a call without executing it.
Half the real questions are asked that way, and they landed on nothing.
- description: add the observation half (460 chars, within the 500 budget).
- new "Step 0.4 — Observing a run": maps each of those questions to the API
that already ships. Its absence is why an agent working in this repo
reinvented parseToolCalls, snapshotTree/diffTrees, interceptTools and the
effect-boundary tier in a single session.
- state the one real asymmetry rather than imply coverage: files have
assertWroteOnly, tools have only the black-list notTool.
- fix the deterministic-tier example, which imported `scriptModel` from
`vigiles/testing`. It is not exported there — deliberately, to keep that
entry point harness-agnostic — so the example returned undefined.
After: 100% on the field-miss set (RU and EN), recall held at 90% on the
authored set with false-positive 0% / precision 100%.
* feat(testing): onlyTools + skillContract — verify a skill against its own declaration
A SKILL.md declares its capability surface in `allowed-tools:` and nothing ever
checked that a run stayed inside it. That is this project's own thesis pointed
at its own file format: the declaration is prose until something mechanical
reads it.
The gap was a missing half of a symmetry the file surface already had:
| "not this one" | "nothing but these"
files | assertNoWrite | assertWroteOnly
tools | notTool | (nothing)
`notTool` can only forbid the calls a test author thought of, and the set of
undeclared tools is unbounded — so "it stayed inside its contract" was not
expressible.
- `onlyTools(allowed)` — the white-list check. Reports every distinct offender,
not just the first. FAILS on a trace that recorded no tool calls, mirroring
`assertWroteOnly` refusing a result that never recorded writes: an uncaptured
run is indistinguishable from a tool-free one, and passing it asserts nothing.
- `skillContract(dir)` — reads the frontmatter and hands back ready checks
(`activation`, `surface`). No new assertion machinery and no third entry
point: a contract is a bag of ordinary `Check<Trace>`s, so it composes into
both the deterministic and eval tiers unchanged.
Two states are findings rather than clean bills, and both fail instead of
passing vacuously:
- `undeclared` — no `allowed-tools:` line, so the skill inherits EVERY tool.
Reporting "no violations" would present the widest surface as the cleanest.
- `malformed` — frontmatter that isn't valid YAML. A strict loader reads no
contract, so the declared set is unknown rather than empty. Observed live
while writing this change: one unquoted `: ` inside a description silently
voided a skill's whole tool contract, and the audit correctly reclassified it
as inheriting everything.
The `Skill` activation call is exempt — it is the harness's own loading
mechanism, appears in every trace, and is never listed in `allowed-tools:`, so
counting it would fail every contract on every run.
Honest limit, stated in the skill rather than implied away: `onlyTools` compares
tool NAMES, so a narrow entry like `Bash(node x.mjs:*)` is satisfied by any
`Bash` call. Scope inside a tool stays unverified.
* docs(dogfood): record the measured numbers, and correct an overstated claim
The commit that added the observation vocabulary quoted "100% RU and EN". That
was measured on a 594-char variant which did not ship: it tripped the linter's
own 500-char description budget, so the text that shipped is the 460-char one,
which measures RU 75% / EN 100%.
The table now records all four variants, and states plainly what the data does
NOT support — at n=8 the 75/88/100 spread across the fixed variants is one to
two runs, so it does not establish that longer wins. The shipped text defers to
the budget rule rather than overriding a documented heuristic on evidence that
cannot distinguish the alternatives.
What the data does support is unchanged and large: naming the observation
vocabulary moved the field-miss set from 2/8 to 6-8/8, and the pre-fix RU and
EN sets scored identically, prompt for prompt, which rules out a language
effect and locates the gap in the concept.
* feat(coverage)!: one detector — colocation. Remove declaration and mention
BREAKING: `untested-*` coverage is now decided by placement alone. A
`vigiles:covers` declaration and a name-mention in a test's code no longer
confer coverage. Repos relying on either will see their untested counts rise.
There were three detectors and they were never three STRENGTHS of evidence —
they were three NAMING CONVENTIONS, all answering "does this surface's name
appear near a test?" and none answering "did anything run against it?".
Ranking them implied a precision that did not exist.
Measured on this repo before the change, `mention` supplied 9 of 10 covered
surfaces. Reading what credited what: three were real dedicated evals that
colocation could not see, and at least three were FALSE — `hooks/pre-edit.sh`
and `hooks/post-edit.sh` were credited by `test-coverage.test.ts`, i.e. the
coverage detector's own suite naming them as fixtures. The detector granted
coverage to hooks because it tested itself against them.
`declared` fared no better in practice. Its first real use in a consumer repo
declared a conformance LINT over 21 skills as coverage OF those 21 skills,
moving that repo 31 untested → 16 while nothing new was tested. It also needed
a guard assertion to stay honest: a mechanism watching the mechanism.
Colocation is kept because it cannot drift by construction — the test lives in
the surface's directory, so deleting the skill deletes its test, renaming moves
both, and `ls` answers the question without running anything.
Consequences, all deliberate:
- the detector no longer READS test files, only lists their paths, so coverage
can no longer be changed by editing text inside a test;
- comment-stripping, declaration parsing and evidence ranking are deleted;
- a good test living elsewhere now counts for nothing until it is moved next to
its surface. That is the intended pressure.
Honest remaining hole, now printed on every run rather than left to be
discovered: colocation says the file EXISTS, not that it RAN — an empty
`foo.eval.mjs` still counts. Closing that is a condition on this rule (a run
reporting >0 checks via the existing CHECK_COUNT channel), not a fourth kind.
Real cost: this repo goes 13 untested → 23, mine goes 31 → 36. Both numbers
are now true.
Tests: the removed tiers are pinned as INVERSIONS rather than deleted, so
re-introducing either fails the suite.
* feat(eval): containment comparison + a diagnostic for the confident zero
Both come from building a consumer's trigger suite (2026-08-11) and hitting the
same class of failure four times.
`compareContainment(weak, strong)` answers a question the model floor makes
people ask and nobody has measured: is a WEAKER model a valid lower bound? The
worst-case argument — tune against the weakest supported model, anything
stronger is covered — is valid only if everything that fires on the weak model
fires on the strong one. If it does not, the weak model is not a floor, it is a
DIFFERENT router, and tuning against it optimises a configuration nobody runs.
Selection is routing, not raw capability, so this is genuinely open.
The verdict separates the two directions on purpose: weak-only prompts are
counterexamples, strong-only prompts are the documented under-selection and NOT
a failure. Conflating them is how a real result gets read as a bug.
`formatTriggerRateReport` now names the setup causes when NOTHING fired on any
prompt. Three separate wiring mistakes each produced a confident, well-formed
0% — a bare id where the namespaced one was required, `pluginDir` where a loose
`skillsDir` was, and a missing `fixture` so every prompt asked about a file that
did not exist. Two were briefly written up as findings about the skills before
being caught. A zero that looks exactly like a result is worth a sentence about
why it usually is not one.
Deliberately only on a TOTAL zero: a partial rate is a real measurement and is
left alone, because a checker that second-guesses good data gets ignored and
then is not there for the case it exists for.
* docs(test-harness): document containment + the confident-zero trap
Answers "did you update the docs and the plugin?" — no, and both needed it.
docs/harness-testing.md gains two subsections under trigger-rate: the total-zero
triage (three setup causes, in the order they bite) and the containment
comparison, with the two directions kept apart — weak-only prompts are
counterexamples, strong-only ones are the expected under-selection the sonnet
floor exists for.
skills/test-harness/SKILL.md gains a row for `compareContainment` and, above the
tier table where an author will actually hit it, the rule that a 0% on EVERY
prompt is a wiring bug until proven otherwise. That is the failure this skill's
reader is most likely to mis-report, because it is well-formed and reads like a
verdict on the description.
`compareContainment` now states its own honest limit: it is harness-agnostic and
will run on Codex reports unchanged, and should not be trusted there yet. Codex
emits no skill-selection event, so `fired` is inferred and the tier is flagged
experimental — and that inference error lands exactly where this function looks,
manufacturing weak-only entries out of noise. A containment verdict is only as
strong as its weakest input signal.
Description budget and frontmatter still lint clean.
* trigger-containment: record the measured verdict — containment does not hold
21 skills x 4 prompts, one trial each, whole-harness against 37 competing
skills, haiku vs sonnet: 3 of 84 shared prompts fired on the weak model and
not the strong one, and one skill scored haiku 1.00 against sonnet 0.75.
The module header claimed nobody had measured it; that is no longer true.
Stated as 'containment is not established' rather than 'it breaks often' —
at one trial per prompt three breaks could be dispersion, and rejecting the
worst-case argument only needs the implication to be unproven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(audit): a skill naming a sibling skill that does not exist
danglingRefs matches PATHS in EXECUTABLE sources and skips prose by design,
so a skill referring to another BY NAME — the reference agents actually
follow — was checked by nothing. A rename breaks every mention silently:
nothing crashes, the model reads a route to a skill that is not there and
does something adjacent.
NEAR-MISS, not a name check, for precision. Flagging every backticked kebab
token would fire on node-fetch, pull-request, claude-code — and a checker
that fires on clean input is muted within a day. So it asks the narrower
question: is this token almost a skill that exists? Measured on a 38-skill
corpus: 0 findings clean, 1 finding with a single character removed from
one reference, naming the intended target.
Folds into the existing broken-reference deduction — no new verb, no new
category. The report field is optional because absence is not a claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* feat(testing): commandsIn + mustInclude/mustNotInclude — a document's rule about its own commands
A skill or runbook states a rule in prose and ships commands that must obey
it: 'always pass curl -g, or bracketed params glob and the request fails
SILENTLY'. Rule and violation sit in the same file three paragraphs apart
and nothing compares them — prose isn't policy, at its smallest, aimed at
the author's own document.
vigiles cannot INFER such a rule; what was missing is a cheap way to DECLARE
it. Measured on a real skill 2026-08-11: expressing one rule took ~95 lines
of hand-rolled markdown parsing around ~5 lines of actual rule, and at that
price the check does not get written. With this, the test IS the rule.
Two decisions that keep it honest: extraction is fenced-blocks-only, so the
prose STATING the rule is not read as a violation of it; and an empty match
FAILS rather than passing vacuously, because a rule that held over zero
commands reads exactly like one that held over twenty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* docs(doc-commands): the prior-art survey, including what is NOT novel here
Four quadrants, and this sits in the empty one: extraction (codedown),
extraction+universal linter (markdown-shellcheck), execution (mdsh, doctests),
matchers (vitest/jest). The nearest neighbour applies a UNIVERSAL correctness
checker; this applies the rule the document states about itself, which no
general linter can know and no doctest can express.
The extraction half is NOT novel and now says so, with the reason it is
reimplemented anyway (30 dependency-free lines vs vendoring an awk script)
and the composition worth stealing: hand the structured commands to a real
shellcheck when one is installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* fix(lethal-trifecta): a skill's fence is `disallowed-tools`, not `allowed-tools`
A skill's `allowed-tools:` is a PRE-APPROVAL, not a restriction. Claude Code's
docs say so outright, under "Pre-approve tools for a skill":
"The `allowed-tools` field grants permission for the listed tools during the
turn that invokes the skill... It does not restrict which tools are available:
every tool remains callable... To remove tools from Claude's available pool
while a skill is active, list them in `disallowed-tools`."
Two bug reports say the same from the field, both closed as not-planned:
anthropics/claude-code#18837 (Jan 2026) and #37683 (Mar 2026) — the second
reproduced INTERACTIVELY, on a live model, on a different CLI version, with the
skill spawning an `Explore` subagent it had told itself never to use. Reproduced
locally under `claude -p` (CLI 2.1.227): a skill declaring
`allowed-tools: WebSearch, WebFetch` read a private file and wrote a new one.
`disallowed-tools` was then measured across 9 runs and DOES restrict. Cleanest
control, one run: the same `Read` succeeds BEFORE the skill activates and is
denied after — "Permission to use Read has been denied." — and through a `Task`
subagent in the same run, "Error: No such tool available: Read. Read is disabled
for this session, in subagents as well as here." Positive controls first (without
the line, `Read` succeeds); refutations failed (a `bogus-tools:` key blocks
nothing; an unactivated skill's fence does not apply).
So the check was reading a field that bounds nothing. It UNDERSTATED risk — "18 of
38 units hold the trifecta" on a corpus where all 38 did — and it CREDITED a narrow
`allowed-tools` with a reduction that does not exist. That is this project's own
thesis, a declaration present is not a rule enforced, violated by this project.
Dogfood, before and after, on this repo: Safety 100 -> 70. All four skills here
declare narrow `allowed-tools` with no web tool, so the old reading called the
whole surface clean.
The SHAPE changed too, not only the arithmetic. Counting every unfenced skill as
its own finding fires on ~100% of every real harness, and a check that fires on the
ecosystem default is muted within a day — taking the hard findings with it. So:
- `fence: "none"` (no `disallowed-tools:` line) is ONE aggregate line plus the
skill names, in the report, the Safety ring, and `lint` (where it also stops
emitting one GitHub annotation per SKILL.md). It is one fact about the harness,
not N facts about N skills.
- `fence: "ineffective"` (a fence that closes no leg) keeps its own line: rare,
per-skill, and a real mistake — same shape as disallowed-tools-contract's
"this entry blocks nothing".
Presentation differs; SEVERITY AND GRADE DO NOT. Both are "advisory" and both count
once toward `trifectaExposure`, so collapsing the presentation never shrinks the
penalty, and an ineffective fence (capability <= no fence) can never grade worse
than no fence — the non-monotonicity this rule was already fixed for once.
A leg is closed only when every built-in that supplies it is denied. That supplier
catalog is deliberately WIDER than the allow-side one: on the allow side a missing
tool under-states risk (safe), on the deny side it would over-state safety. Filtered
against the dialect's own catalog so a Claude Code author is never told to deny
Codex's `shell`. Stated limit, in the rule doc: a session-provided MCP server can
re-supply a closed leg, so "closed" means closed among the built-ins.
SUBAGENTS ARE UNCHANGED. A subagent's `tools:` is a documented rail that really
does bound the unit; `lethalTrifectaIssues` and every agent path are untouched.
Tests pin the point directly: a skill with `allowed-tools: Read, Grep` and one
declaring the entire catalog produce byte-identical findings, and a narrow list
still yields a finding — that test fails if anyone reintroduces the old reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore(api): record commandsIn/mustInclude/mustNotInclude in the testing surface
api:check has been red since
|
||
|
zernie
|
ee204a99c8 |
fix(audit): validate a hook matcher as a pattern, not as a literal tool name (#132)
The rule shape-checked a matcher as though it had to look like mcp__server__tool, so it reported working regexes as never firing and accepted patterns that fire on nothing. A user obeying it replaced a working matcher with a dead one — the failure the rule exists to prevent, reached by complying with it. Semantics were measured against the real CLI rather than inferred, and the reproduction ships as tools/measure-hook-matcher-semantics.mjs: a matcher with no metacharacter is compared by string equality, one with metacharacters is an unanchored regex. Both contested rows were confirmed live against an MCP server connected under two namings. A literal keeps the shape check. A pattern is compiled and tested against probes covering the server namings that actually occur, including an underscore inside the server segment and a hyphenated uuid, plus segments derived positionally from the matcher itself so a deliberately scoped mcp__memory__.* stays clean. Firing on some names but not others is now its own finding that names the missed probes and never says never fires. An uncompilable regex is its own finding rather than silently valid. The suggestion can no longer recurse: a recovered server segment must be name-shaped, and a property test asserts every suggestion satisfies the rule that produced it. Fixes #131 |
||
|
zernie
|
a87d0cf2bc |
fix(audit,testing): three dogfood defects — a heading, a salvage, and a check count (#130)
A bundled script named in a markdown heading was invisible to skill-resource-resolves, because the use-directive gate reads prose and a heading is not prose. An ATX heading now counts; the widening stops there, measured at 1 finding before and after over 28 SKILL.md files. Malformed frontmatter was regex-salvaged into the tool contract the lethal-trifecta and Safety ring read, so a unit whose allowed-tools a strict loader rejects scored as the narrow contract its author meant. A salvage may now only make the verdict worse, never better — the vendored corpus forced that nuance by going red when the first pass refused salvages outright. vigiles test reported a clean exit with zero assertions as a pass. vigiles/testing now counts checks per tier and the runner reports a fourth, non-fatal state. Silence stays a pass: undefined means nobody counted, 0 means counted and found nothing. No new CLI verb or flag. Every mechanism is deterministic. Direction is strictly more conservative: nothing can now pass that could not before. |
||
|
zernie
|
41169e379a |
fix(audit,testing): split Tested/Evaluated, make coverage evidence honest, and stop three silent-success paths (#129)
* feat(audit): split Tested into Tested + Evaluated, with a `not measured` state `Tested` collapsed deterministic harnesses and real-model evals into one number, and its finding read "N surfaces with no vigiles test/eval" — one sentence for two prescriptions three orders of magnitude apart in cost. A harness (`*.harness.mjs`, `*.test.*`) is free, runs in milliseconds on every push, and asks whether a gate still catches what it claims. An eval (`*.eval.mjs`) spends real model calls, runs on a schedule, and asks the one question a deterministic read cannot: does this skill fire at all? Folded together, a repo with complete deterministic coverage and no evals scored identically to a repo with neither. - test-coverage: one discovery pass, partitioned by suffix into a harness tier and an eval tier; the union `covered`/`untested` fields are unchanged. Mirrored in the browser-safe twin. - scan: carries `untestedHarness` / `unevaluated` / `evaluable` (optional and additive; a producer predating the split falls back to the union). - audit-score: `Tested` (deterministic) and `Evaluated` (real-model), both advisory. `Evaluated` has THREE states — a number, a measured `0`, and `not measured` when the executing checks were skipped and no eval exists. Scoring 0 there would report the absence of a check as its result, which is the pattern this tool exists to name elsewhere. In that state the ring names the command that would measure it. - cli: threads the existing execute-skip signal into the ring. - report/HTML/JSON: both rings render; the inventory carries both counts. Grade weighting is unchanged — both rings are advisory and neither enters `reportDeductions`. Audited on this repo before and after: A (100/100) both times; only the diagnostic rings move (Tested 67 → Tested 58 + Evaluated 22). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * docs(site): the audit grades six categories, not five Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * fix(lint): a clean union must not read as 'firing verified' Every surface can have a deterministic harness while nothing has ever measured that a skill fires. The gate is unchanged (still the union), but the ✓ now says so instead of implying the question was asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u * test(site): regenerate the browser-parity fixture for the Evaluated ring The audit now emits a sixth category, so the Node-generated expected report and the in-browser one differ until the fixture is regenerated — which is what the test's own header prescribes for an intentional engine change. Regenerated with `node site/scripts/gen-parity-expected.mjs`. The diff adds the Evaluated ring and its two inventory counts and nothing else: grade and overall are untouched, which is the check that the split did not move the weighting. * fix(coverage): `Tested` measured a STRING, not a test DEFECT. A surface counted as covered when its path or namespace appeared as a SUBSTRING anywhere in any discovered test file (`content-reference`, in src/test-coverage.ts). Two halves, and the second was worse. REPRODUCTION (a real repo: 37 skills, 14 hooks). Appending ONE line to an existing harness file: // probe: skills/argument-arc moved the audit's untested count 33 -> 32. A comment. Not a test. Reverting the line put it back. So `Tested` was trivially gameable and the gaming was indistinguishable from real work. The false NEGATIVE: that repo's `.claude/pipeline/skills.harness.mjs` genuinely asserts over all 21 of its pipeline skills, but builds each path at runtime (`join(root, ".claude", "skills", name, "SKILL.md")`), so it contains ZERO literal `skills/<name>` strings (grep: 0 matches) and covered NOTHING. Generality was PENALISED: 21 trivial files each naming their own subject would have scored better than the one harness that actually tests all 21. This is the exact substitution vigiles names in other people's repos — presence of a surface taken for presence of the property, the way `noExplicitAny` in a config is taken for the rule being enforced. The tool was committing it in its own scoring. FIX, three parts (new src/coverage-evidence.ts — pure and browser-safe, so the disk detector and its in-browser twin cannot drift on what decides a grade): 1. DECLARATION. A test file may carry `vigiles:covers <surface> ...`, mirroring the existing `vigiles:ignore-test` marker convention on the other side. A runtime-path harness can now say what it covers; an accidental mention cannot. Declaration beats inference — the same move as declaring `allowed-tools` instead of inferring capability. 2. PROVENANCE. Every coverage decision now carries HOW it was reached — declared / colocated / mention — and the counts are printed by `formatUntestedReport`, by `formatScanReport`, and carried in `AuditReport.inventory.coverageEvidence` (additive; schema version unchanged). A repo whose coverage rests entirely on substring matches can finally see that about itself; "28 covered" and "28 names that happen to appear in a file" used to print identically. 3. The bare substring detector is KEPT, deliberately, and said so in the code: it is the zero-config path that makes an ordinary `foo.test.ts` naming `skills/foo` count without anyone learning a marker, and dropping it would flip honestly-covered surfaces to "untested" overnight. What it no longer reads is COMMENTS. A path in a comment is prose about a test; a path in executable code is the test reaching for the surface. Prose isn't policy, including ours. It is also now visibly the weakest tier in the report, and when it is the ONLY evidence the report says so in words. Grade weighting is untouched. TESTS prove both halves, each watched failing first: with the fix reverted, "a surface named ONLY in a comment is NOT covered", "a runtime-path harness that DECLARES its surfaces is counted" and "a declaration OUTRANKS a mention" fail (3 failed | 20 passed); with it, 23 passed. Also guarded: a mention in CODE still counts, and a `"https://..."` string literal survives comment-stripping. Docs (untested-skill/-subagent/-hook) updated — untested-skill.md previously recommended "add the skill's path in a comment", which was advice to exploit the defect. * fix(site): key the browser-test prebundle to dist, so a stale cache can't lie DEFECT. `site/src/demo/browser-parity.browser.test.ts` guards node:zlib vs pako divergence by comparing an in-browser AuditReport against one generated in Node. `site/vitest.config.ts` pre-bundles the engine (`optimizeDeps.include` on `@engine/scan-files` / `@engine/audit-report` / `pako`) and Vite caches the result in `site/node_modules/.vite`. That cache SURVIVES `npm run build` — Vite invalidates on config/lockfile changes, not on the mtimes of an aliased path outside node_modules. So the browser side can compute against a PREVIOUS engine while Node computes against the current one. REPRODUCTION (observed 2026-08-07, twice — once by accident, once deliberately). Accidentally: an engine change added a report field; the parity test failed showing the field present on the Node side and absent in the browser. I read that aloud as a genuine engine divergence and started hunting a pako bug that does not exist. `rm -rf site/node_modules/.vite` -> 42/42 pass. Deliberately, to prove the fix: plant an extra `inventory` field in src/audit-report.ts, `npm run build`, regenerate the expected fixture, run the browser suite with a warm cache. - ORIGINAL config: `1 failed | 41 passed`, diff showing `"probeStaleCache": 1` on one side only — a field-level diff that reads exactly like a real divergence and says nothing about the cache. - THIS config: `42 passed`, cache dir rotated `.vite-engine-8baf73f9e373` -> `.vite-engine-a52a27e0137e`. Reverting the plant returns the fixture to its committed bytes. FIX. Key the prebundle cache DIRECTORY on a fingerprint of `dist/` (`site/vite.engine-stamp.ts`; `cacheDir: node_modules/.vite-engine-<stamp>`), sweeping the directories keyed to other fingerprints so they don't pile up. WHY a key and not the alternatives. Comparing the cache's mtime against `dist/` and forcing a re-bundle, or clearing `.vite` in the test script, both work and both leave the bad state REACHABLE — a comparison someone can get wrong, or a script step bypassed by running `vitest` directly. Keying the directory removes the state instead: a different `dist/` is a different path, so "prebundle built from another engine" cannot be reached, and an unchanged `dist/` still gets a warm cache. By construction rather than by check — the argument this project makes everywhere else about removing a capability rather than policing its use. The test's header instruction ("pako and node:zlib disagree — do NOT paper over it") was correct for one of the two states and actively misleading for the other; it now says which state is no longer reachable and why, and the assertion carries the same note in its failure message. * fix(site): make the parity-fixture generator emit prettier-clean JSON DEFECT. `site/scripts/gen-parity-expected.mjs` wrote the fixture with `JSON.stringify(x, null, 2)`, which is NOT what prettier prints for JSON — it keeps short arrays exploded across three lines where prettier collapses them. CI gates on `prettier --check .`, so every regeneration of the fixture committed a file that reddened CI later and far from the cause. It just did: branch head |
||
|
zernie
|
7cf22d012d |
fix(trifecta): a Bash narrowed to a concrete command is not the whole shell (#128)
`classifyTrifectaLegs` stripped everything after `(`, so `Bash(node ./scripts/log.mjs:*)` was classified as plain `Bash` — leg A and leg C, the full shell — and any web tool alongside it produced a lethal-trifecta finding. That made the checker blind to its own prescription. Its advice is "drop at least one leg (Meta's Rule of Two)", and narrowing the grant is how an author does that; before this, taking the advice moved nothing. Deliberately high-precision: the grant loses its legs ONLY when the pattern pins a program AND at least one concrete argument. A program with free arguments (`node:*` runs `node -e`), any shell or wrapper however pinned (sh, bash, eval, env, xargs, sudo, ssh, docker, make), and any program whose whole job is moving bytes out (curl, wget, scp, git, npm) all stay full shell. When in doubt it keeps the legs. Tests assert both directions — the narrowed grant produces no finding where bare Bash does, and ten look-narrowed forms still produce one. No score delta is claimed: a controlled A/B on the repo this was found in left the Safety ring unmoved, because the units it counts are not the narrowed ones. The classification is what was wrong, and what this fixes. Found by dogfooding on zernie/mine. |
||
|
zernie
|
f5aaa8aeae |
fix(test,eval): discover harnesses inside dot-directories (#126)
`vigiles test` and `vigiles eval` printed "no files found" in a repository holding two harnesses, because discovery globbed without `dot: true` and both lived under `.claude/` — which is where a Claude Code harness lives by definition. The scoring consequence is the one that matters: `Tested` then measures visibility rather than coverage, telling an author "you have no tests" when the honest reading is "I looked in the wrong place". This codebase had already learned the lesson and missed one call site — `test-coverage.ts` and `cli.ts` both pass `dot: true` with comments explaining why. Test asserts both halves (dot-directories in, node_modules still out) and was verified by planting the defect: reverting the one-line change turns it red. Found by dogfooding on zernie/mine. |
||
|
zernie
|
86fdb5f141 |
fix(audit): read MCP servers from the layout, not a hard-coded path (#127)
Three defects, plus a cheap bet on the Agent Plugins packaging format. - marketplace.json had drifted a full major behind plugin.json; a CI gate now fails on any future name/version/description/license drift between the manifests. - Both collectMcpServers implementations hard-coded ".mcp.json", bypassing layout.mcpConfigFile and violating adapter-aware-lint-rules. Behaviour is unchanged for both shipping harnesses; the literal is gone. - The report could say "MCP servers: no" while listing an MCP finding. A plugin packaged to the Agent Plugins standard puts its MCP servers in a root mcp.json that no harness layout names, so three MCP rules silently checked nothing on such a plugin. src/core/agent-plugins.ts recognizes the standard by the manifest's $schema, never by filename. Not modelled as a HarnessAdapter: the standard is a packaging format with no tool catalog, hook events or runtime. vigiles also ships as a conformant plugin. Docs are explicit that this is additive — VS Code reads the standard's root manifest; Claude Code and Codex document their own manifest paths. |
||
|
zernie
|
94e9180218 |
fix(hook-runtime): let the repair through so a stale stamp can't wedge the repo (#122)
`verifyStampOrRefuse` makes a compiled hook refuse to run when its source no
longer matches the stamp — correct for tampering. But when that hook is a
PreToolUse Bash gate wired into the same repo, editing the hook source makes
it block EVERY Bash command, including `vigiles compile`, which is the only
command that regenerates the stamp. The repo wedges: you cannot recompile
because the stale hook refuses to let you.
Observed 2026-08-03. The only escape was hand-editing `.claude/settings.json`
to unwire the gate, compile, then rewire.
Fix: exempt the repair action, loudly. `isStampRepairEvent(event, hookFile)`
is true for
- a Bash command invoking `vigiles compile` (AST-backed, so `npx …`,
`pnpm exec …`, `./node_modules/.bin/vigiles …` and a compound command all
count), or
- an edit/write whose target IS the hook's own source file, so a FILE gate
over the repo can also be fixed.
Everything else still fails CLOSED. The same escape covers a hook that no
longer LOADS (a typo mid-edit), which is the same wedge reached a different
way — otherwise the deadlock stays reachable from a normal edit-compile cycle,
which is the property that has to hold.
Why this over failing open or a --force flag: failing open silently disarms
every gate on any edit, and a flag is one more thing to know at exactly the
moment you're stuck. This keeps fail-closed as the default and names the one
door out, in the error message itself.
It also does not weaken what the stamp actually buys. While the stamp is
stale the hook enforces NOTHING (it refuses every call), and anyone able to
rewrite a hook's source can equally rewrite `.claude/settings.json` — the
escape they would otherwise use. The guarantee is that a smuggled capability
can never run SILENTLY; that is unchanged, and the pass-through prints three
lines of stderr saying exactly what it did.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
7b98ae35c4 |
feat(testing): export loadHook + add the react assertions (#123)
The compiled-hook testing story had a hole at each end.
1. NO PUBLIC LOADER. `assertHookDenies(hook, event)` / `assertHookAllows` take
the hook OBJECT, but a `.harness.mjs` test only has its PATH, and the loader
that bridges the two (`loadHookProgram`) lived privately inside `cli.ts`. So
the intended in-process test path was unreachable from the file format
`vigiles test` actually runs, and authors fell back to spawning the runtime
as a subprocess — precisely the plumbing compiled hooks exist to remove.
`loadHook(path)` is now exported from `vigiles/testing` and `vigiles/unit`,
and the CLI runtime calls the SAME function, so a hook that loads in a test
loads identically in production (one loader, no drift). It keeps the
actionable errors: a `.ts` hook on a runtime without type stripping says to
use tsx / Node >= 23.6, and a module with no default export says
`use \`export default defineHook({…})\``.
2. NO ASSERTION FOR A REACT HOOK. A react can't block, so the gate assertions
don't apply, and nothing else did either. That left `notice()` a live trap:
its message goes to STDERR, so a probe built on `execFileSync` (stdout only)
sees nothing and reports a healthy react hook as DEAD — measured against
three real hooks on 2026-08-03.
`assertHookNotices(hook, event, matcher?)` and `assertHookSilent(hook, event)`
read the reaction in-process, so the stream never enters into it. Both name
what they got instead on failure, including when aimed at the wrong role.
Also ships the example the finding asked for:
`examples/harness/compiled-hook-inprocess.harness.mjs` loads a gate and a react
hook BY PATH and asserts over both. It runs under `vigiles test` (so CI proves
the documented pattern works, not just that it parses), and the docs in
compiled-hooks.md / testing-api.md now show this path, with the stderr trap
called out explicitly.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
246be8e9c7 |
feat(hook): add CommandView.writesTo() and capture redirections (#120)
`leafCommandsNormalized("echo x > papers/x/paper.md")` returned
[{head:"echo", argv:["echo","x"], args:["x"], flags:{}, assigns:{}}]
The redirection and its target were DROPPED by the parser — they live on the
enclosing `Stmt`, and the walk only visited `CallExpr` nodes. So the file the
command actually WROTE appeared in no field of any leaf, and `cmd > file` /
`cmd >> file` — the single most common write shape — was invisible to any
compiled gate. A gate written to protect a directory from Bash writes was
silently weaker than the `grep` it replaced.
The only available discriminator was `isSideEffecting()`, which classifies the
WHOLE command line. So `grep -c x notes/S.md 2>/dev/null` classified
side-effecting, `touches` matched, and a plain READ got blocked. Measured
while dogfooding on 2026-08-03: it blocked real work twice within an hour.
Two changes:
1. `NormalizedLeaf.redirects: readonly LeafRedirect[]` — `{op, target, fd,
writes}` per redirection, target quote-unwrapped and $HOME-canonicalized
like every other word (dynamic → null, present but unresolved, never
guessed). `leafCommandsNormalized` now iterates `Stmt`s; every `CallExpr`
in the mvdan AST is a `Stmt.Cmd`, so the same leaves are found.
2. `CommandView.writesTo(prefixes)` — answers the WRITE question precisely:
redirect targets with a write op, plus the argv positions of known
file-writing programs (`sed -i`, `cp`/`mv`/`install` destination, `tee`,
`dd of=`, `truncate`, `shred`). An unlisted head contributes no target, so
it never guesses.
`touches` is unchanged — it still answers "mentioned" — and both doc comments
now say so explicitly, since conflating them is exactly the trap.
Quoting is handled by the PARSER, not a regex, so a write quoted inside
another command does not match: in `echo 'echo y > a/paper.md' > /tmp/note.txt`
the only real target is `/tmp/note.txt`. Pinned by a test.
The mvdan redirect-op codes have no public name mapping, so a test re-derives
the whole `op -> string` + write-classification table from the installed
parser: an upgrade that renumbers a token fails loudly instead of silently
reclassifying a write as a read.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
1194aed561 |
fix(compile): key hook wiring by the FILE, not the string you typed (#121)
`vigiles compile x.hook.ts` followed by `vigiles compile ./x.hook.ts`
appended a SECOND `{matcher, hooks:[…]}` block for the same hook to
`.claude/settings.json` instead of replacing the existing one. After a few
iterations of an edit-compile loop the settings file accumulates duplicate
wirings that all fire.
Root cause: the merge key was the raw path string the user passed. The emitted
command carried that same string, and `managesHook` tested for a substring —
so `"… run-program x.hook.mjs".includes("./x.hook.mjs")` was false, nothing
was replaced, and the block was appended. (The same substring test also had
the opposite failure: `x.hook.mjs` "managed" a neighbouring `my-x.hook.mjs`.)
Fix: `normalizeHookRef()` canonicalizes a hook reference — resolved against
the cwd, made relative when it lives under it, POSIX separators, no `./`.
`compile` emits the canonical ref in the runtime command AND uses it as the
merge key, and `managesHook` compares canonicalized whitespace-separated
tokens instead of testing for a substring. So every spelling of one file is
one wiring, `my-x.hook.mjs` is a different hook, and a settings file already
carrying a differently-spelled entry is de-duplicated on the next compile.
Verified end-to-end: four `compile` runs over `guard.mjs` / `./guard.mjs` /
the absolute path leave exactly one entry, which still enforces.
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
zernie
|
5f7531e5ce |
fix(audit): grade an inherited lethal trifecta like an explicit one (#119)
* fix(audit): grade an inherited lethal trifecta like an explicit one
The Safety score penalized the remediation it recommends.
A unit whose contract NAMES all three trifecta legs produced a graded -10.
A unit that INHERITS all tools — strictly worse, it holds those three legs
implicitly plus every other capability — produced an ungraded advisory. So
declaring an `allowed-tools` contract, a genuine risk reduction, could only
ever LOWER the score.
Measured while dogfooding a real 35-skill repo (2026-08-03):
before: Safety 70 = 100 - 3x10, counting only the 3 units that already
had explicit contracts; the 31 units inheriting everything
contributed ZERO
after: Safety 0 after adding `allowed-tools` to all 35, which took the
units holding the full trifecta from 35/35 down to 17/35
The tool reported the safer configuration as strictly worse, and was silent
on the unsafe one. Two model defects, not a threshold to tune:
1. NON-MONOTONE — only the explicit case counted.
2. SATURATING — with a flat per-unit weight, 35 and 17 exposed units both
blow past the clamp and score 0, so halving exposure shows no change.
Fix: one shared `trifectaExposure()` counts EVERY unit holding all three
legs, whatever the severity, and charges
`min(W_TRIFECTA x exposed, W_TRIFECTA_MAX x exposed/assessable)` — per-unit
cost capped against the share of the model-invocable surface exposed. Both
the Safety ring and the shared `reportDeductions` read that one number, so
the ring and the headline still agree.
The same two measured states now read Safety 70 -> 85.
Existing calibration anchors are preserved: a single explicit all-three
subagent is still -10, and the `feature-dev` 3-of-3 shape is still -30 -> C.
What changes is that a harness whose whole surface inherits everything now
takes the same capped -30 instead of a free pass.
Deliberately NOT changed: `formatScanReport`'s "structural issue(s)" tally
still counts only `hard` findings. That line is about structural defects,
not the safety axis; promoting every idiomatic inherits-all unit to "broken"
in the CLI summary is a louder, separate call.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(site): the demo sample repo now grades F (2 of 3 units hold the trifecta)
The DemoAudit browser test hard-coded the sample map's grade. With the
inherits-all trifecta graded, the sample repo — 2 of its 3 model-invocable
units inherit all tools — drops D(61) to F(41), matching the regenerated
parity fixture.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(site): the Playwright demo assertion follows the sample repo to F
The in-frame report e2e hard-coded the grade badge alongside the browser
test fixed in the previous commit.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Claude
|
f576c74b3d |
chore: drop stale directory references and product-planning prose from the spec
CLAUDE.md.spec.ts still pointed at a directory that no longer exists, so `compile` failed with 48 stale-reference errors and `lint` reported a stale hash. Removes those pointers (49 keyFiles entries, 99 in-prose references), deletes the test that indexed the removed directory, and drops the matching glob from .vigilesrc.json. Prose was rewritten where a pointer was part of a sentence, so no claim is lost. Also removes prose that does not describe shipped behaviour: sequencing and priority notes, hosted-dashboard and pricing plans, the next-phase design section, and competitive framing in rule text. The documentation rules now state the line plainly -- documentation of shipped behaviour belongs in this repository; roadmap, positioning, go-to-market and design notes for unshipped capabilities do not. That rule now lives in the spec rather than in the generated CLAUDE.md, so `compile` no longer erases it. compile: CLAUDE.md.spec.ts clean (the 5 errors in examples/ are pre-existing and unrelated -- paths that moved into src/core/ and .claude/skills/) lint: exit 0, hash valid tests: 2331 passed; the same 6 failures as before this change (pylint/rubocop absent in this environment, report template not built) prettier: clean Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012omt6sirxNMyZN2x5RhK2u |
||
|
zernie
|
103d10bc62 |
refactor(linters): collapse dispatch maps into LINTERS; derive site chips; cedar P1 (#117)
Stage 4 of the LinterAdapter port: inline the four legacy dispatch maps + getCliRuleSet into the LINTERS adapters and route checkLinterRule through capabilities.existenceCheck — LINTERS is now the literal single dispatch source (behavior byte-preserved, api-surface unchanged). Site: Wedge.tsx derives the linter chip strip from BUILTIN_LINTERS via an @engine/spec alias (can't go stale). Cedar: kept + a "verification depth" P1 roadmap item. Stale-ref sweep for the deleted maps across CONTRIBUTING/ci.yml/roadmap/research-index; add-a-linter skill updated. |
||
|
zernie
|
7be092ce19 |
refactor(linters): LinterAdapter port + JVM/Go doc & test parity (#116)
LinterAdapter registry (Record<BuiltinLinter, LinterAdapter>) makes a missing linter a tsc error; generate-types iterates the registry; conformance test set-matches registry vs BUILTIN_LINTERS + docs + site chip list. CI installs detekt/ktlint/checkstyle/golangci-lint so their gated tests run (no silent skips). Docs/site catalogs->linters parity. add-a-linter contributor skill. Report-view teaser cards lightened + linter footnote de-staled. Stage 4 (dispatch-map collapse) tracked as a follow-up. |
||
|
zernie
|
3017f70ef5 |
feat: JVM/Go linters + audit/init/compile hardening + adoption UX (#114)
Closes #107, #109, #110, #111, #113. Vlad's 5 issues (skill allowed-tools round-trip, detekt/ktlint/checkstyle/golangci-lint catalogs, 2 scan false-positives, non-JS harness guide, testGlobs docs), a 14-bug dogfood batch, adoption/demo UX, and prevention refactors (one markdown-it fence oracle replacing 5 regex toggles; one js-yaml detekt-config parser replacing 3). LinterAdapter port added to the roadmap as P0. |