mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
7b8cdecd5e
* fix(loader): a surface that exists on disk was never read, and the report named it anyway
The loader read the repo-root `skills/` **or** the project-level
`.claude/skills/` — never both — and materialized whichever it picked under the
SAME canonical `<materializeRoot>/<surface>/…` key. Two real files, one key: the
loser was never opened, and the winner's bytes sat under the loser's name.
The vendor settles which one the harness loads: BOTH, in two namespaces.
> Plugin skills use a `plugin-name:skill-name` namespace, so they can't
> conflict with other levels. For example, `my-plugin/skills/deploy/SKILL.md`
> becomes `/my-plugin:deploy` and loads alongside a `deploy` skill in your
> project's `.claude/skills/`.
> — https://code.claude.com/docs/en/skills § "Where skills live"
So "pick one" was never a tie-break to get right; it was a question with no
answer, asked because the key shape forced one. The fix removes the question:
every scope present is read, and a scope's key prefix is DERIVED FROM THE SCOPE,
so two files can no longer claim one key — there is no ordering, no
"if already taken", and no last-write-wins left to get wrong.
Measured (36 SHA-pinned public repos, 2026-08-18):
* 573 → 493 surface files that exist on disk and are never opened.
The 80 recovered are the shadowed ones: nyldn/claude-octopus 71 → 0,
hyhmrright/brooks-lint 8 → 0, thedotmack/claude-mem 1 → 0.
* The 493 that remain are a DIFFERENT, pre-existing gap, unchanged by this
commit: directory symlinks under `skills/`, which `entryOf` deliberately
does not descend (the ELOOP containment) but Claude Code does follow.
* nyldn/claude-octopus ships each skill twice — 61 under `skills/`, 57 under
`.claude/skills/`, 50 names in both, and ALL 50 pairs differ. One pair is
vendored as a true-positive fixture.
* vigiles' own repo: 6 skills → 19, lethal-trifecta units 4 → 12. Its own
audit was blind to two thirds of its own machine.
* Grades moved on 1 of 36: brooks-lint C(70) → B(82). NO finding was deleted
(6 before, 6 after) and the COUNT of dangerous units rose (6 → 7); the
Safety ring is a ratio, and this moves its denominator from 6 to the real
12. See the report for the full justification.
Also fixed, because reading both scopes caused it: a name no longer identifies a
surface, so `description-overlap` would have printed `skills "dup" and "dup"` —
50 times on claude-octopus. Overlap labels now carry the real path.
SUBTRACTS: the `root` | `user` | `none` variants of `SurfaceSource` (the either/or
is gone, so "we picked the wrong one" is unrepresentable), the duplicated
precedence rules in `scan-files.ts` (the decision now lives once, IO-free), and
one net lint warning in the two touched files (6 → 4).
Gates: npx vitest run · npm run lint · npx prettier --check . · npm run api:check
· npm run build — all clean, in that order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd
* chore: escape the raw NUL bytes on this branch too
* chore: regenerate the browser-parity fixture for the two-scope disambiguation
The overlap finding now names each skill by PATH, not by bare name — which is
the point of the two-scope work: with root skills/ and .claude/skills both
present, a bare name does not identify a surface. Intentional engine change,
regenerated per the parity test's own instruction.
* chore: regenerate generated.d.ts for the merged tree
Единственный конфликт мёрджа был одной строкой в СГЕНЕРИРОВАННОМ файле — счётчик файлов
проекта, 388 против 387. Разрешено не выбором стороны, а регенерацией: у слитого дерева
счёт 389, то есть ОБЕ стороны были неверны для него. Выбор любой из них дал бы зелёный
мёрдж с неправдой в артефакте.
* fix: four defects found by dogfooding vigiles on a real consumer repo
Все четыре найдены за один день догфуда на `zernie/mine` (43 скилла, 48 харнесс-файлов),
каждый — с замером, а не с рассуждением.
1. `exclude` не доходил до прохода по markdown-рефам
`findDocRefs({ basePath })` вызывался без `ignore`, хотя поле `exclude` документировано
ровно под этот случай («vendored or benchmark fixtures the repo's own lint shouldn't
police»). Замер: 9 битых рефов, 8 из них внутри каталога, явно перечисленного в
`exclude`, и все 9 репортились. Следствие дороже причины: выйти в 0 стало нечем, поэтому
шаг пометили `continue-on-error: true` — а lint, чей код возврата выбрасывают, не гейтит
ничего, и рядом выросли самодельные CI-шаги, чтобы гейтить вместо него. Один непереданный
аргумент — вся эта цепочка. После починки: 9 → 0.
2. Рефы искались ТЕКСТОМ, а не разбором
`/\b(enforce|file|cmd|ref)\(\s*["']…/` — четыре из шести входов классифицировались
неверно (замер 19.08): `ctx.file("OUT")` (метод чужого объекта — живой ложный
срабатыв в реальной репе), `obj.cmd(...)`, `// cmd(...)` в комментарии,
`'cmd("x")'` внутри строки. `\b` спокойно стоит после точки, а у регулярки вообще нет
понятия комментария и строкового литерала. Разбор делает все четыре НЕВЫРАЗИМЫМИ: AST
отдаёт только вызов, у которого callee — голый идентификатор. `typescript` и так
runtime-зависимость пакета — парсер лежал в коробке всё это время.
3. Раннер скриптов был строгой очередью
`files.forEach(... spawnSync ...)` со `stdio: "inherit"`. Замер на 48 харнессах:
270s → 77s (3.5×), результат ПОБАЙТОВО тот же (84 passed, 1 with 0 checks, 7 failed).
Пул по умолчанию выводится из `entry`, а не из нового флага: `test` не передаёт entry и
разъезжается по ядрам (файлы бесплатны и детерминированы по построению — мок-модель, без
API-ключа), `eval` передаёт и остаётся на 1, потому что параллель там — это одновременные
оплачиваемые вызовы модели. Вывод буферизуется на ребёнка: с `inherit` 48 отчётов
шинкуются друг о друга, и это настоящая цена параллели, а не сам цикл.
4. Путь агента в находке брался по ИМЕНИ
`new Map(agents.map(a => [a.name, a.path]))` оставляет последнюю запись, поэтому при
`agents/foo.md` и `.claude/agents/foo.md` (разные неймспейсы у Claude) находка о первом
репортилась против файла второго. Находка lethal-trifecta, указывающая не на тот файл,
хуже отсутствующей: она отправляет читателя проверять невиновный код. До полной
scope-qualified идентичности (правка крупнее, `finding.name` несёт только голое имя)
неоднозначное имя даёт ПУСТОЙ путь вместо произвольного.
Плюс: все 10 джобов в 6 воркфлоу этого репозитория были без `timeout-minutes` — то есть
зависший джоб шёл до шестичасового потолка GitHub. Бонды ~3-4× ЗАМЕРЕННОЙ длительности
собственных прогонов репы, замер процитирован над каждой строкой.
Тесты: два новых на пул, обе половины. ⚠️ Первая версия фикстуры считала маркеры, которые
только СОЗДАВАЛИСЬ, — то есть «сколько стартовало всего», а это достигает N и при строго
последовательном прогоне; тест проходил по причине, не связанной с параллелью. Исправлено
освобождением слота на выходе. Обе мутации (форсировать serial · распараллелить eval)
валят каждая свой ассерт, каждая с доказательством, что патч лёг.
Гейты: `npm run lint` 0 errors · `npx prettier --check .` чисто · `npm run api:check` без
дрейфа · `npm run build` · `npx vitest run` — падает только пре-существующий
`scripts/release-preset.test.ts` (не находит `@semantic-release/commit-analyzer`, объявленный
в package.json, но не установленный в этом окружении; на CI со свежей установкой зелено).
* docs: record what each of the four defects was, in the pages that own them
comparison.md — что СЧИТАЕТСЯ рефом теперь (вызов, у которого callee — голый идентификатор,
а первый аргумент строковый литерал) и почему три формы больше не рефы: метод чужого объекта,
упоминание в комментарии, строка с текстом вызова. Плюс область: проход читает `**/*.md` и
теперь уважает `exclude`, чего не делал — из-за чего репозиторий с вендоренной чужой
разметкой не мог получить чистый lint вообще.
docs/rules/delegation-trifecta.md — новая секция про два агента с одним именем: находка
теперь показывается БЕЗ пути, а не с произвольно выбранным из двух. Явно сказано, почему
это лучше: неверный путь отправляет читателя проверять невиновный файл.
docs/cli.md — почему `test` гоняет несколько скриптов разом, а `eval` по одному, и что
это следует из яруса, а не из флага. С замером 270s → 77s при побайтово одинаковом
результате, и с ценой: вывод буферизуется, живого стрима больше нет.
Все три — то, что было НАЙДЕНО, а не только что изменилось: через месяц полезно именно это.
* fix(docs): mark two illustrative enforce() blocks the old regex could not see
Разбор нашёл на два рефа больше, чем регулярка, и оба оказались СЛОМАННЫМИ — годами,
незаметно: `enforce("rubocop/Custom/NoDirectDbQuery")` и `enforce("pylint/no-direct-db-query")`
в справочных страницах скилла. Это примеры того, как объявить СВОЁ правило, то есть имена
заведомо несуществующие — им место под `<!-- vigiles:ignore -->`.
🔴 Почему регулярка их не видела, и это отдельный класс: она применялась ПОСТРОЧНО
(`for (const {lineNo, text} of blockLines) text.matchAll(CALL_RE)`), поэтому вызов,
разнесённый на несколько строк — а оба именно такие — не совпадал никогда. То есть у прежнего
детектора были не только ложные срабатывания (метод чужого объекта, комментарий, строка), но и
ложные ПРОПУСКИ на самой обычной форматировке. Разбор снимает оба класса разом: у AST нет
понятия «строка файла».
Итог на этом репозитории: 21 реф → 25 (после двух маркеров), все валидны.
* fix(compile): quote frontmatter scalars YAML would otherwise mis-read
`description: ${spec.description}` подставлялся сырым, поэтому описание с двоеточием и
пробелом — «…ловит ошибки, где замер врёт: неаналоги в выборке» — давало frontmatter,
который НЕ является валидным YAML. Замер 19.08 на двух реальных скиллах: сразу после
`vigiles compile` метод `skillContract()` отдаёт `malformed: true` и `declared: []`. Файл
ВЫГЛЯДИТ объявляющим `allowed-tools`, строгий парсер не читает ничего, скилл молча
наследует все инструменты сессии.
🔴 Это ровно тот дефект, который правило `frontmatter-valid` существует чтобы репортить —
то есть благословлённый путь производил то, что продукт ищет. Хуже: он НАКАЗЫВАЛ за
починку. Человек, закавычивший значение руками, получал на следующем линте расхождение
хеша («manually edited after compilation»), а перекомпиляция молча откатывала правку.
Проверка — КРУГОВОЙ ПРОГОН, а не список опасных символов: эмитим, читаем обратно тем же
загрузчиком, которым пользуется линтер, и квотим, только если вернулось не то, что ушло.
Два следствия этого выбора:
· уже безопасное значение эмитится побайтово как раньше — ни один скомпилированный файл не
меняется, ни один интегрити-хеш не двигается (подтверждено: все 1303 core-теста зелены);
· множество «опасных» входов не надо перечислять и поддерживать — решает сам YAML, поэтому
`#`, `[`, `&`, `*`, ведущий пробел, `yes`, `null` покрыты, не будучи названными.
Применено и к субагентам: описание там пишется так же свободно и ломает frontmatter так же.
Тесты — обе половины, и обе несущие (мутации с доказательством, что патч лёг):
· вернуть сырую подстановку → падает «compile quotes a description YAML would mis-read»
· квотить всегда → падает «leaves an already-safe description bare — no hash churn»
Вторая не менее важна первой: избыточное квотирование переписало бы каждый файл в каждой
репе и сдвинуло каждый хеш, что читается как «vigiles переписал мне весь репозиторий».
* feat!: doc-refs is opt-in, because it measured 0 true positives
The markdown code-block ref pass ran unconditionally and could not be turned
off. Measured across two real repositories on 2026-08-19 — 2 582 markdown
files, 52 builder refs — it produced ZERO true positives, and every error it
had ever raised was false: design prose sketching an API that does not exist
yet (`cmd("npm test")` for a package with no such script), and a third-party
CLAUDE.md captured verbatim as benchmark data.
The cause is structural rather than a threshold miss. A fenced block in prose
is a DRAWING of config, and this pass read it as config. No calibration fixes
that, because the block is not wrong — it is not config.
The consumer repo could only reach a passing lint by listing whole directories
in `exclude`, silencing every other check over a third of the tree; measured
afterwards the pass walked 604 files and found 0 refs — inert, still paying for
the walk. Off by default replaces that blunt instrument with an honest default.
BREAKING CHANGE: the markdown code-block ref pass no longer runs by default.
Enable it with {"rules": {"doc-refs": "error"}} — appropriate where markdown
genuinely is the source (a docs site whose fences are copied into live specs).
At "warn" findings print marked ℹ and the exit code is untouched; at "error"
they gate with exit 2 (they previously contributed exit 1, the soft tier,
because the pass fired unasked).
Known gap recorded for whoever raises the default again: the walker globs
without `dot`, so .claude/** — where deployed skills and agents live — has
never been scanned. The rule has never once judged a real instruction file.
* test(doc-refs): the GitHub annotation follows the configured severity
The inline-annotation test drove the pass with no config, which no longer runs
it. Extended to all three tiers rather than just re-enabled: "error" annotates
::error, "warn" annotates ::warning, and absent annotates nothing at all. The
middle case is the one worth asserting — a ::error rendered next to an exit code
of 0 is how an advisory rule gets read as a gate.
* chore: regenerate generated.d.ts, and stop two docs lines claiming doc-refs is on
CI caught what my five local gates could not: `npm run check` runs 18 stages
and I ran 5 of them. The new test file was missing from the generated project-
file union — 17/18, failing on `generated-types:committed`.
Also two sentences in docs/comparison.md still described the pass as
unconditional: the section's opening claim ("vigiles validates … calls inside
fenced TS/JS code blocks") and the Scope paragraph. Both now say opt-in; the
reader hits the opening line four paragraphs before the new note explaining the
default.
* ci(site): stop the browser install from eating the job
The site job was cancelled at its 15-minute bond twice in a row (runs
32300064395 and 32303343213), both times inside `playwright install --with-deps
chromium`, and neither failure was in the diff under review. The log shows the
cause plainly: `--with-deps` shells out to apt, apt hung on
azure.archive.ubuntu.com (`Ign:` on every line, then twelve minutes of silence),
and the orphan reaped at the end is `npm exec playwright install --with-deps`.
Three changes, one per cause:
- Drop `--with-deps`. ubuntu-latest already ships the libraries Chromium needs;
the flag exists for bare containers. Removing it removes the only part of this
step that touches a distro mirror. If a library really is missing, the browser
fails to LAUNCH in the next step within seconds and names the missing .so —
loud, and far better than a silent 15-minute hang.
- Cache ~/.cache/ms-playwright, keyed on the lockfiles. The step's floor was a
full ~150 MB download on every run, cache or no cache, good day or bad.
- Bound the step at 5 minutes. The job bond cannot tell "this step wedged" from
"the suite is slow", so a wedge cost the whole job; now it costs five minutes
and names the step that did it.
Measured baseline for the job: 3m35s.
* ci: run a job only when the commit can reach its inputs
This workflow had no path filter at all, so every PR paid for all five jobs
whatever it touched. Measured on this very PR (16 files across src/ and docs/,
none under site/): the site job built the engine, downloaded a browser and ran
the demo suite against code the diff could not reach — twice, at ~15 minutes
each, because the browser install wedged.
A workflow-level `paths:` cannot express this. It decides whether to run the
workflow AT ALL and says nothing about individual jobs. So the decision moves
into a `changes` job: one API call, no checkout, ~10s, feeding an `if:` on the
jobs whose inputs are disjoint from what changed.
Two flags, each conservative in the direction that costs minutes rather than
correctness:
root false ONLY when every changed file is under site/ — nothing outside
site/ imports anything inside it. Gates test, harness, e2e.
site false ONLY when every changed file is prose or agent config
(docs/, .claude/, root *.md). Gates site.
`check` stays unconditional: it runs types:site and cli-lint, so both halves of
the repo are among its inputs.
The site flag is a measurement, not an assumption. The demo imports the built
engine through three aliases, so any src/ change can reach it and must run it.
What cannot: grepping the three bundles it actually loads for the lint rule table
(DEFAULT_RULES, frontmatter-valid, doc-refs) returns zero in all three, and its
check pages are generated from site/src/checks/checks.ts rather than from docs/.
Tested in src/ci-path-filter.test.ts, which EXTRACTS the patterns from the
workflow rather than restating them — a test that agrees with its own copy of a
rule proves nothing about the rule that runs. A skipped job and a passed job look
identical in the checks list, so both directions are asserted. Three mutations,
each failing on its own assertion with the patch verified in place:
prose arm widened to any *.md → 'a markdown file NESTED in src'
prose arm widened to include src/ → 'runs everything for a normal src change'
empty-list bail flipped to false → 'an empty diff is not a licence to skip'
* test(ci): a job added later cannot silently escape the path gate
The filter itself was tested; the SHAPE of the workflow was not. In the checks
list a skipped job and a passed job are the same tick, so a job added tomorrow
with no `if:` looks exactly like a correctly-gated one that happened to run —
the filter quietly stops meaning anything and nothing goes red.
Exemptions are named with their reason rather than inferred, so granting one is
a visible edit: `changes` is the classifier itself (gating it on itself is a
cycle), `check` runs types:site and cli-lint so both halves of the repo are its
inputs. Plus a vacuity guard — if every job ends up exempt the assertion can no
longer fail, and an empty list would read as health.
Mutation, patch verified in place: stripping the gate off `site` fails with
"job(s) run on every commit with no path gate: site".
Mirrors the same guard added to mine's paper-gates.harness.mjs, where an ungated
19m42s job was what prompted it.
Gates, in order: npx vitest run (8 in this file) · npm run check 18/18.
* Revert "test(ci): a job added later cannot silently escape the path gate"
This reverts commit be267c7133.
* test(compile): state the frontmatter guarantee as a property, not one example
The colon-space test proved the colon-space case and nothing else. YAML has a
dozen ways to read a bare scalar as something other than the string written, and
a description is arbitrary human prose, so the guarantee belongs as a property
over that space: 29 adversarial scalars in, identical strings back; 4 safe ones
left bare so no consumer's integrity hash moves.
This is what lets a downstream repo delete its own check. Before, a consumer
proved the reader failed closed by hand-editing a compiled SKILL.md — text
surgery resting on how that corpus happened to be quoted, an assumption that
stops holding now that quoting is conditional. The emission side is provable
here; the hand-edit side is what frontmatter-valid and the integrity hash cover.
The 'trailing space' row went red on first run and the finding was about the
TEST: frontmatterScalar trims by contract, so it cannot witness a byte-exact
round trip. The compiler had emitted the value correctly. Oracle moved to the raw
parsed value, with the reason recorded inline so nobody 'fixes' it back.
Mutations, patch verified in place both times:
yamlScalar never quotes → 20 rows red, incl. every YAML indicator character
yamlScalar always quotes → all 5 bare-scalar rows red (the hash-churn side)
Gates in order: npx vitest run (44 in this file) · npm run check 18/18.
* fix(coverage): a script that could not LOAD did not run, so it must not retract
The retraction rule is right and stays: a script that RAN and proved nothing must
not keep yesterday's green record alive. It just does not describe a file the
runtime never evaluated. A module that fails to resolve executed no assertion — it
is the same 'did not run' as a missing claude CLI, arriving through another door.
Measured twice in one session on 2026-08-20: a container restore left an old
node_modules behind, every harness in a consumer repo died on `Named export
'recordCheck' not found`, and that repo's ledger fell from 48 records to 34 and
from 47 to 33. Nothing about those surfaces had changed — the machine had.
statusFor now also sees the child's output and classifies a loader error as skip.
Deliberately literal and only the loader's own vocabulary (ERR_MODULE_NOT_FOUND,
Cannot find package/module, the CommonJS named-export SyntaxError,
ERR_UNSUPPORTED_DIR_IMPORT, ERR_PACKAGE_PATH_NOT_EXPORTED) — those strings come
from Node's resolver, not from user code.
⚠️ It does not make a broken environment quiet. A skip still prints ⊘ SKIPPED and
--no-skip, which this repo's own CI passes, still fails the run. What changes is
only whether a machine problem may delete a measurement taken on a machine that
worked.
Mutation with the patch verified in place: reverting to a bare 'fail' fails on
'a module-resolution failure is a skip, not a fail'. The opposite direction is
asserted too — an AssertionError, a zero-check run and a passing run that merely
mentions a loader string all keep their old classification.
Gates: npx vitest run (32 in this file) · npm run check 18/18.
* feat(hook)!: compiled hooks are marked experimental, structurally
docs/experimental.md already states the principle — 'The prefix is the promise,
wherever the symbol lives' — and npm run experimental:check already fails CI when
a declaration tagged @experimental is not named for it. Compiled hooks were never
enrolled: grep found zero @experimental in hook.ts and hook-program.ts, so the
checker had nothing to compare and stayed silent. That is the checker's honest
limit — it catches a mismatch, not an absent marking, which is a human call.
Only the SIX ENTRY POINTS carry the prefix, and the placement is the argument:
every other name in the vocabulary — allow, deny, tool, pathView, commandView,
state, record, notice, run — is reachable only from inside a define* call, so
prefixing the chokepoint makes the marking structural for the whole family. Same
reasoning experimental_skill.input() is already documented with.
BREAKING CHANGE: defineHook, defineFileGate, definePromptGate, defineStopGate,
defineInject and defineReact are exported from vigiles/hook as
experimental_defineHook, experimental_defineFileGate, experimental_definePromptGate,
experimental_defineStopGate, experimental_defineInject and experimental_defineReact.
Alias at the import site so the word crosses the package boundary once and the body
stays readable:
import { experimental_defineInject as defineInject, state } from "vigiles/hook";
Docs, all three places asked for: a banner on docs/compiled-hooks.md (the feature
already has its own guide), a full section in docs/experimental.md, and the private
architecture note in CLAUDE.md.spec.ts. The experimental section states GAPS rather
than a disclaimer — named state landed 2026-08-12 and is untested by anyone but its
author; testing a stateful hook means reconstructing a runtime-private store path,
which the dogfood repo hard-codes and which broke when facts were renamed; compile
is not idempotent; two consumers, both the author's — plus what would have to be
true to drop the prefix.
⚠️ Three of my own defects on the way, all caught by gates rather than by review:
docs-imports flagged that the guide's examples still used the old names; a regex
that ran over one file twice produced 'experimental_defineHook as experimental_
defineHook as defineHook'; and six fixture STRINGS plus one dynamic-import
destructure held the old names in places a rename does not reach.
Gates: npx vitest run 3231 passed · npm run check 18/18.
* fix(examples): the rename missed examples/, and only the harness tier saw it
CI failed on examples/harness/compiled-hook-inprocess.harness.mjs. npm run check
cannot catch this — it type-checks src/ and never runs the example harnesses; the
harness job does, which is why the whole 18/18 suite stayed green over a broken
example.
Fourth instance of the same class in one change, and the pattern is worth naming:
a rename reaches declarations and imports, and does NOT reach a symbol living in
a fixture STRING, in a dynamic-import destructure, in a doc code block, or in a
directory the type-checker does not compile. Each was found by a different gate —
docs-imports, vitest, and now the harness tier — and none by reading the diff.
⚠️ refs-nudge.harness.mjs fails locally and passes in CI, and that is the
environment rather than this branch: CI pins the claude CLI to the
dialect-validated 2.1.187 (read out of src/dialect-drift.ts), this container has
2.1.237, and the file is byte-identical to main's.
* fix(skill-resources): a file the skill WRITES is not a file it ships
A consumer skill's line was reported as a missing bundled resource:
API responses are cached to `scripts/.cite-cache.json` (gitignored) so
re-runs are cheap.
Isolated to one word with a three-line fixture differing only in its tail. The
same sentence ending "so subsequent invocations are cheap" stayed silent, and
so did the same sentence with no tail. Only `re-runs` fired: JavaScript's `\b`
counts a hyphen as a word boundary, so USE_DIRECTIVE's `runs` matches INSIDE
it, and a plural noun about repetition was read as an instruction to run the
file.
Fixing the hyphen was rejected by that same measurement — "re-run
`scripts/x.sh` before submitting" is a genuine directive, so a rule about
hyphens trades this false positive for a false negative. The honest
discriminator is that the line says the file is MADE, not read, and
`gitignored` is its strongest form: a file the author states is outside the
repository cannot be shipped inside it.
GENERATED_FILE_CUE is deliberately separate from ILLUSTRATIVE_CUE. That one
vetoes a HYPOTHETICAL mention; this one vetoes a mention that is entirely
concrete and still cannot name a bundled resource. It is the read the module
already takes: USE_DIRECTIVE excludes authoring verbs (store/create/add/move/
write) for this exact reason, and simply never covered the writing side.
Both candidate shapes consult it — the measured line is an inline path, but
the same sentence as a markdown link would slip past a one-sided fix.
Mutation-proven, each with evidence the patch landed:
| mutation | fails on |
| --- | --- |
| veto removed from the inline shape | the hyphenated-word case |
| veto matches everything (`/./`) | "still flags a REAL directive" (+4 true positives) |
| veto removed from the link shape | the written-at-runtime link case |
End-to-end on the consumer repo: the finding is gone, and the one remaining
`✗` there is unrelated.
Gates, in order: `npm run check` 18/18 · `npx vitest run` 3236 passed. One
suite fails to LOAD in this checkout — `scripts/release-preset.test.ts` cannot
resolve `@semantic-release/commit-analyzer`, which is in package.json but not
in this node_modules. A partial install, not a result: it did not run, so it
proves nothing either way.
---------
Co-authored-by: Claude <noreply@anthropic.com>
13 KiB
13 KiB
What Changes With vigiles
Claude Code
| Without vigiles | With vigiles | |
|---|---|---|
| Instructions | Hand-written CLAUDE.md | Compiled from .spec.ts (build artifact) |
| Linter rule references | Trust-based (nobody checks) | Verified at compile time against real config |
| File paths | Rot silently when renamed | file() references checked against filesystem |
| Commands | Stale scripts go unnoticed | cmd() references checked against package.json |
| Direct edits to CLAUDE.md | Anyone can, nobody knows | PreToolUse hook blocks edits, redirects to spec |
| Linter config changes | CLAUDE.md drifts out of sync | PostToolUse hook auto-regenerates types |
| Spec edits | N/A | PostToolUse hook auto-compiles to markdown |
| guidance → enforce upgrades | Manual guesswork | /strengthen reads per-linter docs, suggests upgrades |
| New lint rules from PR feedback | Copy-paste from review | (planned) a rule-synthesis skill generates rule + tests + spec entry |
| Does a skill/plugin actually help? | Unknown — stars + vibes | A/B measured on real tasks: bill + correctness (measuring-skills) |
| CI | Nothing to verify | vigiles lint catches hash drift, disabled rules, stale refs |
Codex
| Without vigiles | With vigiles | |
|---|---|---|
| Instructions | Hand-written AGENTS.md | Compiled from .spec.ts |
| Linter rule references | Trust-based | Verified at compile time |
| File paths / commands | Rot silently | Checked at compile time |
| Direct edits to AGENTS.md | Undetected | CI catches hash mismatch |
| Hooks / auto-compile | Not available (no plugin system) | Not available — run vigiles compile manually or in CI |
| CI | Nothing to verify | Same vigiles lint pipeline as Claude |
Codex has no hook or plugin system. The compile-time verification and CI enforcement still work — the difference is there's no auto-recompilation on edit. You run vigiles compile before committing, and CI catches drift.
What's Deterministic vs What's Not
Everything vigiles compiles and lints is deterministic — same input, same output, no LLM in the loop. The non-deterministic parts (authoring specs, suggesting upgrades, writing custom rules) are agent skills that run outside the compilation pipeline.
| Check | Deterministic? | How |
|---|---|---|
| Linter rule exists | ✅ Yes | Node API (builtinRules) or CLI (ruff rule, rubocop --show-cops) |
| Linter rule is enabled in config | ✅ Yes | calculateConfigForFile (ESLint), --show-settings (Ruff), --show-cops (RuboCop) |
| Cedar policy exists | ✅ Yes | Scan .cedar/ and cedar/ for @id("...") annotations, with filename fallback |
| File path exists | ✅ Yes | fs.existsSync |
| npm script exists | ✅ Yes | Parsed from package.json |
| SHA-256 hash matches | ✅ Yes | Recompute and compare |
| Duplicate rule detection | ✅ Yes | Normalized Compression Distance (NCD) with fixed threshold |
| Orphan docs detection | ✅ Yes | Scan configured doc directories for .md files no other markdown references |
| guidance → enforce suggestion | ❌ No | Agent reads linter docs, reasons about intent — /strengthen skill |
| PR comment → lint rule | ❌ No | (planned) agent generates custom rule code via a rule-synthesis skill |
| Spec content authoring | ❌ No | Agent or human writes the spec — vigiles verifies it |
What vigiles Does and Doesn't Validate in Markdown
vigiles validates vigiles-specific things in .md files: <!-- vigiles:enforce ... --> comments (inline mode) and vigiles: YAML frontmatter rules (frontmatter mode) — both on by default. It can also validate enforce("...") / file("...") / cmd("...") / ref("...") calls inside fenced TS/JS code blocks, but that one is opt-in (doc-refs, default off — see below). Same engines used for .spec.ts references.
Out of scope — use other tools:
| Concern | Use instead |
|---|---|
| Markdown formatting / structure | markdownlint |
| TS / JS syntax in code blocks | eslint-plugin-markdown |
| TS type-checking of code blocks | twoslash (TS official docs, Astro, etc.) |
| Markdown link validity (URLs, paths) | markdown-link-check |
| Spell / prose / grammar | Vale, alex |
Opt-in as of 2026-08-19 (doc-refs, default off). Measured across two real repositories —
2 582 markdown files, 52 builder refs — this pass produced 0 true positives, and every error it
had ever raised was false: design prose sketching an API that doesn't exist yet, and a third-party
CLAUDE.md vendored as benchmark data. The cause is structural, not a threshold: a fenced block in
prose is a drawing of config, and the pass read it as config. Turn it on with
{"rules": {"doc-refs": "error"}} where markdown genuinely is the source. Full measurement and the
known gap: docs/rules/doc-refs.md.
What counts as a ref (2026-08-19). Blocks are PARSED, not pattern-matched. A ref is a call
whose callee is a bare enforce / file / cmd / ref identifier with a string-literal first
argument. That means a method call on some other object (ctx.file("OUT")), a mention inside a
comment (// cmd("npm test")), and a string containing one ('cmd("x")') are all NOT refs — none
of them is a call expression to the builder. Until this release those three were matched by a
regex and reported as broken refs; \b sits happily after a ., and a regex has no notion of a
comment or a string literal. Parsing makes all three inexpressible rather than individually
excused.
Scope (when enabled). The pass reads **/*.md from the repo root and honours the top-level exclude in
.vigilesrc.json, so vendored or benchmark markdown (a third-party CLAUDE.md captured verbatim
as test data) can be kept out of it. Before 2026-08-19 exclude was not applied here at all, which
meant a repository vendoring other people's markdown had no way to reach a clean lint — and a
lint that cannot exit 0 gets its exit code discarded, at which point it gates nothing.
Illustrative code blocks (typo demos, template placeholders, speculative refs in design docs) opt out via <!-- vigiles:ignore --> immediately before the fence, or <!-- vigiles:ignore-file --> anywhere in a file that's entirely illustrative. Placeholders containing < or > are auto-skipped. Refs that can't be verified because the underlying tool isn't installed (e.g. pylint/X on a machine without pylint) are reported separately from real errors.
What vigiles composes with
vigiles owns one thing: compile-time verification of typed specs against real
linter configs, filesystems, and package.json, plus testing the harness those
specs describe. Everything else, compose:
- Architectural linting — ast-grep, Dependency Cruiser, Steiger. Reference their rules via
enforce(). - File sync across agents — Ruler, rulesync, block/ai-rules. vigiles compiles the source; sync tools distribute. For non-markdown formats (
.cursorrules, Copilot), rule-porter or rulesync convert the compiled output. - Markdown linting — markdownlint. vigiles generates markdown; structure is correct by construction.
- Code-block linting in docs — eslint-plugin-markdown for syntax, twoslash for TS type-checking.
- Prose quality — Vale. Different concern.
- Runtime LLM rule checking — opposite paradigm: those tools send your code to a model on every check (tokens, non-reproducible verdicts); vigiles compiles once and checks deterministically forever after with
eslint,ruff,tsc, Cedar evaluation.
Specs compile to CLAUDE.md by default; set target: "AGENTS.md" or
target: ["CLAUDE.md", "AGENTS.md"] for multiple outputs from one spec. See the
spec format reference.
Flow
DETERMINISTIC AGENT-ASSISTED
┌─────────────────────────┐ ┌──────────────────────────┐
│ │ │ │
.spec.ts ──────┤ vigiles compile │ │ /strengthen │
│ │ ✓ linter rules exist │ │ guidance → enforce │
│ │ ✓ rules enabled │ │ │
│ │ ✓ file paths valid │ │ (planned) │
│ │ ✓ commands valid │ │ rule synthesis skill │
│ │ → CLAUDE.md + hash │ │ │
│ └─────────────────────────┘ │ /edit-spec │
│ │ agent edits .spec.ts │
│ ┌─────────────────────────┐ └──────────────────────────┘
└────────▶│ vigiles lint │ │
│ ✓ hash integrity │ │
│ ✓ inline rule checks │ ▼
│ ✓ duplicate detection │ ┌──────────────────────────┐
│ ✓ orphan docs check │ │ hooks (Claude Code) │
│ ✓ coverage gaps │ │ auto-compile on edit │
└─────────────────────────┘ │ auto-regen types │
│ block direct md edits │
└──────────────────────────┘