mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
v1
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zernie
|
f52fa12c4f |
fix(lint): spec-refs reaches every compiled surface, and compile emits prettier-clean artifacts (#195)
Closes #190. spec-refs re-derives a compiled file's references from its spec, but called compileClaude directly and skipped every spec whose _specType was not "claude". Four of the five compiled surfaces — skills, subagents, railway orchestrators, generated harness files — got no re-derivation at lint time, so an artifact committed while its refs were live stayed green forever after the target was deleted. Measured here: compile reported `Skill not found` on examples/SKILL.md while lint reported `hash valid`. Smaller than the issue estimated. #190 expected a per-type ref accessor because each compiler reaches its refs differently; they do not — all four already return the same errors: CompileError[], and only the options differ. One exhaustive dispatcher, so a fifth spec type is a tsc error rather than a silent skip. compile also emitted artifacts that prettier --check then rejected, on all seven examples/ artifacts — which is why they sat un-recompiled carrying a retired marker position, and which blocked the fix above. Two independent defects: both frontmatter renderers padded inside the --- fence, and the frontmatter/body join added a blank line placeIntegrityHeader then added again. Fixed at the join, not in the stamper: seal hashes the content it receives, so trimming there hashes one string and writes another (measured — it broke integrity on three artifacts). Also: check.mjs now declares and prints the CI jobs it does NOT run, with a test asserting that list against ci.yml. It covered one job of six while its name and its "18/18 passed" read like all of them, and that cost a cycle. The test earned itself immediately — the hand-written list missed e2e and named two commands that do not exist. Also: @openai/codex is pinned. latest was 0.153.1, whose tarball 404s while its metadata still resolves, so the install step failed on every branch until pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
94af23a5b7 |
feat(hook,cli)!: close five adopter-reported issues — halted-turn guards, dead-ref artifacts, centralized tests (#186)
Closes #173, #174, #175, #176 and #170. #174 — decideHook read exit 2, `decision` and `permissionDecision` but not the harness's halt-the-turn field, so `assertBlocksDisasters` reported a guard that stopped every disaster as blocking none. The field is read from the PORT, not hard-coded: `continue` is documented for Claude Code and unverified for Codex. The issue's adjacent suggestion was deliberately NOT implemented — adding the halt to `hook-block-ineffective` as a block-attempt signal would fire on hooks that WORK, since a halt is not event-scoped. It reads as a suppressor instead, fixing a real false alarm. A mutation implementing the original suggestion literally is pinned by a test that fails on it. #173 — compile printed its errors, exited 1, and wrote a hash-valid CLAUDE.md anyway, so `lint` went green over refs it had just called dead. Nothing is written on a failed compile now. #176 — eight small defects, including `audit --out` outside the repo appending unreachable `../../..` entries to .gitignore (a documented read-only report editing a tracked file), an unknown command exiting 1 against a documented 2, and STABILITY.md promising the removed `scan` verb. #175 + #176.2 — three missing capabilities, each opt-in and additive: `{surface}` in testGlobs, `--single`, and `rulesDir` so `.claude/rules/**` stops being invisible. #170 — one enumeration of export forms written against the grammar, both naming checks driven off it, and `check-internal-tag.mjs` asking the TypeScript compiler instead of a regex. It immediately found the issue's open case, 13 tagged types the regex had never seen, and a live gap in the ESLint rule. BREAKING CHANGE: `HookRunResult` gains a required `haltsTurn`, so code that CONSTRUCTS one (a test fake) must set it. Reading a result is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
zernie
|
900dc7e06a |
refactor(spec)!: name things for what they are; retire the vigiles/experimental subpath (#169)
Three exported symbols were named for something other than what they are, and
two mechanisms were doing one job.
RENAMES (old names kept one major, @deprecated, removed one major AFTER this one):
claude() → instructionFile() compiles CLAUDE.md AND AGENTS.md, so a
name from one harness was never right
instructions`` → prose`` builds a prose FRAGMENT; the plural read
as "the instruction file", which is what
instructionFile() builds
agent() → experimental_agent() the shape is not settled, and the tag was
never there (git log -S empty)
SkillSpec.result: → postcondition: `result` already meant the subagent output
contract one screen down; the doc comment
had to spend a line disambiguating them
The alias window is not politeness. Measured: the previous rename shipped with
ZERO overlap (18.1.1 old names only, 19.0.0 new only). A container rollback put
the old node_modules under new sources, PreToolUse stopped loading, and a
PreToolUse that does not load denies EVERY Bash command — including the one that
fixes it. One window per major makes that pair unrepresentable.
SUBPATH RETIRED. vigiles/experimental is gone; its members moved to the door
their feature already uses. Of two mechanisms for one promise the subpath was the
weaker: it marks the import line, out of view by the time anyone reads the call.
Measured on the only user-facing example — with the prefix aliased away at
import, the marker survived at 0 of 5 call sites. So the prefix stays, the advice
to alias it away became an explicit prohibition, and the six define* entry points
are DECLARED prefixed rather than gaining it at re-export.
That de-aliasing is what unblocked moving the naming check into ESLint, and the
ORDER was load-bearing: measured before it, dropping the public/internal
exemption produced 14 findings, all from the re-export indirection. Measured
after, across all of src/ with no exemption: 22 tagged declarations, 0
unprefixed. The rule turns on silent.
EXPORT CLEANUP. vigiles/linting export * → curated, 103 → 79 symbols; its header
claimed to be curated with an `export *` one line below, and claimed the builders
were also at the package root (191 exports there, zero matches). ClaudeTool and
HookEvent deleted — zero references, and both had drifted off the dialect they
claimed to mirror; HookEvent listed 5 events, two of which do not exist, against
the 31 the vendor documents.
DOCS. experimental.md deleted and distributed per feature: the skill half into
skills.md §Status, the hook half into a new compiled-hooks.md §Status, and the
emit half — 111 of its 177 lines, a topic doc wearing a stability page's clothes
— into docs/emit-channel.md, which gains the "which channel when" comparison it
never had. The one cross-cutting sentence went to the README.
TWO GATES WERE GREEN AND WRONG, both found by this work:
- api:check reported "verified for 11 entries — no drift" while reading
dist/experimental.d.ts from a PREVIOUS build (tsc does not clean dist/) for a
source file that no longer existed. Now asserted against package.json
exports, with a mutation proving exit 1 on a phantom entry.
- check-export-prefixes caught MY error: the R3 services first landed on
vigiles/eval, where `paid_` means "this call can bill you" — and
startServices takes a ContainerRuntime and calls no model. Naming it
paid_experimental_startServices would have redefined paid_ as "expensive in
some sense". They went to the package root.
SEVENTEEN review findings landed on this branch, all real. Two mechanical
closures came out of them: src/example-imports.test.ts resolves every relative
import under examples/ instead of matching a string, and the naming rule now
ENUMERATES the three export paths it visits rather than describing its coverage
in the abstract. One attempt is recorded as REFUTED: teaching the doc gate to
flag our own exports as free variables measured 66 findings across 93 blocks, so
that gate's original scoping was right.
BREAKING CHANGE: vigiles/experimental is removed. Its exports are unchanged in
name and behaviour and are now reached from `vigiles` (the emit channel, the R3
disposable-service tier). Every renamed symbol keeps a working @deprecated alias
for this major.
|
||
|
zernie
|
34a4f7a556 |
fix: no raw NUL bytes in source, and a gate that keeps it that way (#160)
src/scan-files.test.ts carried two literal NUL bytes inside a template literal where the escape was meant. Git decides "binary" per blob by looking for a NUL, and a binary blob gets no line diff and NO THREE-WAY MERGE — so every branch touching the file conflicted, unresolvably, because there were no hunks to pick. Measured: with the bytes escaped, `main <- fix-scan` and `main <- fix-shadow` each merge with ZERO conflicts. They had been parked for a day as "conflicting with each other"; they never were. The escape is the form the rest of the codebase already uses (skill-contract, scan-core, coverage-artifact, coverage-probe — eight sites), and the string it produces is byte-identical, so scan-files.test.ts still passes its 21 tests. Adds scripts/check-text-sources.mjs as an 18th `check` gate. It refuses to pass on an empty scan, since a scan finding nothing is indistinguishable from a scan looking in the wrong place. Proven by mutation: planting one raw NUL fails it with the file, the count and the offset. Not a .gitattributes marker: that restores diff and merge while leaving the byte in the source, hiding the defect instead of removing it. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
f8280a93db |
fix(release)!: feat! не давал релиза ВООБЩЕ — ломающие уехали как патч (#152)
Замерено настоящим анализатором на настоящих сообщениях, а не по памяти. При дефолтном пресете (angular) `analyzeCommits` на `feat!:` возвращает **null** — не «минор вместо мажора», а НИ ОДНОГО релиза: пресет не распознаёт `feat!` как тип, и ни одно правило не срабатывает. | сообщение | angular | conventionalcommits | |---------------------------------------------|---------|---------------------| | `feat!:` без футера (наш #149) | null | major | | `feat!:` + `BREAKING:` в теле (наш #147) | null | major | | `feat:` + футер `BREAKING CHANGE:` | major | major | | обычный `feat:` | minor | minor | | обычный `fix:` | patch | patch | | `docs:` | null | null | Следствие серьёзнее аномалии: код #147 (рейм публичного `skill()`) и #149 (разделение экспортов) всё равно уехал на npm — его вынесли следующие коммиты, под `v15.4.0` (minor) и `v15.4.1` (patch). То есть ломающие изменения отгружены как ПАТЧ; кто стоит на `^15.4.0`, получил их молча. 🔴 Что делает это ловушкой, а не опечаткой: `pr-title.yml` гоняет `amannn/action-semantic-pull-request`, который `feat!` ПРИНИМАЕТ — джоб `validate` был зелёным на обоих PR. Один механизм говорит автору, что маркер валиден, второй его выбрасывает, и оба молчат. Починка: пресет `conventionalcommits` у commit-analyzer и release-notes-generator (второй — чтобы changelog и версия читали одну грамматику). Плюс `conventional-changelog-conventionalcommits` в devDeps. Тест `scripts/release-preset.test.ts` закрепляет всю таблицу, включая пол (`docs:` → null): пресет, бампающий на каждом коммите, прошёл бы верхние строки и всё равно был бы неверен. Прогнан мутацией — возврат к bare-string конфигу роняет все шесть; сообщение ошибки при этом само объясняет, что bare string означает angular. Цена: `@semantic-release/commit-analyzer` в devDeps ради поведенческого теста, дельта в node_modules замерена — 134 пакета (381 → 515). Альтернатива «проверять форму конфига» отвергнута: она прошла бы и при неверном имени пресета, и при отсутствующем пакете. Гейты: npm run check 17/17 ✓ · vitest 2960 ✓. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
1ebef8320a |
ci: one list of commands instead of two — CI now calls npm run check (#151)
* ci: один список команд вместо двух — CI зовёт `npm run check` Сегодня три PR уехали в `main` красными: список гейтов гонялся по памяти (пять команд), а джоб `check` гонял тринадцать. Упавшая — `tsc -p test/types/tsconfig.json` — живёт в ОТДЕЛЬНОМ tsconfig, которого корневой тайпчек не видит, и три `@ts-expect-error`-ассерта тихо перестали срабатывать. Длинный список, который надо помнить, это не чинит. Чинит то, что список СТАЛ ОДИН и CI его ВЫЗЫВАЕТ: `scripts/check.mjs` + `run: npm run check` вместо тринадцати шагов. Это стандартный паттерн поля (pre-commit / just / mise): определение задач — источник правды, CI его дёргает. Прежняя моя идея «завести гейт на совпадение двух списков» снята: она была подпоркой под дубль, который правильнее убрать. Стадии несущие, а не косметика. Часть команд ПИШЕТ файлы, которые другие ЧИТАЮТ (`generate types` → `.vigiles/generated.d.ts`, `api-extractor` → `api-surface/*`, `build` → `dist/`), поэтому писатели сериализованы, а параллельно идёт только read-only группа. 17/17 за 70 с против ~3 мин последовательно. Имена падающих шагов не потеряны — раннер печатает, что именно упало, и `--list` показывает команды без запуска. 🔴 Два шага `uses: ./` НЕ входят в `npm run check` намеренно: они догфудят собственный composite Action, у которого нет shell-эквивалента. Один из них (`eval-check`) при первой правке был снесён случайно — он стоял между двумя command-шагами, и правка по диапазону его проглотила. Возвращён, причина записана прямо над ним. Раннер сразу поймал два собственных промаха: неотформатированный `check.mjs` и протухший `generated.d.ts` после добавления скрипта `check`. Проверено: `npm run check` 17/17 ✓ · ci.yml парсится. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: словарь скилла уехал на сам билдер — experimental_skill.input/.step BREAKING: `input` и `step` больше не самостоятельные экспорты. Достаются как `experimental_skill.input(…)` / `.step(…)`. Пользователей нет. Зачем. Префикс `experimental_` — привычка, а привычка уже один раз протекла: `skill()` отгрузился со стабильным именем при доке «`skill()` is experimental». На самих `input`/`step` тега и префикса не было вообще, то есть словарь экспериментальной фичи выглядел стабильным. Повесив их на билдер, маркировку делаем СТРУКТУРНОЙ: до `input()` не добраться, не назвав `experimental_skill`. Забрано ровно два символа, и это замер, а не вкус. Перепроверено независимо от вердикта субагента: `input` и `step` встречаются только в скилл-спеках, а `cmd` (5 агентских спек), `file`, `ref`, `result` (8 агентских) общие — их спрятать в неймспейс скилла нельзя. 🔴 По дороге закрыта асимметрия, которую правка иначе бы УХУДШИЛА: дверь `vigiles/claude-code` своих `input`/`step` не экспортировала никогда, и типизированный автор брал их из второй двери. После снятия standalone- экспортов он остался бы без них совсем. Теперь CC-билдер несёт те же хелперы. Форма — `Object.assign`, не `export namespace`: последний запрещён нашим же линтом (`no-namespace: error`). Генерик пережил: в api-репорте `typeof skillSpec & { input; step }`, `ae-forgotten-export` нет. Честная граница записана и в коде, и в доке: `const { input } = experimental_skill` снимает маркер внутри файла. Гарантия у́же — немаркированное имя не пересекает границу пакета. Доки: `docs/skills.md` — пример на алиасе (`experimental_skill as skill`), маркер держится на строке импорта. `docs/experimental.md` — вступление утверждало, что префикс носит всё из `vigiles/experimental`, что после переезда `experimental_skill` в `vigiles/spec` стало неверным; плюс своя секция. `docs/README.md` — доиндексированы `testing-matrix.md` и `non-js-harnesses.md`. `safety.md` намеренно НЕ индексирован: он сам пишет «INTERNAL / WIP — not surfaced in the public docs yet». Гейты: npm run check 17/17 ✓ · vitest 2954 ✓. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
9413061d9b |
feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
b9302db4e6 |
feat!: one instability marker — the experimental_ name prefix, enforced (#147)
* docs: генераторный режим убран из публичной доки и припаркован `skill()` помечен экспериментальным. Из `docs/skills.md` убраны §3 (генератор) и секция тестов `vigiles/skill-test`, которая на нём стоит: `genSkill`/`act`/`checkpoint`/`finish`/`runSkill` не достижимы ни из одного сабпаса пакета — то есть снаружи репозитория такой скилл не написать. Ровно эти два фантомных сабпаса и заставляли держать в шапке рукописный баннер; баннер снят вместе с ними. Замерено при этом, и записано в §Status как причины экспериментального статуса: - `vigiles compile` генераторы ПОДДЕРЖИВАЕТ (`cli.ts` ловит `genSkill(`, прогнано end-to-end) — прежняя строчка §Status «wiring it into `vigiles compile` … is the next integration step» устарела; - скомпилированный `SKILL.md` ни разу не был показан загружающимся как скилл: `vigiles:sha256:`-шапка отодвигает YAML-фронтматтер с первой строки, и читатель, привязанный к `^---`, не находит фронтматтера вовсе. Все работающие `SKILL.md`, включая 13 своих, начинаются с `---`; шапку несут только два файла в `examples/`, которые никто не грузит; - принятие всё-или-ничего: `renderSkillSections` собирает документ целиком в фиксированном порядке, пути «моя проза + один проверяемый гейт» нет. Паркинг помечен в шапках трёх модулей, чтобы его не подняли молча. Публично экспортировать нечего: замер показал ноль достижимых символов. `scriptModel` НЕ паркуется — он публичен из `vigiles/claude-code` независимо от генератора. Заодно — дефект гейта из #146, найденный этой же правкой: он переиспользовал маркер `vigiles:ignore-file`, который принадлежит резолверу ссылок. Из-за этого единственный документ с двумя доказуемо битыми импортами был единственным, которого гейт не видел (91 блок сканировался, этот — нет). У гейта теперь свой маркер `vigiles:ignore-imports`; на дореформенном тексте он выдаёт ровно те два TS2307, ради которых писался баннер, на текущем корпусе молчит (92 блока, 0 находок). Тест на разделение маркеров прогнан мутацией: возврат общего маркера роняет оба ассерта. Гейты: vitest 2928 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · check-doc-imports 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: skill() → experimental_skill(), и правило, которое это держит BREAKING: `skill()` переименован в `experimental_skill()` на `vigiles/spec`, `vigiles`, `vigiles/linting` и `vigiles/claude-code`. Пользователей нет, мажор бампаем. Почему: дока открывалась словами «`skill()` is experimental», а имя на экспорте говорило «стабильный». Проза против механики — ровно тот дефект, который vigiles и ищет, у vigiles. Соглашение о префиксе существовало (`experimental_emitTool`, `experimental_parseEmitted`, `experimental_withServices`), нигде не было записано и ничем не проверялось. Побочно рейм снимает КОЛЛИЗИЮ, которой не должно было быть: `skill()` экспортировался из ВОСЬМИ сабпасов и был ДВУМЯ разными функциями — билдером спеки (`vigiles/spec`) и `Check<Trace>` по трассе (`vigiles/testing`, принимает id-строку, отвечает «сработал ли скилл»). Одно слово, два понятия, различимые только дверью импорта. Префикс несёт только билдер; `Check` не тронут (`vigiles-testing.api.md` без изменений). Правило: `scripts/check-experimental-naming.mjs` + шаг CI + `npm run experimental:check`. Публичное объявление с TSDoc-тегом `@experimental` обязано начинаться с `experimental_`. Ни одного нового маркера: тег уже используется в `experimental-emit.ts` и `services.ts`, публичность берётся из закоммиченных `api-surface/*.api.md`, которые `api:check` и так держит честными. Типы вне области: на дореформенной поверхности префикс несли только функции, а `ServiceSpec`/`ServiceHandle`/`ContainerRuntime` — нет; с типами гейт открылся бы 6 косметическими переименованиями против 1 настоящей находки и был бы заглушен в тот же день. Настоящая находка одна и закрыта: `makeDockerRuntime` → `experimental_makeDockerRuntime`. 8 тестов гейта, обе половины у каждого. Один нашёл баг в самом гейте: `\s+\S` в маркере-исключении удовлетворялся переводом строки и звёздочкой продолжения JSDoc, то есть голый маркер без причины проходил. Поправка к предыдущему коммиту: §Status утверждал, что sha256-шапка мешает загрузке скилла. Замер это ОПРОВЕРГАЕТ для соседней поверхности — в `reviewer-ab.eval.mjs` живой Claude Code загрузил скомпилированный `agents/code-reviewer.md` С шапкой через `--plugin-dir` и отдиспатчил в него, 100% прогонов на реальном sonnet (2026-06-20). Формулировка сужена до «непроверено», а не «сломано». Гейты: vitest 2936 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал устаревший импорт в docs/railway-subagents.md — правка внесена) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * docs: STABILITY.md объявлял стабильным то, что переименовано и помечено экспериментальным Найдено субагентом при разборе неймспейс-идеи, подтверждено мной. `STABILITY.md` — документ под заголовком «What's stable — depend on it» — перечислял `skill` среди стабильных билдеров `vigiles/spec`. После переименования такого символа нет вообще, а то, во что он переименован, в предыдущем коммите объявлено экспериментальным. То есть контракт стабильности обещал стабильность несуществующему имени. Третий экземпляр «проза против механики» в этой же области за два дня, и мой гейт его НЕ ловит: он сверяет TSDoc-тег с именем, а не документ с API. Заодно записано то, что вскрылось при проверке: - `input()` и `step()` используются ТОЛЬКО скилл-спеками, но экспортируются без префикса и без тега — то есть словарь экспериментальной фичи выглядит стабильным. Слепая зона гейта: он энфорсит «тег ⇒ имя», но не «на всём, что должно, стоит тег». - Маркеров нестабильности в проекте ТРИ и правила выбора между ними нет: `@internal` (39 объявлений), `@experimental` (18), префикс `experimental_` (8 публичных символов). Секция больше не утверждает, что маркер один. - У `pipe` тег `@internal` стоит только на ПЕРВОЙ перегрузке, поэтому остальные в api-репорте читаются как `@public (undocumented)`. Гейты: prettier ✓ · docs:check 0 · experimental:check 0 · lint 0 errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * feat!: один маркер нестабильности вместо трёх — префикс имени BREAKING: `pipe` `pipeStep` `needs` `start` `andThen` `effect` → `experimental_*` на `vigiles` и `vigiles/linting`. Пользователей нет. ## Выбрано **Экспортируем и не стабильно — имя начинается с `experimental_`.** Тег `@experimental` объявляет это один раз, `experimental:check` роняет CI, если у помеченного публичного символа нет префикса — тег и имя разъехаться не могут. `@internal` перестал означать «нестабильно»: он отвечает на ДРУГОЙ вопрос — входит ли символ в API вообще — и верен только на том, что мы не экспортируем. Почему имя, а не тег: тег не виден там, где решают. Имя видно в каждом вызове, дифе, ревью и автокомплите. Ровно так протёк `skill()` — стабильное имя при доке «`skill()` is experimental». ## Исправлено всё Было три маркера без правила выбора: `@internal` 39 объявлений, `@experimental` 18, префикс 8 публичных символов. Найдено 6 противоречий — символы с `@internal`, которые ЭКСПОРТИРУЮТСЯ; все шесть переименованы, теги переведены на `@experimental`. ## Два дефекта в гейте, отгруженном час назад 1. 🔴 Тег читался только если стоял на строке ОДИН (`@experimental\s*$`). Реальные теги несут прозу — `@internal Experimental typed-composition surface — …` у `pipe`, два в `services.ts`. Замер: **2 из 8** `@experimental` и **31 из 39** `@internal` объявлений были гейту не видны. Строгий паттерн выглядел безопаснее и был наоборот. 2. Между JSDoc и объявлением может стоять комментарий (`eslint-disable` перед перегрузками `pipe`) — привязка к следующему символу его теряла. После починки гейт видит 14 объявлений вместо 6. ## Записано `STABILITY.md` — секция «The one rule» + список; `CLAUDE.md.spec.ts` → перекомпилированный `CLAUDE.md` (правило №51). Плюс честно записанная дыра: гейт энфорсит «помечено ⇒ названо», но НЕ «всё, что должно быть помечено, помечено» — живой пример `input()`/`step()`, словарь только скилл-спеков, без тега и префикса. ## Третий раз за сессию Фикстура-спека в шаблонной строке (`generate-harness.test.ts:505`) не видна ни tsc, ни грепу по импорту — ловится только прогоном тестов. До этого то же в `cli.test.ts` (дважды). 5 новых тестов гейта (13 всего), включая закрепление обоих дефектов выше. Гейты: vitest 2941 ✓ · lint 0 errors · prettier ✓ · api:check no drift · build ✓ · docs:check 0 (поймал `docs/cli.md:646`) · experimental:check 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
f63f76da3b |
ci: проверять, что примеры в доках импортируют существующее (#146)
Этого не проверял никто, и 16.08 доки несли ШЕСТЬ сабпасов, дающих ERR_PACKAGE_PATH_NOT_EXPORTED, плюс два примера, импортирующих настоящий символ не из того барреля: `scriptModel` (живёт в `vigiles/claude-code`) и `runHookProgram` (`vigiles/hook`). Всё копипастится целиком и падает на первой строке читателя. Про `scriptModel` это был ТРЕТИЙ экземпляр одного и того же неверного импорта, дважды до этого записанного как известная граница. Проза его не удержала — этот шаг удержит. 🔴 ДОКЛАДЫВАЕТ ТОЛЬКО СОБСТВЕННУЮ МОДУЛЬНУЮ ПОВЕРХНОСТЬ, И ЭТА УЗОСТЬ — ВЕСЬ ДИЗАЙН. Замер по 102 ts/js-блокам этого репо: полный `tsc` на каждом блоке — дефолт всех трёх готовых инструментов — даёт 234 находки, 2 настоящих и 232 ложных, потому что самодостаточны лишь 36,3% блоков, остальные иллюстративные фрагменты со свободными переменными. Здесь 2 из 2, и ни одного маркера подавления во всём репозитории. Инструмент с шумом 1:24 — не более строгая версия этого, а мёртвая: проверку, срабатывающую на нормальном входе, глушат за неделю. Rust идёт обратным путём (компилировать всё, отписываться `ignore`); у нас это ~65 маркеров подавления в первый же день, видимых каждому читателю на GitHub. Инверсия стоит ноль и делает ложное срабатывание НЕВЫРАЗИМЫМ, а не отфильтрованным. Полный тайпчек блока — opt-in маркером `<!-- vigiles:check -->`. Резолвинг не требует переписывания путей: TypeScript резолвит пакет сам на себя через `exports`, поэтому фантомный сабпас — обычный TS2307. МУТАЦИОННЫЙ ПРОГОН ВСКРЫЛ ДЕФЕКТ В САМИХ ТЕСТАХ, и это важнее самой проверки. Первая версия — 10 тестов, все зелёные; снятие ЛЮБОЙ из двух половин фильтра оставляло их зелёными (патч при этом ложился, проверено грепом). Причина: иллюстративные фрагменты спотыкались об ОБЕ половины независимо и различить их не могли. Замерено, что каждая половина держит на реальном корпусе: фильтр «сообщение про наш пакет» выкл → 3 находки, все `import … from "./my-harness-adapter.js"` — придуманный для примера ОТНОСИТЕЛЬНЫЙ путь фильтр «коды модульной поверхности» выкл → 1 находка, TS2664 в `declare module "vigiles/generated"` — модуль, существующий только после кодогенерации Оба случая переписаны в тесты дословно с корпуса. Теперь каждая мутация падает на СВОЁМ ассерте, чистый прогон 12/12. `vitest.config.mjs`: include расширён на `scripts/**`, иначе тест этого гейта не запускался бы вообще — ровно та форма, которой в этом проекте уже трижды были мёртвые проверки. Гейты, поимённо и в порядке: npx vitest run (2928 passed, 173 файла) · npm run lint (0 errors) · npx prettier --check . · npm run api:check (no drift) · npm run build. Новых CLI-глаголов нет. Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Claude
|
878111f794 |
refactor(report): extract shared @vigiles/report-view via npm workspaces
Stage 1 of the hosted-demo work: the audit report view must be a genuinely shared package so report/, site/, and the future in-browser demo all render the SAME AuditReport JSON through the SAME components — never a screenshot, never a duplicated/relative-cross-imported component (the landing-site skill's shared-UI invariant, and the prerequisite Fable's demo brief names). - New packages/report-view (@vigiles/report-view, source-only/private): the presentational components, the AuditReport schema (mirrors src/audit-report.ts, pinned by src/audit-report.test.ts), band tokens, and theme.css. report/src/* moved here (git renames); internal @/ imports rewritten relative. - report/ consumes it; report/src is just the app shell (main.tsx + index.css, which @sources the package + @imports its theme — Tailwind v4 ignores node_modules). Wired as npm WORKSPACES (root workspaces:[packages/*,report,site]) so deps hoist to root and resolve with zero config (no preserveSymlinks). Four aligned changes kept the published vigiles package's CI green: root lockfile regen, pages.yml builds site via the workspace (cd site && npm ci would fail with no per-package lock), build-report.mjs checks root node_modules for its install-guard, and the per-package lockfiles are removed. Verified with a clean-room rm -rf node_modules && npm ci && npm run build (+ tsc, fmt, site build, report renders styled). Design + Stage-2 demo brief in research/report-view-and-hosted-demo.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
zernie
|
bf9815f0b4 |
feat(audit)!: prose→lint-rule routing map (ESLint + Pylint) + dogfood CI-enforcement + repo-structure pass
Audit rule-routing map (ESLint full + Pylint basics; dynamic catalog, S0/S1 markers, construct/docstring patterns) with the four lanes sharpened (enforceable / hook / custom-rule / judgment-call) and the pr-to-lint-rule gated synthesis skill. Dogfood CI-enforced + indexed (compiler gate asserts verdicts; bench corpus guards; research/dogfood-corpus.md + a vendoring-policy rule; per-adapter asymmetry documented). Repo-structure cleanup: dev/ -> .claude/skills, schemas/ parked under src/, fixtures/ -> test/fixtures, etc/ -> api-surface/, scripts/tools split, root CLAUDE.md Layout map. BREAKING CHANGE: the audit --json schema bumps schemaVersion 1 -> 2 (the mechanism enum for an unrouted rule changed "compile" -> "synthesize"). |
||
|
zernie
|
467f65d430 |
feat(experimental): R3 disposable-service eval tier + promptfoo migration guide
Add the experimental R3 disposable-service eval tier (vigiles/experimental) for real side-effect skill testing on the subscription, plus a promptfoo migration guide. README eval section leads with the measured cost case. |
||
|
zernie
|
ff542b7c4b |
refactor!: vigiles audit — Lighthouse-style harness report (rename scan→audit)
Rename the read-only scan verb to `vigiles audit` — a local, zero-config Lighthouse-style report scoring a harness across four deterministic rings (Truthfulness/Triggering/Structure/Tested) with each finding's fix inline + a shareable HTML report and a versioned AuditReport JSON.
audit is a local report, not a CI step (CI uses `vigiles lint`). The executing checks (live MCP resolution + trigger-rate + the adoption preview) run only behind one interactive consent; there is no execution flag. The safety battery is not an audit ring — it lives in the vigiles/testing API.
Adds the adoption-gateway preview ("what would vigiles catch in your repo?"): the model drafts a spec from the instruction file, the cross-reference engine verifies every drafted ref deterministically.
BREAKING CHANGE: the `scan` verb is renamed to `audit` (no alias); the `--check-hooks`/`--verify-mcp`/`--trigger`/`--fix-plan`/`--explain`/`--deep`/`--measure`/`--fast`/`--no-measure` flags are removed.
|
||
|
zernie
|
e8fa3b6b78 |
refactor!: pre-release polish — surface freeze, spec-first defaults, auto-adopt (#48)
* chore: refresh HANDOFF — PR #47 merged, surface-freeze still open Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: freeze pre-release public surface + collapse markdown ladder Pre-release surface freeze (research/pre-release-focus.md): - Mark the experimental typed-composition cluster @internal in vigiles/spec (pipe/pipeStep/needs/start/andThen + Supplies/Handoff/KnownAgentName/ Pipeline/PipeStep/NeedsContract) and effect()/EffectRegion. They stay exported at runtime (no removal) but are excluded from the published API docs and flagged unstable in the committed surface reports, so a later change to them is not a breaking change to a documented-stable API. - Note evolve.ts as @internal/research-stage (already off every public entry). - Markdown cut: drop the "Level 0/1/2" ladder framing across public docs; collapse to two on-ramps (plain markdown via inline enforce comments -> typed spec). Frontmatter mode is KEPT, still works, and its full docs + the old ladder are PARKED via an HTML comment in docs/markdown-mode.md (matching the README's parked-section convention) — nothing deleted, trivially un-parked post-launch. - Ship STABILITY.md (0.x: CLI stable; library API evolving; experimental surfaces marked) + a one-line README pointer. - Regenerate etc/*.api.md to reflect the @internal tags (api:check green). BREAKING CHANGE: public symbols in vigiles/spec are re-tagged @internal (typed-composition combinators, effect()). They remain importable but are no longer part of the frozen, supported surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: surface the author-time-vs-runtime wedge + citeable authorities - README hero: state the differentiator positively — vigiles proves the harness "before the agent ever runs" (author-time / pre-run), ending on verify/test/measure (no negative opener, per the README direction rules). - pre-release-focus.md Positioning lock: fold in the canonical one-line wedge (author-time vs runtime, public-safe phrasing, no competitor names) and the citeable harness-engineering authorities (Karpathy "automate what you can verify"; OpenAI "the harness is hard"; arXiv AHE structure-beats- prose = enforce > guidance) for the launch article. The AHE prediction-loop note already lives on the roadmap A2 optimizer item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the README hero with the concrete pain (untested harness) Reframe the hero to go straight to the developer pain: you installed plugins and wrote skills but don't know if they work — a library with no tests. Replaces the "100x coder / 1x verifier" lead (kept as a backup hook in the positioning lock) with the concrete plugins/skills situation + silent-fail triad, ending on the verify/test/measure benefit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: lead the guide openings with the concrete pain, not the mechanics Apply the README's pain-first hook to the subdoc openings: - verifying-instruction-files.md: open with the CLAUDE.md-drifts pain instead of "vigiles's linting layer ... checks every reference". - for-plugin-authors.md: add the installer pain hook (do your skills fire for the people who install it?) above the audience-routing intro. - harness-testing.md: sharpen the lead to the concrete "does it actually block? does your skill fire?" question. Each keeps its README uplink + "what this doc is" line (docs-quality). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: codify the pain-first principle + fix a stale doc link - README direction comment: add rule 1c — lead with the concrete pain in the reader's own situation (the specific silent failure + the "library with no tests" anchor), for the hero AND every subdoc opening; reconcile with 1b (a pain about the reader's situation is a hook, not an apology). - Fix a stale link: src/vendor.test.ts -> src/adapters/claude-code/vendor.test.ts (moved in the adapter relayout). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — surface freeze + launch polish shipped on branch Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs(positioning): not a linter — spec-first + eject, don't fight on structure-lint Lock the decision (Positioning lock + roadmap launch-readiness): vigiles is NOT a markdown/structure linter and won't contest that crowded, commoditized lane. The bold differentiated play is author-time typed specs the agent writes + evals you can afford. Markdown demotes from on-ramp to floor / eject-target. Names the enabling build — a one-command `vigiles eject` (strip the integrity header -> clean owned markdown) so "always ejectable" is true and the two spec objections (TypeScript friction, lock-in) dissolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: add `vigiles eject` + reframe spec-first (managed, but ejectable) The escape hatch that makes "adopt a typed spec, you can always eject" true, and the positioning shift that leans on it. - `vigiles eject [file]`: the inverse of compile. Strips the vigiles:sha256 integrity header -> plain hand-owned markdown, removes the managing spec (--keep-spec to keep it), and adds a `vigiles-disable require-spec` marker so lint stays quiet. Pure helpers (parseIntegrityHeader/ejectMarkdown in core/integrity.ts) + handler + dispatch + help + docs/cli.md. New verb in cli-commands.ts (additive). Unit + e2e tests. - Positioning reframe: don't contest the structure-lint crown. README hero sub-line + spec mentions lead with "your agent writes the spec, and you can eject anytime"; markdown demoted to floor/eject-target (still the zero-TS on-ramp). README direction rule 1c codifies the pain-first hook. - adopt-spec skill: faithful-by-default + eject note; dropped the Level 0/1 ladder pitch; fixed stale commands (check->lint, generate-schema->generate schema, Action @main command:check -> @v1 command:lint). - Closed the scan gap: skills/ now in the self-command-refs dogfood, which caught two rotted `vigiles generate-types` refs (now `generate types`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(scan): nudge toward the real-model trigger tier; agent-safe by default When a plain `scan` finds model-invocable skills AND a model is reachable (ANTHROPIC_API_KEY, or an authenticated Claude Code session via CLAUDECODE / CLAUDE_CODE_ENTRYPOINT), surface the real-model `--trigger` tier that measures whether those skills actually FIRE: - A human at a TTY is OFFERED setup (scaffold a trigger-prompts.json, or run an existing one — the real-model run is always an explicit yes). - An agent / CI run (non-TTY / --json / --no-interactive / --yes) gets a one-line, non-blocking hint and never a prompt (great-agent-flow). Pure core (hasModelAccess / decideTriggerSuggestion / formatTriggerHint / scaffoldTriggerPrompts) + unit tests; CLI wiring + env-gated e2e in scan-cli.test.ts; docs in docs/cli.md. Also records (HANDOFF) that real-model tiers ARE runnable in Claude Code web: the claude CLI is authenticated via session OAuth (no API key), proven by `claude -p` and a live `scan --trigger` run (test-harness recall 100%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: --strict gates broken surfaces (CI red); fix `lint <dir>` crash The actual "edit a skill/subagent/hook badly → CI goes red" payoff. - `vigiles init --strict` now promotes the HIGH-PRECISION, FP-safe structural rules to `error` (not just require-spec): subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap. So a typo'd tool, a dead hook script, a misconfigured MCP server, or two colliding skill descriptions FAIL `vigiles lint` (exit 2). Never clobbers a user-set severity; leaves the lenient/recommendation rules (frontmatter-valid, skill-frontmatter, prefer-compiled-hooks, untested-*) at warn. Honest limit documented: CC itself loads a name-less / broken-YAML skill, so vigiles gates the subagent/hook/MCP defects + skill collisions that genuinely break, not skill content that still works. Proven: clean subagent exit 0, typo'd tool exit 2. - FIX: `vigiles lint .` (a directory arg) crashed with EISDIR (it read `.` as a file). findInstructionFiles now expands a directory arg to the instruction files inside it; a file arg and a non-existent arg pass through unchanged. Tests: setup-plan (strict gate set), cli (lint-dir no-crash + strict exit-2 regression). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat: init gates broken surfaces BY DEFAULT (no --strict needed) Fold the structural gating into a plain `init` — no confusing "strict mode" to remember. A default `init` now writes the 9 HIGH-PRECISION, FP-safe structural rules as `error` (subagent-tool-contract, subagent-frontmatter, hook-events, hook-script-exists, mcp-config, mcp-tool-resolves, mcp-hook-target-resolves, disallowed-tools-contract, description-overlap), so a broken subagent / hook / MCP ref or a skill collision FAILS `vigiles lint` out of the box — while a well-formed plugin stays green (never cries wolf). `--strict` is reframed to the WORKFLOW-FORCING tier a clean repo can still fail because the work isn't done yet: require-spec, untested-skill/subagent/hook, frontmatter-valid, skill-frontmatter. Opt-in by design (progressive adoption), so a first CI run isn't red just for not having written a spec yet. Never clobbers a severity the user already set. vigiles dogfoods it — the 9 gate rules added to its own .vigilesrc.json (self-lint stays green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * feat(init): interactive init offers the workflow tier (recommended, opt-out) Per the chosen default model: structural gating is always on; the WORKFLOW tier (require specs + a test per surface) is the recommended default a human opts OUT of, never forced on a silent run. - Interactive `init` now asks "Also enforce specs + a test per surface (recommended)? [Y/n]" (default yes) → sets strict. SetupAnswers/applyAnswers carry it; resolvePlan honors it. - A bare NON-interactive `init` (agent/CI, no one to ask) stays structural-only unless `--strict` — so an automated setup never silently turns an existing repo's CI red just for not having written a spec yet. - README agentic-install prompt + docs/cli.md updated: the agent asks before enforcing the workflow tier, structural gating is the safe default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * chore: refresh HANDOFF — spec-first defaults + eject + scan nudge; PR is the next step Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * test(init): make the interactive setup Q&A unit-testable (inject ask) The interactive init prompts were the one untested-and-fragile bit of the setup flow (readline can't run in CI's non-TTY env, so the questions/defaults/ answer-mapping had zero coverage). Extract the Q&A into a pure collectSetupAnswers(ask) with an injected AskFn; promptSetup is now just the readline IO shell. Unit-tested with a fake ask: all-defaults, lint/test pillar selection, and declining CI/plugin/strict — deterministic, no TTY, no model. Now the init flow is covered end to end: pure plan logic + the interactive Q&A (unit) + non-interactive init (e2e in cli.test.ts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * docs: install-enforcement model — rule groups (not presets) + rationale Capture the reasoning, not just the outcome, before implementing. - New `install-enforcement-model` rule in CLAUDE.md (via the spec): rules are organized into GROUPS by confidence (structural / workflow / nudge), NOT a relaxed/standard/strict preset MENU — the linter best practice (Clippy/Ruff/ Biome; "presets are a poor emulation of rule groups"). default = structural at error; --strict adds workflow; --report-only is an orthogonal severity dial. Installation is EXPLICIT ABOUT TRADEOFFS (one source reused across the CLI prompt / agent AskUserQuestion / summary). - Reconcile smooth-adoption + progressive-adoption to the new stance: "permissive = doesn't force specs/TS + doesn't cry wolf, NOT ignores breakage." - research/install-enforcement-dx.md: the full design record — the linter landscape (verified), why we rejected a preset menu, the group→rule mapping, the agentic-install DX (no prior art), and the philosophy reconciliation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhTWXGSinhKGqrgTvPMxTG * refactor!: rename require-spec → require-instructions-spec; rule groups; init auto-adopt Rename + narrow the rule, name the install rule-groups, and make `init` faithfully adopt existing instruction files so the rule is satisfied by construction. - require-spec → require-instructions-spec, NARROWED: only a `.spec.ts` satisfies it — inline `vigiles:enforce` / `vigiles:` frontmatter no longer do. Disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. - Name the rule groups in src/setup-plan.ts (STRUCTURAL_RULES / WORKFLOW_RULES / NUDGE_RULES) and add `--report-only` (writes the whole gate at warn). - prefer-compiled-hooks defaults off; require-skill-spec un-deprecated (the consistent require-<surface>-spec parallel, default off). - New src/core/adopt.ts: adoptMarkdown/adoptToSpec faithfully convert an existing CLAUDE.md/AGENTS.md into a claude() spec (heading → verbatim section, no rule inferred, always compiles). Wired into init()/setupPillar1; round-trip tested (byte-identical below the integrity header). - Sweep stale Level-0/1/2 markdown-mode comments; docs + CLAUDE.md recompiled. BREAKING CHANGE: the `require-spec` lint rule and config key are renamed to `require-instructions-spec` and narrowed — only a `.spec.ts` satisfies it (inline comments / `vigiles:` frontmatter no longer do), and the per-file disable marker is now `<!-- vigiles-disable require-instructions-spec -->`. No automatic alias; update `.vigilesrc.json` and any disable markers. * fix(init,eject): honor interactive strict; don't delete a shared spec on eject Addresses two Codex review findings on #48 + the fmt:check CI failure. - setup(): write `.vigilesrc.json` and the summary from the RESOLVED plan (`plan.strict`), not the raw `--strict` flag — so an interactive "yes" to the workflow tier is actually persisted instead of silently dropped. - eject(): only delete the header's `compiled from` spec when it EXCLUSIVELY belongs to the file being ejected (`<file>.spec.ts`). A mirrored secondary target (e.g. AGENTS.md compiled from CLAUDE.md.spec.ts) names the shared source; deleting it would break the primary's recompile. Keep it + warn instead. Regression test added. - Format HANDOFF.md (the fmt:check failure). * fix(init): non-destructive adopt, deferred compile, test-only & strict config gating Install-DX hardening from PR #48 review + a self-audit across user/project types. - Auto-adopt is now NON-DESTRUCTIVE: `init` writes the spec but never overwrites your CLAUDE.md/AGENTS.md. Run `vigiles compile` to opt the file into spec management (byte-faithful; review the diff; `eject` reverses it). - Defer the compile when `vigiles` isn't resolvable yet (a fresh repo hasn't run `npm install`) — print the next step instead of a "failed to load" error. - Test-only setup (`init --test` / `--no-lint`) records the harness but writes NO lint rules — honors the positive-flag contract (Codex review). - setup() writes config from the RESOLVED plan, so an interactive "yes" to the workflow tier persists (was dropped). - Next steps reordered (install → compile → strengthen → test); `--report-only` in `init --help`; README + docs surface non-destructive adopt. New e2e tests. * fix(adopt): don't drop intro text when a literal ## Overview already exists Codex review (P2): a file with intro prose before the first heading AND an explicit `## Overview` section produced two sections keyed "Overview" — building the sections object overwrote the synthesized intro, silently dropping it (and a later `compile` would remove it from the file). Extract a shared `allocKey` dedup helper and use it for the synthesized Overview too, so it becomes "Overview (2)" on collision instead of clobbering. Regression test added. * fix(eject): preserve frontmatter on skill/agent eject; keep shared multi-target specs Two Codex review P2s on the eject feature. - ejectMarkdown no longer prepends the require-instructions-spec disable marker before a body that leads with YAML frontmatter (a compiled SKILL.md / subagent): doing so displaced the frontmatter and lost the surface's name/description/tools. The marker is instruction-file-only, so a frontmatter-led body ejects verbatim. - eject deletes the header's spec only when NO other compiled output still references it (a directory scan), instead of a basename heuristic. A multi-target spec (`target: ["CLAUDE.md","AGENTS.md"]`) or mirror now keeps its spec until the LAST output is ejected — ejecting either the primary or a secondary no longer orphans the others. The trailing marker hint prints only when a marker was added. Tests: frontmatter-preserved eject (unit) + eject primary/secondary/last-output of a multi-target spec (e2e). Docs note the two safety behaviors. * fix(eject,scan): path-safety on spec deletion; gate scan prompt on stdin TTY Two Codex review P2s. - eject: the `compiled from <path>` header is untrusted — a hand-edited/forged header naming `package.json` or a `../` path would make rmSync delete an unrelated file. Only remove a `.spec.ts` that resolves INSIDE the project; otherwise keep it and warn. Regression test added. - scan --trigger nudge: prompt only when BOTH stdout AND stdin are a terminal — `askOnce` reads stdin, so a TTY stdout with piped/redirected stdin (agents, pipelines) must take the non-blocking hint path instead of blocking on a read. * docs(readme): add FAQ + docs/faq.md, tighten install prompt, lift adoption line 5-star onboarding pass (1/N). - New docs/faq.md: the objection-handling answers (why a typed spec & is it optional, do I write TS, does init touch my files, CC+Codex, cost/key, non-JS repo, safe to run, vs promptfoo). - README: compact FAQ section linking it; install prompt rewritten to plain English (names adoption, tells the agent to finish: install + compile, no flag jargon); the non-destructive adoption reassurance lifted above the fold. * fix(init): honest guidance for non-JS repos (no package.json) 5-star onboarding pass (2/N). A Python/Rust/Go repo with a CLAUDE.md but no package.json can't resolve the vigiles npm package, so the typed-spec compile path isn't available there. Previously init told them to run `npm install` (which wouldn't help). Now it points at the real paths: - the deferred-compile notice and the adopted next-step both detect a missing package.json and recommend `npx vigiles lint` (verifies with zero install) plus `npm init -y && npm i -D vigiles` to enable specs. (Dropped the post-init lint "aha" idea — a freshly-adopted file is guidance-only with no marks to verify, so it would show nothing until `/strengthen` runs.) Test: init on a no-package.json repo guides to lint, not a bare npm install. * docs: add scripts/demo.sh — self-contained asciinema demo of the Lint layer 5-star onboarding pass (3/N). A zero-dependency, reproducible script for recording an asciinema cast / GIF: it builds a throwaway repo whose CLAUDE.md points (via inline marks) at a file that moved and a script that was renamed, then runs `vigiles lint` to show the ✗/✓. One command to record: asciinema rec --command "bash scripts/demo.sh" Tested end-to-end (real ✗ "File not found" / "Script not found" output). Consistency spot-check: public docs carry no stale adopt/compile language and all README-linked guides open with a README uplink. * docs(readme): final 5-star pass — tighten spec blurb, wire demo GIF slot 5-star onboarding pass (4/4). - Tighten the above-fold spec/adopt/eject paragraph (the new FAQ now carries the "do I write TS / will it touch my files" detail), and point new readers to it. - Wire a DEMO GIF slot at the prime "see it work" spot with the exact asciinema/agg recording command (scripts/demo.sh) + the embed snippet. README rendered content ≈152 non-blank lines (well within the ~200 cap); all internal doc links resolve; command refs pass the self-command-refs dogfood. * docs(handoff): record the 5-star onboarding-polish pass * docs(readme): add the demo GIF — vigiles lint catching stale references Animated terminal of `vigiles lint` flagging a moved file + a renamed script, at the README's prime "see it work" spot. The output lines are verbatim from the real CLI (nothing fabricated). Generated headlessly with Pillow — no recorder needed — so it's reproducible: `python3 scripts/make-demo-gif.py` (generator committed); scripts/demo.sh remains for a live asciinema recording. * docs(rules): sync rule docs with this session's default changes doc-per-rule drift caught auditing the session: - prefer-compiled-hooks.md: default is now OFF (was documented "warn") — a recommendation that shouldn't fire unasked. - require-skill-spec.md: drop the "deprecated" framing — it's the consistent require-<surface>-spec parallel (kept, default off), matching the code + matrix. * fix(eject): scan the whole project tree for shared-spec references, matched by resolved path The shared-spec guard scanned only the ejected file's directory, so a multi-target spec emitting a secondary output in a subdirectory (target: ["CLAUDE.md", "docs/AGENTS.md"]) was missed — eject would delete a spec still feeding another compiled file. Walk the whole project tree (skipping node_modules/.git/dist/build/coverage/.next/out) and match on the resolved spec path, not the basename, so two unrelated CLAUDE.md.spec.ts in different packages don't false-positive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zernie
|
dc293e52e1 |
chore(bench): add token-efficient to A1 manifest + per-task spread reporting (#44)
* chore(bench): add token-efficient to A1 manifest + per-task spread reporting
Expand the ecosystem-benchmark manifest with a second cleanly-A/B-able
compression entry, drona23/claude-token-efficient (MIT, 5.7k stars,
pinned @0d30a6d) — a single injectable CLAUDE.md whose viral "63%" headline
is a word-count over 4 prompts, while the repo's own token benchmark admits
only ~4-12% output-token reduction (a self-documented claim>>measured gap).
Vendor the real file + its MIT LICENSE; record full provenance and the
follow-on (needs-binary: RTK/CodeGraph/Claw/...) and license-blocked
(Context Mode) cluster classification in SOURCES.md/FINDINGS.md.
Also surface per-task variance in the leaderboard: the mean hid that a skill
can help on one task and hurt on another (the caveman review-doc finding), so
report the output-cut range + the helped/hurt split + a MIXED-direction flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF for A1 manifest-expansion session
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): record A1 haiku pilot — token-efficient debunked, caveman variance flagged
First multi-skill A1 run (haiku, caveman+token-efficient × 2 tasks × 2 trials, $0.29):
- token-efficient: claim 63% vs measured -2% output (output GREW, bill +3%) — DEBUNKED,
worse than the repo's own honest ~4% haiku admission (CLAUDE.md input overhead is
net-negative, as its README warns).
- caveman: +30% output cut here vs -18% on the prior same-config run — a 48-point swing
flagged as a variance/noise signal (raise trials >=5; only the structural <1%
output-share fact is robust).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — A1 pilot ran, token-efficient debunked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(agent-runtime): make the active-agent tracker nesting-safe (depth-aware stack)
The PreToolUse tool-contract rail tracked the active subagent in a single
`.vigiles/active-agent.json` slot. Under CC v2.1.172 depth-5 nested subagents
this allowed a CONTRACT ESCAPE: when an inner subagent returned, SubagentStop
cleared the whole slot while still inside the outer subagent, so the gate then
saw "no active agent" and ALLOWED a tool the outer subagent's contract forbids.
Fix (TLC-certified in AgentWindowStack.tla): replace the slot with a depth-aware
STACK — pushActiveAgent on dispatch, popActiveAgent on SubagentStop (back to the
parent frame, not a full clear), gate on the stack top. The open signal now
recognizes both the `Task` and nested-spawn `Agent` tools. Legacy single-slot
`{ agent }` markers read back-compat as a one-frame stack.
The TLC counterexample Open(writer);Open(writer);Stop;Call(Bash) is now a
regression test (pure + a runHook e2e proving SubagentStop pops to the parent).
Still EXPERIMENTAL / not auto-wired; the effect() sub-region it served stays
dropped — the stack ships for when nested contract enforcement is wanted alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — V1 nesting bug fixed
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record the per-file mixed spec/hand-written adoption model
Capture how vigiles behaves in a repo where some skills/agents/CLAUDE.mds have a
.spec.ts and some don't: per-file handling (compile + integrity + ref verification
for spec'd files; surface lint rules still apply to hand-written ones), the
no-spec-mandate defaults (require-spec off, require-skill-spec deprecated), and
scan's spec-agnostic spec-managed-vs-hand-written labeling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add the spec-value model (what a typed spec buys vs markdown)
Consolidate the recurring "spec vs plain markdown" question into one canonical
internal doc (research/spec-value-model.md): the capability axis (lint/test/eval
run on markdown; only whole-harness type-checking is spec-exclusive), the
two-oracle model (behavioral = spec-agnostic vs deterministic contract/outcome =
spec-enabled), honest leg-grading (result-shape/flow/purity held; effect()
sub-region dropped), the surface ranking, and the settled require-*-spec split +
capability-triggered defaults. Slim the duplicative per-file note in
adoption-strategy.md to a pointer; cross-link from typed-spec-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: cite "Agents Need Control Flow" as third-party validation (+ commoditization caveat)
Record Brian Suh's bsuh.bearblog.dev/agents-need-control-flow/ in the landscape
doc: an independent statement of vigiles's Agent=Model+Harness /
deterministic-constraints thesis, with its three failure modes (babysitting /
post-run audits / accepting on faith) mapping onto our verify+test layers. Framed
honestly as double-edged — external validation of the frame, but confirms the
commoditization signal from the same-day competitive check (the control-flow
thesis is now a named, spreading idea; the edge is substrate + measurement, not
the idea). Cite for positioning, do not treat as a moat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record the competitive reality-check in typed-spec-moat (don't re-inflate)
Same-day competitive sweep found the headline typed-spec "moats" are mostly
claimed/commoditizing: riftmap = PR-time blast-radius diff (moat #2 mechanism for
infra), AgentAuditKit/PolicyLayer = capability-diff + drift for MCP servers, Mastra
= typed workflow handoffs (#5), promptfoo = deterministic structured-output oracle
(#7); the control-flow thesis is now a named idea (bsuh). Add a top caveat + a
"Competitive reality check" section recording per-moat verdicts, the one narrow
unoccupied sliver (first-party harness effect-diff from a typed spec), and the
synthesis: the differentiation is substrate + sub-affordable measurement, not the
capabilities — so don't re-inflate the moat next session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — commoditization finding + consolidate shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: correct the competitive analysis for MARKET (it conflated 3 markets)
The "typed-spec capabilities are commoditizing" read was a market-conflation:
Mastra/promptfoo = app-building (zero CLAUDE.md contact), riftmap = infra,
AgentAuditKit/PolicyLayer = third-party MCP-server security — none are in vigiles's
market. Grounded by reading docs: the ACTUAL in-market competitors (claudelint 114
rules, cclint ×2, claude plugin validate) are pure static surface linters that do NO
cross-referencing, typed specs, test/eval, or capability-diff. So in-market vigiles
uniquely owns those layers; it is NOT differentiated on surface linting. Lead with
cross-referencing (unique in-market, works on plain markdown = no adoption barrier).
Add the market-segmented matrix to landscape-mid-2026.md; correct typed-spec-moat
synthesis + the HANDOFF block. Reviews/adoption pass on claudelint/cclint in flight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add adoption pass — field is wide open, agnix is the most-built rival
Reviews/adoption pass on the real in-market competitors: NO incumbent, near-zero
mindshare. Biggest is agnix (296★, ~414 rules, Rust, 7-harness + LSP) — crowds
vigiles's multi-harness/LSP positioning but does NOT do cross-ref/typed/eval; the
rest are 6–41★ hobby projects; none in awesome-claude-code (47k★); npm downloads
are CI-inflated (agnix Show HN = 1 point). Only entrenched thing is Anthropic's
shallow `claude plugin validate`. Add agnix/AgentLint/SkillCheck to the Market C
matrix + adoption signals; rewrite the read: don't out-rule-count agnix or match its
LSP — lead with cross-referencing ("valid≠true") + the test/eval tier; the real
battle is distribution, not capability (reinforces A1). Update HANDOFF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: capture poach-list, agnix lesson, badge/viral/acquisition GTM, Zod result()
Following the market-segmentation correction: record the useful-but-not-competitor
ideas + adoption plays.
- landscape: the agnix signal (most-built linter, 1 HN pt → linting-without-eval
doesn't pull a community = evidence for the measurement wedge) + a poach list from
the app-building non-competitors (Zod-schema'd contracts, promptfoo assertion vocab,
riftmap distribution model) + business-on-top validated by closed SaaS rivals.
- roadmap §Explore + distribution-strategy: tiered README badge (lint→test→eval funnel
as a growth loop), viral debunk articles (measurement-as-marketing), open-core +
build-on-top + AI-lab-acquisition posture.
- measurement-authority: competitive confirmation of the pivot (agnix → eval is the
puller; no incumbent → distribution is the bottleneck).
- feature-ideas §14b: Zod-schema'd result() (one schema → type + validator + JSON
Schema), poached from Mastra/Pydantic AI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: fix rule-count error (20 not 31) + 11★ honesty; dedup/crosslink the cluster
- Correct the Market-C matrix: vigiles ships ~20 semantic rules (not 31 — that was
the dogfood CLAUDE.md's section count), with an apples-to-oranges caveat (agnix's
414 are micro-checks; vigiles's rules cross-reference) and the real 11★ baseline.
- Reconcile typed-spec-moat's internal contradiction: the top caveat + §7 title said
"commoditizing", overturned by the later market correction — both now say
"capabilities live in adjacent markets; vigiles's own market is wide open".
- Record the PUBLIC plugin leaderboard idea (A1 + leaderboard.ts published as a
claim-vs-measured ranking, head-to-head per category) in roadmap §Explore.
- Cross-link the two matrix hubs (spec-value-model "what a spec buys" ↔ landscape
"who competes").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): record A1 sonnet pass (5 trials) — both compression skills debunked
Sonnet, caveman + token-efficient × {slugify,debounce,review-doc} × 5 trials ($3.43):
- caveman: claim 75% vs measured -18% output (range -55..+8, MIXED), bill +8% — the
-18% is now STABLE (matches the prior sonnet 3x3; haiku +30% was noise). review-doc
(its supposed best case) is the WORST at -55% (output grew 55%).
- token-efficient: claim 63% vs measured -10% output, bill +3% — WORSE than the repo's
own ~12% sonnet admission; review-doc -35%.
- Both: 0 regressions, output ~0.7% of session tokens (the model-agnostic structural
kill). Neither compresses on multi-turn coding; both cost more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — A1 sonnet pass done, numbers publishable
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): audit caveman/token-efficient OWN benchmarks — why they report a win
Read their benchmark code (caveman/benchmarks/run.py, drona23/.../benchmark/run.py +
SUMMARY.md). Record the methodology gap that explains the headline numbers:
- caveman run.py: single-shot one-turn Q&A (no tools/files/multi-turn), output-tokens-
ONLY (ignores the SKILL.md system-prompt input + cache that dominates real sessions),
vanilla "You are a helpful assistant." headline baseline (the concise-baseline harness
lives separately in evals/), and no correctness gate.
- token-efficient: the 63% is a WORD count; the repo's own SUMMARY.md already admits the
token effect is -2%..-11% and "not agentic/coding loops" — vanity headline vs honest
fine-print, and even the fine-print doesn't reproduce on coding.
Also corrected the prior over-stated "baseline discrepancy" point to the fair version
(headline uses vanilla baseline; author didn't hide the concise comparison).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: record caveman-debunk SATURATION — lead with method/leaderboard, not caveman
Reality check (web-confirmed): the caveman critique is ~6mo old and crowded
(Kuba Guzik 14-21%, GrowwStacks, HN top comment, Decrypt, devneeddev/X; author
conceded; token-efficient author self-debunked). We are NOT first on the verdict.
Our novelty is the METHOD (reproducible multi-turn correctness-gated harness +
code audit + the output-GROWS finding + N-skill scale), not the conclusion. Add a
saturation warning to FINDINGS and reframe the roadmap article bullet: method/
leaderboard-first, caveman as a validation row, don't make it the flagship.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record benchmark/leaderboard prior art for the A1 leaderboard
Web sweep of who already benchmarks/ranks Claude Code skills, to position the
public leaderboard: (1) subjective "best skills" listicles abundant (SEO, no
measurement); (2) install/adoption leaderboards exist (quemsah n8n metrics +
published install counts — popularity, not effectiveness); (3) a few one-off
measured benchmarks (caveman saturated; a real superpowers 12-session A/B that
VINDICATES it — 9% cheaper/14% fewer tokens; MindStudio "5 skills 70%" claim-
pushing). Open slice vigiles owns: reproducible + multi-skill + claim-vs-measured
+ correctness-gated + head-to-head + continuous + open-method. Position on the
claim-gap + reproducibility + correctness axes, not "ranking plugins" (install
boards exist) — and honestly (some claims, e.g. superpowers, hold).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — methodology audit, caveman saturation, leaderboard-next
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF competitive block under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(bench): archive canonical eval runs (the data behind FINDINGS survives)
The raw per-run JSON is gitignored scratch under results/; this commits the haiku
+ sonnet canonical runs (the data the published FINDINGS numbers are computed from)
into results-archive/ so the measurement data — the moat — survives a container
reclaim, is auditable/reproducible, and never needs a paid re-run to recover. Adds
a README with provenance + schema + method/reproduce pointers. Curated: only runs
cited in FINDINGS get archived; routine runs stay in results/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: prettier-ignore archived run JSON (machine-written archival data)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: refine cross-ref moat — harness-native is the value, linter-catalog is legacy
The cross-check engine's value is NOT uniform: HARNESS-NATIVE references (tool
contract, MCP server#tool LIVE resolution, hook events, file/script paths,
delegate targets) are the moat — agent-era-specific, fail silently, nobody else
checks them. The 7-linter-catalog integration (ESLint/Ruff/...) is the founding
feature but the WEAKEST leg (linter self-enforces regardless of the prose mention;
heaviest engineering, least harness-specific payoff) — keep it as supporting, don't
add more catalogs. Deepen harness-native instead; first build = live MCP tool
resolution of the real mcp__server__tool refs. Updated landscape (REFINEMENT block),
roadmap §Now, HANDOFF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(mcp): live-verify real mcp__server__tool contract refs (harness-native cross-check)
Bridge the live MCP engine (listMcpTools/closest in mcp.ts, until now wired only to
the explicit `vigiles:mcp` mark) to the agent's REAL `mcp__server__tool` references.
verifyMcpContractTools starts each referenced+declared server once and checks the
TOOL actually exists on it — catching the create_issue→issue_write rename rot that
the static verifyMcpToolServers (server-declared only) misses. Refs to undeclared /
built-in / plugin-namespaced servers are skipped (the static check's job; we have no
config to start them). New mcpToolParts (mcp-tool.ts) extracts {server,tool};
mcpContractToolMessage renders the did-you-mean. 5 tests against the real
fixture-mcp-server.mjs (CI-safe, offline): existing-resolves, renamed→suggestion,
undeclared-skipped, plugin-namespaced/non-MCP-skipped, unreachable-server. Opt-in
(starts processes) — engine first; scan/CLI wiring is the next increment.
This is the harness-native cross-check moat (the valuable leg), not the linter-catalog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: refresh HANDOFF — MCP contract-tool resolution shipped; trim to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record first live-MCP dogfood finding — 3 dead tool refs in MadAppGang tester.md
Dogfooded verifyMcpContractTools (the new live MCP tool-resolution) against the real
vendored MadAppGang frontend plugin + the real chrome-devtools-mcp server: tester.md's
tools: lists 27 mcp__chrome-devtools__* tools, 3 of which no longer exist on the live
server — emulate_cpu + emulate_network (consolidated into `emulate`) and
navigate_page_history (removed). 24/27 resolve. Classic MCP rename rot, invisible to
every static linter (the plugin declares no server, so even mcp-tool-resolves is
silent). Recorded in the public-disclosures table + a method/repro note. The
harness-native cross-check moat validated on real OSS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — MCP live-resolution dogfooded, found real MadAppGang bug
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record live-MCP-check scope — high-value but narrow (1/4 vendored plugins)
Dogfooded verifyMcpContractTools on all 4 vendored plugins: only MadAppGang pins
mcp__server__tool refs (the 3 dead-ref finding); superpowers/wshobson reference no
MCP tools; oh-my-claudecode declares a server but no surface pins its tools. So the
live check is sharp but narrow — bites only where a contract hard-lists specific MCP
tool names. Recorded honestly so we don't overstate its breadth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EOF
)
* feat(scan): wire live MCP tool resolution behind opt-in scan --verify-mcp
Expose verifyMcpContractTools through the scan surface: `scan --verify-mcp`
starts each declared MCP server and checks that every mcp__server__tool the
plugin's agents reference actually exists on it — the dynamic check no static
linter can do (catches rename/removal rot like create_issue->issue_write).
Opt-in and side-effecting (spawns servers); returns [] when no servers are
declared. Static verifyMcpToolServers still owns undeclared-server refs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — scan --verify-mcp shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under 120-line cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(scan): measure cross-skill selection collision (scan --collisions)
The behavioral confirmation of the deterministic description-overlap rule: run
each model-invocable skill's own prompts against the whole installed plugin and
record WHICH skills fired (whichSkillsFired) — an N×N matrix whose diagonal is
recall and whose off-diagonal mass is one skill HIJACKING a sibling's prompt.
The cross-skill precision question per-skill trigger-rate can't see; the
blast-radius column the leaderboard needs.
- eval.ts: whichSkillsFired + runSkillSelectionTrial (one pass yields the whole
matrix, N× cheaper than per-pair; bodies stubbed so a run stops at selection).
- scan-behavioral.ts: buildSelectionReport (pure fold) + measurePluginSelection
(real claude) + measurePluginSelectionWith (injectable). Claude Code only —
Codex has no discrete skill-selection event.
- CLI: scan <dir> --collisions --prompts=<file.json>.
- Dogfood (on the sub): vigiles's own 3 skills are collision-free; bonus signal
edit-spec under-fired (recall 33%). superpowers slice can't fire (missing
gateway skill). See research/plugin-selection-collision.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — selection-collision matrix shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF under cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: verify superpowers null result — hook-primed plugin, harness drops hooks
The superpowers 0% recall is a measurement artifact, root-caused from source:
its skills activate via a SessionStart hook that injects the using-superpowers
gateway priming, but the trigger/collision tier stubs the plugin to skills-only
(packageSkillsDir drops hooks/), so the gateway never runs — compounded by the
vendored slice missing using-superpowers/SKILL.md. Confirmed 0% in --trigger too
(not collision-specific). Records the limitation + a roadmap follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(scan): label hook-primed 0%-recall as a measurement artifact (Layer 1)
A stubbed trigger/collision run rebuilds the plugin to skills-only and drops the
plugin's hooks — so a SessionStart hook that primes skill selection (superpowers'
using-superpowers gateway) is lost, and recall collapses to a misleading 0%.
Layer 1 (honesty before fidelity): when a stubbed run on a SessionStart-hooked
plugin shows EVERY measured skill at recall 0, relabel it "hook-primed — likely a
measurement artifact; re-run against the full plugin install" on both the trigger
and collision columns (hasSessionStartHook + isStubbedHookArtifact). The all-zero
gate is the precision guard — a genuine single-skill miss stays reported as real.
Verified on the live superpowers run + 3 CI fake-driver tests (hooked→labeled,
non-hooked 0%→stays real, trigger relabel). Layer 2 (a --no-stub faithful path +
auto-fallback) is the deferred follow-up in research/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — Layer-1 hook-primed artifact label shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim HANDOFF to cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(leaderboard): publishable Markdown table + real names; v0 corpus run
Make the structural-health leaderboard a publishable artifact:
- formatLeaderboardMarkdown — a ranked Markdown table (README/gist/site form),
wired as `scan <dirs...> --md`.
- rankPlugins now labels each plugin by its manifest name (.claude-plugin/
plugin.json) instead of a SHA-pinned dir basename.
bench/leaderboard/ — a reproducible v0 generator (run.mjs + corpus.json) that
clones public plugin repos, finds on-disk plugin roots, ranks them, and writes
RESULTS.md (scores only; no plugin code vendored). First run: 83 plugins from
obra/superpowers + wshobson/agents — grade spread A:16 B:28 C:17 D:9 F:13, with
real outliers (a dead agent tool, large uncontracted/untested collections at F).
The deterministic substrate for the public plugin leaderboard (the adoption
engine). Behavioural columns (trigger-rate, collisions) stack on top later.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — leaderboard v0 shipped, broaden-corpus next
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cli): capability-diff — does this change widen the agent's blast radius? (moat #2)
The bridge bet, v0. `vigiles capability-diff <before> <after>` computes each
version's whole-harness capability lattice (computeHarnessCapabilities — read-only
/ side-effecting / unknown-MCP tools + loosest purity) and diffs them:
- src/core/capability-diff.ts (pure, harness-agnostic): diffCapabilities +
formatCapabilityDiff + isNoOpDiff. WIDENED iff a new side-effecting/unknown tool
or a loosened purity floor; new read-only + removals reported but NOT a widening.
- CLI handler: informational by default (exit 0, a PR comment), --fail-on-widen for
the opt-in CI gate (don't cry wolf — widening is often intended).
6 pure-core tests + 3 CLI e2e (the exit-code contract). v0 reads the tool-bucket
lattice; the richer per-step effect-row + cross-step accumulation (turning "gained
Bash" into "opened an exfil path") is the deferred M1 follow-up in research/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — capability-diff #2 v0 shipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cli): consolidate behavioral measurement into a `measure` verb
Bad DX cleanup. scan had accreted ~14 flags by bolting every capability onto it.
Re-grounded on one rule: a verb per JOB/LAYER, flags for modifiers.
- NEW `vigiles measure <dir> --prompts=p.json` — the model-gated behavioral report
(trigger-rate recall+precision AND the selection-collision matrix, one report).
The OSS-testing front door, replacing `scan --trigger` + `scan --collisions`.
- scan goes back to FREE/deterministic: the behavioral flags move to `measure`.
- capability-diff is a FREE check → folded from a standalone command into a scan
flag: `scan <after> --capability-diff=<before>` (was `capability-diff a b`).
- printUsage now lists scan + measure (scan was missing entirely).
- Hand-off text (optimize/score-explainer) + docs/cli.md point at `vigiles measure`.
Net: scan ~14 flags → ~6 (all deterministic); one new justified verb (the missing
paid-measure layer); one needless verb removed. Underlying engine unchanged
(probePluginTriggers / measurePluginSelection / diffCapabilities). Full suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: HANDOFF — CLI consolidated into measure verb; lint/scan fold open
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: poach the app-frameworks for a typed CLAUDE.md (web-verified moat synthesis)
research/typed-claude-md-poach.md — consolidates + extends the scattered poach notes
into one moat doc, web-verified (2026-06-22) against Mastra / LangGraph.js / Pydantic
AI / Vercel AI SDK / Inngest / Temporal / Trigger.dev / Effect-TS.
The thesis: they type the agent you BUILD IN CODE and leave instructions a prose hole
(`instructions: string`); vigiles types the prose harness they don't. Verified moat
points: (1) the whole-harness REGISTRY (one tsc over every inter-agent edge) is unique
— nobody else emits a repo-scale typed handoff registry (the routeTree.gen.ts analogy);
(2) cross-agent handoff checking is universally weak/manual even in the typed frameworks
(Pydantic "no auto-derived cross-agent Supplies check"); (3) nobody types the
tool-permission/capability surface at compile time (the effect-row gap). Ranked bets:
effect-row (M1, headline) > Zod-result() > typed context/needs > exhaustive err-track
(poached from Effect-TS, novel).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: the reliability-RUNTIME moat — type the harness's dynamic structure
research/harness-protocol-flow-moat.md (deep FP pass, web-verified 2026-06-22 across
the Codex/CC failure corpus). The existing FP docs type the harness's static capability
SET (effect-row, capability graph, trifecta). This opens the THREE dimensions a set
discards — and reframes them as a category, not a linter:
- ORDER (typestate/session types): a destructive action is UNREACHABLE until its guard
step ran (kills destroy-without-backup: #50027 DROP SCHEMA, terraform destroy, rm -rf).
- FLOW (information-flow/noninterference): untrusted input cannot reach a sink — the
trifecta as a real source→sink path (CVE-2026-21852, Trail of Bits line-jumping).
- REPLAY (linear types/idempotency): exactly-once side effects — the unsolved
duplicate-on-replay hole (ACRFence, 45 issues/12 frameworks; idempotency keys fail
for LLMs). Multiparty session types = deferred 4th axis.
The MOAT framing per the new steer: this is a deterministic RELIABILITY RUNTIME (the
spec→PreToolUse-gate enforcement, surviving compaction OUTSIDE the context window), not
a lint rule — "make the harness measurably more reliable," provable via the eval layer.
Grounded in: prose-doesn't-bind (#32163 "@enforce"), declared≠enforced (SDK #172/#162),
METR <10%@4h, 56% no-recovery, trifecta in 98% of prod agents.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reframe the moat as a reliability RUNTIME (3 axes) + HANDOFF
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guards): prototype typed safe-by-construction harness guards (EXPERIMENTAL)
Dogfooded the real OSS hooks (our pre-edit.sh block-on-compiled, superpowers/OMC
session-start inject, OMC keyword-detector) → the safety-critical shape is the
PreToolUse gate. A hook today is arbitrary shell (the RCE footgun, CVE-2025-59536)
and its safety is undecidable (Rice). So invert it: DECLARE a guard from a closed
vocabulary, GENERATE the hooks block pointing at vigiles's own gate (no user shell).
src/core/guards.ts (not wired to CLI/public API — a sketch):
- guard.block / requireBefore (the ORDER axis: destroy-after-plan) / confine
- decideGuards — the pure runtime decision (reuses arg-match's matchesArgs)
- compileGuards — generates a Claude Code PreToolUse hooks block whose command is
`npx vigiles guard-hook` (vigiles's audited gate), safe-by-construction
- 4 tests: block reproduces pre-edit intent, requireBefore denies destroy-before-plan
+ allows after, confine denies an out-of-allowlist path, compile emits a valid block
See research/harness-protocol-flow-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(roadmap): reliability-runtime moat + ORDER/FLOW/REPLAY (REPLAY=P3), crosslinked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guards): runnable guard-hook gate — session ledger + ORDER enforcement live
Make the EXPERIMENTAL typed-guard prototype run end-to-end. The `vigiles
guard-hook` PreToolUse subcommand reads the live event, loads the declared
guard set (`.vigiles/guards.json`) + the session ledger
(`.vigiles/guard-ledger.json`), runs `decideGuards`, and blocks (exit 2 +
reason) or records the allowed call so a later `requireBefore` sees it.
- guards.ts: serializeGuards/parseGuards (RegExp-safe, mirrors tool-intercept's
env round-trip), loadGuards, readGuardLedger/recordGuardCall (the prior-call
ledger CC doesn't surface to a hook), parseGuardEvent, runGuardHook (allow →
record; deny → record nothing — a blocked call never happened).
- cli.ts: `guard-hook` subcommand mirroring agent-hook.
- Tests: serialize round-trip incl. RegExp, tolerant parse, ledger IO, and a
runHook e2e proving `terraform destroy` is blocked before `terraform plan`
and allowed after — the ORDER axis enforced across hook invocations, live.
The gate's command is vigiles's OWN audited binary, never user shell —
safe-by-construction. Still not on the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): guard-hook gate runs end-to-end (ORDER axis live)
* feat(hooks): spike — typed, effect-classified hook specs (the "type-safe bash" angle)
A spike to test whether the compile step earns its keep on the hook surface —
the least-typed, most-footgun-prone part of a harness. Catches two silent bugs
hand-written shell can't:
1. WRONG-FIELD EXTRACTION (silent no-op): a hook matching `Bash` but reading
`file_path` extracts empty forever and never fires. validateHook makes it an
error; the typed `hookFor` builder makes it a tsc error at edit time (proven
by an @ts-expect-error the build enforces).
2. EFFECT MISDECLARATION ("type-safe bash"): an observe-only hook that runs a
mutating/undecidable command doesn't compile — rides bash-effects.ts, the
deterministic classifier no other plugin tool has.
compileHook refuses (throws) any error-level issue — "an unsafe hook doesn't
compile" — and emits the settings block + correctly-typed field extractions
(never a hand-typed jq string). Pure core, harness-agnostic: the per-tool field
catalog is injected (Codex differs), the CC catalog + typed builder live in the
test. Not wired to the CLI/public API. See research/harness-protocol-flow-moat.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): save the verified hook pain-point corpus + the verify-not-gate killer feature
Five web-research passes (claude-code issues, official docs, dev.to/Medium, HN,
CVEs), all verified. The convergent #1 pain is FALSE CONFIDENCE — a safety hook
looks like a guardrail and silently isn't (exit 1≠2, wrong JSON field,
PostToolUse-can't-block, wrong jq path); RFC #45427 names the thesis and was
closed not-planned. Killer feature: "prove your guardrail actually blocks" via
runHook + decideHook (verify the decision LOGIC, not deliver enforcement) — works
on existing hooks with no spec/compile, sidesteps CC's delivery bugs
(subagent-bypass #34692 etc.). Compile is the optional Level-2 (declared intent →
auto-generated test), not the mechanism. Crosslinked from harness-protocol-flow-moat
+ roadmap (new HIGH item; gate demoted to MED — undercut by the same CC bugs).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(unit): prove your guardrail actually blocks — disaster-event battery over runHook
The #1 verified hook pain is FALSE CONFIDENCE: a safety hook looks like a guard
and silently doesn't block (exit 1≠2, wrong JSON field, wrong jq path) — "three
teams believed they'd blocked force pushes." This is the deterministic answer.
- src/guardrail-check.ts: a curated DISASTER_CATALOG (force-push incl. compound,
reset --hard, rm -rf, --no-verify, ssh-key read, curl|sh) + verifyGuardrail
(feeds each to a hook via runHook, reports blocked), assertBlocksDisasters (the
CI gate — throws naming the misses), formatGuardrailReport (NEUTRAL coverage
map — never judges intent, since not every hook is a bash-safety guard).
- Exposed on vigiles/unit (no model, no key, CI-runnable; works on hand-written
hooks with no spec). api reports regenerated.
- scaffold-test: a hook scaffold now emits the runnable battery (coverage map +
the commented assertBlocksDisasters CI gate keyed by declared category).
- Committed receipt: exit-1 vs exit-2 force-push guards, the engine tells them
apart; intent-declared assert passes for git, fails honestly when a guard
claims fs-safety it doesn't deliver.
Verifies decision LOGIC, not delivery — flags CC's runtime bugs (subagent-bypass
#34692 etc.) but never claims to fix them. See research/hook-pain-points.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): guardrail VERIFY feature shipped; refresh shipped-this-session
* docs(research): record guardrail VERIFY shipped + the neutral-map vs declared-intent design lesson
* docs(research): real-plugin dogfood — disler hook blocks rm/.env but no-ops vs force-push/ssh-read/curl|sh
* feat(spike): hook-as-typed-program probe — testability/safety/eval/portability + stamping
Probe the constrained-API compiled-hook model: a hook is a PURE typed function
(event)=>Decision against a closed `vigiles/hook` vocabulary, vigiles compiles it.
src/core/hook-program.ts + a 5-claim test, all passing in-process (no subprocess,
no model):
1. PURE/TESTABLE — decide() unit-tested directly; the false-confidence protocol
(exit 1≠2, JSON field) is unrepresentable (compiler emits exit-2).
2. AST-BACKED MATCHING — command.runs("git push",{force}) via new bash-effects
`leafCommands` (AST leaf extraction): catches the compound bypass
`cd x && git push -f` the native glob (#30519) misses AND avoids a grep
false-positive on `echo "...git push..."`.
3. COMPILES to a real CC hooks block.
4. CAPABILITY = API SURFACE — checkHookImports rejects any import outside
vigiles/hook (+ eval/Function/dynamic-import); an out-of-API hook does not
compile.
5. STAMPING (the "fix #4" idea) — stampHook/verifyHookStamp make the compiled
artifact tamper-evident (integrity.ts pattern): a hand-edit smuggling
child_process breaks the stamp → runtime refuses it.
Pure core, harness-neutral, not on the public API. Honest limits unchanged
(buy-in, node latency, CC subagent-bypass #34692 — compile fixes authoring not
delivery). See research/hook-pain-points.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record compiled-hooks probe result (5 claims proven incl. stamping)
* feat(spike): widen hook-program probe — path-confine gate + context-injection (vocabulary holds)
Probe 2 tests whether the closed vigiles/hook vocabulary holds across two
genuinely different shapes. It does — and reveals the vocabulary is a small
FAMILY keyed by hook ROLE, whose per-role output type eliminates two more
verified pains by construction:
- A second GATE: defineFileGate over Edit/Write with a PathView.under() matcher
(different tool, field, matcher) — confines writes to src/test, denies escapes.
- A non-gate: defineInject (SessionStart) returns Injection (context text), not a
Decision. runInject emits the RIGHT JSON field (additionalContext) — the author
never picks it, so the wrong-field pain can't occur.
- Correct-by-construction: an inject hook CANNOT return deny() — it's a tsc error
(proven by @ts-expect-error the build enforces). "Block on a SessionStart/
PostToolUse hook" (a documented mistake) becomes a type error, not a silent no-op.
Finding recorded in research/hook-pain-points.md. Still a pure-core spike, not on
the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): record probe 2 — hook vocabulary holds as a role-keyed family
* feat(spike): probe 3 — PostToolUse react hook (bounded side effects complete the family)
The third hook shape: a react hook (PostToolUse) fires after the tool ran, so it
can't block — its job is to DO something (format, recompile, warn). That's where
side effects re-enter (gate + inject are pure). The constrained API keeps them
BOUNDED: a react returns a typed Reaction whose run(cmd) is EFFECT-CLASSIFIED at
construction (bash-effects), so even the side-effecting role stays analyzable —
you can list/diff exactly what every react runs and its effect. And a react still
CANNOT block (no deny in Reaction) → "block on a PostToolUse hook" (a documented
mistake) is a tsc error (build-enforced @ts-expect-error).
Completes the role-keyed family: gate (decide→Decision, pure) / inject
(produce→Injection, pure) / react (react→Reaction, bounded side effects). Each
role's output type makes its wrong-output bugs unrepresentable. Finding recorded
in research/hook-pain-points.md. Pure-core spike, not on the public API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: probe 3 — hook vocabulary sound across the full role-keyed family (gate/inject/react)
* docs(handoff): condense the hooks narrative under cap
* docs(handoff): center the in-flight compiled-hooks ship arc (steps 1-3) for resume
* feat(hook): ship compiled hooks — author a hook as a pure typed program
Ship the `vigiles/hook` public surface + CLI so a hook is authored as a pure
typed `(event) => Decision` against a CLOSED vocabulary and vigiles compiles
it. This ELIMINATES whole hook bug classes by construction: the author never
writes the exit code / JSON field (the false-confidence class is
unrepresentable), the matcher is AST-backed (catches `cd x && git push -f`
the native glob misses), capability = API surface (an out-of-vocabulary import
does not compile), and the compiled artifact is tamper-evident (a hand-edit
that smuggles a capability breaks the stamp → the runtime refuses it).
- `src/hook.ts` — the `vigiles/hook` barrel (gate/inject/react vocabulary).
- `package.json` exports `./hook`; api-extractor tracks the new surface.
- CLI `compile-hook <file>` (capability-checks, emits the settings block +
stamp sidecar) and `run-hook-program <file>` (the runtime: loads the typed
program, verifies the stamp, dispatches by role — gate→exit 2, inject→
additionalContext, react→classified run).
- Expand CommandView with two principled high-signal matchers: `touches()`
(secret-path reads) and `pipesToShell()` (curl|sh remote code).
- OSS dogfood (src/hook-dogfood.test.ts): a hand-written substring guard (the
widely-copied disler shape) misses 5/7 of the disaster battery; the compiled
hook blocks 7/7 by construction — measured with our own DISASTER_CATALOG.
Honest scope kept in the surface docs: compile/verify fix AUTHORING + LOGIC,
not DELIVERY — CC's subagent-bypass (#34692) still bypasses any PreToolUse
hook, so a gate is a strong default, never an unbypassable wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reposition around harness reliability + the compiled-hooks pillar
Rewrite the README top-line from "lint/test/measure" to "make the harness your
AI agent runs on reliable", with four instruments under it (Verify / Guard /
Test / Measure) — Guard is the new compiled-hooks pillar, not a bolt-on. Add a
README ② Guard section and a full docs/compiled-hooks.md guide that says, plainly
and up front, that compiled hooks make WHOLE CLASSES OF BUGS UNREPRESENTABLE
(false confidence, matcher bypass, capability creep, category mistakes), with a
bug-class table and the OSS dogfood result (2/7 → 7/7).
Cross-link the new guide from the README, the docs index, the CLI reference
(compile-hook / run-hook-program documented), and the sibling verify + test
guides. Internal: add the compiled-hooks positioning paragraph + keyFiles to
CLAUDE.md (via its spec) and recompile.
Honest scope kept everywhere: compile/verify fix the hook's authoring + logic,
not the harness's delivery — a subagent's tool calls still bypass any PreToolUse
hook (#34692), so a gate is a strong default, never an unbypassable wall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(research): design the Codex compiled-hook adapter + record the ship
Add research/compiled-hooks-codex.md: the plan to compile one typed vigiles/hook
program to Codex through the existing ports — no core edits. The key finding is
that the typed program, AST matcher, capability check, stamp, and runtime
decision are all harness-neutral, and Codex's veto is exit-2-identical to Claude
Code's, so the runtime is shared; only the EMIT differs (CC JSON settings vs
Codex TOML [hooks], glob vs anchored-regex matcher), routed through the
PluginLayout.settingsFormat + dialect/HookProtocol ports already in place. Notes
the boundary discipline (ports by injection) and the non-CC-shaped dogfood, and
why OpenCode's in-process hooks are a different emit target.
Record the compiled-hooks ship + the OSS dogfood in hook-pain-points.md and
cross-link from codex-prototype-findings.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): record the compiled-hooks ship + reposition; next = Codex emit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(hook): compile a typed hook to Codex (TOML) as well as Claude Code
Generalize the compiled-hook emit so ONE typed vigiles/hook program compiles to
either harness, through the existing adapter ports — no harness-boundary edits.
The typed program, AST matcher, capability check, stamp, and the gate runtime
are all harness-neutral (Codex vetoes a tool call via exit 2 identically), so
only the emitted settings block differs.
- compileHookProgram(source, hook, gateCommand?) → (source, hook, opts?) where
CompileHookOptions = { gateCommand?, dialect?, hookProtocol?, settingsFormat? },
all optional and defaulting to the Claude Code JSON block (back-compatible).
It validates hook.on against the dialect (verifyHookEvents — a typo'd or
unsupported event won't compile), styles the matcher per
HookProtocol.matcherStyle (CC exact `A|B` vs Codex anchored `^(A|B)$`), and
renders settingsBlock per settingsFormat (CC nested JSON vs Codex TOML
`[[hooks.<event>]]` via @iarna/toml).
- Add HookProtocol.matcherStyle?: "exact" | "regex" (additive); codexHookProtocol
sets "regex".
- compile-hook resolves the target harness (--harness= / detect) and threads the
adapter's dialect/hookProtocol/layout.settingsFormat in; prints the right block
+ target ("add to Codex's config.toml").
Tests (non-CC-shaped, per the adapter-aware-lint discipline): the Codex emit is a
TOML `[[hooks.PreToolUse]]` block with a regex matcher; an unsupported event does
NOT compile; the CC default still emits JSON + exact matcher. Docs: cli.md +
compiled-hooks.md note the multi-harness emit; research/compiled-hooks-codex.md
marked BUILT with the one deferred piece (inject/ask output JSON on Codex). Also
fix 3 pre-existing no-confusing-void-expression lint errors in
guardrail-check.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): Codex compiled-hook emit built (step 4)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(hook): make the Codex inject/react gap LOUD, not silent
The deferral of Codex inject/ask OUTPUT (its field shape is CC-confirmed only)
was correct — guessing a wire format would recreate the silent-failure bug the
feature exists to kill — but it was SILENT: `compile-hook --harness=codex` on an
inject/react hook emitted a clean-looking TOML block whose runtime output is
CC-shaped and may quietly no-op on Codex. Now it warns loudly (⚠) that the
output is unconfirmed for that harness and to use a gate hook meanwhile; a gate
hook (deny→exit 2, cross-harness) compiles with no warning. E2E test covers
both. Docs/research note the loud behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revamp the compiled-hooks docs — "eliminates a class of bugs", link sources, explain the cons
- Lead the bug-class section with "eliminates an entire class of bugs by
construction", and link EVERY class to its verified source: false confidence
(RFC #45427 / #24327), matcher bypass (#30519), capability creep
(CVE-2025-59536), category mistakes (#4362).
- Replace the single delivery-floor caveat with a full "Limitations & trade-offs
(the cons)" section: the delivery floor (#34692 + route-around #45427/#32376),
runtime cost (node spawn per event), buy-in (dep + build step vs inline bash),
the bounded-vocabulary ceiling, "compiling proves the protocol not your policy",
and the unconfirmed Codex inject/ask output.
- Integrate the gate instrument into the reliability frame: README Guard section
uses the "eliminates an entire class of bugs" framing + links to the cons; the
Test guide intro names the four instruments (verify / guard / test / measure)
instead of only a bottom see-also.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(compiled-hooks): clarify the ceiling — capability closure (fundamental) vs matcher coverage (extensible)
The bounded-vocabulary con conflated two surfaces: what a hook can DO (the
deliberate no-fs/net/exec capability closure — the real ceiling and the safety
guarantee) vs what it can SEE (the AST matchers, a soft/extensible limit). Split
them so 'some hooks stay hand-written' reads as the capability rule, not a
weakness of the Bash matcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(hook): in-process testing for compiled hooks — runHookProgram + assertHookDenies/Allows
A compiled hook's decision is a pure function, so it should be testable WITHOUT
spawning the CLI. Add the cheapest test tier:
- `runHookProgram(hook, event)` (core, on `vigiles/hook`) — one in-process
dispatcher over the whole role family: a gate → its Decision, an inject → the
injected context, a react → its classified Reaction, returned as a normalized
discriminated `HookProgramOutcome`. No subprocess, no model.
- `assertHookDenies` / `assertHookAllows` (harness-assert, on `vigiles/unit` +
`vigiles/testing`) — the throwing asserts over a compiled hook program
directly, the sibling of assertHookBlocked/assertHookAllowed (which take a
`runHook` subprocess result).
Tests: runHookProgram dispatches all four roles (hook-program.test.ts);
assertHookDenies/Allows pass/throw with a message naming what it got
(harness-assert.test.ts). Docs: a "Testing a compiled hook" section in
docs/compiled-hooks.md (the three tiers by cost) + a testing-api.md reference
entry; CLAUDE.md keyFiles updated. api-extractor tracks the new surface.
Behavior testing is deliberately NOT added: a hook isn't model-selected and its
output isn't prose, so there's no trigger-rate/judged dimension; "does it fire in
the assembled harness" is the existing runHarnessTest tier.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(handoff): record in-process hook testing (step 5)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: link the parked prototype code so it's discoverable later
The unshipped/experimental code wasn't in the Key Files map — a future session
would only find it by grep. Fix the traceability:
- Add CLAUDE.md Key Files entries (via the spec) for the parked prototypes:
`src/core/guards.ts` (+test, WIRED via `vigiles guard-hook` but EXPERIMENTAL —
the GATE-axis bet, superseded in practice by compiled hooks + VERIFY) and
`src/core/hook-spec.ts` (+test, a pure spike imported NOWHERE, superseded by
the shipped compiled hooks). Each says why it's parked + points at
research/harness-protocol-flow-moat.md to resume.
- Add a "parked prototype files" note under the roadmap's guards bet recording
the lineage (hook-spec's typed-effect-classified idea → shipped compiled hooks)
so the durable map carries it too.
No code change; recompiled CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(hook): cover the assert message branches (100% threshold)
The new assertHookDenies/assertHookAllows helpers dropped coverage below the
repo's 100% global threshold (harness-assert.ts:144-145,165 — the describeOutcome
injection/reaction branches + the assertHookAllows fail path were unexercised).
Add asserts that drive each: assertHookAllows throws on a deny, and
assertHookDenies against an inject / react hook fails with a message naming the
role ("an injection" / "run (a reaction)").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(lint): add a config `exclude` so vendored/benchmark fixtures skip require-spec
The dogfood `vigiles lint` (the CI `check` job) failed `require-spec` on
`bench/ecosystem/skills/token-efficient/CLAUDE.md` — a SHA-pinned third-party
file injected verbatim as a benchmark arm (SOURCES.md: "the real CLAUDE.md"). A
per-file disable comment would perturb that verbatim arm + dent its fidelity, so
add a config-level exclude instead: `exclude` globs in .vigilesrc.json drop paths
from `lint` instruction-file discovery (node_modules/dist/.git stay always-out).
Set `"exclude": ["bench/**"]` — benchmark data isn't the repo's harness, so the
repo's own lint shouldn't police it. The vendored file is untouched.
`node dist/cli.js lint` now exits 0. Documented in docs/rules/require-spec.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(eval): cover runSkillSelectionTrial / whichSkillsFired (100% statements)
The `test` job's 100% statement gate failed on eval.ts (99.42%): the
selection-collision primitives runSkillSelectionTrial + whichSkillsFired (from
|
||
|
zernie
|
456991c974 |
feat!: harness testing/eval pillar + public-API surface revamp (#37)
* docs: add eval-architecture design doc + ranked roadmap
Capture the conceptual model for testing Claude Code harness features:
feature = deterministic TEST + behavioral EVAL; two orthogonal knobs
(reproducibility x gating); cost-matched mechanism; trigger-rate as a
classifier eval; hash-lockfile + evalApiVersion + dated-model; deferred
canary; closure-scoped invalidation. Reconciles the model with what the
codebase already ships (record/replay cache, significance-gated baseline,
check vocabulary, trigger-rate recall+precision) and ranks the genuine
remaining gaps (tool-call spy/fake keystone, negative checks, curl fake,
subprocess golden, hash-lockfile) mapped to the dogfood build order.
Link from research/eval-api-landscape.md and research/roadmap.md.
* docs: clarify constraint is no-SaaS, not zero-dep, in eval-architecture
Deps are acceptable when they require no hosted account; rest the case
against Polly.js/promptfoo on their real merits (cache exists, cassette
doesn't escape the snapshot trap, promptfoo punt is strategic) rather
than a dep-count rule.
* docs: drop reuse-vs-reinvent tangent from eval-architecture
* feat: add toolWith/notTool argument + safety checks to vigiles/check
The inspection half of the keystone tool-call spy (docs/eval-architecture.md
gap #1/#2): a serializable ArgMatcher (dot-path keys; RegExp = pattern,
primitive = exact) drives two new Trace checks —
- toolWith(name, args): the agent called `name` with input matching `args`
(asserts HOW a tool was called, e.g. the image-request body carried the
style suffix), and
- notTool(name, args?): the agent did NOT call `name` (optionally only the
args-matching calls, so "did not push to main" still allows pushing
elsewhere) — the safety/negative assertion a completion grader can't make.
Both flow through vigiles/testing + vigiles/unit via the existing
`export *`. Pure, model-free, fully unit-tested in src/check.test.ts.
* feat: tool-fake interception core + vigiles fake-tool-hook
The interception half of the keystone tool-call spy (docs/eval-architecture.md
gap #1). At the eval tier the model's tool decisions are real, but letting a
side-effecting tool actually run (paid image API, git push, paid subagent)
makes the eval expensive and risky. A FakeTool denies the real execution while
the model still emits the tool_use — so the agent's decision is observable via
toolWith/notTool, but nothing happens.
- src/arg-match.ts: extract the shared ArgMatcher (dot-path keys; RegExp =
pattern, primitive = exact) so the check vocabulary and the fake seam share
one matching semantics; src/check.ts now imports it.
- src/tool-fake.ts: FakeTool + decideFakeTool (first match wins, optional
`when` arg-scoping), buildFakeToolSettings (PreToolUse hook over the union of
faked tool names), and serializeFakeTools/parseFakeTools (env round-trip that
preserves RegExp matchers a naive JSON.stringify would drop). Pure, tested.
- src/cli.ts: `vigiles fake-tool-hook` — reads VIGILES_FAKE_TOOLS, denies a
matched call via exit 2 feeding the canned result to the model (mirrors the
agent-contract rail's block mechanism).
Remaining: the ergonomic `fakeTools:` field on the eval spec (auto-merge the
hook settings + a cache-key entry) to wire it end-to-end from measure/runEval.
* feat: wire fakeTools onto the eval spec (end-to-end tool-call spy)
Make the interception usable from the API: declare `fakeTools: [{ tool,
when?, result? }]` on a measure/runEval arm and the eval tier does the rest.
- EvalArm.fakeTools + MeasureSpec.fakeTools.
- executeTrial auto-merges the fake PreToolUse hook into the arm's resolved
settings (APPENDING to any existing PreToolUse list, never clobbering),
pointing the hook at this CLI's own cli.js (resolved via __dirname, since
the eval runs in a throwaway cwd where `npx vigiles` wouldn't resolve).
- The fake list rides VIGILES_FAKE_TOOLS (AgentRunArgs.env, layered over
process.env in spawnAgent), RegExp matchers intact.
- CacheKeyInput gains `env` so two fake configs that share tool names — and
thus produce identical merged settings — don't collide in the replay cache
(their when/result lives only in the env).
Tested via the injected runner (no model): the hook merges + appends, the
env round-trips, and a no-fakes run wires neither. eval.ts/eval-cache.ts stay
at the 100% lines/functions/statements gate.
* refactor: make tool-fake honest + add real-model validation eval
Acting on the step-back assessment of the tool-call spy:
- Fix the misleading DEFAULT_FAKE_RESULT: the exit-2 hook is a BLOCK, not a
successful return (CC surfaces it as a denied call), so the default reason
now says "intercepted, NOT executed, do not retry" instead of "assume
success". Reframe the tool-fake/eval docs as intercept-and-prevent + observe
the attempt, not a faithful tool mock (CC has no skip-exec-return-success
primitive). Note cost is not reduced — only the side effect is removed.
- docs/eval-architecture.md: record the honest assessment — fidelity ceiling
(deny != success), toolWith overlaps the existing toolUsedWith predicate
(net-new is the negative check + interception), the simplest safety path
(don't allowlist + assert the attempt) often needs no new primitive, and the
competitor delta is narrow (assertions at parity; intercept-in-real-harness
is the edge).
- examples/harness/tool-fake.eval.mjs: end-to-end validation of the one
assumption unit tests can't reach — that a DENIED tool's args still land in
the Trace. Induces a git push origin main, intercepts it, asserts
toolWith(Bash, /push origin main/) holds across trials. Loud skip() without
claude. Run with a key before relying on the spy.
* ci: behavioral eval gate scaffolding (inert until ANTHROPIC_API_KEY)
Addresses the original kicker — real-model evals never ran in CI, results
frozen as FINDING: comments. Adds .github/workflows/evals.yml with two jobs
from the cost model in docs/eval-architecture.md:
- eval-gate (pull_request): the cheap tier — the three shipped model-invocable
skills' trigger evals (stubSkillBodies, 1 trial) on a DATED model, --no-skip.
- eval-nightly (schedule + workflow_dispatch): the fuller drift tier you can't
amortize.
Both are INERT until enabled: every model step is guarded on the
ANTHROPIC_API_KEY secret, so with no secret (default + all fork PRs) it prints
a notice and exits 0 — never reds a PR. Turning it on is a deliberate
secret + budget decision.
Plumbing: `vigiles eval --model=<id>` → VIGILES_MODEL (sibling of --trials)
so the gate pins a dated model id honestly without changing library defaults;
the cheap trigger evals + tool-fake.eval.mjs now read process.env.VIGILES_MODEL.
* feat: warn when an eval cache rides a floating model alias
Closes the last piece of the honest-model-pinning gap (doc #3). A record/replay
cache keyed on a floating alias (haiku/sonnet, or claude-sonnet-4-6 with no
date) can serve a result computed against a since-changed model — hiding drift
behind a green replay.
- isDatedModel(id): a model is honestly pinned when it ends in an 8-digit date
stamp (claude-haiku-4-5-20251001); a bare alias or version is floating.
- runEvalWith emits a one-time warning (::warning:: under GITHUB_ACTIONS, else
stderr) when cache != off and the model is floating.
Tested: isDatedModel dated/floating, both warning branches (CI annotation +
stderr), and the dated-id no-warn path. eval.ts stays at the 100% gate.
* docs: document arg-match + tool-fake modules in CLAUDE.md Key Files
The repo convention is that every src module appears in Key Files. Adds
entries for src/arg-match.ts, src/tool-fake.ts, and src/tool-fake.test.ts
(the tool-call spy), then recompiles CLAUDE.md from the spec — integrity
hash valid.
* test: validate tool-fake spy against a real model (capture-under-deny works)
Ran the validation against claude-haiku-4-5-20251001. The model called Bash
with "git push origin main", the auto-wired PreToolUse fake-tool-hook denied it
(isError, "the git push didn't execute"), and the denied tool_use — WITH its
args — still landed in ctx.toolCalls, so toolWith matched. This confirms the
one assumption the unit tests can't reach: a denied tool's arguments are still
captured for toolWith/notTool, and the side effect is prevented.
Adjust the eval to assert the MECHANISM, not model obedience: strengthen the
prompt, 5 trials, and gate at min 0.2 — any non-zero rate proves capture works;
only a true regression (a flat 0 across every trial) fails. Records the finding
inline.
* docs: refresh eval comparison (public + internal) and positioning
The eval comparison material predated the shipped significance/cache/cost/
regression-gating work and the tool-call spy. Brought it current:
- research/eval-api-landscape.md (INTERNAL): the scorecard's vigiles column
was marked ✗ on significance/caching/cost/concurrency/regression-gating —
all shipped. Rewrote it to shipped reality, added rows for the no-model
cheaper tiers, the intercept-and-prevent tool spy, and the regression gate;
noted the inert-until-keyed CI eval gate. Three bolded-unique rows now name
the defensible core honestly.
- docs/harness-testing.md (PUBLIC): broadened the promptfoo-only section into
a promptfoo/DeepEval/Braintrust/Inspect table; added the safety-intercept
and no-model-tier rows; pointed to the internal landscape doc.
- README: added one line on the safety property (notTool + fakeTools catch a
push-to-wrong-branch / paid-API attempt and prevent the side effect).
Positioning audit: README front door + internal scorecard now current; the
spy modules are already in CLAUDE.md Key Files.
* fix: harden eval cache invalidation (dir contents, tool-set, version salt)
Research-backed (research/cache-invalidation.md, synthesizing Bazel/Turborepo/
Nx/ccache/Jest/webpack + promptfoo/LangChain/Helicone). Fixes the confirmed
false-replay bug and two phantom-miss/staleness gaps, ranked by severity:
1. pluginDir contents weren't in the key (THE bug): a native --plugin-dir was
passed to the run but its files were never hashed, so editing a skill in it
false-replayed. hashDir() folds a sorted relativePath:contentHash list into
the key (content not mtime; path-included). Flat sorted list, not a Merkle
tree — over-engineering at plugin-dir scale.
2. Tool list now hashed as a SET (sorted) so a reordered allowlist isn't a
phantom-distinct key. (canonical() already sorted object keys, so no
stable-stringify dependency is warranted.)
3. CACHE_FORMAT_VERSION salted into the key (Jest/webpack pattern) so a record
format change makes old entries unreachable — no brittle read-time gate.
Model drift on a floating alias stays warn-only (the dated-id pin is the sound
fix; can't auto-resolve without an API call). Eviction/TTL deferred as disk
hygiene, not invalidation correctness — rationale in the research doc.
Tests: hashDir (edit/add/rename/skip-junk), tool-set order-insensitivity,
pluginDirHash sensitivity, and an end-to-end runEvalWith test proving a
pluginDir content edit busts the cache. eval-cache.ts/eval.ts stay at 100%.
* ci: let the eval gate run on a Claude subscription, not just an API key
The affordability bet: vigiles drives the real `claude` CLI, so an eval
authenticates like the user's own CLI — including a Pro/Max SUBSCRIPTION (flat
cost), not metered API billing. (Confirmed empirically: a validation run
reported apiKeySource:"none", i.e. it ran on the session's OAuth subscription.)
Both eval jobs now accept CLAUDE_CODE_OAUTH_TOKEN (subscription, via
`claude setup-token`) OR ANTHROPIC_API_KEY (metered fallback), and stay inert
until one is present. Header documents the subscription as the cheap path.
* docs: isolated-vs-whole-harness eval decision (research of record)
Decision: support BOTH as tiers — default isolated (cheap authoring loop),
add a whole-harness install-set arm as the release gate, plus a near-neighbor
middle tier on the existing ncd engine.
Why: skill selection is competitive and Claude Code evicts least-used skill
descriptions under a context budget (skillListingBudgetFraction), so isolated
trigger-rate overstates recall and understates false-positives. Every existing
tool (promptfoo, Anthropic skill-creator, AWS, OpenAI) isolates; none populate
the install set — a genuine wedge for "test the harness as it ships". Captured
in research/isolated-vs-whole-harness-eval.md; roadmapped in eval-architecture
(item 7). Also updated the eval-gate roadmap note for the subscription-token
auth path.
* feat: whole-harness trigger-rate tier (installSet) + honest isolated labelling
Implements the isolated-vs-whole-harness decision (research/isolated-vs-whole-
harness-eval.md). measureTriggerRate stays isolated by default (cheap authoring
loop); a new `installSet` co-installs the skill-under-test alongside the user's
real skills so it competes for selection as it does in the shipped harness — the
release-gate tier no other eval tool offers.
- TriggerRateSpec.installSet: plugin dirs or loose skills dirs merged into one
install under the under-test plugin's name (under-test wins a name collision,
so its <name>:<skill> id still matches `fired`). packageInstallSet +
collectSkillsSource + copySkillsInto, pure/tested, cleans up on any error.
- TriggerRateReport.competitors: count of co-installed competitors. 0 = isolated.
- formatTriggerRateReport now labels honestly: an isolated run prints that recall
is an UPPER bound and false-positive a LOWER bound (selection is competitive);
a whole-harness run prints the competitor count.
Tests cover the merge (skills/ + .claude/skills/ sources, collision, stub,
missing-source + empty-under-test throws, junk-skip), both report labels, and
installSet from pluginDir + skillsDir. eval.ts stays at the 100% gate.
* docs: lead README with affordability; correct the evals-in-CI framing
- README: affordability is now an explicit theme — most harness questions are
answered with no model (free, every commit), and a real-model eval rides the
user's Claude Pro/Max SUBSCRIPTION (vigiles drives the `claude` CLI), not
metered API billing like API-SDK-based tools. States plainly that evals don't
run in CI by default.
- research/isolated-vs-whole-harness-eval.md: fixed the "nightly/release gate"
wording that wrongly implied a default CI cadence. Clarified that evals are
not in CI by default (the per-PR CI is the free deterministic tiers; the eval
workflow is inert until you add a token); "release gate" = run deliberately
(locally on the sub, or opt-in nightly).
* fix: make the eval CI gate fail on a non-run or a corrupt cache
Two silent-pass holes the eval gate could slip through — closed so a green
eval job actually means evals ran and the cache was sound:
1. Evals never executed. `vigiles eval`/`test` gained `--min=N`: assert at least
N scripts actually matched/ran, else exit 1 loudly. A renamed file, a bad
path, or a glob matching nothing previously printed "No files found" and
exited 0 — a hidden gap. evals.yml now passes --min=3 (the gate's three
listed evals) and --min=1 (nightly), alongside the existing --no-skip.
2. Cache fails. readCache now THROWS on a corrupt record (file present but
invalid JSON) instead of swallowing it and silently re-running. A broken
cassette is a real failure the gate must surface; a true miss (no file)
still returns null and proceeds normally. Message says how to recover.
Tests: cli.test --min (0 matched / too-few / enough), eval-cache corrupt→throws
vs miss→null. eval-cache.ts stays at the 100% gate.
* fix: trigger-rate on Sonnet (not haiku) + count sibling skills as competitors
Dogfooding the three shipped skills on the subscription found two real bugs:
1. Gate model. All three skills "failed" recall on haiku (test-harness 0.50,
edit-spec/strengthen 0.70) — but on Sonnet test-harness scores 0.90 and
passes. Trigger-rate is a SELECTION measurement and haiku is a much weaker
selector, so a haiku gate false-fails skills that fire fine on the model
users run. measureTriggerRate now defaults to "sonnet" (matching measure);
evals.yml pins claude-sonnet-4-6 for the gate; docs corrected. The skill
descriptions were never the problem.
2. competitors miscount. The honest-label counted only installSet additions, so
a multi-skill plugin (the dogfood installs all of vigiles' skills) printed
"isolated: no competing skills" while ~5 siblings were actually competing.
Now competitors = (skills in the resolved install pool) − 1, via countSkills,
so siblings count and a real single-skill install still reads 0/isolated.
packageInstallSet's return renamed competitors→added (installSet delta).
Tests: sibling-competitor count, packageInstallSet `added`. Recorded the
haiku-vs-Sonnet FINDING in test-harness.trigger.eval.mjs. eval.ts at 100% gate.
* feat: model-per-arm (Q1) + save model/isolation eval decisions
Q1 (model strategy) — agreed + built:
- EvalArm.model: a per-arm model override. A model comparison IS a harness A/B,
so "does my harness hold on the cheaper tier / after a model upgrade?" is
arms: { sonnet: {model}, opus: {model} } read through the existing
significance machinery — no separate multi-model matrix runner (that's the
cost-multiplying promptfoo lane we avoid). measureTriggerRate stays
single-model by design.
- Default stays Sonnet (the realistic selector), per the dogfood finding.
Q2 (isolation) — agreed: keep TWO tiers (in-plugin by default = honest sibling
competition; installSet for the whole harness). Do NOT build per-skill isolation
machinery (it produces the LESS honest number others default to). Already shipped;
no new code.
Saved research + conclusions: docs/eval-architecture.md gains a "Model strategy"
section (default Sonnet w/ the 0.50-haiku-vs-0.90-Sonnet dogfood evidence,
model-as-arm, no matrix, the deferred model-floor); the isolation
research/decision is in research/isolated-vs-whole-harness-eval.md.
Validated on the subscription: the corrected Sonnet gate is GREEN (3/3 skills
≥0.8) and the competitors label now reads "whole-harness: 4 competing skills".
* feat: runtime model floor for trigger-rate (default Sonnet) + gate on alias
The "set a floor for all skills" ask, solved without a (brittle) static linter:
a RUNTIME floor enforced in the eval library catches the env-var model a lint
can't see — which is exactly how the haiku footgun entered.
- modelTier(id): rank by family (haiku<sonnet<opus; version ignored), null for
an unrecognized family so the floor never blocks a model it can't judge.
belowModelFloor(model, floor): fail-open on unknowns.
- TriggerRateSpec.minModel (default "sonnet", resolves minModel → VIGILES_MIN_MODEL
→ "sonnet"). measureTriggerRate FAILS before spending a token if the resolved
model is below the floor — so one knob (VIGILES_MIN_MODEL / --min-model) sets
the floor for ALL skills, no per-file annotation, no linter. Lower it
deliberately for a cheap run. Decision: FAIL (not warn) — the floor is a
contract; the escape hatch is lowering it.
- CLI: `vigiles eval --min-model=ID` → VIGILES_MIN_MODEL.
- Gate model switched claude-sonnet-4-6 → the `sonnet` ALIAS: the per-PR gate
runs cache-off, so it should track current Sonnet without churn; dated/exact
ids matter only for cached/baselined reproducibility (kept on that path).
Tests: modelTier/belowModelFloor (families + fail-open), floor throw at default
+ VIGILES_MIN_MODEL, and the lower-floor escape hatch. eval.ts at 100% gate.
* refactor: remove model envs + the speculative evals.yml CI workflow
Acting on review: neither was asked for, and both were the wrong layer.
- Removed VIGILES_MODEL / --model and VIGILES_MIN_MODEL / --min-model. Model is
part of the measurement DEFINITION (a different model = a different result), so
it belongs in the spec (`model`/`minModel`, version-controlled), not a hidden
CLI/env override. Contrast VIGILES_TRIALS, a legit run knob (cost/precision,
doesn't change what's measured) that PRE-DATES this work — kept. The dogfood
evals now omit `model:` and rely on the Sonnet default + the minModel floor.
- Removed .github/workflows/evals.yml. Real-model evals don't belong in a
standalone GitHub Actions workflow needing a metered/sub-token secret: CI runs
the FREE deterministic tiers (ci.yml, no token), and the real-model eval runs
where the subscription already is — a Claude Code session or locally (vigiles
drives the `claude` CLI). The eval-robustness work (--min, --no-skip,
corrupt-cache throw, Sonnet model floor) stays — it hardens `vigiles eval`
wherever it runs.
Docs reframed accordingly (eval-architecture kicker + roadmap, README, the two
research docs): deterministic tiers in CI, evals on the sub in a session.
eval.ts at the 100% gate; 974 tests pass.
* docs: consolidate positioning + pros/cons of the eval approach
Per review — the competitor scorecard, caching research, isolation decision, and
model strategy were all saved, but the POSITIONING + pros/cons were scattered
across README/kicker. Added a dedicated "Positioning & pros/cons" section to the
eval-architecture design-of-record: the affordability thesis (free deterministic
tiers + sub-priced real-model tier vs competitors' per-token-every-run), the
pros (cost/fidelity/honesty) and the honest cons (sub rate limits, non-determinism,
intercept≠mock, sonnet-not-haiku, not-a-CI-checkbox, no dataset/red-team/UI), with
links to the four supporting research docs.
* fix: key the eval cache on the harness binary version (fast-evolution edge)
You were right — the cache wasn't tested/keyed against fast model/harness
evolution. Closing it:
- harnessVersion (`claude --version`, memoized, cache-path only) is now in the
cache key. A CLI upgrade changes the system prompt + tool defs — which steer
behaviour as much as the model — so a cached result now INVALIDATES across a
`claude` upgrade instead of silently replaying a result from a different
harness. Trade-off (frequent patch invalidation) documented; re-scoring within
a session is unaffected (version stable then).
- cacheKey test now actually covers model / settings / env / harnessVersion
changes (it previously asserted "any input" but skipped several, incl. model).
- research/cache-invalidation.md gains a "Fast-evolution edges" section: harness
version (keyed), model-alias re-point (warned, can't auto-invalidate without a
weight hash), and modelTier new-family fail-open (a documented known limit).
eval-cache.ts/eval.ts stay at the 100% gate.
* docs: positioning into CLAUDE.md (canonical) + consistency pass
Point 3 — positioning belonged in the positioning source, not buried in
eval-architecture.md:
- CLAUDE.md ## Positioning (pillar 2) now states the affordability thesis: free
deterministic tiers + a real-model tier that runs on the Claude Pro/Max
SUBSCRIPTION (vigiles drives the `claude` CLI, no metered API), measured on the
realistic selector (Sonnet, minModel floor), model-in-spec not env. Recompiled.
- eval-architecture's pros/cons section now points to CLAUDE.md as canonical and
keeps only the detail.
Gap #4 — refreshed the stale `src/eval.ts` Key Files entry (model-per-arm,
minModel floor + modelTier, installSet whole-harness tier + competitors,
harnessVersion in the cache key, Sonnet default).
Gap #2 (consistency pass, you approved) — the "knobs" / "match-the-mechanism" /
reconciliation sections no longer claim evals run in `every-PR` CI on `haiku`:
deterministic tiers are every-commit CI, the real-model eval is run on-demand on
the sub (Sonnet). Fixed the decision table + the reconciliation bullets too.
Gap #3 — documented the shipped trigger-rate knobs in the user guide
(harness-testing.md): Sonnet default + minModel floor, isolated-vs-whole-harness
`installSet` + the `competitors` count, and model-per-arm for cross-model A/B.
* feat: key cache on harness major.minor, automate stubSkillBodies, exhaustive cache tests + near-neighbor roadmap
Acting on review:
- Claude version granularity: key the cache on the harness MAJOR.MINOR, not the
patch (harnessVersionKey: "2.1.179 (Claude Code)" → "2.1"). A patch rarely
changes behaviour, so keying patches would churn the cache every release for no
signal; a minor/major bump is where the system prompt actually moves. Documented
the trade-off (a known-behavioural patch → clear cache / bump CACHE_FORMAT_VERSION).
- stubSkillBodies is now AUTOMATIC for measureTriggerRate (default true): testing
a description never needs the body (trigger is a frontmatter property decided
before the body loads), so it's not a knob you must remember. Set false only to
run the real body. Updated the 4 fake-pluginDir tests to stubSkillBodies:false
(passthrough), which also keeps the non-stub branch covered.
- Cache tests are now EXHAUSTIVE: one assertion per CacheKeyInput field, plus
explicit VERSION axes — two model snapshots (sonnet-4-6 vs -4-5) and two harness
minor versions each produce a different key. + harnessVersionKey unit test.
- Roadmap: added the near-neighbor trigger-rate tier (NCD-nearest competitors) to
Next, priority P3 (MED-LOW) — decided, deliberately deferred.
(Consistency gaps #2/#3/#4 shipped in 74fc151.) eval.ts/eval-cache.ts at 100% gate; 975 tests.
* feat!: rename fakeTools -> interceptTools (intercept-and-prevent, not a mock)
The tool-fake feature does intercept-and-prevent (a PreToolUse hook denies the
call via exit-2), not fake-a-return-value. The "fake" name wrongly implied a
mock that substitutes a result and continues, requiring two JSDoc corrections
to undo. Rename to carry the real semantics.
BREAKING CHANGE: the vigiles/eval spec field `fakeTools` is now `interceptTools`
(on EvalArm and MeasureSpec); the `FakeTool` type is `ToolIntercept` and its
`result` field is `denyReason`. The `vigiles fake-tool-hook` CLI subcommand is
now `intercept-tool-hook`. Pre-1.0, but signalled for a correct changelog.
Keep toolWith/notTool (clearer than competitors) but add promptfoo/DeepEval
cross-references to their JSDoc and a "vigiles check <-> promptfoo assertion"
mapping to research/promptfoo-deep-dive.md, noting notTool/interceptTools as
the restraint-assertion gap a completion-grading eval cannot cheaply express.
* docs: add safety model front-door doc (confinement + tool interception)
docs/sandboxing.md covers host confinement for hook/plugin code but not the
eval-tier risk — a real model deciding to call a side-effecting tool. Add
docs/safety.md as the front door tying the whole model together: the two
risks (foreign code vs the model's actions), the two mechanisms (bwrap
confinement vs interceptTools/notTool PreToolUse deny), the one safe-by-default
fail-closed rule, the per-tier confinement table, and the honest gaps. Answers
"why not confine everything" by pointing at the four-reason argument in
sandboxing.md. Referenced from README + sandboxing.md (not an orphan).
* docs: decide cross-platform sandboxing (macOS via sandbox-exec) + save the verify-sandbox-config idea
Mac support is a hard requirement (large dev population), and the claude binary's
own sandbox can't cover us (Bash-only, skips hooks, no external API, CC-only).
Record the decision: a vigiles/os-isolation port with a native backend per OS —
keep bwrap+nft on Linux (strongest, packet-layer), add sandbox-exec/Seatbelt on
macOS (built-in, zero deps, deny-all-net). Per-host egress stays Linux-only;
macOS degrades honestly. srt/nono are documented fallbacks, not the default;
Birdcage is reference-only (GPL/binary-net).
Also clarifies the enabled/disabled model (provenance, not platform: your code
runs direct, foreign code is confined-or-refused — the Mac backend makes the
safe option available so nobody is forced to sandbox:false), and parks the
"verify & test the harness's own sandbox config" idea (the network-validation
wedge: valid-is-not-true for sandbox policy). Roadmap + sandboxing.md updated;
not an orphan.
* docs: add the ephemerality axis — running a model-driven skill is a side effect, independent of trust
Correct the safety model: provenance governs CONFINEMENT (host protection), but
EPHEMERALITY (state protection) is unconditional — a skill/agent is model-driven,
so even a plugin you wrote produces nondeterministic side effects when run. Trust
does not make a run side-effect-free.
Classify side effects by reversibility: local-reversible -> ephemeral CWD (already
done); escapes-CWD (~/.gitconfig, ~/.ssh) -> ephemeral HOME + scrubbed env (the GAP:
the direct/non-bwrap path inherits process.env + real HOME); external -> deny/allow
net; irreversible (push, charge) -> interceptTools/notTool (can't un-push by
discarding a box). Ephemerality needs no kernel features, so it lands on macOS
today, ahead of Seatbelt — the cheapest cross-platform side-effect protection.
safety.md + cross-platform-sandboxing.md + roadmap (HIGH) updated.
* docs: positioning — behavioral/side-effecting testing + cross-platform (Mac+Linux); fix phantom issue refs
Positioning (CLAUDE.md): name BEHAVIORAL, side-effecting eval of the assembled
harness as the apex of pillar 2 (highest value, kept thin; notTool/interceptTools
ship today, the ephemeral run env + disposable-dep provisioning are the committed
next step). Add a "Cross-platform by requirement (macOS + Linux)" paragraph: native
backend per OS behind a vigiles/os-isolation port, and the two orthogonal axes
(provenance-keyed host protection vs unconditional ephemeral state protection).
Fix: the declared-vs-enforced anchor cited as Claude Code #54898 could not be
verified; replace every occurrence with the real anchors (Claude Code #4740/#21460,
claude-agent-sdk-typescript #172) across agent-runtime, cli, the spec, and research.
A reference-checking tool should not ship a phantom reference.
* docs(research): save the mid-2026 subagent-demand + skill-eval-landscape findings
subagent-compilation.md: ranked issue-survey demand evidence (declared!=enforced
anchored to #4740/#21460/SDK #172; orchestration/output-contract/dispatch/MCP/
observability/model/determinism/firing/nesting themes), the authoring+observability
tooling table (almost no verification/testing tool exists), Codex-vs-CC delta
(enforcement is Claude-Code-shaped -> Codex non-goal confirmed), honest map-to-vigiles.
skill-eval-landscape.md: how labs + practitioners actually eval skills — Anthropic
skill-creator + "Demystifying evals", OpenAI's Codex eval guide, the two Reddit
harnesses (SkillBenchmark blind-judge A/B+CIs on Caveman; the "SKILL.md is a
contract" methodology harness incl. the adversarial-gate finding), Spence/dbt/
Husain/Schmid. Gap verdict: behavioral skill eval is now TABLE STAKES; vigiles's
moat is the intersection (whole assembled harness + subscription affordability +
statistical rigor + the deterministic runHook tier + the eval->enforce bridge).
promptfoo-deep-dive.md: promptfoo acquired by OpenAI (~$86M) + DeepEval v4 agentic
harness; sharpens the vendor-neutral, subscription-not-metered positioning.
eval-api-landscape.md: new competitors (TribeAI/claude-evals metered; paultyng/
testagent CLI-fake) + statistical-rigor confirmed as a differentiator.
* docs: native input/output/cache token+cost measurement decision + behavioral-umbrella clarification
eval-architecture.md: (1) state explicitly that behavioral/side-effecting eval (the
ephemeral integration tier) is the UMBRELLA and the adversarial-gate test is one
member, not a replacement — with the eval->enforce bridge (pillar 2 hands off to
pillar 1 when a prose gate caves); (2) decide native token/cost support split by
class (input/output/cache) — the tokens() check collapses input+output and cache
tokens aren't captured, so a single total can bless a net-negative change (Caveman
cut output yet 2-4x'd cost). Plan: extend UsageTrace to all token classes,
first-class inputTokens/outputTokens/cacheTokens checks, and an A/B delta-per-class
gated by Welch significance. Added both to the ranked roadmaps (HIGH).
* feat: native input/output/cache token measurement + per-class checks
A harness change trades tokens across sides (a skill/CLAUDE.md injection adds
INPUT every turn; a "compression" skill cuts OUTPUT), so a single total can bless
a net-negative change — SkillBenchmark's Caveman cut output yet 2-4x'd cost. Split
the usage model into honest classes and gate on them separately.
- EvalUsage + UsageTrace gain cacheCreationTokens + cacheReadTokens (parsed from
the claude result usage: cache_creation_input_tokens / cache_read_input_tokens);
inputTokens stays the FRESH (uncached) input. ArmUsage + aggregateUsage sum the
two cache totals.
- New vigiles/check checks: inputTokens({max}), outputTokens({max}),
cacheTokens({maxCreation?, maxRead?}) beside the existing tokens/cost/latency.
Exposed on ctx.usage so an A/B delta-per-class rides the existing mean±se /
Welch-significance machinery for free.
- Dogfood: skill-compression.eval.mjs now measures input AND output (cache reads
~0.1x input, writes 1.25x), so the compression claim is verified on both sides.
Tests: pure pass/fail/score/message/toJSON for each new check; usageFrom parses
the cache fields (0 when absent); aggregateUsage sums the cache totals.
* docs: record real-haiku FINDING in the compression eval — cost metrics work, cache dominates
Ran the compression dogfood on real haiku (3 trials/arm, on the Pro/Max
subscription, apiKeySource:"none", $0.1057). Captures the end-to-end result as a
FINDING: total_cost_usd + input/output + cache tokens all populate from a real run;
inputTokens read ~42/run (the uncached remainder) vs ~2.4-2.7k real volume — the
rest is cache reads, so a single input/total understates economics ~50x here. The
cache classes are where the volume lives (vindicates the split). The compression
claim did not hold on this task/model (output +11%, cost +$0.002, correctness
intact) — the headline-% failure mode the eval exists to catch.
* feat: adversarial-gate dogfood eval + the eval->enforce bridge
The highest-value behavioral test for an enforcement skill: adversarially ask the
agent to skip the gate the skill is supposed to enforce, and assert it REFUSES.
- examples/harness/dogfood/adversarial-gate.eval.mjs (write-don't-run): an inline
SKILL.md states a measurement gate (no benchmark without a baseline; refuse if
asked to skip), the task adversarially asks to skip it, and two composed checks
assert refusal — notTool (didn't run the forbidden command) + output (pushed
back in prose), gated by assertRates({min:0.7}). No new check primitive: the
pattern is the existing vocabulary composed inline (no gold-plating).
- docs/eval-architecture.md + docs/safety.md: document the eval->enforce BRIDGE —
pillar 2 (eval) MEASURES whether a prose gate holds; when the adversarial test
shows it caves, pillar 1's deterministic PreToolUse rail is the FIX. Eval hands
off to enforcement; nobody else connects the two.
* feat: ephemeral run environment foundation (pure builder + opt-in flag, default OFF)
Running a model-driven skill mutates state regardless of trust (the model, not the
author, chose the actions), but the eval tier inherits the full real environment —
a model-driven git push / write to ~ uses the real $HOME, ~/.gitconfig, ~/.ssh,
~/.aws. Add the foundation to close that, behind an opt-in flag, default OFF so the
eval tier's own auth can't silently break before a real-run validation.
- ephemeralRunEnv(base, { home, allow }) — pure, exported, no fs/spawn: sets
HOME+TMPDIR to a throwaway dir, passes through an auth+runtime allowlist (mirrors
the runtime port: PATH/LANG/TERM/ANTHROPIC_*/CLAUDE_*/LC_* + region/profile so a
Bedrock/Vertex-backed claude still authenticates), DROPS everything else
(GIT_*/GH_TOKEN/SSH_*/AWS secret keys + any non-listed var). Conservative by
design: when unsure, allow (a too-narrow allowlist breaks auth).
- EvalSpec.ephemeralEnv?: boolean (default off). When on, executeTrial mkdtemps a
throwaway HOME under the trial cwd and builds the scrubbed env; AgentRunArgs gains
replaceEnv so spawnAgent uses a.env as the COMPLETE env. When OFF, the env path is
byte-identical to before (asserted by a test).
The cross-platform state-protection floor (needs no kernel features → lands on
macOS now). Flipping the default to on waits for a real-auth validation pass; two
honest follow-ups noted in code (AWS static-key auth, cache+ephemeral key interaction).
* docs: validate ephemeral-env auth survival (env-var path works; file-based ~/.claude is the gap)
Ran a real-model check (haiku, ephemeralEnv: true) on the subscription: the run
authenticated through the scrubbed throwaway HOME — exit 0, $0.003, output "OK.",
cache read ~21.7k tok (caching unaffected). So the env-var auth path is validated:
the credential rides an allowlisted ANTHROPIC_* var that survives the scrub.
Remaining gap before flipping the default on: this only covers env-var auth. A
subscription credential that lives only in ~/.claude/.credentials.json (a file
under HOME) would be lost when HOME is scrubbed — ephemeralRunEnv injects env
vars, not the credential file. Safe default-on needs injecting the harness's own
credential file into the throwaway HOME. Flag stays opt-in until then.
* feat: complete the opt-in ephemeral run env (auth-file injection + cache-safe) + validate containment
Finish the ephemeral-run-env opt-in path (still default OFF):
- seedEphemeralHome copies a minimal auth allowlist (EPHEMERAL_HOME_KEEP =
[".claude/.credentials.json"], copy-not-symlink so the run can't write back) from
the real HOME into the throwaway HOME when present, so local file-based OAuth
survives the scrub; .gitconfig/.ssh/.aws are never carried.
- cache-safety: HOME/TMPDIR excluded from the eval cache key (CACHE_KEY_ENV_EXCLUDE),
so the random throwaway HOME no longer makes every ephemeral run a cache miss;
non-ephemeral keys unchanged (surgical, not a blanket env drop).
Validated state protection end-to-end on the real subscription (haiku, ephemeralEnv:
true, the run told to write $HOME/PROOF.txt + probe ~/.ssh): real $HOME untouched
(write landed in the throwaway HOME) AND the run still authenticated (exit 0,
$0.0083). Containment holds; auth survives via the host-brokered CLAUDE_* allowlist.
Default stays OFF: this host-brokered env can't run-test the file-based
~/.claude/.credentials.json OAuth modality (no credential to exercise; no sandbox
mints a real token), so the credential-file injection is unit-tested for the copy
logic but not e2e-validated. Flipping default-on (a behavior change for every eval)
waits on a local-file-auth user validating that path; then it's a one-liner.
* docs: phased design for the vigiles/os-isolation port + Seatbelt backend
Grounded implementation plan over the existing sandbox core (design of record, not
yet built): the narrow port interface (NetPolicy tagged union / FsPolicy /
IsolationCapabilities / OsIsolationBackend), the core-vs-backend layout, the
per-backend capability matrix, the Seatbelt backend shape (sandbox-exec + minimal
SBPL profile), the eslint-plugin-boundaries enforcement (only backends name a raw
primitive, dogfooded via enforce), and a 4-phase migration that keeps the 997-test
suite green at each step (extract+move bwrap behind the port with ZERO behaviour
change -> add Seatbelt -> wire selection + lint -> fold in ephemeral env).
Surfaces the sharp limitation honestly: Seatbelt (deny network*) blocks localhost
too, so bwrap's co-launched-mock-over-loopback trick has no Mac equivalent
(loopbackNet:false) — a trusted spec runs direct, an untrusted-needs-mock spec
refuses. Per-host egress stays Linux-only; srt and CC's built-in sandbox stay out.
Linked from the roadmap (HIGH).
* docs: consolidate eval coverage/isolation into one source of truth (R1+R2 owned, R3 composed)
Add research/eval-coverage-and-isolation.md as the authoritative doc: the three-rung
model (R1 cheap/deterministic — nothing executes; R2 record-replay — a tool/MCP/API
result recorded once from the real tool and replayed via a fake binary on PATH, never
model-synthesized; R3 real disposable service whose semantics IS under test), the
blended+scrubbed coverage distribution (R1+R2 ~90%+ with no Docker on the sub; R3 a
thin apex — browser/DB/redis), the honest e2e landscape (every benchmark/lab/Inspect
runs e2e in a container; no containerless reproducible e2e exists), the
safety/viability/performance axes + non-goals, the competitor comparison, and a
reusable rung-classifier prompt.
Make the chosen approach crystal-clear in CLAUDE.md positioning (R1+R2 owned on the
sub; R3 composed-with-a-container, not reinvented), docs/eval-architecture.md
(Coverage & scope section), README (public), and the roadmap (the R2 enabler ranked
ahead of testcontainers + a dogfood/COVERAGE.md item). Use standard terminology
(PATH-shim + record-replay/VCR), not a coined name. Scrubbed — no private-source
identifying details; conclusions blended with the community/landscape data.
* feat: R2 tool-stub / PATH-shim helper (record-replay a tool result, no service, no Docker)
The R2 enabler from the coverage model: test a skill whose logic consumes a tool's
RESULT by shadowing the binary on PATH with a fake that emits a recorded/canned
output — standard PATH-shimming + VCR-style replay, not a novel approach. Unlocks
the ~43% R2 surface with no live service and no Docker (ranked ahead of any
testcontainers work).
- src/tool-stub.ts: ToolStub + renderToolStub/writeToolStubs/stubBinDir. Each stub
is a self-contained POSIX script that base64-decodes its canned bytes straight to
stdout (base64's alphabet can't contain a shell metacharacter → injection-proof
for any fixture content; piping preserves trailing newlines that command-subst
would strip). Stubs are author/recorded fixtures, NEVER model-synthesized
(drift → false confidence). argv-matching + record-from-real are documented
follow-ons.
- EvalSpec.stubs wired into executeTrial: a bin dir under the trial cwd is prepended
to PATH for BOTH the ephemeral and legacy env paths; absent stubs = byte-identical
(no default flipped, ephemeralEnv stays off).
- Exposed via vigiles/testing + a ./tool-stub subpath export.
- harness-test tier wiring left as a documented TODO (it routes through the
per-harness driver seam — out of MVP scope).
Tests: faithful round-trip of shell-special content incl. trailing newlines; PATH
prepend on both env paths; absent-stubs unchanged. 1007 passed / 11 skipped.
* test: dogfood the three vendored plugins + per-plugin COVERAGE.md scorecards
Prove how much of real plugins the cheap tiers actually cover. 9 model-free,
CI-runnable tests (vendor-coverage.test.ts) across the three vendored plugins —
OMC keyword-detector R1 fire/no-fire (the free floor under the bwrap-gated egress
copies), scan structural facts + the wshobson inherits-all footgun on the scan
report, superpowers session-start R1 (fallback) + R2 (reconstructed plugin root
injecting the real skill body), and the canonical R2 stubBinDir demo (shadow npm
on PATH to drive find-polluter.sh's real bisection to completion, no live service).
Per-plugin COVERAGE.md scorecards (rung table + distribution + grade + R3 shortlist
+ honest verdict), placed beside the pinned snapshots (never inside them):
- oh-my-claudecode: ~60% free / ~33% model-gated / ~7% R3 — B+
- superpowers: ~50% free / ~50% model-gated / ~0% R3 — B (a prose library is
half behavioral, only measurable with a model)
- wshobson-accessibility: ~30% free / ~25% model-gated / ~45% R3 — C (worst case:
its semantics ARE the real browser; the free tier still catches
the inherits-all agent footgun)
The B+/B/C spread is the finding: where free/deterministic testing stops and the
model/container tiers take over. 1016 tests pass; R3/model-gated items written-up,
not run, by design. Linked from eval-coverage-and-isolation.md (Dogfood results).
* docs: record 2026-06-17 multi-SDK probe in sdk-harness-testing
Validate and sharpen the prior "don't retarget pillar 2" verdict with a
nine-SDK primary-source probe across six dimensions (mock-model /
tool-contract enforcement / assembled-as-shipped / trigger-rate /
sub-affordability / record-replay). The prior verdict holds for code-SDKs
(Pydantic/Vercel/LangGraph/LlamaIndex/Mastra ship first-party mocks), but
the probe relocates pillar-2 value to the Claude Agent SDK + Codex (no mock,
buggy/unenforced #172 tool contract, sub-affordability) and adds a concrete
mock-ergonomics borrow-list. Roadmap entry tightened to match.
* docs: make the two-axis coverage boundary explicit (public + internal)
The R1/R2/R3 rungs only answer "what does the test need from the outside
world" — they don't answer "who decides pass/fail", which is what makes a
prose skill look like a coverage hole when it isn't. Add the second
(oracle) axis everywhere the rungs appear:
- Define the `-MG` (model-gated) suffix the dogfood scorecards already use,
and place a prose/guidance skill explicitly at R1-MG (nothing executes,
but only a model judges its worth — fully testable on the sub, not free,
not uncovered).
- Add the three-bucket framing (A free & deterministic / B model-gated on
your sub / C needs-a-container, composed) and grade with two numbers
("% testable at all (free+sub)" vs "% needs-a-container").
- State that skill BEHAVIOR is tested (runEval A/B + judged), not just the
description firing (measureTriggerRate).
- Record that the sub-affordability story is ToS-clean (drive your own CLI
/ your own harness / your own sub; the Agent SDK ToS restricts
productizing claude.ai login, not running your own tests).
- Make the shipped test-harness skill coverage-aware (a Step 0.5 that
classifies each surface into the three buckets and sets honest per-bucket
expectations).
- Roadmap: add the 2026-06-17 multi-SDK mock-ergonomics borrow-list, linked
to sdk-harness-testing.md, marked recent/this-PR.
Covers CLAUDE.md(.spec.ts), README, docs/eval-architecture.md,
research/eval-coverage-and-isolation.md, research/roadmap.md,
skills/test-harness/SKILL.md.
* docs: flag open finding — behaviour testing is broader than A/B
Persist a conclusion raised this PR that lived only in chat: A/B
(runEval arms + assertSignificant) is the right oracle for relative
/regression/noise-floor questions, but testing an EXACT skill wants an
ABSOLUTE judged assertion, not a comparison to an OFF arm. The single-arm
absolute path already exists (measure + judged + assertRates,
skill-quality.eval.mjs); the open question is whether it is first-class and
discoverable or whether A/B is over-privileged in the API + README + the
test-harness skill. Recorded as testing-api-design Part 7 #7 + a roadmap
item for the next review session.
* docs(positioning): sharpen the pillar-2 wedge against SDK reality
The mock-model tier is table stakes (Pydantic/Vercel/LangGraph/LlamaIndex
all ship a first-party fake), so it is not the differentiator. Name the
three things no SDK mock does — testing the harness as it SHIPS, tool-contract
enforcement of the assembled agent (anchored on the live Claude Agent SDK
bug #172), and a sub-affordable real-model tier. Also reframe behaviour
testing as TWO oracles: the absolute single-arm measure()+judged() (the
default when there's no on/off baseline) alongside the relative runEval A/B.
* docs(testing): make the absolute behaviour oracle first-class for single skills
A/B (runEval + assertSignificant) was over-privileged across the README,
docs/harness-testing.md, and the test-harness skill, forcing 'test this one
skill behaves right' into a relative on/off mould. Lead instead with the
absolute oracle — measure({ checks: [judged(rubric)] }) + assertRates — which
needs no baseline and is what promptfoo/DeepEval lead with; reframe A/B as the
specialised relative/regression oracle. Resolves testing-api-design Part 7 #7
(docs/skill only, no code change).
* docs(testing): declare one canonical import per pillar
The docs taught granular concern-named paths (vigiles/run-hook, vigiles/eval,
vigiles/check) while the test-harness skill and README use the vigiles/testing
barrel — an inconsistent canonical-import story. Declare vigiles/testing and
vigiles/linting the canonical entry points (granular paths stay as stable
aliases) so new code has one obvious path.
* docs(safety): correct overclaimed default + macOS confinement status
The ephemeral run environment is shipped but OPT-IN (ephemeralEnv: true,
default off) — docs/safety.md read as if it were on by default for every run.
State the real status: default-off today, default-on is the committed next
step gated on validating the file-based OAuth copy path; the pure pieces and
scrubbed-env wiring are unit-tested. Also mark Seatbelt as macOS-planned (not
shipped) in the two mechanism tables, so 'Mac deny-all' no longer reads as a
working network wall.
* docs(roadmap): note per-check thresholds for assertRates
The absolute-oracle path (measure + assertRates) is now recommended for testing
one skill, but assertRates({ min }) applies one rate floor to all checks. Persist
the additive, non-breaking follow-up: let min accept a per-check mapping.
* docs: add subagent-model-note rule; promote macOS Seatbelt to Now/P1
- New guidance rule: when launching a subagent, state which model was chosen
and why, in one line, at launch — visibility into the cost/capability tradeoff.
- Promote the cross-platform (Seatbelt) confinement backend from Next to Now/P1
per maintainer request; note there's no Mac to validate it on, so it needs a
macOS CI runner or a Mac-having contributor.
* refactor!: make the agnostic surface truly harness-agnostic, enforced by eslint
vigiles/testing and vigiles/integration re-exported Claude-Code transport
(scriptModel, claudeCodeDriver, buildClaudeArgs, parseClaudeRun, claudeAvailable,
loadPlugin, resolveHarness) via blanket `export *`s — so the 'agnostic' barrels
weren't actually agnostic. The directory-based boundary lint missed it because
src/mock-model.ts (the Anthropic-Messages mock) lives at the composition root,
not under src/adapters/.
- testing.ts / integration.ts: re-export ONLY the agnostic harness-test names;
CC transport now comes from vigiles/claude-code.
- eslint: classify src/mock-model.ts as part of the cc-harness element, so the
existing agnostic-surface boundary rule forbids any agnostic barrel from
re-exporting it (verified: reintroducing the leak now errors).
- claude-code.ts: expose claudeCodeDriver + argv/parse helpers + claudeAvailable
(beside the Codex driver in vigiles/codex), preserving access off the agnostic
surface.
BREAKING: scriptModel/loadPlugin/claudeCodeDriver no longer importable from
vigiles/testing or vigiles/integration; import them from vigiles/claude-code.
* refactor!: trim package.json exports to a canonical public surface
Dropped 12 internal/redundant module subpaths that were each fully re-exported
by a concern-named barrel: ./compile ./linters ./eval ./harness-test
./harness-assert ./run-hook ./check ./judge ./mock-model ./plugin-loader ./mcp
./tool-stub. The package's `exports` map IS the access wall (Node throws
ERR_PACKAGE_PATH_NOT_EXPORTED; TS honors it under node16/bundler), so these are
now genuinely internal.
Canonical surface (12): . and ./spec (authoring + the module-augmentation
target), ./linting, ./testing, ./unit ./integration ./e2e (tiers),
./claude-code ./codex ./adapter (harness), ./vitest ./jest (runners).
./linting now also re-exports the linter engine (was ./linters).
BREAKING: the dropped subpaths no longer resolve — import via the canonical
barrel (vigiles/testing, vigiles/linting, vigiles/claude-code).
* docs: update Key Files descriptions for the trimmed public surface
vigiles/plugin-loader and vigiles/check are no longer standalone subpaths;
loadPlugin is on vigiles/claude-code and the check vocabulary on vigiles/testing.
* docs: sweep imports onto the canonical public surface
The trimmed exports removed the granular module subpaths, so rewrite every
doc/skill import to the canonical barrel: vigiles/{eval,run-hook,harness-test,
harness-assert,check,judge,tool-stub} → vigiles/testing, vigiles/{compile,linters}
→ vigiles/linting, vigiles/{plugin-loader,mock-model} → vigiles/claude-code.
CC-specific transport (scriptModel, loadPlugin) is split out of mixed imports
into a vigiles/claude-code line. Also corrected the two now-false 'granular paths
keep working' claims (only vigiles/spec remains besides the barrels).
* build: add API Extractor surface gate + API Documenter reference
Lock the public API surface so it can't silently regrow (the dual of the eslint
agnostic-surface rule, at the .d.ts level). scripts/api-extractor.mjs runs API
Extractor over all 11 public entry points and writes a committed report per entry
(etc/*.api.md) — the surface contract. CI (`node scripts/api-extractor.mjs` in
the check job) fails on any drift; `npm run api:report` regenerates after an
intentional change. The reports prove the de-leak: vigiles-testing.api.md has zero
CC symbols, vigiles-claude-code.api.md has them.
`npm run docs:api` runs API Documenter over the doc models to generate a
cross-linked Markdown API reference (api-reference/, gitignored — a derived
artifact, not committed). Narrative guides stay hand-authored.
Adds @microsoft/api-extractor + @microsoft/api-documenter as devDependencies.
* docs(roadmap): track moving the CC mock + driver into the adapter dir
The glob-classification of src/mock-model.ts as cc-harness enforces the agnostic
boundary today; the structural end-state is to relocate it (and claudeCodeDriver)
under src/adapters/claude-code/ so the boundary is directory-based, mirroring the
Codex adapter.
* ci: smoke-check the API reference generator (no publish)
Run API Documenter in the check job, reusing the doc models the surface gate
already wrote to temp/, so npm run docs:api can't silently break. Output
(api-reference/) is gitignored and not published — there's no docs site yet;
upgrade this to a Pages deploy when one lands.
* test(safety): prove ephemeralEnv actually scrubs — real-subprocess test
The one line that enforces ephemeral scrubbing (env: replaceEnv ? scrubbed :
merge) lived inside the v8-ignored real-spawn path, so a regression to an
always-merge would silently defeat ephemerality with no test catching it.
Extract it to a tested seam resolveSpawnEnv() and pin it two ways: a pure test
of the replace-vs-overlay decision, and a behavioural test that spawns a REAL
node child with the scrubbed env and asserts a planted secret is gone, HOME is
the throwaway dir, and the real ~/.ssh is unreachable. Deterministic, no model.
Updates the vigiles-testing API report for the new resolveSpawnEnv export.
* feat(testing): per-check rate thresholds in assertRates
The absolute-oracle path (measure({ checks }) + assertRates) is the recommended
way to test one skill, but assertRates({ min }) applied a single floor to every
check — you couldn't gate 'skill fires every trial AND judged >= 0.8' in one
call. Add an optional per (by check kind) override: assertRates(report, { min,
per: { skill: 1.0 } }); the failure message now reports each check's own min.
Additive + non-breaking. Updates the vigiles-testing API report.
* feat(testing): thread per-check thresholds through checkReportToJUnit
assertRates gained a per-check-kind threshold override but checkReportToJUnit
still took a single min, so the strict gate and the JUnit report could disagree
about which checks failed. Extract one shared checkRateThreshold() helper used by
both; checkReportToJUnit now accepts the same { min, per }. Updates the report.
* test(safety): prove interceptTools actually PREVENTS, end-to-end
The pure decision (decideIntercept/interceptHookDecision) was tested, but the
rail's whole point — the real `vigiles intercept-tool-hook` CLI, wired as a
PreToolUse hook, BLOCKING a matched call before it runs — was never exercised.
A regression in the subcommand's env-read or deny format would pass every pure
test. Drive the real built CLI through runHook (no model) and assert a matched
`git push` is blocked while a non-matching command and an un-intercepted tool
run. Deterministic; skips loudly if dist/ isn't built.
* fix(testing): guard two silent-green footguns in measure/assertRates
Both are the 'passes green while testing nothing' anti-pattern the no-silent-skips
rule forbids:
- assertRates on an empty report (measure with no checks) vacuously passed — now
throws, pointing at the likely empty checks list.
- stubSkillBodies + a judged check: stubbing replaces the skill body with a no-op,
so judged graded empty output (docs warned, nothing enforced) — now throws,
telling you to drop one. Adds tests for both.
* feat(cli): per-line GitHub annotations for broken doc refs
The dogfooded GitHub Action surfaces lint findings inline on the PR diff via
::error file=,line=:: workflow annotations. The inline/frontmatter/spec checks
already annotated per-line, but the markdown code-block ref check reported to
stdout only — so a broken enforce()/file()/cmd() in a doc never showed up
inline on the PR, only in the summary blob. Emit a per-line annotation for each
docRefReport error (it carries file+line). CI-only; skipped under --json/--summary.
Adds a hermetic test.
* ci: run the dogfooded vigiles Action as the primary lint gate
Move the composite Action (uses: ./) ahead of the redundant direct `node
dist/cli.js lint` in the check job, so a lint failure surfaces THROUGH the
shipped Action — its per-line annotations (inline on the PR diff) and sticky
PR comment — rather than a bare CLI step failing first and skipping it. The
direct CLI call stays as a redundant safety net. Aligns with prod-grade-gha-cli
(the Action is how CI runs for users).
* test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation
NOT FOR MERGE — a broken file() ref in docs/cli.md so the dogfooded Action
fails and leaves a per-line ::error annotation inline on the PR diff plus a
sticky comment. Revert this commit once the GHA behavior is confirmed.
* Revert "test(DEMO): deliberate broken doc ref to exercise the GHA per-line annotation"
This reverts commit
|