zernie
9413061d9b
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>
22 KiB
Testing matrix — what's covered, and how
Every use case of the harness-testing API, mapped to the tier that tests it and the file that proves it. Two kinds of coverage:
- Unit — deterministic, model-free, runs in
npm test/npm run test:vitest/npm run test:jest/npm run test:types. These are the contract. - Integration (CI) — needs the real
claudeCLI and/or a model, so it can't be a unit test. Covered by the canonical examples run in theharnessCI job (deterministic tier, no API key) or bybench/+examplesevals (real model).
Coverage
| Use case | Tier | Where |
|---|---|---|
Hook unit tier — runHook (exit codes / stdin event / env / JSON decision) |
unit | src/run-hook.test.ts |
Hook decision logic (parseHookOutput / decideHook) |
unit | src/run-hook.test.ts |
Subagent PreToolUse tool-contract rail — hook-runtime agent (parse contract / allow-deny / hook ⇄ allowlist agree / real built-CLI hook via runHook; grounded on the vendored ui-visual-validator) |
unit | src/adapters/claude-code/agent-runtime.test.ts |
Plugin loader — inline plugin.json hooks |
unit | src/adapters/claude-code/plugin-loader.test.ts |
Plugin loader — hooks string-path |
unit | src/adapters/claude-code/plugin-loader.test.ts |
Plugin loader — hooks/hooks.json convention |
unit | src/adapters/claude-code/plugin-loader.test.ts |
Plugin loader — repo .claude/settings.json |
unit | src/adapters/claude-code/plugin-loader.test.ts |
| Plugin loader — manifest-wins precedence | unit | src/adapters/claude-code/plugin-loader.test.ts |
| Plugin loader — bare dir (no hooks) | unit | src/adapters/claude-code/plugin-loader.test.ts |
Plugin loader — agents/ + commands/ materialized + surface warnings |
unit | src/adapters/claude-code/plugin-loader.test.ts |
| Plugin loader — empty-machine + MCP + dangling-intra-plugin-ref warnings | unit | src/adapters/claude-code/plugin-loader.test.ts |
| Plugin loader — in-repo dogfood | unit | src/adapters/claude-code/plugin-loader.test.ts |
Vendored real-plugin conformance (loadPlugin invariants, pinned + offline) |
unit | src/adapters/claude-code/vendor.test.ts |
Dogfood conformance over vigiles's own skills (every SKILL.md loads + has name + non-empty description; hook scripts resolve at the plugin root) |
unit | src/adapters/claude-code/skills-dogfood.test.ts |
Untested-surface detector (untested-skill/untested-subagent/untested-hook rules: colocation + content-reference coverage, vigiles:ignore-test opt-out, hook-script discovery) |
unit | src/test-coverage.test.ts |
resolveHarness — merge / passthrough / undefined |
unit | src/adapters/claude-code/plugin-loader.test.ts |
| Eval aggregation (mean / std / se / n / pass^k) + report formatting | unit | src/eval.test.ts |
Eval usage capture + aggregation (parseUsage / aggregateUsage, cost/latency/tokens) |
unit | src/eval.test.ts |
Eval record/replay cache (cacheKey / snapshot+restore / replay skips the model) |
unit | src/eval-cache.test.ts, src/eval.test.ts |
Eval concurrency + rate-limit retry + maxCostUsd abort (runPool / isRateLimited) |
unit | src/eval.test.ts |
Trigger-rate orchestration (measureTriggerRateWith) |
unit | src/eval.test.ts |
Significance stats (welchTTest / compareArms / tPValueTwoSided / incomplete-beta vs t-table) |
unit | src/stats.test.ts |
| Assert/predicate helpers (create/turns/hook-block; tool used/notUsed/count/sequence/with; skillResolved; output/request-contains; hookFired; improvement/improves; reliable; significant; triggerRate) | unit | src/harness-assert.test.ts |
Matchers register + pass under vitest (via the vigiles/vitest entry) |
unit | test/runners/matchers.vitest.mjs |
Matchers register + pass under jest (via the vigiles/jest entry) |
unit | test/runners/matchers.jest.cjs |
Matcher types augment vitest/jest expect |
unit | test/types/smoke.vitest.ts, smoke.jest.ts |
CLI runner (discoverScripts/runScripts/summary) |
unit | src/adapters/claude-code/run-scripts.test.ts |
Judge verdict parsing (parseJudgeOutput) |
unit | src/judge.test.ts |
runHarnessTest end-to-end, incl. plugin: |
integration (CI) | examples/harness/policy-gate.harness.mjs, plugin-cohesion.harness.mjs |
withHarness (auto-cleanup wrapper) |
integration (CI) | examples/harness/plugin-cohesion.harness.mjs |
runEval end-to-end, incl. plugin arm |
integration (real model) | bench/evals/refs-hook.eval.mjs, examples/harness/skill-outcome.eval.mjs |
measureTriggerRate end-to-end (skill activation via pluginDir) |
integration (real model) | examples/harness/skill-trigger-rate.eval.mjs |
Dogfood trigger-rate on vigiles's own model-invocable skills (recall + precision via irrelevantPrompts) |
integration (real model) | examples/harness/dogfood/{test-harness,generate-logo}.trigger.eval.mjs |
judge() model call |
integration (real model) | (parsing is unit-tested; the spawn is not) |
Surface coverage — which plugin surface is reachable at which tier
A Claude Code plugin/repo has several surfaces. They are reachable at different tiers, because some only do anything under a real model:
| Surface | Hook unit (runHook) |
Deterministic (runHarnessTest) |
Eval (runEval) |
|---|---|---|---|
| Hooks — SessionStart / Stop / UserPromptSubmit | ✅ logic | ✅ fires in machine | ✅ |
| Hooks — Bash PreToolUse / PostToolUse | ✅ logic | ✅ fires in machine | ✅ |
| Hooks — Edit/Write PreToolUse / PostToolUse | ✅ logic | ⚠ headless-gated (drive via Bash) | ✅ |
| Hooks — PreCompact / Notification / SessionEnd / SubagentStop | ✅ logic | — (mock can't trigger) | partial |
| CLAUDE.md / instruction files | — | ✅ present in context | ✅ moves behaviour |
| Skills | — | ✅ resolves via pluginDir |
✅ activation (measureTriggerRate) |
Subagents (agents/) |
✅ tool-contract rail | ⚠ materialized; rail not live-armed | ✅ (Task) |
Slash commands (commands/) |
— | ⚠ materialized, not invoked | ✅ |
| MCP servers | — | — (not wired; warned) | bring-your-own |
loadPlugin(...).warnings surfaces the ⚠ rows for a given plugin, so a "load the
whole plugin" test never silently runs an empty machine (e.g. a subagents-only
plugin like wshobson/agents tdd-workflows, which ships 2 agents + 4 commands
and no hooks).
Below the three runtime tiers sits a static, model-free floor the table
doesn't have a column for: skills load and resolve with a usable description
(src/adapters/claude-code/skills-dogfood.test.ts), and vigiles lint's
per-kind untested-skill / untested-subagent / untested-hook rules flag any skill/hook/subagent
that ships with no test or eval at all — the "is there even a test?" check that
precedes "does the test pass?".
The subagent tool-contract rail (hook-runtime agent) makes the agents/ Hook-unit
cell ✅: the generated PreToolUse hook that enforces an agent's declared
tools: is runHook-testable like any other hook, and the declared list and the
enforced rail are compiled from one source (proven by a round-trip test). The
deterministic cell stays ⚠ for an honest reason — Claude Code doesn't surface the
active subagent to hooks, so arming the rail in a live session
(.vigiles/active-agent.json) is still unsolved; the logic is proven at the unit
tier, not yet end-to-end in a real dispatch.
What is intentionally not unit-tested
Only the real-claude subprocess is out of the gate: runHarnessTest,
withHarness, judge()'s model call, and runEval/measureTriggerRate's
default spawnAgent runner spawn the CLI (and, for evals, a real model). A unit
test can't drive that deterministically — but everything around it is pinned.
The eval orchestration (runEvalWith / measureTriggerRateWith) takes an
injected runner, so the loop, cache, concurrency, budget, usage aggregation,
and significance run against canned stream-json with no model; the mock model
(src/mock-model.ts), the loader, the parsing, and the matchers are unit-tested
too. End-to-end behaviour is exercised by the example suite in CI: the
deterministic examples (*.harness.mjs) run with no API key (real claude +
scripted mock model), so they're CI-affordable; the evals (*.eval.mjs) cost
real model calls and run manually / in a keyed job.
JavaScript or TypeScript? Both.
vigiles test / vigiles eval accept harness/eval scripts in either
language — the glob is *.harness.* / *.eval.* over .mjs .cjs .js
.mts .cts .ts (src/adapters/claude-code/run-scripts.ts). Discovery is
name-only and scoped to those infixes — never *.test.* / *.spec.* — so it
won't pick up your vitest/jest/Playwright files (and it ignores node_modules +
dist). A JavaScript file runs as a plain node <file>; a TypeScript file is run
through tsx when it's installed, else Node's native type stripping
(Node ≥ 22.6), with an actionable error if neither is available. So you can author
your own tests in TS and get types end-to-end, or in JS for zero setup.
A bare vigiles eval (no target) discovers the whole tree, and each eval runs the
real model on your subscription — so it asks before fanning out: name the
eval(s), pass --all, or answer the terminal prompt; headless with none of those,
it refuses (exit 2) rather than spend quota. vigiles test is free/deterministic
and always runs. And the deterministic API folds into an existing vitest/jest
suite (vigiles/vitest · vigiles/jest matchers) — the CLI is the zero-dep floor,
not a required second runner; eval stays its own surface (real-model, statistical).
The API itself is fully TypeScript — src/*.ts, shipped with .d.ts; the
whole unit suite (src/*.test.ts) and the type smokes (test/types/*.ts) are
TypeScript, so the project dogfoods the typed path. Under a runner (vitest /
jest) the vigiles/vitest / vigiles/jest entries add typed matchers.
The canonical examples under examples/harness/ stay .mjs on purpose —
not because TS is unsupported, but because they're the lowest-common-denominator
demos: they must run with nothing but Node (no tsx, no loader, no build) so a
copy-paste into any repo Just Works and the CLI-fallback tier stays
dependency-free. Your own scripts are under no such constraint — reach for .ts
whenever you want types.
- Zero-dep CLI tier →
.mjs(or.tsif you havetsx/Node ≥ 22.6), viavigiles test. - Runner tier →
.ts, full types viavigiles/vitest/vigiles/jest.
If you want a typed worked example, the type smokes in
test/types/ show the API used from .ts with the matcher
types applied.
Running the tiers in CI
CI runs every tier except evals — not one at a time, and not all crammed into one job. Each tier runs where it's cheapest, so a fast unit failure surfaces before the slow, privileged ones.
| Tier | Run it with | This repo's CI job | Needs |
|---|---|---|---|
| Unit (+ coverage gate) | npm test / npm run coverage |
test |
nothing (model-free) |
| Reference verification | vigiles lint (+ the Action via uses: ./) |
check |
nothing |
| Deterministic harness | vigiles test (*.harness.{mjs,ts}) |
harness |
the real claude CLI, no key |
| e2e (allowlisted egress) | npm run test:e2e |
e2e (privileged container) |
bubblewrap + slirp4netns |
| Eval (real model) | npm run test:eval (*.eval.{mjs,ts}) |
manual — not in CI | model auth ($) |
Best practice (what this maps to): keep the model-free tiers (unit /
reference / deterministic) on every push and PR so feedback is fast and free;
isolate the slow or privileged tiers (real-egress e2e) in their own jobs; and run
the paid eval tier on demand, not on every commit — its non-determinism and
cost make it a release/regression gate, not a per-push check. The per-tier
npm run test:unit | test:integration | test:e2e scripts exist for exactly this
split (see vitest.config.mjs).
Skips are loud, never a silent green. vigiles test classifies each script
pass / skip / fail. A unit-tier runHook test needs no claude and always runs;
a tier that can't run (deterministic with no claude, e2e with no bubblewrap)
reports a loud ⊘ SKIPPED, tallied separately as "N skipped" — it never counts
as a ✓ passed. A skip passes by default (capabilities differ per job — this
repo runs the egress tier under e2e, not harness), but in a CI job that
asserts the capability is present, add vigiles test --no-skip so a
skipped tier fails: a green-with-skips is itself untested surface. A standalone
script emits a skip with skip(reason) from vigiles.
And so is a file that verified nothing. A script that exits clean having
recorded zero checks reports ∅ … 0 CHECKS, tallied apart from passed — the
shape being a harness that defines tests (export default { … }) and never calls
them, which no exit code can distinguish from a pass. Non-fatal by design: it never
turns a build red, and a script that doesn't import vigiles cannot report,
so it stays a plain pass. recordCheck() reports assertions made another way.
See also
docs/harness-testing.md— the full guide (four layers: verify-refs / hook unit / deterministic / eval, runner-agnostic usage, plugin loader, variance, judge, CLI fallback).