mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
7859fb1a6f
* feat: vigiles test/eval CLI + canonical harness examples + two-pillar positioning
Make the harness-testing pillar usable and concrete:
- CLI: `vigiles test` (discovers *.harness.mjs) and `vigiles eval`
(discovers *.eval.mjs) run the two-tier harness scripts as child node
processes and aggregate exit codes, so they work as a CI command, not
just `node x.mjs`. `vigiles test` skips clean without the claude CLI;
`--trials=N` is forwarded to eval scripts via VIGILES_TRIALS.
New module src/run-scripts.ts (discover/run/format) + test suite.
- Canonical examples under examples/harness/, real-plugin-shaped:
policy-gate.harness.mjs (deterministic PreToolUse Bash block-no-verify
gate + SessionStart setup hook) and skill-outcome.eval.mjs (does a
skill change the agent's output?). refs-hook.eval.mjs reads
VIGILES_TRIALS so it runs under the CLI too.
- Positioning: README + CLAUDE.md(.spec.ts) reframed around "test &
verify your Claude Code harness" — verification and evals as two
pillars; testing section links the canonical examples and CLI commands.
* chore: dogfood the harness-testing pillar on the in-repo build
- npm scripts test:harness / test:eval run the new CLI via the in-repo
build (`node dist/cli.js`), never the published `npx vigiles`, so the
repo dogfoods its own current code.
- CI `harness` job installs the claude CLI and runs the deterministic
tier against the scripted mock model — no API key, zero cost. The
real-model eval tier stays manual (`npm run test:eval`).
- Document both scripts in CLAUDE.md (compile-verified against
package.json); regenerate .vigiles/generated.d.ts.
Verified: `npm run test:harness` builds and runs examples/harness/
policy-gate.harness.mjs green (PreToolUse Bash gate + SessionStart hook).
* feat: cohesion testing (plugin loader), runner-agnostic helpers, eval variance, LLM-judge
Make the harness-testing pillar test the *whole machine*, not one hook, and
fit whatever runner you already have.
- Plugin/repo loader (src/plugin-loader.ts): runHarnessTest/runEval accept a
`plugin` path that loads the real harness — hooks from
.claude-plugin/plugin.json (or .claude/settings.json) with
${CLAUDE_PLUGIN_ROOT} resolved to the real scripts, plus CLAUDE.md and
skills — so you test what ships. Inline settings/files layer on top
(per-event hook arrays concatenated). resolveHarness shared by both tiers.
- Runner-agnostic helpers (src/harness-assert.ts): withHarness (auto-cleanup),
throwing assert* helpers usable in any runner incl. node:test, and
vigilesMatchers for vitest/jest expect.extend (one object, both runners).
The library depends on no test runner.
- Eval variance (src/eval.ts): aggregateStats returns mean/std/se/n; reports
render `metric=mean±se` so an A/B gap can be read against the noise floor.
- LLM-as-judge (src/judge.ts): thin synchronous judge() for subjective
outcomes inside measure(); deliberately not a platform.
- Canonical cohesion example: examples/harness/plugin-cohesion.harness.mjs +
fixture-plugin — one plugin load, SessionStart + PreToolUse Bash gate both
fire. docs/harness-testing.md ties the three layers together.
Verified: full suite 514 pass; plugin loader dogfooded on vigiles's own
plugin.json (unit) and end-to-end (the cohesion example passes via `vigiles
test`); audit/lint/fmt clean. New package exports: ./plugin-loader,
./harness-assert, ./judge.
* test: enforce multi-source + multi-runner support (plugin/repo, vitest/jest)
Lock in the testing API's reach with constraints, not just docs:
- Source support: plugin-loader tests now cover a plain repo
(.claude/settings.json, no plugin manifest), manifest-wins precedence
when both exist, and a bare dir (CLAUDE.md only → empty settings) —
alongside the existing plugin-manifest + in-repo dogfood cases.
- Runner support: the same vigilesMatchers object is exercised under BOTH
vitest (test/runners/matchers.vitest.mjs) and jest
(test/runners/matchers.jest.cjs), proving the library is runner-agnostic.
The dist is CommonJS, so jest requires it natively (no ESM flags / babel;
transform: {} in jest.config.cjs). Each runner is scoped to test/runners
so it never picks up the src/*.test.ts node:test suites. New scripts
test:vitest / test:jest; both run in CI. vitest + jest added as dev deps.
Verified: node:test 516 pass, vitest 2 pass, jest 2 pass; lint/audit/fmt clean.
* feat: opt-in vigiles/vitest + vigiles/jest integration entries (subpath exports)
Per the subpath-exports decision, ship per-runner integration entries that
register the matchers AND add their TypeScript types — without coupling the
core to any runner.
- vigiles/vitest (src/vitest.mts → ESM, since vitest is ESM-only): registers
vigilesMatchers + augments @vitest/expect's Matchers so toHaveCreated /
toBeatBaseline type-check.
- vigiles/jest (src/jest.ts → CJS, required natively): same, augmenting
@jest/expect's Matchers.
- vitest + jest declared as OPTIONAL peer dependencies — only the entry you
import pulls one in.
- Type-level constraint: test/types/{smoke.vitest,smoke.jest}.ts compile under
`tsc --noEmit` (npm run test:types), with @ts-expect-error guards proving the
matchers are real types, not any. Wired into CI alongside the vitest/jest
runner tests.
- tsconfig include now matches .mts; package files ship .mjs/.d.mts.
Verified: node:test 516 pass, vitest 2, jest 2, type smoke compiles, lint/fmt/
audit clean.
* feat: plugin loader handles real-world hook layouts (dogfooded on obra/superpowers)
Dogfooding the loader on a real popular plugin (obra/superpowers, 5.1.0)
surfaced that it keeps hooks in hooks/hooks.json — the Claude Code plugin
convention — which our loader didn't read (it only handled inline plugin.json
hooks). Now loadPlugin resolves, in order:
1. inline `hooks` object in .claude-plugin/plugin.json,
2. a `hooks` string path in plugin.json,
3. the hooks/hooks.json convention (superpowers, etc.),
4. a plain repo's .claude/settings.json.
Verified live: loadPlugin("/tmp/superpowers") surfaces its SessionStart hook
with ${CLAUDE_PLUGIN_ROOT} resolved and materializes its 14 skills. Added
fixture-based tests for the convention + string-path layouts (no network in CI).
Also fixes a latent require-spec=error failure (since the cohesion commit): the
example fixture-plugin/CLAUDE.md now carries <!-- vigiles-disable require-spec -->
like the other fixtures, since it represents a third-party plugin, not a
vigiles-managed instruction file. Backticked the glob tokens (*.harness.mjs etc.)
in the spec so the compiled CLAUDE.md is prettier-stable (hash holds).
Verified: node:test 518, vitest 2, jest 2, type smoke + lint + fmt + audit clean.
* test: close the harness-testing coverage gaps (judge, entries, assertServedTurns)
A coverage audit found three untested use cases; this fills them:
- judge.ts had zero tests. Extracted parseJudgeOutput() as a pure verdict
parser and tested it: claude --output-format json result-field unwrap,
bare/prose-wrapped JSON, custom threshold, [0,1] clamping, and safe
fallback on empty/unparseable/score-less output. (The model spawn stays
integration-only.)
- The vitest/jest runner tests registered matchers by hand, so the opt-in
entries' auto-register was never exercised. They now load `vigiles/vitest`
via setupFiles and `vigiles/jest` via setupFilesAfterEnv (the way users
do), so the entries themselves are under test.
- assertServedTurns was untested — added.
Verified: node:test 524 pass / 1 skip, vitest 2, jest 2, type smoke + lint +
fmt + audit clean.
Remaining intrinsically-integration cases (need claude/model, covered by the
examples in the `harness` CI job, not unit): runHarnessTest incl. plugin:,
runEval incl. plugin arm, withHarness, and judge()'s model call.
* docs: testing-matrix doc + exercise withHarness in the cohesion example
- examples/harness/plugin-cohesion.harness.mjs now uses withHarness (+ the
assertCreated helper), so the auto-cleanup wrapper is actually exercised in
the `harness` CI job — closing the last coverage gap. Comments state the
trade-off plainly: withHarness guarantees sandbox cleanup even on a thrown
assertion (no leaked temp dirs) but assertions run in a callback and the
first failure aborts the rest; use raw runHarnessTest + r.cleanup() when you
want per-check reporting (see policy-gate).
- New docs/testing-matrix.md: every use case → its test tier (unit /
cross-runner / type / integration-CI) and the file that proves it, what's
intentionally not unit-tested and why, and why the CLI examples are .mjs
while the API + runner usage are TypeScript. Linked from the guide + CLAUDE.md.
Verified: cohesion example passes via `vigiles test`; fmt/audit/lint clean.
* chore: sync package-lock with optional jest/vitest peerDeps
The lockfile did not record the optional jest/vitest peerDependencies
declared in package.json; npm install reconciled it.
* feat: add hook unit tier (runHook) and make plugin loader surface-aware
Two gaps surfaced by dogfooding the testing API against obra/superpowers
and the wshobson/agents marketplace:
1. No unit tier. Both existing tiers spawn the real claude CLI, so even
'does guard.sh block --no-verify' needed the binary, and Edit/Write +
PreCompact/Notification/SessionEnd/SubagentStop events were unreachable
(the mock can't trigger them). runHook pipes a synthesized event JSON
straight to the hook process and reports the normalized block/allow
decision — no claude, no model, milliseconds, every event. Pure
parseHookOutput/decideHook are unit-tested; assertHookBlocked/
assertHookAllowed + a toBlock matcher added.
2. loadPlugin silently dropped agents/ and commands/ — so a subagents-only
plugin like tdd-workflows (2 agents + 4 commands, no hooks) loaded an
empty machine and any test 'passed' having exercised nothing. The loader
now materializes agents/ and commands/ too and exposes LoadedPlugin.
warnings flagging surfaces only a real model can drive (subagents,
slash commands, MCP) or an outright empty load.
Docs: harness-testing.md gains the unit tier + surface-coverage note;
testing-matrix.md gains a surface x tier matrix. Canonical example
hook-unit.harness.mjs runs in CI with no claude. CLAUDE.md recompiled.
* docs: wire the unit tier through README/guides + fix jest config key
- Fix a real bug: the jest setup key is setupFilesAfterEnv, not
setupFilesAfterEach (docs/harness-testing.md + src/jest.ts docstring).
- Reconcile tier/layer counts now that runHook is a third testing tier:
README 'two levels' -> 'three levels'; testing-matrix 'three layers' ->
'four layers'; harness-testing see-also drops the stale 'two-tier' label.
- Add the unit tier (runHook) to the README with a snippet, and note
loadPlugin().warnings + subagent/command materialization in the
'test the whole machine' section.
* docs: bring harness-testing research doc + example spec up to date
- research/harness-testing.md: rewrite from 'two tiers' to the shipped
three-tier design (unit runHook / deterministic runHarnessTest / eval
runEval), add a 'plugin loader' section covering the assembled-machine
loader + surface warnings, and fold in the wshobson/agents dogfood
finding (subagents/commands dominate; tdd-workflows empty-machine trap).
- examples/CLAUDE.md(.spec.ts): fix stale API in the teaching example —
there is no check() builder (the real rule kinds are enforce/guard/
guidance), and the CLI is init/compile/audit/test/eval/generate-types,
not check/discover/adopt. Recompiled.
- CLAUDE.md.spec.ts: refresh the research-doc key-file description to the
three-tier framing. Recompiled.
* docs: reconcile rule-type count to three (enforce/guard/guidance)
guard() is a real, exported rule builder with three live uses in the root
spec, and spec-format.md already documents it as one of 'three builders' —
but the root CLAUDE.md architecture said 'Two rule types' and two sample
lines in spec-format lagged. Align them all on enforce()/guard()/guidance()
so the canonical doc, the format reference, and the example agree. Root
CLAUDE.md recompiled.
---------
Co-authored-by: Claude <noreply@anthropic.com>