mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
ceb810e881
* docs: add TOC and subheadings to README for readability
Add a collapsible Contents block organized around the two pillars
(reference verification + harness testing) so the testing section is
easy to jump to. Promote the testing tier lead-ins to ### subheadings
so each tier is scannable and linkable from the TOC.
* docs: trim README to pitch + quick start, extract reference into docs/
Restructure README around the two pillars back-to-back (verify
instruction files → test the harness), then a slim reference tail.
Cuts 511 → 389 lines toward the ~300 brevity target.
- Move CLI, GitHub Action, plugin, and validation rules into a new
docs/cli.md; README keeps a condensed "CLI & CI" section + link.
- Move the related-tools/compose list and output-targets into a new
docs/related-tools.md; README keeps a one-paragraph summary + link.
- Fold the standalone "Type-Safe Rule References" section into a
two-line note under Verified References (deep link to linter-support).
- Condense the guard() and symbol-reference subsections.
- Drop the duplicate init-flags table from Quick Start (now in cli.md).
- Update the Contents TOC to match the new structure.
* test(harness): dogfood the plugin loader against real pinned plugins
Adds runnable dogfood tests that point loadPlugin at vendored snapshots of
two real, MIT-licensed Claude Code plugins instead of synthetic look-alikes:
- real-superpowers.harness.mjs — obra/superpowers: validates the
hooks/hooks.json convention + ${CLAUDE_PLUGIN_ROOT} expansion.
- real-wshobson.harness.mjs — a wshobson/agents sub-plugin: the no-hooks
subagents+commands+skills shape, asserting the loader materializes the
surfaces and warns rather than silently testing an empty machine.
Reliability: the plugins are PINNED, VENDORED snapshots under
examples/harness/vendor/ (upstream LICENSE + a SOURCE recording repo +
commit), so the tests run offline and deterministically — no clone at test
time. tools/refresh-vendor.sh re-fetches them deliberately.
Safety: loadPlugin only parses; it never executes a hook. The dogfood
asserts superpowers' side-effectful SessionStart hook is correctly WIRED
and does not run it — executing untrusted third-party hooks needs a real
sandbox (the CI container, or an opt-in bwrap/Docker boundary), documented
in docs/harness-testing.md.
Both tests are key-free and claude-free (pure loadPlugin), so they run in
CI for free. Vendored dirs are excluded from prettier + eslint.
* chore: regenerate .vigiles/generated.d.ts
Refresh the generated type unions to match current project state (adds the
test:types npm script and recent project files). `generate-types --check`
now passes; no vendored third-party paths leak into the unions.
* docs: track sandboxed harness-test execution as a feature idea
Add feature-ideas.md #13 — an opt-in `sandbox:` option (bwrap/sandbox-exec/
docker) on runHook/runHarnessTest so the execution tiers can safely run
untrusted third-party plugin hooks, turning the dogfood from parse-only into
execute-and-verify. Cross-link it from the harness-testing sandbox-boundary note.
* docs: add harness-testing coverage matrix (shipped vs. should-build)
A roadmap/gap-analysis companion to docs/testing-matrix.md: maps the whole
potential surface of testing a Claude Code harness (hooks by event, skills,
subagents, slash commands, MCP, CLAUDE.md, settings) across the unit /
deterministic-integration / eval tiers, plus cross-cutting capabilities
(sandboxed exec, native plugin-install fidelity, MCP wiring, coverage tooling).
Marks each cell ✅ shipped / 🟡 partial / 🔴 gap, records the fidelity caveats
(skill activation is faked even at the eval tier; Edit/Write headless-gated;
no sandbox for untrusted exec), and gives a prioritized build list. Linked
from docs/testing-matrix.md and research/harness-testing.md (no orphan).
* docs: make coverage matrix use the unit/integration/e2e pyramid honestly
Decouple the columns from a single vigiles function each. The prose surfaces
(skills, agents, commands, CLAUDE.md) DO have a unit tier — static
reference/contract verification via `audit`/`refs` (vigiles' first pillar) —
which the old table hid by equating "Unit" with `runHook`. Rename columns to
Unit/static · Integration (assembled, mock model) · E2E (real model), and fill
the unit column for every surface. Sharpens the read: every surface has a
static check, but the integration/wiring middle is hooks+settings only.
* docs: frame eval as a separate axis (measurement) in the coverage matrix
Eval isn't a "test" in the unit/integration sense — it's measurement (A/B
mean ± se you read, not a pass/fail you assert), non-deterministic and costly.
Keep it as the third column for surface-coverage completeness (it's the only
handle for skills/subagents/commands), but flag the distinction and point its
machinery (trials/variance/judge) to the eval docs instead of the cells.
* docs: make eval's non-determinism explicit in the coverage matrix
* docs: record skill-activation spike — feasible via --plugin-dir
Spike against real claude 2.1.169: --plugin-dir natively registers a plugin's
skills and the model activates them via a real Skill tool call
(superpowers:test-driven-development). Closes the gating unknown for roadmap #1.
Also surfaces a bonus deterministic path (scriptModel a Skill tool_use to test
wiring) and a fidelity rule (vendor whole plugins). Updates caveat #1 + the
native-install row.
* docs: spike finds Edit/Write deterministic tier already works (caveat #2 stale)
Second spike probe on claude 2.1.169: Edit/Write tool-event hooks DO fire in the
mock-model deterministic tier — PostToolUse 3/3, a PreToolUse Edit block held.
The 'headless-gated' limitation is no longer reproducible (--allowedTools
allowlists the edit tools past the prompt). So roadmap #3 collapses from 'build a
driver' to 'add a regression test'. Updates the Edit/Write cell, caveat #2, the
spike section, and the build list.
* test(harness): lock in Edit/Write deterministic-tier regression tests
Replaces the stale `test.skip`ped "Edit/Write headless-gated" placeholder with
two real, reliable tests (claude 2.1.169):
- Write -> PostToolUse(Write|Edit) hook fires (file written + marker), 5/5.
- Read -> Edit, PreToolUse(Edit) blocks (marker proves the hook fired AND the
edit didn't apply), 4/4.
Found three false-pass/no-op traps while writing them, now documented in the
coverage matrix: Edit needs a prior Read; a "file unchanged" assertion is
trivial without a hook-fired marker; a custom `prompt` makes the mocked turn
no-op (default "go" is reliable).
The second planned lock-in test (deterministic skill-wiring) is coupled to the
--plugin-dir feature (#1): probing showed .claude/skills materialization does
NOT register a skill for the Skill tool, so it ships with #1. Updates the matrix
accordingly (Edit/Write row -> shipped; skill-wiring -> coupled to #1).
* feat(harness): native plugin install via pluginDir — skills activate + resolve
Adds `runHarnessTest({ pluginDir })` which passes `claude --plugin-dir`, so a
plugin's skills register and ACTIVATE the real way (file-materialization does
not register skills for the Skill tool — only native install does). Plus a
`transcript: true` option (stream-json) to assert on what the agent's tools
returned.
This ships the deterministic half of roadmap #1: a scripted `Skill` tool_use
now resolves a plugin's skill (its body is injected), so skill WIRING is
testable with no API key. Skills integration cell 🔴→✅.
- src/harness-test.ts: pluginDir + transcript options, threaded into the argv.
- examples/harness/fixture-skill-plugin/: a 3-file self-contained plugin (one
`greet` skill) for the deterministic test.
- src/harness-test.test.ts: a test asserting `demo:greet` resolves (3/3 reliable).
- docs/harness-testing.md + the coverage matrix: document pluginDir, mark what
shipped vs. remaining (runEval pluginDir for real activation; subagents/commands).
* feat(eval): native plugin install via pluginDir — closes the skill-activation caveat
Adds `pluginDir` to `runEval` arms (mirrors runHarnessTest): an arm can install a
plugin natively (`claude --plugin-dir`) so the REAL model activates its skills by
description, not via the old "tell the agent to read a SKILL.md" trick.
Verified end-to-end with a real-model A/B on the fixture plugin: arm `off`
(no plugin) writes NO_SKILL; arm `on` (pluginDir) activates demo:greet and writes
its marker. So caveat #1 (skill activation faked) is now closed at BOTH tiers —
deterministic resolve (runHarnessTest) and real activation (runEval).
- src/eval.ts: pluginDir on EvalArm, threaded into the agent argv.
- examples/harness/skill-outcome.eval.mjs: note now points at pluginDir as the
real-activation path.
- docs + coverage matrix: caveat #1 marked closed; Skills e2e cell -> ✅; also
corrected the stale "Edit/Write headless-gated" line in the guide.
* docs: probe confirms subagents + slash commands register via --plugin-dir
Probed end-to-end (real model): a /cmd slash command wrote its marker, and an
echoer subagent invoked via Task wrote its marker — both off a --plugin-dir
install. So all model-driven surfaces (skills, subagents, commands) activate
natively at the eval tier (their e2e cells ✅). The open work is DETERMINISTIC
testing: slash commands need capturing the mock-received (expanded) prompt;
subagents are hard (nested mock sessions). Matrix rows + build list updated.
* docs: refresh the 'honest read' after subagent/command probe
* test(mcp): MCP governance is testable at the unit tier (no server)
A PreToolUse hook that blocks a destructive github-MCP tool
(mcp__github__merge_pull_request) but allows read-only ones — the dominant real
MCP test. The hook only sees the tool name, so runHook covers it with no MCP
server and no model. Locks in MCP slice #1 from the coverage roadmap.
* feat(harness): assert on the agent's actions (r.toolCalls + tool/skill asserts)
Adds the invariant vocabulary the skill/MCP/command surfaces actually need —
asserting on what the agent DID, not a brittle stdout substring:
- parseToolCalls + HarnessTestResult.toolCalls: parse the stream-json transcript
(transcript:true) into the tools the agent invoked, each paired with its result.
- assertToolUsed / assertToolNotUsed (the safety negative — "the destructive tool
was never called", which "file unchanged" can't prove) / assertSkillResolved.
Grounded in REAL plugins for immediate payoff: the skill-wiring test now uses
assertSkillResolved (no injected marker), and two new tests assert that real
obra/superpowers and wshobson/agents skills resolve via --plugin-dir — something
the old marker-grep couldn't do (it only works on fixtures you control).
Pure unit tests for parseToolCalls + the three asserts run with no claude.
docs/matrix updated. Full suite green; lint + fmt clean.
* docs(readme): rewrite the testing section, plainer + more detail
Reframe the harness-testing pitch around 'three levels, cheapest first' in plain
language (test a hook alone → does it fire in a real session → does it change
behaviour), and surface the newly-shipped capabilities: MCP governance at the
unit tier, native skill testing via pluginDir, and asserting on the agent's
actions (assertSkillResolved / assertToolNotUsed — the safety negative) against
real superpowers/wshobson plugins. TOC + anchors updated.
* feat(harness): sequence & budget invariants over the agent's actions (idea 1)
Adds three assertions over r.toolCalls, beyond the single-bool "did it use X":
- assertToolSequence(r, [...]) — ordering, in-order subsequence (e.g. Read→Edit)
- assertToolCount(r, name, {min,max,exactly}) — budgets (≤1 Write, 0 of X)
- assertToolCalls(r, predicate) — escape hatch for custom invariants like
"every Edit was preceded by a Read" (the rule Claude enforces).
Pure unit tests (no claude) cover all three; a grounded integration test asserts
the workflow on a real Read→Edit run. Documented in docs/harness-testing.md.
Parks the two siblings on the roadmap (coverage matrix, "What we should build"):
#7 property-based hook fuzzing, #8 monotonic/relational eval invariants.
Lint + fmt clean; suites green.
* docs(roadmap): triage the next-candidates against existing tooling
After checking what already exists:
- Skill frontmatter lint + slash-command lint → DROPPED (commoditized:
skill-validator, agentskills.io Skill Spec Linter, plugin-validator already do
it; vigiles only adds deep ref resolution it already has). Recorded under
"Evaluated and dropped".
- MCP harness → sharpened into the standout item: a new mcp() reference kind
(verify mcp__server__tool against the live server's tool list) + an mcpConfig
harness option. The only candidate that's vigiles-shaped, not commoditized.
- Triggering eval (measureTriggerRate) → added to the build list (the #1 skill
activation pain, built on pluginDir).
- Sandbox exec stays parked (feature-ideas §13, out of scope, may reconsider).
* feat(mcp): verify MCP tool references against a live server (src/mcp.ts)
The vigiles-shaped half of MCP support: a minimal MCP stdio client that starts a
server, does the JSON-RPC handshake, and lists its tools — so a referenced
`mcp__server__tool` can be verified against the REAL server (like enforce()
resolves a linter rule), catching a skill/CLAUDE.md that cites a tool the server
renamed or removed (e.g. github MCP create_issue → issue_write). On a miss it
suggests the closest tool name via Levenshtein ("did you mean issue_write?").
- src/mcp.ts: listMcpTools + verifyMcpTool (+ ./mcp export).
- examples/harness/fixture-mcp-server.mjs: a real minimal stdio MCP server, so
tests are deterministic and offline.
- src/mcp.test.ts: 5 tests (handshake/list, resolve, typo→suggestion, renamed
ref, bad command) — wired into npm test. Also probed live against
@modelcontextprotocol/server-everything (13 real tools; "ecko"→["echo"]).
Matrix: MCP unit/static cell → tool refs verifiable; build item 5 part (a) done,
remaining = spec mcp() builder + audit wiring + mcpConfig harness option.
* docs: fix stale 'MCP unwired at every tier' after shipping mcp.ts ref verification
* feat(mcp): wire MCP tool-reference verification into `vigiles audit`
Adds the `vigiles:mcp server#tool` mark (the MCP analogue of vigiles:symbol) and
verifies it in `audit`: parse marks from instruction files' inline spans, read
the server config from .mcp.json, start the referenced server, and flag a
missing tool (with a Levenshtein "did you mean") or an undeclared server —
exit 2, like a broken symbol ref. Only runs when a mark references a server, so
zero overhead otherwise.
- src/mcp.ts: parseMcpRefs / loadMcpServers / verifyMcpRefs / mcpRefMessage.
- src/mcp.test.ts: parse (skips fenced blocks) + verify (ok / typo→suggest /
undeclared) against the real fixture server. 7/7.
- src/cli.ts: audit step 10 + mcpRefErrors in the report / exit code / summary.
Verified end-to-end: a CLAUDE.md citing `vigiles:mcp demo#ekho` →
'not found — did you mean "echo", "add"?'; cli suite 53/53, lint+fmt clean.
* docs: MCP unit/static cell → verified in audit (vigiles:mcp mark)
* docs(readme): surface the mark family (symbol + mcp) and fold in the matrix
- Verified References now lists the inline mark family: `vigiles:symbol file#name`
and the new `vigiles:mcp server#tool` (audit starts the server and flags a
renamed/missing tool with a "did you mean").
- Add the coverage matrix (surface × tier) to the testing section inside a
<details>, so the at-a-glance "what's covered" lives in the README without
bloating it; drop the now-redundant matrix link.
* docs(readme): keep the coverage matrix visible; tuck reference tables under <details>
Matrix (surface × tier) is now a plain visible subsection, not collapsed.
Skills list and Maturity-Levels table moved under <details> (with the gist —
the 4 level names — left inline), since those are reference, not pitch.
* docs(demo): add a runnable 60-second demo (`npm run demo`)
The README's "your CLAUDE.md lies" hook, made runnable: examples/demo ships an
instruction file that reads fine but whose two references lie —
✗ "refreshSession" is not defined in src/auth.ts (renamed symbol)
✗ MCP tool "helper#purge_all" not found — did you mean "purge"?
— while the two truthful refs (verifyToken, helper#log) pass silently. The MCP
check starts a real self-contained stdio server from the demo's .mcp.json. This
is the distribution lever from research/distribution-strategy.md, now backed by
the symbol + MCP verification shipped this session.
- examples/demo/: INSTRUCTIONS.md, src/auth.ts, mcp-server.mjs, .mcp.json,
run.sh, README.md (self-contained, no network).
- `npm run demo` + a README "See it in 60 seconds" callout.
- Isolated: tsc only compiles src/**, the repo's own audit doesn't discover it,
and its generated .vigiles artifact is git/prettier-ignored.
* docs(strategy): close the loop — the runnable demo (proposal A) shipped
Append an update to distribution-strategy.md: the '60-second value, output is
the product' artifact now exists as examples/demo (npm run demo), built curated
+ deterministic to sidestep the heuristic-scan trap that got proposal A
reconsidered. Notes MCP reference verification as a new distinctive Reach hook,
and that the real-plugin dogfood (superpowers/wshobson) is now the evidence base
for an E1 'we scanned the top plugins' post. Next gate is still measurement.
* docs: add research/ + docs/ indexes, TOC, and README navigation
24 research docs (5.7k lines) and 16 user docs had no index — the real
navigation pain with "tons of docs". Rather than risky merges:
- research/README.md — themed index of all 24 research docs (positioning,
the two pillars, skills, proofs, runtime enforcement, FP, distribution/ideas).
- docs/README.md — themed index of the user-facing how-to/reference docs.
- README.md — a Documentation section linking both indexes + the coverage roadmap.
- A Contents line on the harness-testing coverage matrix (the session centerpiece).
Every doc is now reachable from its index, and the indexes from the README
(orphan-docs still green, 43 scanned). Left the docs themselves intact —
consolidation deferred since the clusters (e.g. harness-testing design vs
roadmap vs benchmarks vs user-guide) serve distinct purposes.
* docs(feature-ideas): add a ranked, grouped status overview up top
Keeps all 13 ideas in place but leads with a "Status & priority" section:
marks the 5 already-shipped/partial (dead-enforcement, stale-refs, init,
reverse-coverage, annotation-typo), groups the open ones by theme (catch-more-
lies / change-safety / ergonomics / skills+harness), and ranks them value × cost.
Also clarifies — in the doc — that this is the product backlog (pillar 1), NOT
the harness-testing roadmap (which is the coverage matrix). Top open pick: finish
#2 Reverse Coverage (moat + distribution artifact).
* fix(docs): avoid doc-refs false positive (ctx.file → ctx.sh in an eval example)
The doc-refs validator matched `ctx.file("out.txt")` as the spec `file()`
builder and flagged a missing file — making `vigiles audit` exit 1 (a CI gate).
Switch the example to `ctx.sh("grep …")` (matching the README), which is the
real RunContext API and doesn't trip the matcher. audit back to exit 0.
* docs(research): prior art on testing non-deterministic AI + corrected model
Adds research/testing-nondeterministic-ai.md: a survey of how the field tests
non-deterministic AI/agent tools (DeepEval, Braintrust, Promptfoo, τ-bench's
pass^k + state-over-self-report, AgentAssay, procedure-aware "corrupt success"),
and the model it implies for vigiles. Corrects two conflations from an earlier
sketch:
- predicates (bare, return values) are SEPARATE from assert* (throw); only the
testing helpers carry the assert prefix.
- testing (pass/fail, every commit) and eval (mean±se/pass^k, paid) are SEPARATE
consumers that share predicates over one Trace — not one dual-purpose API.
Folds the gaps into the coverage-matrix roadmap (#10–14): unify the Trace,
capture output + hooks, split predicates from assert*, add pass^k + tool-arg
assertions. Linked from the research index (no orphan).
* docs: fix markdown rendering (stray '- throw' bullet) in the prior-art note
* fix(docs): resolve orphan-doc — reference new doc via full-path backtick
The orphan checker matches `research/foo.md` (it doesn't resolve relative
links), so the bare-basename link to testing-nondeterministic-ai.md didn't count
and audit exited 1 (failing CI's 'check' job). Use the repo convention —
full path in the backtick span, bare href — like the other research cross-links.
* docs(roadmap): save the Trace-unification execution plan for the follow-up
Records the concrete plan for the next session: after #26 merges, branch off
main; do the additive pieces (export predicates + assert* wrappers, unify
runEval ctx with tools/output, r.output, pass^k, tool-arg asserts) which are
non-breaking; defer trace.hooks (needs new instrumentation); keep predicates vs
assert* and testing vs eval separate; acceptance = suites green.
---------
Co-authored-by: Claude <noreply@anthropic.com>