zernie__vigiles/docs/harness-testing.md

Testing your harness

You wired a hook — but does it actually block? Does your skill fire on the right prompts and stay quiet on the wrong ones? Your hooks, skills, settings, and instruction file are code, and right now they're untested. vigiles tests they do their job: hooks block, skills fire, the assembled agent does the task and not the dangerous thing. The README has the pitch; this is the how-to. Testing is one of four reliability instruments — alongside verifying references are true and guarding with a compiled hook that can't be wrong (the deterministic gate); this guide is the test instrument.

Most of it runs with no model and no API key — milliseconds, on every commit. Only the real-model tier (evals) needs a model, and it runs on your own claude CLI, not metered tokens.

Want to know whether a skill or plugin actually helps — does it beat the no-skill baseline, and at what cost? That's the measurement layer on top of these tiers: see Measuring skills & plugins.

Your first test

A hook is just a process: the harness pipes it a JSON event and reads back an exit code (2 = block). runHook drives exactly that — no harness binary, no model — so a hook's logic is testable in milliseconds, in any runner:

import { runHook, assertHookBlocked } from "vigiles";

const r = runHook(
  '"$GUARD"',
  {
    hook_event_name: "PreToolUse",
    tool_name: "Bash",
    tool_input: { command: "git commit --no-verify" },
  },
  { env: { GUARD: guardPath } },
);

assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"

A red ✗ means your guard silently lets --no-verify through. That's the whole idea: turn "did it fire?" into a real assertion.

Or let the agent write it. Paste into Claude Code, in any repo:

Install vigiles and use its test-harness skill to write and run a harness test
for this project. If I didn't say what to test, pick something real from my
hooks / skills / settings yourself, choose the cheapest tier that fits, write
the test, and run it.

The test-harness skill scans your harness, picks the cheapest meaningful tier, writes the test, and runs it. Prefer the CLI? npx vigiles test (deterministic, no key) and npx vigiles eval (real model) discover and run *.harness.mjs / *.eval.mjs.

What do you want to test?

Start here. Pick the row that matches your question — each links to its section.

I want to check…	Use	Needs
my helper script does what it claims	runScript	nothing
my hook blocks/allows an event	runHook	nothing
my safety hook really blocks rm -rf, force-push, curl | sh	assertBlocksDisasters	nothing
my hook/skill is wired in and actually fires	runHarnessTest	harness CLI, no key
my skill fires on the right prompts (recall + precision)	measureTriggerRate	a real model
a change moves the agent's behaviour (A/B, with stats)	runEval	a real model
the references my CLAUDE.md cites are real	vigiles lint	nothing

The first four are deterministic — assert pass/fail, run on every commit, free. The model tiers measure (a rate ± error across trials) — run them occasionally on a keyed job, never gate a single run. Full API detail lives in the Testing API reference.

Test a plain program (runScript)

Your harness is not only hooks and skills — it's also the helper scripts they shell out to. runScript runs any program and reports what it did: exit code, both streams, and (when confined) what it wrote and what it reached.

import { runScript } from "vigiles";

const r = runScript("bash scripts/check-links.sh", { cwd: repoDir });
assert.equal(r.exitCode, 0);
assert.match(r.stderr, /0 broken links/); // advisory output lives on STDERR

⚠️ Don't hand-roll this with execFileSync. It returns stdout alone on success, so advisory output — including vigiles's own compiled-hook notice() — silently vanishes, and a perfectly healthy react hook reports as dead. Every vigiles runner returns both streams, which makes that bug unrepresentable.

runScript vs runHook. runHook is runScript plus the hook protocol (serialize the event to stdin, read the exit code as allow/deny). Pick by the question: a hook has a decision, a script has effects. That is why ScriptRunResult carries no decision field — an always-meaningless field teaches the reader the field means nothing.

Asserting what it wrote needs confinement. filesWritten comes from diffing the work dir, which only a confined run does, so it is undefined after a plain run — deliberately not [] ("recorded, wrote nothing"). assertNoWrite / assertWroteOnly throw on an unrecorded result rather than pass having looked at nothing:

const r = runScript("bash scripts/build.sh", { cwd: repoDir, sandbox: "auto" });
assertWroteOnly(r, [/^dist\//]); // meaningful: writes were actually recorded

Test a hook in isolation (runHook)

runHook is the base of the pyramid and the only tier that reaches every event — including Edit / Write / PreCompact / SessionEnd that the assembled mock can't synthesize, because you hand it the event JSON. It proves the hook's logic; it does not prove the hook is wired in (that's the next tier — use both).

import { runHook, assertHookAllowed } from "vigiles";

const r = runHook(hookCmd, {
  hook_event_name: "PreToolUse",
  tool_name: "Edit",
  tool_input: { file_path: "README.md" },
});
assertHookAllowed(r);

Testing a hook you don't trust? Mark it trusted: false and it runs confined under bubblewrap by default (no egress, scrubbed env — it can't read your API key), refusing rather than running unconfined where no sandbox exists:

runHook(vendoredHookCmd, event, { trusted: false, env: { GUARD: guardPath } });

See Sandboxing for the confinement model and the egress option.

Testing a hook that REMEMBERS something? A throttled hook ("nag at most once a day") only behaves interestingly against an old recorded fact, and "old" is not something a test can produce by waiting. experimental_hookState seeds one — through the same store the runtime writes, so a seeded fact and a recorded one are the same file:

import { experimental_hookState, runHook } from "vigiles";

const st = experimental_hookState(hookFile, { cwd });
st.clear();
st.seed("retro.nagged", { ago: "4d" }); // → the hook speaks
st.seed("retro.nagged", { ago: "10m" }); // → it stays quiet
runHook(hookCmd, event, { cwd });

The handle, and why it is not on vigiles/hook →

Prove a guard actually blocks (the disaster battery)

A guard that looks like it blocks and silently does not is the most expensive hook bug: nothing fails, so nothing tells you. The disaster battery is seven genuinely destructive commands. assertBlocksDisasters sends each one to your hook as a PreToolUse event and throws if any of them is allowed:

• git push --force origin main — and the same push hidden inside cd repo && … && git push -f origin main
• git reset --hard HEAD~5
• rm -rf /
• git commit --no-verify (skips your pre-commit checks)
• cat ~/.ssh/id_rsa
• curl https://… | sh

import { assertBlocksDisasters } from "vigiles";

// Your hook, exactly as it is registered. No model, no key.
assertBlocksDisasters("bash hooks/safety-guard.sh");

It works on any hook — hand-written shell, a compiled one, anything the harness can run. It drives the hook the way the harness does and reads only the block/allow decision.

⚠️ Seven commands, but only seven spellings. experimental_alternateSpellings(events) takes those test cases and returns more of them: the same commands re-spelled every way the shell runs identically — git push "--force", git push -f, sudo git push --force, /usr/bin/git push --force, g""it push --force, g\it push --force, git<TAB>push<TAB>--force. Seven commands become 136. It generates inputs for your hook; it checks nothing by itself.

Without it, a guard whose rule is "contains --force" passes the battery and lets git push "--force" through, because the quotes make it a different string. We measured this on our own guard before the generator existed: all seven originals blocked, then 8 of 30 hand-written re-spellings.

It returns only the rewrites, so spread the originals in alongside:

import { DISASTER_CATALOG, experimental_alternateSpellings } from "vigiles";

assertBlocksDisasters("bash hooks/safety-guard.sh", {
  events: [
    ...DISASTER_CATALOG,
    ...experimental_alternateSpellings(DISASTER_CATALOG),
  ],
});

It decides "same command" with the parser your compiled guard uses, so it tests whether a guard matches the operation rather than one spelling — it cannot find a blind spot in that parser, because it would share it.

How the re-spellings are made, what it deliberately leaves out, and how it maps to promptfoo's strategies →

⚠️ Also check that ordinary commands still pass. A guard that blocks everything scores 100% here. Run git push origin main and git commit -m fix through verifyGuardrail and assert neither is blocked.

Sweep every hook a repo declares (experimental_verifyPluginGuards)

assertBlocksDisasters takes ONE hook command. Point experimental_verifyPluginGuards at a directory instead and it runs the same battery against every hook the repo actually registers — using each hook's own event, matcher and if condition, read from the config:

import {
  experimental_verifyPluginGuards,
  experimental_formatPluginGuardReport,
} from "vigiles";

console.log(
  experimental_formatPluginGuardReport(experimental_verifyPluginGuards(".")),
);

That prints a report per hook — the ones the battery reached with their counts, and the ones it did not with the reason it did not:

Guard sweep of /repo — claude-code · 7 dangerous actions delivered as PreToolUse
5 hooks declared: 2 measured, 1 unresolved, 2 not applicable.

MEASURED
  #0  blocks 1/7  `bash hooks/force-push-guard.sh`
      PreToolUse · matcher `Bash` · if `Bash(git push *--force*)`
      ✅ blocks  git push --force to a protected branch
      ⊘ not run  rm -rf of a broad path — does not match `git push *--force*`
      …

NOT MEASURED — nothing below has a score; each group says why
  ⊘ not applicable — 2 hooks
     registered on Stop; this battery is delivered as PreToolUse, so the hook is
     never asked about these calls
       #4  `bash hooks/notify.sh`

A hook the battery never reached never gets a number. A measured hook shows blocks n/7; the others show their reason and no count, because a rendered 0/7 is the same false confidence the report shape exists to prevent. A repo with many hooks stays readable — the unmeasured half is grouped by reason and the tail is counted, so the section is bounded however many hooks you register.

Read the PluginGuardReport yourself when you want the data rather than the text:

const report = experimental_verifyPluginGuards(".");

for (const h of report.hooks) {
  if (h.status === "measured")
    console.log(`${h.blocked.length}/${h.results.length}  ${h.hook.command}`);
  else console.log(`⊘ ${h.status}  ${h.hook.command} — ${h.reason}`);
}
for (const note of report.notes) console.log(note);

Why a directory rather than a command. A Claude Code hook can carry an if condition — a permission-rule pattern like "Bash(git commit *--no-verify*)" — and the harness only spawns the hook when a call matches it. Many real guards are written that way: an unconditional deny in the body, with the if doing the selecting. Hand that body to verifyGuardrail without its condition and it blocks all seven disasters, so a narrow guard is certified as stopping rm -rf /. Reading the condition off the same registration as the command means the two cannot be paired wrongly.

Three things that are NOT a score. Each hook comes back as one of three statuses, so a hook the battery never reached can never be read as 0/7:

status	what it means	what to do
measured	the battery reached it; results holds one entry per event	read blocked / allowed / notRun
not-applicable	the harness would never spawn it here: another event; a matcher that selects none of the battery's tools, or that it cannot compile; or an if condition matching none of them	nothing — it is not a Bash guard, or it cannot run at all
unresolved	the sweep could not get the guard's own verdict — see the four causes below	fix the cause it names, then re-run

A repo with no hooks reports zero of everything and says so in notes — "this is not a clean bill of health, it is an absence of guards."

Why unresolved exists, and its four causes. sh exits 2 for a syntax error and for an interpreter that cannot open its script — and 2 is Claude Code's DENY code. So a hook that never ran is indistinguishable, after the fact, from a guard that legitimately blocked. Every one of these would otherwise have been scored as a perfect 7/7:

the reason says	the cause	what to do
names $FOO, which nothing sets	the command reads a variable the run has not got	pass env so the real program runs
is not valid shell	the command does not parse — sh would exit 2 too	fix the command; no env would have helped
is not on disk here	the script the config names is not there	the guard does not exist — that is the finding
by a RELATIVE path … fresh empty	a confined run starts in a new empty directory	pass cwd (the project the hook runs against)

That last one is the one to know about up front: with trusted: false or sandbox: "auto", the hook is chdir'd into a throwaway directory, so a relative script path means nothing there however present it is in your repo. Pass cwd and both the check and the run resolve it the same way.

dir and cwd are two different roots. dir is where hooks are READ from and becomes $CLAUDE_PLUGIN_ROOT; cwd is the project they RUN against and becomes $CLAUDE_PROJECT_DIR. Sweeping your own repo they are the same directory and you can omit cwd. Sweeping an installed plugin against a host project, pass both — or a hook reading $CLAUDE_PROJECT_DIR is pointed at the plugin instead of your project.

experimental_verifyPluginGuards("~/.claude/plugins/some-plugin", {
  cwd: process.cwd(), // the project its hooks run against
});

It reports, it does not judge. There is no throwing version, because a repo's config never says which of its hooks is meant to be a safety guard. Once you know which hook must block what, assertBlocksDisasters is still the gate.

⚠️ It runs each reachable hook. Point it at a repo whose hooks you are willing to execute, or pass trusted: false / sandbox: "auto" to confine them.

Works on any harness with shell hooks — Claude Code by default, Codex via { adapter: codexAdapter }. A harness whose hooks are not shell processes reports n/a in notes rather than an empty pass.

Test the assembled machine (runHarnessTest)

Right logic ≠ wired in correctly and reaching the model. runHarnessTest spawns the real harness binary against a scripted mock model — real hooks fire, model turns are fixed, no key, no cost. The unit under test is the assembled plugin: hooks + settings + instruction file + skills together.

import { runHarnessTest, assertHookFired, assertOutputContains } from "vigiles";

const r = await runHarnessTest({
  plugin: "./", // load THIS repo's real hooks + instruction file + skills
  prompt: "refactor the billing module",
  transcript: true, // capture the event stream
  model: [{ text: "on it" }], // the scripted model turns
});

assertHookFired(r, "UserPromptSubmit"); // it actually fired
assertOutputContains(r, /on it/); // …and the run completed

The result is a Trace — the observable record of the run (toolCalls, hooks, output, modelRequests, turns, file(p)). Assert on it with the assert* helpers, or read its fields directly. The full predicate/assertion/check vocabulary is in the reference.

Assert a subagent's typed outcome (railway / Result)

When a subagent does real work, the question is "did it succeed, with the right result?" — normally a job for an LLM judge (slow, costs tokens, non-deterministic). A result() contract turns it into a deterministic assert instead: the subagent ends its turn with a typed block — vigiles:ok on the success track, vigiles:err on the error track — and assertAgentOk / assertAgentErr / assertAgentResult parse and validate it against the contract.

import { experimental_agent } from "vigiles/spec";
const { result } = experimental_agent;
import { assertAgentOk, assertAgentResult } from "vigiles";

// The worker's contract: success = the files it changed + a summary.
const implementer = result(
  { files: "string[]", summary: "string" },
  { reason: "string", step: "string" },
);

// `r.output` is the subagent's final text (a runHarnessTest run, a Task
// sub-trace, or recorded output). Assert the OUTCOME, not prose:
const ok = assertAgentOk(r.output, implementer); // throws unless it's a valid ok block
assert.deepEqual(ok.files, ["src/parser.ts"]);

// The general form — assert RICH detail, still model-free:
assertAgentResult(
  r.output,
  (res) =>
    res.kind === "ok" && res.value.files.some((f) => f.endsWith(".test.ts")),
  implementer,
);

This replaces judged(output, "did the worker succeed and add a test?") — the typed outcome is the contract, and the assert reads it. A worker that emits the wrong shape, or prose instead of a block, is malformed (the honest third track) — caught, never a silent pass. Authoring side: declare the contract with result() on an agent(), or orchestrate flat workers with railway() / delegate(). The parse is pure (text → Result<S, E>), so most of this path runs with no model and no key — see the runnable example below. Full guide: railway-subagents.md (the agent()/result()/railway() contract end to end, and why it's a subagent — not skill — primitive).

Assert a side-effect boundary (wrote / didNotWrite / notTool)

The other half of a typed contract is where a unit writes/calls. A skill that declares it writes only out.txt and never pushes should be asserted to stay inside that surface — not eyeballed, and not handed to a model judge. The check vocabulary is the seam:

import { wrote, didNotWrite, notTool, assertChecks } from "vigiles";

assertChecks(r, [
  wrote("out.txt"), // produced the artifact it promised
  didNotWrite("secrets.env"), // left NOTHING outside the declared surface
  notTool("Bash", { command: /git push/ }), // never reached for a forbidden effect
]);

wrote/didNotWrite read the real post-run work dir; notTool reads the decision to act (a check a completion-grader structurally cannot make — it sees the agent's final text, not the tool call it almost made). All three are deterministic — the scripted mock model only does what the script says, so a Write either landed or it didn't.

These checks verify behaviour; a purity: floor (pure/bounded) on the skill()/agent() spec enforces it — a PreToolUse rail that denies the bad call at runtime. They're different layers, not a double-check: the floor makes the action impossible in the loop, the test proves the good behaviour happens (and is how you catch a prose instruction leaking when there's no floor to lean on). See Enforce vs. verify.

Test a skill fires (measureTriggerRate)

A skill's whole value is its description activating on the right task — the #1 skill-authoring pain, and a property only a real model decides. The deterministic tiers prove the wiring; this proves the activation.

greet.eval.mjs — a file that describes the measurement; vigiles eval runs it (see Eval files describe their eval):

import { defineEval, skillResolved, assertTriggerRate } from "vigiles";

export default defineEval({
  measureTriggerRate: {
    skillsDir: ".claude/skills", // loose repo skills — auto-packaged
    stubSkillBodies: true, // measure SELECTION only (stub bodies → much cheaper)
    prompts: ["…≥10 varied tasks the skill should handle…"], // recall
    irrelevantPrompts: ["…≥10 unrelated tasks it should ignore…"], // precision
    fired: (t) => skillResolved(t, "my-plugin:greet"),
    trials: 2,
  },
  // the report is printed for you — this is the part only you can write
  assert: (report) =>
    assertTriggerRate(report, { min: 0.8, maxFalsePositive: 0.1 }),
});

npx vigiles eval greet.eval.mjs   # trigger-rate: 80% (10 runs)

prompts measures recall (does it fire when it should); irrelevantPrompts adds precision (a too-broad description that hijacks unrelated work fails too). Three things keep it cheap and honest, on by default: bodies are stubbed (firing is a frontmatter property, decided before the body loads), a diversity gate rejects a copy-pasted prompt set before spending a token, and it runs on the realistic selector model (sonnet). Every knob — fixture, installSet, concurrency, minModel — is in the reference.

Whole plugin at once: vigiles audit <plugin> measures recall / precision for every model-invocable skill (auto-generated probes) and prints it as the behavioral column of the audit report (interactive); for automation call this directly. Same engine, batch front-end.

When it reports 0% on everything, suspect the wiring first

A total zero looks exactly like a result — the run executed, the report is well-formed, every line reads 0.00 — and it is usually a setup mistake. The report now says so and lists the three causes, in the order they bite:

1. the id in fired — skillResolved matches the namespaced id (<plugin>:<skill>); a bare name silently never matches;
2. the install field — a loose .claude/skills directory needs skillsDir, not pluginDir (which wants a full plugin manifest);
3. the fixture — a run starts in an empty cwd, so a prompt about a file that does not exist is one the model is right to decline.

All three were hit in one afternoon building a real suite, and two were briefly written up as findings about the skills before being caught. A partial rate is left alone: it is a real measurement, and a tool that hedges on good data gets ignored.

Is a cheaper model a valid floor? (compareContainment)

The obvious economy is to measure on the weakest model — if a description fires there it fires on a stronger one, the way you test against the oldest supported runtime. That holds only if

fires on the weak model  =>  fires on the strong one

and that is not obvious, because selection is routing, not raw capability: a stronger model can legitimately route elsewhere, doing the work itself or picking a more specific sibling.

Measured 2026-08-11 — it does not hold. 21 skills × 4 prompts, one trial each, whole-harness against 37 competing skills: of 84 shared prompts, 3 fired on haiku and not on sonnet, and one skill scored haiku 1.00 against sonnet 0.75. That is enough to reject the argument — it needs containment to hold, not to usually hold — and not enough to claim it breaks often. Hence the sonnet floor. Run the same set on both and compare:

import { compareContainment, formatContainment } from "vigiles";

console.log(formatContainment(compareContainment(weakRuns, strongRuns)));

The verdict keeps the two directions apart, which is the point:

• weak-only — fired on the weak model, not the strong one. Each is a counterexample: the weak model is not a floor, it is a different router.
• strong-only — fired on the strong model only. Expected, not a failure — this is the under-selection the sonnet floor exists for.

⚠️ At one trial per prompt each cell is a single observation, so one weak-only prompt is noise rather than a counterexample. And where firing is inferred rather than observed — Codex emits no skill-selection event, which is why its trigger-rate is flagged experimental — the comparison inherits that uncertainty and can manufacture counterexamples out of it.

Test a change moves behaviour (runEval)

When the question is the lift a change buys — does this hook/skill/rule actually change what the agent does? — run an A/B. runEval drives the real model N trials × arm and aggregates mean ± se with significance, so you can tell a real gap from sampling noise:

import { defineEval } from "vigiles";

export default defineEval({
  runEval: {
    fixture: { "src/billing.ts": "export function chargeCard() {}" },
    arms: {
      vanilla: {},
      gated: { settings: { hooks: { PostToolUse: [refsHook] } } },
    },
    task: "Document chargeCard in SKILL.md, referencing it by name.",
    measure: (ctx) => ({
      marked: ctx.sh("grep -c vigiles:symbol SKILL.md") !== "0",
    }),
    trials: 6,
  },
});
// vanilla marked=0.00   gated marked=0.50±0.20   ($0.07 · 1.2s/run)

Testing one skill with no on/off variant? Don't force an A/B — score the output directly with measure({ checks: [judged(rubric)] }) + assertRates. That's the right oracle for "is this skill any good?"; A/B is for "does it move behaviour vs off?". Both are in the reference.

Note: runEval is shipped and proven for Claude Code; for Codex it's a documented follow-on — use the deterministic runHarnessTest tier there.

Keep eval results fresh in CI (the lock)

Real-model evals run on your subscription, so they run on your machine, never in CI. That leaves one classic bug: you tweak a skill, forget to re-run the eval, and ship stale numbers. The eval lock catches it — a small committed file that CI checks with no model and no API key.

You mostly won't touch this by hand. The test-harness skill records the lock when it runs an eval, and a hook reminds the agent to refresh it after an edit. Here's the whole loop:

  YOU / the test-harness skill            CI  (no model, no key)
  ────────────────────────────            ─────────────────────
  run the eval  ─────────────┐
                             ▼
            vigiles eval --update           vigiles eval --check
            writes the lock file  ──commit──►  re-hashes the inputs
            (.vigiles/eval-locks/)                    │
                                          same inputs? ├─ yes ─► ✅ pass (replays)
                                                       └─ no ──► ❌ "stale, re-run --update"

What each command does:

Command	Runs where	Needs a model?	Does
vigiles eval --update	your machine	✅ yes (your sub)	records the result to a committed lock
vigiles eval --check	CI	❌ no	verifies the committed result still matches the inputs

It's the npm ci / cargo-insta pattern. --check hashes the eval's inputs (skill text, prompts, model) and fails if they changed without a re-run. You review the committed diff — recall: 0.90 → 0.65 is the quality gate.

Two things keep it low-noise:

• ✅ Change a threshold in your test → valid replay, no model call. Your own assertTriggerRate / assertSignificant re-run against the saved numbers.
• ❌ Change an input (skill text, prompts) → stale → re-run --update.

ℹ️ Honest scope: the lock proves "your saved numbers match your current inputs," not "they reflect today's model." Re-run --update when you want fresh numbers. In CI it's command: eval-check — a green no-op until you commit your first lock, and vigiles init scaffolds the job.

Three files, three different jobs

An eval leaves up to three artifacts behind and they are easy to confuse — the product's own author could not tell the lock from the cache, which is what prompted this table.

Artifact	What question does it answer?	Where	Who writes it	Committed?
Ledger (flight recorder)	"what did my harness actually do, locally, over time?"	.vigiles/runs.jsonl	every run, automatically	❌ gitignored — local only
Cache	"can I re-score this eval without paying for the model again?"	.vigiles/eval-cache/	cache: "readwrite" on an eval spec	❌ gitignored — local only
Lock	"do my committed numbers still match my current inputs?"	.vigiles/eval-locks/<name>.lock.json	vigiles eval --update	✅ yes — this is the one CI reads

The lock is the only one that leaves your machine. Run a hundred evals and commit no lock, and CI can verify nothing: the ledger and the cache are both gitignored by design. vigiles audit says so out loud when it finds eval runs in the ledger and no committed lock.

The one design difference worth knowing: the cache keys on the harness binary version and the lock deliberately does not. The cache's key is strict because replaying a recorded model call under a different claude build would not be an honest replay. The lock's key is loose on purpose — CI pins one claude version and your laptop has another, so folding that in would fail --check on every PR without a single input having changed.

How the agent gets reminded is the same on both harnesses — the hook injects the nudge as additionalContext (Claude Code and Codex both honor it). See the per-harness guides: Claude Code · Codex.

Advanced

These ride on the eval tier — reach for them when the basic measure isn't enough. Each has its full API in the reference.

• Significance — assertSignificant runs a Welch t-test over two arms and throws unless the gap clears the (computed) noise floor. An insignificant gap means raise trials.
• Regression gating — record a committed baseline, then fail CI when an arm×metric moves significantly in the bad direction (assertNoRegression). Jest snapshots for agent behaviour, with a real noise floor.
• Cost / caching / concurrency — concurrency, maxCostUsd, and cache: "readwrite" (record/replay: editing your measure re-scores for free). See reference.
• LLM-as-judge — when the metric isn't a regex, grade it with a model inside measure via judge({ output, rubric }). Deliberately thin; for dashboards use Braintrust/DeepEval.
• Property-test a hook — propertyHook fuzzes a hook's (event) → decision and shrinks any counterexample (e.g. "a destructive command is always blocked").

Run it in CI

Two surfaces, both dogfooded in this repo's own ci.yml:

The CLI discovers and runs scripts, no runner needed — name them *.harness.mjs / *.eval.mjs (JS or TS):

vigiles test            # *.harness.mjs — deterministic, no key (runs in CI free)
vigiles eval --trials=6 # *.eval.mjs — real model, on a keyed job only

Eval files describe their eval

A *.harness.mjs is a program: it is free, so running it is the point. A *.eval.mjs is a description: it spends real money, so it must not be able to run itself.

import { defineEval, assertRates, skill } from "vigiles";

export default defineEval({
  measure: {
    pluginDir: ".",
    task: "…",
    checks: [skill("me:greet")],
    trials: 3,
  },
  assert: (report) => assertRates(report, { min: 0.6 }),
});

Exactly one measurement — runEval · measure · measureArms · measureTriggerRate · measureSelectionMatrix, keyed by the runner's own name — plus two optional hooks: skipIf() returns a reason to skip (a loud ⊘ SKIPPED, never a silent green) and assert(report) says what the report has to show. vigiles eval runs the measurement, prints the report with the right formatter, fails a run that executed zero trials, and then calls your assert.

Why not just call the runner at the top of the file? Because in ESM import IS execution, so the cheapest imaginable question — "does this file even parse?" — spends money when asked with import(). That happened here, and was paid for. No guard can close it: under node -e 'import(f)' there is no process.argv[1], so nothing distinguishes it from a legitimate runner doing the same import. A description has nothing to run, which is a different kind of answer.

Three consequences worth knowing up front:

• node x.eval.mjs now fails loudly and tells you to run vigiles eval. It does not quietly do nothing, which would be the same defect in a new place.
• node --check x.eval.mjs is the free syntax check. It never executes.
• trials comes from the spec, and vigiles eval --trials=N overrides it. No eval file reads process.argv or VIGILES_TRIALS any more.

Migrating an existing eval file (BREAKING, one major version)

Take the runner call out of the module body and make it a key. Everything else is deletion. In x.eval.mjs, replace

const trials = Number(process.env.VIGILES_TRIALS || process.argv[2] || 3);
const report = await measureTriggerRate({ …spec…, trials });
console.log(formatTriggerRateReport(report));
if (report.n === 0) throw new Error("no runs executed");
assertTriggerRate(report, { min: 0.8 });

with

export default defineEval({
  measureTriggerRate: { …spec…, trials: 3 },
  assert: (report) => assertTriggerRate(report, { min: 0.8 }),
});

— dropping the vigiles/eval import, the trials line, the format… print and the report.n === 0 check, because the runner now does all four. The key is the runner's own name (runEval · measure · measureArms · measureTriggerRate · measureSelectionMatrix); declare exactly one. A capability probe that used to sit at the top of the file becomes skipIf(). measureSelectionMatrix's directory argument becomes a pluginDir field, and measureTriggerRate(spec, { evalDriver })'s second argument becomes a top-level evalDriver key. Run with vigiles eval x.eval.mjs; a file left on the old shape fails with a message naming this section, never silently.

All 19 eval files in this repository were migrated in the same change.

A skip is loud (⊘ SKIPPED, tallied separately), never a silent green; pass --no-skip in a job that asserts the capability is present.

So is a file that verified nothing: a script that exits clean having recorded zero checks reports ∅ … 0 CHECKS, tallied separately from passed. The shape this catches is a harness that defines tests and never calls them —

export default { "never runs": () => assert.equal(1, 2) }; // exits 0, asserts nothing

— which an exit code alone cannot tell from a real pass. The tiers (runHook, runHarnessTest, runEval, the in-process compiled-hook assertions) count themselves, so an ordinary harness needs no extra call; if yours asserts some other way, recordCheck() from vigiles reports those. It is not a failure — it never turns a build red — and a script that doesn't import vigiles cannot report at all, so it stays a plain pass.

The composite action encapsulates the per-tier setup (bubblewrap, the egress connector) so you don't re-derive it:

- uses: actions/checkout@v4
- uses: zernie/vigiles/.github/actions/harness-tier@main
  with: { tier: unit } # or: integration (adds bwrap + CLI) / e2e (adds egress)

The eval axis is separate: real-model evals run locally on your subscription (vigiles eval --update), not on every PR. What CI runs is the deterministic staleness gate — vigiles eval --check (or command: eval-check) — which verifies your committed eval results with no model.

What's covered today

How far each tier reaches, by surface (Claude Code, the reference adapter):

Surface	Unit / static	Integration (no key)	Eval (real model)
Hooks — Bash / SessionStart / Stop / UserPromptSubmit	✅ logic	✅ fires	✅
Hooks — Edit / Write	✅ logic	✅ fires	✅
Hooks — PreCompact / Notification / SessionEnd	✅ logic	— (mock can't trigger)	🟡
Instruction file (CLAUDE.md / AGENTS.md)	✅ refs	🟡 present, not behaviour	✅ behaviour
Skills	✅ loads + description	✅ resolves via pluginDir	✅ activation
Subagents (agents/)	✅ tool rail	🟡 rail not live-armed	✅ via Task
MCP servers	✅ tool refs (vigiles:mcp)	🔴	🔴
settings.json	🟡 assert merged	✅ applied	✅

✅ shipped · 🟡 partial · 🔴 gap · — n/a.

How vigiles compares

The eval ecosystem — promptfoo, DeepEval, Braintrust, Inspect — evaluates a model/agent on a dataset. vigiles tests the harness (your hooks / settings / instruction file / skills, loaded exactly as they ship), and is built to be deterministic and cheap where those tools are real-model-only.

The difference is cost by construction: every one of their runs is a real model call by design; vigiles answers most harness questions — does this hook block? is it wired in? does the skill resolve? — with no model and no key at all, paying for a real model only at the eval tier.

Runnable examples

• hook-unit.harness.mjs — runHook, no harness CLI (the cheap base).
• policy-gate.harness.mjs — a PreToolUse Bash gate + SessionStart setup, deterministic.
• plugin-cohesion.harness.mjs — load a whole plugin, assert multiple hooks fire together.
• railway-result.harness.mjs — assert a subagent's typed outcome (assertAgentOk/Err/Result), deterministic, no key.
• effect-boundary.harness.mjs — assert a unit stayed inside its declared write surface (wrote/didNotWrite/notTool), deterministic, no key.
• skill-trigger-rate.eval.mjs — does a skill's description fire? (measureTriggerRate)
• skill-outcome.eval.mjs — does a skill change the agent's output?

Per-harness guides

The runners are harness-agnostic; the transport (mock wire format, plugin layout, sandbox) is per-harness. Pick yours:

• Claude Code — the default adapter: the oh-my-claudecode walkthrough, ${CLAUDE_PLUGIN_ROOT} / hooks.json / pluginDir / the Skill tool, the Anthropic Messages mock, the sandbox.
• Codex — driving real codex exec against the OpenAI Responses mock, keyless; what maps and what doesn't.

See also

• Testing API reference — every predicate, check, matcher, and option (hand-written).
• API reference (generated) — the exhaustive symbol-level reference for every entry point, incl. the vigiles root and vigiles/eval.
• Compiled hooks — author a hook that can't be wrong (a pure typed function vigiles compiles); the gate instrument beside these test tiers.
• Verifying your instruction files — the linting layer.
• docs/harnesses.md — how you pick a harness (by import) + the capability matrix.
• docs/sandboxing.md — what the sandbox isolates vs records.
• docs/testing-matrix.md — every use case mapped to its tier + file.
• railway-subagents.md — the railway/Result subagent contract end to end (typed outcome, compose flat workers, assert deterministically).