zernie__vigiles/docs/testing-api.md

Testing API reference

The complete surface behind Testing your harness — every predicate, assertion, check, matcher, and option. The guide has the task-first how-to; this is the reference you reach for when you need the exact name or knob.

Contents

• Picking a runner
• And one that asks about the tests themselves
• The Trace model
• Predicates
• Assertions
• The check vocabulary
• vitest / jest matchers
• measureTriggerRate options
• Selection-collision matrix (Claude Code only)
• runEval, measure, measureArms
• Significance & regression gating
• Imports & harness selection

Picking a runner

Four runners, keyed on the question you actually have. The first three are free and deterministic; only the last needs a model.

Your question	Runner	Result	Cost
does my helper script do what it claims?	runScript	ScriptRunResult	free
does my hook block what it says it blocks?	runHook	HookRunResult	free
does the assembled harness behave (hooks+settings+skills)?	runHarnessTest	HarnessTestResult	free, needs the CLI
does the real model pick my skill / do the job?	measureTriggerRate / runEval	reports	paid

runHarnessTest is deterministic and free despite its model: field — those are scripted turns (ModelTurn[]) served by a mock. Nothing reaches a real model at that tier.

runScript is the primitive; runHook is it plus the hook protocol (event → stdin, exit code → allow/deny). A hook has a decision; a script has effects — so ScriptRunResult deliberately carries no decision field.

Both streams, always. ScriptRunResult, HookRunResult and HarnessTestResult all carry stdout and stderr. That matters more than it looks: advisory output — including vigiles's own compiled-hook notice() — goes to stderr, so a hand-rolled execFileSync runner (which returns stdout alone on success) silently deletes it and reports a healthy react hook as dead.

filesWritten is undefined until something records it. Writes are captured by diffing the work dir, which only a confined run does. undefined means "nobody looked"; [] means "looked, wrote nothing" — assertNoWrite / assertWroteOnly throw on the former instead of passing vacuously.

The Trace model

Every tier produces one Trace: the observable record of a run. A runHarnessTest result is a Trace, and so is the ctx handed to a runEval measure. Its fields:

Field	What it holds
toolCalls	every tool the agent invoked (name + input)
hooks	each HookFire — name, event, exitCode, blocked, output
output	the agent's final answer
modelRequests	what actually reached the model (did injected context land?)
turns	number of agent turns
file(p)	read a file from the post-run filesystem
sh(cmd)	run a shell command against the post-run filesystem (eval ctx)
usage	{ costUsd, durationMs, inputTokens, outputTokens } (eval)

trace.hooks is recorded, not inferred: each HookFire comes from the CLI's stream events, so a test asserts a hook actually fired and blocked — no marker file the hook had to write. Capture it with transcript: true on the harness tier (always on at the eval tier).

Predicates

Pure functions over a Trace — no assert prefix, no throw. Use them in a measure as metrics, or wherever you want a boolean.

import {
  usedTool,
  toolCount,
  skillResolved,
  toolUsedWith,
  outputContains,
  hookFired,
  hookBlocked,
} from "vigiles";

usedTool(trace, "Skill"); // boolean
usedTool(trace, /^mcp__github__merge/); // boolean (regex)
toolCount(trace, "Write"); // number
skillResolved(trace, "demo:greet"); // boolean
toolUsedWith(trace, "Edit", (i) => isRightFile(i)); // boolean (tool argument)
outputContains(trace, /done/i); // boolean (the final answer)
hookFired(trace, "PreToolUse:Edit"); // boolean (recorded from the stream)
hookBlocked(trace, "PreToolUse"); // boolean (fired AND exit ≠ 0)

Assertions

Each assert* is a predicate wrapped in a throw (assertToolUsed is usedTool then throw; assertSkillResolved is skillResolved then throw). They're plain functions, so they work in node:test, vitest, jest, or any runner. Beyond a single tool you can assert on sequence and budget:

assertToolSequence(r, ["Read", "Edit"]); // ordering — Read before Edit
assertToolCount(r, "Write", { max: 1 }); // budget — no runaway writes
assertToolUsedWith(r, "Edit", (i) => i.file_path === "src/billing.ts"); // argument
assertToolCalls(r, (calls) => /* any custom rule over the list */ true);

assertHookBlocked(r); // exit 2 / decision:"block" / permissionDecision:"deny"
assertBlocksDisasters("bash hooks/guard.sh"); // the hook denies every disaster (force-push, rm -rf, curl|sh …)
assertHookFired(r, "UserPromptSubmit");
assertOutputContains(r, /on it/);
assertCreated(r, "DONE"); // a file was created

withHarness(spec, fn) wraps a deterministic run with try/finally cleanup so a test doesn't leak temp dirs.

Test a compiled hook in-process (no subprocess)

A compiled hook (a vigiles/hook program) has a pure decision, so you can test it without spawning the CLI at all — pass the hook's default export and a raw event:

import {
  loadHook,
  assertHookDenies,
  assertHookAllows,
  assertHookNotices,
  assertHookSilent,
} from "vigiles";
import { runHookProgram } from "vigiles/hook";

// loadHook takes the hook's PATH — what a .harness.mjs file actually has. It is
// the same loader the runtime uses, and it handles a .hook.ts under tsx / Node
// >= 23.6. Already holding the object (a static import)? Pass it directly.
const guard = await loadHook(".vigiles/hooks/guard.mjs");

assertHookDenies(guard, {
  tool_name: "Bash",
  tool_input: { command: "git push -f" },
});
assertHookAllows(guard, {
  tool_name: "Bash",
  tool_input: { command: "git status" },
});

// A REACT hook can't block, so it gets its own pair. NB `notice()` writes to
// STDERR — a probe that reads stdout reports a healthy react hook as dead.
// These read the reaction itself.
const warn = await loadHook(".vigiles/hooks/warn-on-failure.mjs");
assertHookNotices(warn, failedBashEvent, /read the error/); // message matcher optional
assertHookSilent(warn, successfulBashEvent);

// the raw primitive: a normalized, role-dispatched outcome
runHookProgram(guard, event); // → { kind: "decision" | "injection" | "reaction", … }

assertHookBlocked / assertHookAllowed (above) are the sibling assertions over a runHook result (the hook run as a real subprocess); assertHookDenies / assertHookAllows evaluate the typed program directly — cheaper, and they work for inject/react hooks too via runHookProgram.

Assert a subagent's typed outcome

For the railway/result subagent contract, assertAgentOk / assertAgentErr / assertAgentResult test a subagent's outcome via parseAgentResult — a deterministic assert that replaces an LLM judge. A subagent with a result() contract ends its turn with a vigiles:ok / vigiles:err block; these helpers parse it and validate it against the declared shape:

import { experimental_agent } from "vigiles/spec";
const { result } = experimental_agent;
import { assertAgentOk, assertAgentErr, assertAgentResult } from "vigiles";

const contract = result(
  { files: "string[]", summary: "string" }, // the ok track
  { reason: "string", step: "string" }, // the err track
);

const value = assertAgentOk(r.output, contract); // returns the validated `value`
const error = assertAgentErr(r.output, contract); // returns the validated `error`

// The general predicate form, for rich detail:
assertAgentResult(
  r.output,
  (res) => res.kind === "ok" && res.value.files.length > 0,
  contract,
);

Pass the contract and a wrong/missing field fails the assertion; omit it and any well-formed JSON block is accepted. Output that isn't a valid block is malformed (throws) — never a silent pass. Worked example: railway-result.harness.mjs.

The check vocabulary

A check is data, not a throwing assert: tool("Bash") is an object that knows how to eval itself against a Trace (or a hook decision) and toJSON. The same check is evaluated two ways — write the assertion once:

import { tool, skill, output, hookFired, blocked, assertChecks } from "vigiles";

// STRICT (deterministic tiers): throws, collecting ALL failures with messages.
assertChecks(await runHarness(spec), [tool("Bash"), output(/done/)]);
assertChecks(runHook(cmd, event), [blocked()]);

// SCORED (eval): the SAME checks, as a rate ± se across trials.
import { paid_measure } from "vigiles/eval"; // the `paid_` prefix = it calls a model
import { assertRates, checkReportToJUnit } from "vigiles"; // reading the report is free
const report = await paid_measure({
  pluginDir: "./my-plugin",
  task: "…",
  checks: [skill("vigiles:test-harness")],
  stubSkillBodies: true, // firing check: stub bodies → a fraction of the tokens
  trials: 10,
  model: "sonnet",
});
assertRates(report, { min: 0.8 }); // gate the rate, not one noisy run
writeFileSync("checks.junit.xml", checkReportToJUnit(report, { min: 0.8 }));

A check's failure message is the product (expected the agent to use tool "Bash", but it used [Read, Edit]), and because it serializes, CI reports and regression baselines fall out for free.

The vocabulary (from vigiles — except paid_judged, which calls a model and so lives on vigiles/eval):

Check	Holds when…	Over
tool(name)	the agent used that tool	Trace
skill(id)	a Skill resolved to id (no error)	Trace
mcp(server, tool)	the agent used mcp__server__tool	Trace
subagent(name, [checks])	the Task subagent ran + passed nested checks	Trace (nested)
output(str | /re/)	the final answer contains/matches	Trace
received(str | /re/)	text reached the model (slash-command / injected)	Trace (mock)
hookFired(event)	a hook fired for that event	Trace
turns({ min, max })	the agent took N turns (multi-turn)	Trace
wrote(path)	a file was created	Trace
paid_judged(rubric, {…})	a model grades the output ≥ min (LLM rubric)	Trace (eval)
cost/latency/tokens({…})	the run stayed under budget	run usage (eval)
blocked() / allowed()	the hook blocked / allowed	runHook result

paid_measureArms({ arms, checks }) scores the same checks per arm (a hook/skill/rule on vs off); compareCheck(report, baseline, arm, i) returns a Welch verdict. It takes stubSkillBodies too. propertyHook({ seed, mutate, decide, invariants }) fuzzes a hook's (event) → decision and shrinks any counterexample.

vitest / jest matchers

The testing API is runner-agnostic; these are an optional convenience. Register by hand:

import { expect } from "vitest"; // or "@jest/globals"
import { vigilesMatchers } from "vigiles";
expect.extend(vigilesMatchers);

…or use the opt-in integration entries, which register the matchers and add their TypeScript types:

// vitest.config.ts →  test: { setupFiles: ["vigiles/vitest"] }
// jest.config.js   →  setupFilesAfterEnv: ["vigiles/jest"]
import "vigiles/vitest"; // …or import once at the top of a test file

	vitest	jest
Register (config)	test: { setupFiles: ["vigiles/vitest"] }	setupFilesAfterEnv: ["vigiles/jest"]
…or per-file	import "vigiles/vitest"	import "vigiles/jest"
Manual (no types)	expect.extend(vigilesMatchers)	expect.extend(vigilesMatchers)

toPass fronts the whole check vocabulary, so reach for it first; the rest are shorthands.

Matcher	Asserts	Over
expect(r).toPass(check)	the check holds (its own message)	any result
expect(r).toPassAll([checks])	every check holds	any result
expect(r).toHaveCreated(path)	a file was created	harness result
expect(r).toBlock()	the hook blocked	runHook result
expect(report).toBeatBaseline(b, a, m)	arm a beats baseline b on metric m	eval report

So toPass(tool("Bash")), toPass(skill("x:y")), toPass(blocked()), toPass(cost({ maxUsd: 0.05 })) all work with one matcher. vitest and jest are optional peer dependencies — only the entry you import pulls one in; the seam is tested under both runners in test/runners/.

measureTriggerRate options

const report = await measureTriggerRate({
  skillsDir: ".claude/skills", // loose skills — auto-packaged. XOR pluginDir.
  pluginDir: "./my-plugin", //    already-packaged plugin. XOR skillsDir.
  prompts: [...], //              recall set (≥ minPrompts, NCD-diverse)
  irrelevantPrompts: [...], //    precision set (→ falsePositiveRate / precision)
  fired: (t) => skillResolved(t, "my-plugin:greet"),
  stubSkillBodies: true, //       stop at SELECTION (frontmatter decides firing)
  fixture: { "path": "contents" }, // seed repo STATE a skill triggers on
  installSet: [otherPluginsOrDirs], // co-install the rest of the harness (release gate)
  trials: 2,
  concurrency: 4, //              parallelize the prompts × trials grid
  minModel: "sonnet", //          fail before spending a token if pointed below
  effort: "low", //               reasoning budget — pin it to reproduce a number
  minPrompts: 10, //              diversity-gate floor
});
assertTriggerRate(report, { min: 0.8, maxFalsePositive: 0.1 });

• skillsDir vs pluginDir — exactly one. skillsDir packages a loose .claude/skills into a throwaway plugin for you.

• stubSkillBodies — triggering is decided by frontmatter before the body loads, so stubbing the body (descriptions kept) lets a run stop at selection instead of executing a multi-step procedure. Don't combine with judged/quality checks — the body is gone.

• Diversity gate — minPrompts (default 10) + an NCD near-duplicate check rejects a too-small / copy-pasted prompt set before spending a token.

• Model — defaults to "sonnet" (the realistic selector; a weaker model under-selects — dogfooded 0.50 on haiku vs 0.90 on sonnet). The minModel floor fails a too-weak run up front. Model lives in the spec, not an env override.

• effort — the reasoning budget the run is pinned to ("low", "high", an integer — whatever your harness build accepts). Set it when you are reproducing or publishing a number: a result measured at one budget is not a result at another, so effort is hashed into both the eval lock and the local cache, and a recorded report is stale for a run at a different budget. It is also an arm in its own right — arms: { cheap: { effort: "low" }, rich: { effort: "high" } } answers "does my harness still hold on the cheaper budget?" through the same significance machinery, exactly as model-as-an-arm does.

  Three things worth knowing before you rely on it:

  It is pinned, not merely passed	The harness also reads CLAUDE_CODE_EFFORT_LEVEL, and that env var outranks the flag. vigiles sets it for the run — and deletes an inherited one when you omit effort — so an exported value in your shell can never silently become the budget a lock records.
  A value the harness rejects fails the run	It does not fail on a bad level; it warns and quietly uses its default. vigiles turns that warning into an error, because a number produced by a configuration nobody asked for is the problem this field exists to avoid.
  Omitting it means the harness default	Not a fixed level: the default is per-model and can move between builds. An effort-less lock is reproducible only modulo that — the same kind of caveat as the harness version.

  Not supported on Codex: no mapping has been measured, so a spec declaring effort there fails loudly rather than recording a budget the run did not use.

• fixture — each run defaults to an empty cwd (faithful for opening-move skills); pass fixture to seed the repo state a skill claims to fire on.

• installSet — by default the skill competes against the other skills in the plugin you point at (honest — selection is competitive); installSet co-installs the rest of the user's harness for a release-gate measurement. report.competitors is the pool size (0 = isolated, an upper bound on recall).

• Comparing models is a harness A/B — set a per-arm model in measureArms / runEval, no separate matrix runner.

Selection-collision matrix (Claude Code only)

🔵 Claude Code only — import from vigiles/claude-code, not the root vigiles surface. This reads which skill the selector chose, and only Claude Code surfaces that (Codex has no skill-selection event). On any other harness it returns available: false rather than a fake pass.

Per-skill trigger-rate asks each skill in isolation. It can't catch the failure that breaks a multi-skill plugin: one skill hijacking a sibling's prompt. measureSelectionMatrix runs each skill's own prompts against the whole installed set and records which skill fired — an N×N matrix whose diagonal is recall and whose off-diagonal mass is collision.

import { measureSelectionMatrix, assertNoCollision } from "vigiles/claude-code";

const report = await measureSelectionMatrix("./my-plugin", {
  // prompts auto-derived from each skill's description (zero setup);
  // pass `prompts` (a { skill: { prompts: [...] } } map) for a curated set.
  trials: 1,
});

// Fail the build when a sibling steals a skill's prompt.
assertNoCollision(report, { maxOffDiagonal: 0.2 }); // per-skill collision ceiling
// or gate the plugin-wide rate: assertNoCollision(report, { maxPluginCollision: 0.1 })

• Zero-setup — omit prompts and they're derived from descriptions (the same generator the audit trigger tier uses). Body-stubbed, so it measures selection cheaply (the skill's procedure never runs).
• assertNoCollision — with no options it demands zero collision; maxOffDiagonal caps each skill's collision rate, maxPluginCollision caps the plugin-wide rate. It throws on a green that tested nothing (unavailable harness / zero runs), never a silent pass.
• It's the behavioral confirmation of the deterministic description-overlap lint rule: the rule says "these two look confusable," the matrix says "they collide X% of the time."

runEval, measure, measureArms

runEval drives the real model N trials × arm and aggregates: mean for numbers, fraction-true for booleans, with std / se and pass^k (did the metric succeed on every trial? — the reliability question "worked every time" ≠ "worked on average"). An arm is a fixture + settings, or a whole plugin / pluginDir.

The measure ctx is a full Trace, so a metric reads the agent's actions (ctx.toolCalls), its final answer (ctx.output), and the filesystem (ctx.file, ctx.sh) — reuse the bare predicates to compute them.

Two questions, two oracles — don't default to A/B for both:

• Absolute — "is this exact skill / output any good?" Testing one skill with no on/off variant: score it directly with measure({ checks: [judged(rubric)] })
  • assertRates({ min }). The oracle promptfoo/DeepEval lead with.
• Relative — "does this change move behaviour vs off?" The lift a change buys (regression gating, proving a gap isn't noise): runEval with off/on arms + assertSignificant.

Cost / caching / concurrency:

• concurrency: N — run N trials at once (default 1). Rate-limit responses back off and retry automatically (rateLimitRetries, retryBackoffMs).
• maxCostUsd: N — stop launching once measured cost crosses the cap; in-flight trials finish and report.aborted is set.
• cache: "readwrite" — record/replay. Each trial's output and post-run filesystem are recorded; a matching re-run replays without calling the model. The key excludes measure, so editing your metric and re-running re-scores for free — the model is re-called only when a model-affecting input changes.

report.arms[a].usage aggregates per arm; report.totalCostUsd sums the run.

LLM-as-judge — grade with a model inside measure (synchronous, shells out via the harness CLI):

import { paid_judge } from "vigiles/eval";

measure: (ctx) => {
  const v = paid_judge({
    output: ctx.file("PLAN.md") ?? "",
    rubric: "1 if the plan lists concrete, ordered steps; else 0.",
  });
  return { quality: v.score, ok: v.pass };
};

Deliberately thin — for datasets, tracing, and dashboards use a dedicated eval platform (Braintrust, DeepEval). vigiles owns the harness A/B, not the judging platform.

Significance & regression gating

Significance. se gives the spread; assertSignificant runs a Welch's t-test over two arms' summary stats and throws unless the arm beats the baseline at alpha (default 0.05) — the noise floor is computed, not hand-fed:

import { assertSignificant, significantlyBeats, compareArms } from "vigiles";

assertSignificant(report, {
  baseline: "vanilla",
  arm: "gated",
  metric: "marked",
});
significantlyBeats(report, "vanilla", "gated", "marked"); // bare predicate
const c = compareArms(report, "vanilla", "gated", "marked");
// → { delta, seDelta, t, df, pValue, significant }  (reads mean/se/n, no raw rows)

For 0/1 metrics this is the t approximation to the two-proportion test — close at eval trial counts. An insignificant gap means raise trials until the noise floor drops below it. assertReliable(report, { arm, metric }) is the stricter bar: the metric succeeded on every trial (pass^k = 1).

Regression gating compares one run against a committed baseline — "jest snapshots for agent behaviour, with a real noise floor":

import {
  writeBaseline,
  readBaseline,
  assertNoRegression,
  diffToJUnit,
  diffReports,
} from "vigiles";

// Record once (commit .vigiles/eval-baseline.json):
writeBaseline(".vigiles/eval-baseline.json", [report]);

// In CI thereafter — throws on a significant regression:
const baseline = readBaseline(".vigiles/eval-baseline.json");
if (baseline) {
  assertNoRegression(report, baseline, { lowerIsBetter: ["cost"] });
  // or emit JUnit: diffToJUnit(diffReports(baseline, [report]))
}

Higher is better by default; list lowerIsBetter metrics (cost/latency) to flip them. A new arm/metric absent from the baseline is skipped (not a regression).

Imports & harness selection

One canonical entry point per layer — import from these:

Layer	Canonical import	What it re-exports
Test your harness	vigiles	every check, assertion, matcher, hook/harness runner and tool stub — everything that costs nothing to run
Measure with a model	vigiles/eval	paid_runEval · paid_measure · paid_measureArms · paid_measureTriggerRate · paid_judge · paid_judged · paid_claudeEvalDriver
Declare an eval file	vigiles	defineEval (free — a description cannot spend; see eval files)
Lint instruction files	vigiles/linting	the spec builders + the compiler

In a *.eval.mjs file the runners below are DECLARED, not called — export default defineEval({ measure: spec }) — because importing a file that calls one spends real money. See eval files describe their eval.

The testing surface splits on COST, not on test tier. Free is the package root; anything that can call a model is vigiles/eval. There is no vigiles/test — the bare package name is the testing surface — and the old vigiles/unit / vigiles/integration / vigiles/e2e / vigiles/testing barrels are gone. Tiers were never a property of the module graph: express them where they belong, in test-file naming and runner config (vitest --project integration), the way @playwright/test does.

Every runtime export on vigiles/eval is also named paid_. The import path warns once, at the top of the file; the name warns at every call site, which is where the money is spent. Same device the package already uses for experimental_ on every call site. ⚠️ The prefix overstates slightly — paid_judged(rubric, { judge: myFn }) runs your function and bills nothing, and the measure* family takes an injectable evalDriver; only the DEFAULT path calls a model. metered_ would be exact but reads a beat slower, and a warning that isn't absorbed at a glance isn't a warning. Types are not prefixed: a type cannot be called, so it cannot bill.

Two consequences worth stating plainly:

• Free is not fast. runHarnessTest spawns a real claude under bubblewrap and can take ~40 seconds. It bills nothing, so it is on the free root, and the import no longer warns you about the wall clock.
• The eval-analysis helpers stay free and unprefixed. assertSignificant, assertNoRegression, diffReports, cost / latency / tokens and friends read a report without running anything, so they are on vigiles. Their argument TYPES (EvalReport, CheckReport, …) are re-exported from both barrels, so you never import vigiles/eval for a type alone.

The remaining entry points are deliberate surfaces, not aliases:

• Authoring — vigiles/spec: the spec builders for .spec.ts files.
• CC-specific transport — scriptModel, loadPlugin, resolveHarness, LoadedPlugin live in vigiles/claude-code, not on the root surface.
• CC-only measurement — measureSelectionMatrix / assertNoCollision live in vigiles/claude-code too. They read which skill the selector chose (Codex has no such event), so they can't sit on the agnostic surface. Everything else measured (paid_measureTriggerRate, paid_measure, paid_runEval) is on vigiles/eval.
• Runner integration — vigiles/vitest / vigiles/jest.
• Harness selection — vigiles/claude-code / vigiles/codex / vigiles/adapter.

Selecting a harness. The runners (runHook, runHarnessTest, runEval) are harness-agnostic; pick the harness with one option, defaulting to Claude Code:

import { runHarnessTest } from "vigiles";

await runHarnessTest(spec); // Claude Code (default)
await runHarnessTest(spec, { adapter: codexAdapter }); // a second harness

codexAdapter comes from vigiles/codex. Nothing in vigiles changes when the harness changes; unused adapters tree-shake out. The CLI can't take an import, so it auto-detects the harness from the repo. For the full adapter/import model and the capability matrix, see docs/harnesses.md.

See also

• Testing your harness — the task-first how-to guide.
• docs/harnesses.md — harness selection + the capability matrix.