mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
e78841174d
* feat: sweep the disaster battery across every hook a repo declares `verifyGuardrail` takes a command string. #211 taught it to honour a hook's `if` condition, but only when the caller passes it — its own comment has to ask. That left "forgot to pass the condition" reachable, and reaching it reproduces the exact false 7/7 #211 existed to kill: a narrow force-push guard certified as stopping `rm -rf /` and `cat ~/.ssh/id_rsa`. `experimental_verifyPluginGuards(dir, opts?)` takes a DIRECTORY instead. It reads the command, event, matcher and condition off the same registration (loadPlugin -> normalizeHooks), so the pairing cannot be made wrongly. Per hook it returns a discriminated union, not a score with nullable fields: measured the battery reached it; results/blocked/allowed/notRun not-applicable another event, or a matcher selecting no battery tool unresolved the command names a variable nothing sets Reading the score of a hook that never ran is a tsc error rather than a 0/7 someone quotes. A repo with no hooks — or none reachable — says so in `notes`, because `hooks: []` on its own reads as a clean bill of health, which is the same false confidence this battery exists to remove. It reports and does not judge: no throwing sibling, because a repo's config never declares which of its hooks is meant to be a guard. `assertBlocksDisasters` stays the gate once you have said so yourself. Harness-agnostic with Claude Code as the default, the runHarnessTest shape — every piece it composes already takes its harness by injection, and a shellHooks:false adapter reports n/a in words rather than an empty pass. Also exports `hookMatcherSelects` from core/hook-matcher.ts so the measured matcher semantics have one home; it fails open in the same direction `decideHookCondition` does, so a wrong answer can only ever say "ran and did not block", never invent a skip. Tested in both directions over a fixture plugin carrying all five shapes, the real vendored davila7 guard, a Codex-shaped repo, and the opencode prototype. Two mutations verified: dropping each hook's own condition brings the 7/7 back (3 assertions red), and ignoring the hook's event turns a Stop hook into a measured 0/7 (4 red). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: render a plugin-guard sweep, without giving an unmeasured hook a number `experimental_verifyPluginGuards` returned a discriminated union and nothing to print it with, so the one use case it exists for — "point the battery at YOUR hooks" — was a six-line fold over that union at every call site. That is a worse code block than the two-liner it replaces, and `cohesive-feature-delivery` row 9 (output parity) could not be claimed. `experimental_formatPluginGuardReport(report)` renders it as terminal text, and its contract is that the union's honesty survives becoming a string: measured prints `blocks n/7` and one row per battery event not-applicable prints its REASON under a ⊘ heading, and no count unresolved likewise notes printed FIRST and in full A rendered `0/7` beside a hook the battery never reached would reintroduce, one layer above the type system, exactly the false confidence the union was built to make unrepresentable. So it is asserted directly and in both directions: for a sweep that measured nothing the output must match no `n/m` at all, and must carry the sentence saying nothing was measured. Mutating the formatter to print `blocks 0/7` for a not-applicable hook fails that assertion and three others. MANY HOOKS stay readable by grouping the unmeasured half BY REASON — a repo with thirty hooks usually has two or three — and naming at most three hooks per reason before counting the rest. That is the only lossy step, and it is confined to the half whose payload is the reason rather than the hook; the measured half is never collapsed, since those are the hooks you came for. The per-event rows come from a shared `guardrailRow`, so this renderer and `formatGuardrailReport` cannot drift into two vocabularies for the same three outcomes; `formatGuardrailReport`'s output is byte-identical. The prefix is on the name because the report SHAPE is documented as likely to move, and a stable formatter pinned to an unstable shape is a lie. It comes off with the same change that takes it off the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: four ways the guard sweep answered for a hook it could not run An automated review of #212 found four, and all four move the same way: toward measuring less. Each has a test that failed before the fix. 1. An UNCOMPILABLE matcher scored 7/7. `hookMatcherSelects` failed open on every uncertain input, reasoning that imprecision can only add a run the harness makes. That reasoning holds where the harness fails open, and this is where it fails closed: the same module already reports `Bash(` as invalid-regex, "the harness can't compile it, so the hook never fires". So the sweep ran the battery against a hook the harness never spawns, and an unconditional-deny body came back blocking all seven disasters — the exact false 7/7 #211 removed, arriving through the matcher instead of the condition. The predicate is now `hookMatcherReach`, three-valued, and an uncompilable matcher is `not-applicable` with the matcher named. 2. Codex matcher semantics were ignored. `HookProtocol.matcherStyle` already says all Codex matchers are regexes; the helper applied Claude Code's literal-equality rule globally, so `ash` — which regex-matches Bash — was classified n/a and never tested. The style is threaded through. 3. Variables the command sets for itself read as unresolved. A `$NAME` regex over shell called two self-contained commands unresolvable: one that assigns the name before expanding it, and one written inside single quotes, where the shell expands nothing. Both are real guards that went unmeasured for a reason untrue of them. Reads now come from the shell's own parser (core/shell-vars.ts, the mvdan-sh parse bash-effects uses); a parse failure falls back to the regex and says so. 4. Adapter-declared event variables were not supplied. `eventEnvVars` exists to say what a synthesized hook event carries, and the sweep synthesizes one, so a Codex hook reading `$hook_event_name` or `$cwd` was unresolvable for lack of a variable the harness would have set. Now populated from the declaration. Deliberately NOT populated: `permission_mode`, `session_id`, `model` — the sweep does not know those, and a guard branching on an invented permission mode is a measurement of a program we made up. The direction is the same in the ambiguous places: refuse to score rather than score optimistically. `*` and `**` on a regex-matcher harness come back uncompilable rather than assumed to be special-cased; an expansion with a default counts as a dependency; a parse failure over-reports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: five ways the guard sweep answered for a hook it never ran Two came from pointing the instrument at a real OSS corpus; three more from an automated review of #212. All five are one class — we run a DIFFERENT program than the harness runs, and record a score anyway — and all five move the same way: refuse rather than answer. Each has a test that failed first. 1. A hook whose SCRIPT IS MISSING scored 7/7. `python3 <missing>.py` exits 2, and 2 is Claude Code's deny code, so a guard that does not exist was certified as stopping every disaster in the battery. Worse than the uncompilable-matcher door fixed last round, which at least needs a malformed config: a relative script path is the commonest hook shape in the wild, and the campaign hit this three times in one run. THE EXIT CODE CANNOT DECIDE IT, and this repo already knew: run-script.ts says so in prose for its own coverage attribution ("`sh <missing>` exits 2 ... indistinguishable from a gate legitimately denying, so it cannot be encoded"). The fact was written down one module away and the sweep scored on it regardless. So the question is asked in two halves that no single signal covers. BEFORE the run, core/command-files.ts names the files the command hands a program to execute, and an absent one is `unresolved` -- no score at all, which is the assertion the mutation targets. AFTER it, a shell exit of 126/127 means the program never launched (missing interpreter, no execute bit, bad shebang); that is the shell's own code, not a language's convention and not a guess about stderr text, and `shellNeverLaunched` is now shared with run-script.ts rather than copied. The pre-flight narrowing is MEASURED, not taste. Over davila7's 107 real hook registrations, "any path-shaped operand" hits 31 with 8 wrong -- files a hook writes or later reads, plus `echo N/A`, which contains a slash -- while "interpreter head OR script extension" hits 23 with 0 wrong. 2. `**` was in MATCH_ALL on no evidence, and that direction MANUFACTURES AN ACCUSATION: a guard registered under it came back "allows 0/7", reported as letting seven disasters through, about a hook the harness never spawned -- the exact mirror of scoring one that could not start. Measured here against claude 2.1.263 with a marker file as the oracle: `*`, `.*` and `""` fire on a Bash call and `**` does not, with `.*` as the in-run control and a hand-run `*` swap ruling out "only the first hook per event fires". Pinned by src/hook-matcher-delivery.test.ts, the shape subagent-delivery already uses: prose cannot notice that someone else's product moved. 3. Variables were checked against `process.env`, which a CONFINED run does not have. `trusted: false` starts from --clearenv and restores only HOME/TMPDIR/PATH plus the caller's env, so a foreign hook reading an ambient variable was measured with it missing. The availability set now follows the execution MODE, decided by the same `decideSandbox` policy the runner uses, and `hookEnvironment` returns three fields because they answer three different questions: what is passed, what the hook will find set, and the subset whose value is known here (a confined HOME points at a sandbox temp dir, so it is present without being resolvable). 4. `layout.projectRootTokens` were never supplied, so `$CLAUDE_PROJECT_DIR` -- what an ordinary project hook reads, including this repo's own -- read as unresolvable while the harness supplies exactly the root being swept. 5. Assignment subtraction ignored execution order, and this one is worth naming plainly: the parser rewrite an hour earlier removed two ways to be wrong and added a third, in the worse direction. Subtracting every assigned name globally excused a read the assignment never reached. Measured against /bin/sh with the name exported first, `echo "$X"; X=1` and `FOO=1 sh -c "echo $FOO"` both print the AMBIENT value -- so both really do read the environment. Two existing tests asserted the opposite; they were the defect written down, and are corrected with the measurement in place of the argument. The rule is now dominance, not membership, and a prefix, subshell or function-body assignment excuses nothing. Where a claim about someone else's product is load-bearing it is pinned by a test against the real binary, not restated: #2 was re-measured here rather than taken from the report that found it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: three more doors to a score for a hook that never ran Review found five; four were real and one measurement had to be redone before the fifth could be. All four are the same class the branch exists to close: `sh` exits 2 both for a hook that legitimately denied and for a hook that never ran, so anything reaching the spawn that should not have comes back as a perfect 7/7. SYNTAX ERROR SCORED AS A BLOCK. `commandFileRefs` returns `{ parsed: false, refs: [] }` for a command the shell parser rejects, and the caller read only `refs` — so an unparseable command looked exactly like a clean one. Measured on the build before this commit: hook `echo "unterminated` → MEASURED blocks=7/7 exits=2,2,2,2,2,2,2 The parse verdict is now decided first, ahead of the variable check too: `shellVarReads` falls back to a regex when the parse fails, so it still named `$FOO` in `echo "$FOO` and the sweep blamed an unset variable for a syntax error, telling the caller to pass `env` for a command no `env` could have saved. PRE-FLIGHT AND RUNNER LOOKED IN DIFFERENT DIRECTORIES. `sandboxedSpawn` chdirs into a `work/` directory it has just created and left empty when no `cwd` is given, while the missing-file check resolved against `process.cwd()`. Same guard, same command, two directories: project dir → exit 0 (the guard's real verdict: allow) fresh empty dir → exit 2 (cannot open its script = DENY = a block) So a guard that allows was reported as blocking, out of nothing but the mismatch. `effectiveCwd` now derives the run's directory from the same policy the runner uses; a relative ref under confinement-without-`cwd` is unresolvable by construction, and an absolute one is still tested on disk because `/` stays ro-bound in there. THE PROJECT IS NOT THE PLUGIN. `$CLAUDE_PROJECT_DIR` and `$CLAUDE_PROJECT` were bound to the swept dir, which is right only while the two coincide. Sweeping an installed plugin against a host project — `dir` the plugin, `cwd` the project — pointed an ordinary project hook at the wrong tree: measured before this commit, a hook reading `$CLAUDE_PROJECT_DIR` came back `unresolved — not on disk here` for a script sitting in the project all along. The plugin-root token still follows `dir`; the project tokens and Codex's `$cwd` follow the run. SOURCE ORDER IS NOT CONTROL FLOW. An assignment excused a later read whenever its offset came first, so `false && MODE=safe; [ "$MODE" = safe ]` suppressed `MODE` although the shell goes on reading the environment — and under confinement the sweep clears it, so it would have measured the other arm of the guard. The denylist of non-persisting assignments is now an allowlist of unconditional ones (a top-level `NAME=value`, or an `export`/`readonly`/`declare`), which makes the unlisted construct default to excusing nothing rather than to excusing everything. The old rule was wrong on four of five measured shapes: `&&`, `if`, `&`, and a pipeline. AN EXTENSION IS NOT A LICENCE TO EXECUTE. `rm -f /tmp/stale.sh` — an ordinary cleanup hook whose file is meant to be absent — was reported as running a missing script, because the extension test ran independently of the head. It is evidence about the FILE, not about the command, so it now requires a head that could execute: a known data-only head silences it, an unknown head still carries it. Re-measured before narrowing, since narrowing that loses a true positive would be worse than the false positive it fixes. The pinned corpus behind the original table (107 registrations, 23 hits, 0 wrong) has moved, so this is a second sample rather than a reproduction: today's davila7/claude-code-templates gives 134 registrations / 120 distinct commands and 5 hits, all true positives reached by an interpreter head or a head that is itself a path — 5 before the clause, 5 after. Both numbers are recorded in the module header, labelled as different corpora. Each fix is pinned by a test proven with a mutation: dropping the parse gate, ignoring confinement in the pre-flight, collapsing project onto plugin, restoring source-order dominance, and removing the head check each fail on their own assertion, with the patch grepped in source and dist. Two existing tests caught the allowlist being too narrow while it was being written — `export NAME=value` is a DeclClause, not a CallExpr — and that gap is now handled and covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: three hand-kept copies of a fact that already had one source Review's second round left five findings. Three are the same defect wearing different clothes — a fact with ONE authoritative source, restated by hand somewhere else, where the copy fell behind — so they are closed as classes rather than patched as places. Two are their own bugs. DERIVE, DON'T RESTATE CONFINEMENT. The sweep kept its own copy of the runner's mode expression and did not know `recordEgress` confines. So a `recordEgress` sweep pre-flighted a relative script against the host's cwd and env, accepted it, and handed it to a run that starts in a fresh empty directory with a cleared environment — where `sh` cannot open the script and exits 2, this harness's DENY code, scored as a block. `routeScriptRun` is now the one decision: `runScriptWith` dispatches on it and owns no policy of its own, and the pre-flight asks it. There is no list here left to fall behind — a future option that selects confinement is added once, in the runner, and both readers inherit it. WRAPPERS. `commandFileRefs` read `args[0]` as the head, so `env python3 guard` named no file at all and the same exit-2 door opened. `stripWrappers` already owns the wrapper list for the effect classifier, so it is exported and ASKED, not copied: env / command / timeout / sudo / nice / xargs / nohup, recursively, skipping each wrapper's own options and `NAME=value` words. NOT fixed, and said out loud in the code and pinned by a test so it is recorded rather than rediscovered: `exec` is a pass-through that list does not carry, and `stripWrappers` has its own recorded remainder (an escaped head behind a wrapper, `sudo g\it push`). Both belong in that module, where the safety matcher reads the same list and a change deserves the measuring a matcher change deserves — not in a private copy here. EXECUTION CWD. The pre-flight tested files against the CALLER's directory while the project-root variables named the swept root: two trees, one run. A guard on disk exactly where its config says was reported `unresolved`; and where the caller happened to hold the same relative path, the sweep ran and SCORED the caller's file under the swept repo's name. `effectiveCwd(opts, root)` answers once, defaults to the swept repository, and the runner is handed that same answer. An explicit `cwd` still overrides — the installed-plugin case, where hooks are READ from the plugin and RUN against a project elsewhere. TWO ORDERING BUGS, THEIR OWN A read inside an assignment's own initializer is never dominated by that assignment: the shell expands the RHS and only then binds the name. Keyed on the assignment's START offset it was, because the walk is preorder — the `Assign` was recorded before its own `ParamExp` was visited. So `MODE="$MODE"; [ "$MODE" = strict ]` reported NO dependency, and under confinement the sweep cleared an ambient value without reporting it unresolved and scored whichever branch the empty value took. Keyed on the END, the containment falls out of the arithmetic instead of needing a special case. `prompt` / `http` / `mcp_tool` / `agent` are real hook actions. `normalizeHooks` drops them correctly — nothing that spawns a shell can drive one — but the sweep read the empty list as "No hooks are declared", which is an accusation about the repository rather than a limit of the tier, written in the one sentence this report exists to get right. They are carried now as `report.unmeasurable` and reported declared-and-NOT-measured in the notes, the census and the rendering; the mixed case is counted too, because a reader who sees six measured hooks cannot otherwise know a seventh went unexamined. MUTATIONS Six, each reverting a fix to its old shape, each failing on its own assertion. Two passed GREEN first, and both were findings about the tests: - `routeScriptRun`'s unit tests pinned the ROUTER, but nothing pinned that the SWEEP asks it, so restoring the hand-kept copy went green. Added. - the cwd fixture's guard DENIED — and `sh missing.sh` also exits 2, so a correct measurement and a total failure to find the file both read 7/7. The fixture now ALLOWS, making the right answer 0/7: the one thing the false 7/7 cannot be confused with. That is this branch's own bug class, found inside its own test. No published number moves. `WRAPPER_HEADS` is untouched (the bash-effects diff is a doc comment and an `export` keyword): 7 seeds, 136 alternate spellings and the 2/7 -> 7/7 dogfood all unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: name the one class five review findings belong to `blocked` is decided by exit code 2, and an interpreter that cannot start its script also exits 2 — so a guard that never ran and a guard that denied are the same observation. The preflight that resolves a hook's script is a proxy for "did it run", and review has defeated that proxy five times. Two are fixed on this branch; three are not, and patching them individually would buy one spelling each while leaving the class open — extracting a script path from an arbitrary shell command is the same undecidable problem bash-effects.ts already documents for itself. Names the three open spellings and points at the fix that closes the class: a control probe, one benign command the guard must allow, run beside the battery. Blocked too means the program is failing to start, not blocking, and the hook is unmeasurable rather than scored. Comment only; no behaviour change. Follow-up: #213 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: a hook rejected by its own condition was scored, not excused Matcher `Bash` with `if: "Bash(terraform apply*)"` reaches the battery by matcher and matches none of it by condition. Every entry in `ran` came back `ran: false`, and the hook was still returned `measured` — so the renderer printed `blocks 0/7` and `sweepNotes`, seeing a measured hook, stayed silent. A guard the harness never spawned, reported as a guard that blocks nothing. That is the same false verdict as the other three non-verdicts and in the unsafe direction, so it gets the same shape: `not-applicable` with its own reason naming the condition, and no score anywhere. The module header said THREE things are not a verdict; the condition is the fourth and it now says so. Decided AFTER the run rather than before, deliberately: `ran: false` IS the runner's own condition verdict, so this reads the decision the harness would make instead of re-deciding it beside `decideHookCondition`. It costs nothing, because a rejected condition never spawns a process either. `ran` is non-empty (an empty `reachable` returned earlier) and a launch failure already returned `unresolved`, so every `ran: false` left is a condition rejection. The sole-hook path falls out: a repo whose only hook was that one now measures nothing, and `sweepNotes` reaches its "none of them reachable / Nothing was measured" branch instead of rendering an empty score. Four tests, and the boundary is pinned in both directions — a condition matching PART of the battery must still score. Mutations: removing the classification reddens three assertions (`expected 'measured' to be 'not-applicable'`); widening `every` to `some` reddens seven, including the partial-condition guard and the vendored davila7 hooks. Also records two review findings as KNOWN REMAINDER (#213) rather than patching them. Both fail toward UNDER-reporting — a real guard reads `unresolved`, nothing is falsely scored — and both were reproduced: an unexpanded `~` probes `<cwd>/~/.claude/…` so a guard present at $HOME is called missing, and `$PWD` is absent from the confined name set though /bin/sh sets it after bubblewrap's --chdir (same command: 7/7 unconfined, `unresolved` confined). Fixing either by resolving harder is the reflex the block already argues against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
38 KiB
38 KiB
API Report File for "vigiles"
Do not edit this file. It is a report generated by API Extractor.
// @public
export interface AgentRunArgs {
// (undocumented)
readonly cwd: string;
readonly effort?: string | number;
readonly env?: Record<string, string>;
// (undocumented)
readonly hasSettings: boolean;
// (undocumented)
readonly model: string;
// (undocumented)
readonly pluginDir: string | undefined;
readonly replaceEnv?: boolean;
// (undocumented)
readonly task: string;
// (undocumented)
readonly timeoutMs: number;
// (undocumented)
readonly tools: readonly string[];
}
// @public
export type AgentRunner = (args: AgentRunArgs) => Promise<RunOut>;
// @public
export function allowed(): Check<HookRunResult>;
// @public
export type ArgMatcher = Record<string, string | number | boolean | RegExp>;
// @public (undocumented)
export interface ArmReport {
readonly metrics: Record<string, number>;
// (undocumented)
readonly runs: number;
readonly stats: Record<string, MetricStat>;
readonly usage: ArmUsage;
}
// @public
export interface ArmsCheckReport {
// (undocumented)
readonly arms: Record<string, CheckReport>;
}
// @public
export interface ArmsMeasureSpec {
// (undocumented)
readonly allowedTools?: readonly string[];
readonly arms: Record<string, EvalArm>;
// (undocumented)
readonly checks: readonly Check<RunContext>[];
readonly effort?: string | number;
// (undocumented)
readonly fixture?: Record<string, string>;
// (undocumented)
readonly model?: string;
// (undocumented)
readonly spacingSec?: number;
readonly stubSkillBodies?: boolean;
// (undocumented)
readonly task: string;
// (undocumented)
readonly timeoutMs?: number;
// (undocumented)
readonly trials?: number;
}
// @public
export interface ArmUsage {
// (undocumented)
readonly meanCostUsd: number;
// (undocumented)
readonly meanDurationMs: number;
// (undocumented)
readonly totalCacheCreationTokens: number;
// (undocumented)
readonly totalCacheReadTokens: number;
// (undocumented)
readonly totalCostUsd: number;
// (undocumented)
readonly totalInputTokens: number;
// (undocumented)
readonly totalOutputTokens: number;
}
// @public
export function assertAgentErr(output: string, contract?: OutputContract): Record<string, unknown>;
// @public
export function assertAgentOk(output: string, contract?: OutputContract): Record<string, unknown>;
// @public
export function assertAgentResult(output: string, predicate: (r: ParsedAgentResult) => boolean, contract?: OutputContract): void;
// @public
export function assertBlocksDisasters(hookCommand: string, opts?: VerifyGuardrailOptions): void;
// @public
export function assertChecks<T>(target: T, checks: readonly Check<T>[]): void;
// @public
export function assertCreated(r: HarnessTestResult, path: string): void;
// @public
export function assertEgressOnly(r: HasEgress, allowed: ReadonlyArray<string | RegExp>): void;
// @public
export function assertHookAllowed(r: HookRunResult): void;
// @public
export function assertHookAllows(hook: AnyHook, event: RawHookEvent): void;
// @public
export function assertHookBlocked(r: HookRunResult): void;
// @public
export function assertHookDenies(hook: AnyHook, event: RawHookEvent): void;
// @public
export function assertHookFired(trace: Trace, name: string | RegExp, opts?: {
blocked?: boolean;
}): void;
// @public
export function assertHookNotices(hook: AnyHook, event: RawHookEvent, matcher?: string | RegExp): void;
// @public
export function assertHookSilent(hook: AnyHook, event: RawHookEvent): void;
// @public
export function assertImproves(report: EvalReport, opts: {
baseline: string;
arm: string;
metric: string;
by?: number;
significant?: boolean;
alpha?: number;
}): void;
// @public
export function assertNoEgress(r: HasEgress): void;
// @public
export function assertNoRegression(current: EvalReport | readonly EvalReport[], baseline: BaselineFile, opts?: DiffOptions): void;
// @public
export function assertNotCreated(r: HarnessTestResult, path: string): void;
// @public
export function assertNoWrite(r: HasWrites, pattern: string | RegExp): void;
// @public
export function assertOutputContains(trace: Trace, needle: string | RegExp): void;
// @public
export function assertPromptDiversity(prompts: readonly string[], opts?: {
minPrompts?: number;
minDistance?: number;
label?: string;
}): void;
// @public
export function assertRates(report: CheckReport, opts: {
min: number;
per?: Readonly<Record<string, number>>;
}): void;
// @public
export function assertReliable(report: EvalReport, opts: {
arm: string;
metric: string;
}): void;
// @public
export function assertRequestContains(trace: Trace, needle: string | RegExp): void;
// @public
export function assertServedTurns(r: HarnessTestResult, n: number): void;
// @public
export function assertSignificant(report: EvalReport, opts: {
baseline: string;
arm: string;
metric: string;
alpha?: number;
}): void;
// @public
export function assertSkillResolved(trace: Trace, skill: string): void;
// @public
export function assertToolCalls(trace: Trace, predicate: (calls: readonly ToolCall[]) => boolean, message?: string): void;
// @public
export function assertToolCount(trace: Trace, name: string | RegExp, bounds: {
min?: number;
max?: number;
exactly?: number;
}): void;
// @public
export function assertToolNotUsed(trace: Trace, name: string | RegExp): void;
// @public
export function assertToolSequence(trace: Trace, names: ReadonlyArray<string | RegExp>): void;
// @public
export function assertToolUsed(trace: Trace, name: string | RegExp): void;
// @public
export function assertToolUsedWith(trace: Trace, name: string | RegExp, inputMatcher: (input: unknown) => boolean, message?: string): void;
// @public
export function assertTriggerRate(report: TriggerRateReport, opts: {
min?: number;
maxFalsePositive?: number;
minPrecision?: number;
}): void;
// @public
export function assertWroteOnly(r: HasWrites, allowed: ReadonlyArray<string | RegExp>): void;
// @public (undocumented)
export interface BaselineDiff {
readonly entries: readonly MetricDiff[];
readonly improvements: readonly MetricDiff[];
readonly passed: boolean;
readonly regressions: readonly MetricDiff[];
}
// @public
export interface BaselineFile {
readonly recordedAt: string;
readonly reports: Record<string, EvalReport>;
// (undocumented)
readonly version: number;
}
// @public
export function blocked(): Check<HookRunResult>;
// @public
export type BlockMechanism = "exit-code" | "deny-decision" | "halt-field";
// @public
export function cacheTokens(opts: {
maxCreation?: number;
maxRead?: number;
}): Check<UsageTrace>;
// @public
export interface Check<T> {
eval(target: T): CheckResult;
readonly kind: string;
toJSON(): CheckJSON;
}
// @public
export interface CheckJSON {
// (undocumented)
readonly [field: string]: unknown;
// (undocumented)
readonly kind: string;
}
// @public
export function checkPromptDiversity(prompts: readonly string[], opts?: {
minPrompts?: number;
minDistance?: number;
label?: string;
}): PromptDiversityIssue[];
// @public
export interface CheckRate {
// (undocumented)
readonly check: CheckJSON;
readonly n: number;
readonly passK: number;
readonly rate: number;
readonly se: number;
}
// @public (undocumented)
export interface CheckReport {
// (undocumented)
readonly n: number;
// (undocumented)
readonly perCheck: readonly CheckRate[];
readonly usage: ArmUsage;
}
// @public
export function checkReportToJUnit(report: CheckReport, opts?: {
min?: number;
per?: Readonly<Record<string, number>>;
name?: string;
}): string;
// @public
export interface CheckResult {
readonly message: string;
readonly pass: boolean;
readonly score: number;
}
// @public
export function commandsIn(md: string, matching: RegExp): DocCommand[];
// @public
export function compareArms(report: EvalReport, baseline: string, arm: string, metric: string, alpha?: number): Comparison | null;
// @public
export function compareContainment(weak: readonly ContainmentInput[], strong: readonly ContainmentInput[]): ContainmentVerdict;
// @public
export interface Comparison {
readonly delta: number;
readonly df: number;
readonly pValue: number;
readonly seDelta: number;
readonly significant: boolean;
readonly t: number;
}
// @public
export interface ContainerRuntime {
available(): boolean | Promise<boolean>;
readonly name: string;
start(name: string, spec: ServiceSpec): Promise<ServiceHandle>;
stop(handle: ServiceHandle): Promise<void>;
}
// @public
export interface ContainmentInput {
// (undocumented)
readonly perPrompt: readonly {
readonly prompt: string;
readonly rate: number;
}[];
readonly subject: string;
}
// @public (undocumented)
export interface ContainmentVerdict {
readonly both: readonly string[];
readonly compared: number;
readonly holds: boolean;
readonly neither: readonly string[];
readonly strongOnly: readonly string[];
readonly weakOnly: readonly string[];
}
// @public
export function cost(opts: {
maxUsd: number;
}): Check<UsageTrace>;
// @public
export function decideHook(exitCode: number, json: HookOutput | null, protocol?: HookProtocol): {
blocked: boolean;
decision: HookRunResult["decision"];
haltsTurn: boolean;
blockedBy: readonly BlockMechanism[];
};
// @public
export function decideSandbox(opts: {
trusted: boolean;
mode: SandboxMode;
available: boolean;
}): SandboxDecision;
// @public
export function defineEval<K extends EvalKind>(def: EvalDefinitionInput<K>): EvalDefinition<K>;
// @public
export function didNotWrite(path: string): Check<Trace>;
// @public (undocumented)
export interface DiffOptions {
readonly alpha?: number;
readonly lowerIsBetter?: readonly string[];
}
// @public
export function diffReports(baseline: BaselineFile, current: readonly EvalReport[], opts?: DiffOptions): BaselineDiff;
// @public
export type DiffStatus = "regressed" | "improved" | "unchanged";
// @public
export function diffToJUnit(diff: BaselineDiff): string;
// @public
export const DISASTER_CATALOG: readonly DisasterEvent[];
// @public
export type DisasterCategory = "destructive-git" | "destructive-fs" | "bypass-verification" | "secret-exfiltration" | "remote-code";
// @public
export interface DisasterEvent {
readonly category: DisasterCategory;
readonly id: string;
readonly input: Record<string, unknown>;
readonly label: string;
readonly tool: string;
}
// @public
export interface DocCommand {
readonly line: number;
readonly text: string;
}
// @public
export type DockerExec = (args: readonly string[]) => {
readonly stdout: string;
readonly stderr: string;
readonly code: number;
};
// @public
export type Duration = `${number}${"s" | "m" | "h" | "d"}`;
// @public
export function egressHosts(r: HasEgress): string[];
// @public (undocumented)
export function egressRoutes(): boolean;
// @public
export type EmitFieldSchema = {
readonly type: "string";
} | {
readonly type: "number";
} | {
readonly type: "boolean";
} | {
readonly type: "array";
readonly items: {
readonly type: "string";
};
} | {
readonly type: "string";
readonly enum: readonly string[];
};
// @public
export interface EmitObjectSchema {
// (undocumented)
readonly additionalProperties: false;
// (undocumented)
readonly properties: Readonly<Record<string, EmitPropertySchema>>;
// (undocumented)
readonly required: readonly string[];
// (undocumented)
readonly type: "object";
}
// @public
export type EmitPropertySchema = EmitFieldSchema | EmitTrackSchema | EmitObjectSchema;
// @public
export interface EmitToolDefinition {
// (undocumented)
readonly description: string;
// (undocumented)
readonly inputSchema: EmitObjectSchema;
// (undocumented)
readonly name: string;
}
// @public
export interface EmitTrackSchema {
// (undocumented)
readonly enum: readonly ["ok", "err"];
// (undocumented)
readonly type: "string";
}
// @public
export interface EvalArm {
readonly effort?: string | number;
readonly files?: Record<string, string>;
readonly interceptTools?: readonly ToolIntercept[];
readonly model?: string;
readonly plugin?: string;
readonly pluginDir?: string;
readonly settings?: unknown;
}
// @public
export function evalChecks<T>(target: T, checks: readonly Check<T>[]): CheckResult[];
// @public
export type EvalDefinition<K extends EvalKind = EvalKind> = EvalDefinitionInput<K> & {
readonly [EVAL_DEFINITION]: true;
};
// @public
export type EvalDefinitionInput<K extends EvalKind> = Pick<EvalMeasurements, K> & EvalHooks<K>;
// @public
export interface EvalDriver {
readonly experimental?: string;
readonly harness?: string;
// (undocumented)
readonly parse: ModelOutputParser;
// (undocumented)
readonly runError?: (out: RunOut) => string | null;
// (undocumented)
readonly runner: AgentRunner;
}
// @public
export interface EvalHooks<K extends EvalKind> {
readonly assert?: (report: EvalReports[K]) => void | Promise<void>;
readonly evalDriver?: EvalDriver;
readonly skipIf?: () => string | false | undefined | null;
}
// @public
export type EvalKind = keyof EvalMeasurements;
// @public
export interface EvalMeasurements {
readonly measure?: MeasureSpec;
readonly measureArms?: ArmsMeasureSpec;
readonly measureSelectionMatrix?: SelectionMatrixSpec;
readonly measureTriggerRate?: TriggerRateSpec;
readonly runEval?: EvalSpec<Metrics>;
}
// @public (undocumented)
export interface EvalReport {
readonly aborted: boolean;
// (undocumented)
readonly arms: Record<string, ArmReport>;
// (undocumented)
readonly name: string;
readonly totalCostUsd: number;
// (undocumented)
readonly trials: number;
}
// @public
export interface EvalReports {
// (undocumented)
readonly measure: CheckReport;
// (undocumented)
readonly measureArms: ArmsCheckReport;
// (undocumented)
readonly measureSelectionMatrix: SelectionReport;
// (undocumented)
readonly measureTriggerRate: TriggerRateReport;
// (undocumented)
readonly runEval: EvalReport;
}
// @public (undocumented)
export interface EvalSpec<M extends Metrics> {
readonly allowedTools?: readonly string[];
readonly arms: Record<string, EvalArm>;
readonly cache?: CacheMode;
readonly cacheDir?: string;
readonly concurrency?: number;
readonly effort?: string | number;
readonly ephemeralEnv?: boolean;
readonly fixture?: Record<string, string>;
readonly lock?: EvalLockOptions;
readonly maxCostUsd?: number;
readonly measure: (ctx: RunContext) => M;
readonly model?: string;
// (undocumented)
readonly name?: string;
readonly rateLimitRetries?: number;
readonly retryBackoffMs?: number;
readonly spacingSec?: number;
readonly stubs?: readonly ToolStub[];
readonly task: string;
readonly timeoutMs?: number;
readonly trials?: number;
}
// @public
export interface EvalUsage {
readonly cacheCreationTokens: number;
readonly cacheReadTokens: number;
readonly costUsd: number;
readonly durationMs: number;
readonly inputTokens: number;
// (undocumented)
readonly outputTokens: number;
}
// @public
export function experimental_alternateSpellings(events: readonly DisasterEvent[]): readonly DisasterEvent[];
// @public
export function experimental_assertEmittedOk(toolCalls: readonly ToolCall[], contract: OutputContract, options?: {
readonly name?: string;
}): Record<string, unknown>;
// @public
export const experimental_dockerRuntime: ContainerRuntime;
// @public
export function experimental_emitTool(contract: OutputContract, options?: {
readonly name?: string;
}): ExperimentalEmitTool;
// @public
export function experimental_formatPluginGuardReport(report: PluginGuardReport): string;
// @public
export function experimental_hookState(hookFile: string, opts?: {
readonly cwd?: string;
}): HookStateHandle;
// @public
export function experimental_makeDockerRuntime(deps?: {
exec?: DockerExec;
netProbe?: NetProbe;
sleep?: (ms: number) => Promise<void>;
readyTimeoutMs?: number;
}): ContainerRuntime;
// @public
export function experimental_parseEmitted(toolCalls: readonly ToolCall[], contract: OutputContract, options?: {
readonly name?: string;
}): ParsedAgentResult;
// @public
export function experimental_startServices(services: Readonly<Record<string, ServiceSpec>>, runtime: ContainerRuntime): Promise<ServiceSession>;
// @public
export function experimental_verifyPluginGuards(dir: string, opts?: VerifyPluginGuardsOptions): PluginGuardReport;
// @public
export function experimental_withServices<T>(services: Readonly<Record<string, ServiceSpec>>, runtime: ContainerRuntime, fn: (session: ServiceSession) => Promise<T>): Promise<T>;
// @public
export interface ExperimentalEmitTool {
readonly instruction: string;
readonly tool: EmitToolDefinition;
}
// @public
export interface FileToolEventOptions {
readonly event?: string;
readonly extra?: Readonly<Record<string, unknown>>;
readonly input?: Readonly<Record<string, unknown>>;
readonly root?: string;
readonly tool?: string;
}
// @public
export function fileToolEvents(path: string, opts?: FileToolEventOptions): readonly [HookInput, HookInput];
// @public
export function formatBaselineDiff(diff: BaselineDiff): string;
// @public
export function formatCheckReport(report: CheckReport): string;
// @public
export function formatContainment(v: ContainmentVerdict): string;
// @public
export function formatEvalReport(report: EvalReport): string;
// @public
export function formatGuardrailReport(hookCommand: string, results: readonly GuardrailResult[]): string;
// @public
export function formatTriggerRateReport(report: TriggerRateReport): string;
// @public
export interface GuardrailResult {
readonly blocked: boolean;
// (undocumented)
readonly event: DisasterEvent;
readonly exitCode: number;
readonly ran: boolean;
readonly reason: string;
}
// @public
export interface HarnessTestDriver {
available(): boolean;
buildArgs(ctx: HarnessDriverContext): readonly string[];
parseRun(stdout: string): ParsedRun;
readonly runtime: HarnessRuntime;
startMock(script: readonly ModelTurn[]): Promise<HarnessMockHandle>;
}
// @public (undocumented)
export interface HarnessTestResult extends Trace {
cleanup(): void;
readonly cwd: string;
// (undocumented)
readonly exitCode: number;
readonly stderr: string;
// (undocumented)
readonly stdout: string;
readonly turns: number;
}
// @public (undocumented)
export interface HarnessTestSpec {
readonly allowedTools?: readonly string[];
readonly files?: Record<string, string>;
readonly model: readonly ModelTurn[];
readonly plugin?: string;
readonly pluginDir?: string;
readonly prompt?: string;
readonly sandbox?: SandboxMode;
readonly settings?: unknown;
readonly timeoutMs?: number;
readonly transcript?: boolean;
}
// @public
export function hookBlocked(trace: Trace, name: string | RegExp): boolean;
// @public
export interface HookFire {
readonly blocked: boolean;
readonly event: string;
readonly exitCode: number | undefined;
readonly name: string;
readonly output: string;
}
// @public
export function hookFired(event: string): Check<Trace>;
// @public
export interface HookInput {
readonly [k: string]: unknown;
readonly hook_event_name?: string;
readonly prompt?: string;
readonly source?: string;
readonly stop_hook_active?: boolean;
// (undocumented)
readonly tool_input?: unknown;
readonly tool_name?: string;
// (undocumented)
readonly tool_response?: unknown;
}
// @public
export interface HookOutput {
// (undocumented)
readonly [k: string]: unknown;
// (undocumented)
readonly continue?: boolean;
// (undocumented)
readonly decision?: "approve" | "block";
// (undocumented)
readonly hookSpecificOutput?: {
readonly hookEventName?: string;
readonly permissionDecision?: "allow" | "deny" | "ask";
readonly permissionDecisionReason?: string;
readonly additionalContext?: string;
};
// (undocumented)
readonly reason?: string;
// (undocumented)
readonly stopReason?: string;
// (undocumented)
readonly suppressOutput?: boolean;
// (undocumented)
readonly systemMessage?: string;
}
// @public
export interface HookPropertyResult<E> {
readonly counterexample?: E;
readonly failedInvariant?: string;
// (undocumented)
readonly iterations: number;
// (undocumented)
readonly passed: boolean;
}
// @public (undocumented)
export interface HookRunResult extends ScriptRunResult {
readonly blocked: boolean;
readonly blockedBy: readonly BlockMechanism[];
readonly conditionReason: string;
readonly decision: HookOutput["decision"] | "allow" | "deny" | "ask" | undefined;
readonly haltsTurn: boolean;
readonly json: HookOutput | null;
readonly ran: boolean;
}
// @public
export interface HookStateHandle {
clear(): void;
readonly dir: string;
read(key: string): StateFact;
seed(key: string, opts?: SeedStateOptions): StateFact;
}
// @public
export function improvement(report: EvalReport, baseline: string, arm: string, metric: string): number;
// @public
export function inputTokens(opts: {
max: number;
}): Check<UsageTrace>;
// @public
export type JudgeFn = (opts: {
output: string;
rubric: string;
threshold?: number;
}) => {
score: number;
pass: boolean;
reason?: string;
};
// @public
export function latency(opts: {
maxMs: number;
}): Check<UsageTrace>;
// @public
export function loadHook(file: string): Promise<AnyHook>;
// @public
export function mcp(server: string, toolName: string): Check<Trace>;
// @public
export interface MeasureSpec {
readonly allowedTools?: readonly string[];
readonly checks: readonly Check<RunContext>[];
readonly effort?: string | number;
readonly fixture?: Record<string, string>;
readonly interceptTools?: readonly ToolIntercept[];
readonly model?: string;
readonly plugin?: string;
readonly pluginDir?: string;
readonly settings?: unknown;
readonly spacingSec?: number;
readonly stubSkillBodies?: boolean;
readonly task: string;
readonly timeoutMs?: number;
readonly trials?: number;
}
// @public
export interface MetricDiff {
// (undocumented)
readonly arm: string;
readonly comparison: Comparison;
// (undocumented)
readonly metric: string;
readonly report: string;
// (undocumented)
readonly status: DiffStatus;
}
// @public (undocumented)
export type Metrics = Record<string, number | boolean>;
// @public
export interface MetricStat {
readonly mean: number;
readonly n: number;
readonly passK: number;
readonly se: number;
readonly std: number;
}
// @public (undocumented)
export type ModelOutputParser = (out: RunOut) => ParsedModelRun;
// @public
export interface ModelRequest {
readonly messages: readonly {
readonly role: string;
readonly text: string;
}[];
readonly sideChannel?: boolean;
readonly system: string;
}
// @public
export interface ModelTurn {
readonly input?: Record<string, unknown>;
readonly text?: string;
readonly tool?: string;
}
// @public
export function mustInclude(fragment: string, why: string): Check<readonly DocCommand[]>;
// @public
export function mustNotInclude(fragment: string, why: string): Check<readonly DocCommand[]>;
// @public
export type NetProbe = (port: number) => Promise<boolean>;
// @public
export interface NonCommandHookAction {
readonly event: string;
readonly matcher: string | null;
readonly type: string;
}
// @public
export function notTool(name: string, args?: ArgMatcher): Check<Trace>;
// @public
export function onlyTools(allowed: readonly string[]): Check<Trace>;
// @public
export function output(matcher: string | RegExp): Check<Trace>;
// @public
export function outputContains(trace: Trace, needle: string | RegExp): boolean;
// @public
export function outputTokens(opts: {
max: number;
}): Check<UsageTrace>;
// @public
export function parseBaselineFile(json: string): BaselineFile;
// @public
export function parseClaudeRun(out: RunOut): ParsedModelRun;
// @public
export interface ParsedModelRun {
// (undocumented)
readonly hooks: ReturnType<typeof parseHooks>;
// (undocumented)
readonly output: string;
// (undocumented)
readonly subagents: ReturnType<typeof parseSubagents>;
// (undocumented)
readonly toolCalls: ReturnType<typeof parseToolCalls>;
// (undocumented)
readonly turns: number;
// (undocumented)
readonly usage: EvalUsage;
}
// @public
export function parseHookOutput(stdout: string): HookOutput | null;
// @public (undocumented)
export function parseHooks(stdout: string): HookFire[];
// @public
export function parseOutput(stdout: string): string;
// @public
export function parseResultEvent(stdout: string): Record<string, unknown> | null;
// @public
export function parseSubagents(streamJson: string): SubagentTrace[];
// @public
export function parseToolCalls(streamJson: string): ToolCall[];
// @public
export interface PluginGuardReport {
readonly dir: string;
readonly event: string;
readonly events: readonly DisasterEvent[];
readonly harness: string;
readonly hooks: readonly SweptHookOutcome[];
readonly notes: readonly string[];
readonly unmeasurable: readonly NonCommandHookAction[];
}
// @public (undocumented)
export interface PromptDiversityIssue {
// (undocumented)
readonly kind: "too-few" | "too-similar";
// (undocumented)
readonly message: string;
}
// @public
export interface PromptTriggerStat {
// (undocumented)
readonly fired: number;
// (undocumented)
readonly prompt: string;
readonly rate: number;
// (undocumented)
readonly trials: number;
}
// @public
export function propertyHook<E, D>(opts: {
readonly seed: E;
readonly mutate: (event: E, rng: number) => E;
readonly decide: (event: E) => D;
readonly invariants: Record<string, (decision: D, event: E) => boolean>;
readonly iterations?: number;
}): HookPropertyResult<E>;
// @public
export function readBaseline(path: string): BaselineFile | null;
// @public
export function received(matcher: string | RegExp): Check<Trace>;
// @public
export function recordCheck(n?: number): void;
// @public
export function reliable(report: EvalReport, arm: string, metric: string): boolean;
// @public
export function renderToolStub(stub: ToolStub): string;
// @public
export function requestContains(trace: Trace, needle: string | RegExp): boolean;
// @public
export interface RunContext extends Trace {
// (undocumented)
readonly cwd: string;
// (undocumented)
readonly exitCode: number;
sh(command: string): string;
// (undocumented)
readonly stdout: string;
readonly toolCalls: readonly ToolCall[];
readonly turns: number;
readonly usage: EvalUsage;
}
// @public
export function runHarness(spec: HarnessTestSpec, opts?: RunHarnessTestOptions & {
model?: "mock" | "real";
}): Promise<HarnessTestResult>;
// @public
export function runHarnessTest(spec: HarnessTestSpec, opts?: RunHarnessTestOptions): Promise<HarnessTestResult>;
// @public
export interface RunHarnessTestOptions {
readonly adapter?: HarnessAdapter;
}
// @public
export function runHook(command: string, input: HookInput, opts?: RunHookOptions): HookRunResult;
// @public
export type RunHookOptions = Omit<RunScriptOptions, "stdin"> & {
readonly condition?: string;
readonly protocol?: HookProtocol;
};
// @public
export interface RunOut {
// (undocumented)
code: number;
stderr?: string;
// (undocumented)
stdout: string;
}
// @public
export function runScript(command: string, opts?: RunScriptOptions): ScriptRunResult;
// @public (undocumented)
export interface RunScriptOptions {
readonly cwd?: string;
readonly egress?: {
readonly allow: readonly string[];
};
readonly env?: Record<string, string>;
readonly recordEgress?: boolean;
readonly sandbox?: SandboxMode;
readonly stdin?: string;
readonly timeoutMs?: number;
readonly trusted?: boolean;
}
// @public
export function sandboxAvailable(): boolean;
// @public
export type SandboxMode = "auto" | "strict" | false;
// @public
export interface ScriptRunResult {
readonly egress: readonly EgressAttempt[];
readonly egressDropped?: {
readonly packets: number;
readonly bytes: number;
};
readonly exitCode: number;
readonly filesWritten?: readonly string[];
readonly stderr: string;
// (undocumented)
readonly stdout: string;
}
// @public
export type SeedStateOptions =
/** Recorded that long ago — the reason this exists (a throttle needs an OLD fact). */
{
readonly value?: string;
readonly ago: Duration;
readonly at?: never;
}
/** Recorded at exactly this instant. */
| {
readonly value?: string;
readonly at: Date;
readonly ago?: never;
}
/** Recorded just now. */
| {
readonly value?: string;
readonly ago?: never;
readonly at?: never;
};
// @public
export interface SelectionMatrixSpec extends SelectionMatrixOptions {
readonly pluginDir: string;
}
// @public
export interface SelectionTrialResult {
// (undocumented)
readonly errored: boolean;
// (undocumented)
readonly fired: readonly string[];
}
// @public
export interface ServiceHandle {
exec(command: string): {
readonly stdout: string;
readonly stderr: string;
readonly code: number;
};
readonly host: string;
readonly port?: number;
readonly url?: string;
}
// @public
export type ServiceReady = {
readonly tcp: number;
} | {
readonly log: RegExp;
} | {
readonly exec: string;
};
// @public
export type ServiceReset = "per-trial" | "per-arm";
// @public
export interface ServiceSession {
readonly endpoints: readonly string[];
readonly handles: Readonly<Record<string, ServiceHandle>>;
teardown(): Promise<void>;
}
// @public
export interface ServiceSpec {
readonly env?: Readonly<Record<string, string>>;
readonly image: string;
readonly port?: number;
readonly ready?: ServiceReady;
readonly reset?: ServiceReset;
readonly seed?: string;
}
// @public
export function significantlyBeats(report: EvalReport, baseline: string, arm: string, metric: string, alpha?: number): boolean;
// @public
export function skill(id: string): Check<Trace>;
// @public (undocumented)
export interface SkillContract {
readonly activation: Check<Trace>;
readonly declared: readonly string[];
readonly id: string;
readonly malformed: boolean;
readonly name: string;
readonly surface: readonly Check<Trace>[];
readonly undeclared: boolean;
}
// @public
export function skillContract(skillDir: string, options?: SkillContractOptions): SkillContract;
// @public (undocumented)
export interface SkillContractOptions {
readonly plugin?: string;
}
// @public
export function skillResolved(trace: Trace, skill: string): boolean;
// @public
export function skip(reason?: string): never;
// @public
export function specTrusted(spec: {
plugin?: string;
pluginDir?: string;
}): boolean;
// @public
export interface StateFact {
readonly ageSeconds: number;
readonly at: string;
fresherThan(within: Duration): boolean;
olderThan(within: Duration): boolean;
readonly recorded: boolean;
readonly value: string;
}
// @public
export function stubBinDir(stubs: readonly ToolStub[], parentDir: string): string;
// @public
export function stubSkillBody(skillMd: string): string;
// @public
export function subagent(name: string, checks: readonly Check<Trace>[]): Check<Trace>;
// @public
export interface SubagentTrace {
readonly name: string;
readonly output: string;
readonly toolCalls: readonly ToolCall[];
}
// @public
export interface SweptHook {
readonly command: string;
readonly condition: string | null;
readonly event: string;
readonly index: number;
readonly matcher: string | null;
}
// @public
export type SweptHookOutcome = {
readonly status: "measured";
readonly hook: SweptHook;
readonly results: readonly GuardrailResult[];
readonly blocked: readonly string[];
readonly allowed: readonly string[];
readonly notRun: readonly string[];
} | {
readonly status: "not-applicable";
readonly hook: SweptHook;
readonly reason: string;
} | {
readonly status: "unresolved";
readonly hook: SweptHook;
readonly reason: string;
};
// @public
export function toBaselineFile(reports: readonly EvalReport[], recordedAt?: string): BaselineFile;
// @public
export function tokens(opts: {
max: number;
}): Check<UsageTrace>;
// @public
export function tool(name: string): Check<Trace>;
// @public
export interface ToolCall {
// (undocumented)
readonly input: unknown;
readonly isError: boolean;
// (undocumented)
readonly name: string;
readonly resultText: string;
}
// @public
export function toolCount(trace: Trace, name: string | RegExp): number;
// @public
export interface ToolStub {
readonly exitCode?: number;
readonly name: string;
readonly stderr?: string;
readonly stdout?: string;
}
// @public
export function toolUsedWith(trace: Trace, name: string | RegExp, inputMatcher: (input: unknown) => boolean): boolean;
// @public
export function toolWith(name: string, args: ArgMatcher): Check<Trace>;
// @public
export interface Trace {
file(path: string): string | null;
readonly hooks: readonly HookFire[];
readonly modelRequests: readonly ModelRequest[];
readonly output: string;
readonly subagents?: readonly SubagentTrace[];
readonly toolCalls: readonly ToolCall[];
readonly turns: number;
}
// @public (undocumented)
export interface TriggerRateReport {
readonly competitors: number;
readonly errored?: number;
readonly experimental?: string;
readonly falsePositiveRate?: number;
readonly n: number;
readonly namespace?: string;
readonly perIrrelevant?: readonly PromptTriggerStat[];
// (undocumented)
readonly perPrompt: readonly PromptTriggerStat[];
readonly precision?: number;
readonly rate: number;
readonly usage: ArmUsage;
}
// @public
export interface TriggerRateSpec {
readonly allowedTools?: readonly string[];
readonly concurrency?: number;
readonly effort?: string | number;
readonly fired: (trace: Trace) => boolean;
readonly fixture?: Record<string, string>;
readonly installSet?: readonly string[];
readonly irrelevantPrompts?: readonly string[];
readonly lock?: EvalLockOptions;
readonly minDistance?: number;
readonly minModel?: string;
readonly minPrompts?: number;
readonly model?: string;
readonly name?: string;
readonly pluginDir?: string;
readonly prompts: readonly string[];
readonly skillsDir?: string;
readonly spacingSec?: number;
readonly stubSkillBodies?: boolean;
readonly timeoutMs?: number;
readonly trials?: number;
}
// @public
export function turns(opts: {
min?: number;
max?: number;
}): Check<Trace>;
// @public
export function unblockedDisasters(results: readonly GuardrailResult[]): GuardrailResult[];
// @public
export function usedTool(trace: Trace, name: string | RegExp): boolean;
// @public
export function verifyGuardrail(hookCommand: string, opts?: VerifyGuardrailOptions): GuardrailResult[];
// @public (undocumented)
export interface VerifyGuardrailOptions extends RunHookOptions {
readonly categories?: readonly DisasterCategory[];
readonly event?: string;
readonly events?: readonly DisasterEvent[];
}
// @public
export interface VerifyPluginGuardsOptions extends Omit<RunHookOptions, "condition" | "protocol"> {
readonly adapter?: HarnessAdapter;
readonly categories?: readonly DisasterCategory[];
readonly event?: string;
readonly events?: readonly DisasterEvent[];
}
// @public
export const vigilesMatchers: {
toHaveCreated(received: HarnessTestResult, path: string): MatcherOutput;
toBlock(received: HookRunResult): MatcherOutput;
toBeatBaseline(received: EvalReport, baseline: string, arm: string, metric: string, by?: number): MatcherOutput;
toPass<T>(received: T, check: Check<T>): MatcherOutput;
toPassAll<T>(received: T, checks: readonly Check<T>[]): MatcherOutput;
};
// @public
export function withHarness<T>(spec: HarnessTestSpec, fn: (r: HarnessTestResult) => T | Promise<T>): Promise<T>;
// @public
export function writeBaseline(path: string, reports: readonly EvalReport[]): void;
// @public
export function writeToolStubs(binDir: string, stubs: readonly ToolStub[]): void;
// @public
export function wrote(path: string): Check<Trace>;
// (No @packageDocumentation comment for this package)