mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
4f0e6e8037
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com>
10 KiB
10 KiB
API Report File for "vigiles-eval"
Do not edit this file. It is a report generated by API Extractor.
// @public
export interface AgentRunArgs {
// (undocumented)
readonly cwd: string;
readonly effort?: string | number;
readonly env?: Record<string, string>;
// (undocumented)
readonly hasSettings: boolean;
// (undocumented)
readonly model: string;
// (undocumented)
readonly pluginDir: string | undefined;
readonly replaceEnv?: boolean;
// (undocumented)
readonly task: string;
// (undocumented)
readonly timeoutMs: number;
// (undocumented)
readonly tools: readonly string[];
}
// @public
export type AgentRunner = (args: AgentRunArgs) => Promise<RunOut>;
// @public (undocumented)
export interface ArmReport {
readonly metrics: Record<string, number>;
// (undocumented)
readonly runs: number;
readonly stats: Record<string, MetricStat>;
readonly usage: ArmUsage;
}
// @public
export interface ArmsCheckReport {
// (undocumented)
readonly arms: Record<string, CheckReport>;
}
// @public
export interface ArmsMeasureSpec {
// (undocumented)
readonly allowedTools?: readonly string[];
readonly arms: Record<string, EvalArm>;
// (undocumented)
readonly checks: readonly Check<RunContext>[];
readonly effort?: string | number;
// (undocumented)
readonly fixture?: Record<string, string>;
// (undocumented)
readonly model?: string;
// (undocumented)
readonly spacingSec?: number;
readonly stubSkillBodies?: boolean;
// (undocumented)
readonly task: string;
// (undocumented)
readonly timeoutMs?: number;
// (undocumented)
readonly trials?: number;
}
// @public
export interface ArmUsage {
// (undocumented)
readonly meanCostUsd: number;
// (undocumented)
readonly meanDurationMs: number;
// (undocumented)
readonly totalCacheCreationTokens: number;
// (undocumented)
readonly totalCacheReadTokens: number;
// (undocumented)
readonly totalCostUsd: number;
// (undocumented)
readonly totalInputTokens: number;
// (undocumented)
readonly totalOutputTokens: number;
}
// @public
export interface Check<T> {
eval(target: T): CheckResult;
readonly kind: string;
toJSON(): CheckJSON;
}
// @public
export interface CheckRate {
// (undocumented)
readonly check: CheckJSON;
readonly n: number;
readonly passK: number;
readonly rate: number;
readonly se: number;
}
// @public (undocumented)
export interface CheckReport {
// (undocumented)
readonly n: number;
// (undocumented)
readonly perCheck: readonly CheckRate[];
readonly usage: ArmUsage;
}
// @public
export interface CheckResult {
readonly message: string;
readonly pass: boolean;
readonly score: number;
}
// @public
export interface EvalArm {
readonly effort?: string | number;
readonly files?: Record<string, string>;
readonly interceptTools?: readonly ToolIntercept[];
readonly model?: string;
readonly plugin?: string;
readonly pluginDir?: string;
readonly settings?: unknown;
}
// @public
export interface EvalDriver {
readonly experimental?: string;
readonly harness?: string;
// (undocumented)
readonly parse: ModelOutputParser;
// (undocumented)
readonly runError?: (out: RunOut) => string | null;
// (undocumented)
readonly runner: AgentRunner;
}
// @public (undocumented)
export interface EvalReport {
readonly aborted: boolean;
// (undocumented)
readonly arms: Record<string, ArmReport>;
// (undocumented)
readonly name: string;
readonly totalCostUsd: number;
// (undocumented)
readonly trials: number;
}
// @public (undocumented)
export interface EvalSpec<M extends Metrics> {
readonly allowedTools?: readonly string[];
readonly arms: Record<string, EvalArm>;
readonly cache?: CacheMode;
readonly cacheDir?: string;
readonly concurrency?: number;
readonly effort?: string | number;
readonly ephemeralEnv?: boolean;
readonly fixture?: Record<string, string>;
readonly lock?: EvalLockOptions;
readonly maxCostUsd?: number;
readonly measure: (ctx: RunContext) => M;
readonly model?: string;
// (undocumented)
readonly name?: string;
readonly rateLimitRetries?: number;
readonly retryBackoffMs?: number;
readonly spacingSec?: number;
readonly stubs?: readonly ToolStub[];
readonly task: string;
readonly timeoutMs?: number;
readonly trials?: number;
}
// @public
export interface EvalUsage {
readonly cacheCreationTokens: number;
readonly cacheReadTokens: number;
readonly costUsd: number;
readonly durationMs: number;
readonly inputTokens: number;
// (undocumented)
readonly outputTokens: number;
}
// @public
export type JudgeFn = (opts: {
output: string;
rubric: string;
threshold?: number;
}) => {
score: number;
pass: boolean;
reason?: string;
};
// @public
export interface MeasureSpec {
readonly allowedTools?: readonly string[];
readonly checks: readonly Check<RunContext>[];
readonly effort?: string | number;
readonly fixture?: Record<string, string>;
readonly interceptTools?: readonly ToolIntercept[];
readonly model?: string;
readonly plugin?: string;
readonly pluginDir?: string;
readonly settings?: unknown;
readonly spacingSec?: number;
readonly stubSkillBodies?: boolean;
readonly task: string;
readonly timeoutMs?: number;
readonly trials?: number;
}
// @public (undocumented)
export type Metrics = Record<string, number | boolean>;
// @public
export interface MetricStat {
readonly mean: number;
readonly n: number;
readonly passK: number;
readonly se: number;
readonly std: number;
}
// @public (undocumented)
export type ModelOutputParser = (out: RunOut) => ParsedModelRun;
// @public
export const paid_claudeEvalDriver: EvalDriver;
// @public
export function paid_judge(opts: JudgeOptions): JudgeResult;
// @public
export function paid_judged(rubric: string, opts?: {
min?: number;
judge?: JudgeFn;
}): Check<Trace>;
// @public
export function paid_measure(spec: MeasureSpec): Promise<CheckReport>;
// @public
export function paid_measureArms(spec: ArmsMeasureSpec): Promise<ArmsCheckReport>;
// @public
export function paid_measureTriggerRate(spec: TriggerRateSpec, opts?: {
evalDriver?: EvalDriver;
}): Promise<TriggerRateReport>;
// @public (undocumented)
export function paid_runEval<M extends Metrics>(spec: EvalSpec<M>): Promise<EvalReport>;
// @public
export interface ParsedModelRun {
// (undocumented)
readonly hooks: ReturnType<typeof parseHooks>;
// (undocumented)
readonly output: string;
// (undocumented)
readonly subagents: ReturnType<typeof parseSubagents>;
// (undocumented)
readonly toolCalls: ReturnType<typeof parseToolCalls>;
// (undocumented)
readonly turns: number;
// (undocumented)
readonly usage: EvalUsage;
}
// @public (undocumented)
export interface PromptDiversityIssue {
// (undocumented)
readonly kind: "too-few" | "too-similar";
// (undocumented)
readonly message: string;
}
// @public
export interface PromptTriggerStat {
// (undocumented)
readonly fired: number;
// (undocumented)
readonly prompt: string;
readonly rate: number;
// (undocumented)
readonly trials: number;
}
// @public
export interface RunContext extends Trace {
// (undocumented)
readonly cwd: string;
// (undocumented)
readonly exitCode: number;
sh(command: string): string;
// (undocumented)
readonly stdout: string;
readonly toolCalls: readonly ToolCall[];
readonly turns: number;
readonly usage: EvalUsage;
}
// @public
export interface RunOut {
// (undocumented)
code: number;
stderr?: string;
// (undocumented)
stdout: string;
}
// @public
export interface SelectionTrialResult {
// (undocumented)
readonly errored: boolean;
// (undocumented)
readonly fired: readonly string[];
}
// @public
export interface Trace {
file(path: string): string | null;
readonly hooks: readonly HookFire[];
readonly modelRequests: readonly ModelRequest[];
readonly output: string;
readonly subagents?: readonly SubagentTrace[];
readonly toolCalls: readonly ToolCall[];
readonly turns: number;
}
// @public (undocumented)
export interface TriggerRateReport {
readonly competitors: number;
readonly errored?: number;
readonly experimental?: string;
readonly falsePositiveRate?: number;
readonly n: number;
readonly namespace?: string;
readonly perIrrelevant?: readonly PromptTriggerStat[];
// (undocumented)
readonly perPrompt: readonly PromptTriggerStat[];
readonly precision?: number;
readonly rate: number;
readonly usage: ArmUsage;
}
// @public
export interface TriggerRateSpec {
readonly allowedTools?: readonly string[];
readonly concurrency?: number;
readonly effort?: string | number;
readonly fired: (trace: Trace) => boolean;
readonly fixture?: Record<string, string>;
readonly installSet?: readonly string[];
readonly irrelevantPrompts?: readonly string[];
readonly lock?: EvalLockOptions;
readonly minDistance?: number;
readonly minModel?: string;
readonly minPrompts?: number;
readonly model?: string;
readonly name?: string;
readonly pluginDir?: string;
readonly prompts: readonly string[];
readonly skillsDir?: string;
readonly spacingSec?: number;
readonly stubSkillBodies?: boolean;
readonly timeoutMs?: number;
readonly trials?: number;
}
// (No @packageDocumentation comment for this package)