mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
4f0e6e8037
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com>
9.7 KiB
9.7 KiB
API Report File for "vigiles-claude-code"
Do not edit this file. It is a report generated by API Extractor.
// @public
export function assertNoCollision(report: SelectionReport, opts?: {
maxOffDiagonal?: number;
maxPluginCollision?: number;
}): void;
// @public
export function buildClaudeArgs(spec: HarnessTestSpec, hasSettings: boolean): string[];
// @public
export function claudeAvailable(): boolean;
// @public (undocumented)
export const claudeCodeAdapter: HarnessAdapter;
// @public
export type ClaudeCodeBoundedTool = ClaudeCodeReadOnlyTool | "Write" | "Edit" | "NotebookEdit" | "Bash" | "PowerShell";
// @public
export const claudeCodeBuiltinAgentTools: readonly ["Read", "Grep", "Glob", "Bash", "PowerShell", "Edit", "Write", "NotebookEdit", "WebFetch", "WebSearch", "TodoWrite", "Skill", "ToolSearch", "EnterWorktree", "ExitWorktree", "Monitor", "TaskStop", "SendMessage", "Artifact", "Agent", "Task", "ExitPlanMode", "ListAgents", "LSP", "ShareOnboardingGuide", "CronCreate", "CronDelete", "CronList", "TaskCreate", "TaskGet", "TaskList", "TaskUpdate"];
// @public
export type ClaudeCodeBuiltinTool = (typeof claudeCodeBuiltinAgentTools)[number];
// @public (undocumented)
export const claudeCodeDialect: HarnessDialect;
// @public
export const claudeCodeDriver: HarnessTestDriver;
// @public (undocumented)
export const claudeCodeHookProtocol: HookProtocol;
// @public (undocumented)
export const claudeCodeLayout: PluginLayout;
// @public (undocumented)
export const claudeCodeModelMock: ModelMock;
// @public
export type ClaudeCodeReadOnlyTool = Exclude<ClaudeCodeBuiltinTool, ClaudeCodeSideEffectingTool>;
// @public (undocumented)
export const claudeCodeRuntime: HarnessRuntime;
// @public
export type ClaudeCodeSideEffectingTool = (typeof claudeCodeSideEffectingTools)[number];
// @public
export const claudeCodeSideEffectingTools: readonly ["Bash", "PowerShell", "Edit", "Write", "NotebookEdit", "WebFetch", "WebSearch", "Skill", "TodoWrite", "Monitor", "SendMessage", "Artifact", "EnterWorktree", "ExitWorktree", "TaskStop", "Agent", "Task", "TaskCreate", "TaskUpdate", "CronCreate", "CronDelete", "ShareOnboardingGuide"];
// @public
export interface ClaudeCodeToolVocabulary extends ToolVocabulary {
// (undocumented)
readonly bounded: ClaudeCodeBoundedTool;
// (undocumented)
readonly readOnly: ClaudeCodeReadOnlyTool;
}
// @public
const experimental_agent: typeof agentSpec & {
result: <const Ok extends Shape, const Err extends Shape>(ok: Ok, err: Err) => OutputContract<Ok, Err>;
delegate: (agent: string, task?: string, needsContract?: Shape) => RailwayStep;
railway: (spec: Omit<Railway, "_specType">) => Railway;
needs: <const N extends Shape>(shape: N) => NeedsContract<N>;
pipeStep: <Needs extends Shape, Ok extends Shape, Err extends Shape>(a: TypedAgentSpec<Ok, Err>, needsContract?: Needs) => PipeStep<Needs, Ok, Err>;
start: <Ok extends Shape, Err extends Shape>(first: PipeStep<Record<string, never>, Ok, Err> | TypedAgentSpec<Ok, Err>) => Pipeline<Ok, Err>;
andThen: <PriorOk extends Shape, PriorErr extends Shape, Needs extends Shape, Ok extends Shape, Err extends Shape>(prior: Pipeline<PriorOk, PriorErr>, next: Supplies<PriorOk, Needs> extends true ? PipeStep<Needs, Ok, Err> : {
readonly __HANDOFF_ERROR: Supplies<PriorOk, Needs>;
}) => Pipeline<Ok, PriorErr | Err>;
pipe: {
<A extends Shape, AE extends Shape>(a: TypedAgentSpec<A, AE>): Pipeline<A, AE>;
<A extends Shape, AE extends Shape, BN extends Shape, B extends Shape, BE extends Shape>(a: TypedAgentSpec<A, AE>, b: Supplies<A, BN> extends true ? PipeStep<BN, B, BE> : {
readonly __HANDOFF_ERROR: Supplies<A, BN>;
}): Pipeline<B, AE | BE>;
<A extends Shape, AE extends Shape, BN extends Shape, B extends Shape, BE extends Shape, CN extends Shape, C extends Shape, CE extends Shape>(a: TypedAgentSpec<A, AE>, b: Supplies<A, BN> extends true ? PipeStep<BN, B, BE> : {
readonly __HANDOFF_ERROR: Supplies<A, BN>;
}, c: Supplies<B, CN> extends true ? PipeStep<CN, C, CE> : {
readonly __HANDOFF_ERROR: Supplies<B, CN>;
}): Pipeline<C, AE | BE | CE>;
<A extends Shape, AE extends Shape, BN extends Shape, B extends Shape, BE extends Shape, CN extends Shape, C extends Shape, CE extends Shape, DN extends Shape, D extends Shape, DE extends Shape>(a: TypedAgentSpec<A, AE>, b: Supplies<A, BN> extends true ? PipeStep<BN, B, BE> : {
readonly __HANDOFF_ERROR: Supplies<A, BN>;
}, c: Supplies<B, CN> extends true ? PipeStep<CN, C, CE> : {
readonly __HANDOFF_ERROR: Supplies<B, CN>;
}, d: Supplies<C, DN> extends true ? PipeStep<DN, D, DE> : {
readonly __HANDOFF_ERROR: Supplies<C, DN>;
}): Pipeline<D, AE | BE | CE | DE>;
};
};
export { experimental_agent as agent }
export { experimental_agent }
// @public (undocumented)
export const experimental_skill: typeof skillSpec & {
input: (name: string, hint: string, opts?: {
required?: boolean;
}) => SkillInput;
step: (instr: string | InstructionFragment[], opts?: {
gate?: Gate;
retry?: number;
}) => SkillStep;
};
// @public
export function extractRequest(body: {
system?: unknown;
messages?: unknown;
}): ModelRequest;
// @public
export function formatSelectionReport(r: SelectionReport): string;
// @public (undocumented)
export interface HarnessDialect {
readonly builtinAgentTools: readonly string[];
readonly hookEvents: readonly string[];
readonly hookEventVocabulary?: HarnessVocabulary;
readonly instructionTargets: readonly string[];
readonly knownMcpServers?: readonly string[];
readonly mcpToolPattern: RegExp;
readonly name: string;
readonly neverAvailableTools: readonly string[];
readonly noEffectHookEvents?: readonly string[];
readonly permissionDecisionHookEvents?: readonly string[];
readonly pluginRootToken: string;
readonly sideEffectingTools?: readonly string[];
readonly skillFrontmatter: SkillFrontmatterProfile;
readonly subagentToolVocabulary?: HarnessVocabulary;
}
// @public
export function isMainLoopRequest(body: {
tools?: unknown;
}): boolean;
// @public (undocumented)
export interface LoadedPlugin {
readonly files: Record<string, string>;
readonly settings: {
hooks?: unknown;
};
readonly sources: Record<string, string>;
readonly warnings: readonly string[];
}
// @public
export function loadPlugin(pluginPath: string, layout?: PluginLayout): ReturnType<typeof loadPlugin_2>;
// @public
export function measureSelectionMatrix(dir: string, opts?: SelectionMatrixOptions): Promise<SelectionReport>;
// @public (undocumented)
export interface MockHandle {
// (undocumented)
close(): void;
readonly count: number;
readonly requests: readonly ModelRequest[];
readonly sideChannelCount: number;
// (undocumented)
readonly url: string;
}
// @public
export function mockModelEnv(runtime: HarnessRuntime, baseUrl: string, base?: NodeJS.ProcessEnv): NodeJS.ProcessEnv;
// @public
export interface ModelRequest {
readonly messages: readonly {
readonly role: string;
readonly text: string;
}[];
readonly sideChannel?: boolean;
readonly system: string;
}
// @public
export interface ModelTurn {
readonly input?: Record<string, unknown>;
readonly text?: string;
readonly tool?: string;
}
// @public
export function parseClaudeRun(stdout: string): ParsedRun;
// @public
export function resolveHarness(opts: {
plugin?: string;
settings?: unknown;
files?: Record<string, string>;
}, layout?: PluginLayout): ReturnType<typeof resolveHarness_2>;
// @public
export function scriptModel(turns: readonly ModelTurn[]): ModelTurn[];
// @public
export function scriptUnconsumedWarning(count: number, sideChannelCount: number): string | undefined;
// @public
export interface SelectionMatrixOptions extends SelectionOptions {
readonly prompts?: TriggerPromptSet;
}
// @public
export interface SelectionOptions {
readonly concurrency?: number;
readonly effort?: string | number;
readonly harness?: ProbeHarness;
readonly model?: string;
readonly trials?: number;
}
// @public (undocumented)
export interface SelectionReport {
readonly available: boolean;
readonly collisionRate: number;
readonly matrix: readonly (readonly number[])[];
// (undocumented)
readonly n: number;
// (undocumented)
readonly note?: string;
// (undocumented)
readonly perSkill: readonly SkillSelectionStat[];
readonly skills: readonly string[];
}
// @public (undocumented)
export interface SkillSelectionStat {
readonly collidesWith: readonly {
readonly skill: string;
readonly rate: number;
}[];
readonly collisionRate: number;
readonly n: number;
readonly recall: number;
// (undocumented)
readonly skill: string;
}
// @public
export function splitRequestCounts(requests: readonly ModelRequest[]): {
count: number;
sideChannelCount: number;
};
// @public
export function startMock(script: readonly ModelTurn[], opts?: {
onTurn?: (info: TurnInfo) => void;
onRequest?: (req: ModelRequest) => void;
}): Promise<MockHandle>;
// @public (undocumented)
export interface TurnInfo {
// (undocumented)
readonly hasToolResult: boolean;
// (undocumented)
readonly n: number;
// (undocumented)
readonly stream: boolean;
}
// (No @packageDocumentation comment for this package)