Files
zernie__vigiles/api-surface/vigiles-claude-code.api.md
T
zernie 4f0e6e8037 feat(eval): pin --effort as a measurement input, hashed in lock and cache (#191)
`claude --effort` sets the reasoning budget per turn. vigiles could not set it,
so an A/B could not reproduce a benchmark that pins one: the arm you ran was not
the arm the author measured (reported in #189, with a real case — a skill whose
published results pin "effort low").

Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not
the sample size. So it lives on the spec beside `model` (never an env), it is an
arm in its own right, and it is hashed into BOTH the eval lock and the local
cache — the reporter named only the lock; the cache key carried `model` without
effort in exactly the same way, and fixing one would have left half the hole.

Three things that are not in the reported shape:

- THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings
  key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while
  EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient
  value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning
  it would make the lock confidently wrong: `low` recorded over a run that
  executed at the shell's `max` — the defect this feature exists to prevent,
  reintroduced by the fix for it. Declared sets the var; OMITTED deletes an
  inherited one, so "omit" means the harness default rather than whatever the
  machine exported.

- NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and
  accepts an integer budget; its valid set already MOVED between builds (2.1.42
  had no `xhigh`). A hard-coded union would reject a valid level after any
  upstream addition. The silent fallback a union was meant to prevent is not
  silent — the binary warns on stderr, which is already captured — so the wrong
  value is caught at runtime by `effortRejection`, and a rejected value THROWS
  rather than counting as `runError`: an unusable value is deterministic, so
  every trial fails it and the report would be a rate over zero samples.

- `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a
  hard-coded model; effort has no observable effect there. The issue names it;
  adding the flag would have been dead argv.

Codex refuses a declared effort loudly instead of dropping it — no mapping has
been measured, and silently recording a budget the run did not use is worse than
refusing. `audit` deliberately gets no CLI flag: it is a local report, not a
reproducibility surface.

`buildAgentArgs` is extracted so the flag is provable at all — the argv was built
inside the `v8 ignore`d spawn, where nothing could assert it.

Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures.
Eight mutations, each proven to have LANDED. Seven fail on their own assertion.
The eighth stays green and that is a finding about the TEST, recorded in place:
effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and
the per-arm inputs), so removing either alone is green and removing BOTH fails —
overlap across two populations, not a dead line.

One regression caught by the full suite and fixed here: `withEffortGuard` was
written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into
rejected promises — downgrading the foreign-test-runner guard that stops a paid
eval from billing. Now pinned by a test.


Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-02 14:37:27 +05:00

9.7 KiB

API Report File for "vigiles-claude-code"

Do not edit this file. It is a report generated by API Extractor.


// @public
export function assertNoCollision(report: SelectionReport, opts?: {
    maxOffDiagonal?: number;
    maxPluginCollision?: number;
}): void;

// @public
export function buildClaudeArgs(spec: HarnessTestSpec, hasSettings: boolean): string[];

// @public
export function claudeAvailable(): boolean;

// @public (undocumented)
export const claudeCodeAdapter: HarnessAdapter;

// @public
export type ClaudeCodeBoundedTool = ClaudeCodeReadOnlyTool | "Write" | "Edit" | "NotebookEdit" | "Bash" | "PowerShell";

// @public
export const claudeCodeBuiltinAgentTools: readonly ["Read", "Grep", "Glob", "Bash", "PowerShell", "Edit", "Write", "NotebookEdit", "WebFetch", "WebSearch", "TodoWrite", "Skill", "ToolSearch", "EnterWorktree", "ExitWorktree", "Monitor", "TaskStop", "SendMessage", "Artifact", "Agent", "Task", "ExitPlanMode", "ListAgents", "LSP", "ShareOnboardingGuide", "CronCreate", "CronDelete", "CronList", "TaskCreate", "TaskGet", "TaskList", "TaskUpdate"];

// @public
export type ClaudeCodeBuiltinTool = (typeof claudeCodeBuiltinAgentTools)[number];

// @public (undocumented)
export const claudeCodeDialect: HarnessDialect;

// @public
export const claudeCodeDriver: HarnessTestDriver;

// @public (undocumented)
export const claudeCodeHookProtocol: HookProtocol;

// @public (undocumented)
export const claudeCodeLayout: PluginLayout;

// @public (undocumented)
export const claudeCodeModelMock: ModelMock;

// @public
export type ClaudeCodeReadOnlyTool = Exclude<ClaudeCodeBuiltinTool, ClaudeCodeSideEffectingTool>;

// @public (undocumented)
export const claudeCodeRuntime: HarnessRuntime;

// @public
export type ClaudeCodeSideEffectingTool = (typeof claudeCodeSideEffectingTools)[number];

// @public
export const claudeCodeSideEffectingTools: readonly ["Bash", "PowerShell", "Edit", "Write", "NotebookEdit", "WebFetch", "WebSearch", "Skill", "TodoWrite", "Monitor", "SendMessage", "Artifact", "EnterWorktree", "ExitWorktree", "TaskStop", "Agent", "Task", "TaskCreate", "TaskUpdate", "CronCreate", "CronDelete", "ShareOnboardingGuide"];

// @public
export interface ClaudeCodeToolVocabulary extends ToolVocabulary {
    // (undocumented)
    readonly bounded: ClaudeCodeBoundedTool;
    // (undocumented)
    readonly readOnly: ClaudeCodeReadOnlyTool;
}

// @public
const experimental_agent: typeof agentSpec & {
    result: <const Ok extends Shape, const Err extends Shape>(ok: Ok, err: Err) => OutputContract<Ok, Err>;
    delegate: (agent: string, task?: string, needsContract?: Shape) => RailwayStep;
    railway: (spec: Omit<Railway, "_specType">) => Railway;
    needs: <const N extends Shape>(shape: N) => NeedsContract<N>;
    pipeStep: <Needs extends Shape, Ok extends Shape, Err extends Shape>(a: TypedAgentSpec<Ok, Err>, needsContract?: Needs) => PipeStep<Needs, Ok, Err>;
    start: <Ok extends Shape, Err extends Shape>(first: PipeStep<Record<string, never>, Ok, Err> | TypedAgentSpec<Ok, Err>) => Pipeline<Ok, Err>;
    andThen: <PriorOk extends Shape, PriorErr extends Shape, Needs extends Shape, Ok extends Shape, Err extends Shape>(prior: Pipeline<PriorOk, PriorErr>, next: Supplies<PriorOk, Needs> extends true ? PipeStep<Needs, Ok, Err> : {
        readonly __HANDOFF_ERROR: Supplies<PriorOk, Needs>;
    }) => Pipeline<Ok, PriorErr | Err>;
    pipe: {
        <A extends Shape, AE extends Shape>(a: TypedAgentSpec<A, AE>): Pipeline<A, AE>;
        <A extends Shape, AE extends Shape, BN extends Shape, B extends Shape, BE extends Shape>(a: TypedAgentSpec<A, AE>, b: Supplies<A, BN> extends true ? PipeStep<BN, B, BE> : {
            readonly __HANDOFF_ERROR: Supplies<A, BN>;
        }): Pipeline<B, AE | BE>;
        <A extends Shape, AE extends Shape, BN extends Shape, B extends Shape, BE extends Shape, CN extends Shape, C extends Shape, CE extends Shape>(a: TypedAgentSpec<A, AE>, b: Supplies<A, BN> extends true ? PipeStep<BN, B, BE> : {
            readonly __HANDOFF_ERROR: Supplies<A, BN>;
        }, c: Supplies<B, CN> extends true ? PipeStep<CN, C, CE> : {
            readonly __HANDOFF_ERROR: Supplies<B, CN>;
        }): Pipeline<C, AE | BE | CE>;
        <A extends Shape, AE extends Shape, BN extends Shape, B extends Shape, BE extends Shape, CN extends Shape, C extends Shape, CE extends Shape, DN extends Shape, D extends Shape, DE extends Shape>(a: TypedAgentSpec<A, AE>, b: Supplies<A, BN> extends true ? PipeStep<BN, B, BE> : {
            readonly __HANDOFF_ERROR: Supplies<A, BN>;
        }, c: Supplies<B, CN> extends true ? PipeStep<CN, C, CE> : {
            readonly __HANDOFF_ERROR: Supplies<B, CN>;
        }, d: Supplies<C, DN> extends true ? PipeStep<DN, D, DE> : {
            readonly __HANDOFF_ERROR: Supplies<C, DN>;
        }): Pipeline<D, AE | BE | CE | DE>;
    };
};
export { experimental_agent as agent }
export { experimental_agent }

// @public (undocumented)
export const experimental_skill: typeof skillSpec & {
    input: (name: string, hint: string, opts?: {
        required?: boolean;
    }) => SkillInput;
    step: (instr: string | InstructionFragment[], opts?: {
        gate?: Gate;
        retry?: number;
    }) => SkillStep;
};

// @public
export function extractRequest(body: {
    system?: unknown;
    messages?: unknown;
}): ModelRequest;

// @public
export function formatSelectionReport(r: SelectionReport): string;

// @public (undocumented)
export interface HarnessDialect {
    readonly builtinAgentTools: readonly string[];
    readonly hookEvents: readonly string[];
    readonly hookEventVocabulary?: HarnessVocabulary;
    readonly instructionTargets: readonly string[];
    readonly knownMcpServers?: readonly string[];
    readonly mcpToolPattern: RegExp;
    readonly name: string;
    readonly neverAvailableTools: readonly string[];
    readonly noEffectHookEvents?: readonly string[];
    readonly permissionDecisionHookEvents?: readonly string[];
    readonly pluginRootToken: string;
    readonly sideEffectingTools?: readonly string[];
    readonly skillFrontmatter: SkillFrontmatterProfile;
    readonly subagentToolVocabulary?: HarnessVocabulary;
}

// @public
export function isMainLoopRequest(body: {
    tools?: unknown;
}): boolean;

// @public (undocumented)
export interface LoadedPlugin {
    readonly files: Record<string, string>;
    readonly settings: {
        hooks?: unknown;
    };
    readonly sources: Record<string, string>;
    readonly warnings: readonly string[];
}

// @public
export function loadPlugin(pluginPath: string, layout?: PluginLayout): ReturnType<typeof loadPlugin_2>;

// @public
export function measureSelectionMatrix(dir: string, opts?: SelectionMatrixOptions): Promise<SelectionReport>;

// @public (undocumented)
export interface MockHandle {
    // (undocumented)
    close(): void;
    readonly count: number;
    readonly requests: readonly ModelRequest[];
    readonly sideChannelCount: number;
    // (undocumented)
    readonly url: string;
}

// @public
export function mockModelEnv(runtime: HarnessRuntime, baseUrl: string, base?: NodeJS.ProcessEnv): NodeJS.ProcessEnv;

// @public
export interface ModelRequest {
    readonly messages: readonly {
        readonly role: string;
        readonly text: string;
    }[];
    readonly sideChannel?: boolean;
    readonly system: string;
}

// @public
export interface ModelTurn {
    readonly input?: Record<string, unknown>;
    readonly text?: string;
    readonly tool?: string;
}

// @public
export function parseClaudeRun(stdout: string): ParsedRun;

// @public
export function resolveHarness(opts: {
    plugin?: string;
    settings?: unknown;
    files?: Record<string, string>;
}, layout?: PluginLayout): ReturnType<typeof resolveHarness_2>;

// @public
export function scriptModel(turns: readonly ModelTurn[]): ModelTurn[];

// @public
export function scriptUnconsumedWarning(count: number, sideChannelCount: number): string | undefined;

// @public
export interface SelectionMatrixOptions extends SelectionOptions {
    readonly prompts?: TriggerPromptSet;
}

// @public
export interface SelectionOptions {
    readonly concurrency?: number;
    readonly effort?: string | number;
    readonly harness?: ProbeHarness;
    readonly model?: string;
    readonly trials?: number;
}

// @public (undocumented)
export interface SelectionReport {
    readonly available: boolean;
    readonly collisionRate: number;
    readonly matrix: readonly (readonly number[])[];
    // (undocumented)
    readonly n: number;
    // (undocumented)
    readonly note?: string;
    // (undocumented)
    readonly perSkill: readonly SkillSelectionStat[];
    readonly skills: readonly string[];
}

// @public (undocumented)
export interface SkillSelectionStat {
    readonly collidesWith: readonly {
        readonly skill: string;
        readonly rate: number;
    }[];
    readonly collisionRate: number;
    readonly n: number;
    readonly recall: number;
    // (undocumented)
    readonly skill: string;
}

// @public
export function splitRequestCounts(requests: readonly ModelRequest[]): {
    count: number;
    sideChannelCount: number;
};

// @public
export function startMock(script: readonly ModelTurn[], opts?: {
    onTurn?: (info: TurnInfo) => void;
    onRequest?: (req: ModelRequest) => void;
}): Promise<MockHandle>;

// @public (undocumented)
export interface TurnInfo {
    // (undocumented)
    readonly hasToolResult: boolean;
    // (undocumented)
    readonly n: number;
    // (undocumented)
    readonly stream: boolean;
}

// (No @packageDocumentation comment for this package)