mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
4f0e6e8037
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com>
2.8 KiB
2.8 KiB
API Report File for "vigiles-codex"
Do not edit this file. It is a report generated by API Extractor.
// @public
export function buildCodexArgs(ctx: HarnessDriverContext): string[];
// @public
export const CODEX_TRIGGER_RATE_EXPERIMENTAL: string;
// @public (undocumented)
export const codexAdapter: HarnessAdapter;
// @public (undocumented)
export const codexDialect: HarnessDialect;
// @public (undocumented)
export const codexDriver: HarnessTestDriver;
// @public
export function codexEvalAgentRunner(args: AgentRunArgs): Promise<RunOut>;
// @public
export const codexEvalDriver: EvalDriver;
// @public
export function codexEvalRunner(args: {
task: string;
cwd: string;
timeoutMs: number;
}): {
code: number;
stdout: string;
};
// @public (undocumented)
export const codexHookProtocol: HookProtocol;
// @public (undocumented)
export const codexLayout: PluginLayout;
// @public
export function codexMockArgs(baseUrl: string): string[];
// @public
export function codexMockEnv(): Record<string, string>;
// @public (undocumented)
export interface CodexMockHandle {
// (undocumented)
close(): Promise<void>;
// (undocumented)
readonly port: number;
readonly requests: readonly CodexRequest[];
// (undocumented)
readonly url: string;
}
// @public (undocumented)
export const codexModelMock: ModelMock;
// @public
export interface CodexRequest {
// (undocumented)
readonly model: string;
// (undocumented)
readonly prompt: string;
// (undocumented)
readonly toolNames: readonly string[];
}
// @public
export function codexRunError(out: {
stdout: string;
}): string | null;
// @public (undocumented)
export const codexRuntime: HarnessRuntime;
// @public
export function codexSkillFired(run: {
toolCalls: readonly ToolCall[];
}, name: string): boolean;
// @public
export interface CodexTurn {
readonly text: string;
}
// @public
export function installCodexSkills(pluginDir: string, cwd: string): number;
// @public
export function parseCodexEvalRun(out: {
stdout: string;
}): ParsedModelRun;
// @public
export function parseCodexRun(stdout: string): ParsedRun;
// @public
export function parseResponsesRequest(body: string): {
prompt: string;
model: string;
toolNames: string[];
};
// @public
export function refuseCodexEffort(effort: string | number | undefined): void;
// @public
export function renderResponsesSSE(text: string, opts?: {
model?: string;
}): string;
// @public
export function startCodexMock(script: readonly CodexTurn[], opts?: {
onRequest?: (req: CodexRequest) => void;
}): Promise<CodexMockHandle>;
// (No @packageDocumentation comment for this package)