mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
4f0e6e8037
`claude --effort` sets the reasoning budget per turn. vigiles could not set it, so an A/B could not reproduce a benchmark that pins one: the arm you ran was not the arm the author measured (reported in #189, with a real case — a skill whose published results pin "effort low"). Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not the sample size. So it lives on the spec beside `model` (never an env), it is an arm in its own right, and it is hashed into BOTH the eval lock and the local cache — the reporter named only the lock; the cache key carried `model` without effort in exactly the same way, and fixing one would have left half the hole. Three things that are not in the reported shape: - THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning it would make the lock confidently wrong: `low` recorded over a run that executed at the shell's `max` — the defect this feature exists to prevent, reintroduced by the fix for it. Declared sets the var; OMITTED deletes an inherited one, so "omit" means the harness default rather than whatever the machine exported. - NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and accepts an integer budget; its valid set already MOVED between builds (2.1.42 had no `xhigh`). A hard-coded union would reject a valid level after any upstream addition. The silent fallback a union was meant to prevent is not silent — the binary warns on stderr, which is already captured — so the wrong value is caught at runtime by `effortRejection`, and a rejected value THROWS rather than counting as `runError`: an unusable value is deterministic, so every trial fails it and the report would be a rate over zero samples. - `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a hard-coded model; effort has no observable effect there. The issue names it; adding the flag would have been dead argv. Codex refuses a declared effort loudly instead of dropping it — no mapping has been measured, and silently recording a budget the run did not use is worse than refusing. `audit` deliberately gets no CLI flag: it is a local report, not a reproducibility surface. `buildAgentArgs` is extracted so the flag is provable at all — the argv was built inside the `v8 ignore`d spawn, where nothing could assert it. Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures. Eight mutations, each proven to have LANDED. Seven fail on their own assertion. The eighth stays green and that is a finding about the TEST, recorded in place: effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and the per-arm inputs), so removing either alone is green and removing BOTH fails — overlap across two populations, not a dead line. One regression caught by the full suite and fixed here: `withEffortGuard` was written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into rejected promises — downgrading the foreign-test-runner guard that stops a paid eval from billing. Now pinned by a test. Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4 Co-authored-by: Claude <noreply@anthropic.com>
126 lines
2.8 KiB
Markdown
126 lines
2.8 KiB
Markdown
## API Report File for "vigiles-codex"
|
|
|
|
> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
|
|
|
|
```ts
|
|
|
|
// @public
|
|
export function buildCodexArgs(ctx: HarnessDriverContext): string[];
|
|
|
|
// @public
|
|
export const CODEX_TRIGGER_RATE_EXPERIMENTAL: string;
|
|
|
|
// @public (undocumented)
|
|
export const codexAdapter: HarnessAdapter;
|
|
|
|
// @public (undocumented)
|
|
export const codexDialect: HarnessDialect;
|
|
|
|
// @public (undocumented)
|
|
export const codexDriver: HarnessTestDriver;
|
|
|
|
// @public
|
|
export function codexEvalAgentRunner(args: AgentRunArgs): Promise<RunOut>;
|
|
|
|
// @public
|
|
export const codexEvalDriver: EvalDriver;
|
|
|
|
// @public
|
|
export function codexEvalRunner(args: {
|
|
task: string;
|
|
cwd: string;
|
|
timeoutMs: number;
|
|
}): {
|
|
code: number;
|
|
stdout: string;
|
|
};
|
|
|
|
// @public (undocumented)
|
|
export const codexHookProtocol: HookProtocol;
|
|
|
|
// @public (undocumented)
|
|
export const codexLayout: PluginLayout;
|
|
|
|
// @public
|
|
export function codexMockArgs(baseUrl: string): string[];
|
|
|
|
// @public
|
|
export function codexMockEnv(): Record<string, string>;
|
|
|
|
// @public (undocumented)
|
|
export interface CodexMockHandle {
|
|
// (undocumented)
|
|
close(): Promise<void>;
|
|
// (undocumented)
|
|
readonly port: number;
|
|
readonly requests: readonly CodexRequest[];
|
|
// (undocumented)
|
|
readonly url: string;
|
|
}
|
|
|
|
// @public (undocumented)
|
|
export const codexModelMock: ModelMock;
|
|
|
|
// @public
|
|
export interface CodexRequest {
|
|
// (undocumented)
|
|
readonly model: string;
|
|
// (undocumented)
|
|
readonly prompt: string;
|
|
// (undocumented)
|
|
readonly toolNames: readonly string[];
|
|
}
|
|
|
|
// @public
|
|
export function codexRunError(out: {
|
|
stdout: string;
|
|
}): string | null;
|
|
|
|
// @public (undocumented)
|
|
export const codexRuntime: HarnessRuntime;
|
|
|
|
// @public
|
|
export function codexSkillFired(run: {
|
|
toolCalls: readonly ToolCall[];
|
|
}, name: string): boolean;
|
|
|
|
// @public
|
|
export interface CodexTurn {
|
|
readonly text: string;
|
|
}
|
|
|
|
// @public
|
|
export function installCodexSkills(pluginDir: string, cwd: string): number;
|
|
|
|
// @public
|
|
export function parseCodexEvalRun(out: {
|
|
stdout: string;
|
|
}): ParsedModelRun;
|
|
|
|
// @public
|
|
export function parseCodexRun(stdout: string): ParsedRun;
|
|
|
|
// @public
|
|
export function parseResponsesRequest(body: string): {
|
|
prompt: string;
|
|
model: string;
|
|
toolNames: string[];
|
|
};
|
|
|
|
// @public
|
|
export function refuseCodexEffort(effort: string | number | undefined): void;
|
|
|
|
// @public
|
|
export function renderResponsesSSE(text: string, opts?: {
|
|
model?: string;
|
|
}): string;
|
|
|
|
// @public
|
|
export function startCodexMock(script: readonly CodexTurn[], opts?: {
|
|
onRequest?: (req: CodexRequest) => void;
|
|
}): Promise<CodexMockHandle>;
|
|
|
|
// (No @packageDocumentation comment for this package)
|
|
|
|
```
|