`claude --effort` sets the reasoning budget per turn. vigiles could not set it,
so an A/B could not reproduce a benchmark that pins one: the arm you ran was not
the arm the author measured (reported in #189, with a real case — a skill whose
published results pin "effort low").
Effort is a MEASUREMENT, not a run knob: it moves the output distribution, not
the sample size. So it lives on the spec beside `model` (never an env), it is an
arm in its own right, and it is hashed into BOTH the eval lock and the local
cache — the reporter named only the lock; the cache key carried `model` without
effort in exactly the same way, and fixing one would have left half the hole.
Three things that are not in the reported shape:
- THE ENV PIN. Effort has three inputs — the flag, the `effortLevel` settings
key, and CLAUDE_CODE_EFFORT_LEVEL — and the env var outranks the flag, while
EPHEMERAL_ALLOW_PREFIXES passes `CLAUDE_*` through by design. So an ambient
value survives even the scrubbed ephemeral env. Hashing effort WITHOUT pinning
it would make the lock confidently wrong: `low` recorded over a run that
executed at the shell's `max` — the defect this feature exists to prevent,
reintroduced by the fix for it. Declared sets the var; OMITTED deletes an
inherited one, so "omit" means the harness default rather than whatever the
machine exported.
- NOT A LITERAL UNION. The binary is case-insensitive, keeps an alias map, and
accepts an integer budget; its valid set already MOVED between builds (2.1.42
had no `xhigh`). A hard-coded union would reject a valid level after any
upstream addition. The silent fallback a union was meant to prevent is not
silent — the binary warns on stderr, which is already captured — so the wrong
value is caught at runtime by `effortRejection`, and a rejected value THROWS
rather than counting as `runError`: an unusable value is deterministic, so
every trial fails it and the report would be a rate over zero samples.
- `harness-test.ts` IS NOT A SITE. That tier runs a scripted mock with a
hard-coded model; effort has no observable effect there. The issue names it;
adding the flag would have been dead argv.
Codex refuses a declared effort loudly instead of dropping it — no mapping has
been measured, and silently recording a budget the run did not use is worse than
refusing. `audit` deliberately gets no CLI flag: it is a local report, not a
reproducibility surface.
`buildAgentArgs` is extracted so the flag is provable at all — the argv was built
inside the `v8 ignore`d spawn, where nothing could assert it.
Measured: `npm run check` 18/18 in 73s; 3462 unit tests, 0 failures.
Eight mutations, each proven to have LANDED. Seven fail on their own assertion.
The eighth stays green and that is a finding about the TEST, recorded in place:
effort reaches the lock hash through two fills (the `withEvalLock` chokepoint and
the per-arm inputs), so removing either alone is green and removing BOTH fails —
overlap across two populations, not a dead line.
One regression caught by the full suite and fixed here: `withEffortGuard` was
written `async`, which converts the wrapped runner's SYNCHRONOUS refusals into
rejected promises — downgrading the foreign-test-runner guard that stops a paid
eval from billing. Now pinned by a test.
Claude-Session: https://claude.ai/code/session_01PrS34j2M8XR12P2QHaBer4
Co-authored-by: Claude <noreply@anthropic.com>
The Codex-experimental caveat added in #79 introduced additive public API
(CODEX_TRIGGER_RATE_EXPERIMENTAL on vigiles/codex; optional `experimental?`
fields on EvalDriver and TriggerRateReport on vigiles/testing) but the
committed api-surface snapshots were never regenerated, so the API surface
gate has been failing on main. Regenerate to match the shipped surface.
All additive — no exports removed, not a breaking change.
Co-authored-by: Claude <noreply@anthropic.com>
zernie