The Codex CLI defaults both `model_reasoning_effort` and `model_verbosity`
to "low", but `gpt-5.2-codex` (the default Codex model) only accepts
"medium" for both. Out-of-the-box `codex exec` against the AI Gateway
fails with:
Unsupported value: 'low' is not supported with the 'gpt-5.2-codex'
model. Supported values are: 'medium'.
The error covers both the `reasoning.effort` and `text.verbosity`
request parameters, depending on which the model rejects first.
Set both fields to "medium" in two places:
- the generated profile config in ~/.codex/default.config.toml
- explicit -c flags on `codex exec`, since CLI flags have the highest
precedence and we observed the profile-only setting being silently
overridden by the CLI's "low" default in some Codex versions.
`generateCodexConfig` now accepts an optional `reasoningEffort`
parameter so callers can override per-run via
`model: "gpt-5.2-codex?reasoningEffort=high"`.
Verified end-to-end against the Vercel AI Gateway: a previously-failing
`codex exec` smoke run now completes in ~31s and returns a real
response instead of erroring at `turn.failed`.
Also added `vercel-agent-eval-*.tgz` to .gitignore so local `npm pack`
artifacts don't leak into commits.
Allows experiment config files to pass agent-specific options (like
binaryUrl and extraProviders) at runtime via a new agentOptions field.
Previously these could only be set at agent registration time, making
configs for unreleased models non-replicable.
The options flow: ExperimentConfig → runner → AgentRunOptions → agent.run().
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Use built-in OpenAI provider for Codex
* Strip openai/ prefix from --model CLI flag for direct API path
The config.toml correctly strips the provider prefix for direct OpenAI API
usage, but the --model CLI flag still passed the prefixed name (e.g.
"openai/gpt-5.2-codex"), causing a "model not found" error.
* feat: support CLAUDE_CODE_OAUTH_TOKEN for Claude Code agent
Allow Claude Pro/Max subscribers to authenticate using their OAuth token
instead of requiring a separate ANTHROPIC_API_KEY. When CLAUDE_CODE_OAUTH_TOKEN
is set in the environment, it takes precedence over ANTHROPIC_API_KEY for
non-AI-Gateway configurations.
Closes#54
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: address PR review comments for OAuth token support
Refactor nested ternary for sandbox env to if/else for readability,
add clarifying comment about credential consistency, and add unit
tests for getApiKeyEnvVar() precedence (gateway > oauth > direct).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Classifier] Upgrade classification model to Claude Sonnet 4.6
* [Classifier] Parallelize classification with p-limit and add dashboard progress
Run up to 4 classifications concurrently using p-limit instead of a
sequential for-loop. The dashboard now shows "classifying N/M…" so
users can track per-eval progress during the classification phase.
* [Classifier] Switch to Haiku 4.5 for classification
Haiku is faster and cheaper while still capable enough for the
classification task, especially with 4 concurrent classifications.
* [Runner] Retry eval attempts on 429 rate limiting with exponential backoff
* Add changeset
* Add StartRateLimiter and switch to anomaly-based retry detection
- Add StartRateLimiter class to throttle sandbox starts across experiments
(20 starts per 2s window) to prevent 429s at the source
- Replace 429 string matching with anomaly detection: retry any failure
that completes in <5s, since real evals take minutes
- Move progress events (eval:start, eval:complete, earlyExit) outside
runAttempt so they fire correctly after retries
* Fix runner tests: exclude timeouts from retry, bump mock durations above anomaly threshold
Removes a debug console.log statement that was left in the saveResults function. The statement was logging internal state (copyFiles, hasGeneratedFiles, hasDeletedFiles, options, and runData) during result persistence, which is unnecessary noise in production output.
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
github-actions[bot]