Allows experiment config files to pass agent-specific options (like
binaryUrl and extraProviders) at runtime via a new agentOptions field.
Previously these could only be set at agent registration time, making
configs for unreleased models non-replicable.
The options flow: ExperimentConfig → runner → AgentRunOptions → agent.run().
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Use built-in OpenAI provider for Codex
* Strip openai/ prefix from --model CLI flag for direct API path
The config.toml correctly strips the provider prefix for direct OpenAI API
usage, but the --model CLI flag still passed the prefixed name (e.g.
"openai/gpt-5.2-codex"), causing a "model not found" error.
* feat: support CLAUDE_CODE_OAUTH_TOKEN for Claude Code agent
Allow Claude Pro/Max subscribers to authenticate using their OAuth token
instead of requiring a separate ANTHROPIC_API_KEY. When CLAUDE_CODE_OAUTH_TOKEN
is set in the environment, it takes precedence over ANTHROPIC_API_KEY for
non-AI-Gateway configurations.
Closes#54
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: address PR review comments for OAuth token support
Refactor nested ternary for sandbox env to if/else for readability,
add clarifying comment about credential consistency, and add unit
tests for getApiKeyEnvVar() precedence (gateway > oauth > direct).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Classifier] Upgrade classification model to Claude Sonnet 4.6
* [Classifier] Parallelize classification with p-limit and add dashboard progress
Run up to 4 classifications concurrently using p-limit instead of a
sequential for-loop. The dashboard now shows "classifying N/M…" so
users can track per-eval progress during the classification phase.
* [Classifier] Switch to Haiku 4.5 for classification
Haiku is faster and cheaper while still capable enough for the
classification task, especially with 4 concurrent classifications.
* [Runner] Retry eval attempts on 429 rate limiting with exponential backoff
* Add changeset
* Add StartRateLimiter and switch to anomaly-based retry detection
- Add StartRateLimiter class to throttle sandbox starts across experiments
(20 starts per 2s window) to prevent 429s at the source
- Replace 429 string matching with anomaly detection: retry any failure
that completes in <5s, since real evals take minutes
- Move progress events (eval:start, eval:complete, earlyExit) outside
runAttempt so they fire correctly after retries
* Fix runner tests: exclude timeouts from retry, bump mock durations above anomaly threshold
Allen Zhou