* [Classifier] Upgrade classification model to Claude Sonnet 4.6
* [Classifier] Parallelize classification with p-limit and add dashboard progress
Run up to 4 classifications concurrently using p-limit instead of a
sequential for-loop. The dashboard now shows "classifying N/M…" so
users can track per-eval progress during the classification phase.
* [Classifier] Switch to Haiku 4.5 for classification
Haiku is faster and cheaper while still capable enough for the
classification task, especially with 4 concurrent classifications.
* [Runner] Retry eval attempts on 429 rate limiting with exponential backoff
* Add changeset
* Add StartRateLimiter and switch to anomaly-based retry detection
- Add StartRateLimiter class to throttle sandbox starts across experiments
(20 starts per 2s window) to prevent 429s at the source
- Replace 429 string matching with anomaly detection: retry any failure
that completes in <5s, since real evals take minutes
- Move progress events (eval:start, eval:complete, earlyExit) outside
runAttempt so they fire correctly after retries
* Fix runner tests: exclude timeouts from retry, bump mock durations above anomaly threshold
Removes a debug console.log statement that was left in the saveResults function. The statement was logging internal state (copyFiles, hasGeneratedFiles, hasDeletedFiles, options, and runData) during result persistence, which is unnecessary noise in production output.
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
* [O11y] Add transcript parsers for Gemini and Cursor agents
* Handle Gemini direct-API transcript format
The Gemini agent has two output formats: the CLI format (via OpenCode
framework with part.tool/part.state) and the direct-API format (with
tool_name/parameters and separate tool_result events). The initial
parser only handled the CLI format, leaving direct-API transcripts
with all-zero metrics. This adds handling for both.
* Aggregate Gemini direct-API delta messages into single turns
Contiguous assistant delta messages are now merged into one message
event so totalTurns reflects actual conversation turns instead of 0.
* Add changeset
* [Agents] Add Cursor CLI agent with direct API support
Add Cursor CLI as a new agent to the eval framework, enabling testing against Cursor models using direct API access via `CURSOR_API_KEY`. The implementation follows the established pattern from existing agents (Claude Code, Codex, Gemini) and integrates seamlessly with the sandbox infrastructure.
Cursor CLI uses a different interaction model than other agents. Unlike the `agent chat` subcommand syntax, the correct invocation is to pass the prompt as the first positional argument, then use the `--print` flag to enable non-interactive mode (required for scripts and headless execution). The `--force` flag is added to automatically approve all CLI operations without waiting for user confirmation. This combination ensures the agent completes successfully in sandboxed environments without hanging on interactive prompts.
The agent installation uses Cursor's official shell-based installation method (`curl https://cursor.com/install -fsSL | bash`), which handles proper setup of the CLI environment. Once installed, prompts are executed as `agent "<prompt>" --print --force`, capturing both stdout and stderr as output. The implementation includes transcript extraction that filters for JSON-formatted events when available, falling back to raw text output if no structured format is detected. This matches the approach used by other agents but adapts to Cursor's plain-text output characteristics.
The agent uses `composer-1.5` as the default model and integrates with all eval framework features including file capture, script validation, and test execution. Integration tests verify both basic eval execution and result output structure matching. The agent is fully typed in the framework's type system and added to the configuration schema alongside other direct-API agents.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Fix missing --model and --output-format flags, update env var docs
- Pass --model to Cursor CLI so user-configured model is actually used
- Add --output-format stream-json for structured JSONL transcripts
- Update changeset to reflect correct default model (composer-1.5)
- Add GEMINI_API_KEY and CURSOR_API_KEY to README env var table
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
`npx agent-eval` with no args now discovers and runs all experiments
(same as `agent-eval run-all`). Running a single experiment still
works: `npx agent-eval my-experiment`.
New run-all command auto-discovers experiments/*.ts and orchestrates
the full pipeline: fingerprint reuse, failure classification,
auto-retry of infra failures, and housekeeping.
Housekeeping runs after each experiment: keeps only the latest valid
result per (experiment, eval) pair, removes duplicates and incomplete
results, cleans up empty timestamp directories.
Supports --dry (preview), --force (ignore fingerprints), --smoke
(1 eval per experiment), and optional experiment name/glob arguments.
New classifier.ts module classifies failed evals as model, infra, or
timeout. Rule-based fast path catches ~80% of cases (timeout errors,
missing transcripts, rate limits, tiny transcripts with API errors).
Falls back to AI classification via gateway(anthropic/claude-sonnet-4-5)
with sandboxed read-only tools. Gracefully degrades to rule-based only
when AI_GATEWAY_API_KEY is not set.
shouldRetry() returns true when all failures are non-model, enabling
auto-retry of infra-failed evals.
summary.json now supports classification and valid fields. Results
marked valid: false are excluded from fingerprint reuse.
Computes a SHA-256 fingerprint from all eval directory files plus the
config fields that affect results (agent, model, scripts, timeout,
earlyExit, runs). The fingerprint is stored in summary.json.
scanReusableResults() scans existing results and returns matches where
the fingerprint is identical, the result is valid, and passedRuns > 0.
Functions like setup and editPrompt can't be hashed — use --force to
re-run when these change.
Picks the first eval alphabetically and runs it once per model.
Useful for verifying API keys, model IDs, and sandbox connectivity
before committing to a full suite, especially with model arrays.
When `scripts: ['eval']` is configured, the script output was written
to `outputs/eval.txt` which collided with the EVAL.ts test output at
the same path. Script outputs now go to `outputs/scripts/{name}.txt`.
github-actions[bot]