* [Agents] Add Cursor CLI agent with direct API support
Add Cursor CLI as a new agent to the eval framework, enabling testing against Cursor models using direct API access via `CURSOR_API_KEY`. The implementation follows the established pattern from existing agents (Claude Code, Codex, Gemini) and integrates seamlessly with the sandbox infrastructure.
Cursor CLI uses a different interaction model than other agents. Unlike the `agent chat` subcommand syntax, the correct invocation is to pass the prompt as the first positional argument, then use the `--print` flag to enable non-interactive mode (required for scripts and headless execution). The `--force` flag is added to automatically approve all CLI operations without waiting for user confirmation. This combination ensures the agent completes successfully in sandboxed environments without hanging on interactive prompts.
The agent installation uses Cursor's official shell-based installation method (`curl https://cursor.com/install -fsSL | bash`), which handles proper setup of the CLI environment. Once installed, prompts are executed as `agent "<prompt>" --print --force`, capturing both stdout and stderr as output. The implementation includes transcript extraction that filters for JSON-formatted events when available, falling back to raw text output if no structured format is detected. This matches the approach used by other agents but adapts to Cursor's plain-text output characteristics.
The agent uses `composer-1.5` as the default model and integrates with all eval framework features including file capture, script validation, and test execution. Integration tests verify both basic eval execution and result output structure matching. The agent is fully typed in the framework's type system and added to the configuration schema alongside other direct-API agents.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Fix missing --model and --output-format flags, update env var docs
- Pass --model to Cursor CLI so user-configured model is actually used
- Add --output-format stream-json for structured JSONL transcripts
- Update changeset to reflect correct default model (composer-1.5)
- Add GEMINI_API_KEY and CURSOR_API_KEY to README env var table
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Rewrites the README to document all features added in v0.3.0:
default run-all behavior, --smoke/--force flags, content fingerprinting
and result reuse, AI failure classification, auto-retry, and
housekeeping. Fixes outdated result file paths and summary.json format.
New classifier.ts module classifies failed evals as model, infra, or
timeout. Rule-based fast path catches ~80% of cases (timeout errors,
missing transcripts, rate limits, tiny transcripts with API errors).
Falls back to AI classification via gateway(anthropic/claude-sonnet-4-5)
with sandboxed read-only tools. Gracefully degrades to rule-based only
when AI_GATEWAY_API_KEY is not set.
shouldRetry() returns true when all failures are non-model, enabling
auto-retry of infra-failed evals.
summary.json now supports classification and valid fields. Results
marked valid: false are excluded from fingerprint reuse.
OpenCode requires the vercel/ prefix in model strings (e.g., vercel/minimax/minimax-m2.1)
because the agent's opencode.json config sets up a vercel provider.
Also fixes Docker sandbox file permissions by adding chown after uploadFiles.
Add OpenCode as a new agent option for running evals. OpenCode uses
Vercel AI Gateway for model access and supports non-interactive
execution via `opencode run` command.
- Add src/lib/agents/opencode.ts with full agent implementation
- Register agent in index.ts as 'vercel-ai-gateway/opencode'
- Add type to AgentType union in types.ts
- Add validation in config.ts schema
- Add integration tests for OpenCode sandbox execution
- Update README.md with OpenCode documentation
- Add EVAL.tsx to TEST_FILE_PATTERNS and EXCLUDED_FILES
- Update fixture validation to accept either EVAL.ts or EVAL.tsx
- Remove *.test.tsx and *.test.ts from TEST_FILE_PATTERNS
- Document EVAL.ts vs EVAL.tsx convention in README with examples
- Bump version to 0.0.2
This allows using React Testing Library and other JSX-based testing
approaches in eval fixtures.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Update package name from agent-eval to @vercel/agent-eval
- Fix version test to read dynamically from package.json
- Update integration tests to load from .env.local
- Update README with @vercel/agent-eval usage
- Clarify Vercel employee setup instructions
- Fix lint error in runner.test.ts
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Rename package from @judegao/eval to agent-eval
- Reset version to 0.0.1
- Update all imports, templates, and docs to use new package name
- Point repository and git remote to vercel-labs/agent-eval
- Fix duplicate evals keys in README config example
- Rewrite environment variables docs with summary table
- Add Vercel employees section for vc env pull workflow
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
Remove the `eval` npm script from generated projects and update all
docs/tests to use `npx agent-eval` directly, eliminating the confusing
`--` separator needed with `npm run`. Also sync cli.ts version to 1.1.1
and fix stale integration test assertion for removed scripts field.
- Add support for direct ANTHROPIC_API_KEY and OPENAI_API_KEY authentication
- New agent naming: 'vercel-ai-gateway/claude-code' vs 'claude-code' for direct
- Change default model to opus
- Add ESLint configuration
- Add runner tests
- Clean up legacy agent.ts code
- Update README with dual authentication documentation
- Add Codex CLI agent alongside Claude Code
- Route all agents through Vercel AI Gateway (single API key)
- Extract shared agent utilities to reduce duplication
- Simplify model types (accept any string, let CLI validate)
- Remove list command (use --dry instead)
- Update README for framework authors and A/B testing
Update transcriptPath and outputPaths to use explicit relative paths
(./transcript.jsonl, ./outputs/*.txt) for clarity.
- Update results.ts path generation
- Update test expectations
- Update README documentation
- Bump version to 0.1.12
- outputs/ contains tests.txt, build.txt, etc. (not generated files)
- result.json includes paths: transcriptPath and outputPaths
- Update README to reflect correct output structure
- Bump version to 0.1.10
Allen Zhou