New classifier.ts module classifies failed evals as model, infra, or
timeout. Rule-based fast path catches ~80% of cases (timeout errors,
missing transcripts, rate limits, tiny transcripts with API errors).
Falls back to AI classification via gateway(anthropic/claude-sonnet-4-5)
with sandboxed read-only tools. Gracefully degrades to rule-based only
when AI_GATEWAY_API_KEY is not set.
shouldRetry() returns true when all failures are non-model, enabling
auto-retry of infra-failed evals.
summary.json now supports classification and valid fields. Results
marked valid: false are excluded from fingerprint reuse.
OpenCode requires the vercel/ prefix in model strings (e.g., vercel/minimax/minimax-m2.1)
because the agent's opencode.json config sets up a vercel provider.
Also fixes Docker sandbox file permissions by adding chown after uploadFiles.
Add OpenCode as a new agent option for running evals. OpenCode uses
Vercel AI Gateway for model access and supports non-interactive
execution via `opencode run` command.
- Add src/lib/agents/opencode.ts with full agent implementation
- Register agent in index.ts as 'vercel-ai-gateway/opencode'
- Add type to AgentType union in types.ts
- Add validation in config.ts schema
- Add integration tests for OpenCode sandbox execution
- Update README.md with OpenCode documentation
- Add EVAL.tsx to TEST_FILE_PATTERNS and EXCLUDED_FILES
- Update fixture validation to accept either EVAL.ts or EVAL.tsx
- Remove *.test.tsx and *.test.ts from TEST_FILE_PATTERNS
- Document EVAL.ts vs EVAL.tsx convention in README with examples
- Bump version to 0.0.2
This allows using React Testing Library and other JSX-based testing
approaches in eval fixtures.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Update package name from agent-eval to @vercel/agent-eval
- Fix version test to read dynamically from package.json
- Update integration tests to load from .env.local
- Update README with @vercel/agent-eval usage
- Clarify Vercel employee setup instructions
- Fix lint error in runner.test.ts
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Rename package from @judegao/eval to agent-eval
- Reset version to 0.0.1
- Update all imports, templates, and docs to use new package name
- Point repository and git remote to vercel-labs/agent-eval
- Fix duplicate evals keys in README config example
- Rewrite environment variables docs with summary table
- Add Vercel employees section for vc env pull workflow
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
Remove the `eval` npm script from generated projects and update all
docs/tests to use `npx agent-eval` directly, eliminating the confusing
`--` separator needed with `npm run`. Also sync cli.ts version to 1.1.1
and fix stale integration test assertion for removed scripts field.
- Add support for direct ANTHROPIC_API_KEY and OPENAI_API_KEY authentication
- New agent naming: 'vercel-ai-gateway/claude-code' vs 'claude-code' for direct
- Change default model to opus
- Add ESLint configuration
- Add runner tests
- Clean up legacy agent.ts code
- Update README with dual authentication documentation
- Add Codex CLI agent alongside Claude Code
- Route all agents through Vercel AI Gateway (single API key)
- Extract shared agent utilities to reduce duplication
- Simplify model types (accept any string, let CLI validate)
- Remove list command (use --dry instead)
- Update README for framework authors and A/B testing
Update transcriptPath and outputPaths to use explicit relative paths
(./transcript.jsonl, ./outputs/*.txt) for clarity.
- Update results.ts path generation
- Update test expectations
- Update README documentation
- Bump version to 0.1.12
- outputs/ contains tests.txt, build.txt, etc. (not generated files)
- result.json includes paths: transcriptPath and outputPaths
- Update README to reflect correct output structure
- Bump version to 0.1.10
Jude Gao