- Change wire_api from "chat" to "responses" (required by newer Codex CLI)
- Fix direct OpenAI API auth by using codex login --with-api-key
- Fix model name for direct API (remove openai/ prefix)
- Update integration tests to assert agent success (not just completion)
- Fix test isolation with beforeAll hooks for project initialization
Codex via Vercel AI Gateway still has a separate tools.7.type API error
that needs to be fixed on the AI Gateway side.
Adds Docker as a sandbox backend for running evals locally without
requiring Vercel credentials.
Features:
- Auto-detection: uses Vercel if VERCEL_TOKEN present, else Docker
- Explicit override via SANDBOX_BACKEND=docker|vercel env var
- Non-root execution (uid 1000) for security and Claude Code compatibility
- npm global installs configured for non-root user
New files:
- src/lib/docker-sandbox.ts - Docker sandbox implementation
- src/lib/docker-sandbox.test.ts - Integration tests
- scripts/test-docker-sandbox.ts - Manual test script
- scripts/test-docker-eval-flow.ts - Full eval flow test
- DOCKER.md - Beginner-friendly setup guide
API additions:
- createSandbox() - Factory function with auto-detection
- resolveBackend() - Determines which backend to use
- getSandboxBackendInfo() - Returns backend info for display
New npm scripts:
- test:sandbox - Quick Docker sandbox test
- test:docker - Docker unit tests
- test:docker:integration - Full integration tests with Docker
- Add EVAL.tsx to TEST_FILE_PATTERNS and EXCLUDED_FILES
- Update fixture validation to accept either EVAL.ts or EVAL.tsx
- Remove *.test.tsx and *.test.ts from TEST_FILE_PATTERNS
- Document EVAL.ts vs EVAL.tsx convention in README with examples
- Bump version to 0.0.2
This allows using React Testing Library and other JSX-based testing
approaches in eval fixtures.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Update package name from agent-eval to @vercel/agent-eval
- Fix version test to read dynamically from package.json
- Update integration tests to load from .env.local
- Update README with @vercel/agent-eval usage
- Clarify Vercel employee setup instructions
- Fix lint error in runner.test.ts
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Rename package from @judegao/eval to agent-eval
- Reset version to 0.0.1
- Update all imports, templates, and docs to use new package name
- Point repository and git remote to vercel-labs/agent-eval
- Fix duplicate evals keys in README config example
- Rewrite environment variables docs with summary table
- Add Vercel employees section for vc env pull workflow
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
- Run all (fixtures × runs) concurrently via Promise.all
- Add AbortController per fixture for earlyExit behavior
- When one attempt passes, abort remaining in-flight attempts
- Add signal?: AbortSignal to AgentRunOptions
- Agents stop sandbox when abort signal fires
- Aborted results excluded from final counts
- Results sorted by runIndex, count only up to first pass with earlyExit
Remove the `eval` npm script from generated projects and update all
docs/tests to use `npx agent-eval` directly, eliminating the confusing
`--` separator needed with `npm run`. Also sync cli.ts version to 1.1.1
and fix stale integration test assertion for removed scripts field.
- Add support for direct ANTHROPIC_API_KEY and OPENAI_API_KEY authentication
- New agent naming: 'vercel-ai-gateway/claude-code' vs 'claude-code' for direct
- Change default model to opus
- Add ESLint configuration
- Add runner tests
- Clean up legacy agent.ts code
- Update README with dual authentication documentation
- Add Codex CLI agent alongside Claude Code
- Route all agents through Vercel AI Gateway (single API key)
- Extract shared agent utilities to reduce duplication
- Simplify model types (accept any string, let CLI validate)
- Remove list command (use --dry instead)
- Update README for framework authors and A/B testing
Update transcriptPath and outputPaths to use explicit relative paths
(./transcript.jsonl, ./outputs/*.txt) for clarity.
- Update results.ts path generation
- Update test expectations
- Update README documentation
- Bump version to 0.1.12
- outputs/ contains tests.txt, build.txt, etc. (not generated files)
- result.json includes paths: transcriptPath and outputPaths
- Update README to reflect correct output structure
- Bump version to 0.1.10
- Save transcript.jsonl separately (not in result.json)
- result.json contains only: status, failedStep, error, duration
- summary.json uses design format: passRate as string, no runs array
- Create outputs/ directory per run
- Capture Claude Code transcript from sandbox
- Update EvalRunResult/EvalRunData types
- Bump version to 0.1.7
- Add @types/node and vitest to generated project devDependencies
- Add root tsconfig.json to generated projects
- Add JSDoc @default annotations to ExperimentConfig fields
- Add typecheck test to verify IDE experience works after init
- Bump version to 0.1.5
- Rename package from @vercel/eval-framework to @judegao/eval
- Update all references in init templates and docs
- Fix unused SandboxFile import in agent.ts
- Fix Sandbox interface to include options parameter
- Use EXCLUDED_FILES constant in fixture.ts
- Use dirname() instead of join(path, '..') in init.ts
- Make experimentConfigSchema private in config.ts
- Remove unused existsSync import in results.ts
- Consolidate duplicate imports in cli.ts
- Use TEST_FILE_PATTERNS constant in sandbox.ts
- Fix JSDoc comment ordering in sandbox.ts
CLI commands:
- `eval init <name>` - placeholder for project scaffolding
- `eval run <config>` - run experiments with dry run support
- `eval list` - list available eval fixtures
Fixture discovery:
- Discover fixtures in evals/ directory
- Validate required files (PROMPT.md, EVAL.ts, package.json)
- Validate package.json has type: module
- Load fixture contents and prompt text
- Get/read fixture files with exclusion patterns
Includes comprehensive tests for all functionality.
Jude Gao