Collected project trees were not faithful copies. Two independent paths
decoded every file as UTF-8, replacing each invalid byte with U+FFFD:
- captureGeneratedFiles read the agent's diff via sandbox.readFile, which
hands back command stdout -- a string the sandbox API decodes as UTF-8.
- readFixtureFiles read fixtures with 'utf-8', corrupting every binary
asset a fixture ships when copyFiles: 'all' is used.
Both failures are silent: no exception, no warning, and text-only projects
never show them. Binary files were destroyed and inflated ~1.8x.
Sandbox gains readFileBuffer(path), routed through `base64` so the existing
transport round-trips losslessly, and copyFiles: 'all' now uses copyFileSync
instead of read-then-write -- bytes never enter the heap, so a whole fixture
tree is no longer buffered in memory.
BREAKING: generatedFiles is now Record<string, Buffer> on AgentRunResult and
EvalRunData; readFixtureFiles returns Map<string, Buffer>.
Closes#176
Co-Authored-By: Claude <noreply@anthropic.com>
An explicit --model like anthropic/claude-sonnet-5 was passed to the
OpenCode CLI verbatim, which reads the first segment as its provider id.
The generated opencode.json only configures the vercel (AI Gateway)
provider, so every explicit-model run using a canonical gateway id died
at session start with "Unexpected server error" and a null
observedModel.
- resolveOpenCodeModel (host-side): prefix vercel/ unless the caller
already targets vercel/... or a configured extraProviders key.
- Threaded to the runner via runnerExtra -> input.extra.cliModel,
mirroring codex's host-computed model (the judge path ships extra but
not agentOptions, so the resolution must happen host-side).
- normalizeObservedModel (runner-side): un-apply exactly that prefix on
the observation, so observedModel === requestedModel holds for
canonical gateway ids and a gateway substitution still surfaces as a
clean gateway id. Native-default observations are untouched, and the
documented vercel/-prefixed form still passes through verbatim.
- README: the OpenCode model format section now documents both forms
(it previously stated unprefixed ids fail with provider-not-found).
Adopting a changed or new eval shouldn't force re-running every model. This adds a
small, explicit workflow and fixes the fingerprinting that made it impossible.
- Fingerprint split: results store a content-only hash alongside the combined
(content+config) one. A real eval change is never masked; a benign config change
(timeout bump, pinning a judge) is carried forward by `refingerprint` instead of
re-running. Existing `fingerprint` values are byte-identical.
- `agent-eval status` — read-only: new vs changed evals, per experiment. `--check`
exits non-zero on any new/changed eval; `--json` emits per-experiment new/changed.
- `agent-eval run <experiments...>` — run the named experiments' new/changed evals.
- Bare `agent-eval` shows status, then lets you multi-select what to run. Removes
`run-all` and `--dry`. No in-framework accept/keep — staleness acceptance is the
consumer's policy (filter `status --json` in CI).
Stacked on #164.
Judge the final state or the transcript from inside EVAL.ts, reusing the
SAME agent + sandbox as codegen (no fresh sandbox, no copied evidence):
import { environment, transcript } from '@vercel/agent-eval/eval';
await expect(environment).toSatisfyCriterion('uses Server Components');
await expect(transcript).toSatisfyCriterion('used DevTools, not guesswork');
await expect(environment).toScoreAtLeast('code quality', 0.8);
- eval-helper.mjs: shipped in-sandbox (aliased to @vercel/agent-eval/eval +
vitest setup file); each assertion re-invokes __agent_eval__/run.mjs with a
judge prompt and parses the verdict. You give only the criterion.
- orchestrator/shared: ship the helper, materialize the raw transcript to a
file (read by path, never dumped into a prompt), record judge-config (same
model), pass auth env to validation, gitignore __agent_eval__/.
- Failures are attributable: '[judge:environment] FAIL (score): reason'.
Verified: unit tests + real e2e (Vercel sandbox + Claude via gateway) — judge
confirms true criteria (eval passes) and rejects a false one (eval fails).
* [Agents] Add Cursor CLI agent with direct API support
Add Cursor CLI as a new agent to the eval framework, enabling testing against Cursor models using direct API access via `CURSOR_API_KEY`. The implementation follows the established pattern from existing agents (Claude Code, Codex, Gemini) and integrates seamlessly with the sandbox infrastructure.
Cursor CLI uses a different interaction model than other agents. Unlike the `agent chat` subcommand syntax, the correct invocation is to pass the prompt as the first positional argument, then use the `--print` flag to enable non-interactive mode (required for scripts and headless execution). The `--force` flag is added to automatically approve all CLI operations without waiting for user confirmation. This combination ensures the agent completes successfully in sandboxed environments without hanging on interactive prompts.
The agent installation uses Cursor's official shell-based installation method (`curl https://cursor.com/install -fsSL | bash`), which handles proper setup of the CLI environment. Once installed, prompts are executed as `agent "<prompt>" --print --force`, capturing both stdout and stderr as output. The implementation includes transcript extraction that filters for JSON-formatted events when available, falling back to raw text output if no structured format is detected. This matches the approach used by other agents but adapts to Cursor's plain-text output characteristics.
The agent uses `composer-1.5` as the default model and integrates with all eval framework features including file capture, script validation, and test execution. Integration tests verify both basic eval execution and result output structure matching. The agent is fully typed in the framework's type system and added to the configuration schema alongside other direct-API agents.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Fix missing --model and --output-format flags, update env var docs
- Pass --model to Cursor CLI so user-configured model is actually used
- Add --output-format stream-json for structured JSONL transcripts
- Update changeset to reflect correct default model (composer-1.5)
- Add GEMINI_API_KEY and CURSOR_API_KEY to README env var table
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Rewrites the README to document all features added in v0.3.0:
default run-all behavior, --smoke/--force flags, content fingerprinting
and result reuse, AI failure classification, auto-retry, and
housekeeping. Fixes outdated result file paths and summary.json format.
New classifier.ts module classifies failed evals as model, infra, or
timeout. Rule-based fast path catches ~80% of cases (timeout errors,
missing transcripts, rate limits, tiny transcripts with API errors).
Falls back to AI classification via gateway(anthropic/claude-sonnet-4-5)
with sandboxed read-only tools. Gracefully degrades to rule-based only
when AI_GATEWAY_API_KEY is not set.
shouldRetry() returns true when all failures are non-model, enabling
auto-retry of infra-failed evals.
summary.json now supports classification and valid fields. Results
marked valid: false are excluded from fingerprint reuse.
OpenCode requires the vercel/ prefix in model strings (e.g., vercel/minimax/minimax-m2.1)
because the agent's opencode.json config sets up a vercel provider.
Also fixes Docker sandbox file permissions by adding chown after uploadFiles.
Add OpenCode as a new agent option for running evals. OpenCode uses
Vercel AI Gateway for model access and supports non-interactive
execution via `opencode run` command.
- Add src/lib/agents/opencode.ts with full agent implementation
- Register agent in index.ts as 'vercel-ai-gateway/opencode'
- Add type to AgentType union in types.ts
- Add validation in config.ts schema
- Add integration tests for OpenCode sandbox execution
- Update README.md with OpenCode documentation
- Add EVAL.tsx to TEST_FILE_PATTERNS and EXCLUDED_FILES
- Update fixture validation to accept either EVAL.ts or EVAL.tsx
- Remove *.test.tsx and *.test.ts from TEST_FILE_PATTERNS
- Document EVAL.ts vs EVAL.tsx convention in README with examples
- Bump version to 0.0.2
This allows using React Testing Library and other JSX-based testing
approaches in eval fixtures.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Update package name from agent-eval to @vercel/agent-eval
- Fix version test to read dynamically from package.json
- Update integration tests to load from .env.local
- Update README with @vercel/agent-eval usage
- Clarify Vercel employee setup instructions
- Fix lint error in runner.test.ts
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Rename package from @judegao/eval to agent-eval
- Reset version to 0.0.1
- Update all imports, templates, and docs to use new package name
- Point repository and git remote to vercel-labs/agent-eval
- Fix duplicate evals keys in README config example
- Rewrite environment variables docs with summary table
- Add Vercel employees section for vc env pull workflow
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
Remove the `eval` npm script from generated projects and update all
docs/tests to use `npx agent-eval` directly, eliminating the confusing
`--` separator needed with `npm run`. Also sync cli.ts version to 1.1.1
and fix stale integration test assertion for removed scripts field.
- Add support for direct ANTHROPIC_API_KEY and OPENAI_API_KEY authentication
- New agent naming: 'vercel-ai-gateway/claude-code' vs 'claude-code' for direct
- Change default model to opus
- Add ESLint configuration
- Add runner tests
- Clean up legacy agent.ts code
- Update README with dual authentication documentation
molebox