Adopting a changed or new eval shouldn't force re-running every model. This adds a
small, explicit workflow and fixes the fingerprinting that made it impossible.
- Fingerprint split: results store a content-only hash alongside the combined
(content+config) one. A real eval change is never masked; a benign config change
(timeout bump, pinning a judge) is carried forward by `refingerprint` instead of
re-running. Existing `fingerprint` values are byte-identical.
- `agent-eval status` — read-only: new vs changed evals, per experiment. `--check`
exits non-zero on any new/changed eval; `--json` emits per-experiment new/changed.
- `agent-eval run <experiments...>` — run the named experiments' new/changed evals.
- Bare `agent-eval` shows status, then lets you multi-select what to run. Removes
`run-all` and `--dry`. No in-framework accept/keep — staleness acceptance is the
consumer's policy (filter `status --json` in CI).
Stacked on #164.
Judge the final state or the transcript from inside EVAL.ts, reusing the
SAME agent + sandbox as codegen (no fresh sandbox, no copied evidence):
import { environment, transcript } from '@vercel/agent-eval/eval';
await expect(environment).toSatisfyCriterion('uses Server Components');
await expect(transcript).toSatisfyCriterion('used DevTools, not guesswork');
await expect(environment).toScoreAtLeast('code quality', 0.8);
- eval-helper.mjs: shipped in-sandbox (aliased to @vercel/agent-eval/eval +
vitest setup file); each assertion re-invokes __agent_eval__/run.mjs with a
judge prompt and parses the verdict. You give only the criterion.
- orchestrator/shared: ship the helper, materialize the raw transcript to a
file (read by path, never dumped into a prompt), record judge-config (same
model), pass auth env to validation, gitignore __agent_eval__/.
- Failures are attributable: '[judge:environment] FAIL (score): reason'.
Verified: unit tests + real e2e (Vercel sandbox + Claude via gateway) — judge
confirms true criteria (eval passes) and rejects a false one (eval fails).
Review cleanups (no behavior change):
- contract.ts: drop dead AgentRunResult re-export; fix wrong
'editPrompt'd by the runner' comment; de-dup the extra/runnerExtra docs.
- codex/agent.ts: extract parseOptionsModel() — configFiles() and
runnerExtra() shared the same parse (DRY).
- run.mjs headers: honest, terse export rationale (the old wording claimed
helpers were test-verified when codex/gemini/cursor have none).
Port the remaining four adapters onto the generic orchestrator: each becomes a
host-side definition (agent.ts) + an in-sandbox runner (run.mjs); the old
single-file adapters are deleted. The Agent interface + registry are unchanged.
- codex: host-computed values (resolved --model, reasoning effort, verbosity from
parseModelString) must match the TOML profile, so AgentDefinition gains an
optional runnerExtra(options) threaded to the runner as input.extra (the only
contract addition). generateCodexConfig/parseModelString stay host-side + exported.
login is a separate spawn piping the key on stdin (no secret in a shell string);
the && short-circuit is preserved.
- opencode: opencode.json config + log-scrape then `opencode export` observedModel
fallback chain preserved; generators stay host-side + exported.
- gemini/cursor: stdout-only stream-json, no config, observedModel null. cursor
binary is 'agent', prompt-first arg order.
Adversarially reviewed each plugin vs its origin/main adapter: zero critical/major
findings. Live-validated end to end against a real Vercel sandbox + AI Gateway:
claude 7/7 and codex 6/6 integration tests pass. 216 unit + lint + build green.
Introduce the agent plugin contract + a single generic host orchestrator, and
migrate claude-code onto it as the first vertical slice.
Each agent is now: a host-side DEFINITION (agent.ts: install/configFiles/authEnv/
getApiKeyEnvVar/defaultModel) + an in-sandbox RUNNER (run.mjs: invoke the CLI +
capture the transcript, returning a typed RunnerResult). The generic
runWithDefinition() drives the shared, agent-agnostic flow (sandbox lifecycle,
git baseline, neutral workspace, validation, generated-file capture, o11y parse,
abort/timeout) — reproducing the old claude-code.ts run() exactly.
run.mjs is self-contained (only node:*), dual-mode (importable + runnable), and
exports its pure helpers so host tests verify exactly what the sandbox runs. The
Agent interface is unchanged, so registry/runner/results are untouched.
- plugin/contract.ts: AgentDefinition, AgentRunInput, RunnerResult (+ InstallStep/ConfigFile)
- plugin/orchestrator.ts: runWithDefinition() — the generic run()
- claude-code/agent.ts + claude-code/run.mjs; index.ts points at the new path
- build: scripts/copy-runners.mjs ships each run.mjs into dist
- claude-code.test.ts: import paths updated (helpers from run.mjs, factory from agent.ts)
Host<->sandbox result contract: run.mjs writes __agent_eval__/agent-result.json
(+ a __AGENT_RESULT__ stdout marker fallback); host reads it, splitting agent
failure (ok) from runner crash (node exit). Validated end-to-end against a real
Vercel sandbox + AI Gateway: 7/7 claude integration tests pass (gateway + direct
API + parallel + invalid-model error path); 216 unit tests + lint green.
The comma-separated token from #150 prevents the tools list itself from
splitting, but --allowedTools is variadic: it keeps capturing positional
tokens until the next flag. With the prompt directly after the value,
claude 2.1.112 consumed it as another tool name and failed with 'Input
must be provided either through stdin or as a prompt argument when using
--print' — caught by a live a0-local smoke run, invisible to the unit
tests because they asserted the broken order.
Emit --allowedTools first so the always-present
--dangerously-skip-permissions terminates the variadic capture before
the trailing prompt. Verified live: the reordered invocation accepts the
prompt. Default-off argument construction is byte-identical. New test
asserts the token after the allowedTools value is always a flag.
Conditional include (same pattern as native-default modelPolicy) so
default-off fingerprints are byte-identical to existing releases, while
research and non-research configs never share a fingerprint — result
reuse must not serve a cached parametric-only result for a research
run, or vice versa.
The adapters read options.webResearch, but runExperiment built
AgentRunOptions from an explicit field list that never included it, so
the option was unreachable for experiment-config consumers (a0-local
calls runExperiment, not executeAgent). Adds the field to
ExperimentConfig/ResolvedExperimentConfig/RunnableExperimentConfig, the
zod schema (z.object strips unknown keys, so schema membership is
required for validateConfig not to drop it), resolveConfig, and both
agent.run call sites (runExperiment and runSingleEval).
Still default-off: absent config yields webResearch: undefined, which
leaves every adapter branch untaken.
Safe redo of #141 (reverted in #144). webResearch defaults to false, so
command construction is byte-identical for existing consumers; coding
evals are unaffected unless they opt in.
The #141 breakage is fixed and regression-tested: Claude Code's
--allowedTools is variadic, so WebSearch/WebFetch are passed as a single
comma-separated value instead of separate tokens that consumed the
trailing positional prompt.
Verified against AI Gateway with live spikes: Claude Code WebSearch
executes (tool_use/tool_result events), OpenCode Exa websearch executes,
and Codex researches via shell even though no web_search items appear
through the responses wire (setting kept for direct-OpenAI runs and
future gateway support).
OpenCode 1.17.0 rewrote its logging pipeline and removed the
service=llm log lines the adapter scraped for providerID/modelID,
so native-default runs silently lost model observation.
Fall back to 'opencode export <sessionID>' when log scraping yields
nothing: the session id comes from the --format json event stream and
the exported assistant message carries providerID/modelID. Observation
never fails the run. The log scrape stays as the first, cheaper source
for OpenCode <= 1.16.x.
The Codex CLI defaults both `model_reasoning_effort` and `model_verbosity`
to "low", but `gpt-5.2-codex` (the default Codex model) only accepts
"medium" for both. Out-of-the-box `codex exec` against the AI Gateway
fails with:
Unsupported value: 'low' is not supported with the 'gpt-5.2-codex'
model. Supported values are: 'medium'.
The error covers both the `reasoning.effort` and `text.verbosity`
request parameters, depending on which the model rejects first.
Set both fields to "medium" in two places:
- the generated profile config in ~/.codex/default.config.toml
- explicit -c flags on `codex exec`, since CLI flags have the highest
precedence and we observed the profile-only setting being silently
overridden by the CLI's "low" default in some Codex versions.
`generateCodexConfig` now accepts an optional `reasoningEffort`
parameter so callers can override per-run via
`model: "gpt-5.2-codex?reasoningEffort=high"`.
Verified end-to-end against the Vercel AI Gateway: a previously-failing
`codex exec` smoke run now completes in ~31s and returns a real
response instead of erroring at `turn.failed`.
Also added `vercel-agent-eval-*.tgz` to .gitignore so local `npm pack`
artifacts don't leak into commits.
Allows experiment config files to pass agent-specific options (like
binaryUrl and extraProviders) at runtime via a new agentOptions field.
Previously these could only be set at agent registration time, making
configs for unreleased models non-replicable.
The options flow: ExperimentConfig → runner → AgentRunOptions → agent.run().
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
github-actions[bot]