Judge the final state or the transcript from inside EVAL.ts, reusing the
SAME agent + sandbox as codegen (no fresh sandbox, no copied evidence):
import { environment, transcript } from '@vercel/agent-eval/eval';
await expect(environment).toSatisfyCriterion('uses Server Components');
await expect(transcript).toSatisfyCriterion('used DevTools, not guesswork');
await expect(environment).toScoreAtLeast('code quality', 0.8);
- eval-helper.mjs: shipped in-sandbox (aliased to @vercel/agent-eval/eval +
vitest setup file); each assertion re-invokes __agent_eval__/run.mjs with a
judge prompt and parses the verdict. You give only the criterion.
- orchestrator/shared: ship the helper, materialize the raw transcript to a
file (read by path, never dumped into a prompt), record judge-config (same
model), pass auth env to validation, gitignore __agent_eval__/.
- Failures are attributable: '[judge:environment] FAIL (score): reason'.
Verified: unit tests + real e2e (Vercel sandbox + Claude via gateway) — judge
confirms true criteria (eval passes) and rejects a false one (eval fails).
The comma-separated token from #150 prevents the tools list itself from
splitting, but --allowedTools is variadic: it keeps capturing positional
tokens until the next flag. With the prompt directly after the value,
claude 2.1.112 consumed it as another tool name and failed with 'Input
must be provided either through stdin or as a prompt argument when using
--print' — caught by a live a0-local smoke run, invisible to the unit
tests because they asserted the broken order.
Emit --allowedTools first so the always-present
--dangerously-skip-permissions terminates the variadic capture before
the trailing prompt. Verified live: the reordered invocation accepts the
prompt. Default-off argument construction is byte-identical. New test
asserts the token after the allowedTools value is always a flag.
The adapters read options.webResearch, but runExperiment built
AgentRunOptions from an explicit field list that never included it, so
the option was unreachable for experiment-config consumers (a0-local
calls runExperiment, not executeAgent). Adds the field to
ExperimentConfig/ResolvedExperimentConfig/RunnableExperimentConfig, the
zod schema (z.object strips unknown keys, so schema membership is
required for validateConfig not to drop it), resolveConfig, and both
agent.run call sites (runExperiment and runSingleEval).
Still default-off: absent config yields webResearch: undefined, which
leaves every adapter branch untaken.
Safe redo of #141 (reverted in #144). webResearch defaults to false, so
command construction is byte-identical for existing consumers; coding
evals are unaffected unless they opt in.
The #141 breakage is fixed and regression-tested: Claude Code's
--allowedTools is variadic, so WebSearch/WebFetch are passed as a single
comma-separated value instead of separate tokens that consumed the
trailing positional prompt.
Verified against AI Gateway with live spikes: Claude Code WebSearch
executes (tool_use/tool_result events), OpenCode Exa websearch executes,
and Codex researches via shell even though no web_search items appear
through the responses wire (setting kept for direct-OpenAI runs and
future gateway support).
OpenCode 1.17.0 rewrote its logging pipeline and removed the
service=llm log lines the adapter scraped for providerID/modelID,
so native-default runs silently lost model observation.
Fall back to 'opencode export <sessionID>' when log scraping yields
nothing: the session id comes from the --format json event stream and
the exported assistant message carries providerID/modelID. Observation
never fails the run. The log scrape stays as the first, cheaper source
for OpenCode <= 1.16.x.
Allows experiment config files to pass agent-specific options (like
binaryUrl and extraProviders) at runtime via a new agentOptions field.
Previously these could only be set at agent registration time, making
configs for unreleased models non-replicable.
The options flow: ExperimentConfig → runner → AgentRunOptions → agent.run().
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Use built-in OpenAI provider for Codex
* Strip openai/ prefix from --model CLI flag for direct API path
The config.toml correctly strips the provider prefix for direct OpenAI API
usage, but the --model CLI flag still passed the prefixed name (e.g.
"openai/gpt-5.2-codex"), causing a "model not found" error.
* feat: support CLAUDE_CODE_OAUTH_TOKEN for Claude Code agent
Allow Claude Pro/Max subscribers to authenticate using their OAuth token
instead of requiring a separate ANTHROPIC_API_KEY. When CLAUDE_CODE_OAUTH_TOKEN
is set in the environment, it takes precedence over ANTHROPIC_API_KEY for
non-AI-Gateway configurations.
Closes#54
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: address PR review comments for OAuth token support
Refactor nested ternary for sandbox env to if/else for readability,
add clarifying comment about credential consistency, and add unit
tests for getApiKeyEnvVar() precedence (gateway > oauth > direct).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
github-actions[bot]