Add sandboxImage and sandboxUser to ExperimentConfig, AgentRunOptions,
and SandboxOptions (image / user). Both are unset by default, so the
legacy node24 runtime and default account are unchanged.
image boots from a Vercel Container Registry image instead of the
legacy runtime and creates the wrapper's working directory, which
managed images do not ship. The Docker backend rejects it.
user creates a Linux user per sandbox and routes every command and file
operation through it: workspace under the user's home, a user-owned npm
prefix so global CLI installs work, and the SUDO_* variables left by the
SDK's sudo -u transition cleared. Command env is written to a
user-owned file and sourced by a bash bootstrap rather than passed as
argv, so agent auth tokens stay out of process listings and sandbox
command records. The Docker backend already runs as node and ignores it.
Both options join the result-reuse fingerprints so cached default
environment results are not reused for them.
Bump @vercel/sandbox from ^1.2.0 to ^3.3.0. The only API break that
reaches the wrapper is Sandbox.sandboxId -> Sandbox.name, mapped behind
the existing SandboxManager.sandboxId getter.
v3 sandboxes are persistent by default and transparently resume a
stopped session on the next call. Both would change eval semantics:
snapshots on every stop, and a mid-run session death continuing on an
empty filesystem where 1.x failed hard. Pass persistent: false, and turn
any resume into SandboxSessionRecycledError (stopping the replacement
session first so an abort race cannot leave one idling), backed by a
session-id guard for the case where the SDK does not flag the resume.
Default runtime (node24), options, credentials, and the Docker backend
are unchanged.
toSatisfyCriterion and toScoreAtLeast ran the in-sandbox judge with spawnSync,
freezing the worker's event loop for the length of a model run. The worker
answers the main process over an RPC channel whose per-call timeout is birpc's
default 60 seconds, hardcoded — no vitest option or env var reaches it. Any
judge slower than that left an in-flight call like onTaskUpdate unanswered, and
when the loop finally unblocked, the expired timer fired first: an unhandled
"[vitest-worker]: Timeout calling" error and a non-zero exit for a file whose
every assertion had passed. A green eval recorded as red, more often the slower
the judge model, and invisible on genuinely failing runs since those were
already red.
The judge now runs via async spawn and the matchers are async, so the loop
keeps servicing the RPC channel however long the judge takes; the sandbox
timeout remains the real bound. Judge calls must be awaited, which every
published example already shows — the await is load-bearing now, since vitest
does not track a custom matcher's promise and an un-awaited call floats free of
its test. toContainText is deterministic and stays sync.
Verified three ways: unit tests with a fake runner, including an event-loop
liveness test that fails against the old implementation; and a 70-second judge
under vitest 3 in the harness's generated-config shape, where the old helper
exits 1 with the RPC timeout despite its test passing and the new one exits 0.
Validation runs `npx vitest`, and the generated vitest.config.ts imports
`vitest/config`, resolved from the workspace. Fixtures supply vitest as a
devDependency, but package.json belongs to the agent for the length of a run. An
agent scaffolding into an empty directory tends to replace that file rather than
edit it, the next install prunes vitest, and npx then fetches it into a cache
that does not satisfy the config's import.
The run dies at startup with `Cannot find package 'vitest'`. Nothing downstream
distinguishes that from a bad result: the eval never executes, and the harness
records a failure for work it never graded. A wrong answer and an ungraded one
score the same.
runValidation now checks for the runner and installs it when missing. The repair
must not read as the agent's work, so `--no-save` keeps it out of the manifest
and `--no-package-lock` out of the lockfile, which is not gitignored and would
otherwise be captured as a file the agent wrote.
Verified end to end against a fixture with no vitest: without this the run fails
to start, with it the eval executes and grades.
The fix is user-facing but shipped with no changeset, so it would have landed
without a changelog entry or a release of its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `exports["./eval"].types` path pointed at
`dist/lib/agents/eval-helper.d.mts`, but `tsc` does not emit input declaration
files to outDir and copy-runners.mjs only copied `.mjs`. The file therefore
reached neither dist nor the published tarball, so consumers still got no types
and the subpath's types target dangled. In-repo type-checking resolved the source
file, which is why CI stayed green.
Verified by packing the tarball, installing it into a scratch project and running
tsc: `import { environment, transcript, type JudgeVerdict } from
'@vercel/agent-eval/eval'` plus both matchers now type-check with no manual
`declare module 'vitest'`.
Also bumped the changeset from patch to minor, since this adds a new public
subpath export.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `Writable` was imported but never used, which fails `npm run lint` (the
repo's no-unused-vars rule) and would have blocked CI.
- Declare the chunk accumulators before the handlers that close over them.
- Revert the incidental package-lock.json edit so this PR touches no lockfile.
- Add a patch changeset so the fix appears in the changelog and ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging main brought in redact.ts (#186), which redacts run credentials out of
everything a run hands back to the host. It maps `generatedFiles` through
`redactSecrets`, typed `(text: string)` and implemented with `split`/`join`, so
this branch stopped compiling: two type errors at redact.ts:108 and :111.
The tempting fix is `redactSecrets(content.toString('utf-8'), ...)` wrapped back
into a Buffer. That silently defeats this whole change, reintroducing U+FFFD for
every non-UTF-8 byte on the orchestrator's main path. Worse, the naive shape
without the decode does not merely corrupt, it throws: Buffer has no `split`, so
redaction would crash on any run that captured a generated file while a usable
credential was set. The existing test hid both outcomes because its fixture
passed a string into a field the type says is a Buffer.
So `redactSecretsBuffer` locates the credential's UTF-8 byte sequence and splices
it out, copying every other byte through untouched. The fixture now holds a real
Buffer, and the new cases cover a secret embedded between invalid-UTF-8 bytes,
multiple occurrences, absent and too-short secrets, and overlapping secrets. One
case asserts the byte-preserving path and the decode-and-re-encode path actually
differ, so the shortcut cannot be reintroduced without failing.
Build, lint and the full suite pass: 318 passed, 12 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Several agents are configured through a file the framework writes into the
sandbox with the live credential in it: opencode's `opencode.json` carries it at
`provider.vercel.options.apiKey`, and codex's TOML is the same shape. Those files
sit in the agent's cwd, so models read them as a matter of course while
orienting, and the read lands in the transcript. Consumers commit transcripts, so
the credential ends up in their repo.
That is not hypothetical. It is how this was found: vercel/next-evals-oss is
public and tripped secret scanning on a committed transcript. Sweeping that repo
turned up 287 occurrences of 5 distinct credentials across 82 files in 8
experiments, going back ten weeks. All 5 were Vercel OIDC tokens with a 12h TTL
and all had expired, so the exposure was limited, but nothing about the mechanism
guarantees that — a consumer authenticating with a long-lived Gateway key would
have published a live credential.
Adds redactRunResult, applied at the host boundary in the plugin orchestrator, to
every text-bearing field of AgentRunResult: output, transcript, error, test and
script output, and generated file contents. Generated files matter because an
agent that copies its config into a new file would otherwise route around a
transcript-only check.
Three deliberate choices:
Exact-string matching, not patterns. The framework knows the precise value it
injected, so there is nothing to infer and no false positives. A pattern would
have to guess, and credential-shaped substrings genuinely occur in transcripts —
the agents' own `ses_…` session IDs contain base64url runs that a JWT prefix
match flags. The tradeoff is that a credential the framework never saw is not
covered, e.g. if a run refreshes its own token mid-flight.
On the way out, not before the judge. The judge runs inside the sandbox where the
credential is present anyway, and rewriting the transcript before it is judged
would change what the judge reads and therefore the score. Redacting at the
boundary keeps in-sandbox behavior byte-identical and only affects what the host
persists.
Wrapping runWithDefinition rather than patching each return. It has eight return
paths; a ninth added later must not be able to leak by omission. The wrapper also
redacts a pinned judge's key, which differs from the codegen key when the judge
is pinned to another agent, and tolerates judge resolution failing — that is the
run's error to report, not redaction's, so it must not become a throw on a path
that previously returned a result.
Verified against the real leaked data: this redaction applied to the pre-scrub
transcript from the flagged commit targets byte-identical ranges to the manual
scrub done in that repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collected project trees were not faithful copies. Two independent paths
decoded every file as UTF-8, replacing each invalid byte with U+FFFD:
- captureGeneratedFiles read the agent's diff via sandbox.readFile, which
hands back command stdout -- a string the sandbox API decodes as UTF-8.
- readFixtureFiles read fixtures with 'utf-8', corrupting every binary
asset a fixture ships when copyFiles: 'all' is used.
Both failures are silent: no exception, no warning, and text-only projects
never show them. Binary files were destroyed and inflated ~1.8x.
Sandbox gains readFileBuffer(path), routed through `base64` so the existing
transport round-trips losslessly, and copyFiles: 'all' now uses copyFileSync
instead of read-then-write -- bytes never enter the heap, so a whole fixture
tree is no longer buffered in memory.
BREAKING: generatedFiles is now Record<string, Buffer> on AgentRunResult and
EvalRunData; readFixtureFiles returns Map<string, Buffer>.
Closes#176
Co-Authored-By: Claude <noreply@anthropic.com>
- modelRepair now flows RunnerResult -> AgentRunResult -> EvalRunResult
(mirroring observedModel) so persisted results record which runs needed
the shell-tool repair; repairs dropping to zero is the workaround's
removal signal.
- The canary fail-loud path now returns the captured canary CLI output
and transcript instead of discarding them, matching the login and
real-exec failure paths.
- The verified canary outcome is memoized in ~/.codex/agent-eval-canary.json
for the sandbox's lifetime: judge assertions re-invoke run.mjs once per
assertion and previously would each have paid a canary exec. The marker
re-reports the original repairedModel so every invocation carries the
same evidence.
Verified live against the gateway: first invocation repairs and records
modelRepair (14.6s); second invocation skips the canary via the marker
(1.7s) and still reports the repair. 279 tests green (2 new for the
marker parser).
Codex CLI >= 0.144.0 (published 2026-07-09; 0.143.0 is the last good
version) exposes no shell/exec tool to the model when config.toml uses a
custom model_provider and omits the model key — exactly the shape
native-default gateway runs write. The model answers, but cannot run
commands, read files, or use installed skills, and it was observed
fabricating command output instead of reporting the missing tool.
run.mjs now pre-verifies native-default runs with a fabrication-proof
shell canary (a command_execution item must carry a random nonce),
repairs by re-stating the CLI's own resolved default model as an
explicit top-level model key in the profile config (prepended — a line
appended after a [table] header would join that table), re-verifies,
and fails loudly if the tool is still unavailable. The repair is
surfaced as an optional modelRepair field on RunnerResult.
Verified end to end against the AI Gateway with codex-cli 0.145.0:
canary fails -> repair writes model = "openai/gpt-5.6-sol" -> canary
passes -> real task runs with a genuine command_execution.
An explicit --model like anthropic/claude-sonnet-5 was passed to the
OpenCode CLI verbatim, which reads the first segment as its provider id.
The generated opencode.json only configures the vercel (AI Gateway)
provider, so every explicit-model run using a canonical gateway id died
at session start with "Unexpected server error" and a null
observedModel.
- resolveOpenCodeModel (host-side): prefix vercel/ unless the caller
already targets vercel/... or a configured extraProviders key.
- Threaded to the runner via runnerExtra -> input.extra.cliModel,
mirroring codex's host-computed model (the judge path ships extra but
not agentOptions, so the resolution must happen host-side).
- normalizeObservedModel (runner-side): un-apply exactly that prefix on
the observation, so observedModel === requestedModel holds for
canonical gateway ids and a gateway substitution still surfaces as a
clean gateway id. Native-default observations are untouched, and the
documented vercel/-prefixed form still passes through verbatim.
- README: the OpenCode model format section now documents both forms
(it previously stated unprefixed ids fail with provider-not-found).
Adopting a changed or new eval shouldn't force re-running every model. This adds a
small, explicit workflow and fixes the fingerprinting that made it impossible.
- Fingerprint split: results store a content-only hash alongside the combined
(content+config) one. A real eval change is never masked; a benign config change
(timeout bump, pinning a judge) is carried forward by `refingerprint` instead of
re-running. Existing `fingerprint` values are byte-identical.
- `agent-eval status` — read-only: new vs changed evals, per experiment. `--check`
exits non-zero on any new/changed eval; `--json` emits per-experiment new/changed.
- `agent-eval run <experiments...>` — run the named experiments' new/changed evals.
- Bare `agent-eval` shows status, then lets you multi-select what to run. Removes
`run-all` and `--dry`. No in-framework accept/keep — staleness acceptance is the
consumer's policy (filter `status --json` in CI).
Stacked on #164.
Judge the final state or the transcript from inside EVAL.ts, reusing the
SAME agent + sandbox as codegen (no fresh sandbox, no copied evidence):
import { environment, transcript } from '@vercel/agent-eval/eval';
await expect(environment).toSatisfyCriterion('uses Server Components');
await expect(transcript).toSatisfyCriterion('used DevTools, not guesswork');
await expect(environment).toScoreAtLeast('code quality', 0.8);
- eval-helper.mjs: shipped in-sandbox (aliased to @vercel/agent-eval/eval +
vitest setup file); each assertion re-invokes __agent_eval__/run.mjs with a
judge prompt and parses the verdict. You give only the criterion.
- orchestrator/shared: ship the helper, materialize the raw transcript to a
file (read by path, never dumped into a prompt), record judge-config (same
model), pass auth env to validation, gitignore __agent_eval__/.
- Failures are attributable: '[judge:environment] FAIL (score): reason'.
Verified: unit tests + real e2e (Vercel sandbox + Claude via gateway) — judge
confirms true criteria (eval passes) and rejects a false one (eval fails).
The comma-separated token from #150 prevents the tools list itself from
splitting, but --allowedTools is variadic: it keeps capturing positional
tokens until the next flag. With the prompt directly after the value,
claude 2.1.112 consumed it as another tool name and failed with 'Input
must be provided either through stdin or as a prompt argument when using
--print' — caught by a live a0-local smoke run, invisible to the unit
tests because they asserted the broken order.
Emit --allowedTools first so the always-present
--dangerously-skip-permissions terminates the variadic capture before
the trailing prompt. Verified live: the reordered invocation accepts the
prompt. Default-off argument construction is byte-identical. New test
asserts the token after the allowedTools value is always a flag.
The adapters read options.webResearch, but runExperiment built
AgentRunOptions from an explicit field list that never included it, so
the option was unreachable for experiment-config consumers (a0-local
calls runExperiment, not executeAgent). Adds the field to
ExperimentConfig/ResolvedExperimentConfig/RunnableExperimentConfig, the
zod schema (z.object strips unknown keys, so schema membership is
required for validateConfig not to drop it), resolveConfig, and both
agent.run call sites (runExperiment and runSingleEval).
Still default-off: absent config yields webResearch: undefined, which
leaves every adapter branch untaken.
Safe redo of #141 (reverted in #144). webResearch defaults to false, so
command construction is byte-identical for existing consumers; coding
evals are unaffected unless they opt in.
The #141 breakage is fixed and regression-tested: Claude Code's
--allowedTools is variadic, so WebSearch/WebFetch are passed as a single
comma-separated value instead of separate tokens that consumed the
trailing positional prompt.
Verified against AI Gateway with live spikes: Claude Code WebSearch
executes (tool_use/tool_result events), OpenCode Exa websearch executes,
and Codex researches via shell even though no web_search items appear
through the responses wire (setting kept for direct-OpenAI runs and
future gateway support).
OpenCode 1.17.0 rewrote its logging pipeline and removed the
service=llm log lines the adapter scraped for providerID/modelID,
so native-default runs silently lost model observation.
Fall back to 'opencode export <sessionID>' when log scraping yields
nothing: the session id comes from the --format json event stream and
the exported assistant message carries providerID/modelID. Observation
never fails the run. The log scrape stays as the first, cheaper source
for OpenCode <= 1.16.x.
github-actions[bot]