toSatisfyCriterion and toScoreAtLeast ran the in-sandbox judge with spawnSync,
freezing the worker's event loop for the length of a model run. The worker
answers the main process over an RPC channel whose per-call timeout is birpc's
default 60 seconds, hardcoded — no vitest option or env var reaches it. Any
judge slower than that left an in-flight call like onTaskUpdate unanswered, and
when the loop finally unblocked, the expired timer fired first: an unhandled
"[vitest-worker]: Timeout calling" error and a non-zero exit for a file whose
every assertion had passed. A green eval recorded as red, more often the slower
the judge model, and invisible on genuinely failing runs since those were
already red.
The judge now runs via async spawn and the matchers are async, so the loop
keeps servicing the RPC channel however long the judge takes; the sandbox
timeout remains the real bound. Judge calls must be awaited, which every
published example already shows — the await is load-bearing now, since vitest
does not track a custom matcher's promise and an un-awaited call floats free of
its test. toContainText is deterministic and stays sync.
Verified three ways: unit tests with a fake runner, including an event-loop
liveness test that fails against the old implementation; and a 70-second judge
under vitest 3 in the harness's generated-config shape, where the old helper
exits 1 with the RPC timeout despite its test passing and the new one exits 0.
Validation runs `npx vitest`, and the generated vitest.config.ts imports
`vitest/config`, resolved from the workspace. Fixtures supply vitest as a
devDependency, but package.json belongs to the agent for the length of a run. An
agent scaffolding into an empty directory tends to replace that file rather than
edit it, the next install prunes vitest, and npx then fetches it into a cache
that does not satisfy the config's import.
The run dies at startup with `Cannot find package 'vitest'`. Nothing downstream
distinguishes that from a bad result: the eval never executes, and the harness
records a failure for work it never graded. A wrong answer and an ungraded one
score the same.
runValidation now checks for the runner and installs it when missing. The repair
must not read as the agent's work, so `--no-save` keeps it out of the manifest
and `--no-package-lock` out of the lockfile, which is not gitignored and would
otherwise be captured as a file the agent wrote.
Verified end to end against a fixture with no vitest: without this the run fails
to start, with it the eval executes and grades.
The `exports["./eval"].types` path pointed at
`dist/lib/agents/eval-helper.d.mts`, but `tsc` does not emit input declaration
files to outDir and copy-runners.mjs only copied `.mjs`. The file therefore
reached neither dist nor the published tarball, so consumers still got no types
and the subpath's types target dangled. In-repo type-checking resolved the source
file, which is why CI stayed green.
Verified by packing the tarball, installing it into a scratch project and running
tsc: `import { environment, transcript, type JudgeVerdict } from
'@vercel/agent-eval/eval'` plus both matchers now type-check with no manual
`declare module 'vitest'`.
Also bumped the changeset from patch to minor, since this adds a new public
subpath export.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `Writable` was imported but never used, which fails `npm run lint` (the
repo's no-unused-vars rule) and would have blocked CI.
- Declare the chunk accumulators before the handlers that close over them.
- Revert the incidental package-lock.json edit so this PR touches no lockfile.
- Add a patch changeset so the fix appears in the changelog and ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging main brought in redact.ts (#186), which redacts run credentials out of
everything a run hands back to the host. It maps `generatedFiles` through
`redactSecrets`, typed `(text: string)` and implemented with `split`/`join`, so
this branch stopped compiling: two type errors at redact.ts:108 and :111.
The tempting fix is `redactSecrets(content.toString('utf-8'), ...)` wrapped back
into a Buffer. That silently defeats this whole change, reintroducing U+FFFD for
every non-UTF-8 byte on the orchestrator's main path. Worse, the naive shape
without the decode does not merely corrupt, it throws: Buffer has no `split`, so
redaction would crash on any run that captured a generated file while a usable
credential was set. The existing test hid both outcomes because its fixture
passed a string into a field the type says is a Buffer.
So `redactSecretsBuffer` locates the credential's UTF-8 byte sequence and splices
it out, copying every other byte through untouched. The fixture now holds a real
Buffer, and the new cases cover a secret embedded between invalid-UTF-8 bytes,
multiple occurrences, absent and too-short secrets, and overlapping secrets. One
case asserts the byte-preserving path and the decode-and-re-encode path actually
differ, so the shortcut cannot be reintroduced without failing.
Build, lint and the full suite pass: 318 passed, 12 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Several agents are configured through a file the framework writes into the
sandbox with the live credential in it: opencode's `opencode.json` carries it at
`provider.vercel.options.apiKey`, and codex's TOML is the same shape. Those files
sit in the agent's cwd, so models read them as a matter of course while
orienting, and the read lands in the transcript. Consumers commit transcripts, so
the credential ends up in their repo.
That is not hypothetical. It is how this was found: vercel/next-evals-oss is
public and tripped secret scanning on a committed transcript. Sweeping that repo
turned up 287 occurrences of 5 distinct credentials across 82 files in 8
experiments, going back ten weeks. All 5 were Vercel OIDC tokens with a 12h TTL
and all had expired, so the exposure was limited, but nothing about the mechanism
guarantees that — a consumer authenticating with a long-lived Gateway key would
have published a live credential.
Adds redactRunResult, applied at the host boundary in the plugin orchestrator, to
every text-bearing field of AgentRunResult: output, transcript, error, test and
script output, and generated file contents. Generated files matter because an
agent that copies its config into a new file would otherwise route around a
transcript-only check.
Three deliberate choices:
Exact-string matching, not patterns. The framework knows the precise value it
injected, so there is nothing to infer and no false positives. A pattern would
have to guess, and credential-shaped substrings genuinely occur in transcripts —
the agents' own `ses_…` session IDs contain base64url runs that a JWT prefix
match flags. The tradeoff is that a credential the framework never saw is not
covered, e.g. if a run refreshes its own token mid-flight.
On the way out, not before the judge. The judge runs inside the sandbox where the
credential is present anyway, and rewriting the transcript before it is judged
would change what the judge reads and therefore the score. Redacting at the
boundary keeps in-sandbox behavior byte-identical and only affects what the host
persists.
Wrapping runWithDefinition rather than patching each return. It has eight return
paths; a ninth added later must not be able to leak by omission. The wrapper also
redacts a pinned judge's key, which differs from the codegen key when the judge
is pinned to another agent, and tolerates judge resolution failing — that is the
run's error to report, not redaction's, so it must not become a throw on a path
that previously returned a result.
Verified against the real leaked data: this redaction applied to the pre-scrub
transcript from the flagged commit targets byte-identical ranges to the manual
scrub done in that repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collected project trees were not faithful copies. Two independent paths
decoded every file as UTF-8, replacing each invalid byte with U+FFFD:
- captureGeneratedFiles read the agent's diff via sandbox.readFile, which
hands back command stdout -- a string the sandbox API decodes as UTF-8.
- readFixtureFiles read fixtures with 'utf-8', corrupting every binary
asset a fixture ships when copyFiles: 'all' is used.
Both failures are silent: no exception, no warning, and text-only projects
never show them. Binary files were destroyed and inflated ~1.8x.
Sandbox gains readFileBuffer(path), routed through `base64` so the existing
transport round-trips losslessly, and copyFiles: 'all' now uses copyFileSync
instead of read-then-write -- bytes never enter the heap, so a whole fixture
tree is no longer buffered in memory.
BREAKING: generatedFiles is now Record<string, Buffer> on AgentRunResult and
EvalRunData; readFixtureFiles returns Map<string, Buffer>.
Closes#176
Co-Authored-By: Claude <noreply@anthropic.com>
- results.test.ts/runner.test.ts now pin modelRepair propagation at the
same layers where observedModel (its precedent) is tested, so a refactor
cannot silently drop the repair evidence that serves as the workaround's
removal signal.
- The __AGENT_RESULT__ status line and readRunnerResult's fallback
reconstruction now carry modelRepair, matching observedModel on that
channel (previously the field was lost whenever the result file could
not be read back).
- Canary spawnSync timeout 180s -> 60s: a single echo round-trip never
needs more, and the worst-case pre-task budget drops from 360s to 120s
of the 600s default sandbox lifetime.
- Cross-reference comments at the orchestrator's sandbox-creation step and
eval-helper's header document the canary marker's dependence on the
shared per-run sandbox lifetime.
- modelRepair now flows RunnerResult -> AgentRunResult -> EvalRunResult
(mirroring observedModel) so persisted results record which runs needed
the shell-tool repair; repairs dropping to zero is the workaround's
removal signal.
- The canary fail-loud path now returns the captured canary CLI output
and transcript instead of discarding them, matching the login and
real-exec failure paths.
- The verified canary outcome is memoized in ~/.codex/agent-eval-canary.json
for the sandbox's lifetime: judge assertions re-invoke run.mjs once per
assertion and previously would each have paid a canary exec. The marker
re-reports the original repairedModel so every invocation carries the
same evidence.
Verified live against the gateway: first invocation repairs and records
modelRepair (14.6s); second invocation skips the canary via the marker
(1.7s) and still reports the repair. 279 tests green (2 new for the
marker parser).
Codex CLI >= 0.144.0 (published 2026-07-09; 0.143.0 is the last good
version) exposes no shell/exec tool to the model when config.toml uses a
custom model_provider and omits the model key — exactly the shape
native-default gateway runs write. The model answers, but cannot run
commands, read files, or use installed skills, and it was observed
fabricating command output instead of reporting the missing tool.
run.mjs now pre-verifies native-default runs with a fabrication-proof
shell canary (a command_execution item must carry a random nonce),
repairs by re-stating the CLI's own resolved default model as an
explicit top-level model key in the profile config (prepended — a line
appended after a [table] header would join that table), re-verifies,
and fails loudly if the tool is still unavailable. The repair is
surfaced as an optional modelRepair field on RunnerResult.
Verified end to end against the AI Gateway with codex-cli 0.145.0:
canary fails -> repair writes model = "openai/gpt-5.6-sol" -> canary
passes -> real task runs with a genuine command_execution.
An explicit --model like anthropic/claude-sonnet-5 was passed to the
OpenCode CLI verbatim, which reads the first segment as its provider id.
The generated opencode.json only configures the vercel (AI Gateway)
provider, so every explicit-model run using a canonical gateway id died
at session start with "Unexpected server error" and a null
observedModel.
- resolveOpenCodeModel (host-side): prefix vercel/ unless the caller
already targets vercel/... or a configured extraProviders key.
- Threaded to the runner via runnerExtra -> input.extra.cliModel,
mirroring codex's host-computed model (the judge path ships extra but
not agentOptions, so the resolution must happen host-side).
- normalizeObservedModel (runner-side): un-apply exactly that prefix on
the observation, so observedModel === requestedModel holds for
canonical gateway ids and a gateway substitution still surfaces as a
clean gateway id. Native-default observations are untouched, and the
documented vercel/-prefixed form still passes through verbatim.
- README: the OpenCode model format section now documents both forms
(it previously stated unprefixed ids fail with provider-not-found).
Adopting a changed or new eval shouldn't force re-running every model. This adds a
small, explicit workflow and fixes the fingerprinting that made it impossible.
- Fingerprint split: results store a content-only hash alongside the combined
(content+config) one. A real eval change is never masked; a benign config change
(timeout bump, pinning a judge) is carried forward by `refingerprint` instead of
re-running. Existing `fingerprint` values are byte-identical.
- `agent-eval status` — read-only: new vs changed evals, per experiment. `--check`
exits non-zero on any new/changed eval; `--json` emits per-experiment new/changed.
- `agent-eval run <experiments...>` — run the named experiments' new/changed evals.
- Bare `agent-eval` shows status, then lets you multi-select what to run. Removes
`run-all` and `--dry`. No in-framework accept/keep — staleness acceptance is the
consumer's policy (filter `status --json` in CI).
Stacked on #164.
Judge the final state or the transcript from inside EVAL.ts, reusing the
SAME agent + sandbox as codegen (no fresh sandbox, no copied evidence):
import { environment, transcript } from '@vercel/agent-eval/eval';
await expect(environment).toSatisfyCriterion('uses Server Components');
await expect(transcript).toSatisfyCriterion('used DevTools, not guesswork');
await expect(environment).toScoreAtLeast('code quality', 0.8);
- eval-helper.mjs: shipped in-sandbox (aliased to @vercel/agent-eval/eval +
vitest setup file); each assertion re-invokes __agent_eval__/run.mjs with a
judge prompt and parses the verdict. You give only the criterion.
- orchestrator/shared: ship the helper, materialize the raw transcript to a
file (read by path, never dumped into a prompt), record judge-config (same
model), pass auth env to validation, gitignore __agent_eval__/.
- Failures are attributable: '[judge:environment] FAIL (score): reason'.
Verified: unit tests + real e2e (Vercel sandbox + Claude via gateway) — judge
confirms true criteria (eval passes) and rejects a false one (eval fails).
Review cleanups (no behavior change):
- contract.ts: drop dead AgentRunResult re-export; fix wrong
'editPrompt'd by the runner' comment; de-dup the extra/runnerExtra docs.
- codex/agent.ts: extract parseOptionsModel() — configFiles() and
runnerExtra() shared the same parse (DRY).
- run.mjs headers: honest, terse export rationale (the old wording claimed
helpers were test-verified when codex/gemini/cursor have none).
Port the remaining four adapters onto the generic orchestrator: each becomes a
host-side definition (agent.ts) + an in-sandbox runner (run.mjs); the old
single-file adapters are deleted. The Agent interface + registry are unchanged.
- codex: host-computed values (resolved --model, reasoning effort, verbosity from
parseModelString) must match the TOML profile, so AgentDefinition gains an
optional runnerExtra(options) threaded to the runner as input.extra (the only
contract addition). generateCodexConfig/parseModelString stay host-side + exported.
login is a separate spawn piping the key on stdin (no secret in a shell string);
the && short-circuit is preserved.
- opencode: opencode.json config + log-scrape then `opencode export` observedModel
fallback chain preserved; generators stay host-side + exported.
- gemini/cursor: stdout-only stream-json, no config, observedModel null. cursor
binary is 'agent', prompt-first arg order.
Adversarially reviewed each plugin vs its origin/main adapter: zero critical/major
findings. Live-validated end to end against a real Vercel sandbox + AI Gateway:
claude 7/7 and codex 6/6 integration tests pass. 216 unit + lint + build green.
Introduce the agent plugin contract + a single generic host orchestrator, and
migrate claude-code onto it as the first vertical slice.
Each agent is now: a host-side DEFINITION (agent.ts: install/configFiles/authEnv/
getApiKeyEnvVar/defaultModel) + an in-sandbox RUNNER (run.mjs: invoke the CLI +
capture the transcript, returning a typed RunnerResult). The generic
runWithDefinition() drives the shared, agent-agnostic flow (sandbox lifecycle,
git baseline, neutral workspace, validation, generated-file capture, o11y parse,
abort/timeout) — reproducing the old claude-code.ts run() exactly.
run.mjs is self-contained (only node:*), dual-mode (importable + runnable), and
exports its pure helpers so host tests verify exactly what the sandbox runs. The
Agent interface is unchanged, so registry/runner/results are untouched.
- plugin/contract.ts: AgentDefinition, AgentRunInput, RunnerResult (+ InstallStep/ConfigFile)
- plugin/orchestrator.ts: runWithDefinition() — the generic run()
- claude-code/agent.ts + claude-code/run.mjs; index.ts points at the new path
- build: scripts/copy-runners.mjs ships each run.mjs into dist
- claude-code.test.ts: import paths updated (helpers from run.mjs, factory from agent.ts)
Host<->sandbox result contract: run.mjs writes __agent_eval__/agent-result.json
(+ a __AGENT_RESULT__ stdout marker fallback); host reads it, splitting agent
failure (ok) from runner crash (node exit). Validated end-to-end against a real
Vercel sandbox + AI Gateway: 7/7 claude integration tests pass (gateway + direct
API + parallel + invalid-model error path); 216 unit tests + lint green.
The comma-separated token from #150 prevents the tools list itself from
splitting, but --allowedTools is variadic: it keeps capturing positional
tokens until the next flag. With the prompt directly after the value,
claude 2.1.112 consumed it as another tool name and failed with 'Input
must be provided either through stdin or as a prompt argument when using
--print' — caught by a live a0-local smoke run, invisible to the unit
tests because they asserted the broken order.
Emit --allowedTools first so the always-present
--dangerously-skip-permissions terminates the variadic capture before
the trailing prompt. Verified live: the reordered invocation accepts the
prompt. Default-off argument construction is byte-identical. New test
asserts the token after the allowedTools value is always a flag.
Conditional include (same pattern as native-default modelPolicy) so
default-off fingerprints are byte-identical to existing releases, while
research and non-research configs never share a fingerprint — result
reuse must not serve a cached parametric-only result for a research
run, or vice versa.
github-actions[bot]