Several agents are configured through a file the framework writes into the
sandbox with the live credential in it: opencode's `opencode.json` carries it at
`provider.vercel.options.apiKey`, and codex's TOML is the same shape. Those files
sit in the agent's cwd, so models read them as a matter of course while
orienting, and the read lands in the transcript. Consumers commit transcripts, so
the credential ends up in their repo.
That is not hypothetical. It is how this was found: vercel/next-evals-oss is
public and tripped secret scanning on a committed transcript. Sweeping that repo
turned up 287 occurrences of 5 distinct credentials across 82 files in 8
experiments, going back ten weeks. All 5 were Vercel OIDC tokens with a 12h TTL
and all had expired, so the exposure was limited, but nothing about the mechanism
guarantees that — a consumer authenticating with a long-lived Gateway key would
have published a live credential.
Adds redactRunResult, applied at the host boundary in the plugin orchestrator, to
every text-bearing field of AgentRunResult: output, transcript, error, test and
script output, and generated file contents. Generated files matter because an
agent that copies its config into a new file would otherwise route around a
transcript-only check.
Three deliberate choices:
Exact-string matching, not patterns. The framework knows the precise value it
injected, so there is nothing to infer and no false positives. A pattern would
have to guess, and credential-shaped substrings genuinely occur in transcripts —
the agents' own `ses_…` session IDs contain base64url runs that a JWT prefix
match flags. The tradeoff is that a credential the framework never saw is not
covered, e.g. if a run refreshes its own token mid-flight.
On the way out, not before the judge. The judge runs inside the sandbox where the
credential is present anyway, and rewriting the transcript before it is judged
would change what the judge reads and therefore the score. Redacting at the
boundary keeps in-sandbox behavior byte-identical and only affects what the host
persists.
Wrapping runWithDefinition rather than patching each return. It has eight return
paths; a ninth added later must not be able to leak by omission. The wrapper also
redacts a pinned judge's key, which differs from the codegen key when the judge
is pinned to another agent, and tolerates judge resolution failing — that is the
run's error to report, not redaction's, so it must not become a throw on a path
that previously returned a result.
Verified against the real leaked data: this redaction applied to the pre-scrub
transcript from the flagged commit targets byte-identical ranges to the manual
scrub done in that repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- results.test.ts/runner.test.ts now pin modelRepair propagation at the
same layers where observedModel (its precedent) is tested, so a refactor
cannot silently drop the repair evidence that serves as the workaround's
removal signal.
- The __AGENT_RESULT__ status line and readRunnerResult's fallback
reconstruction now carry modelRepair, matching observedModel on that
channel (previously the field was lost whenever the result file could
not be read back).
- Canary spawnSync timeout 180s -> 60s: a single echo round-trip never
needs more, and the worst-case pre-task budget drops from 360s to 120s
of the 600s default sandbox lifetime.
- Cross-reference comments at the orchestrator's sandbox-creation step and
eval-helper's header document the canary marker's dependence on the
shared per-run sandbox lifetime.
- modelRepair now flows RunnerResult -> AgentRunResult -> EvalRunResult
(mirroring observedModel) so persisted results record which runs needed
the shell-tool repair; repairs dropping to zero is the workaround's
removal signal.
- The canary fail-loud path now returns the captured canary CLI output
and transcript instead of discarding them, matching the login and
real-exec failure paths.
- The verified canary outcome is memoized in ~/.codex/agent-eval-canary.json
for the sandbox's lifetime: judge assertions re-invoke run.mjs once per
assertion and previously would each have paid a canary exec. The marker
re-reports the original repairedModel so every invocation carries the
same evidence.
Verified live against the gateway: first invocation repairs and records
modelRepair (14.6s); second invocation skips the canary via the marker
(1.7s) and still reports the repair. 279 tests green (2 new for the
marker parser).
Codex CLI >= 0.144.0 (published 2026-07-09; 0.143.0 is the last good
version) exposes no shell/exec tool to the model when config.toml uses a
custom model_provider and omits the model key — exactly the shape
native-default gateway runs write. The model answers, but cannot run
commands, read files, or use installed skills, and it was observed
fabricating command output instead of reporting the missing tool.
run.mjs now pre-verifies native-default runs with a fabrication-proof
shell canary (a command_execution item must carry a random nonce),
repairs by re-stating the CLI's own resolved default model as an
explicit top-level model key in the profile config (prepended — a line
appended after a [table] header would join that table), re-verifies,
and fails loudly if the tool is still unavailable. The repair is
surfaced as an optional modelRepair field on RunnerResult.
Verified end to end against the AI Gateway with codex-cli 0.145.0:
canary fails -> repair writes model = "openai/gpt-5.6-sol" -> canary
passes -> real task runs with a genuine command_execution.
An explicit --model like anthropic/claude-sonnet-5 was passed to the
OpenCode CLI verbatim, which reads the first segment as its provider id.
The generated opencode.json only configures the vercel (AI Gateway)
provider, so every explicit-model run using a canonical gateway id died
at session start with "Unexpected server error" and a null
observedModel.
- resolveOpenCodeModel (host-side): prefix vercel/ unless the caller
already targets vercel/... or a configured extraProviders key.
- Threaded to the runner via runnerExtra -> input.extra.cliModel,
mirroring codex's host-computed model (the judge path ships extra but
not agentOptions, so the resolution must happen host-side).
- normalizeObservedModel (runner-side): un-apply exactly that prefix on
the observation, so observedModel === requestedModel holds for
canonical gateway ids and a gateway substitution still surfaces as a
clean gateway id. Native-default observations are untouched, and the
documented vercel/-prefixed form still passes through verbatim.
- README: the OpenCode model format section now documents both forms
(it previously stated unprefixed ids fail with provider-not-found).
Adopting a changed or new eval shouldn't force re-running every model. This adds a
small, explicit workflow and fixes the fingerprinting that made it impossible.
- Fingerprint split: results store a content-only hash alongside the combined
(content+config) one. A real eval change is never masked; a benign config change
(timeout bump, pinning a judge) is carried forward by `refingerprint` instead of
re-running. Existing `fingerprint` values are byte-identical.
- `agent-eval status` — read-only: new vs changed evals, per experiment. `--check`
exits non-zero on any new/changed eval; `--json` emits per-experiment new/changed.
- `agent-eval run <experiments...>` — run the named experiments' new/changed evals.
- Bare `agent-eval` shows status, then lets you multi-select what to run. Removes
`run-all` and `--dry`. No in-framework accept/keep — staleness acceptance is the
consumer's policy (filter `status --json` in CI).
Stacked on #164.
Judge the final state or the transcript from inside EVAL.ts, reusing the
SAME agent + sandbox as codegen (no fresh sandbox, no copied evidence):
import { environment, transcript } from '@vercel/agent-eval/eval';
await expect(environment).toSatisfyCriterion('uses Server Components');
await expect(transcript).toSatisfyCriterion('used DevTools, not guesswork');
await expect(environment).toScoreAtLeast('code quality', 0.8);
- eval-helper.mjs: shipped in-sandbox (aliased to @vercel/agent-eval/eval +
vitest setup file); each assertion re-invokes __agent_eval__/run.mjs with a
judge prompt and parses the verdict. You give only the criterion.
- orchestrator/shared: ship the helper, materialize the raw transcript to a
file (read by path, never dumped into a prompt), record judge-config (same
model), pass auth env to validation, gitignore __agent_eval__/.
- Failures are attributable: '[judge:environment] FAIL (score): reason'.
Verified: unit tests + real e2e (Vercel sandbox + Claude via gateway) — judge
confirms true criteria (eval passes) and rejects a false one (eval fails).
Review cleanups (no behavior change):
- contract.ts: drop dead AgentRunResult re-export; fix wrong
'editPrompt'd by the runner' comment; de-dup the extra/runnerExtra docs.
- codex/agent.ts: extract parseOptionsModel() — configFiles() and
runnerExtra() shared the same parse (DRY).
- run.mjs headers: honest, terse export rationale (the old wording claimed
helpers were test-verified when codex/gemini/cursor have none).
Port the remaining four adapters onto the generic orchestrator: each becomes a
host-side definition (agent.ts) + an in-sandbox runner (run.mjs); the old
single-file adapters are deleted. The Agent interface + registry are unchanged.
- codex: host-computed values (resolved --model, reasoning effort, verbosity from
parseModelString) must match the TOML profile, so AgentDefinition gains an
optional runnerExtra(options) threaded to the runner as input.extra (the only
contract addition). generateCodexConfig/parseModelString stay host-side + exported.
login is a separate spawn piping the key on stdin (no secret in a shell string);
the && short-circuit is preserved.
- opencode: opencode.json config + log-scrape then `opencode export` observedModel
fallback chain preserved; generators stay host-side + exported.
- gemini/cursor: stdout-only stream-json, no config, observedModel null. cursor
binary is 'agent', prompt-first arg order.
Adversarially reviewed each plugin vs its origin/main adapter: zero critical/major
findings. Live-validated end to end against a real Vercel sandbox + AI Gateway:
claude 7/7 and codex 6/6 integration tests pass. 216 unit + lint + build green.
Introduce the agent plugin contract + a single generic host orchestrator, and
migrate claude-code onto it as the first vertical slice.
Each agent is now: a host-side DEFINITION (agent.ts: install/configFiles/authEnv/
getApiKeyEnvVar/defaultModel) + an in-sandbox RUNNER (run.mjs: invoke the CLI +
capture the transcript, returning a typed RunnerResult). The generic
runWithDefinition() drives the shared, agent-agnostic flow (sandbox lifecycle,
git baseline, neutral workspace, validation, generated-file capture, o11y parse,
abort/timeout) — reproducing the old claude-code.ts run() exactly.
run.mjs is self-contained (only node:*), dual-mode (importable + runnable), and
exports its pure helpers so host tests verify exactly what the sandbox runs. The
Agent interface is unchanged, so registry/runner/results are untouched.
- plugin/contract.ts: AgentDefinition, AgentRunInput, RunnerResult (+ InstallStep/ConfigFile)
- plugin/orchestrator.ts: runWithDefinition() — the generic run()
- claude-code/agent.ts + claude-code/run.mjs; index.ts points at the new path
- build: scripts/copy-runners.mjs ships each run.mjs into dist
- claude-code.test.ts: import paths updated (helpers from run.mjs, factory from agent.ts)
Host<->sandbox result contract: run.mjs writes __agent_eval__/agent-result.json
(+ a __AGENT_RESULT__ stdout marker fallback); host reads it, splitting agent
failure (ok) from runner crash (node exit). Validated end-to-end against a real
Vercel sandbox + AI Gateway: 7/7 claude integration tests pass (gateway + direct
API + parallel + invalid-model error path); 216 unit tests + lint green.
The comma-separated token from #150 prevents the tools list itself from
splitting, but --allowedTools is variadic: it keeps capturing positional
tokens until the next flag. With the prompt directly after the value,
claude 2.1.112 consumed it as another tool name and failed with 'Input
must be provided either through stdin or as a prompt argument when using
--print' — caught by a live a0-local smoke run, invisible to the unit
tests because they asserted the broken order.
Emit --allowedTools first so the always-present
--dangerously-skip-permissions terminates the variadic capture before
the trailing prompt. Verified live: the reordered invocation accepts the
prompt. Default-off argument construction is byte-identical. New test
asserts the token after the allowedTools value is always a flag.
Conditional include (same pattern as native-default modelPolicy) so
default-off fingerprints are byte-identical to existing releases, while
research and non-research configs never share a fingerprint — result
reuse must not serve a cached parametric-only result for a research
run, or vice versa.
The adapters read options.webResearch, but runExperiment built
AgentRunOptions from an explicit field list that never included it, so
the option was unreachable for experiment-config consumers (a0-local
calls runExperiment, not executeAgent). Adds the field to
ExperimentConfig/ResolvedExperimentConfig/RunnableExperimentConfig, the
zod schema (z.object strips unknown keys, so schema membership is
required for validateConfig not to drop it), resolveConfig, and both
agent.run call sites (runExperiment and runSingleEval).
Still default-off: absent config yields webResearch: undefined, which
leaves every adapter branch untaken.
Safe redo of #141 (reverted in #144). webResearch defaults to false, so
command construction is byte-identical for existing consumers; coding
evals are unaffected unless they opt in.
The #141 breakage is fixed and regression-tested: Claude Code's
--allowedTools is variadic, so WebSearch/WebFetch are passed as a single
comma-separated value instead of separate tokens that consumed the
trailing positional prompt.
Verified against AI Gateway with live spikes: Claude Code WebSearch
executes (tool_use/tool_result events), OpenCode Exa websearch executes,
and Codex researches via shell even though no web_search items appear
through the responses wire (setting kept for direct-OpenAI runs and
future gateway support).
Jude Gao