The fix is user-facing but shipped with no changeset, so it would have landed
without a changelog entry or a release of its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `exports["./eval"].types` path pointed at
`dist/lib/agents/eval-helper.d.mts`, but `tsc` does not emit input declaration
files to outDir and copy-runners.mjs only copied `.mjs`. The file therefore
reached neither dist nor the published tarball, so consumers still got no types
and the subpath's types target dangled. In-repo type-checking resolved the source
file, which is why CI stayed green.
Verified by packing the tarball, installing it into a scratch project and running
tsc: `import { environment, transcript, type JudgeVerdict } from
'@vercel/agent-eval/eval'` plus both matchers now type-check with no manual
`declare module 'vitest'`.
Also bumped the changeset from patch to minor, since this adds a new public
subpath export.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `Writable` was imported but never used, which fails `npm run lint` (the
repo's no-unused-vars rule) and would have blocked CI.
- Declare the chunk accumulators before the handlers that close over them.
- Revert the incidental package-lock.json edit so this PR touches no lockfile.
- Add a patch changeset so the fix appears in the changelog and ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging main brought in redact.ts (#186), which redacts run credentials out of
everything a run hands back to the host. It maps `generatedFiles` through
`redactSecrets`, typed `(text: string)` and implemented with `split`/`join`, so
this branch stopped compiling: two type errors at redact.ts:108 and :111.
The tempting fix is `redactSecrets(content.toString('utf-8'), ...)` wrapped back
into a Buffer. That silently defeats this whole change, reintroducing U+FFFD for
every non-UTF-8 byte on the orchestrator's main path. Worse, the naive shape
without the decode does not merely corrupt, it throws: Buffer has no `split`, so
redaction would crash on any run that captured a generated file while a usable
credential was set. The existing test hid both outcomes because its fixture
passed a string into a field the type says is a Buffer.
So `redactSecretsBuffer` locates the credential's UTF-8 byte sequence and splices
it out, copying every other byte through untouched. The fixture now holds a real
Buffer, and the new cases cover a secret embedded between invalid-UTF-8 bytes,
multiple occurrences, absent and too-short secrets, and overlapping secrets. One
case asserts the byte-preserving path and the decode-and-re-encode path actually
differ, so the shortcut cannot be reintroduced without failing.
Build, lint and the full suite pass: 318 passed, 12 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Several agents are configured through a file the framework writes into the
sandbox with the live credential in it: opencode's `opencode.json` carries it at
`provider.vercel.options.apiKey`, and codex's TOML is the same shape. Those files
sit in the agent's cwd, so models read them as a matter of course while
orienting, and the read lands in the transcript. Consumers commit transcripts, so
the credential ends up in their repo.
That is not hypothetical. It is how this was found: vercel/next-evals-oss is
public and tripped secret scanning on a committed transcript. Sweeping that repo
turned up 287 occurrences of 5 distinct credentials across 82 files in 8
experiments, going back ten weeks. All 5 were Vercel OIDC tokens with a 12h TTL
and all had expired, so the exposure was limited, but nothing about the mechanism
guarantees that — a consumer authenticating with a long-lived Gateway key would
have published a live credential.
Adds redactRunResult, applied at the host boundary in the plugin orchestrator, to
every text-bearing field of AgentRunResult: output, transcript, error, test and
script output, and generated file contents. Generated files matter because an
agent that copies its config into a new file would otherwise route around a
transcript-only check.
Three deliberate choices:
Exact-string matching, not patterns. The framework knows the precise value it
injected, so there is nothing to infer and no false positives. A pattern would
have to guess, and credential-shaped substrings genuinely occur in transcripts —
the agents' own `ses_…` session IDs contain base64url runs that a JWT prefix
match flags. The tradeoff is that a credential the framework never saw is not
covered, e.g. if a run refreshes its own token mid-flight.
On the way out, not before the judge. The judge runs inside the sandbox where the
credential is present anyway, and rewriting the transcript before it is judged
would change what the judge reads and therefore the score. Redacting at the
boundary keeps in-sandbox behavior byte-identical and only affects what the host
persists.
Wrapping runWithDefinition rather than patching each return. It has eight return
paths; a ninth added later must not be able to leak by omission. The wrapper also
redacts a pinned judge's key, which differs from the codegen key when the judge
is pinned to another agent, and tolerates judge resolution failing — that is the
run's error to report, not redaction's, so it must not become a throw on a path
that previously returned a result.
Verified against the real leaked data: this redaction applied to the pre-scrub
transcript from the flagged commit targets byte-identical ranges to the manual
scrub done in that repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collected project trees were not faithful copies. Two independent paths
decoded every file as UTF-8, replacing each invalid byte with U+FFFD:
- captureGeneratedFiles read the agent's diff via sandbox.readFile, which
hands back command stdout -- a string the sandbox API decodes as UTF-8.
- readFixtureFiles read fixtures with 'utf-8', corrupting every binary
asset a fixture ships when copyFiles: 'all' is used.
Both failures are silent: no exception, no warning, and text-only projects
never show them. Binary files were destroyed and inflated ~1.8x.
Sandbox gains readFileBuffer(path), routed through `base64` so the existing
transport round-trips losslessly, and copyFiles: 'all' now uses copyFileSync
instead of read-then-write -- bytes never enter the heap, so a whole fixture
tree is no longer buffered in memory.
BREAKING: generatedFiles is now Record<string, Buffer> on AgentRunResult and
EvalRunData; readFixtureFiles returns Map<string, Buffer>.
Closes#176
Co-Authored-By: Claude <noreply@anthropic.com>
- results.test.ts/runner.test.ts now pin modelRepair propagation at the
same layers where observedModel (its precedent) is tested, so a refactor
cannot silently drop the repair evidence that serves as the workaround's
removal signal.
- The __AGENT_RESULT__ status line and readRunnerResult's fallback
reconstruction now carry modelRepair, matching observedModel on that
channel (previously the field was lost whenever the result file could
not be read back).
- Canary spawnSync timeout 180s -> 60s: a single echo round-trip never
needs more, and the worst-case pre-task budget drops from 360s to 120s
of the 600s default sandbox lifetime.
- Cross-reference comments at the orchestrator's sandbox-creation step and
eval-helper's header document the canary marker's dependence on the
shared per-run sandbox lifetime.
- modelRepair now flows RunnerResult -> AgentRunResult -> EvalRunResult
(mirroring observedModel) so persisted results record which runs needed
the shell-tool repair; repairs dropping to zero is the workaround's
removal signal.
- The canary fail-loud path now returns the captured canary CLI output
and transcript instead of discarding them, matching the login and
real-exec failure paths.
- The verified canary outcome is memoized in ~/.codex/agent-eval-canary.json
for the sandbox's lifetime: judge assertions re-invoke run.mjs once per
assertion and previously would each have paid a canary exec. The marker
re-reports the original repairedModel so every invocation carries the
same evidence.
Verified live against the gateway: first invocation repairs and records
modelRepair (14.6s); second invocation skips the canary via the marker
(1.7s) and still reports the repair. 279 tests green (2 new for the
marker parser).
Codex CLI >= 0.144.0 (published 2026-07-09; 0.143.0 is the last good
version) exposes no shell/exec tool to the model when config.toml uses a
custom model_provider and omits the model key — exactly the shape
native-default gateway runs write. The model answers, but cannot run
commands, read files, or use installed skills, and it was observed
fabricating command output instead of reporting the missing tool.
run.mjs now pre-verifies native-default runs with a fabrication-proof
shell canary (a command_execution item must carry a random nonce),
repairs by re-stating the CLI's own resolved default model as an
explicit top-level model key in the profile config (prepended — a line
appended after a [table] header would join that table), re-verifies,
and fails loudly if the tool is still unavailable. The repair is
surfaced as an optional modelRepair field on RunnerResult.
Verified end to end against the AI Gateway with codex-cli 0.145.0:
canary fails -> repair writes model = "openai/gpt-5.6-sol" -> canary
passes -> real task runs with a genuine command_execution.
An explicit --model like anthropic/claude-sonnet-5 was passed to the
OpenCode CLI verbatim, which reads the first segment as its provider id.
The generated opencode.json only configures the vercel (AI Gateway)
provider, so every explicit-model run using a canonical gateway id died
at session start with "Unexpected server error" and a null
observedModel.
- resolveOpenCodeModel (host-side): prefix vercel/ unless the caller
already targets vercel/... or a configured extraProviders key.
- Threaded to the runner via runnerExtra -> input.extra.cliModel,
mirroring codex's host-computed model (the judge path ships extra but
not agentOptions, so the resolution must happen host-side).
- normalizeObservedModel (runner-side): un-apply exactly that prefix on
the observation, so observedModel === requestedModel holds for
canonical gateway ids and a gateway substitution still surfaces as a
clean gateway id. Native-default observations are untouched, and the
documented vercel/-prefixed form still passes through verbatim.
- README: the OpenCode model format section now documents both forms
(it previously stated unprefixed ids fail with provider-not-found).
Rich Haines