Rich Haines
8fea011ea6
Merge pull request #147 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@1.0.0
2026-06-01 19:12:13 +02:00
github-actions[bot]
1ee6ee852e
Version Packages
2026-06-01 17:11:11 +00:00
Rich Haines
17f5987722
Merge pull request #146 from vercel-labs/feat/native-default-model-policy
...
Use native model defaults when model is omitted
2026-06-01 19:10:24 +02:00
molebox
cedf84b4bc
Use native default when model is omitted
2026-06-01 18:48:13 +02:00
molebox
aa66c4d35b
Add native default model policy
2026-06-01 17:01:36 +02:00
Allen Zhou
e4587b5434
Merge pull request #145 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@0.14.5
2026-05-29 17:35:24 -07:00
github-actions[bot]
3db1716a00
Version Packages
2026-05-30 00:34:54 +00:00
Allen Zhou
fb4005237c
Merge pull request #144 from vercel-labs/revert-141-enable-web-sources
...
Revert "Enable source-capable web tools for agent runs" (#141 )
2026-05-29 17:34:02 -07:00
Allen Zhou
450bed3b94
Add changeset for revert of #141
2026-05-29 17:30:46 -07:00
Allen Zhou
2b7eddf90f
Revert "Merge pull request #141 from vercel-labs/feat/enable-agent-web-sources"
...
This reverts commit daaeccb42c , reversing
changes made to cb2280db61 .
2026-05-29 17:30:14 -07:00
Rich Haines
ff51aaf83d
Merge pull request #143 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@0.14.4
2026-05-29 17:06:41 +02:00
github-actions[bot]
e0db9a3813
Version Packages
2026-05-29 15:03:07 +00:00
Rich Haines
daaeccb42c
Merge pull request #141 from vercel-labs/feat/enable-agent-web-sources
...
Enable source-capable web tools for agent runs
2026-05-29 17:02:23 +02:00
molebox
74be0aa25b
Merge main into feat/enable-agent-web-sources
2026-05-29 16:51:52 +02:00
Allen Zhou
cb2280db61
Merge pull request #142 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@0.14.3
2026-05-28 14:04:44 -07:00
github-actions[bot]
a53721752b
Version Packages
2026-05-28 18:45:17 +00:00
Allen Zhou
43bec69aba
Merge pull request #140 from vercel-labs/fix/codex-reasoning-effort-default
...
Fix Codex profile reasoning_effort and verbosity defaults
2026-05-28 11:44:28 -07:00
molebox
2d27942cc3
Enable source-capable web tools for agent runs
2026-05-28 11:15:10 +02:00
Allen Zhou
af62bc3fe9
Remove vercel-agent-eval-*.tgz from .gitignore
...
Per review feedback. Local `npm pack` artifacts can be cleaned up by
hand; no need for a project-level ignore rule.
2026-05-27 21:34:43 -07:00
Allen Zhou
a9efa3ad1f
Fix Codex profile reasoning_effort and verbosity defaults
...
The Codex CLI defaults both `model_reasoning_effort` and `model_verbosity`
to "low", but `gpt-5.2-codex` (the default Codex model) only accepts
"medium" for both. Out-of-the-box `codex exec` against the AI Gateway
fails with:
Unsupported value: 'low' is not supported with the 'gpt-5.2-codex'
model. Supported values are: 'medium'.
The error covers both the `reasoning.effort` and `text.verbosity`
request parameters, depending on which the model rejects first.
Set both fields to "medium" in two places:
- the generated profile config in ~/.codex/default.config.toml
- explicit -c flags on `codex exec`, since CLI flags have the highest
precedence and we observed the profile-only setting being silently
overridden by the CLI's "low" default in some Codex versions.
`generateCodexConfig` now accepts an optional `reasoningEffort`
parameter so callers can override per-run via
`model: "gpt-5.2-codex?reasoningEffort=high"`.
Verified end-to-end against the Vercel AI Gateway: a previously-failing
`codex exec` smoke run now completes in ~31s and returns a real
response instead of erroring at `turn.failed`.
Also added `vercel-agent-eval-*.tgz` to .gitignore so local `npm pack`
artifacts don't leak into commits.
2026-05-27 20:19:58 -07:00
Allen Zhou
096192204e
Merge pull request #139 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@0.14.2
2026-05-27 19:55:33 -07:00
github-actions[bot]
8bb80230c4
Version Packages
2026-05-28 02:55:06 +00:00
Allen Zhou
79d0d3b4a0
Merge pull request #138 from vercel-labs/fix/codex-profile-config
...
Fix Codex profile config
2026-05-27 19:54:11 -07:00
molebox
5950d74405
Fix Codex profile config
2026-05-27 21:36:53 +02:00
Allen Zhou
330b743167
Merge pull request #130 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@0.14.1
2026-05-20 08:26:39 -07:00
github-actions[bot]
10cfa27c51
Version Packages
2026-05-13 08:02:51 +00:00
Rich Haines
e66e522f13
Merge pull request #126 from vercel-labs/neutralize-sandbox-workspace
...
Neutralize sandbox workspace path
2026-05-13 10:01:50 +02:00
Allen Zhou
ea8d7abba6
Neutralize sandbox workspace path
2026-05-06 16:52:56 -07:00
Allen Zhou
475b4bc879
Merge pull request #124 from vercel-labs/changeset-release/main
...
Version Packages
@vercel/agent-eval@0.14.0
2026-05-05 16:03:42 -07:00
github-actions[bot]
5294461898
Version Packages
2026-05-05 23:02:49 +00:00
Allen Zhou
9bc474f783
Merge pull request #123 from vercel-labs/a0-local-harness-support
...
Add response-only harness support
2026-05-05 16:01:59 -07:00
Allen Zhou
c52126f198
Keep agent config validation strict
2026-05-05 15:58:10 -07:00
Allen Zhou
07614ec3b7
Add response-only harness support
2026-05-05 15:37:24 -07:00
github-actions[bot]
f5cba1ea67
Version Packages ( #122 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.13.1
2026-04-26 08:51:13 -04:00
Jude Gao
384133b982
[CLI] Surface AI Gateway errors during failure classification ( #121 )
2026-04-26 02:22:46 -04:00
github-actions[bot]
f7e79f7a1f
Version Packages ( #120 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.13.0
2026-04-24 23:33:25 -04:00
Jude Gao
748fcc08d3
[Release] Use github-api commit mode for signed commits ( #119 )
...
* [CLI] Remove auto-retry of non-model failures
* [Release] Use github-api commit mode for signed commits
2026-04-24 23:31:51 -04:00
Jude Gao
660ea3ea20
[CLI] Remove auto-retry of non-model failures ( #118 )
2026-04-24 21:52:45 -04:00
github-actions[bot]
4b81d0f676
Version Packages ( #116 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.12.1
2026-04-17 03:09:12 +08:00
Jude Gao
a3c2136f03
[Sandbox] Reconnect on terminated streams to avoid spurious failures on long commands ( #115 )
2026-04-16 15:06:15 -04:00
github-actions[bot]
69db6fca7b
Version Packages ( #114 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.12.0
2026-04-16 11:39:53 -04:00
Jude Gao
a209ae099f
[Claude Code] Forward cliPackage and effort agentOptions to the CLI ( #113 )
2026-04-16 11:38:02 -04:00
github-actions[bot]
4385c53c84
Version Packages ( #111 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.11.0
2026-04-15 10:53:20 -04:00
Jude Gao
f838bd7363
[OpenCode] Pass timeout to provider config ( #112 )
2026-04-15 10:41:25 -04:00
Jude Gao
481637dd6e
Auto-retry non-model failures with configurable retry rounds ( #110 )
2026-04-14 23:26:27 -04:00
github-actions[bot]
01c6e88322
Version Packages ( #109 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.10.1
2026-04-14 22:01:08 -04:00
Jude Gao
5185640dde
[OpenCode] Deep-merge vercel provider config and use user-space binary path ( #108 )
2026-04-14 21:57:07 -04:00
Jude Gao
cb441f58be
fix(ci): remove stale pnpm-lock.yaml to fix changeset publish
...
Changesets was detecting pnpm-lock.yaml and trying to publish with
pnpm, which isn't installed in CI. The project uses npm.
@vercel/agent-eval@0.10.0
2026-04-14 15:42:58 -04:00
github-actions[bot]
38faa84631
Version Packages ( #107 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-14 15:31:11 -04:00
Rune Botten
df0dfb6657
fix(codex): append to config.toml instead of overwriting ( #104 )
2026-04-14 15:28:28 -04:00