Commit Graph

272 Commits

Author SHA1 Message Date
Rich Haines ff51aaf83d Merge pull request #143 from vercel-labs/changeset-release/main
Version Packages
@vercel/agent-eval@0.14.4
2026-05-29 17:06:41 +02:00
github-actions[bot] e0db9a3813 Version Packages 2026-05-29 15:03:07 +00:00
Rich Haines daaeccb42c Merge pull request #141 from vercel-labs/feat/enable-agent-web-sources
Enable source-capable web tools for agent runs
2026-05-29 17:02:23 +02:00
molebox 74be0aa25b Merge main into feat/enable-agent-web-sources 2026-05-29 16:51:52 +02:00
Allen Zhou cb2280db61 Merge pull request #142 from vercel-labs/changeset-release/main
Version Packages
@vercel/agent-eval@0.14.3
2026-05-28 14:04:44 -07:00
github-actions[bot] a53721752b Version Packages 2026-05-28 18:45:17 +00:00
Allen Zhou 43bec69aba Merge pull request #140 from vercel-labs/fix/codex-reasoning-effort-default
Fix Codex profile reasoning_effort and verbosity defaults
2026-05-28 11:44:28 -07:00
molebox 2d27942cc3 Enable source-capable web tools for agent runs 2026-05-28 11:15:10 +02:00
Allen Zhou af62bc3fe9 Remove vercel-agent-eval-*.tgz from .gitignore
Per review feedback. Local `npm pack` artifacts can be cleaned up by
hand; no need for a project-level ignore rule.
2026-05-27 21:34:43 -07:00
Allen Zhou a9efa3ad1f Fix Codex profile reasoning_effort and verbosity defaults
The Codex CLI defaults both `model_reasoning_effort` and `model_verbosity`
to "low", but `gpt-5.2-codex` (the default Codex model) only accepts
"medium" for both. Out-of-the-box `codex exec` against the AI Gateway
fails with:

  Unsupported value: 'low' is not supported with the 'gpt-5.2-codex'
  model. Supported values are: 'medium'.

The error covers both the `reasoning.effort` and `text.verbosity`
request parameters, depending on which the model rejects first.

Set both fields to "medium" in two places:
- the generated profile config in ~/.codex/default.config.toml
- explicit -c flags on `codex exec`, since CLI flags have the highest
  precedence and we observed the profile-only setting being silently
  overridden by the CLI's "low" default in some Codex versions.

`generateCodexConfig` now accepts an optional `reasoningEffort`
parameter so callers can override per-run via
`model: "gpt-5.2-codex?reasoningEffort=high"`.

Verified end-to-end against the Vercel AI Gateway: a previously-failing
`codex exec` smoke run now completes in ~31s and returns a real
response instead of erroring at `turn.failed`.

Also added `vercel-agent-eval-*.tgz` to .gitignore so local `npm pack`
artifacts don't leak into commits.
2026-05-27 20:19:58 -07:00
Allen Zhou 096192204e Merge pull request #139 from vercel-labs/changeset-release/main
Version Packages
@vercel/agent-eval@0.14.2
2026-05-27 19:55:33 -07:00
github-actions[bot] 8bb80230c4 Version Packages 2026-05-28 02:55:06 +00:00
Allen Zhou 79d0d3b4a0 Merge pull request #138 from vercel-labs/fix/codex-profile-config
Fix Codex profile config
2026-05-27 19:54:11 -07:00
molebox 5950d74405 Fix Codex profile config 2026-05-27 21:36:53 +02:00
Allen Zhou 330b743167 Merge pull request #130 from vercel-labs/changeset-release/main
Version Packages
@vercel/agent-eval@0.14.1
2026-05-20 08:26:39 -07:00
github-actions[bot] 10cfa27c51 Version Packages 2026-05-13 08:02:51 +00:00
Rich Haines e66e522f13 Merge pull request #126 from vercel-labs/neutralize-sandbox-workspace
Neutralize sandbox workspace path
2026-05-13 10:01:50 +02:00
Allen Zhou ea8d7abba6 Neutralize sandbox workspace path 2026-05-06 16:52:56 -07:00
Allen Zhou 475b4bc879 Merge pull request #124 from vercel-labs/changeset-release/main
Version Packages
@vercel/agent-eval@0.14.0
2026-05-05 16:03:42 -07:00
github-actions[bot] 5294461898 Version Packages 2026-05-05 23:02:49 +00:00
Allen Zhou 9bc474f783 Merge pull request #123 from vercel-labs/a0-local-harness-support
Add response-only harness support
2026-05-05 16:01:59 -07:00
Allen Zhou c52126f198 Keep agent config validation strict 2026-05-05 15:58:10 -07:00
Allen Zhou 07614ec3b7 Add response-only harness support 2026-05-05 15:37:24 -07:00
github-actions[bot] f5cba1ea67 Version Packages (#122)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.13.1
2026-04-26 08:51:13 -04:00
Jude Gao 384133b982 [CLI] Surface AI Gateway errors during failure classification (#121) 2026-04-26 02:22:46 -04:00
github-actions[bot] f7e79f7a1f Version Packages (#120)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.13.0
2026-04-24 23:33:25 -04:00
Jude Gao 748fcc08d3 [Release] Use github-api commit mode for signed commits (#119)
* [CLI] Remove auto-retry of non-model failures

* [Release] Use github-api commit mode for signed commits
2026-04-24 23:31:51 -04:00
Jude Gao 660ea3ea20 [CLI] Remove auto-retry of non-model failures (#118) 2026-04-24 21:52:45 -04:00
github-actions[bot] 4b81d0f676 Version Packages (#116)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.12.1
2026-04-17 03:09:12 +08:00
Jude Gao a3c2136f03 [Sandbox] Reconnect on terminated streams to avoid spurious failures on long commands (#115) 2026-04-16 15:06:15 -04:00
github-actions[bot] 69db6fca7b Version Packages (#114)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.12.0
2026-04-16 11:39:53 -04:00
Jude Gao a209ae099f [Claude Code] Forward cliPackage and effort agentOptions to the CLI (#113) 2026-04-16 11:38:02 -04:00
github-actions[bot] 4385c53c84 Version Packages (#111)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.11.0
2026-04-15 10:53:20 -04:00
Jude Gao f838bd7363 [OpenCode] Pass timeout to provider config (#112) 2026-04-15 10:41:25 -04:00
Jude Gao 481637dd6e Auto-retry non-model failures with configurable retry rounds (#110) 2026-04-14 23:26:27 -04:00
github-actions[bot] 01c6e88322 Version Packages (#109)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.10.1
2026-04-14 22:01:08 -04:00
Jude Gao 5185640dde [OpenCode] Deep-merge vercel provider config and use user-space binary path (#108) 2026-04-14 21:57:07 -04:00
Jude Gao cb441f58be fix(ci): remove stale pnpm-lock.yaml to fix changeset publish
Changesets was detecting pnpm-lock.yaml and trying to publish with
pnpm, which isn't installed in CI. The project uses npm.
@vercel/agent-eval@0.10.0
2026-04-14 15:42:58 -04:00
github-actions[bot] 38faa84631 Version Packages (#107)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-14 15:31:11 -04:00
Rune Botten df0dfb6657 fix(codex): append to config.toml instead of overwriting (#104) 2026-04-14 15:28:28 -04:00
Jerilyn Zheng 8d138a28e7 Add agentOptions support to ExperimentConfig (#106)
Allows experiment config files to pass agent-specific options (like
binaryUrl and extraProviders) at runtime via a new agentOptions field.
Previously these could only be set at agent registration time, making
configs for unreleased models non-replicable.

The options flow: ExperimentConfig → runner → AgentRunOptions → agent.run().

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 15:27:00 -04:00
Jude Gao d4c0a01a08 gpt 5.4 integration test 2026-03-21 13:43:14 -04:00
github-actions[bot] 90f33d8e6c Version Packages (#100)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.9.5
2026-03-21 11:20:08 -04:00
Jude Gao ec11c4a6b5 [CLI] Add override flag to dotenv config to allow shell env vars to take precedence (#99) 2026-03-21 11:07:10 -04:00
github-actions[bot] 7c6fee55eb Version Packages (#98)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.9.4
2026-03-19 10:51:29 -04:00
Jude Gao 4815babe17 Bump minimatch to 10.2.4 to resolve ReDoS CVE (#97)
* Bump minimatch to 10.2.4 to resolve ReDoS CVE

* Fix lockfile: use npm instead of pnpm
2026-03-19 10:49:47 -04:00
github-actions[bot] c9c0db1d7d Version Packages (#93)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@vercel/agent-eval@0.9.3
2026-03-19 01:21:31 -04:00
Jude Gao bb69d68f47 Fix changeset package name to match workspace (#96) 2026-03-19 01:18:42 -04:00
Jude Gao 9dcc63abc6 Use built-in OpenAI provider for Codex (#94) 2026-03-19 01:11:09 -04:00
Jude Gao 6ced2ea189 Strip model prefix from Codex CLI flag for direct API (#95)
* Use built-in OpenAI provider for Codex

* Strip openai/ prefix from --model CLI flag for direct API path

The config.toml correctly strips the provider prefix for direct OpenAI API
usage, but the --model CLI flag still passed the prefixed name (e.g.
"openai/gpt-5.2-codex"), causing a "model not found" error.
2026-03-19 01:03:52 -04:00