1b009bbbda
Merge pull request #19 from vercel-labs/changeset-release/main
...
Version Packages
2026-02-06 13:10:57 -08:00
f7978683c8
Version Packages
2026-02-06 19:40:38 +00:00
ef7e40617c
Merge pull request #18 from vercel-labs/edit-prompt
...
feat: add `editPrompt` config to experiment
2026-02-06 20:40:00 +01:00
85bfb21b54
feat: add editPrompt config to experiment
2026-02-06 20:27:02 +01:00
6ce61c80d8
Merge pull request #17 from vercel-labs/allen/playground
...
[playground] add a playground for visualizing eval results
2026-02-06 08:46:22 -08:00
1b6945a9e7
docs update
2026-02-05 14:49:06 -08:00
27bfad6346
docs
2026-02-05 14:43:49 -08:00
8888c8af0b
remove cursor skills
2026-02-05 14:38:27 -08:00
dcc6aedcf2
remove loading and paginate
2026-02-05 14:34:37 -08:00
92d2cea3db
loading
2026-02-05 14:25:52 -08:00
48f4a7da4c
clean up
2026-02-05 14:19:31 -08:00
ffc65ea271
.
2026-02-05 14:06:38 -08:00
561da6e0c4
playground + skills
2026-02-05 13:52:23 -08:00
b29ee92a8e
init ui
2026-02-05 12:57:39 -08:00
b5367158e7
Merge pull request #16 from vercel-labs/changeset-release/main
...
Version Packages
2026-02-05 10:13:52 -08:00
07fe53fd4e
Version Packages
2026-02-05 18:12:29 +00:00
3fb6ec46ea
Merge branch 'main' of https://github.com/vercel-labs/agent-eval
2026-02-05 10:11:43 -08:00
c5d6fac6ae
fix build
2026-02-05 10:10:40 -08:00
558abe5960
chore: changeset for #10
2026-02-05 19:09:45 +01:00
1a830ffcd2
Merge branch 'main' of https://github.com/vercel-labs/agent-eval
2026-02-05 10:08:41 -08:00
4fc79391de
return changesets for publishing
2026-02-05 10:08:33 -08:00
cb14077fe2
Merge pull request #10 from paoloricciuti/array-model
...
feat: accept array of models in experiment
2026-02-05 19:07:03 +01:00
b6c1db7624
release
2026-02-05 10:03:48 -08:00
be0ea788b4
Add CONTRIBUTING.md ( #15 )
2026-02-05 12:57:53 -05:00
9cfa555dfb
use provenance
2026-02-05 09:46:14 -08:00
bb9b9eeba4
Version Packages ( #14 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-05 12:40:47 -05:00
8398eb7e0e
Merge pull request #13 from vercel-labs/allen/o11y-changeset
...
[changeset] o11y feat
2026-02-05 09:37:26 -08:00
bb3c09bf5d
o11y changeset
2026-02-05 09:36:36 -08:00
7c17e5cdd5
[publishing] use oidc for release flow
2026-02-05 09:32:55 -08:00
e8041959ef
changesets ( #12 )
2026-02-05 12:27:04 -05:00
2530301c1d
[o11y] Support normalized and raw transcripts from agent runs ( #9 )
...
* start o11y work
* default to normalized transcripts
* handle lack of agent parser error
* fix codex gateway config
* include transcript raw path
2026-02-05 12:17:32 -05:00
474991cb5a
Remove AI SDK harness agent ( #11 )
2026-02-05 12:10:09 -05:00
2a53c40864
feat: accept array of models in experiment
2026-02-05 16:42:55 +01:00
c4551d2c4d
fix: Codex CLI wire_api and auth configuration
...
- Change wire_api from "chat" to "responses" (required by newer Codex CLI)
- Fix direct OpenAI API auth by using codex login --with-api-key
- Fix model name for direct API (remove openai/ prefix)
- Update integration tests to assert agent success (not just completion)
- Fix test isolation with beforeAll hooks for project initialization
Codex via Vercel AI Gateway still has a separate tools.7.type API error
that needs to be fixed on the AI Gateway side.
2026-02-04 20:48:00 -05:00
df205bbcbc
chore: bump version to 0.0.8
2026-02-03 19:11:25 -05:00
59826e7b9a
[Agents] Add AI SDK Harness for evaluating any AI Gateway model ( #8 )
2026-02-03 19:10:44 -05:00
02a5a48a7b
chore: bump version to 0.0.7
2026-02-03 15:55:30 -05:00
4b2040f190
chore: change default timeout from 5 minutes to 10 minutes
2026-02-03 15:49:12 -05:00
0bb8dbe39d
fix: read CLI version from package.json instead of hardcoding
2026-02-03 15:26:38 -05:00
c08fff35a1
fix: load .env.local in CLI before .env
2026-02-03 15:25:40 -05:00
c5105339cc
chore: bump version to 0.0.6
2026-02-03 15:17:21 -05:00
6844159ca7
[OpenCode] Fix model format and Docker file permissions for agent compatibility ( #7 )
...
OpenCode requires the vercel/ prefix in model strings (e.g., vercel/minimax/minimax-m2.1)
because the agent's opencode.json config sets up a vercel provider.
Also fixes Docker sandbox file permissions by adding chown after uploadFiles.
2026-02-03 15:14:30 -05:00
97b9cc5ce4
[Config] Move sandbox backend selection from env var to experiment config ( #6 )
2026-02-04 03:07:00 +08:00
9d6c0cf6d9
Merge pull request #5 from vercel-labs/fix/timeout-enforcement
...
[Runner] Enforce configured timeout at runner level
2026-02-04 02:58:32 +08:00
438611ff12
fix: signal abort to agent on timeout for resource cleanup
...
When timeout fires, the agent's AbortController is now triggered so
agents can clean up their sandboxes instead of continuing to run
as orphaned processes.
- Always pass AbortSignal to agents (not just for earlyExit)
- Call abort() before rejecting on timeout
- Clear timeout timer when agent completes successfully
- Add test verifying abort signal is triggered on timeout
2026-02-03 13:56:32 -05:00
0f5562dac6
fix: enforce timeout at runner level with Promise.race
...
The timeout parameter was being passed to agents but never enforced at the
runner level. If an agent didn't respect the timeout, evals could run
indefinitely.
This fix wraps agent.run() with Promise.race() to ensure the runner enforces
the configured timeout, returning a failed result if the timeout is exceeded.
2026-02-03 13:48:41 -05:00
5315e425f0
chore: bump version to 0.0.5
2026-02-03 12:48:54 -05:00
49623c248f
Merge pull request #4 from vercel-labs/feat/docker-sandbox
...
[Sandbox] Add Docker backend with auto-detection fallback from Vercel
2026-02-04 01:45:45 +08:00
3db517f731
chore: add test:integration:docker and test:integration:vercel scripts
2026-02-03 12:40:33 -05:00
3df0816f98
chore: remove docs and test scripts
2026-02-03 12:39:52 -05:00
Allen Zhou