skills/vercel-labs__agent-eval
1
0
Fork 0
mirror of https://github.com/vercel-labs/agent-eval.git synced 2026-09-19 07:11:21 +08:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 7e9aae4f77 Merge pull request #202 from vercel-labs/changeset-release/main main @vercel/agent-eval@2.3.0 Rich Haines 2026-09-18 07:03:48 +02:00
  • 4a44e5c639 Version Packages #202 github-actions[bot] 2026-09-18 05:01:17 +00:00
  • e4c0b8894a Merge pull request #201 from vercel-labs/rh/upgrade-sandbox-v3 Rich Haines 2026-09-18 07:00:27 +02:00
  • fed031c2cc feat: opt-in sandbox image and user for the Vercel backend #201 molebox 2026-09-17 21:43:14 +02:00
  • 2cc99ee6bf fix: upgrade @vercel/sandbox to v3 without changing sandbox behavior molebox 2026-09-17 20:07:45 +02:00
  • 7a4b029980 fix: nested eval directory with housekeeping and result checking #198 fix/nested-eval-results Rihan Arfan 2026-09-03 15:30:29 +01:00
  • fb01d911a0 Merge pull request #197 from vercel-labs/changeset-release/main @vercel/agent-eval@2.2.1 Jude Gao 2026-08-25 12:28:26 -04:00
  • c29fc1da79 Version Packages #197 github-actions[bot] 2026-08-25 16:23:20 +00:00
  • 701c1e1448 Merge pull request #195 from vercel-labs/ensure-validation-runner Jude Gao 2026-08-25 12:21:42 -04:00
  • af0ef7a9f2 Merge pull request #196 from vercel-labs/async-judge-matchers Jude Gao 2026-08-25 12:21:32 -04:00
  • c1cd91f151 Merge remote-tracking branch 'origin/main' into ensure-validation-runner #195 Jude Gao 2026-08-25 11:48:39 -04:00
  • 3cc55a7764 Judge matchers must not block the vitest worker #196 Jude Gao 2026-08-24 23:42:48 -04:00
  • 2bcb34a79a Install vitest before validating when the agent removed it Jude Gao 2026-08-24 23:23:32 -04:00
  • d425388914 Merge pull request #194 from vercel-labs/changeset-release/main @vercel/agent-eval@2.2.0 Rich Haines 2026-08-24 18:08:47 +02:00
  • 5a43a071ca Version Packages #194 github-actions[bot] 2026-08-24 16:07:01 +00:00
  • bd5c186821 Merge pull request #193 from vercel-labs/rh/add-fx-agent Rich Haines 2026-08-24 18:06:00 +02:00
  • d208267116 fix: validate fx judge config path #193 molebox 2026-08-24 17:24:20 +02:00
  • 5d7aef26a7 feat: add fx research agent molebox 2026-08-24 17:08:17 +02:00
  • 46102860ff Merge pull request #192 from vercel-labs/changeset-release/main @vercel/agent-eval@2.1.0 Rich Haines 2026-08-21 13:51:06 +02:00
  • 352ded78bf Version Packages #192 github-actions[bot] 2026-08-21 11:45:29 +00:00
  • a210dd1292 Merge pull request #191 from vercel-labs/rh/opt-in-agent-capabilities-pr Rich Haines 2026-08-21 13:44:39 +02:00
  • ceba203690 feat: add opt-in agent runtime controls #191 molebox 2026-08-19 18:42:54 +02:00
  • 32dc9a1ade Merge pull request #188 from vercel-labs/changeset-release/main @vercel/agent-eval@2.0.0 Jude Gao 2026-08-19 11:46:02 -04:00
  • 338f6adbb0 Version Packages #188 github-actions[bot] 2026-08-19 15:16:09 +00:00
  • fa4a97a79d Merge pull request #169 from huang-julien/fix/codex_transcript Jude Gao 2026-08-19 11:14:40 -04:00
  • 1835669fb9 Merge pull request #166 from huang-julien/fix/provide_type_augment Jude Gao 2026-08-19 11:14:24 -04:00
  • d9fcd25054 Merge pull request #177 from Sidnioulz/claude/distracted-poincare-5057e4 Jude Gao 2026-08-19 11:14:10 -04:00
  • a11801125b Merge pull request #158 from huang-julien/fix/docker_multiplexes_exe_stream Jude Gao 2026-08-19 11:13:33 -04:00
  • f867b13401 Add changeset for the Codex transcript fallback #169 Jude Gao 2026-08-19 10:58:47 -04:00
  • 65524edcc9 fix: codex transciprt fallback in runner Julien Huang 2026-07-01 16:08:50 +02:00
  • ba8149fa75 Ship eval-helper.d.mts in dist so the ./eval types resolve #166 Jude Gao 2026-08-19 10:58:29 -04:00
  • e37d8f0d7d Drop unused import, revert lockfile churn, add changeset #158 Jude Gao 2026-08-19 10:57:03 -04:00
  • 974cda2689 fix(agent-eval): provide type augmentation for agent-eval/eval Julien Huang 2026-06-30 17:08:40 +02:00
  • 48ba69098c refactor: use PassThrough Julien Huang 2026-06-30 11:20:53 +02:00
  • 5bf88bb093 fix(agent-aevla): uise demucStream Julien Huang 2026-06-26 16:59:20 +02:00
  • 1bf90ea590 Keep credential redaction working now that generatedFiles holds bytes #177 Jude Gao 2026-08-19 11:01:47 -04:00
  • 29fc098a89 Merge branch 'main' into fix-177 Jude Gao 2026-08-19 10:59:05 -04:00
  • a2f741f916 Merge pull request #187 from vercel-labs/changeset-release/main @vercel/agent-eval@1.5.1 Jude Gao 2026-08-17 16:52:04 -04:00
  • 0d838b3757 Version Packages #187 github-actions[bot] 2026-08-17 20:46:35 +00:00
  • 5f4ed44faa Merge pull request #186 from vercel-labs/jude/redact-run-credentials Jude Gao 2026-08-17 16:45:46 -04:00
  • c8ac6841c2 Redact run credentials from agent run results #186 Jude Gao 2026-08-17 15:53:17 -04:00
  • b4b69f00d6 Merge pull request #185 from vercel-labs/changeset-release/main @vercel/agent-eval@1.5.0 OwenKephart 2026-08-11 17:30:03 -07:00
  • 1345eaa3ec Version Packages #185 github-actions[bot] 2026-08-12 00:14:58 +00:00
  • 22045330a2 Merge pull request #181 from vercel-labs/feat/custom-agent-registration OwenKephart 2026-08-11 17:14:02 -07:00
  • 95a5e159c7 feat: add natural multi-turn interaction #183 feat/natural-multi-turn-interaction owenkephart 2026-08-11 23:00:18 +00:00
  • 4c9bf92fe6 fix: snapshot exact fixture starting states #182 feat/reusable-sandbox-templates owenkephart 2026-08-11 23:25:55 +00:00
  • 5224c485e8 fix: reset template fixture files before overlay owenkephart 2026-08-11 22:45:06 +00:00
  • 1527f0eada feat: add reusable sandbox templates owenkephart 2026-08-11 22:44:36 +00:00
  • 88c3c4ee00 test: require definitions for registered agents #181 #178 owenkephart 2026-08-11 21:40:35 +00:00
  • cc57b91910 feat: support registered custom agents owenkephart 2026-08-11 21:19:01 +00:00
  • 93e3131c58 Adjust comments Steve Dodier-Lazaro 2026-07-29 16:02:00 +02:00
  • 3be067057d Fix silent binary-file corruption when collecting results Steve Dodier-Lazaro 2026-07-29 12:20:58 +02:00
  • 8d886792a0 Merge pull request #173 from vercel-labs/changeset-release/main @vercel/agent-eval@1.4.0 Rich Haines 2026-07-22 17:30:38 +02:00
  • 092dfc7a35 Version Packages #173 github-actions[bot] 2026-07-22 13:40:24 +00:00
  • 074f157b6b Merge pull request #175 from vercel-labs/rh/codex-native-default-tools Rich Haines 2026-07-22 15:39:41 +02:00
  • dd84f36f41 Address second review pass: test modelRepair plumbing, carry it on the fallback channel, tighten canary timeout #175 molebox 2026-07-22 15:33:01 +02:00
  • f820e3317a Address review: propagate modelRepair, preserve canary output, memoize canary per sandbox molebox 2026-07-22 14:46:23 +02:00
  • f661766173 agent-eval: verify and repair the shell tool for native-default Codex runs molebox 2026-07-22 14:11:13 +02:00
  • 21419b22a5 Merge pull request #174 from vercel-labs/jude/to-contain-text-regex Jude Gao 2026-07-17 15:39:36 -04:00
  • e45907c90b Accept RegExp needles in toContainText #174 Jude Gao 2026-07-17 15:36:30 -04:00
  • c2c458e11e Merge pull request #172 from vercel-labs/jude/transcript-to-contain-text Jude Gao 2026-07-16 20:48:40 -04:00
  • 4f8732ccdd Add deterministic transcript matcher expect(transcript).toContainText() #172 Jude Gao 2026-07-16 20:41:24 -04:00
  • d33f7cddac Merge pull request #171 from vercel-labs/changeset-release/main @vercel/agent-eval@1.3.1 Rich Haines 2026-07-06 18:42:27 +02:00
  • fddf0a39fe Version Packages #171 github-actions[bot] 2026-07-06 16:41:41 +00:00
  • 435879923d Merge pull request #170 from vercel-labs/rh/opencode-gateway-model-prefix Rich Haines 2026-07-06 18:40:57 +02:00
  • d6e9c86c76 agent-eval: route OpenCode explicit model overrides through the AI Gateway #170 molebox 2026-07-06 12:06:21 +02:00
  • 1d1f4f60b2 Merge pull request #167 from vercel-labs/changeset-release/main @vercel/agent-eval@1.3.0 Jude Gao 2026-07-02 10:02:52 -04:00
  • a9a1d16e6a Version Packages #167 github-actions[bot] 2026-06-30 17:15:02 +00:00
  • 6ebfe82f39 Merge pull request #165 from vercel-labs/jude/agent-eval-eval-staleness Jude Gao 2026-07-01 01:14:13 +08:00
  • b35873ca4a [agent-eval] Incremental eval staleness: status / run + content-aware fingerprints #165 Jude Gao 2026-06-30 12:36:39 -04:00
  • b8bd4ec7f1 Merge pull request #164 from vercel-labs/jude/agent-eval-pinnable-judge Jude Gao 2026-07-01 00:51:25 +08:00
  • 2905905a27 [agent-eval] Pin the agentic judge to a fixed agent and model #164 Jude Gao 2026-06-29 23:45:45 -04:00
  • 6867dc1cf7 Merge pull request #163 from vercel-labs/changeset-release/main @vercel/agent-eval@1.2.0 Jude Gao 2026-06-30 06:02:53 +08:00
  • 9b2caa0106 Version Packages #163 github-actions[bot] 2026-06-29 21:33:53 +00:00
  • 4c67a083a7 Merge pull request #162 from vercel-labs/jude/agent-eval-eval-matcher Jude Gao 2026-06-30 05:33:06 +08:00
  • 122a329c44 Merge pull request #160 from vercel-labs/jude/agent-eval-plugin-migration Jude Gao 2026-06-30 05:32:58 +08:00
  • 7021c4b914 agent-eval: agentic LLM-judge matcher for EVAL.ts #162 Jude Gao 2026-06-29 14:28:25 -04:00
  • 821cc726e9 agent-eval: add judge() — agentic LLM judge on the plugin model #161 jude/agent-eval-judge-api Jude Gao 2026-06-26 18:24:01 -04:00
  • d9b0cb1696 agent-eval: deslop plugin migration #160 Jude Gao 2026-06-26 20:20:18 -04:00
  • 1bdd92462f agent-eval: migrate codex/opencode/gemini/cursor to the plugin model Jude Gao 2026-06-26 17:18:06 -04:00
  • 45e22f6b4e agent-eval: plugin model — generic orchestrator + claude-code migrated Jude Gao 2026-06-26 17:07:20 -04:00
  • fec27a21f2 agent-eval judge: codebase() — judge explores the workspace #159 jude/agent-eval-judge-codebase Jude Gao 2026-06-26 11:37:57 -04:00
  • c9bb1f3aaa agent-eval judge: real e2e + fixes it surfaced #157 jude/agent-eval-llm-judge Jude Gao 2026-06-26 10:38:12 -04:00
  • e3f73b3a7f feat(agent-eval): LLM-as-judge assertions for EVAL.ts Jude Gao 2026-06-26 10:24:11 -04:00
  • a9dcc9a8c5 Merge pull request #155 from vercel-labs/jude/publish-reconcile Jude Gao 2026-06-11 10:36:14 -07:00
  • ac1bc65c01 Reconcile against registry when changeset publish reports failure #155 Jude Gao 2026-06-11 12:52:51 -04:00
  • ed5fdc4c69 Merge pull request #154 from vercel-labs/changeset-release/main @vercel/agent-eval@1.1.1 Rich Haines 2026-06-11 18:49:03 +02:00
  • bb7cbed5dd Version Packages #154 github-actions[bot] 2026-06-11 16:46:15 +00:00
  • 220a788a7a Merge pull request #152 from vercel-labs/rh/fix-claude-allowed-tools-order Rich Haines 2026-06-11 18:45:20 +02:00
  • c016ea0319 Fix Claude prompt consumed by variadic --allowedTools #152 molebox 2026-06-11 17:00:37 +02:00
  • 090c8290de Merge pull request #151 from vercel-labs/changeset-release/main @vercel/agent-eval@1.1.0 Rich Haines 2026-06-11 16:14:41 +02:00
  • 7a23a1377a Version Packages #151 github-actions[bot] 2026-06-11 14:13:27 +00:00
  • 086a33e0fb Merge pull request #150 from vercel-labs/rh/opt-in-web-research Rich Haines 2026-06-11 16:12:39 +02:00
  • 17d246965c Include webResearch in result-reuse fingerprints when enabled #150 molebox 2026-06-11 15:43:16 +02:00
  • 1d8321f0c6 Thread webResearch through ExperimentConfig and the runner molebox 2026-06-11 15:37:29 +02:00
  • 084d895f66 Add opt-in webResearch option for agent web tools molebox 2026-06-11 15:03:59 +02:00
  • 966aea0b25 Merge pull request #149 from vercel-labs/changeset-release/main @vercel/agent-eval@1.0.1 Rich Haines 2026-06-11 08:29:29 +02:00
  • c4961d7cbc Version Packages #149 github-actions[bot] 2026-06-11 06:26:45 +00:00
  • 1d580086bf Merge pull request #148 from vercel-labs/rh/opencode-observed-model-export Rich Haines 2026-06-11 08:25:50 +02:00
  • b4841d6791 Fix OpenCode observed model extraction for OpenCode >= 1.17.0 #148 molebox 2026-06-10 16:54:43 +02:00