mirror of
https://github.com/vercel-labs/agent-eval.git
synced 2026-09-19 07:11:21 +08:00
Jude Gao
087415c73d
* [O11y] Add transcript parsers for Gemini and Cursor agents * Handle Gemini direct-API transcript format The Gemini agent has two output formats: the CLI format (via OpenCode framework with part.tool/part.state) and the direct-API format (with tool_name/parameters and separate tool_result events). The initial parser only handled the CLI format, leaving direct-API transcripts with all-zero metrics. This adds handling for both. * Aggregate Gemini direct-API delta messages into single turns Contiguous assistant delta messages are now merged into one message event so totalTurns reflects actual conversation turns instead of 0. * Add changeset
1.9 KiB
1.9 KiB
Agent Eval
Development Guidelines
No New Environment Variables
Configuration should be done through the experiment config file, not environment variables.
- All experiment settings belong in
ExperimentConfig(seesrc/lib/types.ts) - The only acceptable env vars are API keys (e.g.,
AI_GATEWAY_API_KEY,ANTHROPIC_API_KEY,VERCEL_TOKEN) - When adding new configuration options, add them to the config schema in
src/lib/config.ts
DRY & Colocation
- Don't duplicate logic across files. If the same check exists in multiple places, extract it into a single function and import it.
- Colocate shared helpers in the module that owns the concept (e.g. classification logic belongs in
classifier.ts, not scattered acrosshousekeeping.tsandresults.ts).
Testing
- Always use the existing integration test framework (
src/integration.test.ts) for testing - Do not create standalone test scripts in
/tmp- they won't have proper module resolution - Run integration tests with:
INTEGRATION_TEST=1 npx vitest run src/integration.test.ts --testNamePattern="<pattern>"
Adding a New Agent
When adding a new agent to the framework, follow this checklist:
- Agent implementation in
src/lib/agents/ - Register in
src/lib/agents/index.ts - Add to
AgentTypeinsrc/lib/types.ts - Add to config schema in
src/lib/config.ts - Add API key config in
src/lib/agents/shared.ts - Add transcript parser in
src/lib/o11y/parsers/ - Register parser in
src/lib/o11y/parsers/index.ts - Export parser from
src/lib/o11y/index.ts - Add parser tests in
src/lib/o11y/o11y.test.ts - Add integration tests in
src/integration.test.ts - Update README.md
Pull Requests
- Every PR that changes user-facing behavior should update the README
- Every PR should include a changeset (
npx changeset) for version management