mirror of https://github.com/vercel-labs/agent-eval.git synced 2026-09-19 07:11:21 +08:00

Files

T

Jude Gao 3e57444699 Add run-all command with housekeeping (#41 )

New run-all command auto-discovers experiments/*.ts and orchestrates
the full pipeline: fingerprint reuse, failure classification,
auto-retry of infra failures, and housekeeping.

Housekeeping runs after each experiment: keeps only the latest valid
result per (experiment, eval) pair, removes duplicates and incomplete
results, cleans up empty timestamp directories.

Supports --dry (preview), --force (ignore fingerprints), --smoke
(1 eval per experiment), and optional experiment name/glob arguments.

2026-02-08 00:21:05 -05:00

941 B

Raw Permalink Blame History

Agent Eval

Development Guidelines

No New Environment Variables

Configuration should be done through the experiment config file, not environment variables.

All experiment settings belong in ExperimentConfig (see src/lib/types.ts)
The only acceptable env vars are API keys (e.g., AI_GATEWAY_API_KEY, ANTHROPIC_API_KEY, VERCEL_TOKEN)
When adding new configuration options, add them to the config schema in src/lib/config.ts

Testing

Always use the existing integration test framework (src/integration.test.ts) for testing
Do not create standalone test scripts in /tmp - they won't have proper module resolution
Run integration tests with: INTEGRATION_TEST=1 npx vitest run src/integration.test.ts --testNamePattern="<pattern>"

Pull Requests

Every PR that changes user-facing behavior should update the README
Every PR should include a changeset (npx changeset) for version management