Rewrites the README to document all features added in v0.3.0:
default run-all behavior, --smoke/--force flags, content fingerprinting
and result reuse, AI failure classification, auto-retry, and
housekeeping. Fixes outdated result file paths and summary.json format.
`npx agent-eval` with no args now discovers and runs all experiments
(same as `agent-eval run-all`). Running a single experiment still
works: `npx agent-eval my-experiment`.
New run-all command auto-discovers experiments/*.ts and orchestrates
the full pipeline: fingerprint reuse, failure classification,
auto-retry of infra failures, and housekeeping.
Housekeeping runs after each experiment: keeps only the latest valid
result per (experiment, eval) pair, removes duplicates and incomplete
results, cleans up empty timestamp directories.
Supports --dry (preview), --force (ignore fingerprints), --smoke
(1 eval per experiment), and optional experiment name/glob arguments.
New classifier.ts module classifies failed evals as model, infra, or
timeout. Rule-based fast path catches ~80% of cases (timeout errors,
missing transcripts, rate limits, tiny transcripts with API errors).
Falls back to AI classification via gateway(anthropic/claude-sonnet-4-5)
with sandboxed read-only tools. Gracefully degrades to rule-based only
when AI_GATEWAY_API_KEY is not set.
shouldRetry() returns true when all failures are non-model, enabling
auto-retry of infra-failed evals.
summary.json now supports classification and valid fields. Results
marked valid: false are excluded from fingerprint reuse.
Computes a SHA-256 fingerprint from all eval directory files plus the
config fields that affect results (agent, model, scripts, timeout,
earlyExit, runs). The fingerprint is stored in summary.json.
scanReusableResults() scans existing results and returns matches where
the fingerprint is identical, the result is valid, and passedRuns > 0.
Functions like setup and editPrompt can't be hashed — use --force to
re-run when these change.
Picks the first eval alphabetically and runs it once per model.
Useful for verifying API keys, model IDs, and sandbox connectivity
before committing to a full suite, especially with model arrays.
When `scripts: ['eval']` is configured, the script output was written
to `outputs/eval.txt` which collided with the EVAL.ts test output at
the same path. Script outputs now go to `outputs/scripts/{name}.txt`.
Allen Zhou