Commit Graph

5 Commits

Author SHA1 Message Date
Jude Gao 087415c73d [O11y] Add transcript parsers for Gemini and Cursor agents (#74)
* [O11y] Add transcript parsers for Gemini and Cursor agents

* Handle Gemini direct-API transcript format

The Gemini agent has two output formats: the CLI format (via OpenCode
framework with part.tool/part.state) and the direct-API format (with
tool_name/parameters and separate tool_result events). The initial
parser only handled the CLI format, leaving direct-API transcripts
with all-zero metrics. This adds handling for both.

* Aggregate Gemini direct-API delta messages into single turns

Contiguous assistant delta messages are now merged into one message
event so totalTurns reflects actual conversation turns instead of 0.

* Add changeset
2026-02-12 16:50:18 -05:00
Jude Gao bc5114cea6 [CLI] Live terminal dashboard for parallel experiment runs (#63)
* [CLI] Live terminal dashboard for parallel experiment runs

* Fix lint errors: remove unused renderDetailLine and verbose
2026-02-10 17:55:04 -05:00
Jude Gao 3e57444699 Add run-all command with housekeeping (#41)
New run-all command auto-discovers experiments/*.ts and orchestrates
the full pipeline: fingerprint reuse, failure classification,
auto-retry of infra failures, and housekeeping.

Housekeeping runs after each experiment: keeps only the latest valid
result per (experiment, eval) pair, removes duplicates and incomplete
results, cleans up empty timestamp directories.

Supports --dry (preview), --force (ignore fingerprints), --smoke
(1 eval per experiment), and optional experiment name/glob arguments.
2026-02-08 00:21:05 -05:00
Jude Gao 59826e7b9a [Agents] Add AI SDK Harness for evaluating any AI Gateway model (#8) 2026-02-03 19:10:44 -05:00
Jude Gao 97b9cc5ce4 [Config] Move sandbox backend selection from env var to experiment config (#6) 2026-02-04 03:07:00 +08:00