Files
Colby McHenry dc19eaba3e test(agent-eval): exclude MCP-cold-start-raced runs from bench medians
A headless run where the agent fired before `codegraph serve --mcp` finished
registering tools ("No such tool available") floods grep/Read and spirals — it
measures CodeGraph's startup latency, not its steady-state value. parse-bench-
readme.mjs now flags such runs (a tool_result containing "No such tool
available") and excludes them from the per-repo medians, printing "[N raced runs
excluded]". CG_INCLUDE_RACED=1 keeps them to inspect the raw distribution. On the
last sweep this lifted the (artifact-contaminated) average time 9%→17% and tools
64%→66% — the raced vscode/django runs were dragging the medians.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 20:53:11 -05:00
..