Files
Colby McHenry 03069c9118 docs(benchmarks): current-build A/B on all 7 README repos + fix token-measurement bug
Re-ran the README benchmark on the current build (7 repos reindexed, median of 4): avg 35% cost / 57% tokens / 46% time / 71% tool calls saved — reproduces the published README (35/59/49/70), no regression. Adds bench-readme.sh + parse-bench-readme.mjs harness.

Fixes a token-measurement bug: result.usage is last-turn-only in current Claude Code; must sum per-turn assistant usage for cumulative tokens. Corrects the earlier excalidraw note (its '-34% tokens' was off this bug; real ~90%) and the cost MECHANISM (volume/fewer-turns, not cache-ability — the without-arm's huge token volume is mostly cheap cache-reads, so token savings 57% > cost savings 35%). Cost/time were always correct.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 03:41:16 -05:00
..