Colby McHenry 04c0f8eab5 test(agent-eval): sum tokens per turn — result.usage stopped being cumulative ...

"Tokens processed" was read off result.usage. That was correct when the README
figures were measured; in current Claude Code the field reports the LAST turn
only. Nothing here changed — the host did, silently — and the harness kept
reporting the smaller number.

The error is one-sided, which makes it worse than noise: it under-counts
whichever arm takes more turns, and that is always the WITHOUT arm. On the
2026-08-05 campaign it turned a real 62% token saving into 19% and invented a
token REGRESSION on tokio (-41%) and alamofire (-25%) that does not exist. Those
numbers were one push away from the README.

Now summed per assistant request and deduped by message.id, the same rule the
occupancy timeline already used — Claude Code emits one event per content block
carrying identical usage, so summing per event double-counts (~1.7x measured).

CLAUDE.md already warned about this field. The code did not follow; it does now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-05 21:27:02 -05:00

71 KiB

Raw Permalink Blame History

View Raw