Files
Colby McHenry 1b527aaf0f chore(agent-eval): add agent-behavior eval harness for codegraph MCP usage
Tooling to measure how a Claude Code agent actually uses the codegraph
MCP tools on a real repo — does it lead with codegraph_explore, how many
Read/Grep follow-ups, token cost — for validating tool-guidance changes
(server-instructions, tool descriptions) against real agent behavior.

- itrun.sh drives the real interactive TUI via tmux (the faithful
  Explore path). Hardened for unattended runs: type-and-verify prompt
  delivery (the ❯ glyph is drawn ~6s before the input accepts keys),
  auto-accepts the "trust this folder" dialog, busy-detection keys on
  the universal "(Ns · …)" spinner so the pre-stream thinking phase
  counts as busy, and fails loudly instead of capturing an empty pane.
- parse-session.mjs reports the tool breakdown + token accounting
  (gen / fresh-in / cached-in / billable) from the session and subagent
  logs, consistent across main-thread and subagent runs; counts
  main-thread Bash in the grep verdict.
- run-agent.sh / parse-run.mjs are the headless stream-json complement
  (exact per-tool tokens/cost via claude -p).
- run-interactive-test.md documents how to run it and how completion is
  detected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 22:05:23 -05:00
..