Re-ran the README benchmark on the current build (7 repos reindexed, median of 4): avg 35% cost / 57% tokens / 46% time / 71% tool calls saved — reproduces the published README (35/59/49/70), no regression. Adds bench-readme.sh + parse-bench-readme.mjs harness.
Fixes a token-measurement bug: result.usage is last-turn-only in current Claude Code; must sum per-turn assistant usage for cumulative tokens. Corrects the earlier excalidraw note (its '-34% tokens' was off this bug; real ~90%) and the cost MECHANISM (volume/fewer-turns, not cache-ability — the without-arm's huge token volume is mostly cheap cache-reads, so token savings 57% > cost savings 35%). Cost/time were always correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Records why codegraph read savings (-75%) under-convert to wall-clock (-16%): the bottleneck is round-trips + the synthesis turn, not reads. Ablation (arms A-I) shows explore is 68% of payload but load-bearing, trace is path-scoped but under-adopted, instruction/description steering cannot match an append-prompt's salience (and regresses), and the shippable win is making the trace output sufficient (arm I). Adds harness: seq-matrix, run-arms/arms-*, parse-arms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dev-only validation harness for the dynamic-dispatch coverage work:
- probe-{trace,node,context,explore}.mjs: drive MCP tools against a built index
without a full agent run.
- block-read-hook.sh + hook-settings.json: PreToolUse experiment that denies
source Reads to measure codegraph sufficiency (forced Read-0).
- docs/design/: callback-edge-synthesis + dynamic-dispatch-coverage playbook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the old interactive publish.js script with two Claude skills and
a full agent-evaluation harness:
- `.claude/skills/audit/` — `/audit` skill drives `scripts/agent-eval/audit.sh`
to benchmark retrieval quality (with vs. without codegraph) on a chosen
real-world repo from the new `corpus.json` (17 repos across 14 languages).
- `.claude/skills/publish/` — `/publish` skill orchestrates the full release
workflow (preflight → changelog → confirmation gate → bump/build → npm
publish → GitHub release), replacing `publish.js`.
- `scripts/agent-eval/` — headless (`run-agent.sh`, `run-all.sh`) and
interactive tmux (`itrun.sh`) harnesses with stream-json parsers
(`parse-run.mjs`, `parse-session.mjs`) that report tool calls, token
usage, and a VERDICT line summarising codegraph_explore vs. Read/Grep counts.
- `run-interactive-test.md` — documents the two harnesses, idle-detection
approach, and what "good" agent behavior looks like after explore-first
guidance.
Colby McHenry