Colby McHenry 7ddd3fa7eb test(agent-eval): persist offload accuracy/adoption eval harness + front-load hook ...

Reproducible suite measuring the managed CodeGraph AI offload and the front-load
UserPromptSubmit hook (approach 1) vs raw codegraph and no-codegraph, across repo
sizes, on time / main-session tokens+cost / CodeGraph-AI tokens+cost / accuracy.
All agent arms run claude -p sonnet --effort high; eval-only, nothing shipped.

- offload-eval-setup.sh: clone + index 4 memory-probe-verified "not-trained-on" repos
  (mtkruto/postybirb/shapeshift/trezor — small→large) so the no-codegraph baseline is honest.
- offload-eval-3arm.sh / -frontload.sh: one repo, the arms (offload/raw/nocg, frontload).
- offload-eval-matrix.sh / -frontload-matrix.sh: drive all 4 tiers.
- offload-eval-hook.mjs: the front-load hook (self-locates its engine; CG_FRONTLOAD_DEBUG to log).
- offload-eval-metrics.mjs / -judge.mjs (Sonnet) / -summarize.mjs: extract, score, aggregate.
- offload-eval-ground-truth.json: source-verified canonical flows (the judge's reference).
- offload-eval.md: usage + the 2026-06 findings (raw = the win; offload least-accurate;
  front-load solves adoption but exposes explore's dynamic-dispatch gaps).

Scripts are path-portable (self-locating $HERE/$ENGINE; AGENT_EVAL_OUT scratch dir).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-18 22:30:30 -05:00

1.1 KiB

Executable File

Raw Permalink Blame History

View Raw