zernie
ea9a0a693d
* Retire buggy legacy evals; document the faithful reproduce command The bench/evals caveman-claim + ecosystem-pilot evals used the old bare-SKILL.md delivery (dropped in the run cwd, which Claude Code never registers), so the skill never loaded — and their docstrings still broadcast the resulting discredited numbers (output up, costs more, ~1% bill) that contradict the corrected canonical run in bench/ecosystem/FINDINGS.md. - Delete bench/evals/caveman-claim.eval.mjs and ecosystem-pilot.eval.mjs. They are fully superseded by bench/ecosystem/benchmark.mjs, which installs skills the faithful way (--plugin-dir + activation hook) over the same corpus. - Repoint every reference (corpus + CLAUDE.md.spec.ts + benchmark header + headroom-pilot header + research/roadmap + research/benchmark-methodology) to the faithful benchmark / FINDINGS.md, and refresh roadmap's stale numbers to the corrected finding (~6% output cut, pooled bill flat, output ~20% of the dollar bill). - Regenerate CLAUDE.md from the spec (fresh integrity hash). - Add bench/ecosystem/README.md documenting the reproduce commands (cheap pilot + the published 7x5 Sonnet run), named explicitly so there is no accidental full-manifest fan-out. Verified: npm run build, vigiles self-lint (integrity + refs, exit 0), prettier --check. * Remove remaining buggy-era pilots: headroom-pilot eval + launch-article draft - Delete bench/evals/headroom-pilot.eval.mjs — same June-2026 bare-SKILL.md delivery bug as the retired caveman/ecosystem pilots (skill never registers). Not superseded content-wise (it measured correctness-lift, a distinct claim), but the delivery is broken; rebuild faithfully if headroom becomes a priority. - Delete research/launch-article-draft.md — a publicly-visible early draft of the now-published, corrected article, still carrying the discredited pre-correction numbers. Remove its index entry (research/README.md) and Key-File entry (research/CLAUDE.md.spec.ts) + recompile research/CLAUDE.md. Verified: vigiles self-lint (integrity + refs) exit 0, prettier --check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zGXEtUA57J6FwW9VzMWyz --------- Co-authored-by: Claude <noreply@anthropic.com>
The ecosystem benchmark
A/B the most-hyped Claude Code skills and plugins over a neutral real-task corpus,
and report the metric triple — bill (costUsd), target (whatever the skill
claims to move), and blast radius (correctness) — leading with the debunks
(claimed ≫ measured).
Skills install the faithful way (--plugin-dir + their real activation hook),
so the treatment actually loads into the session — not a bare SKILL.md dropped in
the working directory, which Claude Code never registers. See
CAVEMAN-INVESTIGATION.md for how we caught and fixed
that delivery bug, and FINDINGS.md for the corrected canonical run.
Reproduce
Runs use your own claude CLI on your Pro/Max subscription (apiKeySource: "none")
— $0 metered; the $ figures are API-equivalent yardsticks. Name the file
explicitly so there's no accidental fan-out across the whole manifest.
# Cheap sanity pilot (seconds, pennies) — 2 tasks × 2 trials
VIGILES_SKILLS=caveman VIGILES_TASKS=2 VIGILES_TRIALS=2 \
node bench/ecosystem/benchmark.mjs
# The published caveman result: 7 tasks × 5 trials on Sonnet (~140 runs, ~$10
# API-equivalent, $0 on a subscription) — matches FINDINGS.md's canonical run
VIGILES_SKILLS=caveman VIGILES_MODEL=sonnet VIGILES_TASKS=7 VIGILES_TRIALS=5 \
node bench/ecosystem/benchmark.mjs
Point it at any skill in the manifest (skills.mjs) via
VIGILES_SKILLS=<name>. Output is an incremental per-skill table plus a JSON dump
under results/. Method: ../../research/benchmark-methodology.md;
the general "measure your own skill" guide: ../../docs/measuring-skills.md.