mirror of
https://github.com/EveryInc/compound-engineering-plugin.git
synced 2026-09-19 01:09:55 +08:00
Kieran Klaassen
6b7a0e15fd
A model upgrade can make a working skill corpus worse: workflows stall, halt partway through, or burn several times the tokens for the same task. The instinct is to read the prose and rewrite what looks wrong. That produces a plausible fix list and no way to tell whether any item mattered, because the failure is stochastic and a corpus large enough to need retuning is one whose prose nobody can reason about reliably. This skill runs the measurement path instead: mine the existing run archive for a baseline at zero model cost, establish a noise floor on two identical builds before crediting anything, audit the corpus with a second agent whose job is to defend the existing prose from the project's own learnings and tests, then cut in surgical passes until a bar registered in advance clears. Distilled from a real engagement rather than composed from principle. The methodology already existed as two solutions docs; what was missing was a skill that executes it. Design decisions worth reviewing: Broken runs are a first-class outcome. Empty transcripts and error exits score as model failures and silently inflate every effect; one archive was 20% of them, and excluding them falsified the first headline finding. "Followed the process" and "did the job" stay separate metrics. A run can complete a task while skipping the workflow entirely, which reads as success when the two are collapsed. Kept apart it reads as its own defect, which is how the source engagement caught a regression its own cutting had introduced. The audit is adversarial by construction. "A weaker model might need it" is not grounds for keeping a line; only citable provenance is. The synthesis is instructed to report what contradicts the starting premise, because confirmation of a thesis you already hold teaches nothing. It refuses without a harness. No archive, no build selector, no repeatable task: the skill stops and names what to build rather than degrading into a static audit presented as retuning. An audit-only pass is a legitimate request and a different one. User-invoked only. It spends many paid runs and refuses without a harness, so model-routing it would let a cheap request escalate into an expensive measurement program. Co-Authored-By: Claude <noreply@anthropic.com>