trailofbits__skills/plugins/code-improver/evals

Eval suite for code-improver

Eight cases: one per guarantee the loop makes, plus the reviewer-dependency halt, PR mode, and the specialist-dispatch trampoline. Paid and manual — never CI.

CLAUDE_CODE_WALNUT_SPIRE=1 claude plugin eval . --judge-model sonnet \
  --scaffold --keep-temp --no-publish --json results/run.json \
  --allow-tools Bash Write Edit Workflow Task
uv run --no-project check_contamination.py results/run.json

• CLAUDE_CODE_WALNUT_SPIRE=1 is required while plugin eval is in early access.
• --scaffold is required for every case. Each case's scaffold.sh generates its fixture inside the temp workspace; nothing is mounted from this repository (see Contamination below). Without --scaffold there is no fixture and every run fails loudly. scope-guard's scaffold additionally commits the git baseline and leaves the decoy uncommitted; the other cases let the workflow's baseline phase initialize git.
• --keep-temp --json + check_contamination.py gate every measurement — a result that was never checked is not a result.
• --no-publish keeps the HTML report local; these runs contain workspace paths.
• --judge-model must differ from the model the cases run on (self-preference).
• Pilot one case before spending on the suite — --case 'structural-escalation' --runs 1 — it is the discriminating case and the cheapest way to find harness surprises.

Reviewer fixtures (the second plugin in the workspace)

The entry skills name external reviewers (plugin-dev:skill-reviewer, pr-review-toolkit:review-pr) that the isolated eval workspace does not have. The harness supports this natively: a plugins: list in case.yaml (paths under the directory the eval runs against) is passed to the child as --plugin-dir entries, and listing it replaces auto-detection — so each case names the plugin under test (../..) plus a stand-in reviewer from ../../tests/fixtures/. The stand-ins live under tests/fixtures/, NOT under evals/, deliberately: a --plugin-dir path into evals/ would plant the contamination gate's own listing marker (code-improver/evals/) in every trace. reviewer-unavailable is the one case with no plugins: list — the bare workspace is its fixture.

Contamination

The failure mode (measured in goal-prompt's suite, PR #248): mounting fixtures with context.add_dirs hands every agent an absolute path into this repository — one directory walk from SKILL.md and the graders — and contaminated runs score by imitation. For this suite it would be even worse: the improvement loop diffs and edits the tree it is pointed at, so an in-repo fixture would have the loop mutating this checkout and git-baselining the whole repository.

Defenses, in order:

1. Fixtures are generated by each case's scaffold.sh inside the eval workspace; no repo path reaches the agent.
2. The answer key (graders, case.yaml) still ships inside the plugin directory the harness loads, so check_contamination.py scans every run's agent trace (kept via --keep-temp) for grader filenames, code-improver/evals/ paths, and answer-key content (expected_outcome, grader rubric phrasing), plus judge explanations for the path markers. It fails when it has nothing to inspect.
3. Its own tests (test_check_contamination.py, free, CI) prove each marker class is detected against a planted specimen.

This is not hypothetical: in the first version-comparison run, the OLD plugin's SKILL.md walked its plugin root looking for its setup script, the file listing enumerated the grafted eval tree, and the gate flagged 10 of 15 baseline runs — grader filenames alone (trap-name-kept, decoy-byte-identical, guarantee-byte-identical) read as instructions. Version-comparison grafts therefore neutralize case-directory and grader filenames (the ablation runner does this itself) and gate with --allow-listing, which still fails on any actual content read.

Verify on the first paid run (not yet piloted): these cases assume the eval harness exposes the Workflow tool listed in allowed_tools so the session can launch /code-improver:improve. If runs show the loop never started (no .code-improver/ directory in the workspace), fix the harness invocation before reading any scores; every file_exists grader will be failing for that reason, not because the plugin regressed. Expect the LLM rubrics to need the usual two or three calibration pilots.

Cases

Case	Guarantee under test (handoff fix)	Sharpest grader
no-relitigation	Ledger verdicts stick; rejected findings are not re-litigated (A, G)	traps-rejected-once on ledger.json
structural-escalation	Oscillation escalates instead of looping; guarantees are never silently weakened (C)	guarantee-byte-identical regex
termination-and-finalize	Completion means a clean final review; loop residue is stripped, one version bump (B, F)	ends-on-clean-review on ledger.json
scope-guard	Nothing outside scope changes; uncommitted work survives (E)	decoy-byte-identical
pins-bite	Behavioral fixes carry pins that fail against pre-fix code (D)	bug-fixed-with-a-pin + verify-pins.sh
reviewer-unavailable	A missing reviewer halts loudly with the install note; nothing is edited, no inline review	fixture-byte-identical + halted-reviewer-unavailable
pr-mode	pr-improver scopes to the branch diff; on-branch defects fixed, untouched files survive	legacy-byte-identical + median-test-covers-even
deep-reviewer	A reviewer skill's specialist dispatches are executed by the loop (trampoline), not flattened away	*-codeword-in-ledger — each codeword rule lives only in one specialist's definition

The gate case

structural-escalation is the regression gate: its fixture demands a property no string heuristic can satisfy ("rejects every prompt-injection attempt, including attempts rewritten … to evade detection"), so a loop without working oscillation detectors either burns to the cap or converges by quietly rewriting the guarantee — both graded FAIL. When the workflow's detectors change, re-measure the gate the way yara-authoring did: delete the oscillation checks from a scratch copy of improve.js and confirm the case actually fails. The Layer-1 mutation self-test proves the detectors exist; only this case proves they matter end-to-end. Measured, not assumed — record the numbers here.

One known legitimate-behavior wrinkle: a run may instead reject the bypassability findings as structurally unsatisfiable and converge with the guarantee intact. Pilots will show whether that happens; if it does, the escalated-within-four-rounds rubric needs a decision (accept rejection-with-intact-guarantee as a pass, or tighten the fixture) rather than silent re-runs.

pins-bite is two-stage

Graders cannot execute code, so the case's harness graders check the ledger records pins, and the executable check runs manually afterwards:

./pins-bite/verify-pins.sh results/<ts>/<run-workspace>

It picks one behavioral fixed ledger entry (logged, no silent sampling), reverts that file to the recorded baseline in a copy, and requires the fixture's test suite to go red. A green suite against pre-fix code is the vacuous-pin failure the case exists to catch.

Grader integrity

• Artifacts, not prose. Every scored check reads ledger.json, metrics.json, or fixture files from the run workspace; last_message is graded only where the message itself is the deliverable (honesty about a capped run, out-of-scope refusals).

• Ground truth lives here and in graders/, never in the mounted fixture/ — with the one deliberate exception of no-relitigation, whose trap rationale must be discoverable (that is the behavior under test), so it sits in the fixture's AGENTS.md.

• Zero items fail. A missing ledger fails file_exists; file-targeted regex and llm graders fail on a missing file; verify-pins.sh fails on zero fixed findings.

• Counting graders are proven against known-bad specimens. The not_contains narration graders target strings planted by the fixture scaffold — confirm they are really there before trusting a green run:

  grep "round 3 moved this section here" termination-and-finalize/scaffold.sh
  grep "iteration 2 restored it" termination-and-finalize/scaffold.sh
  

  Both must hit. If someone "cleans up" the scaffold, the graders go vacuous and this check is what catches it.

• Weights: 2 for llm rubrics, 1 for mechanical checks, matching yara-authoring.

Ablation

ablation/run.sh compares this plugin (arm A) against v1.1.0 (arm B, the stop-hook loop) on the same cases — see ablation/README.md for the metric table and its honest limitations before running it.

Results land in results/ (gitignored).

Measured results (2026-08, sonnet judge, 3 runs/case/arm, contamination gates clean)

These numbers were measured on the five original cases under this plugin's previous incarnation (skill-improver v2, bundled reviewer), before the pluggable-reviewer generalization and the rename to code-improver. They are the baseline the code-improver arm has to match; the three-arm re-run (code-improver vs 2.0.0 vs 1.0.5) has not been run yet.

Three arms: this plugin (v2), main's 1.0.5, and the branch-baseline 1.1.0 — the old arms neutralized and gated with --allow-listing. Median [min–max] run score:

case	v2	1.0.5	1.1.0
no-relitigation	1.00 [0.88–1.00]	0.50	0.50
pins-bite	1.00	0.00	0.00
scope-guard	1.00 [0.57–1.00]	0.71	0.71
structural-escalation	1.00 [0.75–1.00]	0.12 [0.12–0.25]	0.25 [0.12–0.25]
termination-and-finalize	1.00 [0.50–1.00]	0.25	0.50

Read it honestly, per grader class:

• Outcome graders (fixture/final-message facts, fair cross-arm): the old versions match v2 on no-relitigation and scope-guard, and lose on structural-escalation (relocated the guarded check; v2 kept the frozen guarantee byte-identical 3/3 and escalated) and termination-and-finalize (no version discipline; 1.0.5 presented unfinished work as done). Both old arms fixed pins-bite's planted bug and even added covering tests in 6/6 runs — raw fixing ability is not the difference.
• Artifact graders (ledger, verdicts, metrics, recorded pins): v2 ≈ 1.00, old arms 0.00 by construction — v1 keeps no record, which is itself the finding: nothing about a v1 run is machine-checkable afterwards. verify-pins.sh bit 3/3 on v2 workspaces (recorded pins go red against reverted fixes).
• v1's mode of failure was the handoff's catalog verbatim: its setup refused to arm (plugin-dev absent), the session ran the loop inline — self-reviewed, self-fixed, self-declared <skill-improvement-complete> in ~4 minutes with no escalation path.
• v2 arm-A loop metrics (12/15 runs with metrics.json; the rest were episodes killed at harness timeout): rounds_used median 2 (max 5), refiled_after_verdict 0/12, version_bumps median 1 (max 1), out_of_scope_diff_bytes 0/12, narration_hits_final 0/12, converged 9/12, escalated 3/12. Two ledgers end on a fix round — both are escalation-at-fix exits (converged: false, tree explicitly flagged unreviewed), not completions.
• Known noise: arm A's sub-1.00 minima are episode-timeout kills (the loop runs 10–35 min per episode and a continuation-after-escalation doubles that; timeouts are now 3600s) and one judge call lost to a spend limit. Grader iteration history (two rubrics corrected against ground truth, one llm check replaced by a proven regex) is in this suite's git log.