- neutral prompt title (issue→user request); context block adds goal-driver
waiver (headless has no visible Task agent — agents were rule-abidingly
handing off and exiting with zero implementation) and an explicit "process
artifacts still required" clause (deconfound: sonnet artifact 0→0.22)
- scenario.artifact_glob: fixtures carry their own artifact contract
(cs-feat features/**/*design*), fix-note default kept for compat
- bug_id now optional (feature scenarios inject nothing); .codestable tree
snapshot in scorer evidence
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
P0 L2 finding: the shared output-requirements block told BOTH variants to
write fix-note and run pytest, handing the skill's process contract to the
bare control group and flattening the artifact metric. The shared template
now stays neutral (fix on disk + summarize); process requirements come from
the skill text itself. Tests assert the inverse (fix-note/pytest must NOT
appear in the shared template) so the leak cannot silently return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New answerType e2e-outcome: agent runs a full skill workflow inside a real
seed repo workdir (prepare_e2e_workdir: build-seed + bug inject), scored
mechanically by hidden acceptance tests + regression suite + artifact
contract (all [measured]). Scoring moved inside the workdir lifetime;
HarnessResult.workdir carries the repo path to scorers. 10 offline tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Now that build-cs-skill owns skill authoring, this skill is purely the measured
eval/optimize loop, so eval-cs-skill is the clearer name. Physical rename of
.claude/skills/cs-skill-lab -> eval-cs-skill and experiments/cs-skill-lab-001 ->
eval-cs-skill-001, with all self-references, tests, ADR-003, and results updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Manual inspection during the real-model campaign caught a second measurement
bug: planted_defect's token-overlap oracle gives false negatives on prose /
symbol answers (e.g. ">= changed to >", "removed early-return guard"). On
cs-refactor both sonnet and haiku actually caught every behavior-equivalence
violation, but the token matcher scored them 0.62/0.75. A judge-based semantic
recall oracle scores both at 1.00.
- add scorers/recall_judge.py: an independent judge decides, per defect,
whether the output identifies it (wording-agnostic); tagged [soft]; falls
back to token overlap offline / when no judge_model is set
- committed eval suite uses planted_defect + recall_judge
- test for the offline token-fallback path
Corrected fair review-skill baseline (onboard context + recall_judge, sonnet
+ haiku): cs-issue/cs-audit/cs-refactor all ~1.00 across both models. The
earlier "gaps" were eval artifacts (missing context, brittle scorer, k=1
variance), not skill/model defects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dafang