Plan (not yet implemented) to upgrade cs-feat/cs-issue/cs-epic evaluation
from routing-level proxy metrics to outcome-level: seed repos + hidden
acceptance tests + real agent harness, mechanical pass-rate scoring, with
a "skill vs bare agent" control layer. Carries forward the routing
campaign's validity rules (hidden tests, realistic onboarded repos,
solvability pre-check, must-refuse scenarios, triage discipline).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Now that build-cs-skill owns skill authoring, this skill is purely the measured
eval/optimize loop, so eval-cs-skill is the clearer name. Physical rename of
.claude/skills/cs-skill-lab -> eval-cs-skill and experiments/cs-skill-lab-001 ->
eval-cs-skill-001, with all self-references, tests, ADR-003, and results updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The campaign found that every apparent model/skill "gap" was an eval-fixture or
harness deficiency, not a skill defect. Codify the three validity rules so future
fixtures are fair:
- reproduce the skill's designed runtime (inject_context: onboard context)
- score prose/symbol answers with a semantic oracle (recall_judge), not just
token overlap
- embed the subject matter in the fixture (review needs the diff, docs need the
code/config/API; design/plan can proceed from a requirement)
plus the inspection discipline: read per-model, read raw outputs, k=1 has variance.
Recorded in references/eval/protocol.md (三铁律) and ADR-003 consequences.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dafang