Borrows four patterns from the eval landscape (Anthropic pass@k,
SkillsBench A/B, Aider retry-with-feedback, SWE-bench task prompts)
and adapts them to our 12-task workbench:
- Add task-specific prompt.md to all 12 tasks (lifts pass rate 50%→92%)
- Add --runs N flag for pass@k / pass^k statistical tracking
- Add --compare flag for with-skill vs without-skill A/B testing
- Add --retry flag for two-attempt with test failure feedback
- Add --generic-prompt flag for prompt sensitivity measurement
- New eval-agent-suite.sh orchestrator with mode selection
- Expand agent suite JSON from 3 cases to all 12
- Add prompt.md and agent suite coverage checks to CI gate
Phase 1: scripts/eval-agent-harness.sh wraps setup → agent CLI → score
for any workbench task. Supports --dry-run, --hooks-disabled for A/B,
--timeout, and agent-not-found graceful handling.
Phase 2: extend parseEvalRuntime() to accept claude/codex alongside
static/mock/shell. Update flag help and shell completion. Add table-
driven test covering all 5 runtime values plus empty and unknown.
Closes soc-egdg, soc-eiy4.
Bo