Claude 671831dacd feat: add A/B benchmark harness with 20 tasks, gate simulation, and composite scoring ...

Implements the Measurement Plan: Config A (no control plane) vs Config B
(Phase 1 guidance) across 20 representative tasks from Claude Flow repo
history spanning 7 task classes (bug-fix, feature, refactor, security,
deployment, test, performance).

KPIs: success rate, tool calls, token spend, violations, human
interventions. Composite score formula:
score = success_rate - 0.1*cost - 0.2*violations - 0.1*interventions

Category shift detection: B beats A by ≥0.2 on composite across ≥3 task
classes. Gate simulation detects destructive commands, hardcoded secrets,
force push, unsafe types, skipped hooks, missing tests, policy violations.

1,328 tests passing across 26 files (38 new AB benchmark tests).

https://claude.ai/code/session_01HLi9WmfvFgkDLJxLKGAUTf

2026-02-01 22:22:17 +00:00

..

api-quick-reference.md

feat: add A/B benchmark harness with 20 tasks, gate simulation, and composite scoring

2026-02-01 22:22:17 +00:00