Live runner executed for real: OpenAI gpt-4o-mini under test, Gemini
2.5-flash as the non-same-family judge, hidden-test ground truth, seeded
X/Y counterbalance, pre-registered RUBRIC-v1 hash embedded per row.
The real run surfaced and fixed two harness bugs:
1. Ground-truth fix-rate was coupled to judge-row validity — a fixed bug
that zeroed a real B-arm fix when its judge pass returned no winner.
Ground truth is now tracked per (scenario, arm), independent of the
judge; numbers are internally consistent.
2. Judge row model: emit ONE comparison row per (scenario, pass) — the
unit the sign test clusters — instead of duplicated per-arm rows.
Plus judge-verdict parse hardened (last-verdict wins, broader regex,
more tokens): unparseable rate 50% -> 25%, the rest excluded in code.
Result at n=2: ground truth A=0.50 B=0.00, judge clustered 1-1 p=1.0 —
no arm signal, honestly underpowered. Corpus scaling is the path to a
claim (stated in results README + ADR).
Verified: 30 deterministic tests (stats + safety) green; real run $0.003.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome
(hidden acceptance test), style-neutral verbatim judge instruction (no
treatment-revealing framing in either direction), two-pass cross-family
judging with INFERRED-labeled fallback, clustered-stats reporting rule.
Provably rubric-before-data: this commit precedes any results-* artifact.
2 scenarios with seeded defects + personas + hidden tests (off-by-one
pagination; swallowed async rejection). Hidden tests verified both
directions: FAIL on seeded bug with precise reasons, PASS on the fix.
lib/stats.ts encodes the three Pattern Space harness fixes as code:
isGoodRow enforced inside aggregate() (no path to a number that skips
it), seeded per-item counterbalance (their null cell was 72% position-
skewed), exact scenario-clustered sign test (validated against the
known 27/11 → p≈0.0136 re-analysis).
Verified: 14/14 stats tests; hidden-test ground truth both directions;
tsc clean. Remaining: live conversation driver + judge integration
(API-gated, same decision as ADR-106 live mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dragan Spiridonov