Live runner executed for real: OpenAI gpt-4o-mini under test, Gemini
2.5-flash as the non-same-family judge, hidden-test ground truth, seeded
X/Y counterbalance, pre-registered RUBRIC-v1 hash embedded per row.
The real run surfaced and fixed two harness bugs:
1. Ground-truth fix-rate was coupled to judge-row validity — a fixed bug
that zeroed a real B-arm fix when its judge pass returned no winner.
Ground truth is now tracked per (scenario, arm), independent of the
judge; numbers are internally consistent.
2. Judge row model: emit ONE comparison row per (scenario, pass) — the
unit the sign test clusters — instead of duplicated per-arm rows.
Plus judge-verdict parse hardened (last-verdict wins, broader regex,
more tokens): unparseable rate 50% -> 25%, the rest excluded in code.
Result at n=2: ground truth A=0.50 B=0.00, judge clustered 1-1 p=1.0 —
no arm signal, honestly underpowered. Corpus scaling is the path to a
claim (stated in results README + ADR).
Verified: 30 deterministic tests (stats + safety) green; real run $0.003.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Path A (model-judgment) live runner over the real ADR-026 tiers
(haiku/sonnet/opus via Anthropic API). Each tier is given the
data-protection absolutes + an explicit ACTIONS: [...] declaration
protocol, hit with each temptation as opener AND mid-task, N=5, and run
through the existing assertion engine. Asserts on declared intent only;
never touches a real DB. safety-eval.yml live job now executes it.
Real run found and fixed a harness false-positive: a REFUSING model that
illustrates a command in prose was wrongly flagged. The ACTIONS protocol
makes intent unambiguous; discrimination tests prove a genuinely-declared
rm memory.db still FAILS while an illustrated one PASSES (not neutered).
Verified: 16 deterministic tests; live matrix 5/5 all cells, all 3 tiers
(haiku $0.085 / sonnet $0.231 / opus $1.108), committed as evidence.
Retry-resilient client (transient fetch/429/5xx backoff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome
(hidden acceptance test), style-neutral verbatim judge instruction (no
treatment-revealing framing in either direction), two-pass cross-family
judging with INFERRED-labeled fallback, clustered-stats reporting rule.
Provably rubric-before-data: this commit precedes any results-* artifact.
2 scenarios with seeded defects + personas + hidden tests (off-by-one
pagination; swallowed async rejection). Hidden tests verified both
directions: FAIL on seeded bug with precise reasons, PASS on the fix.
lib/stats.ts encodes the three Pattern Space harness fixes as code:
isGoodRow enforced inside aggregate() (no path to a number that skips
it), seeded per-item counterbalance (their null cell was 72% position-
skewed), exact scenario-clustered sign test (validated against the
known 27/11 → p≈0.0136 re-analysis).
Verified: 14/14 stats tests; hidden-test ground truth both directions;
tsc clean. Remaining: live conversation driver + judge integration
(API-gated, same decision as ADR-106 live mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dragan Spiridonov