mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
be28f15f9c
Live runner executed for real: OpenAI gpt-4o-mini under test, Gemini 2.5-flash as the non-same-family judge, hidden-test ground truth, seeded X/Y counterbalance, pre-registered RUBRIC-v1 hash embedded per row. The real run surfaced and fixed two harness bugs: 1. Ground-truth fix-rate was coupled to judge-row validity — a fixed bug that zeroed a real B-arm fix when its judge pass returned no winner. Ground truth is now tracked per (scenario, arm), independent of the judge; numbers are internally consistent. 2. Judge row model: emit ONE comparison row per (scenario, pass) — the unit the sign test clusters — instead of duplicated per-arm rows. Plus judge-verdict parse hardened (last-verdict wins, broader regex, more tokens): unparseable rate 50% -> 25%, the rest excluded in code. Result at n=2: ground truth A=0.50 B=0.00, judge clustered 1-1 p=1.0 — no arm signal, honestly underpowered. Corpus scaling is the path to a claim (stated in results README + ADR). Verified: 30 deterministic tests (stats + safety) green; real run $0.003. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ADR-109 interaction benchmark — run records
results-<stamp>.jsonl — one comparison row per (scenario, pass): the judge's
winner verdict, the X/Y transcripts, rubric hash. summary-<stamp>.json — the
scenario-clustered sign test (judge) PLUS the independent ground-truth fix-rate
(did the developer's post-conversation fix pass the hidden test — computed
separately from any judge verdict).
Latest run (2026-06-11) — cross-family, real
- agent-under-test: OpenAI gpt-4o-mini · judge: Gemini 2.5-flash (non-same-family)
- 2 scenarios × 2 arms (bare vs QE-guided) × 3 turns × 2 judge passes
- Ground truth: A(bare)=0.50, B(QE)=0.00 · Judge clustered: A=1 B=1 tie=0, p=1.0
- No arm signal at n=2 — honestly underpowered, not a result. Scaling the scenario corpus (n→20+) is the path to a statistical claim, per the ADR.
- Judge verdict parsed on 3/4 passes; the unparseable one is excluded by isGoodRow in code (never silently counted).