mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
c02e60a8f3
RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome (hidden acceptance test), style-neutral verbatim judge instruction (no treatment-revealing framing in either direction), two-pass cross-family judging with INFERRED-labeled fallback, clustered-stats reporting rule. Provably rubric-before-data: this commit precedes any results-* artifact. 2 scenarios with seeded defects + personas + hidden tests (off-by-one pagination; swallowed async rejection). Hidden tests verified both directions: FAIL on seeded bug with precise reasons, PASS on the fix. lib/stats.ts encodes the three Pattern Space harness fixes as code: isGoodRow enforced inside aggregate() (no path to a number that skips it), seeded per-item counterbalance (their null cell was 72% position- skewed), exact scenario-clustered sign test (validated against the known 27/11 → p≈0.0136 re-analysis). Verified: 14/14 stats tests; hidden-test ground truth both directions; tsc clean. Remaining: live conversation driver + judge integration (API-gated, same decision as ADR-106 live mode). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2.8 KiB
2.8 KiB
Benchmark Lineage Registry (ADR-108)
Every benchmark or eval whose numbers are quoted anywhere (README, release notes, issues, marketing) gets a row here. Deprecated results are archived, never deleted. A number that gets quoted must have committed data behind it.
Rules (full rationale in ADR-108):
- Rubric before data — any judged/rubric-scored benchmark commits
RUBRIC-v<N>.md(rubric, judge instructions, weights) before result generation. Result rows embed the rubric file's git hash; post-hoc edits are detectable by construction. - Rubric changelog — a rubric revision's LINEAGE entry enumerates ALL changes from the prior version, verifiable against the file diff. Never describe a rebuild as a one-variable change.
- Abandoned runs keep their data — partial results move to
archive/<run>/with a README stating the abandonment reason. - Clustered statistics — report results clustered by experimental unit (scenario-level sign test / mixed model, with CIs). Raw pooled counts may appear only alongside the clustered result.
- Style-neutral judges — judge instructions must not reveal treatment identity or pre-frame any output style as a virtue or fault.
Registry
| Benchmark | Version | Measures | Method | Known limitations / confounds | Status | Cite |
|---|---|---|---|---|---|---|
| qe-arena tournaments | Phase 1 (ADR-104) | Test-suite selection fitness: 0.6·killRate + 0.3·coverage − 0.1·suiteCostRatio, real mutant executions via node --test |
Objective (no judge, no rubric); byte-reproducible under --seed |
Fixture-scoped (suite selection, not generation); runtime term is a suite-size proxy, not measured ms — the measured-duration variant was retracted for jitter-driven rank instability (see ADR-104 "Reproducibility note", the model entry for this registry) | Current | ADR-104 + aqe arena run --seed <s> output |
| Interaction benchmark for qualitative agents | v1 rubric registered (ADR-109) | Developer movement: ground-truth (hidden acceptance test passes) primary; pre-registered rubric dimensions secondary | Reactive interactor; non-same-family two-pass judging; scenario-clustered sign test | Simulated developer ≠ real developer (stated per report); per-conversation cost → release-cadence only | Rubric pre-registered (benchmarks/interaction/RUBRIC-v1.md); 2 scenarios with verified hidden tests; aggregation/stats lib tested; NO DATA YET — live driver pending |
— (no citable results) |
Backfill
Historical performance/benchmark numbers quoted in older docs predate this registry. They are uncited-by-default: before quoting any pre-2026-06 benchmark figure, add its row here first (measures, method, limitations) or treat it as deprecated.