mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
c02e60a8f3
RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome (hidden acceptance test), style-neutral verbatim judge instruction (no treatment-revealing framing in either direction), two-pass cross-family judging with INFERRED-labeled fallback, clustered-stats reporting rule. Provably rubric-before-data: this commit precedes any results-* artifact. 2 scenarios with seeded defects + personas + hidden tests (off-by-one pagination; swallowed async rejection). Hidden tests verified both directions: FAIL on seeded bug with precise reasons, PASS on the fix. lib/stats.ts encodes the three Pattern Space harness fixes as code: isGoodRow enforced inside aggregate() (no path to a number that skips it), seeded per-item counterbalance (their null cell was 72% position- skewed), exact scenario-clustered sign test (validated against the known 27/11 → p≈0.0136 re-analysis). Verified: 14/14 stats tests; hidden-test ground truth both directions; tsc clean. Remaining: live conversation driver + judge integration (API-gated, same decision as ADR-106 live mode). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
23 lines
2.8 KiB
Markdown
23 lines
2.8 KiB
Markdown
# Benchmark Lineage Registry (ADR-108)
|
||
|
||
Every benchmark or eval whose numbers are quoted anywhere (README, release notes, issues, marketing) gets a row here. Deprecated results are archived, never deleted. A number that gets quoted must have committed data behind it.
|
||
|
||
**Rules** (full rationale in [ADR-108](../implementation/adrs/ADR-108-benchmark-lineage-preregistered-rubrics.md)):
|
||
|
||
1. **Rubric before data** — any judged/rubric-scored benchmark commits `RUBRIC-v<N>.md` (rubric, judge instructions, weights) *before* result generation. Result rows embed the rubric file's git hash; post-hoc edits are detectable by construction.
|
||
2. **Rubric changelog** — a rubric revision's LINEAGE entry enumerates ALL changes from the prior version, verifiable against the file diff. Never describe a rebuild as a one-variable change.
|
||
3. **Abandoned runs keep their data** — partial results move to `archive/<run>/` with a README stating the abandonment reason.
|
||
4. **Clustered statistics** — report results clustered by experimental unit (scenario-level sign test / mixed model, with CIs). Raw pooled counts may appear only alongside the clustered result.
|
||
5. **Style-neutral judges** — judge instructions must not reveal treatment identity or pre-frame any output style as a virtue or fault.
|
||
|
||
## Registry
|
||
|
||
| Benchmark | Version | Measures | Method | Known limitations / confounds | Status | Cite |
|
||
|---|---|---|---|---|---|---|
|
||
| qe-arena tournaments | Phase 1 (ADR-104) | Test-suite selection fitness: 0.6·killRate + 0.3·coverage − 0.1·suiteCostRatio, real mutant executions via `node --test` | Objective (no judge, no rubric); byte-reproducible under `--seed` | Fixture-scoped (suite selection, not generation); runtime term is a suite-size proxy, not measured ms — the measured-duration variant was **retracted** for jitter-driven rank instability (see ADR-104 "Reproducibility note", the model entry for this registry) | **Current** | ADR-104 + `aqe arena run --seed <s>` output |
|
||
| Interaction benchmark for qualitative agents | v1 rubric registered (ADR-109) | Developer movement: ground-truth (hidden acceptance test passes) primary; pre-registered rubric dimensions secondary | Reactive interactor; non-same-family two-pass judging; scenario-clustered sign test | Simulated developer ≠ real developer (stated per report); per-conversation cost → release-cadence only | **Rubric pre-registered** (`benchmarks/interaction/RUBRIC-v1.md`); 2 scenarios with verified hidden tests; aggregation/stats lib tested; NO DATA YET — live driver pending | — (no citable results) |
|
||
|
||
## Backfill
|
||
|
||
Historical performance/benchmark numbers quoted in older docs predate this registry. They are **uncited-by-default**: before quoting any pre-2026-06 benchmark figure, add its row here first (measures, method, limitations) or treat it as deprecated.
|