RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome
(hidden acceptance test), style-neutral verbatim judge instruction (no
treatment-revealing framing in either direction), two-pass cross-family
judging with INFERRED-labeled fallback, clustered-stats reporting rule.
Provably rubric-before-data: this commit precedes any results-* artifact.
2 scenarios with seeded defects + personas + hidden tests (off-by-one
pagination; swallowed async rejection). Hidden tests verified both
directions: FAIL on seeded bug with precise reasons, PASS on the fix.
lib/stats.ts encodes the three Pattern Space harness fixes as code:
isGoodRow enforced inside aggregate() (no path to a number that skips
it), seeded per-item counterbalance (their null cell was 72% position-
skewed), exact scenario-clustered sign test (validated against the
known 27/11 → p≈0.0136 re-analysis).
Verified: 14/14 stats tests; hidden-test ground truth both directions;
tsc clean. Remaining: live conversation driver + judge integration
(API-gated, same decision as ADR-106 live mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lineage registry with rubric-before-data, changelog, kept-data,
clustered-statistics, and style-neutral-judge rules. Seeded with
arena Phase 1 (incl. its retracted measured-duration variant as the
model entry) and the planned ADR-109 row. Pre-registry numbers are
uncited-by-default pending backfill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports ruflo's perf-M4 (sign-random-projection + Hamming popcount). Adds
src/shared/utils/rabitq.ts: signSignature (384-bit packed into 12 Uint32
words, 48 bytes), hammingDistance (SWAR popcount), hammingSimilarity, and
topKBySignature (Hamming prefilter gated at N>=100 + exact-cosine rerank of a
recall-aware pool, so output equals exact topK when the true top-k fall in the
pool). 16 tests + scripts/benchmark-rabitq.ts (npm run benchmark:rabitq).
HONEST FINDING (verified, does NOT match ruflo's optimistic 2.7x/10.9x):
N=1000: speedup 1.31x, recall@10 100%, index 1.46MB -> 47KB (32x smaller)
N=5000: speedup 1.18x, recall@10 99.9%, 7.32MB -> 234KB (32x smaller)
The 32x INDEX-MEMORY reduction is real and unconditional. The SPEEDUP is only
~1.2-1.5x in JS, not ruflo's figure: a 1-bit sign signature in 384-dim is
coarse, so preserving recall@10 >= 0.95 needs a large exact-rerank pool
(~40-55% of N), which eats most of the prefilter win. ruflo's big win came
from replacing a LINEAR scan; AQE's main search is already native HNSW
(sublinear), so that win doesn't transfer. Tests pin this tradeoff (recall==1.0
with a sufficient pool; small pool sacrifices recall) so it can't be
mis-sold later.
NOT INTEGRATED — no zero-regression consumer exists:
- unified-memory-hnsw search() IS the ranking (no separate exact rerank to
prefilter), so Hamming-prefiltering its beam would drop true neighbors;
- coherence-gate-energy O(N^2) SUM and k-means clustering are numerically
sensitive (Hamming would change results).
Committed as a tested, benchmarked, ready-to-use utility (real memory lever for
patterns.rvf and a future prefilter->exact-rerank consumer), not wired into any
production path. No production code or .agentic-qe stores touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports ruflo's benchmark-self-learning.mjs discipline (the 'reports success
but persists nothing' detector) to AQE. Proves the learning stores actually
persist what they claim, that stat aggregators agree with the underlying
index, that nothing writes accidentally, and that data survives restart.
Sections (each isolated in a temp dir, each gating):
A store persistence — N creates => totalPatterns delta == N
B outcome learning — N outcomes => learningOutcomes delta == N, successRate matches fed ratio
C negative control — read-only ops => 0 delta (no accidental writes)
D restart survival — dispose + reopen on same temp path => counts persist
E consistency — RvfPatternStore.getStats == adapter.totalVectors == bank.totalPatterns
F audit-real — copies real stores read-only, checks banner numbers round-trip
Data protection (CLAUDE.md): synthetic mode uses fresh temp dirs; audit-real
copies stores and opens only copies. Verified real .rvf stores byte-identical
(mtime+size+sha256) before/after a run.
Audit-real finding on the live store: banner numbers corroborated
(72.5% success, 39.7% confidence, 2131 routings) BUT RVF vector count (338)
!= SQLite qe_patterns rows (345), a ~7-pattern index/table drift the banner
hides — exactly the divergence this harness exists to surface (reported as a
diagnostic note, non-gating). Synthetic sections A-E pass; exit 1 on failure.
npm run benchmark:self-learning
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dragan Spiridonov