Acceptance test from the climb plan: "RRF improves or preserves nDCG@10
on both NFCorpus and SciFact, bootstrap CI does not undermine the
claim, defaults fixed before viewing test result."
THE TEST FAILS. Default RRF k=60 equal weights degrades nDCG@10 on
BOTH datasets. We ship the harness + the honest finding anyway.
Measured (full ablation, fixed defaults BEFORE viewing results):
| Config | NFCorpus | SciFact | Mean |
|---------------------------|---------:|--------:|-------:|
| dense alone (BGE-base) | 0.352 | 0.626 | 0.489 |
| BM25 alone (ours) | 0.279 | 0.576 | 0.428 |
| RRF k=60 equal (default) | 0.328 | 0.569 | 0.449 |
| RRF k=30 equal | 0.335 | 0.582 | 0.459 |
| RRF k=60 dense=1.2,bm25=0.8 | 0.334 | 0.577 | 0.456 |
| RRF k=60 dense=0.8,bm25=1.2 | 0.323 | 0.558 | 0.441 |
Every RRF variant UNDERPERFORMS dense-alone on the 2-dataset mean.
Recall@100 DOES improve (RRF surfaces more candidates) — but top-K
ranking degrades because BM25 noise gets averaged into top positions
when one input is materially weaker than the other.
Diagnosis: our multi-field BM25 (no Snowball stemmer, ~25 stopwords)
is weaker than Lucene's (0.279 NFCorpus vs published 0.325). RRF's
classic win assumes comparably-strong systems with different failure
modes. Asymmetric strength → noise averaging instead of cancellation.
Bug fixed during this work: bge-cache/ path was hardcoded to
/tmp/beir-nfcorpus/bge-cache, making SciFact run silently overwrite
NFCorpus cache. Now per-dataset path. (Caught only when first RRF
returned nDCG=0.14.)
Ships:
- scripts/run-beir-rrf-ablation.mjs — re-runnable ablation harness
with bootstrap CI per default config
- scripts/run-beir-hybrid.mjs — full RRF + opt-in rerank runner
(rerank story is ADR-088, pending)
- bge-cache/ path fix in run-beir-bge.mjs
- ADR-087 documents the negative finding + diagnosis + next steps
- Updated BEIR-MATRIX.md with ablation rows
- No default change — dense-only remains the BEIR runner default
Next round (ADR-088): cross-encoder rerank on RRF's wider candidate
pool. Recall@100 IS up (0.95 SciFact), so a stronger pairwise scorer
should pull genuine top-K out. Then Lucene-style BM25 to fix the
asymmetric-strength root cause.
Co-Authored-By: RuFlo <ruv@ruv.net>
ruv