Took the "release hype → benchmark infrastructure" feedback to the wire.
This release IS the infrastructure, not the rank.
What ships:
- docs/benchmarks/BEIR-MATRIX.md — dataset × pipeline × metric grid with
bootstrap CIs and pipeline disclosure (direct dense, no fine-tune,
no rerank)
- scripts/beir-bootstrap-significance.mjs — paired bootstrap, 10k
resamples, mulberry32 seed=42, reports CI overlap vs each baseline
- perQuery metrics now saved in every run JSON (for external bootstrap
verification by anyone)
- ADR-085 hedged ("TOP-2 on BEIR NFCorpus, not BEIR average"; "Best
Listed Baseline"; "direct dense, no fine-tune, no rerank")
- ADR-086 documents the silent-fallback bug story + bootstrap method
+ the two-dataset honest picture
- 2 ruvector issues filed upstream (ruvnet/ruvector#523, #524)
Honest two-dataset picture (95% CIs, 10k bootstrap):
| Dataset | nDCG@10 | 95% CI | Rank | vs BM25 |
|----------|--------:|----------------|------:|------------------------|
| NFCorpus | 0.352 | [0.317, 0.387] | 2/11 | +0.027 (n.s.) |
| SciFact | 0.626 | [0.577, 0.672] | 10/11 | -0.053 (p<0.05) LOSS |
User's acceptance test "ruflo beats BM25 on both" FAILS on SciFact —
we significantly lose to BM25 there (CI excludes baseline). On NFCorpus,
only SBERT msmarco (Δ +0.080) and ColBERT (Δ +0.047) are statistically
significant wins; the "rank-2" gap to SPLADE++ et al is within sampling
noise (CI overlaps).
Two-dataset mean: ours 0.489, BM25 0.502, SPLADE++ 0.526, BGE-large 0.551.
We're below BM25 on the mean. BGE-base zero-shot is competent on
NFCorpus (medical IR), weak on SciFact (fact-verification with rare
scientific terms favouring lexical retrieval). This validates the user's
"single-dataset wins don't generalise" critique with our own numbers.
The release is honest infrastructure, not a SOTA claim.
Co-Authored-By: RuFlo <ruv@ruv.net>
ruv