ruv ef4acc49de feat(beir): 4-dataset BEIR (SciDocs joins) + config divergence finding (ADR-091, 3.10.30) ...

4th BEIR dataset shipped. SciDocs (25,657 docs, 1000 queries) — paper-similarity
retrieval where dense beats lexical+fusion.

| Pipeline             | SciDocs nDCG@10 |
|----------------------|-----------------|
| dense alone (BGE-base)  | 0.211  (RANK 2/11) |
| Lucene RRF (no rerank)  | 0.203  (-0.008, RRF hurt) |

SciDocs joins the 3 prior datasets. The 4-dataset mean leaderboard:

| System                       | Params | Mean |
|------------------------------|-------:|-----:|
| BGE-large-v1.5 (published)   |   335M | 0.491 |
| SPLADE++ (published)         |   110M | 0.433 |
| ruflo best (per-dataset)     |   110M | 0.421 ← us, rank 3/11 |
| GTR-XL (published)           |  1.2B  | 0.405 |
| GenQ (published)             |   110M | 0.400 |
| BM25 (published Lucene)      |      — | 0.390 |
| Contriever                   |   110M | 0.387 |
| TAS-B                        |    66M | 0.385 |
| DocT5query                   |    60M | 0.378 |
| ColBERT                      |   110M | 0.339 |
| SBERT msmarco                |   110M | 0.330 |

Rank 3/11 on 4-dataset mean. Beats GTR-XL (1.2B) with 1/10x the params.
Loses only to SPLADE++ (-0.012, basically tied) and BGE-large (-0.070).

=== Config divergence (the real ADR-091 finding) ===

After 4 datasets, NO single pipeline wins everywhere:

| Dataset    | Best config                | What hurts                  |
|------------|----------------------------|-----------------------------|
| NFCorpus   | Lucene + RRF + CE rerank   | (full pipeline wins)        |
| SciFact    | Lucene + RRF + CE rerank   | (full pipeline wins)        |
| ArguAna    | Lucene + RRF (no CE)       | CE rerank actively HURTS    |
| SciDocs    | dense alone                | RRF + CE both hurt          |

Three of four datasets pick a different best config. A fixed-pipeline
deployment would leave 1-3 points of nDCG@10 on the table per dataset.
Need a per-corpus calibrator to auto-select; tracked.

Co-Authored-By: RuFlo <ruv@ruv.net>

2026-05-31 03:17:53 -04:00

13 KiB

Raw Permalink Blame History

View Raw