ruv 17ce6ba67d feat(intelligence): labelled held-out corpus + nDCG metrics (ADR-081, 3.10.21) ...

Replace the regex-over-subject relevance proxy used in ADRs 077-080 with
a hand-curated labelled corpus. Each of the 10 bench queries gets
expectedSubstrings: string[] — case-insensitive substring matches against
the pattern name. Adds precision@3, nDCG@3, nDCG@5 alongside the existing
hit-rate and MRR metrics. Regex metrics preserved for back-compat.

WHY: The regex proxy was both under- and over-reporting. Honest SOTA
requires honest measurement.

| Config              | Regex top-1 | Label top-1 | Direction         |
|---------------------|------------:|------------:|-------------------|
| Hybrid (3.10.19)    |         80% |        90%  | regex under-reports |
| Hybrid + Rerank     |         90% |        80%  | regex over-reports  |

Labelled metrics across the full 3.10.17 → 3.10.20 push:

| Metric               | Cosine | Hybrid | Hybrid+Rerank |
|----------------------|-------:|-------:|--------------:|
| Label top-1 hit rate |     0% |    90% |           80% |
| Label top-3 hit rate |     0% |    90% |          100% |
| Label MRR@3          |  0.000 |  0.900 |         0.883 |
| Label precision@3    |  0.000 |  0.400 |         0.667 |
| Label nDCG@3         |  0.000 |  0.900 |         0.913 |

The cross-encoder optimises for "find all the right docs" (precision@3
0.40 → 0.67) while hybrid alone optimises for "find THE right doc first"
(top-1 80% → 90%). Default rerank: false remains correct for top-1
callers; opt-in rerank: true for richer top-K consumers.

Implementation:
  - QUERIES gains expectedSubstrings: string[] per item
  - isRelevant(name, expectedSubstrings) — case-insensitive substring match
  - ndcgAtK(rankedRelevance, k) — binary relevance + ideal-DCG normalisation
  - 6 new metrics in summary JSON + console output
  - Old regex metrics kept under "regex proxy" labels for back-compat

Co-Authored-By: RuFlo <ruv@ruv.net>

2026-05-30 14:34:59 -04:00

14 KiB

Raw Permalink Blame History

View Raw