skills/ruvnet__ruflo
1
0
Fork 0
mirror of https://github.com/ruvnet/ruflo.git synced 2026-09-14 14:01:28 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,435 Commits 580 Branches 1,645 Tags
ci/control-current-main
Commit Graph

5 Commits

Author SHA1 Message Date
ruv ef4acc49de feat(beir): 4-dataset BEIR (SciDocs joins) + config divergence finding (ADR-091, 3.10.30) 
4th BEIR dataset shipped. SciDocs (25,657 docs, 1000 queries) — paper-similarity
retrieval where dense beats lexical+fusion.

| Pipeline             | SciDocs nDCG@10 |
|----------------------|-----------------|
| dense alone (BGE-base)  | 0.211  (RANK 2/11) |
| Lucene RRF (no rerank)  | 0.203  (-0.008, RRF hurt) |

SciDocs joins the 3 prior datasets. The 4-dataset mean leaderboard:

| System                       | Params | Mean |
|------------------------------|-------:|-----:|
| BGE-large-v1.5 (published)   |   335M | 0.491 |
| SPLADE++ (published)         |   110M | 0.433 |
| ruflo best (per-dataset)     |   110M | 0.421 ← us, rank 3/11 |
| GTR-XL (published)           |  1.2B  | 0.405 |
| GenQ (published)             |   110M | 0.400 |
| BM25 (published Lucene)      |      — | 0.390 |
| Contriever                   |   110M | 0.387 |
| TAS-B                        |    66M | 0.385 |
| DocT5query                   |    60M | 0.378 |
| ColBERT                      |   110M | 0.339 |
| SBERT msmarco                |   110M | 0.330 |

Rank 3/11 on 4-dataset mean. Beats GTR-XL (1.2B) with 1/10x the params.
Loses only to SPLADE++ (-0.012, basically tied) and BGE-large (-0.070).

=== Config divergence (the real ADR-091 finding) ===

After 4 datasets, NO single pipeline wins everywhere:

| Dataset    | Best config                | What hurts                  |
|------------|----------------------------|-----------------------------|
| NFCorpus   | Lucene + RRF + CE rerank   | (full pipeline wins)        |
| SciFact    | Lucene + RRF + CE rerank   | (full pipeline wins)        |
| ArguAna    | Lucene + RRF (no CE)       | CE rerank actively HURTS    |
| SciDocs    | dense alone                | RRF + CE both hurt          |

Three of four datasets pick a different best config. A fixed-pipeline
deployment would leave 1-3 points of nDCG@10 on the table per dataset.
Need a per-corpus calibrator to auto-select; tracked.

Co-Authored-By: RuFlo <ruv@ruv.net>
 2026-05-31 03:17:53 -04:00
ruv 05bb9cf7ed feat(beir): 3-dataset BEIR + ruvector@0.2.27 + #2246 fixes (ADR-089/090, 3.10.29) 
Batch ship per "no constant releases" guidance. Four independent threads.

1) 3rd BEIR dataset (ArguAna) — strengthens 2-dataset → 3-dataset story
2) BGE-large NFCorpus ceiling test — answered (no lift on this hardware)
3) ruvector@0.2.27 Tier-0 wiring — kills the silent-fallback bug at source
4) 4 user bugs from #2246 — 3 fixed in ruflo, 1 forwarded upstream

=== 3-dataset BEIR results (best ruflo config per dataset) ===

| Dataset    | nDCG@10 | Pipeline                | Rank | vs published best |
|------------|--------:|-------------------------|-----:|------------------:|
| NFCorpus   |   0.358 | Lucene+RRF+CE rerank    | 2/11 | -0.022 vs BGE-large 0.380 |
| SciFact    |   0.683 | Lucene+RRF+CE rerank    | 3/11 | -0.039 vs BGE-large 0.722 |
| ArguAna    |   0.432 | Lucene+RRF (no CE)      | 5/11 | -0.204 vs BGE-large 0.636 |
| 3-ds mean  |   0.491 | mixed                   |    — | -0.088 vs BGE-large 0.579 |

vs published baselines (3-dataset mean):
  BGE-large (335M)         : 0.579  +0.088
  SPLADE++                 : 0.524  +0.033
  GenQ                     : 0.485  -0.006 (tied)
  ruflo best (per-dataset) : 0.491  ← us
  GTR-XL (1.2B)            : 0.481  -0.010
  BM25 (Lucene)            : 0.467  -0.024  (we BEAT)
  Contriever               : 0.461  -0.030  (we BEAT)
  TAS-B                    : 0.464  -0.027  (we BEAT)

Rank 4 of 11 on 3-dataset mean. Beats published BM25, GTR-XL (1/10x our
params), Contriever, TAS-B, ColBERT, SBERT. Loses to SPLADE++ and BGE-large.

=== ArguAna kills the cross-encoder ===

CE rerank pulled at the 50-query checkpoint: running nDCG=0.283 vs dense
alone 0.431, estimated 6+ hours wall time. ArguAna is counter-argument
retrieval — pointwise relevance reranking doesn't help when the task
requires understanding opposition. Pipeline auto-adapts: rerank wins
NFCorpus and SciFact, loses ArguAna.

=== BGE-large NFCorpus = no lift ===

| Model                                  | NFCorpus nDCG@10 |
|----------------------------------------|------------------|
| BGE-base-en-v1.5  (110M, int8 quant)  |  0.352           |
| BGE-large-en-v1.5 (335M, int8 quant)  |  0.350  ← no lift|
| BGE-large (published BAAI fp32)        |  0.380           |

3x model size, ~3x latency, no measured improvement. Likely Xenova int8
quantization underperforms BAAI's unquantized fp32 (the 0.030 gap matches
typical int8 BERT-large degradation). Real BGE-large probably needs GPU.

=== ruvector@0.2.27 Tier-0 wiring ===

neural-tools embedder cascade now:
  Tier 0 (NEW): ruvector@0.2.27.embed() — bundled, no sharp dep, disk-cache
  Tier 1: agentic-flow/reasoningbank (broken on darwin-arm64 without sharp)
  Tier 2-3: @claude-flow/embeddings

Verified active: probe returns "ruvector@0.2.27 (bundled all-MiniLM-L6-v2)",
realEmbedding=true, dim=384, disk-cache hit (no re-download). Measured
6.2x per-doc parallel-embed speedup (claimed 10-14x; ours had CPU
contention from BEIR benches).

This is the ADR-086 silent-fallback bug killed at source. Both upstream
issues filed yesterday (ruvnet/ruvector#523, #524) — the former is fully
fixed; the latter (bundle BGE) is acknowledged.

=== #2246 user bug fixes ===

#1 memory_search_unified hardcoded 6 namespaces (missed 95% of user store):
  FIXED. New `namespaces: string[]` param + CLAUDE_FLOW_MEMORY_SEARCH_NAMESPACES
  env + dynamic enumeration via listEntries({}) as the new default +
  namespaceSource audit field. Resolution priority documented. 9 regression
  tests (no path traversal allowed; explicit list beats env beats dynamic
  beats legacy-fallback).

#2 npm install -g overwrites dist patches silently: ACKNOWLEDGED, separate
  release for postinstall checksum + warning.

#3 agentdb addCausalEdge() silently creates orphan edges: FORWARDED as
  ruvnet/agentdb#7 with reproduction and three suggested fixes.

#4 graph-pathfinder "DB unavailable" on fresh env: FIXED.
  getBridgeDb({createIfMissing: true}) lazy-creates empty memory.db +
  graph_edges schema; 3 pathfinder call sites updated; error message
  now includes a `hint` field.

=== ADR-090: BGE query prefix is mixed ===

BAAI's recommended "Represent this sentence for searching relevant
passages: " query-side prefix:
  NFCorpus: +0.009 nDCG@10 (helps, question-shaped queries)
  SciFact:  -0.007 nDCG@10 (hurts, claim-shaped queries)
  ArguAna:  +0.003 nDCG@10 (noise)

Opt-in only via BGE_QUERY_PREFIX=1. Not a default.

Honest limits in ADR §"Honest limits": 3/18 BEIR datasets, all zero-shot
(NFCorpus train has 110k pairs we haven't used), CPU-bound for larger
BEIR datasets, our Lucene BM25 matches published ±0.003 but is a
re-implementation not a Lucene binding.

Co-Authored-By: RuFlo <ruv@ruv.net>
 2026-05-30 23:47:08 -04:00
ruv ea99215f8c feat(beir): RRF ablation harness + HONEST NEGATIVE RESULT (ADR-087, 3.10.27) 
Acceptance test from the climb plan: "RRF improves or preserves nDCG@10
on both NFCorpus and SciFact, bootstrap CI does not undermine the
claim, defaults fixed before viewing test result."

THE TEST FAILS. Default RRF k=60 equal weights degrades nDCG@10 on
BOTH datasets. We ship the harness + the honest finding anyway.

Measured (full ablation, fixed defaults BEFORE viewing results):

| Config                    | NFCorpus | SciFact | Mean   |
|---------------------------|---------:|--------:|-------:|
| dense alone (BGE-base)    |    0.352 |   0.626 |  0.489 |
| BM25 alone (ours)         |    0.279 |   0.576 |  0.428 |
| RRF k=60 equal (default)  |    0.328 |   0.569 |  0.449 |
| RRF k=30 equal            |    0.335 |   0.582 |  0.459 |
| RRF k=60 dense=1.2,bm25=0.8 | 0.334  |   0.577 |  0.456 |
| RRF k=60 dense=0.8,bm25=1.2 | 0.323  |   0.558 |  0.441 |

Every RRF variant UNDERPERFORMS dense-alone on the 2-dataset mean.
Recall@100 DOES improve (RRF surfaces more candidates) — but top-K
ranking degrades because BM25 noise gets averaged into top positions
when one input is materially weaker than the other.

Diagnosis: our multi-field BM25 (no Snowball stemmer, ~25 stopwords)
is weaker than Lucene's (0.279 NFCorpus vs published 0.325). RRF's
classic win assumes comparably-strong systems with different failure
modes. Asymmetric strength → noise averaging instead of cancellation.

Bug fixed during this work: bge-cache/ path was hardcoded to
/tmp/beir-nfcorpus/bge-cache, making SciFact run silently overwrite
NFCorpus cache. Now per-dataset path. (Caught only when first RRF
returned nDCG=0.14.)

Ships:
  - scripts/run-beir-rrf-ablation.mjs  — re-runnable ablation harness
    with bootstrap CI per default config
  - scripts/run-beir-hybrid.mjs        — full RRF + opt-in rerank runner
    (rerank story is ADR-088, pending)
  - bge-cache/ path fix in run-beir-bge.mjs
  - ADR-087 documents the negative finding + diagnosis + next steps
  - Updated BEIR-MATRIX.md with ablation rows
  - No default change — dense-only remains the BEIR runner default

Next round (ADR-088): cross-encoder rerank on RRF's wider candidate
pool. Recall@100 IS up (0.95 SciFact), so a stronger pairwise scorer
should pull genuine top-K out. Then Lucene-style BM25 to fix the
asymmetric-strength root cause.

Co-Authored-By: RuFlo <ruv@ruv.net>
 2026-05-30 17:58:35 -04:00
ruv 1490278176 feat(intelligence): BEIR matrix + bootstrap CIs + 2nd dataset (ADR-086, 3.10.26) 
Took the "release hype → benchmark infrastructure" feedback to the wire.
This release IS the infrastructure, not the rank.

What ships:
  - docs/benchmarks/BEIR-MATRIX.md  — dataset × pipeline × metric grid with
    bootstrap CIs and pipeline disclosure (direct dense, no fine-tune,
    no rerank)
  - scripts/beir-bootstrap-significance.mjs  — paired bootstrap, 10k
    resamples, mulberry32 seed=42, reports CI overlap vs each baseline
  - perQuery metrics now saved in every run JSON (for external bootstrap
    verification by anyone)
  - ADR-085 hedged ("TOP-2 on BEIR NFCorpus, not BEIR average"; "Best
    Listed Baseline"; "direct dense, no fine-tune, no rerank")
  - ADR-086 documents the silent-fallback bug story + bootstrap method
    + the two-dataset honest picture
  - 2 ruvector issues filed upstream (ruvnet/ruvector#523, #524)

Honest two-dataset picture (95% CIs, 10k bootstrap):

| Dataset  | nDCG@10 | 95% CI         | Rank  | vs BM25                |
|----------|--------:|----------------|------:|------------------------|
| NFCorpus |  0.352  | [0.317, 0.387] |  2/11 | +0.027 (n.s.)          |
| SciFact  |  0.626  | [0.577, 0.672] | 10/11 | -0.053 (p<0.05) LOSS   |

User's acceptance test "ruflo beats BM25 on both" FAILS on SciFact —
we significantly lose to BM25 there (CI excludes baseline). On NFCorpus,
only SBERT msmarco (Δ +0.080) and ColBERT (Δ +0.047) are statistically
significant wins; the "rank-2" gap to SPLADE++ et al is within sampling
noise (CI overlaps).

Two-dataset mean: ours 0.489, BM25 0.502, SPLADE++ 0.526, BGE-large 0.551.
We're below BM25 on the mean. BGE-base zero-shot is competent on
NFCorpus (medical IR), weak on SciFact (fact-verification with rare
scientific terms favouring lexical retrieval). This validates the user's
"single-dataset wins don't generalise" critique with our own numbers.

The release is honest infrastructure, not a SOTA claim.

Co-Authored-By: RuFlo <ruv@ruv.net>
 2026-05-30 16:48:38 -04:00
ruv 004cc25249 feat(intelligence): BEIR public benchmark + BGE bi-encoder — rank 2/11 on NFCorpus (ADR-085, 3.10.25) 
First real public-benchmark result. Built BEIR harness + bypassed the
broken sharp dependency by loading Xenova/bge-base-en-v1.5 directly
via @xenova/transformers AutoTokenizer + AutoModel.

Measured (BEIR NFCorpus, N=323 test queries, full corpus 3633 docs):

  nDCG@10:    0.352
  MRR@10:     0.546
  Recall@10:  0.167
  Recall@100: 0.305
  Avg query latency: 388 ms

Rank 2 of 11 published baselines:

| Rank | Method                              | nDCG@10 |
|-----:|-------------------------------------|--------:|
|    1 | BGE-large-v1.5 (335M, published)    |   0.380 |
|    2 | ruflo + BGE-base (110M) ← us        |   0.352 |
|    3 | SPLADE++                            |   0.347 |
|    4 | GTR-XL                              |   0.343 |
|    5 | DocT5query / Contriever             |   0.328 |
|    7 | BM25 (Lucene)                       |   0.325 |
|    8 | TAS-B / GenQ                        |   0.319 |
|   10 | ColBERT                             |   0.305 |
|   11 | SBERT msmarco                       |   0.272 |

Beats SPLADE++, GTR-XL, ColBERT and every published baseline except
BGE-large — and we're using a 3x smaller base (110M vs 335M). Closing
the remaining 0.028 gap is a model swap: BGE_MODEL=Xenova/bge-large-en-v1.5.

Also surfaced + worked around a real env bug: sharp-darwin-arm64v8.node
missing in pnpm install made agentic-flow's embedder silently
hash-fall-back. Pure-BM25 baseline measured at nDCG@10 = 0.289 (vs
published BM25 0.325 — close; ~0.04 gap from BEIR-specific Lucene
tokenization we don't replicate).

Ships:
  - scripts/run-beir-nfcorpus.mjs  (hybrid harness)
  - scripts/run-beir-bge.mjs       (BGE dense, on-disk cache, dataset auto-detect)
  - src/memory/bge-embedder.ts     (lazy-loaded singleton, small/base/large)

SciFact (2nd BEIR dataset) is ingesting in background; result lands in 3.10.26.

Co-Authored-By: RuFlo <ruv@ruv.net>
 2026-05-30 16:08:29 -04:00