mirror of
https://github.com/ruvnet/ruflo.git
synced 2026-09-14 14:01:28 +08:00
ruv
62df8a184d
Closes the relevance gap exposed by ADR-077: cosine-only neural_patterns
search was returning plausible-but-off-topic results because the bridge ONNX
bi-encoder gets distracted on small corpora by IDF-cheap shared tokens.
Three changes, one release:
1) Hybrid retrieval (src/memory/hybrid-retrieval.ts) — BM25 (k1=1.5,b=0.75)
+ cosine + MMR diversity (λ=0.5). Pure functions, no deps, 21 unit tests.
Wired into neural_patterns search via {mode:'hybrid'|'cosine', alpha,
mmrLambda, limit}. Cosine path preserved for A/B.
2) Pattern.content persistence — adds source text (cap 4096 chars) to the
neural store so BM25 has tokens to score against. Backwards compatible:
pre-3.10.18 patterns fall back to `name`.
3) Outcome signal in pretrain harvester — detects reverted (Revert "<X>"
pattern) and hotfixed (≥50% file overlap with later fix commit) commits,
surfacing the verdict mix in summary + recording original outcome in
trajectory metadata.
Measured (N=385, 10 queries, this checkout):
| Metric | Cosine | Hybrid | Δ |
|-----------------------|-------:|-------:|--------:|
| Top-1 hit rate | 0% | 50% | +50pp |
| Top-3 hit rate | 0% | 70% | +70pp |
| MRR@3 | 0.000 | 0.583 | +0.583 |
| Top-1 diversity | 100% | 80% | -20pp |
| Avg query latency | 28.7ms | 40.6ms | +11.9ms |
Cosine was returning 0% relevant top-3 results — finding "something" but
never the right thing. Hybrid lands relevant top-1 50% / top-3 70% with a
12 ms latency cost.
Co-Authored-By: RuFlo <ruv@ruv.net>
63 lines
1.4 KiB
JSON
63 lines
1.4 KiB
JSON
{
|
|
"runAt": "2026-05-30T17:15:24.767Z",
|
|
"benchmark": "pretrain-from-github",
|
|
"source": "git",
|
|
"config": {
|
|
"COMMITS": 200,
|
|
"ISSUES": 30
|
|
},
|
|
"harvest": {
|
|
"commits": 200,
|
|
"issues": 0,
|
|
"total": 200,
|
|
"harvestMs": 5577.96
|
|
},
|
|
"feed": {
|
|
"trained": 200,
|
|
"failed": 0,
|
|
"avgLatencyMs": 12.38,
|
|
"totalMs": 2475.27,
|
|
"sampleFailures": [],
|
|
"verdictMix": {
|
|
"success": 200,
|
|
"partial": 0,
|
|
"reverted": 0,
|
|
"hotfixed": 0
|
|
}
|
|
},
|
|
"seedNeuralStore": {
|
|
"stored": 200,
|
|
"total": 200,
|
|
"seedMs": 95.03
|
|
},
|
|
"before": {
|
|
"trajectoriesRecorded": 0,
|
|
"patternsLearned": 0,
|
|
"signalsProcessed": 0,
|
|
"neuralPatternCount": 185,
|
|
"memoryBridgeTotal": 0
|
|
},
|
|
"after": {
|
|
"trajectoriesRecorded": 385,
|
|
"patternsLearned": 365,
|
|
"signalsProcessed": 0,
|
|
"neuralPatternCount": 385,
|
|
"memoryBridgeTotal": 0
|
|
},
|
|
"deltas": {
|
|
"trajectoriesRecorded": 385,
|
|
"patternsLearned": 365,
|
|
"signalsProcessed": 0,
|
|
"neuralPatternCount": 200,
|
|
"memoryBridgeTotal": 0
|
|
},
|
|
"consistency": {
|
|
"sonaTracksGlobal": false,
|
|
"sonaTracksGlobalDelta": -385,
|
|
"notes": [
|
|
"sona.trajectoriesTotal (0) drifts from globalStats.trajectoriesRecorded (385) by -385 — expected to track within ±1",
|
|
"memory-bridge unreachable — bridge-dependent counters (post-edit/-command persistence, pretrain bundle) will show 0"
|
|
]
|
|
},
|
|
"passed": true
|
|
} |