 rUv
|
08b3623759
|
fix+perf+docs(intelligence): self-learning audit, hardening fixes, honest perf numbers (3.10.7) (#2228)
* docs(review): empirical capability audit of the intelligence/self-learning system
6 parallel auditors ran real measurements against built dist (not docs).
- Self-learning loop is REAL, measured end-to-end (pattern confidence +
Q-routing change persist cross-process; MoE gating converges; SONA WASM
adapt 0.0042ms; Int8 3.92x; RaBitQ 32x memory — all CONFIRMED).
- Performance multipliers largely UNSUBSTANTIATED: HNSW 150x-12,500x measured
1.48x peak; Flash Attention 2.49-7.47x fabricated via Math.random() at
runtime; embeddings 75x / RaBitQ 2.70x retrieval never benchmarked.
- 1 critical bug: CLI inverts negative reward (route feedback -r -1.0 -> +1.00).
- Real-but-inert: WASM MicroLoRA apply(), MCP consolidation gradient (synthetic),
hooks_intelligence_learn (cosmetic), CLI trajectory-* (no-op stubs), silent
mock-embedding mislabel.
No source modified — audit branch only. Prioritized fix list in the report.
Co-Authored-By: RuFlo <ruv@ruv.net>
* fix+perf(intelligence): harden self-learning system + replace fabricated/unsubstantiated perf numbers with measured ones
Implements the prioritized fix list from docs/reviews/intelligence-system-audit-2026-05-29.md.
FIXES
- CRITICAL (audit #1, #2222 follow-up): CLI inverted negative reward — route feedback
-r -1.0 / --reward -1.0 parsed as +1.00, so negative feedback REINFORCED the bad
agent. Fixed in src/parser.ts (isFlagValue accepts negative numeric literals). All
three syntaxes now yield -1.0; +6 regression assertions; 52/52 parser tests pass.
- audit #2: removed fabricated Flash Attention speedup (randomized/midpoint telemetry)
in attention-coordinator (swarm + integration) -> unmeasured sentinel + unverified.
- audit #3: generateEmbedding returns backend:onnx|mock, surfaced in bridge_status /
import_claude so mock embeddings are never mislabeled as real ONNX.
- audit #4/#5: trajectory-end no longer feeds EWC a synthetic sine gradient;
hooks_intelligence_learn runs a real distill/consolidate cycle. New test added.
OPTIMIZE (HNSW, measured): root cause — HNSW never ran (no storagePath -> native DB
lock held by daemon -> silent catch{} -> brute force). Fixed vector-db.ts (unique
storagePath, hnswConfig m:32/efC:200, visible fallback warning). Same-harness
before->after: N=5000 0.92x->3.2-4.7x, N=20000 0.95x->1.89x; N=1000 below crossover
(reported honestly); reverted m16/efC64 (recall loss).
BENCHMARK: new scripts/benchmark-intelligence.mjs (HNSW N=1k-50k, Int8 3.84x/cos
0.99999, RaBitQ 32x/0.60ms, SONA 0.0043ms, MoE 0.13->0.88, embedding backend).
DOCS: README + 3x CLAUDE.md + READMEs perf tables now show MEASURED values or
unverified/target; HNSW 150x-12,500x & Flash 2.49-7.47x marked NOT reproduced.
Verification: cli build clean (tsc); 161/161 targeted tests pass.
Co-Authored-By: RuFlo <ruv@ruv.net>
* chore(release): bump 3.10.6 → 3.10.7 (intelligence hardening + honest perf docs)
Co-Authored-By: RuFlo <ruv@ruv.net>
|
2026-05-29 12:13:53 -04:00 |
|