mirror of https://github.com/ruvnet/ruflo.git synced 2026-09-14 14:01:28 +08:00

Files

T

ruv 6d6d659ca9 perf(guidance): Phase 1 — hot-path cleanup + scaling benchmark infrastructure

Horizon: guidance-sota-2026-05 (M1 + M2). First commit on the path to
SOTA-with-proof for @claude-flow/guidance.

## What changed

Three localized hot-path cleanups + a benchmark infrastructure that
exposes the real opportunity (M3, HNSW substrate).

### Code changes (no public API surface change)

1. **analyzer.ts extractMetrics** — replace 6 separate `lines.filter()`
   passes + 2 `for-of` loops with a single iteration that accumulates
   every line-derived metric. Hoist 11 in-body regex literals to module
   scope. Net: cleaner code, no measurable speedup on V8 (the JIT
   already optimizes filter chains well).

2. **compiler.ts parseRule** — replace 4 `new RegExp(PATTERN.source, 'gi')`
   constructions per call with `text.matchAll(PATTERN)` against the
   existing module-level global patterns. Net: cleaner code, no
   measurable speedup (V8 internalizes the regex objects).

3. **retriever.ts cosineSimilarity** — drop the 3-accumulator
   sqrt-div-clamp form to a pure dot product. The embeddings this
   retriever consumes are always unit-normalised (HashEmbeddingProvider
   divides by L2 norm at line 134, ONNX providers emit unit vectors by
   design), so `sqrt(normA)*sqrt(normB) === 1` and the result already
   lies in [-1, 1]. Net: +8.2% at N=1000 shards, +10.9% at N=500.

### Benchmark infrastructure (new)

- `scripts/bench-phase-1.mjs` — multi-trial median microbenchmarks for
  the 3 hot paths. Stable to ~2%.
- `scripts/bench-retriever-scale.mjs` — end-to-end `retriever.retrieve()`
  at N ∈ {10, 100, 500, 1000}. Reveals the real scaling cliff.
- `scripts/README.md` — how to run + findings.
- `docs/benchmarks/guidance-{baseline,phase-1}.json` — captured runs.
- `docs/benchmarks/guidance-retriever-scale-{baseline,phase-1}.json` —
  scaling curves at four shard counts.

## Honest M2 status

Original target: ≥2x speedup or ≥30% reduction. Phase 1 delivers ~1.1x
at scale — short of the bar. **But the baseline benchmarks revealed
what the real lever is**: `scoreShards` linear scan goes from 14µs at
N=10 to 760µs at N=1000 — a 55x latency growth that's pure O(n).

M3 (HNSW substrate) targets that bottleneck directly. The infrastructure
landing here is the proof scaffold M3 will measure against.

## Verification

- `npm test` in v3/@claude-flow/guidance: all 1,331 tests pass
- `npm run build`: clean
- Bench scripts runnable end-to-end with JSON artifacts under
  `docs/benchmarks/`

Co-Authored-By: RuFlo <ruflo-bot@users.noreply.github.com>

2026-05-21 23:21:29 -04:00

1.8 KiB

Raw Permalink Blame History

Guidance Performance Benchmarks

Phase 1 benchmarks for the @claude-flow/guidance SOTA optimization horizon (guidance-sota-2026-05).

Scripts

Script	Measures
`bench-phase-1.mjs`	Micro-benchmarks for the 3 hot paths identified by the researcher (analyzer extractMetrics, compiler parseRule, retriever cosine)
`bench-retriever-scale.mjs`	End-to-end `retriever.retrieve()` latency at N ∈ {10, 100, 500, 1000} shards — the production-facing scaling curve

Running

cd v3/@claude-flow/guidance && npm run build
node v3/@claude-flow/guidance/scripts/bench-phase-1.mjs --tag=baseline
node v3/@claude-flow/guidance/scripts/bench-retriever-scale.mjs --tag=baseline

Output lands in docs/benchmarks/guidance-*-<tag>.json.

Findings (Phase 1, 2026-05-22)

Bench	Baseline	Phase 1	Δ
analyzer.analyze (150-line CLAUDE.md)	2,896 ops/s	2,860 ops/s	within noise
compiler.compile (150-line CLAUDE.md)	3,752 ops/s	3,704 ops/s	within noise
retriever.retrieve (N=500)	2,457 ops/s	2,724 ops/s	+10.9%
retriever.retrieve (N=1000)	1,317 ops/s	1,425 ops/s	+8.2%

The micro-optimizations to extractMetrics (single-pass loop) and parseRule (text.matchAll instead of new RegExp(...) per call) are within run-to-run noise on V8 — the JIT already optimizes these patterns heavily. The retriever changes (unit-vector dot-only cosine + same single-pass philosophy) deliver a real 8-11% lift at scale.

The real opportunity is M3: replace scoreShards's O(n) linear scan (retriever.ts:268) with an HNSW ANN query. Baseline shows latency goes from 14µs at N=10 to 760µs at N=1000 — pure O(n) cost. An ANN index will deliver O(log n), which at N=1000 means roughly 100x algorithmic improvement on the dominant bottleneck.

1.8 KiB Raw Permalink Blame History

Guidance Performance Benchmarks

Scripts

Running

Findings (Phase 1, 2026-05-22)

1.8 KiB

Raw Permalink Blame History