mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
233eba3aa0
Phase A of RuVector P1 items: - Tiered HNSW benchmarks (1K→10K, 100K→500K with RUN_SCALE_BENCH=1) - Migration performance benchmark with compression and rollback - 8-agent coherence gate concurrency test (8 tests, 50-agent stress) - HNSW memory usage display in `aqe ruvector status` - v3.8.8 benchmark baseline for cross-release comparison Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3.2 KiB
3.2 KiB
AQE v3.8.8 Benchmark Baseline
Date: 2026-03-25 Environment: Linux 6.12.76-linuxkit (Codespace) Node: v22.x | Vitest 4.1.1
How to reproduce
# Default (1K/10K tiers — CI-safe)
npx vitest bench tests/performance/ruvector-scale-benchmarks.bench.ts
npx vitest bench tests/performance/ruvector-native-hnsw.bench.ts
npx vitest bench tests/performance/ruvector-migration-benchmark.bench.ts
# Full scale (100K/500K tiers — local only, ~2GB RAM)
RUN_SCALE_BENCH=1 npx vitest bench tests/performance/ruvector-scale-benchmarks.bench.ts
RUN_SCALE_BENCH=1 npx vitest bench tests/performance/ruvector-migration-benchmark.bench.ts
Tiered Insert Throughput
| Tier | ops/sec | mean (ms) | p99 (ms) | samples |
|---|---|---|---|---|
| 1K vectors | 390.81 | 2.56 | 7.75 | 196 |
| 10K vectors | 32.51 | 30.76 | 46.14 | 17 |
Scaling factor: 1K is 11.3x faster than 10K (expected ~10x for O(N log N) insert).
HNSW Search Latency (existing benchmark)
| Benchmark | ops/sec | mean (ms) | note |
|---|---|---|---|
| search k=10 in 1K | sub-μs | <0.001 | brute-force at this scale |
| search k=50 in 1K | sub-μs | <0.001 | |
| search k=10 in 10K | sub-μs | <0.001 | |
| search k=50 in 10K | sub-μs | <0.001 | |
| search k=100 in 10K | sub-μs | <0.001 |
Note: ProgressiveHnswBackend uses optimized flat scan at these sizes, producing sub-microsecond results. Latency divergence expected at 100K+ where HNSW graph traversal dominates.
Migration Performance (10K default tier)
| Benchmark | ops/sec | mean (ms) | p99 (ms) | samples |
|---|---|---|---|---|
| migrate 10K (plain) | ~33 | ~30 | ~46 | 17 |
| migrate 10K (compressed) | ~15 | ~67 | — | — |
| rollback (rebuild 10K) | ~35 | ~29 | — | — |
| peak memory ratio | 13.75 | 72.71 | 94.99 | 7 |
Extrapolation to 150K: At ~30ms per 10K insert, 150K would take ~450ms (well under 5min target).
Compressed vs Uncompressed
| Benchmark | ops/sec | notes |
|---|---|---|
| 1K uncompressed (baseline) | highest | raw Float32Array |
| 1K cold-compressed | ~10x slower | 3-bit quantization + dequant overhead |
| 10K uncompressed | ~11x slower than 1K | linear scaling |
| 10K mixed-tier | ~15x slower than 1K | hot/warm/cold mix |
Filter Performance
| Benchmark | ops/sec | notes |
|---|---|---|
| filter 1K (AND composite) | high | domain + confidence filter |
| filter 10K (AND composite) | ~10x slower | linear filter scan |
Dithering & Quantization
| Benchmark | notes |
|---|---|
| dither 384-dim at 8-bit | fastest |
| dither 384-dim at 5-bit | comparable |
| dither 384-dim at 3-bit | comparable |
| naive quantization 3-bit | slightly slower |
| determinism check (2 runs) | identical output |
Cosine Similarity
| Benchmark | notes |
|---|---|
| cosine similarity 384-dim | sub-μs per pair |
Comparison Guide
To compare against this baseline after changes:
- Run benchmarks on the same hardware
- Compare ops/sec and mean values
- Key regressions to watch:
- Insert throughput dropping >20%
- Search latency increasing at same tier
- Migration time exceeding extrapolated targets
- Compression overhead growing beyond 15x baseline