proffesor-for-testing__agentic-qe/tests/benchmarks

Agentic QE v3 Performance Benchmarks

This directory contains performance benchmark suites for validating system performance metrics.

Benchmark Suites

ADR-051 Agentic-Flow Integration Benchmarks

File: agentic-flow-performance.bench.ts

Validates the success metrics defined in ADR-051 for the Agentic-Flow integration:

Component	Target Metric	Benchmark Coverage
Agent Booster	<5ms mechanical edit	✅ 10 transform types
Model Router	<10ms routing decision	✅ 5 complexity levels
ONNX Embeddings	<50ms embedding generation	✅ 9 scenarios
ReasoningBank	<20ms pattern retrieval	✅ 6 operations
Cross-Session Hit Rate	50% pattern hits	✅ Validated
Pattern Retention	100% retention	✅ Validated

Total Scenarios: 41 benchmarks

Run Benchmarks:

# Run all agentic-flow benchmarks
npx vitest bench tests/benchmarks/agentic-flow-performance.bench.ts --run

# Run with detailed output
npx vitest bench tests/benchmarks/agentic-flow-performance.bench.ts --reporter=verbose --run

# Run specific benchmark group
npx vitest bench tests/benchmarks/agentic-flow-performance.bench.ts -t "Agent Booster"

Benchmark Results

Latest benchmark results are documented in:

📊 Performance Report: /workspaces/agentic-qe/docs/reports/adr-051-performance-validation.md

Key Results (Latest Run)

✅ All ADR-051 targets exceeded:

• Agent Booster: 0.2-19μs (1,000-10,000x faster than 5ms target)
• Model Router: 0.14-0.33μs (30,000-71,000x faster than 10ms target)
• ONNX Embeddings: 86-110μs (450-580x faster than 50ms target)
• ReasoningBank: 0.08-8.9μs (2,000-250,000x faster than 20ms target)
• Cross-Session Hit Rate: ~50% (target met)
• Pattern Retention: 100% (target met)

Code Intelligence Token Reduction Benchmarks

File: code-intelligence-token-reduction.bench.ts

Validates O(log n) sublinear coverage analysis performance with knowledge graph-based code understanding.

Target Metric: 80% token reduction vs full file transmission

Run Benchmarks:

npm run benchmark:token-reduction

Writing New Benchmarks

Use Vitest's bench() API for performance testing:

import { bench, describe } from 'vitest';

describe('My Performance Test', () => {
  bench('operation name', () => {
    // Code to benchmark
    myFunction();
  });

  bench('operation with setup', async () => {
    // Setup is included in measurement
    const data = prepareData();
    await processData(data);
  });
});

Best Practices

1. Use descriptive names: Clearly indicate what is being benchmarked
2. Include units: Specify data sizes (e.g., "10 lines", "1000 vectors")
3. Target metrics: Document expected performance in comments
4. Avoid I/O: Keep benchmarks CPU-bound for reproducibility
5. Warm-up: Vitest automatically runs warmup iterations
6. Sample size: Vitest automatically determines optimal sample count

Benchmark Configuration

Benchmarks are configured in vitest.config.ts:

export default defineConfig({
  test: {
    benchmark: {
      include: ['**/*.bench.ts'],
    },
  },
});

Continuous Integration

Benchmarks should be run:

1. Before release: Validate performance targets are met
2. After optimization: Verify improvements
3. Regression detection: Compare against baseline

Performance Baselines

Store baseline results in /docs/reports/ for comparison:

# Generate baseline
npx vitest bench --reporter=json > baseline.json

# Compare against baseline
npx vitest bench --reporter=json > current.json
# (Use tools like hyperfine or custom scripts for comparison)

Performance Targets

ADR-051 Success Metrics

Metric	Target	Status
Agent Booster latency	<5ms	✅ EXCEEDED (0.2-19μs)
Model Router decision	<10ms	✅ EXCEEDED (0.14μs)
ONNX embedding generation	<50ms	✅ EXCEEDED (86-110μs)
ReasoningBank retrieval	<20ms	✅ EXCEEDED (8.9μs)
Cross-session hit rate	50%	✅ MET (~50%)
Pattern retention rate	100%	✅ MET (100%)

ADR-042 Token Reduction

Metric	Target	Status
Token reduction	80%	✅ VALIDATED
Semantic accuracy	>95%	✅ VALIDATED

Troubleshooting

Benchmarks Running Slow

• Check system load: top or htop
• Close other applications
• Disable browser/IDE heavy processes
• Use dedicated benchmark environment

Inconsistent Results

• Run multiple times and average
• Check for background processes
• Ensure stable CPU frequency (disable turbo boost)
• Use --no-parallel flag

Memory Issues

• Reduce sample size in benchmarks
• Split large benchmarks into smaller groups
• Use --max-old-space-size=4096 Node flag

Related Documentation

• ADR-051: Agentic-Flow Integration
• ADR-042: Token Usage Optimization
• Performance Report
• Vitest Benchmark API

Contributing

When adding new benchmarks:

1. Follow naming convention: [feature]-[aspect].bench.ts
2. Document target metrics in comments
3. Include baseline comparisons
4. Update this README with new benchmark info
5. Add results to /docs/reports/

Questions?

Contact the QE Performance team or open an issue in the repository.