proffesor-for-testing__agentic-qe/docs/guides/PERFORMANCE-TESTING.md

Performance Testing Guide

Learn how to test performance, detect bottlenecks, and validate system resilience with load testing and chaos engineering.

Overview

AQE's performance testing capabilities help you:

• Load test your applications with k6, JMeter, and custom tools
• Detect bottlenecks automatically using AI analysis
• Monitor performance trends over time
• Run chaos tests to validate resilience
• Generate performance reports with actionable insights

Key Features:

• Multi-tool support (k6, JMeter, Artillery, custom)
• Real-time performance monitoring
• Bottleneck detection with AI
• Chaos engineering scenarios
• Performance regression detection

Basic Performance Testing

Run Performance Benchmark

aqe benchmark

What it does:

1. Discovers performance-critical endpoints
2. Generates load test scenarios
3. Executes tests with increasing load
4. Monitors response times and throughput
5. Identifies bottlenecks automatically

Output:

⚡ Running Performance Benchmark...

Test Configuration:
  Duration: 60s
  Virtual Users: 1 → 100 (ramp-up)
  Tool: k6

Running tests:
  [00:15] 25 VUs  | Avg: 45ms  | P95: 120ms  | P99: 250ms  | RPS: 250
  [00:30] 50 VUs  | Avg: 52ms  | P95: 145ms  | P99: 310ms  | RPS: 480
  [00:45] 75 VUs  | Avg: 68ms  | P95: 180ms  | P99: 420ms  | RPS: 650
  [00:60] 100 VUs | Avg: 95ms  | P95: 280ms  | P99: 650ms  | RPS: 750

📊 Performance Results:
   Max Throughput: 750 RPS
   Avg Response Time: 95ms
   P95 Response Time: 280ms
   P99 Response Time: 650ms
   Error Rate: 0.2%

✅ Performance benchmark completed!

Load Testing

Using k6 (Recommended)

k6 is a modern load testing tool with excellent scripting capabilities.

Install k6:

npm install -g k6

Generate k6 test script:

aqe generate --type performance --framework k6 --target http://localhost:3000/api

Generated script:

// tests/performance/api-load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 20 },  // Ramp-up to 20 users
    { duration: '1m', target: 50 },   // Stay at 50 users
    { duration: '30s', target: 0 },   // Ramp-down to 0
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95% requests < 500ms
    http_req_failed: ['rate<0.01'],   // Error rate < 1%
  },
};

export default function () {
  const res = http.get('http://localhost:3000/api/users');

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });

  sleep(1);
}

Run k6 test:

k6 run tests/performance/api-load-test.js

Using JMeter

JMeter is a Java-based load testing tool with GUI.

Generate JMeter test plan:

aqe generate --type performance --framework jmeter --target http://localhost:3000/api

Generated: tests/performance/api-load-test.jmx

Run JMeter test:

jmeter -n -t tests/performance/api-load-test.jmx -l results.jtl

Using Artillery

Artillery is a modern, developer-friendly load testing tool.

Install Artillery:

npm install -g artillery

Generate Artillery test:

aqe generate --type performance --framework artillery --target http://localhost:3000/api

Generated script:

# tests/performance/api-load-test.yml
config:
  target: 'http://localhost:3000'
  phases:
    - duration: 60
      arrivalRate: 10
      rampTo: 50
  defaults:
    headers:
      Content-Type: 'application/json'

scenarios:
  - name: 'Load test API'
    flow:
      - get:
          url: '/api/users'
          expect:
            - statusCode: 200
            - contentType: json

Run Artillery test:

artillery run tests/performance/api-load-test.yml

Bottleneck Detection

Auto-Detect Bottlenecks

aqe analyze bottlenecks

What it analyzes:

1. Database query performance
2. API endpoint response times
3. Memory usage patterns
4. CPU utilization
5. Network latency
6. External service dependencies

Output:

🔍 Analyzing Performance Bottlenecks...

Bottlenecks Detected (3):

CRITICAL:
  1. Database Query: getUserOrders()
     └─> Location: src/services/order-service.ts:45
     └─> Issue: N+1 query problem
     └─> Impact: 850ms avg query time
     └─> Recommendation: Use JOIN or eager loading

HIGH:
  2. API Endpoint: POST /api/orders
     └─> Response Time: P95 = 1,200ms
     └─> Issue: Synchronous external API call
     └─> Impact: Blocking request processing
     └─> Recommendation: Use async queue or background job

MEDIUM:
  3. Memory Leak: WebSocket connections
     └─> Location: src/services/websocket-service.ts:78
     └─> Issue: Connections not properly closed
     └─> Impact: Memory grows 50MB/hour
     └─> Recommendation: Add connection cleanup

💡 Recommendations:
   1. Fix N+1 query (expected 90% improvement)
   2. Move external API calls to background
   3. Implement WebSocket cleanup

Analyze Specific Component

aqe analyze bottlenecks --component api-endpoint

Components:

• api-endpoint - API response times
• database - Database queries
• memory - Memory usage
• cpu - CPU utilization

Performance Metrics

Key Metrics Tracked

Response Time Metrics:

• Average (mean) - Typical response time
• P50 (median) - Middle value
• P95 - 95th percentile (most users' experience)
• P99 - 99th percentile (worst-case)
• Max - Slowest response

Throughput Metrics:

• RPS - Requests per second
• Concurrent users - Active users at a time
• Total requests - Cumulative requests

Error Metrics:

• Error rate - Percentage of failed requests
• Timeout rate - Percentage of timeouts
• 5xx errors - Server errors

Resource Metrics:

• CPU usage - Processor utilization
• Memory usage - RAM consumption
• Disk I/O - Read/write operations
• Network bandwidth - Data transfer

Chaos Engineering

What is Chaos Engineering?

Chaos engineering validates that your system can handle failures gracefully by intentionally injecting faults.

Common scenarios:

• Network latency
• Service failures
• Resource exhaustion
• Network partitions

Run Chaos Scenario

aqe chaos latency --duration 60 --intensity medium

What happens:

1. Establishes baseline metrics
2. Injects latency (150ms) into network calls
3. Monitors system behavior
4. Tracks error rates and response times
5. Verifies recovery after chaos stops

Output:

💥 Running chaos test: latency
   Duration: 60s
   Intensity: medium
   Target: all services

[00:00] ✓ Baseline established
        └─> Avg latency: 45ms, P95: 120ms, P99: 250ms

[00:10] ⚡ Injecting latency (150ms)
        └─> Current latency: 195ms, P95: 270ms, P99: 420ms

[00:30] ⚡ Latency active (150ms)
        └─> Current latency: 198ms, P95: 285ms, P99: 445ms
        ⚠️  3 timeouts detected

[00:60] ✓ Chaos stopped, monitoring recovery
        └─> Current latency: 52ms, P95: 130ms, P99: 260ms

[01:30] ✓ Recovery complete
        └─> Final latency: 46ms, P95: 122ms, P99: 255ms

📊 Chaos Test Results:
   Impact:
   • Timeouts: 3 (0.5% of requests)
   • Errors: 0
   • Avg latency increase: +153ms

   Recovery:
   • Recovery time: 30s
   • Data loss: None
   • Final status: ✅ Healthy

✅ Chaos test completed! System resilient.

Chaos Scenarios

1. Latency Injection

Tests system behavior under network delays.

# Low latency (50ms)
aqe chaos latency --intensity low --duration 60

# High latency (500ms)
aqe chaos latency --intensity high --duration 120

Use cases:

• Test timeout handling
• Validate retry logic
• Check user experience under slow networks

2. Service Failures

Simulates random service failures.

aqe chaos failure --target api-gateway --duration 120 --intensity high

Use cases:

• Test circuit breakers
• Validate fallback mechanisms
• Check error handling

3. Resource Exhaustion

Simulates CPU, memory, or disk pressure.

aqe chaos resource-exhaustion --intensity high --duration 180

Use cases:

• Test resource limits
• Validate auto-scaling
• Check graceful degradation

4. Network Partition

Simulates network splits between services.

aqe chaos network-partition --target database --duration 120

Use cases:

• Test distributed system behavior
• Validate data consistency
• Check partition tolerance

Performance Benchmarking

Create Baseline

aqe benchmark --baseline

Output: performance-baseline.json

Compare Against Baseline

aqe benchmark --baseline performance-baseline.json

Output:

📊 Performance Comparison:

Metric            Baseline   Current    Change
────────────────────────────────────────────────
Avg Response      45ms       52ms       +15.6% ⚠️
P95 Response      120ms      145ms      +20.8% ⚠️
P99 Response      250ms      310ms      +24.0% ⚠️
Max Throughput    750 RPS    680 RPS    -9.3% ⚠️
Error Rate        0.1%       0.3%       +200% ⚠️

⚠️  PERFORMANCE REGRESSION DETECTED

Degraded Areas:
  1. Average response time increased 15.6%
  2. P95 response time increased 20.8%
  3. Max throughput decreased 9.3%

💡 Recommendations:
   - Run bottleneck analysis
   - Check for new code introduced
   - Review database query performance

Practical Examples

Example 1: API Load Test

Scenario: Test REST API with 1000 concurrent users

# Generate test
aqe generate --type performance --framework k6 --target http://api.example.com

# Customize load
# Edit generated script to set stages:
# { duration: '2m', target: 1000 }

# Run test
k6 run tests/performance/api-load-test.js

# Analyze results
aqe analyze bottlenecks --component api-endpoint

Example 2: Database Performance

Scenario: Find slow database queries

# Run tests with database profiling
aqe run --coverage

# Analyze bottlenecks
aqe analyze bottlenecks --component database

# Fix N+1 queries and re-test
# (fix code)
aqe benchmark --baseline db-baseline.json

Example 3: Resilience Testing

Scenario: Validate system handles failures

# Test latency resilience
aqe chaos latency --duration 300 --intensity high

# Test failure resilience
aqe chaos failure --duration 300 --intensity medium

# Test resource limits
aqe chaos resource-exhaustion --duration 180

# Generate chaos report
aqe report chaos --format html --output chaos-report.html

CI/CD Integration

Performance Testing in CI

# .github/workflows/performance.yml
name: Performance Tests

on: [push]

jobs:
  performance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Start Application
        run: |
          npm install
          npm start &
          sleep 10

      - name: Run Performance Tests
        run: aqe benchmark --baseline performance-baseline.json

      - name: Check for Regressions
        run: |
          if [ $? -ne 0 ]; then
            echo "Performance regression detected!"
            exit 1
          fi

      - name: Upload Results
        uses: actions/upload-artifact@v3
        with:
          name: performance-report
          path: performance-report.html

Troubleshooting

High Response Times

Problem: Slow API responses

Solutions:

# Detect bottlenecks
aqe analyze bottlenecks --component api-endpoint

# Check database queries
aqe analyze bottlenecks --component database

# Profile memory usage
aqe analyze bottlenecks --component memory

Performance Regressions

Problem: Performance worse than baseline

Solutions:

1. Run bottleneck analysis
2. Check recent code changes
3. Review database query plans
4. Validate caching works

Flaky Performance Tests

Problem: Inconsistent results

Solutions:

1. Increase test duration for stable averages
2. Use percentiles (P95, P99) instead of averages
3. Run multiple iterations and average
4. Isolate test environment

Best Practices

1. Establish baselines early - Measure before optimizing
2. Test realistic scenarios - Use production-like data and load
3. Monitor continuously - Track performance over time
4. Set SLOs - Define service level objectives (e.g., P95 < 200ms)
5. Test resilience - Use chaos engineering regularly
6. Automate testing - Include in CI/CD pipeline
7. Profile before optimizing - Find bottlenecks first

Next Steps

• Set up quality gates → QUALITY-GATES.md
• Improve test coverage → COVERAGE-ANALYSIS.md
• Optimize test suite → See aqe optimize command

Related Commands

aqe benchmark --help           # Full benchmark options
aqe chaos --help              # Chaos testing scenarios
aqe analyze bottlenecks --help # Bottleneck detection
aqe generate --type performance # Generate perf tests