proffesor-for-testing__agentic-qe/docs/guides/TEST-EXECUTION.md

Test Execution Guide

Master parallel test execution, retry logic, and real-time monitoring with AQE.

Overview

AQE's test executor orchestrates your test suite with:

• Parallel execution - Run tests simultaneously across CPU cores
• Smart retry logic - Auto-retry flaky tests
• Real-time reporting - Live progress and coverage tracking
• Framework agnostic - Works with Jest, Mocha, Cypress, Playwright
• CI/CD optimized - Fast, reliable, automation-friendly

Basic Execution

Run All Tests

aqe run

What happens:

1. Auto-detects testing framework (Jest, Mocha, etc.)
2. Discovers all test files in ./tests
3. Spawns parallel workers (based on CPU cores)
4. Executes tests with coverage collection
5. Reports results in real-time

Output:

🧪 Executing test suite: ./tests
📦 Auto-detected framework: jest
⚡ Using 8 parallel workers
🚀 Starting test execution...

Running tests:
  ✓ tests/unit/user-service.test.ts (12 tests, 2.3s)
  ✓ tests/unit/auth-service.test.ts (8 tests, 1.8s)
  ✓ tests/integration/api.test.ts (15 tests, 4.1s)

📊 Execution Summary:
   Total: 120
   Passed: 120
   Failed: 0
   Coverage: 94.5%
   Duration: 12.3s

✅ All tests passed!

Run Specific Test Suite

aqe run tests/integration

Runs only tests in the tests/integration directory.

Run Single Test File

aqe run tests/unit/user-service.test.ts

Parallel Execution

Auto-Scaling (Recommended)

aqe run --parallel auto

Automatically uses optimal worker count based on CPU cores.

Fixed Worker Count

aqe run --parallel 4

Uses exactly 4 parallel workers.

When to use:

• 2-4 workers - Limited CPU, small test suites
• 4-8 workers - Standard development machines
• 8-16 workers - CI/CD servers, large test suites
• auto - Let AQE decide (recommended)

Performance Impact

Example: 120 tests

Workers	Duration	Speedup
1	98s	1x
2	52s	1.9x
4	28s	3.5x
8	15s	6.5x

Diminishing returns after 8 workers for most suites.

Retry Logic

Auto-Retry Flaky Tests

aqe run --retry 2

How it works:

1. Test fails on first run
2. AQE automatically retries up to 2 more times
3. Marks test as flaky if it eventually passes
4. Records flaky test for future optimization

Output with retries:

Running tests:
  ✓ tests/unit/user-service.test.ts (12 tests)
  ⚠ tests/integration/payment.test.ts (5 tests, 1 flaky)
    └─> 'processPayment should handle timeout' passed on retry 2

Disable Retries

aqe run --retry 0

Fail immediately on first test failure (useful for debugging).

Coverage Collection

Enable Coverage (Default)

aqe run --coverage

Collects:

• Line coverage
• Branch coverage
• Function coverage
• Statement coverage

Output:

📊 Coverage Report:
   Lines: 94.5% (1,260/1,334)
   Branches: 91.2% (420/461)
   Functions: 96.8% (122/126)
   Statements: 94.3% (1,248/1,324)

Coverage file: ./coverage/coverage-final.json

Disable Coverage

aqe run --no-coverage

Why disable?

• Faster execution (30-50% speed improvement)
• Debugging test failures
• CI/CD optimization (when coverage not needed)

Test Filtering

Run Tests Matching Pattern

aqe run --filter "user.*test"

Runs only tests matching the regex pattern user.*test.

Examples:

# Run all user-related tests
aqe run --filter "user"

# Run only integration tests
aqe run --filter "integration"

# Run specific test suite
aqe run --filter "payment.*integration"

Watch Mode (TDD)

Enable Watch Mode

aqe run --watch

What happens:

1. Runs tests initially
2. Watches for file changes
3. Re-runs affected tests automatically
4. Shows instant feedback

Perfect for Test-Driven Development (TDD):

🔍 Watch mode active...
Watching for changes...

[File changed: src/services/user-service.ts]
Re-running affected tests...
  ✓ tests/unit/user-service.test.ts (12 tests, 1.2s)

[File changed: tests/unit/user-service.test.ts]
Re-running tests...
  ✗ tests/unit/user-service.test.ts (12 tests, 1 failed)
    └─> 'createUser should validate email' failed

Press Ctrl+C to exit

Watch Mode with Bail

aqe run --watch --bail

Stops on first failure in watch mode (faster feedback).

CI/CD Integration

Optimized for CI

aqe run --reporter junit --bail --parallel 8

CI/CD best practices:

• Use JUnit reporter for pipeline integration
• Enable bail for fast failure feedback
• Disable coverage if not needed (faster)
• Use fixed parallel count (predictable)

Example GitHub Actions:

- name: Run Tests
  run: |
    aqe run --reporter junit --bail --parallel 8

Example GitLab CI:

test:
  script:
    - aqe run --reporter junit --bail
  artifacts:
    reports:
      junit: test-results/junit.xml

Framework-Specific Execution

Jest

aqe run --framework jest

Auto-detected if jest.config.js exists.

Mocha

aqe run --framework mocha

Auto-detected if .mocharc.json exists.

Cypress

aqe run --framework cypress

Requires Cypress installed and configured.

Playwright

aqe run --framework playwright

Runs Playwright tests in parallel.

Reporters

Default Reporter

aqe run --reporter default

Human-readable console output.

JSON Reporter

aqe run --reporter json

Outputs JSON for programmatic parsing.

Output: test-results.json

{
  "numTotalTests": 120,
  "numPassedTests": 120,
  "numFailedTests": 0,
  "coverage": { "total": { "pct": 94.5 } },
  "duration": 12300
}

HTML Reporter

aqe run --reporter html

Generates interactive HTML report.

Output: test-results/index.html

JUnit Reporter

aqe run --reporter junit

Generates JUnit XML (CI/CD standard).

Output: test-results/junit.xml

Timeouts

Set Test Timeout

aqe run --timeout 60000

Sets timeout to 60 seconds (default: 30 seconds).

When to increase:

• Integration tests with external APIs
• E2E tests with slow UI interactions
• Database operations

When to decrease:

• Fast unit tests
• Catching infinite loops quickly

Bail Strategies

Stop on First Failure

aqe run --bail

Use cases:

• Fast feedback in TDD
• Pre-commit hooks
• Debugging specific failures

Run All Tests

aqe run --no-bail

Use cases:

• Full test suite validation
• CI/CD (see all failures)
• Coverage collection

Real-Time Monitoring

Live Progress

During execution, AQE shows:

🚀 Progress: 45/120 tests (37.5%)
   ✓ Passed: 44
   ✗ Failed: 1
   ⏱  Elapsed: 8.2s
   📊 Coverage: 42.3% (partial)

Agent Coordination

Tests run with coordination:

[10:05:12] qe-test-executor: Starting suite execution
[10:05:15] qe-coverage-analyzer: Tracking coverage in real-time
[10:05:23] qe-test-executor: Test failure detected, retrying...
[10:05:25] qe-test-executor: Retry successful, marking as flaky

Advanced Examples

Example 1: Full CI/CD Pipeline

#!/bin/bash
# ci-test.sh

# Run tests with full reporting
aqe run \
  --parallel 8 \
  --reporter junit \
  --coverage \
  --bail \
  --timeout 45000

# Check exit code
if [ $? -eq 0 ]; then
  echo "✅ All tests passed"
  exit 0
else
  echo "❌ Tests failed"
  exit 1
fi

Example 2: TDD Workflow

# Terminal 1: Watch mode
aqe run --watch --bail --filter "user-service"

# Terminal 2: Edit code
vim src/services/user-service.ts

# Tests re-run automatically on save

Example 3: Performance Testing

# Run with timing analysis
aqe run --parallel 8 --reporter json

# Analyze slow tests
cat test-results.json | jq '.tests[] | select(.duration > 1000)'

Troubleshooting

Tests Timeout

Problem: Tests exceed timeout

Solutions:

# Increase timeout
aqe run --timeout 60000

# Check for async issues in tests
# Ensure promises are awaited
# Verify no hanging connections

Parallel Execution Issues

Problem: Tests fail only in parallel mode

Solutions:

# Run serially to debug
aqe run --parallel 1

# Check for shared state issues
# Ensure tests are isolated
# Use unique test data per test

Coverage Lower Than Expected

Problem: Coverage drops during execution

Solutions:

# Generate missing tests
aqe analyze gaps
aqe generate src/services/uncovered-file.ts

# Re-run with coverage
aqe run --coverage

Flaky Tests

Problem: Tests fail randomly

Solutions:

# Enable retry to identify
aqe run --retry 3

# Analyze flaky tests
aqe analyze quality --path tests/

# Fix or remove flaky tests
aqe optimize flakiness --dry-run

Performance Tips

1. Use parallel execution - Always use --parallel for faster runs
2. Disable coverage for debugging - Use --no-coverage when not needed
3. Filter tests during development - Use --filter to run subset
4. Enable watch mode for TDD - Use --watch for instant feedback
5. Optimize worker count - Experiment with worker count for your machine

Next Steps

• Analyze your coverage → COVERAGE-ANALYSIS.md
• Set up quality gates → QUALITY-GATES.md
• Optimize your suite → See aqe optimize command

Related Commands

aqe run --help          # Full command reference
aqe status              # Check executor agent status
aqe analyze coverage    # Analyze test coverage
aqe optimize suite      # Optimize test execution