proffesor-for-testing__agentic-qe/docs/agents/test-executor-modes.md

TestExecutorAgent: Real vs Simulation Modes

Overview

The TestExecutorAgent now supports two distinct modes of operation:

1. REAL MODE (default): Executes actual tests via Jest/Mocha/Playwright/Cypress using child_process
2. SIMULATION MODE (demo only): Simulates test results with random pass/fail for demonstration purposes

Configuration

Real Mode (Default - Recommended for Production)

const agent = new TestExecutorAgent({
  // ... other config
  simulationMode: false, // Default: REAL execution
  workingDir: process.cwd() // Default: current directory
});

Behavior:

• Executes tests via TestFrameworkExecutor using child_process.spawn
• Spawns actual test runners (Jest, Mocha, Playwright, Cypress)
• Parses JSON output from test frameworks
• Returns real test results with actual pass/fail/skip counts
• Includes real coverage data when enabled
• Provides real error messages and stack traces

Logs:

[TestExecutor] ✅ REAL EXECUTION MODE - Tests will be executed via test frameworks
TestExecutorAgent test-executor-123 initializing in REAL mode with frameworks: jest, mocha, cypress, playwright

Simulation Mode (Demo/Testing Only)

const agent = new TestExecutorAgent({
  // ... other config
  simulationMode: true // Enable simulation
});

Behavior:

• Does NOT execute real tests
• Simulates test execution with delays based on test type
• Generates random pass/fail results (90% pass rate)
• Returns simulated coverage data
• Error messages are prefixed with [SIMULATED]

Logs:

[TestExecutor] ⚠️  SIMULATION MODE ENABLED - Tests will NOT be executed for real
TestExecutorAgent test-executor-123 initializing in SIMULATION mode with frameworks: jest, mocha, cypress, playwright

Feature Comparison

Feature	Real Mode	Simulation Mode
Executes actual tests	✅ Yes	❌ No
Uses test frameworks	✅ Yes (child_process)	❌ No
Real pass/fail results	✅ Yes	❌ Random (90% pass)
Real coverage data	✅ Yes (when enabled)	❌ Simulated
Framework validation	✅ Yes (checks package.json)	❌ Skipped
Test discovery	✅ Real (glob search)	❌ Random counts
Exit codes	✅ Real	❌ Simulated
Error messages	✅ Real stack traces	⚠️ Prefixed with [SIMULATED]

Implementation Details

Real Test Execution Flow

// 1. Initialize TestFrameworkExecutor (lazy loaded)
const { TestFrameworkExecutor } = await import('../utils/TestFrameworkExecutor.js');
this.testFrameworkExecutor = new TestFrameworkExecutor();

// 2. Select framework based on test type
const framework = this.selectFramework(test); // e.g., 'jest', 'mocha'

// 3. Build test pattern
const testPattern = this.buildTestPattern(test); // e.g., '**/*.test.js'

// 4. Execute with real framework
const result = await this.testFrameworkExecutor.execute({
  framework: 'jest',
  testPattern: '**/*.test.js',
  workingDir: '/path/to/tests',
  timeout: 300000,
  coverage: false,
  environment: 'test'
});

// 5. Parse results from framework JSON output
// Returns: { status, totalTests, passedTests, failedTests, tests[], coverage }

Simulated Test Execution Flow

// 1. Estimate duration based on test type
const duration = this.estimateTestDuration(test); // 100-5000ms

// 2. Simulate execution delay
await new Promise(resolve => setTimeout(resolve, duration));

// 3. Generate random result
const success = SecureRandom.randomFloat() > 0.1; // 90% pass rate

// 4. Return simulated result with [SIMULATED] prefix
return {
  status: success ? 'passed' : 'failed',
  errors: success ? [] : ['[SIMULATED] Test assertion failed'],
  metadata: { simulated: true }
};

Test Discovery

Real Mode

// Uses glob to find actual test files
const files = await glob('**/*.test.js', {
  cwd: workingDir,
  ignore: ['**/node_modules/**']
});

// Returns: { unitTests: 45, integrationTests: 12, e2eTests: 8, simulated: false }

Simulation Mode

// Returns random counts
return {
  unitTests: Math.floor(SecureRandom.randomFloat() * 50) + 10,
  integrationTests: Math.floor(SecureRandom.randomFloat() * 20) + 5,
  summary: '[SIMULATED] Discovered 65 tests...',
  simulated: true
};

Result Metadata

Test results include metadata to identify execution mode:

Real Mode Metadata

{
  "framework": "jest",
  "retries": 0,
  "totalTests": 45,
  "passedTests": 43,
  "failedTests": 2,
  "exitCode": 1
}

Simulation Mode Metadata

{
  "framework": "jest",
  "retries": 0,
  "simulated": true
}

Usage Examples

Production Use (Real Mode)

import { TestExecutorAgent } from './agents/TestExecutorAgent';

const executor = new TestExecutorAgent({
  frameworks: ['jest', 'mocha'],
  maxParallelTests: 8,
  timeout: 300000,
  retryAttempts: 3,
  simulationMode: false, // REAL execution (default)
  workingDir: '/path/to/project'
});

const result = await executor.execute({
  type: 'parallel-test-execution',
  payload: {
    testSuite: {
      id: 'test-suite-1',
      tests: [/* test objects */]
    }
  }
});

// result.optimizationApplied: true
// result.results: [/* real test results */]

Demo/Testing Use (Simulation Mode)

const simulator = new TestExecutorAgent({
  frameworks: ['jest'],
  simulationMode: true, // SIMULATION mode
  maxParallelTests: 8
});

const result = await simulator.execute({
  type: 'test-discovery',
  payload: {
    searchPath: './tests'
  }
});

// result.simulated: true
// result.summary: '[SIMULATED] Discovered 65 tests...'

Environment Variables

The agent respects the following environment variables:

• NODE_ENV=test: Affects execution environment
• AQE_USE_MOCK_AGENTDB=true: Uses mock AgentDB adapter (affects logging, not execution)

Migration Guide

Before (v1.7.0 and earlier)

All test execution was simulated with no way to run real tests:

// OLD: Always simulated
const agent = new TestExecutorAgent({
  frameworks: ['jest']
});
// Always returned random results

After (v1.8.0+)

Real execution by default, simulation mode optional:

// NEW: Real execution by default
const agent = new TestExecutorAgent({
  frameworks: ['jest'],
  simulationMode: false // Explicit (but default)
});
// Returns real test results

// NEW: Simulation mode available for demos
const simulator = new TestExecutorAgent({
  frameworks: ['jest'],
  simulationMode: true
});
// Returns simulated results with [SIMULATED] prefix

Best Practices

1. Production: Always use Real Mode (simulationMode: false or omit)
2. CI/CD: Verify simulationMode: false in agent configuration
3. Demos: Use Simulation Mode for presentations without requiring test infrastructure
4. Testing: Use Simulation Mode for unit tests of agent coordination logic
5. Validation: Check metadata.simulated flag to detect simulation mode results

Troubleshooting

"Framework 'jest' not found in package.json dependencies"

Cause: Real Mode requires test frameworks to be installed.

Solution:

npm install --save-dev jest  # or mocha, playwright, cypress

Tests return [SIMULATED] prefix in errors

Cause: Agent is running in Simulation Mode.

Solution: Check configuration and set simulationMode: false.

No test frameworks validated during initialization

Cause: Agent is in Simulation Mode.

Solution: Framework validation is skipped in Simulation Mode. Switch to Real Mode if validation is needed.

Related Files

• Implementation: /workspaces/agentic-qe-cf/src/agents/TestExecutorAgent.ts
• Test Framework Executor: /workspaces/agentic-qe-cf/src/utils/TestFrameworkExecutor.ts
• Issue: GitHub Issue #52

---

Version: 1.8.0 Last Updated: 2025-11-17 Status: Production Ready