proffesor-for-testing__agentic-qe/docs/implementation/rl-qlearning-coverage-integration.md

Q-Learning Integration with Coverage Analysis Domain

Summary

Successfully integrated the Q-Learning reinforcement learning algorithm with the coverage-analysis domain for intelligent test prioritization and coverage path optimization.

Architecture

1. Type Definitions

Added coverage-specific Q-Learning types to /v3/src/domains/coverage-analysis/interfaces.ts:

• CoverageQLState: State representation with 12 features for Q-Learning
• CoverageQLAction: Action types (generate-unit, generate-integration, prioritize, skip)
• CoverageQLPrediction: Prediction result with confidence and estimates
• QLPrioritizedTests: List of prioritized tests with metrics
• PrioritizedTest: Single test with priority, coverage gain, and duration

2. Coordinator Integration

Updated /v3/src/domains/coverage-analysis/coordinator.ts:

New Properties

• qLearning: QLearningAlgorithm - Q-Learning instance with coverage-specific config
• qlConfig - Configuration (stateSize: 12, hiddenLayers: [128, 64], doubleDQN: true)

New Methods

getQLRecommendations(gaps, limit?)

• Gets Q-Learning predictions for each coverage gap
• Prioritizes tests by Q-value (predicted reward)
• Returns estimated coverage gains and durations
• Sorts by priority descending

predictQL(gap)

• Converts coverage gap to Q-Learning state
• Gets prediction from Q-Learning model
• Returns action, confidence, value, reasoning, and estimates

trainQL(experience)

• Trains Q-Learning with execution results
• Persists model state every 10 episodes
• Uses coverage-specific reward signals

Helper Methods

State Representation (gapToQLState) Creates 12-dimensional feature vector:

1. Normalized risk score
2. Number of uncovered lines
3. Number of uncovered branches
4. Severity level
5. Estimated complexity
6. Change frequency
7. Business criticality
8. Line to branch ratio
9. Coverage potential
10. File complexity score
11. Test complexity
12. Execution cost

Coverage Gain Estimation

• generate-unit: 30% of base gain
• generate-integration: 50% of base gain
• prioritize: 70% of base gain
• skip: 0% gain

Test Duration Estimation

• Unit tests: 0.5s per line
• Integration tests: 1s per line
• E2E tests: 2.5s per line

3. Neural Network Configuration

Input Layer: 12 neurons (state features)
Hidden Layer 1: 128 neurons (ReLU activation)
Hidden Layer 2: 64 neurons (ReLU activation)
Output Layer: 4 neurons (Q-values for each action)

Training Parameters:

• Learning rate: 0.001
• Discount factor: 0.99
• Batch size: 32
• Replay buffer: 10,000 experiences
• Target network update: Every 50 episodes
• Double DQN: Enabled (reduces overestimation)

Integration Points

1. RL Suite Module

Located at /v3/src/integrations/rl-suite/:

• algorithms/q-learning.ts - Q-Learning with Deep Q-Network
• interfaces.ts - Core RL types and reward signals
• base-algorithm.ts - Abstract base class

2. Coverage Analysis Domain

Located at /v3/src/domains/coverage-analysis/:

• coordinator.ts - Main coordinator with Q-Learning integration
• interfaces.ts - Domain-specific types
• Services: coverage-analyzer, gap-detector, risk-scorer

3. Reward Signals

Uses COVERAGE_REWARDS from RL suite:

• coverage-gain (weight 0.6): Reward for improving coverage
• efficiency (weight 0.4): Reward for efficient test generation

Usage Examples

Basic Prediction

const gap: CoverageGap = {
  id: 'gap-1',
  file: 'src/services/auth.service.ts',
  lines: [10, 15, 20],
  branches: [5, 10],
  riskScore: 8.5,
  severity: 'high',
  recommendation: 'Test authentication logic',
};

const prediction = await coordinator.predictQL(gap);
// Returns: { action, confidence, value, reasoning, estimatedCoverageGain, estimatedTestCount }

Test Prioritization

const gaps: CoverageGap[] = [gap1, gap2, gap3];
const result = await coordinator.getQLRecommendations(gaps, 10);

if (result.success) {
  const { tests, totalEstimatedCoverageGain, totalEstimatedDuration } = result.value;
  // Tests sorted by priority (Q-value) descending
  tests.forEach(test => {
    console.log(`${test.filePath}: ${test.testType} - Priority: ${test.priority}`);
  });
}

Training with Experience

const experience: RLExperience = {
  state: { id: 'gap-1', features: [...] },
  action: { type: 'generate-unit', value: 'standard' },
  reward: 0.8,
  nextState: { id: 'gap-1-next', features: [...] },
  done: true,
};

await coordinator.trainQL(experience);

Test Results

All 10 integration tests passing:

✓ should convert coverage gap to Q-Learning state with correct features
✓ should generate different predictions for different gap types
✓ should prioritize tests based on Q-Learning predictions
✓ should respect limit parameter for recommendations
✓ should train Q-Learning with execution results
✓ should persist training state to memory
✓ should estimate different coverage gains for different actions
✓ should estimate longer duration for integration tests
✓ should map actions to appropriate test types
✓ should provide reasoning for predictions

Key Features

1. Intelligent Prioritization: Uses Q-Learning to prioritize tests based on learned policies
2. Coverage Optimization: Maximizes coverage gain while minimizing execution time
3. Adaptive Learning: Improves predictions over time with experience
4. Real Integration: Actually calls Q-Learning methods and uses predictions
5. Explainable: Provides reasoning and confidence for each prediction

Performance Characteristics

• State Encoding: 12 features normalized to [0, 1]
• Network Architecture: 128 -> 64 -> 4 (Deep Q-Network)
• Training: Online learning with experience replay
• Inference: Single forward pass through network (<1ms)
• Scalability: O(1) prediction per gap, O(n) for n gaps

Future Enhancements

1. Add more sophisticated state features (cyclomatic complexity, code churn)
2. Implement multi-objective optimization (coverage, speed, cost)
3. Add curriculum learning for faster convergence
4. Integrate with test-generation domain for closed-loop learning
5. Add transfer learning from pre-trained models

Files Modified

1. /v3/src/domains/coverage-analysis/interfaces.ts - Added Q-Learning types
2. /v3/src/domains/coverage-analysis/coordinator.ts - Integrated Q-Learning
3. /v3/src/domains/coverage-analysis/index.ts - Exported new types
4. /v3/tests/integration/coverage-analysis-qlearning-integration.test.ts - Integration tests

Related ADRs

• ADR-040: Agentic QE v3 RL Suite (9 algorithms including Q-Learning)
• ADR-003: Sublinear Algorithms for Coverage Analysis (HNSW indexing)