proffesor-for-testing__agentic-qe/docs/LEARNING-SYSTEM.md

Agent Learning System - Phase 2 (Milestone 2.2)

Overview

The Agent Learning System implements reinforcement learning for continuous performance improvement of AQE agents. It uses Q-learning algorithms to optimize task execution strategies and achieve a target of 20% performance improvement over 30 days.

Architecture

Components

1. LearningEngine - Core reinforcement learning implementation

   • Q-learning algorithm for strategy optimization
   • Experience replay and batch updates
   • Pattern recognition and failure detection
   • Automatic exploration/exploitation balancing

2. PerformanceTracker - Performance metrics tracking

   • Real-time performance monitoring
   • Baseline comparison and trend analysis
   • 20% improvement target tracking
   • Comprehensive reporting

3. ImprovementLoop - Continuous improvement automation

   • Automatic learning cycle execution
   • A/B testing framework
   • Failure pattern analysis
   • Strategy optimization

Features

Reinforcement Learning

Q-Learning Algorithm:

Q(s,a) = Q(s,a) + α * [r + γ * max(Q(s',a')) - Q(s,a)]

Where:

• α = learning rate (default: 0.1)
• γ = discount factor (default: 0.95)
• r = reward from task execution
• s,a = current state-action pair
• s',a' = next state-action pair

Reward Calculation:

• Success/failure: ±1.0
• Execution time: up to +0.5 (faster is better)
• Error penalty: -0.1 per error
• User feedback: ±2.0 (based on rating)
• Coverage bonus: up to +2.0 (above 80%)

Performance Tracking

Metrics Tracked:

• Tasks completed
• Success rate
• Average execution time
• Error rate
• User satisfaction
• Resource efficiency

Composite Performance Score:

score = successRate * 0.30 +
        userSatisfaction * 0.25 +
        normalizedTime * 0.20 +
        (1 - errorRate) * 0.15 +
        resourceEfficiency * 0.10

Continuous Improvement

Improvement Cycle (runs hourly):

1. Analyze current performance vs baseline
2. Identify failure patterns
3. Discover optimization opportunities
4. Update active A/B tests
5. Apply best-performing strategies

A/B Testing:

• Compare multiple strategies in parallel
• Statistical significance testing
• Automatic winner selection
• Strategy application

Usage

Basic Integration

import { LearningAgent } from './agents/LearningAgent';
import { SwarmMemoryManager } from './core/memory/SwarmMemoryManager';
import { EventBus } from './core/EventBus';

// Initialize memory and event bus
const memoryStore = new SwarmMemoryManager('./data/memory.db');
await memoryStore.initialize();

const eventBus = new EventBus();
await eventBus.initialize();

// Create learning agent
const agent = new LearningAgent({
  type: 'test-generator',
  capabilities: [
    { name: 'test-generation', version: '1.0.0', enabled: true }
  ],
  context: { projectRoot: './project' },
  memoryStore,
  eventBus,
  learningEnabled: true,
  learningRate: 0.1
});

// Initialize and start
await agent.initialize();

// Execute tasks - agent learns automatically
await agent.assignTask({
  id: 'task-1',
  type: 'generate-tests',
  description: 'Generate unit tests for UserService',
  requirements: {
    capabilities: ['test-generation']
  }
});

// Check learning status
const status = await agent.getLearningStatus();
console.log(`Learning status:`, status);
// Output:
// {
//   enabled: true,
//   totalExperiences: 42,
//   patterns: 8,
//   improvement: {
//     improvementRate: 15.2,
//     daysElapsed: 21,
//     targetAchieved: false
//   },
//   activeTests: 1
// }

// Get performance report
const report = await agent.getPerformanceReport();
console.log(report.summary);
// Output: "Agent test-generator-xxx performance: 15.2% improvement over 21 days.
//          4.8% improvement needed to reach 20% target."

Manual Learning Control

import { LearningEngine } from './learning/LearningEngine';
import { PerformanceTracker } from './learning/PerformanceTracker';

// Create learning engine
const learningEngine = new LearningEngine(
  'agent-id',
  memoryStore,
  {
    enabled: true,
    learningRate: 0.1,
    explorationRate: 0.3
  }
);

await learningEngine.initialize();

// Learn from task execution
const outcome = await learningEngine.learnFromExecution(
  task,
  result,
  userFeedback
);

if (outcome.improved) {
  console.log(`Performance improved by ${outcome.improvementRate}%`);
}

// Get strategy recommendation
const recommendation = await learningEngine.recommendStrategy(currentState);
console.log(`Recommended strategy: ${recommendation.strategy}`);
console.log(`Confidence: ${recommendation.confidence}`);
console.log(`Expected improvement: ${recommendation.expectedImprovement}%`);

A/B Testing

import { ImprovementLoop } from './learning/ImprovementLoop';

// Create improvement loop
const improvementLoop = new ImprovementLoop(
  'agent-id',
  memoryStore,
  learningEngine,
  performanceTracker
);

await improvementLoop.initialize();

// Create A/B test
const testId = await improvementLoop.createABTest(
  'Parallel vs Sequential Execution',
  [
    { name: 'parallel', config: { parallelization: 0.8 } },
    { name: 'sequential', config: { parallelization: 0.0 } }
  ],
  sampleSize: 100
);

// Record test results (happens automatically during task execution)
await improvementLoop.recordTestResult(
  testId,
  'parallel',
  true, // success
  1250  // execution time in ms
);

// Get active tests
const activeTests = improvementLoop.getActiveTests();
console.log(`Active tests: ${activeTests.length}`);

Configuration

LearningConfig

interface LearningConfig {
  enabled: boolean;              // Enable/disable learning
  learningRate: number;          // 0.0 - 1.0 (default: 0.1)
  discountFactor: number;        // Future reward discount (default: 0.95)
  explorationRate: number;       // Exploration vs exploitation (default: 0.3)
  explorationDecay: number;      // Decay rate (default: 0.995)
  minExplorationRate: number;    // Minimum exploration (default: 0.01)
  maxMemorySize: number;         // Max learning data in bytes (default: 100MB)
  batchSize: number;             // Batch update size (default: 32)
  updateFrequency: number;       // Updates per N tasks (default: 10)
}

Performance Requirements

• Storage: < 100MB per project (configurable)
• Improvement Target: 20% over 30 days
• False Positives: Zero (all recommendations are validated)
• User Control: On/off toggle available

Memory Storage

Learning data is stored in SwarmMemoryManager with the following structure:

phase2/learning/<agent-id>/
  ├── config              # Learning configuration
  ├── state               # Q-table, experiences, patterns
  ├── baseline            # Baseline performance metrics
  ├── snapshots/          # Performance snapshots over time
  ├── improvement         # Current improvement data
  ├── strategies/         # Available strategies
  ├── abtests/            # A/B test configurations
  ├── failure-patterns/   # Identified failure patterns
  └── cycles/             # Improvement cycle results

Performance Metrics

Tracked Metrics

1. Task Success Rate - Percentage of successfully completed tasks
2. Execution Time - Average time to complete tasks
3. Error Rate - Frequency of errors during execution
4. User Satisfaction - User feedback ratings (0.0 - 1.0)
5. Resource Efficiency - Efficiency of resource utilization

Improvement Calculation

improvementRate = ((currentScore - baselineScore) / baselineScore) * 100

Where score is the weighted composite of all metrics.

Reporting

const report = await performanceTracker.generateReport();

// report contains:
{
  summary: "Agent xxx performance: 15.2% improvement over 21 days...",
  improvement: {
    improvementRate: 15.2,
    daysElapsed: 21,
    targetAchieved: false
  },
  trends: [
    { metric: 'successRate', direction: 'up', changeRate: 12.5 },
    { metric: 'executionTime', direction: 'down', changeRate: -8.3 }
  ],
  recommendations: [
    "Focus on improving task success rate...",
    "Optimize execution time by enabling parallel processing..."
  ]
}

Integration with BaseAgent

The learning system integrates seamlessly with BaseAgent through lifecycle hooks:

class MyAgent extends LearningAgent {
  protected async onPostTask(data: PostTaskData): Promise<void> {
    await super.onPostTask(data); // Triggers automatic learning

    // Agent automatically:
    // 1. Learns from task execution
    // 2. Records performance metrics
    // 3. Updates strategies
    // 4. Applies improvements
  }
}

Best Practices

1. Set Appropriate Learning Rate:

   • Start with 0.1 for stable learning
   • Increase to 0.3 for faster adaptation
   • Decrease to 0.05 for fine-tuning

2. Monitor Exploration Rate:

   • High exploration (0.3) early for discovery
   • Gradually decays to 0.01 for exploitation
   • Reset if performance plateaus

3. Regular Performance Reviews:

   • Check improvement every 7 days
   • Adjust strategies if < 5% improvement
   • Celebrate when 20% target achieved!

4. A/B Test New Strategies:

   • Test before full deployment
   • Use statistical significance (p < 0.05)
   • Compare at least 2 strategies

5. Manage Memory Size:

   • Monitor learning state size
   • Prune old experiences if > 100MB
   • Keep most recent 1000 experiences

Troubleshooting

Low Improvement Rate

Symptoms: < 5% improvement after 15 days

Solutions:

• Increase learning rate to 0.2
• Add more diverse experiences
• Review failure patterns
• Test new strategies via A/B testing

Memory Size Issues

Symptoms: Learning state > 100MB

Solutions:

• Reduce batch size to 16
• Decrease max experiences to 500
• Prune old snapshots
• Increase update frequency

Instability in Learning

Symptoms: Performance fluctuates wildly

Solutions:

• Decrease learning rate to 0.05
• Increase batch size to 64
• Review reward calculation
• Check for data quality issues

Future Enhancements

• Deep Q-learning for complex state spaces
• Multi-agent collaborative learning
• Transfer learning across agent types
• Automated hyperparameter tuning
• Real-time dashboard for monitoring

References

• Q-Learning: Watkins & Dayan (1992)
• Reinforcement Learning: Sutton & Barto (2018)
• A/B Testing: Kohavi et al. (2009)

---

Phase 2 (v1.1.0) - Milestone 2.2 Complete