proffesor-for-testing__agentic-qe/docs/examples/LEARNING-SYSTEM-EXAMPLES.md

Learning System - Practical Examples

Complete examples for enabling continuous improvement through A/B testing, performance tracking, and automatic strategy optimization.

Table of Contents

• Quick Start
• A/B Testing
• Performance Tracking
• Improvement Loops
• Strategy Selection
• Advanced Scenarios

---

Quick Start

Enable Learning for an Agent

import { LearningEngine, SwarmMemoryManager } from 'agentic-qe';

// Initialize memory and learning engine
const memory = SwarmMemoryManager.getInstance();
const engine = new LearningEngine({
  memoryStore: memory,
  minSampleSize: 10,
  confidenceLevel: 0.95
});

await engine.initialize();

// Enable learning for test-generator agent
await engine.enableLearning('test-generator', {
  strategies: [
    'property-based',
    'example-based',
    'mutation-based'
  ],
  metrics: ['coverage', 'quality', 'speed'],
  improvementThreshold: 0.05  // 5% improvement to accept new strategy
});

console.log('✅ Learning enabled for test-generator');

CLI Usage

# Enable learning
aqe learning enable --agent test-generator

# Check learning status
aqe learning status --agent test-generator

# Run A/B test
aqe learning ab-test --agent test-generator --strategy-a property --strategy-b mutation

---

A/B Testing

Example 1: Basic A/B Test

Compare two test generation strategies:

import { LearningEngine } from 'agentic-qe';

const engine = new LearningEngine({
  memoryStore: memory,
  minSampleSize: 10,
  confidenceLevel: 0.95
});

await engine.initialize();

// Run A/B test: Property-based vs Mutation-based
const result = await engine.runABTest('test-generator', {
  strategyA: 'property-based',
  strategyB: 'mutation-based',
  metric: 'coverage',
  sampleSize: 20
});

console.log(`\n🔬 A/B Test Results\n`);
console.log(`Strategy A (property-based):`);
console.log(`  Coverage: ${(result.strategyA.mean * 100).toFixed(1)}%`);
console.log(`  StdDev: ${(result.strategyA.stdDev * 100).toFixed(1)}%`);
console.log(`  Samples: ${result.strategyA.samples}`);

console.log(`\nStrategy B (mutation-based):`);
console.log(`  Coverage: ${(result.strategyB.mean * 100).toFixed(1)}%`);
console.log(`  StdDev: ${(result.strategyB.stdDev * 100).toFixed(1)}%`);
console.log(`  Samples: ${result.strategyB.samples}`);

console.log(`\n📊 Conclusion:`);
console.log(`  Winner: ${result.winner}`);
console.log(`  Improvement: ${(result.improvement * 100).toFixed(1)}%`);
console.log(`  Confidence: ${(result.confidence * 100).toFixed(1)}%`);
console.log(`  P-value: ${result.pValue.toFixed(4)}`);
console.log(`  Statistically Significant: ${result.significant ? 'Yes ✅' : 'No ❌'}`);

Expected Output:

🔬 A/B Test Results

Strategy A (property-based):
  Coverage: 87.5%
  StdDev: 3.2%
  Samples: 20

Strategy B (mutation-based):
  Coverage: 82.1%
  StdDev: 4.1%
  Samples: 20

📊 Conclusion:
  Winner: property-based
  Improvement: 5.4%
  Confidence: 98.7%
  P-value: 0.0023
  Statistically Significant: Yes ✅

Example 2: Multi-Metric A/B Test

Compare strategies across multiple metrics:

// Test across coverage, quality, and speed
const metrics = ['coverage', 'quality', 'speed'];
const results = [];

for (const metric of metrics) {
  const result = await engine.runABTest('test-generator', {
    strategyA: 'property-based',
    strategyB: 'mutation-based',
    metric: metric,
    sampleSize: 15
  });

  results.push({
    metric,
    winner: result.winner,
    improvement: result.improvement,
    confidence: result.confidence
  });
}

console.log('\n📊 Multi-Metric A/B Test Results\n');

results.forEach(r => {
  console.log(`${r.metric}:`);
  console.log(`  Winner: ${r.winner}`);
  console.log(`  Improvement: ${(r.improvement * 100).toFixed(1)}%`);
  console.log(`  Confidence: ${(r.confidence * 100).toFixed(1)}%`);
  console.log();
});

// Overall best strategy
const overallWinner = results.reduce((acc, r) => {
  acc[r.winner] = (acc[r.winner] || 0) + 1;
  return acc;
}, {});

const bestStrategy = Object.entries(overallWinner)
  .sort((a, b) => b[1] - a[1])[0][0];

console.log(`🏆 Overall Best Strategy: ${bestStrategy}`);

Example 3: Sequential A/B Testing

Test multiple strategies sequentially:

const strategies = [
  'property-based',
  'example-based',
  'mutation-based',
  'hybrid'
];

// Test each strategy against the current best
let currentBest = strategies[0];
let currentBestScore = 0;

for (let i = 1; i < strategies.length; i++) {
  console.log(`\n🔬 Testing ${currentBest} vs ${strategies[i]}`);

  const result = await engine.runABTest('test-generator', {
    strategyA: currentBest,
    strategyB: strategies[i],
    metric: 'coverage',
    sampleSize: 15
  });

  if (result.winner === strategies[i] && result.significant) {
    console.log(`✅ ${strategies[i]} is better! (${(result.improvement * 100).toFixed(1)}% improvement)`);
    currentBest = strategies[i];
    currentBestScore = result.strategyB.mean;
  } else {
    console.log(`❌ ${currentBest} remains the best`);
  }
}

console.log(`\n🏆 Final Best Strategy: ${currentBest}`);
console.log(`   Coverage: ${(currentBestScore * 100).toFixed(1)}%`);

---

Performance Tracking

Example 1: Track Agent Performance Over Time

import { PerformanceTracker, LearningEngine } from 'agentic-qe';

const tracker = new PerformanceTracker({
  memoryStore: memory,
  trackingWindow: 7 * 24 * 60 * 60 * 1000  // 7 days
});

await tracker.initialize();

// Record performance data
await tracker.recordPerformance('test-generator', {
  strategy: 'property-based',
  coverage: 0.89,
  quality: 0.92,
  speed: 145,  // milliseconds
  timestamp: Date.now()
});

// Get performance over time
const performance = await engine.getAgentPerformance('test-generator', {
  timeRange: '7d',
  metrics: ['coverage', 'quality', 'speed']
});

console.log(`\n📈 Agent Performance (Last 7 Days)\n`);

Object.keys(performance.metrics).forEach(metric => {
  const stats = performance.metrics[metric];
  const trend = performance.trends[metric];

  console.log(`${metric}:`);
  console.log(`  Mean: ${(stats.mean * 100).toFixed(1)}%`);
  console.log(`  Min: ${(stats.min * 100).toFixed(1)}%`);
  console.log(`  Max: ${(stats.max * 100).toFixed(1)}%`);
  console.log(`  StdDev: ${(stats.stdDev * 100).toFixed(1)}%`);
  console.log(`  Trend: ${trend} ${trend === 'rising' ? '📈' : trend === 'declining' ? '📉' : '➡️'}`);
  console.log();
});

Expected Output:

📈 Agent Performance (Last 7 Days)

coverage:
  Mean: 87.5%
  Min: 82.1%
  Max: 92.3%
  StdDev: 2.8%
  Trend: rising 📈

quality:
  Mean: 91.2%
  Min: 87.5%
  Max: 95.1%
  StdDev: 2.1%
  Trend: rising 📈

speed:
  Mean: 145ms
  Min: 120ms
  Max: 180ms
  StdDev: 15ms
  Trend: stable ➡️

Example 2: Compare Performance Across Strategies

// Get performance for each strategy
const strategies = ['property-based', 'mutation-based', 'example-based'];
const comparison = [];

for (const strategy of strategies) {
  const perf = await tracker.getStrategyPerformance('test-generator', strategy, {
    timeRange: '7d'
  });

  comparison.push({
    strategy,
    coverage: perf.coverage.mean,
    quality: perf.quality.mean,
    speed: perf.speed.mean,
    samples: perf.samples
  });
}

// Sort by coverage
comparison.sort((a, b) => b.coverage - a.coverage);

console.log('\n📊 Strategy Comparison\n');
console.log('Strategy          Coverage  Quality   Speed    Samples');
console.log('─'.repeat(60));

comparison.forEach(s => {
  console.log(
    `${s.strategy.padEnd(18)} ${(s.coverage * 100).toFixed(1)}%    ` +
    `${(s.quality * 100).toFixed(1)}%    ${s.speed.toFixed(0)}ms    ${s.samples}`
  );
});

Example 3: Performance Alerts

// Set up performance monitoring with alerts
const monitor = new PerformanceMonitor({
  thresholds: {
    coverage: { min: 0.80, max: 1.0 },
    quality: { min: 0.85, max: 1.0 },
    speed: { min: 0, max: 200 }
  },
  alertCallback: async (alert) => {
    console.log(`\n⚠️  Performance Alert: ${alert.metric}`);
    console.log(`   Agent: ${alert.agentId}`);
    console.log(`   Current: ${alert.current}`);
    console.log(`   Threshold: ${alert.threshold}`);
    console.log(`   Severity: ${alert.severity}`);

    // Take action
    if (alert.severity === 'critical') {
      // Revert to known good strategy
      await engine.revertToBestStrategy(alert.agentId, alert.metric);
    }
  }
});

await monitor.start();

---

Improvement Loops

Example 1: Continuous Improvement Loop

Set up automatic improvement based on performance:

import { ImprovementLoop, LearningEngine } from 'agentic-qe';

const loop = new ImprovementLoop({
  memoryStore: memory,
  checkInterval: 3600000,  // Check every hour
  minSampleSize: 20,
  improvementThreshold: 0.05  // 5% improvement required
});

await loop.initialize();

// Start improvement loop for test-generator
await loop.startLoop('test-generator', {
  strategies: ['property-based', 'mutation-based', 'example-based'],
  metric: 'coverage',
  onImprovement: async (improvement) => {
    console.log(`\n✨ Improvement Detected!`);
    console.log(`   Strategy: ${improvement.newStrategy}`);
    console.log(`   Improvement: ${(improvement.improvement * 100).toFixed(1)}%`);
    console.log(`   Old: ${(improvement.oldPerformance * 100).toFixed(1)}%`);
    console.log(`   New: ${(improvement.newPerformance * 100).toFixed(1)}%`);
  },
  onStagnation: async (stagnation) => {
    console.log(`\n⚠️  Performance Stagnation Detected`);
    console.log(`   Duration: ${stagnation.duration}ms`);
    console.log(`   Trying new strategies...`);
  }
});

console.log('✅ Improvement loop started');

// Loop runs in background, checking performance every hour
// and automatically switching to better strategies

Example 2: Manual Improvement Cycle

// Manual improvement cycle
async function improvementCycle(agentId: string) {
  console.log(`\n🔄 Starting improvement cycle for ${agentId}\n`);

  // 1. Get current performance
  const currentPerf = await engine.getAgentPerformance(agentId, {
    timeRange: '24h'
  });

  console.log(`Current Performance:`);
  console.log(`  Coverage: ${(currentPerf.metrics.coverage.mean * 100).toFixed(1)}%`);
  console.log(`  Trend: ${currentPerf.trends.coverage}`);

  // 2. If declining, try new strategies
  if (currentPerf.trends.coverage === 'declining') {
    console.log(`\n⚠️  Declining performance - testing alternatives`);

    const currentStrategy = await engine.getCurrentStrategy(agentId);
    const alternatives = await engine.getAlternativeStrategies(agentId, currentStrategy);

    for (const alt of alternatives) {
      const result = await engine.runABTest(agentId, {
        strategyA: currentStrategy,
        strategyB: alt,
        metric: 'coverage',
        sampleSize: 15
      });

      if (result.winner === alt && result.significant) {
        console.log(`\n✅ Found better strategy: ${alt}`);
        await engine.switchStrategy(agentId, alt);
        return;
      }
    }
  }

  // 3. If rising, continue current strategy
  else if (currentPerf.trends.coverage === 'rising') {
    console.log(`\n✅ Performance is improving - continuing current strategy`);
  }

  // 4. If stable, explore new options
  else {
    console.log(`\n➡️  Performance is stable - exploring new strategies`);

    const newStrategy = await engine.suggestNewStrategy(agentId, {
      based On: 'exploration',
      diversity: 0.7
    });

    console.log(`   Trying: ${newStrategy}`);
    await engine.testStrategy(agentId, newStrategy, { samples: 10 });
  }
}

// Run improvement cycle every 6 hours
setInterval(() => improvementCycle('test-generator'), 6 * 60 * 60 * 1000);

Example 3: Feedback-Driven Learning

// Learn from explicit feedback
async function recordFeedback(agentId: string, feedback: any) {
  await engine.recordFeedback(agentId, {
    strategy: feedback.strategy,
    rating: feedback.rating,  // 1-5
    comments: feedback.comments,
    context: {
      testFile: feedback.testFile,
      coverage: feedback.coverage,
      issues: feedback.issues
    },
    timestamp: Date.now()
  });

  console.log(`✅ Feedback recorded for ${agentId}`);

  // Analyze feedback trends
  const feedbackAnalysis = await engine.analyzeFeedback(agentId, {
    timeRange: '30d'
  });

  console.log(`\n📊 Feedback Analysis (30 days)`);
  console.log(`   Average Rating: ${feedbackAnalysis.avgRating.toFixed(1)}/5`);
  console.log(`   Total Feedback: ${feedbackAnalysis.total}`);
  console.log(`   Positive: ${feedbackAnalysis.positive} (${(feedbackAnalysis.positiveRate * 100).toFixed(1)}%)`);
  console.log(`   Negative: ${feedbackAnalysis.negative} (${(feedbackAnalysis.negativeRate * 100).toFixed(1)}%)`);

  // Adjust strategy based on feedback
  if (feedbackAnalysis.positiveRate < 0.6) {
    console.log(`\n⚠️  Low positive feedback - switching strategy`);
    await engine.switchToHighestRatedStrategy(agentId);
  }
}

// Example usage
await recordFeedback('test-generator', {
  strategy: 'property-based',
  rating: 4,
  comments: 'Good coverage but slow',
  testFile: 'user-service.test.ts',
  coverage: 0.92,
  issues: ['performance']
});

---

Strategy Selection

Example 1: Automatic Best Strategy Selection

// Automatically select best strategy for each metric
const bestStrategies = await engine.selectBestStrategies('test-generator', {
  metrics: ['coverage', 'quality', 'speed'],
  confidenceLevel: 0.95
});

console.log('\n🏆 Best Strategies per Metric\n');

Object.keys(bestStrategies).forEach(metric => {
  const best = bestStrategies[metric];
  console.log(`${metric}:`);
  console.log(`  Strategy: ${best.strategy}`);
  console.log(`  Performance: ${(best.performance * 100).toFixed(1)}%`);
  console.log(`  Confidence: ${(best.confidence * 100).toFixed(1)}%`);
  console.log(`  Samples: ${best.samples}`);
  console.log();
});

// Use composite strategy (best for each metric)
await engine.enableCompositeStrategy('test-generator', {
  coverage: bestStrategies.coverage.strategy,
  quality: bestStrategies.quality.strategy,
  speed: bestStrategies.speed.strategy
});

Example 2: Context-Aware Strategy Selection

// Select strategy based on context
async function selectStrategyForContext(context: any) {
  const recommendation = await engine.recommendStrategy('test-generator', {
    fileType: context.fileType,
    complexity: context.complexity,
    priority: context.priority,
    deadline: context.deadline
  });

  console.log(`\n💡 Strategy Recommendation for Context\n`);
  console.log(`File Type: ${context.fileType}`);
  console.log(`Complexity: ${context.complexity}`);
  console.log(`Priority: ${context.priority}`);
  console.log(`\nRecommended Strategy: ${recommendation.strategy}`);
  console.log(`Reason: ${recommendation.reason}`);
  console.log(`Expected Coverage: ${(recommendation.expectedCoverage * 100).toFixed(1)}%`);
  console.log(`Expected Duration: ${recommendation.expectedDuration}ms`);

  return recommendation.strategy;
}

// Usage
const strategy = await selectStrategyForContext({
  fileType: 'service',
  complexity: 'high',
  priority: 'critical',
  deadline: '1h'
});

Example 3: Ensemble Strategy

// Use multiple strategies together
const ensembleStrategy = await engine.createEnsembleStrategy('test-generator', {
  strategies: [
    { name: 'property-based', weight: 0.4 },
    { name: 'mutation-based', weight: 0.3 },
    { name: 'example-based', weight: 0.3 }
  ],
  combineMethod: 'weighted-average'
});

// Apply ensemble
const result = await engine.applyStrategy('test-generator', ensembleStrategy, {
  sourceFile: 'user-service.ts'
});

console.log(`\n🎯 Ensemble Strategy Results\n`);
console.log(`Coverage: ${(result.coverage * 100).toFixed(1)}%`);
console.log(`Quality: ${(result.quality * 100).toFixed(1)}%`);
console.log(`Tests Generated: ${result.testsGenerated}`);
console.log(`\nContributions:`);

result.contributions.forEach(c => {
  console.log(`  ${c.strategy}: ${c.testsGenerated} tests (${(c.weight * 100).toFixed(0)}%)`);
});

---

Advanced Scenarios

Example 1: Multi-Agent Learning Coordination

// Coordinate learning across multiple agents
const agents = ['test-generator', 'coverage-analyzer', 'quality-gate'];

// Share learning insights across agents
for (const agent of agents) {
  await engine.enableLearning(agent, {
    strategies: await engine.getSuggestedStrategies(agent),
    metrics: ['coverage', 'quality', 'speed'],
    shareInsights: true  // Share with other agents
  });
}

// Cross-agent learning
const insights = await engine.getCrossAgentInsights(agents);

console.log('\n🔗 Cross-Agent Learning Insights\n');

insights.forEach(insight => {
  console.log(`${insight.type}:`);
  console.log(`  Source: ${insight.sourceAgent}`);
  console.log(`  Applicable To: ${insight.applicableAgents.join(', ')}`);
  console.log(`  Insight: ${insight.description}`);
  console.log(`  Impact: ${insight.estimatedImpact}`);
  console.log();
});

Example 2: Learning Dashboard

// Generate learning dashboard
const dashboard = await engine.generateDashboard({
  agents: ['test-generator', 'coverage-analyzer'],
  timeRange: '7d',
  metrics: ['coverage', 'quality', 'speed']
});

console.log('\n📊 Learning Dashboard (Last 7 Days)\n');
console.log('═'.repeat(60));

dashboard.agents.forEach(agent => {
  console.log(`\n${agent.name}:`);
  console.log(`  Current Strategy: ${agent.currentStrategy}`);
  console.log(`  Performance: ${(agent.performance.mean * 100).toFixed(1)}%`);
  console.log(`  Trend: ${agent.trend} ${agent.trend === 'rising' ? '📈' : agent.trend === 'declining' ? '📉' : '➡️'}`);
  console.log(`  A/B Tests: ${agent.abTests.completed}/${agent.abTests.total}`);
  console.log(`  Improvements: ${agent.improvements.count} (+${(agent.improvements.totalGain * 100).toFixed(1)}%)`);
});

console.log('\n' + '═'.repeat(60));
console.log(`\nOverall System Performance:`);
console.log(`  Average Coverage: ${(dashboard.overall.coverage * 100).toFixed(1)}%`);
console.log(`  Average Quality: ${(dashboard.overall.quality * 100).toFixed(1)}%`);
console.log(`  Total Improvements: ${dashboard.overall.improvements}`);

Example 3: Export Learning Report

// Generate comprehensive learning report
const report = await engine.generateReport('test-generator', {
  timeRange: '30d',
  includeCharts: true,
  includeRecommendations: true,
  format: 'html'
});

// Save report
await fs.writeFile('learning-report.html', report.html);

console.log('✅ Learning report saved to learning-report.html');

// Also get summary
console.log(`\n📈 30-Day Learning Summary\n`);
console.log(`Total A/B Tests: ${report.summary.abTests}`);
console.log(`Improvements Found: ${report.summary.improvements}`);
console.log(`Average Improvement: ${(report.summary.avgImprovement * 100).toFixed(1)}%`);
console.log(`Best Strategy: ${report.summary.bestStrategy}`);
console.log(`\nRecommendations:`);

report.recommendations.forEach((rec, idx) => {
  console.log(`  ${idx + 1}. ${rec.title}`);
  console.log(`     ${rec.description}`);
  console.log(`     Impact: ${rec.estimatedImpact}`);
});

---

CLI Reference

Enable/Disable Learning

# Enable learning
aqe learning enable --agent test-generator

# Disable learning
aqe learning disable --agent test-generator

# List agents with learning enabled
aqe learning list

A/B Testing

# Run A/B test
aqe learning ab-test --agent test-generator --strategy-a property --strategy-b mutation

# Run multi-metric A/B test
aqe learning ab-test --agent test-generator --strategy-a property --strategy-b mutation --metrics coverage,quality,speed

# View A/B test history
aqe learning ab-test-history --agent test-generator

Performance Tracking

# View agent performance
aqe learning performance --agent test-generator --time-range 7d

# Compare strategies
aqe learning compare --agent test-generator --strategies property,mutation,example

# Export performance data
aqe learning export --agent test-generator --format csv --output performance.csv

Reports

# Generate learning report
aqe learning report --agent test-generator --format html --output report.html

# View dashboard
aqe learning dashboard --agents test-generator,coverage-analyzer

# Get insights
aqe learning insights --time-range 30d

---

Best Practices

1. A/B Testing

DO:

• ✅ Use sufficient sample sizes (min 10-20)
• ✅ Test one variable at a time
• ✅ Wait for statistical significance
• ✅ Document test context and results

DON'T:

• ❌ Compare incompatible strategies
• ❌ Change strategies too frequently
• ❌ Ignore confidence levels
• ❌ Test without clear metrics

2. Performance Tracking

DO:

• ✅ Track multiple metrics
• ✅ Set realistic baselines
• ✅ Monitor trends over time
• ✅ Alert on significant changes

DON'T:

• ❌ Optimize for single metric
• ❌ Ignore context (file type, complexity)
• ❌ Compare across different environments
• ❌ Over-react to short-term fluctuations

3. Improvement Loops

DO:

• ✅ Start with small improvement thresholds (3-5%)
• ✅ Review improvements regularly
• ✅ Balance exploration vs exploitation
• ✅ Keep human in the loop for major changes

DON'T:

• ❌ Auto-apply without validation
• ❌ Run too many concurrent experiments
• ❌ Forget to document what worked
• ❌ Disable learning after finding good strategy

---

Troubleshooting

Issue: A/B Tests Never Reach Significance

Solutions:

// Reduce confidence level
const result = await engine.runABTest('agent', {
  strategyA: 'a',
  strategyB: 'b',
  confidenceLevel: 0.90  // Down from 0.95
});

// Increase sample size
const result = await engine.runABTest('agent', {
  strategyA: 'a',
  strategyB: 'b',
  sampleSize: 30  // Up from 20
});

// Try different metrics
const result = await engine.runABTest('agent', {
  strategyA: 'a',
  strategyB: 'b',
  metric: 'quality'  // Instead of coverage
});

Issue: Performance Not Improving

Solutions:

// Check if learning is enabled
const status = await engine.getLearningStatus('test-generator');
console.log(`Learning enabled: ${status.enabled}`);

// Review strategy pool
const strategies = await engine.getAvailableStrategies('test-generator');
console.log(`Available strategies: ${strategies.join(', ')}`);

// Try manual strategy switch
await engine.switchStrategy('test-generator', 'mutation-based');

// Reset learning data
await engine.resetLearning('test-generator');

---

Next Steps

• Flaky Detection ML Examples
• Reasoning Bank Examples
• Complete API Reference
• Phase 2 User Guide

---

Learning System Examples | Report Issues