proffesor-for-testing__agentic-qe/docs/implementation-plans/phase4-alerting-implementation-plan.md

Phase 4: Alerting & Feedback Loop - Implementation Plan

Issue: #69 Design Document: phase4-alerting-feedback-design.md Estimated Effort: 11 hours Status: Ready for Implementation

---

Quick Reference

Files to Create

src/alerting/
  ├── AlertManager.ts          # Core alert engine (2.0h)
  ├── AlertRule.ts             # Rule definitions (0.5h)
  ├── AlertChannel.ts          # Notification system (1.0h)
  ├── MetricMonitor.ts         # Metric collection (1.0h)
  └── types.ts                 # Type definitions (0.5h)

src/feedback/
  ├── FeedbackRouter.ts        # Orchestration (2.0h)
  ├── FeedbackEvent.ts         # Event types (0.5h)
  ├── StrategyApplicator.ts   # Strategy application (1.0h)
  └── types.ts                 # Type definitions (0.5h)

config/alerts/
  ├── quality-rules.yaml       # Quality alerts (0.5h)
  ├── performance-rules.yaml   # Performance alerts (0.3h)
  └── learning-rules.yaml      # Learning alerts (0.2h)

src/cli/commands/
  └── alerts.ts                # CLI commands (1.0h)

tests/integration/
  └── alerting-feedback.test.ts  # Integration tests (1.0h)

Integration Points

Component	File	Change Type	Effort
QualityMetrics	src/telemetry/metrics/quality-metrics.ts	Add alert evaluation calls	0.5h
BaseAgent	src/agents/BaseAgent.ts	Add feedback subscription	0.5h
LearningEngine	src/learning/LearningEngine.ts	Add processFeedback() method	1.0h

---

Implementation Timeline (11 hours)

Phase 1: Core Infrastructure (4 hours)

Task 1.1: Type Definitions (1.5h)

Files: src/alerting/types.ts, src/feedback/types.ts

// src/alerting/types.ts
export interface AlertRule {
  id: string;
  name: string;
  description: string;
  metricName: string;
  operator: 'lt' | 'lte' | 'gt' | 'gte' | 'eq' | 'neq';
  threshold: number;
  severity: 'info' | 'warning' | 'error' | 'critical';
  windowMs: number;
  minOccurrences?: number;
  agentScope: string;
  triggerFeedback: boolean;
  feedbackAction?: FeedbackAction;
  cooldownMs: number;
  metadata?: Record<string, unknown>;
}

export interface FeedbackAction {
  type: 'adjust_strategy' | 'retrain_model' | 'escalate' | 'auto_remediate';
  parameters: Record<string, unknown>;
  maxRetries?: number;
}

export interface AlertEvent {
  id: string;
  ruleId: string;
  timestamp: Date;
  metricName: string;
  actualValue: number;
  thresholdValue: number;
  severity: string;
  agentId: string;
  message: string;
  metadata: Record<string, unknown>;
}

// src/feedback/types.ts
export interface FeedbackEvent {
  id: string;
  timestamp: Date;
  alertRuleId?: string;
  agentIds: string[];
  metricName: string;
  currentValue: number;
  expectedValue: number;
  type: 'performance_degradation' | 'quality_improvement' |
        'error_pattern' | 'success_pattern';
  severity: 'low' | 'medium' | 'high' | 'critical';
  suggestions: StrategyRecommendation[];
  context: Record<string, unknown>;
}

export interface StrategyRecommendation {
  strategy: string;
  confidence: number;
  expectedImpact: {
    metricName: string;
    expectedChange: number;
    timeframe: string;
  };
  steps: string[];
  parameters: Record<string, unknown>;
}

export interface FeedbackEffectiveness {
  feedbackId: string;
  metricBefore: number;
  metricAfter: number;
  improvementPercent: number;
  strategyApplied: string;
  timeToImprovement: number;
  success: boolean;
}

Validation:

• Types compile without errors
• Types exported from index.ts
• JSDoc comments complete

---

Task 1.2: Alert Rule Parser (1.0h)

File: src/alerting/AlertRule.ts

import { AlertRule } from './types';
import * as yaml from 'yaml';
import * as fs from 'fs';

export class AlertRuleLoader {
  static loadFromYaml(filePath: string): AlertRule[] {
    const content = fs.readFileSync(filePath, 'utf-8');
    const parsed = yaml.parse(content);
    return this.validateRules(parsed.rules || []);
  }

  static validateRules(rules: any[]): AlertRule[] {
    return rules.map((rule, index) => {
      // Validate required fields
      if (!rule.id || !rule.metricName || !rule.threshold) {
        throw new Error(`Invalid rule at index ${index}: missing required fields`);
      }

      // Validate operator
      const validOperators = ['lt', 'lte', 'gt', 'gte', 'eq', 'neq'];
      if (!validOperators.includes(rule.operator)) {
        throw new Error(`Invalid operator '${rule.operator}' in rule ${rule.id}`);
      }

      // Validate severity
      const validSeverities = ['info', 'warning', 'error', 'critical'];
      if (!validSeverities.includes(rule.severity)) {
        throw new Error(`Invalid severity '${rule.severity}' in rule ${rule.id}`);
      }

      return rule as AlertRule;
    });
  }
}

Validation:

• Can load YAML files
• Validates all required fields
• Throws errors for invalid rules
• Unit tests pass

---

Task 1.3: Alert Manager (2.0h)

File: src/alerting/AlertManager.ts

import { EventEmitter } from 'events';
import { AlertRule, AlertEvent } from './types';
import { Logger } from '../utils/Logger';
import { EventStore } from '../persistence/event-store';

export class AlertManager extends EventEmitter {
  private rules: Map<string, AlertRule>;
  private lastFiredTime: Map<string, number>;
  private occurrenceCount: Map<string, number>;
  private logger: Logger;
  private eventStore: EventStore;

  constructor(rules: AlertRule[], eventStore: EventStore) {
    super();
    this.rules = new Map(rules.map(r => [r.id, r]));
    this.lastFiredTime = new Map();
    this.occurrenceCount = new Map();
    this.logger = Logger.getInstance();
    this.eventStore = eventStore;
  }

  async evaluateMetric(
    metricName: string,
    value: number,
    attributes: Record<string, unknown>
  ): Promise<AlertEvent | null> {
    const agentId = attributes.agent_id as string;

    // Find matching rules
    for (const [ruleId, rule] of this.rules) {
      if (!this.matchesRule(rule, metricName, agentId)) {
        continue;
      }

      // Check if threshold exceeded
      if (!this.checkThreshold(rule, value)) {
        // Reset occurrence count if threshold not exceeded
        this.occurrenceCount.delete(ruleId);
        continue;
      }

      // Increment occurrence count
      const occurrences = (this.occurrenceCount.get(ruleId) || 0) + 1;
      this.occurrenceCount.set(ruleId, occurrences);

      // Check minimum occurrences
      if (rule.minOccurrences && occurrences < rule.minOccurrences) {
        continue;
      }

      // Check cooldown period
      if (this.isInCooldown(ruleId, rule.cooldownMs)) {
        continue;
      }

      // Fire alert
      const alert = await this.fireAlert(rule, value, agentId, attributes);
      return alert;
    }

    return null;
  }

  private matchesRule(rule: AlertRule, metricName: string, agentId: string): boolean {
    if (rule.metricName !== metricName) {
      return false;
    }

    if (rule.agentScope !== '*' && rule.agentScope !== agentId) {
      return false;
    }

    return true;
  }

  private checkThreshold(rule: AlertRule, value: number): boolean {
    switch (rule.operator) {
      case 'lt': return value < rule.threshold;
      case 'lte': return value <= rule.threshold;
      case 'gt': return value > rule.threshold;
      case 'gte': return value >= rule.threshold;
      case 'eq': return value === rule.threshold;
      case 'neq': return value !== rule.threshold;
      default: return false;
    }
  }

  private isInCooldown(ruleId: string, cooldownMs: number): boolean {
    const lastFired = this.lastFiredTime.get(ruleId);
    if (!lastFired) return false;

    const elapsed = Date.now() - lastFired;
    return elapsed < cooldownMs;
  }

  private async fireAlert(
    rule: AlertRule,
    actualValue: number,
    agentId: string,
    attributes: Record<string, unknown>
  ): Promise<AlertEvent> {
    const alert: AlertEvent = {
      id: `alert-${Date.now()}-${rule.id}`,
      ruleId: rule.id,
      timestamp: new Date(),
      metricName: rule.metricName,
      actualValue,
      thresholdValue: rule.threshold,
      severity: rule.severity,
      agentId,
      message: this.formatAlertMessage(rule, actualValue),
      metadata: { ...attributes, rule: rule.name }
    };

    // Store in event store
    await this.eventStore.recordEvent({
      agent_id: agentId,
      event_type: 'custom',
      payload: { alert, type: 'alert_fired' },
      session_id: attributes.session_id as string || 'default'
    });

    // Update last fired time
    this.lastFiredTime.set(rule.id, Date.now());

    // Reset occurrence count
    this.occurrenceCount.delete(rule.id);

    // Emit alert event
    this.emit('alert:fired', alert);

    // Trigger feedback if configured
    if (rule.triggerFeedback && rule.feedbackAction) {
      this.emit('feedback:trigger', {
        alert,
        action: rule.feedbackAction
      });
    }

    this.logger.warn(`Alert fired: ${rule.name}`, {
      ruleId: rule.id,
      severity: rule.severity,
      value: actualValue,
      threshold: rule.threshold
    });

    return alert;
  }

  private formatAlertMessage(rule: AlertRule, value: number): string {
    return `${rule.name}: ${rule.metricName} = ${value} (${rule.operator} ${rule.threshold})`;
  }

  getActiveAlerts(): AlertEvent[] {
    // Return alerts that fired recently (within cooldown period)
    const active: AlertEvent[] = [];
    // Implementation to track active alerts
    return active;
  }
}

Validation:

• Evaluates metrics against rules
• Respects cooldown periods
• Handles minimum occurrences
• Emits events correctly
• Logs alert activity
• Unit tests pass

---

Phase 2: Feedback System (4 hours)

Task 2.1: Feedback Router (2.0h)

File: src/feedback/FeedbackRouter.ts

import { EventEmitter } from 'events';
import { FeedbackEvent, StrategyRecommendation } from './types';
import { LearningEngine } from '../learning/LearningEngine';
import { SwarmMemoryManager } from '../core/memory/SwarmMemoryManager';
import { Logger } from '../utils/Logger';
import { v4 as uuidv4 } from 'uuid';

export class FeedbackRouter {
  private logger: Logger;

  constructor(
    private learningEngine: LearningEngine,
    private memoryStore: SwarmMemoryManager,
    private eventBus: EventEmitter
  ) {
    this.logger = Logger.getInstance();
  }

  async routeFeedback(event: FeedbackEvent): Promise<void> {
    this.logger.info(`Processing feedback: ${event.type}`, {
      metricName: event.metricName,
      agentIds: event.agentIds
    });

    // 1. Store feedback event
    await this.storeFeedback(event);

    // 2. Route to appropriate handler
    switch (event.type) {
      case 'performance_degradation':
        await this.handlePerformanceDegradation(event);
        break;
      case 'quality_improvement':
        await this.handleQualityImprovement(event);
        break;
      case 'error_pattern':
        await this.handleErrorPattern(event);
        break;
      case 'success_pattern':
        await this.handleSuccessPattern(event);
        break;
    }

    // 3. Update learning engine
    await this.updateLearning(event);

    // 4. Emit completion event
    this.eventBus.emit('feedback:processed', event);
  }

  private async handlePerformanceDegradation(event: FeedbackEvent): Promise<void> {
    // Increase exploration to find better strategies
    const config = await this.learningEngine.getConfig();
    await this.learningEngine.updateConfig({
      explorationRate: Math.min(config.explorationRate * 1.5, 0.5)
    });

    // Apply suggestions
    for (const suggestion of event.suggestions) {
      await this.applyStrategy(suggestion, event.agentIds);
    }

    this.logger.info('Performance degradation handled', {
      newExplorationRate: (await this.learningEngine.getConfig()).explorationRate
    });
  }

  private async handleQualityImprovement(event: FeedbackEvent): Promise<void> {
    // Decrease exploration (exploit successful strategy)
    const config = await this.learningEngine.getConfig();
    await this.learningEngine.updateConfig({
      explorationRate: Math.max(config.explorationRate * 0.9, 0.01)
    });

    // Reinforce in learning
    const reward = event.currentValue - event.expectedValue;
    await this.learningEngine.recordExperience({
      taskId: event.id,
      state: event.context as any,
      action: (event.context.lastAction as any) || 'default',
      reward,
      nextState: event.context as any,
      timestamp: event.timestamp
    });

    this.logger.info('Quality improvement reinforced', { reward });
  }

  private async handleErrorPattern(event: FeedbackEvent): Promise<void> {
    // Store failure pattern
    for (const agentId of event.agentIds) {
      await this.memoryStore.store(
        `aqe/feedback/error/${agentId}/${event.id}`,
        {
          pattern: event.context,
          metricName: event.metricName,
          timestamp: event.timestamp,
          severity: event.severity
        },
        { partition: 'feedback' }
      );
    }
  }

  private async handleSuccessPattern(event: FeedbackEvent): Promise<void> {
    // Store success pattern
    for (const suggestion of event.suggestions) {
      await this.memoryStore.storePattern({
        id: uuidv4(),
        agentId: event.agentIds[0],
        taskType: event.context.taskType as string || 'unknown',
        pattern: suggestion.strategy,
        confidence: suggestion.confidence,
        occurrences: 1,
        lastSeen: event.timestamp,
        metadata: {
          feedback: true,
          metricName: event.metricName,
          improvement: event.currentValue - event.expectedValue
        }
      });
    }
  }

  private async applyStrategy(
    suggestion: StrategyRecommendation,
    agentIds: string[]
  ): Promise<void> {
    for (const agentId of agentIds) {
      await this.memoryStore.store(
        `aqe/feedback/strategy/${agentId}`,
        {
          strategy: suggestion.strategy,
          parameters: suggestion.parameters,
          confidence: suggestion.confidence,
          expectedImpact: suggestion.expectedImpact,
          appliedAt: new Date()
        },
        { partition: 'feedback' }
      );

      this.logger.info(`Strategy applied to ${agentId}: ${suggestion.strategy}`);
    }
  }

  private async storeFeedback(event: FeedbackEvent): Promise<void> {
    await this.memoryStore.store(
      `aqe/feedback/events/${event.id}`,
      event,
      { partition: 'feedback' }
    );
  }

  private async updateLearning(event: FeedbackEvent): Promise<void> {
    if (this.learningEngine.processFeedback) {
      await this.learningEngine.processFeedback(event);
    }
  }
}

Validation:

• Routes all feedback types correctly
• Applies strategies to memory
• Updates learning engine
• Emits events
• Logs activity
• Integration test passes

---

Task 2.2: Extend LearningEngine (1.5h)

File: src/learning/LearningEngine.ts (modification)

Add new method:

/**
 * Process feedback from alerting system
 * Converts feedback to learning experiences and patterns
 */
async processFeedback(feedback: FeedbackEvent): Promise<void> {
  this.logger.info(`Processing feedback: ${feedback.type}`, {
    agentId: this.agentId,
    metricName: feedback.metricName
  });

  // Convert feedback to task experience
  const experience: TaskExperience = {
    taskId: feedback.id,
    state: this.extractStateFromFeedback(feedback),
    action: this.extractActionFromFeedback(feedback),
    reward: this.calculateFeedbackReward(feedback),
    nextState: this.extractStateFromFeedback(feedback),
    timestamp: feedback.timestamp
  };

  // Record experience
  await this.recordExperience(experience);

  // Store success patterns
  if (feedback.type === 'success_pattern' || feedback.type === 'quality_improvement') {
    for (const suggestion of feedback.suggestions) {
      await this.memoryStore.storePattern({
        id: uuidv4(),
        agentId: this.agentId,
        taskType: feedback.context.taskType as string || 'feedback',
        pattern: suggestion.strategy,
        confidence: suggestion.confidence,
        occurrences: 1,
        lastSeen: feedback.timestamp,
        metadata: {
          source: 'feedback',
          metricName: feedback.metricName,
          improvement: feedback.currentValue - feedback.expectedValue
        }
      });
    }
  }

  this.logger.info('Feedback processed and stored');
}

private extractStateFromFeedback(feedback: FeedbackEvent): TaskState {
  return {
    complexity: feedback.severity === 'critical' ? 'high' : 'medium',
    priority: feedback.severity === 'critical' ? 'critical' : 'high',
    testCount: feedback.context.testCount as number || 0,
    coverage: feedback.context.coverage as number || 0,
    executionTime: feedback.context.executionTime as number || 0,
    agentLoad: 0,
    queueDepth: 0
  };
}

private extractActionFromFeedback(feedback: FeedbackEvent): AgentAction {
  const primarySuggestion = feedback.suggestions[0];
  return {
    strategy: primarySuggestion?.strategy || 'default',
    batchSize: 1,
    parallelism: 1,
    useCache: true
  };
}

private calculateFeedbackReward(feedback: FeedbackEvent): number {
  // Positive reward for improvement, negative for degradation
  const delta = feedback.currentValue - feedback.expectedValue;

  // Severity multipliers
  const severityMultiplier = {
    low: 0.1,
    medium: 0.5,
    high: 1.0,
    critical: 2.0
  };

  const multiplier = severityMultiplier[feedback.severity];

  // Normalize reward between -10 and +10
  return Math.max(-10, Math.min(10, delta * multiplier));
}

Validation:

• processFeedback() method works
• Converts feedback to experiences correctly
• Calculates appropriate rewards
• Stores patterns
• Unit tests pass

---

Phase 3: Configuration & CLI (3 hours)

Task 3.1: Alert Rule Configuration Files (1.0h)

Files: config/alerts/*.yaml

Create three YAML files with predefined rules (see design document section 2.2).

Validation:

• All YAML files are valid
• Rules can be loaded by AlertRuleLoader
• No duplicate rule IDs
• All metric names exist

---

Task 3.2: CLI Commands (1.0h)

File: src/cli/commands/alerts.ts

import { Command } from 'commander';
import { AlertManager } from '../../alerting/AlertManager';
import { AlertRuleLoader } from '../../alerting/AlertRule';
import { EventStore } from '../../persistence/event-store';
import chalk from 'chalk';

export function createAlertsCommand(): Command {
  const cmd = new Command('alerts');
  cmd.description('Manage alerting and feedback system');

  // List active alerts
  cmd
    .command('list')
    .option('-s, --severity <level>', 'Filter by severity')
    .description('List active alerts')
    .action(async (options) => {
      // Implementation
      console.log(chalk.blue('Active Alerts:'));
      // Query and display
    });

  // Configure alert rules
  cmd
    .command('configure')
    .requiredOption('-f, --rule-file <path>', 'Path to alert rules YAML')
    .description('Load alert rules from file')
    .action(async (options) => {
      const rules = AlertRuleLoader.loadFromYaml(options.ruleFile);
      console.log(chalk.green(`✓ Loaded ${rules.length} alert rules`));
    });

  // View feedback history
  cmd
    .command('feedback')
    .option('-a, --agent <id>', 'Filter by agent ID')
    .option('-w, --window <time>', 'Time window (e.g., 24h)')
    .description('View feedback history')
    .action(async (options) => {
      // Implementation
      console.log(chalk.blue('Feedback History:'));
      // Query and display
    });

  return cmd;
}

Validation:

• aqe alerts list works
• aqe alerts configure loads rules
• aqe alerts feedback shows history
• Help text is clear

---

Task 3.3: Integration with Existing Components (1.5h)

File 1: src/telemetry/metrics/quality-metrics.ts (modification)

// Import AlertManager (injected via singleton)
import { getAlertManager } from '../../alerting';

export function recordTestExecution(result: TestExecutionResult): void {
  const metrics = getQualityMetrics();

  // ... existing code ...

  // NEW: Evaluate alerts
  const failureRate = result.failed / result.total;
  getAlertManager().evaluateMetric(
    METRIC_NAMES.TEST_COUNT,
    failureRate,
    { agent_id: 'qe-test-executor', ...attributes }
  );
}

File 2: src/agents/BaseAgent.ts (modification)

protected async initialize(): Promise<void> {
  // ... existing code ...

  // NEW: Subscribe to feedback events
  this.eventBus.on(`feedback:${this.agentId.id}`, async (event: FeedbackEvent) => {
    await this.handleFeedback(event);
  });
}

protected async handleFeedback(feedback: FeedbackEvent): Promise<void> {
  this.logger.info(`Received feedback: ${feedback.type}`);

  // Check for strategy in memory
  const strategy = await this.memoryStore.retrieve(
    `aqe/feedback/strategy/${this.agentId.id}`
  );

  if (strategy) {
    // Apply strategy adaptation
    this.logger.info(`Applying strategy: ${strategy.strategy}`);
    // Implementation specific to agent type
  }

  // Update learning engine
  if (this.learningEngine?.processFeedback) {
    await this.learningEngine.processFeedback(feedback);
  }
}

Validation:

• Metrics trigger alerts
• Agents receive feedback
• Strategies are applied
• Integration test passes

---

Phase 4: Testing & Documentation (1 hour)

Task 4.1: Integration Tests (1.0h)

File: tests/integration/alerting-feedback.test.ts

describe('Alerting & Feedback Integration', () => {
  let alertManager: AlertManager;
  let feedbackRouter: FeedbackRouter;
  let eventStore: EventStore;

  beforeEach(() => {
    // Setup
  });

  it('should complete full feedback loop for coverage drop', async () => {
    // 1. Record low coverage
    recordCoverage({
      target: 'TestService',
      line: 65,
      branch: 60,
      function: 70
    });

    // 2. Wait for evaluation
    await wait(6000);

    // 3. Verify alert fired
    const alerts = alertManager.getActiveAlerts();
    expect(alerts).toHaveLength(1);
    expect(alerts[0].ruleId).toBe('coverage-drop-critical');

    // 4. Verify feedback generated
    const feedback = await getFeedbackEvent(alerts[0].id);
    expect(feedback.type).toBe('quality_improvement');
    expect(feedback.suggestions[0].strategy).toBe('generate_additional_tests');

    // 5. Verify strategy applied to memory
    const strategy = await memoryStore.retrieve('aqe/feedback/strategy/qe-coverage-analyzer');
    expect(strategy).toBeDefined();
    expect(strategy.strategy).toBe('generate_additional_tests');
  });

  it('should handle performance degradation feedback', async () => {
    // Similar test for performance alerts
  });
});

Validation:

• All integration tests pass
• Test coverage >80%
• Tests run in <10 seconds

---

Acceptance Criteria

Must Have ✅

• AlertManager evaluates metrics against rules
• FeedbackRouter processes feedback events
• LearningEngine integrates feedback
• Alert rules load from YAML
• CLI commands functional
• Integration with QualityMetrics working
• BaseAgent receives and handles feedback
• All tests pass (unit + integration)

Should Have ✅

• 10+ predefined alert rules
• Cooldown periods prevent alert spam
• Feedback effectiveness tracking
• Proper logging throughout
• Documentation complete

Nice to Have ✨

• Webhook notification channel
• Grafana dashboard examples
• Alert correlation analysis

---

Risk Mitigation

Risk	Mitigation	Owner
Alert fatigue (too many alerts)	Tune thresholds, cooldown periods	QE Team
Feedback loops cause instability	Gradual strategy application, rollback	Dev Team
Performance impact of constant evaluation	Batch evaluations, caching	Dev Team
Integration breaks existing flows	Comprehensive integration tests	QA Team

---

Post-Implementation Checklist

• Update README.md with alerting features
• Update CHANGELOG.md
• Create operational runbook for alerts
• Train team on new CLI commands
• Monitor alert precision/recall for 1 week
• Tune thresholds based on real data
• Document common alert scenarios
• Create troubleshooting guide

---

Success Metrics

Week 1 (Post-Implementation):

• Alert precision >70%
• Feedback leads to improvement in >50% of cases
• No system instability from feedback loops

Month 1:

• Alert precision >90%
• Feedback improvement rate >70%
• Learning convergence demonstrated

---

Status: ✅ Ready to Begin Implementation Next Action: Start with Task 1.1 (Type Definitions) Estimated Completion: 11 hours from start