proffesor-for-testing__agentic-qe/docs/plans/TEST-REARCHITECTURE-PLAN.md

Test Re-Architecture Plan

Priority: 🔴 CRITICAL Timeline: 2-4 weeks Goal: Increase coverage from 1.36% → 60%

---

🎯 Executive Summary

Current State: 1.36% coverage despite 150+ test files Root Cause: Tests validate mocks, not real implementations Solution: Rewrite tests to exercise actual code paths

---

📊 Coverage Gap Analysis

Modules with 0% Coverage (Priority Order)

Tier 1: Foundation (IMMEDIATE)

Impact: +20% coverage, blocks all other work

Module	Lines	Priority	Reason
BaseAgent.ts	166	🔴 CRITICAL	Base class for all agents, must work
FleetManager.ts	99	🔴 CRITICAL	Orchestrates everything, 38% → 85%
Task.ts	92	🔴 CRITICAL	Core abstraction, 17% → 80%
SwarmMemoryManager.ts	436	🔴 CRITICAL	All state storage, 18% → 75%
MemoryManager.ts	211	🔴 CRITICAL	Memory operations, 0% → 70%

Sub-total: 1,004 lines, currently 173 covered (17.2%) Target: 800 lines covered (79.7%) Impact: +2.8% overall coverage

Tier 2: Agents (HIGH)

Impact: +15% coverage, enables agent testing

Agent Type	Lines	Tests Exist?	Strategy
TestExecutorAgent	262	✅ Mocked	Rewrite with real executor
TestGeneratorAgent	218	✅ Mocked	Test real generation
CoverageAnalyzerAgent	270	✅ Mocked	Test real analysis
FlakyTestHunterAgent	357	✅ Mocked	Test real detection
PerformanceTesterAgent	221	✅ Mocked	Test real benchmarks
QualityGateAgent	188	✅ Mocked	Test real gates
RegressionRiskAnalyzerAgent	381	✅ Mocked	Test real analysis
SecurityScannerAgent	223	✅ Mocked	Test real scanning
DeploymentReadinessAgent	332	✅ Mocked	Test real checks
RequirementsValidatorAgent	404	✅ Mocked	Test real validation

Sub-total: 2,856 lines, currently 0 covered (0%) Target: 2,285 lines covered (80%) Impact: +10.1% overall coverage

Tier 3: Learning System (HIGH)

Impact: +5% coverage, critical for intelligence

Module	Lines	Current	Strategy
LearningEngine.ts	194	0%	Test real learning loops
PerformanceTracker.ts	119	0%	Test real metrics
ImprovementLoop.ts	213	0%	Test real feedback
FlakyTestDetector.ts	252	0%	Test real ML detection
StatisticalAnalysis.ts	143	0%	Test real statistics

Sub-total: 921 lines, currently 0 covered (0%) Target: 736 lines covered (80%) Impact: +3.3% overall coverage

Tier 4: CLI Commands (MEDIUM)

Impact: +8% coverage, user-facing

60+ command modules (~6,000 lines, 0% coverage) Strategy: Integration tests with real command execution Target: 60% coverage (3,600 lines) Impact: +6.0% overall coverage

Tier 5: MCP Handlers (MEDIUM)

Impact: +3% coverage, API surface

10+ handler modules (~1,500 lines, 0% coverage) Strategy: Test real MCP protocol handling Target: 75% coverage (1,125 lines) Impact: +5.0% overall coverage

Tier 6: Reasoning/ReasoningBank (LOW)

Impact: +2% coverage, advanced features

5 modules (~800 lines, 0% coverage) Strategy: Test real pattern recognition Target: 80% coverage (640 lines) Impact: +2.8% overall coverage

---

🔧 Test Re-Architecture Patterns

Pattern 1: Remove Mock Overuse

❌ WRONG (Current)

// tests/agents/TestExecutorAgent.test.ts
jest.mock('../../src/agents/BaseAgent');
jest.mock('../../src/core/FleetManager');
jest.mock('../../src/core/EventBus');

describe('TestExecutorAgent', () => {
  it('should execute tests', async () => {
    const agent = new TestExecutorAgent({} as any);
    const result = await agent.executeTask({} as any);
    expect(result).toBeDefined(); // 0% coverage
  });
});

Result: Test passes, 0 lines covered

✅ RIGHT (Required)

// tests/agents/TestExecutorAgent.test.ts
import { TestExecutorAgent } from '../../src/agents/TestExecutorAgent';
import { globalEventBus } from '../setup/global-infrastructure';
import { createTestMemoryStore } from '../utils/test-helpers';

describe('TestExecutorAgent', () => {
  let agent: TestExecutorAgent;
  let memoryStore: SwarmMemoryManager;

  beforeEach(async () => {
    memoryStore = await createTestMemoryStore();
    agent = new TestExecutorAgent({
      id: 'test-executor-1',
      type: 'test-executor',
      memoryStore,
      eventBus: globalEventBus,
      logger: createTestLogger()
    });
    await agent.initialize();
  });

  it('should execute real Jest tests', async () => {
    const task = {
      id: 'task-1',
      description: 'Run unit tests',
      framework: 'jest',
      testFiles: ['tests/sample-unit.test.js']
    };

    const result = await agent.executeTask(task);

    expect(result.status).toBe('completed');
    expect(result.testsRun).toBeGreaterThan(0);
    expect(result.framework).toBe('jest');
    expect(result.duration).toBeGreaterThan(0);

    // Verify memory storage
    const storedResult = await memoryStore.retrieve(
      `test-results/${task.id}`
    );
    expect(storedResult).toEqual(result);
  });

  it('should handle test failures gracefully', async () => {
    const task = {
      id: 'task-2',
      framework: 'jest',
      testFiles: ['tests/failing-test.js']
    };

    const result = await agent.executeTask(task);

    expect(result.status).toBe('failed');
    expect(result.failures).toBeGreaterThan(0);
    expect(result.errorDetails).toBeDefined();
  });
});

Result: Test passes, 80-100 lines covered

Pattern 2: Integration Over Isolation

❌ WRONG (Current)

// tests/integration/fleet-coordination.test.ts
const mockFleet = { coordinate: jest.fn() };

test('fleet coordinates agents', async () => {
  await mockFleet.coordinate();
  expect(mockFleet.coordinate).toHaveBeenCalled(); // 0% coverage
});

✅ RIGHT (Required)

// tests/integration/fleet-coordination.test.ts
import { FleetManager } from '../../src/core/FleetManager';
import { globalEventBus, globalMemoryStore } from '../setup/global-infrastructure';

describe('Fleet Coordination Integration', () => {
  let fleet: FleetManager;

  beforeEach(async () => {
    fleet = new FleetManager({
      eventBus: globalEventBus,
      memoryStore: globalMemoryStore,
      config: { maxAgents: 5, topology: 'mesh' }
    });
    await fleet.initialize();
  });

  afterEach(async () => {
    await fleet.shutdown();
  });

  it('should coordinate multiple agents in real workflow', async () => {
    // Spawn agents
    const executor = await fleet.spawnAgent({
      type: 'test-executor',
      capabilities: ['jest', 'mocha']
    });
    const analyzer = await fleet.spawnAgent({
      type: 'coverage-analyzer',
      capabilities: ['istanbul', 'v8']
    });

    // Execute coordinated workflow
    const task = {
      id: 'workflow-1',
      steps: [
        { agent: executor.id, action: 'execute-tests' },
        { agent: analyzer.id, action: 'analyze-coverage' }
      ]
    };

    const result = await fleet.executeWorkflow(task);

    expect(result.status).toBe('completed');
    expect(result.stepsCompleted).toBe(2);
    expect(result.agents).toHaveLength(2);

    // Verify agent coordination via EventBus
    const events = await globalEventBus.getEventLog();
    expect(events.filter(e => e.type === 'agent:coordinated')).toHaveLength(2);

    // Verify state persistence
    const fleetState = await globalMemoryStore.retrieve('fleet/state');
    expect(fleetState.activeAgents).toBe(2);
  });
});

Result: Test passes, 150-200 lines covered across multiple modules

Pattern 3: Real Dependencies, Boundary Mocks

Keep Real

• EventBus (use global singleton)
• MemoryStore (use in-memory SQLite)
• Agents (use real instances)
• Task execution logic

Mock Only Boundaries

• Network requests (axios)
• Filesystem (fs/promises)
• External APIs (Claude API)
• Time-dependent operations (Date.now)

// ✅ GOOD: Mock boundaries, test real logic
import axios from 'axios';
jest.mock('axios');

describe('RequirementsValidatorAgent', () => {
  it('validates requirements from API', async () => {
    // Mock the API boundary
    (axios.get as jest.Mock).mockResolvedValue({
      data: { requirements: [...] }
    });

    // Test real validation logic
    const agent = new RequirementsValidatorAgent(realDeps);
    const result = await agent.validateRequirements('api-url');

    expect(result.validated).toBe(true);
    expect(result.coverage).toBeGreaterThan(0.9);
    // Real validation logic exercised: 50+ lines covered
  });
});

---

📅 Implementation Timeline

Week 1: Foundation (Tier 1)

Goal: 1.36% → 8%

Day 1-2: BaseAgent + Task

• Rewrite tests/agents/BaseAgent.test.ts
• Remove all mocks, use real dependencies
• Target: 80% coverage (133/166 lines)
• Deliverable: BaseAgent.test.ts with 40+ test cases

Day 3-4: FleetManager + MemoryManager

• Enhance tests/core/FleetManager.test.ts
• Rewrite tests/core/MemoryManager.test.ts
• Target: 75% coverage (473/610 lines)
• Deliverable: Real fleet initialization tests

Day 5: SwarmMemoryManager

• Enhance tests/core/memory/SwarmMemoryManager.test.ts
• Test all 12 tables, real database operations
• Target: 75% coverage (327/436 lines)
• Deliverable: Database integration tests

Week 1 Milestone: 8% coverage (+1,500 lines), foundation solid

Week 2: Agents (Tier 2)

Goal: 8% → 25%

Day 6-7: Test Execution Agents

• TestExecutorAgent (262 lines)
• TestGeneratorAgent (218 lines)
• Target: 80% coverage (384/480 lines)

Day 8-9: Quality Agents

• CoverageAnalyzerAgent (270 lines)
• QualityGateAgent (188 lines)
• RegressionRiskAnalyzerAgent (381 lines)
• Target: 80% coverage (671/839 lines)

Day 10-11: Security & Deployment Agents

• SecurityScannerAgent (223 lines)
• DeploymentReadinessAgent (332 lines)
• RequirementsValidatorAgent (404 lines)
• Target: 80% coverage (767/959 lines)

Day 12: Performance & Flaky Test Agents

• PerformanceTesterAgent (221 lines)
• FlakyTestHunterAgent (357 lines)
• Target: 80% coverage (462/578 lines)

Week 2 Milestone: 25% coverage (+3,800 lines), agents working

Week 3: Learning & CLI (Tier 3 & 4)

Goal: 25% → 45%

Day 13-14: Learning System

• LearningEngine.ts (194 lines)
• PerformanceTracker.ts (119 lines)
• ImprovementLoop.ts (213 lines)
• FlakyTestDetector.ts (252 lines)
• StatisticalAnalysis.ts (143 lines)
• Target: 80% coverage (736/921 lines)

Day 15-17: CLI Commands

• Core commands (analyze, fleet, generate, init, run)
• Agent commands (assign, attach, benchmark, etc.)
• Config commands (export, get, import, etc.)
• Target: 60% coverage (1,800/3,000 lines for top 50 commands)

Day 18: Debug & Quality Commands

• Debug commands (agent, diagnostics, health-check)
• Quality commands (baseline, gate, policy, risk)
• Target: 60% coverage (600/1,000 lines)

Week 3 Milestone: 45% coverage (+6,000 lines), CLI + Learning tested

Week 4: MCP, Reasoning, Polish (Tier 5 & 6)

Goal: 45% → 60%

Day 19-20: MCP Handlers

• CoordinationTools, MemoryTools
• AnalysisTools, QualityTools, ChaosTools
• Target: 75% coverage (1,125/1,500 lines)

Day 21: ReasoningBank

• QEReasoningBank.ts (289 lines)
• PatternExtractor.ts (156 lines)
• PatternClassifier.ts (103 lines)
• Target: 80% coverage (438/548 lines)

Day 22-23: Integration Test Enhancement

• Fix existing integration tests to use real code paths
• Add missing integration scenarios
• Target: +5% coverage from better integration tests

Day 24: Polish & Verification

• Fill coverage gaps
• Add edge case tests
• Verify all targets met

Week 4 Milestone: 60% coverage (+9,000 lines total), production-ready

---

🎯 Success Metrics

Coverage Targets

• Week 1: 1.36% → 8% (Foundation)
• Week 2: 8% → 25% (Agents)
• Week 3: 25% → 45% (Learning + CLI)
• Week 4: 45% → 60% (MCP + Polish)

Quality Metrics

• Mock-to-Real Ratio: <20% (currently >80%)
• Integration Test Coverage: 40% (currently ~5%)
• Test File Existence: 200/220 modules (currently 150/220)
• Average Module Coverage: >60% (currently 1.36%)

Verification Gates

• All tests pass with real dependencies
• No coverage regressions on any module
• CI/CD pipeline executes in <10 minutes
• No flaky tests introduced

---

🛠️ Implementation Guidelines

Test File Template

// tests/[category]/[Module].test.ts
import { Module } from '../../src/[category]/[Module]';
import { globalEventBus, globalMemoryStore } from '../setup/global-infrastructure';
import { createTestLogger, waitForEvent } from '../utils/test-helpers';

describe('[Module]', () => {
  let module: Module;

  beforeEach(async () => {
    module = new Module({
      eventBus: globalEventBus,
      memoryStore: globalMemoryStore,
      logger: createTestLogger()
    });
    await module.initialize();
  });

  afterEach(async () => {
    await module.cleanup();
  });

  describe('initialization', () => {
    it('should initialize with valid config', async () => {
      expect(module.isReady()).toBe(true);
    });

    it('should register with EventBus', async () => {
      const events = await globalEventBus.getEventLog();
      expect(events.some(e => e.type === 'module:initialized')).toBe(true);
    });
  });

  describe('core functionality', () => {
    it('should [main use case]', async () => {
      // Arrange
      const input = createTestInput();

      // Act
      const result = await module.process(input);

      // Assert
      expect(result).toMatchObject({
        status: 'completed',
        data: expect.any(Object)
      });

      // Verify side effects
      const stored = await globalMemoryStore.retrieve('module/result');
      expect(stored).toEqual(result);
    });
  });

  describe('error handling', () => {
    it('should handle [error case]', async () => {
      // Test real error handling, not mocked failures
      const invalidInput = createInvalidInput();

      await expect(module.process(invalidInput))
        .rejects.toThrow('Expected error message');
    });
  });
});

Test Helper Pattern

// tests/utils/test-helpers.ts
export function createTestMemoryStore(): Promise<SwarmMemoryManager> {
  return SwarmMemoryManager.create(':memory:'); // In-memory SQLite
}

export function createTestLogger(): Logger {
  return new Logger({ level: 'error', silent: true });
}

export async function waitForEvent(
  eventBus: EventBus,
  eventType: string,
  timeout = 5000
): Promise<any> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => reject(new Error('Event timeout')), timeout);
    eventBus.once(eventType, (data) => {
      clearTimeout(timer);
      resolve(data);
    });
  });
}

---

📋 Deliverables

Week 1

• 5 rewritten test files (BaseAgent, Task, FleetManager, MemoryManager, SwarmMemoryManager)
• Test helper utilities (test-helpers.ts, test-fixtures.ts)
• Coverage: 1.36% → 8%
• Documentation: Test re-architecture guide

Week 2

• 10 rewritten agent test files
• Agent integration test suite
• Coverage: 8% → 25%
• Documentation: Agent testing patterns

Week 3

• 5 learning system test files
• 30+ CLI command test files
• Coverage: 25% → 45%
• Documentation: CLI testing guide

Week 4

• 10 MCP handler test files
• 5 ReasoningBank test files
• Enhanced integration tests
• Coverage: 45% → 60%
• Final report: Coverage achievement

---

🚨 Risk Mitigation

Risk 1: Test Complexity Explosion

Mitigation: Use shared test fixtures and helpers Contingency: Reduce targets (60% → 50%)

Risk 2: Real Dependencies Too Slow

Mitigation: Optimize test setup/teardown, use in-memory databases Contingency: Selective mocking for slowest operations

Risk 3: Flaky Tests Introduced

Mitigation: Use waitForEvent(), avoid timing assumptions Contingency: FlakyTestDetector analysis + quarantine

Risk 4: Coverage Regression

Mitigation: Lock coverage per-module with jest thresholds Contingency: Rollback to last stable state

---

📊 Progress Tracking

Daily Updates

# Run coverage and track progress
npm run test:coverage
cat coverage/coverage-summary.json | jq '.total.lines.pct'

# Track against targets
echo "Target: [Week X Target]%, Current: [Actual]%"

Weekly Reports

• Coverage achieved vs target
• Modules completed
• Blockers identified
• Next week priorities

---

🎓 Team Training

Day 1: Kickoff (2 hours)

• Why 1.36% coverage despite 150+ tests?
• Mock overuse anti-patterns
• Real dependency testing patterns
• Code walkthrough of good vs bad tests

Week 1 Check-in (1 hour)

• Review BaseAgent test rewrite
• Share learnings and challenges
• Adjust timeline if needed

Week 2 Check-in (1 hour)

• Review agent test patterns
• Integration test best practices

Week 3 Check-in (1 hour)

• CLI testing patterns
• Performance optimization

Week 4 Wrap-up (2 hours)

• Final results review
• Lessons learned
• Test maintenance process

---

🔗 References

• Current Coverage Report: /docs/reports/PHASE1-2-COVERAGE-UPDATE.md
• Original Analysis: /docs/reports/PHASE1-COVERAGE-ANALYSIS-20251020.md
• Jest Configuration: /jest.config.js
• Test Setup: /jest.setup.ts

---

Created: 2025-10-20 08:20:00 UTC Owner: QA Team + Test Engineering Status: 🔴 DRAFT - Pending Approval Next Review: 2025-10-21 (1 day)