Files
proffesor-for-testing__agen…/docs/migration/test-suite-restructuring-plan.md
Dragan Spiridonov 1e3fbb5ea8 release: v2.1.0 - Comprehensive QX Analysis & Skills Optimization (#107)
* SwarmMemoryManager Refactoring Plan

* fix: add missing directories to npm package (v1.9.3)

Added templates/, .claude/helpers/, and docs/reference/ to package.json
files array. These were missing from the npm distribution, causing
`aqe init` to fail with "Template file not found" errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* checkpoint improvements

* fix: critical memory/learning system fixes for v1.9.4

## Critical Fixes
- Vector embeddings now stored correctly (was storing NULL)
- SQL parameter style fixed for agentdb's SqlJsDatabase wrapper
- HNSW index schema mismatch resolved with pattern_id column
- Hooks now save to correct database (.agentic-qe/agentdb.db)

## New Features
- Added 5 SwarmMemoryManager methods for learning data retrieval:
  - getBestAction() - Q-learning best action selection
  - getRecentLearningExperiences() - Recent experience retrieval
  - getLearningExperiencesByTaskType() - Task-filtered experiences
  - getHighRewardExperiences() - Successful experience extraction
  - getLearningStats() - Aggregate learning statistics
- Added comprehensive integration test for memory/learning loop
- Added RuVector pattern store integration (experimental)

## Technical Details
- Vector embeddings: 384 dimensions × 4 bytes = 1,536 bytes per pattern
- All 12 integration tests pass
- HNSW index: 150x faster vector search enabled

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: move ARM64-only ruvector packages to optionalDependencies

Fixes CI failures caused by platform-specific dependencies:
- @ruvector/node-linux-arm64-gnu (ARM64 only)
- ruvector-core-linux-arm64-gnu (ARM64 only)

These packages caused npm ci to fail on x64 CI runners with:
"EBADPLATFORM - wanted linux/arm64, current linux/x64"

Solution: Move ARM64 packages to optionalDependencies and add
x64 variants for cross-platform compatibility.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: regenerate package-lock.json with x64 ruvector packages

Updates lockfile to include optional x64 platform packages for
CI compatibility:
- @ruvector/node-linux-x64-gnu@0.1.17
- ruvector-core-linux-x64-gnu@0.1.15

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: replace fake QUIC with proper implementation and clean up skills manifest

QUIC Transport:
- Remove fake HTTP/2-based QUICTransport.ts that pretended to be QUIC
- Add proper QUIC implementation via Rust/WASM (@agentic-flow/quic-wasm)
- Add WebSocket fallback transport for when WASM unavailable
- Add dedicated type declarations for optional WASM module
- Features: 0-RTT connection, stream multiplexing, TLS 1.3 built-in

Skills Manifest:
- Rewrite skills-manifest.json with only QE skills (41 skills)
- Remove Claude Flow platform skills (agentdb-*, flow-nexus-*, github-*, etc.)
- Organize into proper categories: qe-core, testing-methodologies, etc.

Bug Fixes:
- Fix UnifiedMemoryCoordinator.ts TypeScript errors
- Use Logger.getInstance() instead of private constructor
- Use swarmMemory.query() instead of non-existent list() method
- Use correct close() method signatures for backends

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add QX Partner agent, compact subagents, fix CLAUDE.md dynamic counts

- Add QX Partner agent for Quality Experience analysis combining QA + UX perspectives
- Compact 9 subagent definitions to ~80 lines each (88% reduction)
- Fix CLAUDE.md template to dynamically count agents (including subagents) and skills
- Move CLAUDE.md creation after parallel phases for accurate counts
- Remove 'Production-Ready' claim from README and CHANGELOG
- Clean up backup files and move docs to proper directory

Initialization now correctly reports:
- 31 QE Agents (20 main + 11 subagents)
- 53 QE Skills
- 8 Slash Commands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve CI failures and security alert #36

- Replace MD5 with SHA-256 for ETag calculation (security alert #36)
- Fix CRLF line endings in shell scripts causing parse errors
- Remove 'local' keyword from for-loop in test-integration-batched.sh
- Use dynamic process.cwd() instead of hardcoded path in jest.config.js
- Enhance quality-gate.ts to parse test logs when results file missing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Complete Agentic QE Fleet v2.1 implementation (closes #101)

## Implemented Features

### Error Recovery Enhancement (Task 1.3)
- CircuitBreaker class with state management (CLOSED/OPEN/HALF_OPEN)
- RetryStrategy with exponential/linear/constant backoff
- RecoveryOrchestrator with 6 recovery strategies
- Transport fallback mechanisms

### Dynamic Skill Loader (Task 2.2)
- Metadata-only initial loading (98% token reduction)
- On-demand full skill retrieval
- LRU caching with configurable TTL (30min default)
- Preload suggestions for critical workflows

### Swarm Learning Optimizer (Task 4.1)
- Topology recommendation engine (hierarchical/mesh/ring/star)
- Agent allocation with capability matching and load balancing
- Bottleneck detection (agent/memory/coordination/IO)
- Real-time performance adjustments
- Added getLatestRecommendation() method

### Workflow Orchestrator (Task 4.2)
- AdaptiveScheduler with dynamic strategy selection
- Priority-based task queue with age boosting
- Task dependency resolution
- Work stealing and backpressure handling
- Added getQueueDepth() and getStatus() methods

### Unified Memory Coordinator (Task 4.3)
- Consolidated memory managers (SQLite, AgentDB, Vector, JSON)
- Automatic failover chain with health monitoring
- Batch operations and namespace isolation
- Periodic sync between backends

### SwarmCoordinator Improvements
- Real CPU utilization tracking via process.cpuUsage()
- Queue depth from WorkflowOrchestrator
- Optimizer recommendations in feedback data
- Calculated average complexity from metrics

## Additional Changes
- Migration documentation and test restructuring plan
- GitHub issue templates and workflows
- Test suite improvements and cleanup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: optimize 38 QE skills with agent-focused format (#102)

- Add enhanced YAML frontmatter (tokenEstimate, agents, tags, etc.)
- Add <default_to_action> blocks for immediate actionable guidance
- Add Quick Reference Cards with tables for at-a-glance info
- Add Agent Integration examples using Task() pattern
- Add Agent Coordination Hints (memory namespace + fleet coordination)
- Condense content optimized for agent consumption
- Reduce total token footprint by ~65% across skills

Skills updated:
- Core QE: agentic-quality-engineering, holistic-testing-pact
- Testing Types: api-testing, chaos-engineering, compatibility, compliance,
  contract, database, exploratory, localization, mobile, mutation,
  performance, regression, security, visual-testing
- Methodologies: tdd-london-chicago, xp-practices, context-driven,
  risk-based, shift-left, shift-right
- Analysis: code-review-quality, quality-metrics, test-design-techniques
- Strategy: test-automation-strategy, test-data-management,
  test-environment-management, test-reporting-analytics
- CI/CD: cicd-pipeline-qe-orchestrator, accessibility-testing
- Communication: brutal-honesty-review, bug-reporting-excellence,
  consultancy-practices, sherlock-review, six-thinking-hats, technical-writing
- Refactoring: refactoring-patterns

Also includes: package.json updates, docs/optimization/, docs/templates/,
scripts/validate-skill-optimization.ts

Skipped: testability-scoring (pending PR from @fndlalit)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* release: v2.1.0 - Comprehensive QX Analysis & Skills Optimization

## Summary
- Comprehensive QX Analysis with 23+ heuristics (PR #104 by @fndlalit)
- 38 QE Skills optimized with agent-focused format (PR #102)
- Testability Scoring skill v2.1 with optimized format
- QX Partner Agent v2.1 with enhanced capabilities
- Fix MCP SDK compatibility for @modelcontextprotocol/sdk bump (PR #105)

## Changes
- .claude/agents/qx-partner.md: Updated to v2.1 with domain-specific detection
- .claude/skills/testability-scoring/SKILL.md: Optimized format (59% token reduction)
- src/mcp/server.ts: Removed deprecated 'description' field for MCP SDK compatibility
- package.json: Bumped version to 2.1.0
- README.md: Updated version and release notes
- CHANGELOG.md: Added v2.1.0 release notes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-03 11:07:46 +01:00

653 lines
23 KiB
Markdown

# Test Suite Migration: User-Centric Restructuring Plan
**Status**: 🟡 Planning Phase
**Priority**: High
**Effort**: ~40-60 hours (4-6 work sessions)
**Labels**: `enhancement`, `testing`, `technical-debt`, `breaking-change`
---
## 📊 Current State Analysis
### Metrics (as of 2025-12-02)
- **402 test files** with 195,527 lines of test code
- **7 skipped test blocks** (down from 599 - good progress!)
- **18 files exceed 1,000 lines** (monolithic, hard to maintain)
- **Test Focus Distribution**:
- 45% tests focus on implementation details ❌ (should be 10-15%)
- 35% tests focus on user value ✅ (should be 70-80%)
- 20% tests focus on contracts (acceptable)
### Critical Files Identified
#### DELETE (No User Value)
- `tests/unit/agents/TestGeneratorAgent.null-safety.test.ts` (40K) - Tests TypeScript semantics
- `tests/unit/agents/NeuralCapableMixin.skip.test.ts` (11K) - Already skipped, no implementation
#### CONSOLIDATE (Duplicate Coverage)
| File Pattern | Current Files | Current Lines | Target Files | Target Lines | Reduction |
|--------------|---------------|---------------|--------------|--------------|-----------|
| BaseAgent | 6 files | ~4,500 | 2 files | 1,200 | 73% |
| TestGeneratorAgent | 3 files | ~3,000 | 1 file | 800 | 73% |
| LearningEngine | 2 files | ~2,000 | 1 file | 600 | 70% |
| FlakyTestDetector | 2 files | ~1,500 | 1 file | 500 | 67% |
**Total Reduction**: ~8,000 lines → ~3,100 lines (61% reduction in duplicates)
---
## 🎯 Goal Definition
### Success Criteria
1. ✅ **7 Core User Journeys** pass with 100% reliability
2. ✅ **Test execution time** < 2 minutes (currently ~5 minutes)
3. ✅ **70% of tests** focus on user-facing behavior
4. ✅ **Zero skipped tests** in CI/CD pipeline
5. ✅ **File size limit**: No test file > 600 lines
6. ✅ **Test coverage maintained**: ≥85% for critical paths
### Key Principles (Testing Trophy)
```
/\
/ \ E2E (5%) - Critical user flows
/----\ Integration (15%) - Component interactions
/ \ Contract (10%) - API boundaries
/--------\ Unit (70%) - Business logic ONLY
```
**Focus**: Test user value, not implementation details.
---
## 📋 Phase Dependency Graph
```mermaid
graph TD
A[Phase 1: Setup & Analysis] --> B[Phase 2: Create Journey Tests]
B --> C[Phase 3: Consolidate Unit Tests]
C --> D[Phase 4: Delete Redundant Tests]
D --> E[Phase 5: CI/CD Integration]
style A fill:#e1f5ff
style B fill:#fff4e1
style C fill:#e1ffe1
style D fill:#ffe1e1
style E fill:#f0e1ff
```
---
## 🚀 Phase 1: Setup & Analysis (4-6 hours)
### Goal
Establish baseline metrics, create directory structure, and identify all files for migration.
### Actions
- [ ] **1.1 Create Migration Tracking** (30 min)
- **Files**: `docs/migration/progress-tracking.md`
- **Task**: Document current metrics, create tracking template
- **Dependencies**: None
- **Verification**:
```bash
cat docs/migration/progress-tracking.md | grep "Baseline"
```
- [ ] **1.2 Create New Directory Structure** (30 min)
- **Files**: Create empty directories
- **Task**:
```bash
mkdir -p tests/{journeys,contracts,infrastructure,regression/fixed-bugs}
touch tests/journeys/.gitkeep
touch tests/contracts/.gitkeep
touch tests/infrastructure/.gitkeep
touch tests/regression/fixed-bugs/.gitkeep
```
- **Dependencies**: None
- **Verification**: `ls -la tests/journeys tests/contracts tests/infrastructure`
- [ ] **1.3 Analyze Duplicate Test Coverage** (2 hours)
- **Files**: `scripts/analyze-test-duplicates.ts`
- **Task**: Create script to identify overlapping test cases across duplicate files
- **Output**: `docs/migration/duplicate-analysis.json`
- **Dependencies**: None
- **Verification**:
```bash
npx tsx scripts/analyze-test-duplicates.ts
cat docs/migration/duplicate-analysis.json | jq '.duplicates | length'
```
- [ ] **1.4 Create Test Mapping Document** (1 hour)
- **Files**: `docs/migration/test-mapping.md`
- **Task**: Map every existing test to its target location (journey/contract/infra/delete)
- **Dependencies**: 1.3
- **Verification**: `wc -l docs/migration/test-mapping.md` should show ~500 lines
- [ ] **1.5 Run Baseline Coverage Report** (30 min)
- **Task**:
```bash
npm run test:coverage:baseline
cp coverage/coverage-summary.json docs/migration/baseline-coverage.json
```
- **Dependencies**: None
- **Verification**: `cat docs/migration/baseline-coverage.json | jq '.total.lines.pct'`
- [ ] **1.6 Setup Migration CI Job** (1 hour)
- **Files**: `.github/workflows/migration-validation.yml`
- **Task**: Create CI job that runs only migrated tests (separate from main suite)
- **Dependencies**: 1.2
- **Verification**: Push to branch and check GitHub Actions
### Phase 1 Exit Criteria
- ✅ All directories created
- ✅ Baseline metrics documented
- ✅ Test mapping complete (every file accounted for)
- ✅ Migration CI job passes (even with empty tests)
---
## 🧪 Phase 2: Create Journey Tests (8-12 hours)
### Goal
Implement the 7 core user journeys with real database interactions and end-to-end validation.
### Actions
- [ ] **2.1 Journey 1: Init & Bootstrap** (2 hours)
- **Files**: `tests/journeys/init-and-bootstrap.test.ts`
- **Task**: Test `aqe init` command flow
- **Test Cases**:
- ✅ Creates config file at `.aqe/config.json`
- ✅ Initializes database tables (agents, memory, learning)
- ✅ Registers 18 agents in fleet
- ✅ Loads 41 skills
- ✅ Registers 8 slash commands
- ✅ Validates all agents can query database
- **Dependencies**: 1.2
- **Verification**:
```bash
npm run test:journeys -- init-and-bootstrap
# Should complete in < 5 seconds
```
- [ ] **2.2 Journey 2: Test Generation** (2 hours)
- **Files**: `tests/journeys/generate-tests.test.ts`
- **Task**: Test end-to-end test generation with real AI
- **Test Cases**:
- ✅ Accepts TypeScript source file
- ✅ Generates valid Jest test suite
- ✅ Tests are syntactically correct (TypeScript compiles)
- ✅ Generated tests can execute
- ✅ Coverage data is stored in database
- ✅ Patterns are saved for reuse
- **Dependencies**: 2.1
- **Verification**:
```bash
npm run test:journeys -- generate-tests
aqe learn status --agent test-gen | grep "patterns: [1-9]"
```
- [ ] **2.3 Journey 3: Test Execution** (2 hours)
- **Files**: `tests/journeys/execute-tests.test.ts`
- **Task**: Test parallel execution with real-time progress
- **Test Cases**:
- ✅ Executes tests in parallel (4 workers)
- ✅ Emits real-time progress events
- ✅ Retries flaky tests automatically (max 3 attempts)
- ✅ Load balances across workers
- ✅ Generates comprehensive report
- ✅ Stores results in database
- **Dependencies**: 2.2
- **Verification**:
```bash
npm run test:journeys -- execute-tests
# Execution time should be < 30 seconds for 100 tests
```
- [ ] **2.4 Journey 4: Coverage Analysis** (2 hours)
- **Files**: `tests/journeys/coverage-analysis.test.ts`
- **Task**: Test O(log n) gap detection with risk scoring
- **Test Cases**:
- ✅ Analyzes coverage data with Johnson-Lindenstrauss
- ✅ Identifies gaps in < 1 second for 10k LOC
- ✅ Prioritizes gaps by risk (complexity, change frequency)
- ✅ Generates specific test recommendations
- ✅ Stores analysis in database
- ✅ Visualizes gaps in HTML report
- **Dependencies**: 2.3
- **Verification**:
```bash
npm run test:journeys -- coverage-analysis
# Should use O(log n) algorithm
```
- [ ] **2.5 Journey 5: Quality Gate** (1.5 hours)
- **Files**: `tests/journeys/quality-gate.test.ts`
- **Task**: Test automated GO/NO-GO decisions
- **Test Cases**:
- ✅ Evaluates coverage, security, performance metrics
- ✅ Applies policy rules (min coverage 80%, no critical vulns)
- ✅ Generates pass/fail decision with rationale
- ✅ Blocks deployment on failure
- ✅ Creates audit trail in database
- **Dependencies**: 2.4
- **Verification**:
```bash
npm run test:journeys -- quality-gate
aqe quality-gate evaluate --build test-123 | grep "Decision: GO"
```
- [ ] **2.6 Journey 6: Flaky Detection** (1.5 hours)
- **Files**: `tests/journeys/flaky-detection.test.ts`
- **Task**: Test statistical flaky test detection
- **Test Cases**:
- ✅ Detects flaky tests with 100% accuracy (chi-square test)
- ✅ Identifies root causes (timing, race conditions, dependencies)
- ✅ Generates auto-fix recommendations
- ✅ Applies stabilization strategies (retry, wait, isolation)
- ✅ Stores patterns in database
- **Dependencies**: 2.3
- **Verification**:
```bash
npm run test:journeys -- flaky-detection
# Should detect synthetic flaky test in < 2 seconds
```
- [ ] **2.7 Journey 7: Learning & Improvement** (1 hour)
- **Files**: `tests/journeys/learning-improvement.test.ts`
- **Task**: Test Q-learning and pattern recognition
- **Test Cases**:
- ✅ Stores experiences in database (reward, outcome)
- ✅ Updates Q-values for state-action pairs
- ✅ Learns successful patterns over time
- ✅ Improves test generation quality (measured by coverage increase)
- ✅ Adapts to project-specific patterns
- **Dependencies**: 2.2, 2.6
- **Verification**:
```bash
npm run test:journeys -- learning-improvement
aqe learn status --agent test-gen | grep "Q-values: [1-9]"
```
### Phase 2 Exit Criteria
- ✅ All 7 journey tests pass with 100% reliability
- ✅ Journey tests use real database (no mocks)
- ✅ Average journey test execution < 10 seconds each
- ✅ Migration CI job passes all journey tests
---
## 🔧 Phase 3: Consolidate Unit Tests (12-16 hours)
### Goal
Merge duplicate test files, eliminate redundant tests, focus on business logic.
### Actions
- [ ] **3.1 Consolidate BaseAgent Tests** (3 hours)
- **Current Files** (6 total):
- `tests/unit/agents/BaseAgent.test.ts` (36K)
- `tests/unit/agents/BaseAgent.enhanced.test.ts` (26K)
- `tests/unit/agents/BaseAgent.comprehensive.test.ts` (22K)
- `tests/integration/agentdb/BaseAgentIntegration.test.ts`
- `tests/agents/BaseAgent.test.ts`
- `tests/agents/BaseAgent.lifecycle.test.ts`
- **Target**: 2 files (1,200 lines total)
- `tests/infrastructure/base-agent.test.ts` (600 lines) - Core functionality
- `tests/contracts/base-agent-interface.test.ts` (600 lines) - Public API
- **Task**: Merge unique tests, delete duplicates (70% overlap)
- **Dependencies**: 2.7
- **Verification**:
```bash
npm run test:unit -- base-agent
# Should complete in < 3 seconds
```
- [ ] **3.2 Consolidate TestGeneratorAgent Tests** (3 hours)
- **Current Files** (3 total):
- `tests/unit/agents/TestGeneratorAgent.comprehensive.test.ts` (41K)
- `tests/unit/agents/TestGeneratorAgent.test.ts` (21K)
- `tests/agents/TestGeneratorAgent.test.ts`
- **Target**: 1 file (800 lines)
- `tests/infrastructure/test-generator-agent.test.ts`
- **Task**: Focus on core generation logic, remove implementation details
- **Dependencies**: 2.2, 3.1
- **Verification**:
```bash
npm run test:unit -- test-generator-agent
# Should complete in < 3 seconds
```
- [ ] **3.3 Consolidate LearningEngine Tests** (2 hours)
- **Current Files** (2 total):
- `tests/unit/learning/LearningEngine.test.ts`
- `tests/unit/learning/LearningEngine.database.test.ts`
- **Target**: 1 file (600 lines)
- `tests/infrastructure/learning-engine.test.ts`
- **Task**: Focus on Q-learning algorithm, pattern storage
- **Dependencies**: 2.7, 3.1
- **Verification**:
```bash
npm run test:unit -- learning-engine
```
- [ ] **3.4 Consolidate FlakyTestDetector Tests** (2 hours)
- **Current Files** (2 total):
- `tests/unit/learning/FlakyTestDetector.test.ts`
- `tests/unit/learning/FlakyTestDetector.ml.test.ts`
- **Target**: 1 file (500 lines)
- `tests/infrastructure/flaky-test-detector.test.ts`
- **Task**: Focus on statistical detection algorithms
- **Dependencies**: 2.6, 3.3
- **Verification**:
```bash
npm run test:unit -- flaky-test-detector
```
- [ ] **3.5 Create Contract Tests** (2 hours)
- **Files**:
- `tests/contracts/mcp-tools.test.ts` (102 tools)
- `tests/contracts/cli-commands.test.ts` (8 commands)
- `tests/contracts/agent-interfaces.test.ts` (18 agents)
- **Task**: Validate public APIs don't break
- **Test Strategy**: JSON Schema validation, not full execution
- **Dependencies**: 3.4
- **Verification**:
```bash
npm run test:contracts
# Should complete in < 5 seconds (fast validation)
```
### Phase 3 Exit Criteria
- ✅ All consolidated tests pass
- ✅ No test file > 600 lines
- ✅ Unit test execution time < 30 seconds total
- ✅ Contract tests validate all 102 MCP tools + 8 commands + 18 agents
- ✅ Test coverage maintained ≥85% for critical paths
---
## 🗑️ Phase 4: Delete Redundant Tests (4-6 hours)
### Goal
Remove tests that don't provide user value, eliminate duplicates.
### Actions
- [ ] **4.1 Delete Implementation Detail Tests** (1 hour)
- **Files to DELETE**:
- `tests/unit/agents/TestGeneratorAgent.null-safety.test.ts` (40K)
- `tests/unit/agents/NeuralCapableMixin.skip.test.ts` (11K)
- **Reason**: Tests TypeScript compiler behavior, not business logic
- **Task**:
```bash
git rm tests/unit/agents/TestGeneratorAgent.null-safety.test.ts
git rm tests/unit/agents/NeuralCapableMixin.skip.test.ts
```
- **Dependencies**: 3.5
- **Verification**:
```bash
npm run test:all
# Coverage should not drop (these tests didn't add value)
```
- [ ] **4.2 Delete Duplicate BaseAgent Tests** (1 hour)
- **Files to DELETE** (after consolidation in 3.1):
- `tests/unit/agents/BaseAgent.enhanced.test.ts`
- `tests/unit/agents/BaseAgent.comprehensive.test.ts`
- `tests/agents/BaseAgent.lifecycle.test.ts`
- `tests/agents/BaseAgent.edge-cases.test.ts`
- `tests/agents/BaseAgent.race-condition.test.ts`
- **Task**: Delete old files, update imports in remaining tests
- **Dependencies**: 3.1
- **Verification**:
```bash
find tests -name "*BaseAgent*.test.ts" | wc -l
# Should be exactly 2 files
```
- [ ] **4.3 Delete Duplicate TestGeneratorAgent Tests** (1 hour)
- **Files to DELETE** (after consolidation in 3.2):
- `tests/unit/agents/TestGeneratorAgent.comprehensive.test.ts`
- `tests/agents/TestGeneratorAgent.test.ts`
- **Dependencies**: 3.2
- **Verification**:
```bash
find tests -name "*TestGeneratorAgent*.test.ts" | wc -l
# Should be exactly 1 file
```
- [ ] **4.4 Delete Duplicate Learning Tests** (1 hour)
- **Files to DELETE** (after consolidation in 3.3, 3.4):
- `tests/unit/learning/LearningEngine.database.test.ts`
- `tests/unit/learning/FlakyTestDetector.ml.test.ts`
- **Dependencies**: 3.3, 3.4
- **Verification**:
```bash
ls tests/unit/learning/*.test.ts | wc -l
# Should be 0 (all moved to infrastructure/)
```
- [ ] **4.5 Clean Up Old Directory Structure** (30 min)
- **Directories to DELETE**:
- `tests/unit/` (if empty)
- `tests/integration/` (if empty)
- `tests/agents/` (all migrated)
- **Task**:
```bash
# Only delete if empty
find tests/unit tests/integration tests/agents -type d -empty -delete
```
- **Dependencies**: 4.4
- **Verification**:
```bash
tree tests -L 1
# Should show only: journeys, contracts, infrastructure, regression
```
- [ ] **4.6 Update Test Configuration** (1 hour)
- **Files**:
- `jest.config.js`
- `package.json` (test scripts)
- `.github/workflows/test.yml`
- **Task**: Update paths to reflect new structure
- **Dependencies**: 4.5
- **Verification**:
```bash
npm run test:all
# All tests should still pass
```
### Phase 4 Exit Criteria
- ✅ ~60K lines of test code deleted
- ✅ No duplicate test files remain
- ✅ Old directory structure removed
- ✅ All CI/CD pipelines still pass
---
## 🔄 Phase 5: CI/CD Integration (4-6 hours)
### Goal
Optimize test execution for fast feedback, integrate with GitHub Actions.
### Actions
- [ ] **5.1 Create Optimized Test Scripts** (1 hour)
- **Files**: `package.json`
- **Task**: Add fast-feedback test scripts
- **Scripts**:
```json
{
"test:journeys": "jest tests/journeys --maxWorkers=4",
"test:contracts": "jest tests/contracts --maxWorkers=2",
"test:infrastructure": "jest tests/infrastructure --maxWorkers=4",
"test:regression": "jest tests/regression --maxWorkers=2",
"test:fast": "npm run test:journeys && npm run test:contracts",
"test:all": "jest --maxWorkers=4"
}
```
- **Dependencies**: 4.6
- **Verification**:
```bash
npm run test:fast
# Should complete in < 30 seconds
```
- [ ] **5.2 Update GitHub Actions Workflow** (1.5 hours)
- **Files**: `.github/workflows/test.yml`
- **Task**: Optimize CI pipeline with parallel jobs
- **Strategy**:
```yaml
jobs:
journeys:
runs-on: ubuntu-latest
steps:
- run: npm run test:journeys
contracts:
runs-on: ubuntu-latest
steps:
- run: npm run test:contracts
infrastructure:
runs-on: ubuntu-latest
steps:
- run: npm run test:infrastructure
```
- **Dependencies**: 5.1
- **Verification**: Push to branch and check Actions tab
- [ ] **5.3 Add Test Coverage Gates** (1 hour)
- **Files**: `.github/workflows/test.yml`
- **Task**: Enforce 85% coverage minimum
- **Config**:
```yaml
- name: Check Coverage
run: |
npm run test:coverage
npx coverage-check --lines 85 --branches 80
```
- **Dependencies**: 5.2
- **Verification**: Create PR with low coverage, should fail
- [ ] **5.4 Create Test Dashboard** (1.5 hours)
- **Files**: `docs/test-dashboard.md`
- **Task**: Generate markdown dashboard with metrics
- **Metrics**:
- Total tests by category (journeys, contracts, infrastructure)
- Execution time per category
- Coverage by module
- Flaky test count
- **Dependencies**: 5.3
- **Verification**:
```bash
npm run test:dashboard
cat docs/test-dashboard.md
```
- [ ] **5.5 Update Documentation** (1 hour)
- **Files**:
- `README.md`
- `docs/testing/README.md`
- `docs/testing/writing-tests.md`
- **Task**: Document new test structure and best practices
- **Dependencies**: 5.4
- **Verification**: PR review passes
### Phase 5 Exit Criteria
- ✅ CI/CD pipeline runs in < 2 minutes (down from ~5 minutes)
- ✅ Coverage gates enforced automatically
- ✅ Test dashboard generated on every run
- ✅ Documentation updated and reviewed
---
## ⚠️ Risks & Mitigations
| Risk | Impact | Probability | Mitigation |
|------|--------|-------------|------------|
| **Coverage drops during consolidation** | High | Medium | Run coverage report after each consolidation step (Phase 3). Rollback if coverage drops > 2%. |
| **Journey tests are flaky** | High | Medium | Use real database with test isolation. Add retry logic. Run 10x locally before committing. |
| **Migration takes longer than estimated** | Medium | High | Break into smaller PRs (one phase at a time). Use TodoWrite to track progress. |
| **Existing bugs exposed** | Medium | Low | Expected! Create regression tests for discovered bugs. Don't block migration. |
| **Team members blocked** | Low | Medium | Keep old tests working during migration. Delete only after new tests pass. |
---
## 📈 Success Metrics
### Quantitative Metrics
| Metric | Baseline | Target | Current |
|--------|----------|--------|---------|
| **Test Files** | 402 | 50 | 402 |
| **Test Lines** | 195,527 | 40,000 | 195,527 |
| **Test Execution Time** | ~5 min | < 2 min | ~5 min |
| **Files > 1,000 Lines** | 18 | 0 | 18 |
| **Skipped Tests** | 7 | 0 | 7 |
| **Coverage (Critical Paths)** | 85% | ≥85% | 85% |
### Qualitative Metrics
- ✅ **Developer Confidence**: "I trust the test suite to catch regressions"
- ✅ **Onboarding Speed**: New contributors understand tests in < 1 hour
- ✅ **Maintenance Burden**: Test updates take < 10% of feature development time
- ✅ **CI/CD Reliability**: Zero flaky failures in 30 days
---
## 📝 Phase Checkpoints
### After Phase 1
- [ ] All team members can navigate new test structure
- [ ] Baseline metrics documented and agreed upon
- [ ] Migration CI job passes (empty tests)
### After Phase 2
- [ ] Product team validates journey tests match user expectations
- [ ] All journey tests run against staging database
- [ ] Journey tests documented in user-facing docs
### After Phase 3
- [ ] No regressions introduced (all existing tests still pass)
- [ ] Coverage maintained or improved
- [ ] Team reviews consolidated tests (code review)
### After Phase 4
- [ ] Old directory structure deleted
- [ ] CI/CD pipelines updated
- [ ] No broken imports or references
### After Phase 5
- [ ] CI/CD runs in < 2 minutes
- [ ] Coverage gates enforced
- [ ] Documentation complete and reviewed
---
## 🚀 Getting Started
1. **Create Tracking Issue**: Copy this document to GitHub Issues
2. **Assign Owner**: Designate migration lead
3. **Kick-off Meeting**: Review plan with team (30 min)
4. **Start Phase 1**: Begin with setup and analysis
5. **Daily Updates**: Update tracking issue checkboxes
6. **Phase Reviews**: Review with team after each phase
---
## 📚 References
- [Testing Trophy](https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications)
- [Google Testing Blog: Test Sizes](https://testing.googleblog.com/2010/12/test-sizes.html)
- [Martin Fowler: Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html)
- [Project: Test Execution Policy](file:///workspaces/agentic-qe-cf/docs/policies/test-execution.md)
- [Project: Git Operations Policy](file:///workspaces/agentic-qe-cf/docs/policies/git-operations.md)
---
**Migration Lead**: TBD
**Start Date**: TBD
**Target Completion**: TBD (4-6 weeks, 1 phase per week)
---
_Generated by GOAP Specialist • Last Updated: 2025-12-02_