mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
1e3fbb5ea8
* SwarmMemoryManager Refactoring Plan * fix: add missing directories to npm package (v1.9.3) Added templates/, .claude/helpers/, and docs/reference/ to package.json files array. These were missing from the npm distribution, causing `aqe init` to fail with "Template file not found" errors. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * checkpoint improvements * fix: critical memory/learning system fixes for v1.9.4 ## Critical Fixes - Vector embeddings now stored correctly (was storing NULL) - SQL parameter style fixed for agentdb's SqlJsDatabase wrapper - HNSW index schema mismatch resolved with pattern_id column - Hooks now save to correct database (.agentic-qe/agentdb.db) ## New Features - Added 5 SwarmMemoryManager methods for learning data retrieval: - getBestAction() - Q-learning best action selection - getRecentLearningExperiences() - Recent experience retrieval - getLearningExperiencesByTaskType() - Task-filtered experiences - getHighRewardExperiences() - Successful experience extraction - getLearningStats() - Aggregate learning statistics - Added comprehensive integration test for memory/learning loop - Added RuVector pattern store integration (experimental) ## Technical Details - Vector embeddings: 384 dimensions × 4 bytes = 1,536 bytes per pattern - All 12 integration tests pass - HNSW index: 150x faster vector search enabled 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: move ARM64-only ruvector packages to optionalDependencies Fixes CI failures caused by platform-specific dependencies: - @ruvector/node-linux-arm64-gnu (ARM64 only) - ruvector-core-linux-arm64-gnu (ARM64 only) These packages caused npm ci to fail on x64 CI runners with: "EBADPLATFORM - wanted linux/arm64, current linux/x64" Solution: Move ARM64 packages to optionalDependencies and add x64 variants for cross-platform compatibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: regenerate package-lock.json with x64 ruvector packages Updates lockfile to include optional x64 platform packages for CI compatibility: - @ruvector/node-linux-x64-gnu@0.1.17 - ruvector-core-linux-x64-gnu@0.1.15 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: replace fake QUIC with proper implementation and clean up skills manifest QUIC Transport: - Remove fake HTTP/2-based QUICTransport.ts that pretended to be QUIC - Add proper QUIC implementation via Rust/WASM (@agentic-flow/quic-wasm) - Add WebSocket fallback transport for when WASM unavailable - Add dedicated type declarations for optional WASM module - Features: 0-RTT connection, stream multiplexing, TLS 1.3 built-in Skills Manifest: - Rewrite skills-manifest.json with only QE skills (41 skills) - Remove Claude Flow platform skills (agentdb-*, flow-nexus-*, github-*, etc.) - Organize into proper categories: qe-core, testing-methodologies, etc. Bug Fixes: - Fix UnifiedMemoryCoordinator.ts TypeScript errors - Use Logger.getInstance() instead of private constructor - Use swarmMemory.query() instead of non-existent list() method - Use correct close() method signatures for backends 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add QX Partner agent, compact subagents, fix CLAUDE.md dynamic counts - Add QX Partner agent for Quality Experience analysis combining QA + UX perspectives - Compact 9 subagent definitions to ~80 lines each (88% reduction) - Fix CLAUDE.md template to dynamically count agents (including subagents) and skills - Move CLAUDE.md creation after parallel phases for accurate counts - Remove 'Production-Ready' claim from README and CHANGELOG - Clean up backup files and move docs to proper directory Initialization now correctly reports: - 31 QE Agents (20 main + 11 subagents) - 53 QE Skills - 8 Slash Commands 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve CI failures and security alert #36 - Replace MD5 with SHA-256 for ETag calculation (security alert #36) - Fix CRLF line endings in shell scripts causing parse errors - Remove 'local' keyword from for-loop in test-integration-batched.sh - Use dynamic process.cwd() instead of hardcoded path in jest.config.js - Enhance quality-gate.ts to parse test logs when results file missing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Complete Agentic QE Fleet v2.1 implementation (closes #101) ## Implemented Features ### Error Recovery Enhancement (Task 1.3) - CircuitBreaker class with state management (CLOSED/OPEN/HALF_OPEN) - RetryStrategy with exponential/linear/constant backoff - RecoveryOrchestrator with 6 recovery strategies - Transport fallback mechanisms ### Dynamic Skill Loader (Task 2.2) - Metadata-only initial loading (98% token reduction) - On-demand full skill retrieval - LRU caching with configurable TTL (30min default) - Preload suggestions for critical workflows ### Swarm Learning Optimizer (Task 4.1) - Topology recommendation engine (hierarchical/mesh/ring/star) - Agent allocation with capability matching and load balancing - Bottleneck detection (agent/memory/coordination/IO) - Real-time performance adjustments - Added getLatestRecommendation() method ### Workflow Orchestrator (Task 4.2) - AdaptiveScheduler with dynamic strategy selection - Priority-based task queue with age boosting - Task dependency resolution - Work stealing and backpressure handling - Added getQueueDepth() and getStatus() methods ### Unified Memory Coordinator (Task 4.3) - Consolidated memory managers (SQLite, AgentDB, Vector, JSON) - Automatic failover chain with health monitoring - Batch operations and namespace isolation - Periodic sync between backends ### SwarmCoordinator Improvements - Real CPU utilization tracking via process.cpuUsage() - Queue depth from WorkflowOrchestrator - Optimizer recommendations in feedback data - Calculated average complexity from metrics ## Additional Changes - Migration documentation and test restructuring plan - GitHub issue templates and workflows - Test suite improvements and cleanup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: optimize 38 QE skills with agent-focused format (#102) - Add enhanced YAML frontmatter (tokenEstimate, agents, tags, etc.) - Add <default_to_action> blocks for immediate actionable guidance - Add Quick Reference Cards with tables for at-a-glance info - Add Agent Integration examples using Task() pattern - Add Agent Coordination Hints (memory namespace + fleet coordination) - Condense content optimized for agent consumption - Reduce total token footprint by ~65% across skills Skills updated: - Core QE: agentic-quality-engineering, holistic-testing-pact - Testing Types: api-testing, chaos-engineering, compatibility, compliance, contract, database, exploratory, localization, mobile, mutation, performance, regression, security, visual-testing - Methodologies: tdd-london-chicago, xp-practices, context-driven, risk-based, shift-left, shift-right - Analysis: code-review-quality, quality-metrics, test-design-techniques - Strategy: test-automation-strategy, test-data-management, test-environment-management, test-reporting-analytics - CI/CD: cicd-pipeline-qe-orchestrator, accessibility-testing - Communication: brutal-honesty-review, bug-reporting-excellence, consultancy-practices, sherlock-review, six-thinking-hats, technical-writing - Refactoring: refactoring-patterns Also includes: package.json updates, docs/optimization/, docs/templates/, scripts/validate-skill-optimization.ts Skipped: testability-scoring (pending PR from @fndlalit) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * release: v2.1.0 - Comprehensive QX Analysis & Skills Optimization ## Summary - Comprehensive QX Analysis with 23+ heuristics (PR #104 by @fndlalit) - 38 QE Skills optimized with agent-focused format (PR #102) - Testability Scoring skill v2.1 with optimized format - QX Partner Agent v2.1 with enhanced capabilities - Fix MCP SDK compatibility for @modelcontextprotocol/sdk bump (PR #105) ## Changes - .claude/agents/qx-partner.md: Updated to v2.1 with domain-specific detection - .claude/skills/testability-scoring/SKILL.md: Optimized format (59% token reduction) - src/mcp/server.ts: Removed deprecated 'description' field for MCP SDK compatibility - package.json: Bumped version to 2.1.0 - README.md: Updated version and release notes - CHANGELOG.md: Added v2.1.0 release notes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
653 lines
23 KiB
Markdown
653 lines
23 KiB
Markdown
# Test Suite Migration: User-Centric Restructuring Plan
|
|
|
|
**Status**: 🟡 Planning Phase
|
|
**Priority**: High
|
|
**Effort**: ~40-60 hours (4-6 work sessions)
|
|
**Labels**: `enhancement`, `testing`, `technical-debt`, `breaking-change`
|
|
|
|
---
|
|
|
|
## 📊 Current State Analysis
|
|
|
|
### Metrics (as of 2025-12-02)
|
|
- **402 test files** with 195,527 lines of test code
|
|
- **7 skipped test blocks** (down from 599 - good progress!)
|
|
- **18 files exceed 1,000 lines** (monolithic, hard to maintain)
|
|
- **Test Focus Distribution**:
|
|
- 45% tests focus on implementation details ❌ (should be 10-15%)
|
|
- 35% tests focus on user value ✅ (should be 70-80%)
|
|
- 20% tests focus on contracts (acceptable)
|
|
|
|
### Critical Files Identified
|
|
|
|
#### DELETE (No User Value)
|
|
- `tests/unit/agents/TestGeneratorAgent.null-safety.test.ts` (40K) - Tests TypeScript semantics
|
|
- `tests/unit/agents/NeuralCapableMixin.skip.test.ts` (11K) - Already skipped, no implementation
|
|
|
|
#### CONSOLIDATE (Duplicate Coverage)
|
|
| File Pattern | Current Files | Current Lines | Target Files | Target Lines | Reduction |
|
|
|--------------|---------------|---------------|--------------|--------------|-----------|
|
|
| BaseAgent | 6 files | ~4,500 | 2 files | 1,200 | 73% |
|
|
| TestGeneratorAgent | 3 files | ~3,000 | 1 file | 800 | 73% |
|
|
| LearningEngine | 2 files | ~2,000 | 1 file | 600 | 70% |
|
|
| FlakyTestDetector | 2 files | ~1,500 | 1 file | 500 | 67% |
|
|
|
|
**Total Reduction**: ~8,000 lines → ~3,100 lines (61% reduction in duplicates)
|
|
|
|
---
|
|
|
|
## 🎯 Goal Definition
|
|
|
|
### Success Criteria
|
|
1. ✅ **7 Core User Journeys** pass with 100% reliability
|
|
2. ✅ **Test execution time** < 2 minutes (currently ~5 minutes)
|
|
3. ✅ **70% of tests** focus on user-facing behavior
|
|
4. ✅ **Zero skipped tests** in CI/CD pipeline
|
|
5. ✅ **File size limit**: No test file > 600 lines
|
|
6. ✅ **Test coverage maintained**: ≥85% for critical paths
|
|
|
|
### Key Principles (Testing Trophy)
|
|
```
|
|
/\
|
|
/ \ E2E (5%) - Critical user flows
|
|
/----\ Integration (15%) - Component interactions
|
|
/ \ Contract (10%) - API boundaries
|
|
/--------\ Unit (70%) - Business logic ONLY
|
|
```
|
|
|
|
**Focus**: Test user value, not implementation details.
|
|
|
|
---
|
|
|
|
## 📋 Phase Dependency Graph
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Phase 1: Setup & Analysis] --> B[Phase 2: Create Journey Tests]
|
|
B --> C[Phase 3: Consolidate Unit Tests]
|
|
C --> D[Phase 4: Delete Redundant Tests]
|
|
D --> E[Phase 5: CI/CD Integration]
|
|
|
|
style A fill:#e1f5ff
|
|
style B fill:#fff4e1
|
|
style C fill:#e1ffe1
|
|
style D fill:#ffe1e1
|
|
style E fill:#f0e1ff
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 Phase 1: Setup & Analysis (4-6 hours)
|
|
|
|
### Goal
|
|
Establish baseline metrics, create directory structure, and identify all files for migration.
|
|
|
|
### Actions
|
|
|
|
- [ ] **1.1 Create Migration Tracking** (30 min)
|
|
- **Files**: `docs/migration/progress-tracking.md`
|
|
- **Task**: Document current metrics, create tracking template
|
|
- **Dependencies**: None
|
|
- **Verification**:
|
|
```bash
|
|
cat docs/migration/progress-tracking.md | grep "Baseline"
|
|
```
|
|
|
|
- [ ] **1.2 Create New Directory Structure** (30 min)
|
|
- **Files**: Create empty directories
|
|
- **Task**:
|
|
```bash
|
|
mkdir -p tests/{journeys,contracts,infrastructure,regression/fixed-bugs}
|
|
touch tests/journeys/.gitkeep
|
|
touch tests/contracts/.gitkeep
|
|
touch tests/infrastructure/.gitkeep
|
|
touch tests/regression/fixed-bugs/.gitkeep
|
|
```
|
|
- **Dependencies**: None
|
|
- **Verification**: `ls -la tests/journeys tests/contracts tests/infrastructure`
|
|
|
|
- [ ] **1.3 Analyze Duplicate Test Coverage** (2 hours)
|
|
- **Files**: `scripts/analyze-test-duplicates.ts`
|
|
- **Task**: Create script to identify overlapping test cases across duplicate files
|
|
- **Output**: `docs/migration/duplicate-analysis.json`
|
|
- **Dependencies**: None
|
|
- **Verification**:
|
|
```bash
|
|
npx tsx scripts/analyze-test-duplicates.ts
|
|
cat docs/migration/duplicate-analysis.json | jq '.duplicates | length'
|
|
```
|
|
|
|
- [ ] **1.4 Create Test Mapping Document** (1 hour)
|
|
- **Files**: `docs/migration/test-mapping.md`
|
|
- **Task**: Map every existing test to its target location (journey/contract/infra/delete)
|
|
- **Dependencies**: 1.3
|
|
- **Verification**: `wc -l docs/migration/test-mapping.md` should show ~500 lines
|
|
|
|
- [ ] **1.5 Run Baseline Coverage Report** (30 min)
|
|
- **Task**:
|
|
```bash
|
|
npm run test:coverage:baseline
|
|
cp coverage/coverage-summary.json docs/migration/baseline-coverage.json
|
|
```
|
|
- **Dependencies**: None
|
|
- **Verification**: `cat docs/migration/baseline-coverage.json | jq '.total.lines.pct'`
|
|
|
|
- [ ] **1.6 Setup Migration CI Job** (1 hour)
|
|
- **Files**: `.github/workflows/migration-validation.yml`
|
|
- **Task**: Create CI job that runs only migrated tests (separate from main suite)
|
|
- **Dependencies**: 1.2
|
|
- **Verification**: Push to branch and check GitHub Actions
|
|
|
|
### Phase 1 Exit Criteria
|
|
- ✅ All directories created
|
|
- ✅ Baseline metrics documented
|
|
- ✅ Test mapping complete (every file accounted for)
|
|
- ✅ Migration CI job passes (even with empty tests)
|
|
|
|
---
|
|
|
|
## 🧪 Phase 2: Create Journey Tests (8-12 hours)
|
|
|
|
### Goal
|
|
Implement the 7 core user journeys with real database interactions and end-to-end validation.
|
|
|
|
### Actions
|
|
|
|
- [ ] **2.1 Journey 1: Init & Bootstrap** (2 hours)
|
|
- **Files**: `tests/journeys/init-and-bootstrap.test.ts`
|
|
- **Task**: Test `aqe init` command flow
|
|
- **Test Cases**:
|
|
- ✅ Creates config file at `.aqe/config.json`
|
|
- ✅ Initializes database tables (agents, memory, learning)
|
|
- ✅ Registers 18 agents in fleet
|
|
- ✅ Loads 41 skills
|
|
- ✅ Registers 8 slash commands
|
|
- ✅ Validates all agents can query database
|
|
- **Dependencies**: 1.2
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- init-and-bootstrap
|
|
# Should complete in < 5 seconds
|
|
```
|
|
|
|
- [ ] **2.2 Journey 2: Test Generation** (2 hours)
|
|
- **Files**: `tests/journeys/generate-tests.test.ts`
|
|
- **Task**: Test end-to-end test generation with real AI
|
|
- **Test Cases**:
|
|
- ✅ Accepts TypeScript source file
|
|
- ✅ Generates valid Jest test suite
|
|
- ✅ Tests are syntactically correct (TypeScript compiles)
|
|
- ✅ Generated tests can execute
|
|
- ✅ Coverage data is stored in database
|
|
- ✅ Patterns are saved for reuse
|
|
- **Dependencies**: 2.1
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- generate-tests
|
|
aqe learn status --agent test-gen | grep "patterns: [1-9]"
|
|
```
|
|
|
|
- [ ] **2.3 Journey 3: Test Execution** (2 hours)
|
|
- **Files**: `tests/journeys/execute-tests.test.ts`
|
|
- **Task**: Test parallel execution with real-time progress
|
|
- **Test Cases**:
|
|
- ✅ Executes tests in parallel (4 workers)
|
|
- ✅ Emits real-time progress events
|
|
- ✅ Retries flaky tests automatically (max 3 attempts)
|
|
- ✅ Load balances across workers
|
|
- ✅ Generates comprehensive report
|
|
- ✅ Stores results in database
|
|
- **Dependencies**: 2.2
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- execute-tests
|
|
# Execution time should be < 30 seconds for 100 tests
|
|
```
|
|
|
|
- [ ] **2.4 Journey 4: Coverage Analysis** (2 hours)
|
|
- **Files**: `tests/journeys/coverage-analysis.test.ts`
|
|
- **Task**: Test O(log n) gap detection with risk scoring
|
|
- **Test Cases**:
|
|
- ✅ Analyzes coverage data with Johnson-Lindenstrauss
|
|
- ✅ Identifies gaps in < 1 second for 10k LOC
|
|
- ✅ Prioritizes gaps by risk (complexity, change frequency)
|
|
- ✅ Generates specific test recommendations
|
|
- ✅ Stores analysis in database
|
|
- ✅ Visualizes gaps in HTML report
|
|
- **Dependencies**: 2.3
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- coverage-analysis
|
|
# Should use O(log n) algorithm
|
|
```
|
|
|
|
- [ ] **2.5 Journey 5: Quality Gate** (1.5 hours)
|
|
- **Files**: `tests/journeys/quality-gate.test.ts`
|
|
- **Task**: Test automated GO/NO-GO decisions
|
|
- **Test Cases**:
|
|
- ✅ Evaluates coverage, security, performance metrics
|
|
- ✅ Applies policy rules (min coverage 80%, no critical vulns)
|
|
- ✅ Generates pass/fail decision with rationale
|
|
- ✅ Blocks deployment on failure
|
|
- ✅ Creates audit trail in database
|
|
- **Dependencies**: 2.4
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- quality-gate
|
|
aqe quality-gate evaluate --build test-123 | grep "Decision: GO"
|
|
```
|
|
|
|
- [ ] **2.6 Journey 6: Flaky Detection** (1.5 hours)
|
|
- **Files**: `tests/journeys/flaky-detection.test.ts`
|
|
- **Task**: Test statistical flaky test detection
|
|
- **Test Cases**:
|
|
- ✅ Detects flaky tests with 100% accuracy (chi-square test)
|
|
- ✅ Identifies root causes (timing, race conditions, dependencies)
|
|
- ✅ Generates auto-fix recommendations
|
|
- ✅ Applies stabilization strategies (retry, wait, isolation)
|
|
- ✅ Stores patterns in database
|
|
- **Dependencies**: 2.3
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- flaky-detection
|
|
# Should detect synthetic flaky test in < 2 seconds
|
|
```
|
|
|
|
- [ ] **2.7 Journey 7: Learning & Improvement** (1 hour)
|
|
- **Files**: `tests/journeys/learning-improvement.test.ts`
|
|
- **Task**: Test Q-learning and pattern recognition
|
|
- **Test Cases**:
|
|
- ✅ Stores experiences in database (reward, outcome)
|
|
- ✅ Updates Q-values for state-action pairs
|
|
- ✅ Learns successful patterns over time
|
|
- ✅ Improves test generation quality (measured by coverage increase)
|
|
- ✅ Adapts to project-specific patterns
|
|
- **Dependencies**: 2.2, 2.6
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:journeys -- learning-improvement
|
|
aqe learn status --agent test-gen | grep "Q-values: [1-9]"
|
|
```
|
|
|
|
### Phase 2 Exit Criteria
|
|
- ✅ All 7 journey tests pass with 100% reliability
|
|
- ✅ Journey tests use real database (no mocks)
|
|
- ✅ Average journey test execution < 10 seconds each
|
|
- ✅ Migration CI job passes all journey tests
|
|
|
|
---
|
|
|
|
## 🔧 Phase 3: Consolidate Unit Tests (12-16 hours)
|
|
|
|
### Goal
|
|
Merge duplicate test files, eliminate redundant tests, focus on business logic.
|
|
|
|
### Actions
|
|
|
|
- [ ] **3.1 Consolidate BaseAgent Tests** (3 hours)
|
|
- **Current Files** (6 total):
|
|
- `tests/unit/agents/BaseAgent.test.ts` (36K)
|
|
- `tests/unit/agents/BaseAgent.enhanced.test.ts` (26K)
|
|
- `tests/unit/agents/BaseAgent.comprehensive.test.ts` (22K)
|
|
- `tests/integration/agentdb/BaseAgentIntegration.test.ts`
|
|
- `tests/agents/BaseAgent.test.ts`
|
|
- `tests/agents/BaseAgent.lifecycle.test.ts`
|
|
- **Target**: 2 files (1,200 lines total)
|
|
- `tests/infrastructure/base-agent.test.ts` (600 lines) - Core functionality
|
|
- `tests/contracts/base-agent-interface.test.ts` (600 lines) - Public API
|
|
- **Task**: Merge unique tests, delete duplicates (70% overlap)
|
|
- **Dependencies**: 2.7
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:unit -- base-agent
|
|
# Should complete in < 3 seconds
|
|
```
|
|
|
|
- [ ] **3.2 Consolidate TestGeneratorAgent Tests** (3 hours)
|
|
- **Current Files** (3 total):
|
|
- `tests/unit/agents/TestGeneratorAgent.comprehensive.test.ts` (41K)
|
|
- `tests/unit/agents/TestGeneratorAgent.test.ts` (21K)
|
|
- `tests/agents/TestGeneratorAgent.test.ts`
|
|
- **Target**: 1 file (800 lines)
|
|
- `tests/infrastructure/test-generator-agent.test.ts`
|
|
- **Task**: Focus on core generation logic, remove implementation details
|
|
- **Dependencies**: 2.2, 3.1
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:unit -- test-generator-agent
|
|
# Should complete in < 3 seconds
|
|
```
|
|
|
|
- [ ] **3.3 Consolidate LearningEngine Tests** (2 hours)
|
|
- **Current Files** (2 total):
|
|
- `tests/unit/learning/LearningEngine.test.ts`
|
|
- `tests/unit/learning/LearningEngine.database.test.ts`
|
|
- **Target**: 1 file (600 lines)
|
|
- `tests/infrastructure/learning-engine.test.ts`
|
|
- **Task**: Focus on Q-learning algorithm, pattern storage
|
|
- **Dependencies**: 2.7, 3.1
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:unit -- learning-engine
|
|
```
|
|
|
|
- [ ] **3.4 Consolidate FlakyTestDetector Tests** (2 hours)
|
|
- **Current Files** (2 total):
|
|
- `tests/unit/learning/FlakyTestDetector.test.ts`
|
|
- `tests/unit/learning/FlakyTestDetector.ml.test.ts`
|
|
- **Target**: 1 file (500 lines)
|
|
- `tests/infrastructure/flaky-test-detector.test.ts`
|
|
- **Task**: Focus on statistical detection algorithms
|
|
- **Dependencies**: 2.6, 3.3
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:unit -- flaky-test-detector
|
|
```
|
|
|
|
- [ ] **3.5 Create Contract Tests** (2 hours)
|
|
- **Files**:
|
|
- `tests/contracts/mcp-tools.test.ts` (102 tools)
|
|
- `tests/contracts/cli-commands.test.ts` (8 commands)
|
|
- `tests/contracts/agent-interfaces.test.ts` (18 agents)
|
|
- **Task**: Validate public APIs don't break
|
|
- **Test Strategy**: JSON Schema validation, not full execution
|
|
- **Dependencies**: 3.4
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:contracts
|
|
# Should complete in < 5 seconds (fast validation)
|
|
```
|
|
|
|
### Phase 3 Exit Criteria
|
|
- ✅ All consolidated tests pass
|
|
- ✅ No test file > 600 lines
|
|
- ✅ Unit test execution time < 30 seconds total
|
|
- ✅ Contract tests validate all 102 MCP tools + 8 commands + 18 agents
|
|
- ✅ Test coverage maintained ≥85% for critical paths
|
|
|
|
---
|
|
|
|
## 🗑️ Phase 4: Delete Redundant Tests (4-6 hours)
|
|
|
|
### Goal
|
|
Remove tests that don't provide user value, eliminate duplicates.
|
|
|
|
### Actions
|
|
|
|
- [ ] **4.1 Delete Implementation Detail Tests** (1 hour)
|
|
- **Files to DELETE**:
|
|
- `tests/unit/agents/TestGeneratorAgent.null-safety.test.ts` (40K)
|
|
- `tests/unit/agents/NeuralCapableMixin.skip.test.ts` (11K)
|
|
- **Reason**: Tests TypeScript compiler behavior, not business logic
|
|
- **Task**:
|
|
```bash
|
|
git rm tests/unit/agents/TestGeneratorAgent.null-safety.test.ts
|
|
git rm tests/unit/agents/NeuralCapableMixin.skip.test.ts
|
|
```
|
|
- **Dependencies**: 3.5
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:all
|
|
# Coverage should not drop (these tests didn't add value)
|
|
```
|
|
|
|
- [ ] **4.2 Delete Duplicate BaseAgent Tests** (1 hour)
|
|
- **Files to DELETE** (after consolidation in 3.1):
|
|
- `tests/unit/agents/BaseAgent.enhanced.test.ts`
|
|
- `tests/unit/agents/BaseAgent.comprehensive.test.ts`
|
|
- `tests/agents/BaseAgent.lifecycle.test.ts`
|
|
- `tests/agents/BaseAgent.edge-cases.test.ts`
|
|
- `tests/agents/BaseAgent.race-condition.test.ts`
|
|
- **Task**: Delete old files, update imports in remaining tests
|
|
- **Dependencies**: 3.1
|
|
- **Verification**:
|
|
```bash
|
|
find tests -name "*BaseAgent*.test.ts" | wc -l
|
|
# Should be exactly 2 files
|
|
```
|
|
|
|
- [ ] **4.3 Delete Duplicate TestGeneratorAgent Tests** (1 hour)
|
|
- **Files to DELETE** (after consolidation in 3.2):
|
|
- `tests/unit/agents/TestGeneratorAgent.comprehensive.test.ts`
|
|
- `tests/agents/TestGeneratorAgent.test.ts`
|
|
- **Dependencies**: 3.2
|
|
- **Verification**:
|
|
```bash
|
|
find tests -name "*TestGeneratorAgent*.test.ts" | wc -l
|
|
# Should be exactly 1 file
|
|
```
|
|
|
|
- [ ] **4.4 Delete Duplicate Learning Tests** (1 hour)
|
|
- **Files to DELETE** (after consolidation in 3.3, 3.4):
|
|
- `tests/unit/learning/LearningEngine.database.test.ts`
|
|
- `tests/unit/learning/FlakyTestDetector.ml.test.ts`
|
|
- **Dependencies**: 3.3, 3.4
|
|
- **Verification**:
|
|
```bash
|
|
ls tests/unit/learning/*.test.ts | wc -l
|
|
# Should be 0 (all moved to infrastructure/)
|
|
```
|
|
|
|
- [ ] **4.5 Clean Up Old Directory Structure** (30 min)
|
|
- **Directories to DELETE**:
|
|
- `tests/unit/` (if empty)
|
|
- `tests/integration/` (if empty)
|
|
- `tests/agents/` (all migrated)
|
|
- **Task**:
|
|
```bash
|
|
# Only delete if empty
|
|
find tests/unit tests/integration tests/agents -type d -empty -delete
|
|
```
|
|
- **Dependencies**: 4.4
|
|
- **Verification**:
|
|
```bash
|
|
tree tests -L 1
|
|
# Should show only: journeys, contracts, infrastructure, regression
|
|
```
|
|
|
|
- [ ] **4.6 Update Test Configuration** (1 hour)
|
|
- **Files**:
|
|
- `jest.config.js`
|
|
- `package.json` (test scripts)
|
|
- `.github/workflows/test.yml`
|
|
- **Task**: Update paths to reflect new structure
|
|
- **Dependencies**: 4.5
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:all
|
|
# All tests should still pass
|
|
```
|
|
|
|
### Phase 4 Exit Criteria
|
|
- ✅ ~60K lines of test code deleted
|
|
- ✅ No duplicate test files remain
|
|
- ✅ Old directory structure removed
|
|
- ✅ All CI/CD pipelines still pass
|
|
|
|
---
|
|
|
|
## 🔄 Phase 5: CI/CD Integration (4-6 hours)
|
|
|
|
### Goal
|
|
Optimize test execution for fast feedback, integrate with GitHub Actions.
|
|
|
|
### Actions
|
|
|
|
- [ ] **5.1 Create Optimized Test Scripts** (1 hour)
|
|
- **Files**: `package.json`
|
|
- **Task**: Add fast-feedback test scripts
|
|
- **Scripts**:
|
|
```json
|
|
{
|
|
"test:journeys": "jest tests/journeys --maxWorkers=4",
|
|
"test:contracts": "jest tests/contracts --maxWorkers=2",
|
|
"test:infrastructure": "jest tests/infrastructure --maxWorkers=4",
|
|
"test:regression": "jest tests/regression --maxWorkers=2",
|
|
"test:fast": "npm run test:journeys && npm run test:contracts",
|
|
"test:all": "jest --maxWorkers=4"
|
|
}
|
|
```
|
|
- **Dependencies**: 4.6
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:fast
|
|
# Should complete in < 30 seconds
|
|
```
|
|
|
|
- [ ] **5.2 Update GitHub Actions Workflow** (1.5 hours)
|
|
- **Files**: `.github/workflows/test.yml`
|
|
- **Task**: Optimize CI pipeline with parallel jobs
|
|
- **Strategy**:
|
|
```yaml
|
|
jobs:
|
|
journeys:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- run: npm run test:journeys
|
|
contracts:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- run: npm run test:contracts
|
|
infrastructure:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- run: npm run test:infrastructure
|
|
```
|
|
- **Dependencies**: 5.1
|
|
- **Verification**: Push to branch and check Actions tab
|
|
|
|
- [ ] **5.3 Add Test Coverage Gates** (1 hour)
|
|
- **Files**: `.github/workflows/test.yml`
|
|
- **Task**: Enforce 85% coverage minimum
|
|
- **Config**:
|
|
```yaml
|
|
- name: Check Coverage
|
|
run: |
|
|
npm run test:coverage
|
|
npx coverage-check --lines 85 --branches 80
|
|
```
|
|
- **Dependencies**: 5.2
|
|
- **Verification**: Create PR with low coverage, should fail
|
|
|
|
- [ ] **5.4 Create Test Dashboard** (1.5 hours)
|
|
- **Files**: `docs/test-dashboard.md`
|
|
- **Task**: Generate markdown dashboard with metrics
|
|
- **Metrics**:
|
|
- Total tests by category (journeys, contracts, infrastructure)
|
|
- Execution time per category
|
|
- Coverage by module
|
|
- Flaky test count
|
|
- **Dependencies**: 5.3
|
|
- **Verification**:
|
|
```bash
|
|
npm run test:dashboard
|
|
cat docs/test-dashboard.md
|
|
```
|
|
|
|
- [ ] **5.5 Update Documentation** (1 hour)
|
|
- **Files**:
|
|
- `README.md`
|
|
- `docs/testing/README.md`
|
|
- `docs/testing/writing-tests.md`
|
|
- **Task**: Document new test structure and best practices
|
|
- **Dependencies**: 5.4
|
|
- **Verification**: PR review passes
|
|
|
|
### Phase 5 Exit Criteria
|
|
- ✅ CI/CD pipeline runs in < 2 minutes (down from ~5 minutes)
|
|
- ✅ Coverage gates enforced automatically
|
|
- ✅ Test dashboard generated on every run
|
|
- ✅ Documentation updated and reviewed
|
|
|
|
---
|
|
|
|
## ⚠️ Risks & Mitigations
|
|
|
|
| Risk | Impact | Probability | Mitigation |
|
|
|------|--------|-------------|------------|
|
|
| **Coverage drops during consolidation** | High | Medium | Run coverage report after each consolidation step (Phase 3). Rollback if coverage drops > 2%. |
|
|
| **Journey tests are flaky** | High | Medium | Use real database with test isolation. Add retry logic. Run 10x locally before committing. |
|
|
| **Migration takes longer than estimated** | Medium | High | Break into smaller PRs (one phase at a time). Use TodoWrite to track progress. |
|
|
| **Existing bugs exposed** | Medium | Low | Expected! Create regression tests for discovered bugs. Don't block migration. |
|
|
| **Team members blocked** | Low | Medium | Keep old tests working during migration. Delete only after new tests pass. |
|
|
|
|
---
|
|
|
|
## 📈 Success Metrics
|
|
|
|
### Quantitative Metrics
|
|
| Metric | Baseline | Target | Current |
|
|
|--------|----------|--------|---------|
|
|
| **Test Files** | 402 | 50 | 402 |
|
|
| **Test Lines** | 195,527 | 40,000 | 195,527 |
|
|
| **Test Execution Time** | ~5 min | < 2 min | ~5 min |
|
|
| **Files > 1,000 Lines** | 18 | 0 | 18 |
|
|
| **Skipped Tests** | 7 | 0 | 7 |
|
|
| **Coverage (Critical Paths)** | 85% | ≥85% | 85% |
|
|
|
|
### Qualitative Metrics
|
|
- ✅ **Developer Confidence**: "I trust the test suite to catch regressions"
|
|
- ✅ **Onboarding Speed**: New contributors understand tests in < 1 hour
|
|
- ✅ **Maintenance Burden**: Test updates take < 10% of feature development time
|
|
- ✅ **CI/CD Reliability**: Zero flaky failures in 30 days
|
|
|
|
---
|
|
|
|
## 📝 Phase Checkpoints
|
|
|
|
### After Phase 1
|
|
- [ ] All team members can navigate new test structure
|
|
- [ ] Baseline metrics documented and agreed upon
|
|
- [ ] Migration CI job passes (empty tests)
|
|
|
|
### After Phase 2
|
|
- [ ] Product team validates journey tests match user expectations
|
|
- [ ] All journey tests run against staging database
|
|
- [ ] Journey tests documented in user-facing docs
|
|
|
|
### After Phase 3
|
|
- [ ] No regressions introduced (all existing tests still pass)
|
|
- [ ] Coverage maintained or improved
|
|
- [ ] Team reviews consolidated tests (code review)
|
|
|
|
### After Phase 4
|
|
- [ ] Old directory structure deleted
|
|
- [ ] CI/CD pipelines updated
|
|
- [ ] No broken imports or references
|
|
|
|
### After Phase 5
|
|
- [ ] CI/CD runs in < 2 minutes
|
|
- [ ] Coverage gates enforced
|
|
- [ ] Documentation complete and reviewed
|
|
|
|
---
|
|
|
|
## 🚀 Getting Started
|
|
|
|
1. **Create Tracking Issue**: Copy this document to GitHub Issues
|
|
2. **Assign Owner**: Designate migration lead
|
|
3. **Kick-off Meeting**: Review plan with team (30 min)
|
|
4. **Start Phase 1**: Begin with setup and analysis
|
|
5. **Daily Updates**: Update tracking issue checkboxes
|
|
6. **Phase Reviews**: Review with team after each phase
|
|
|
|
---
|
|
|
|
## 📚 References
|
|
|
|
- [Testing Trophy](https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications)
|
|
- [Google Testing Blog: Test Sizes](https://testing.googleblog.com/2010/12/test-sizes.html)
|
|
- [Martin Fowler: Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html)
|
|
- [Project: Test Execution Policy](file:///workspaces/agentic-qe-cf/docs/policies/test-execution.md)
|
|
- [Project: Git Operations Policy](file:///workspaces/agentic-qe-cf/docs/policies/git-operations.md)
|
|
|
|
---
|
|
|
|
**Migration Lead**: TBD
|
|
**Start Date**: TBD
|
|
**Target Completion**: TBD (4-6 weeks, 1 phase per week)
|
|
|
|
---
|
|
|
|
_Generated by GOAP Specialist • Last Updated: 2025-12-02_
|