proffesor-for-testing__agentic-qe/docs/MCP-TESTING-GAP.md

MCP Tools Testing Gap Analysis

Date: 2025-10-30 Version: v1.3.6 Status: 🟡 PARTIAL COVERAGE (19/54 tools tested)

---

Executive Summary

The AQE Fleet has 54 MCP tools registered in src/mcp/server.ts, but only 19 tools (35%) have unit test coverage. The remaining 35 tools (65%) lack unit tests, representing a significant testing gap discovered during PR #25 CI/CD improvements.

This gap was initially masked by bugs in the validation script that only checked flat file paths and didn't search subdirectories or within test files.

---

Current Test Coverage

✅ Tools With Unit Tests (19/54)

Memory & Coordination Tools (10 tools)

• ✅ memory_store - MemoryTools.test.ts
• ✅ memory_retrieve - MemoryTools.test.ts
• ✅ memory_query - MemoryTools.test.ts
• ✅ memory_share - MemoryTools.test.ts
• ✅ memory_backup - MemoryTools.test.ts
• ✅ blackboard_post - MemoryTools.test.ts
• ✅ blackboard_read - MemoryTools.test.ts
• ✅ consensus_propose - MemoryTools.test.ts
• ✅ consensus_vote - MemoryTools.test.ts
• ✅ artifact_manifest - MemoryTools.test.ts

Workflow & Coordination Tools (7 tools)

• ✅ workflow_create - CoordinationTools.test.ts
• ✅ workflow_execute - CoordinationTools.test.ts
• ✅ workflow_checkpoint - CoordinationTools.test.ts
• ✅ workflow_resume - CoordinationTools.test.ts
• ✅ task_status - CoordinationTools.test.ts
• ✅ event_emit - CoordinationTools.test.ts
• ✅ event_subscribe - CoordinationTools.test.ts

Core Fleet Tools (2 tools)

• ✅ test_generate - test-generate.test.ts
• ✅ fleet_status - fleet-status.test.ts

Total Tested: 19 tools across 4 test files

---

❌ Tools Missing Unit Tests (35/54)

Fleet Management (2 tools)

• ❌ fleet_init
• ❌ agent_spawn

Test Lifecycle (7 tools)

• ❌ test_execute
• ❌ test_generate_enhanced
• ❌ test_execute_parallel
• ❌ test_optimize_sublinear
• ❌ test_report_comprehensive
• ❌ test_coverage_detailed
• ❌ task_orchestrate
• ❌ optimize_tests

Quality Tools (6 tools)

• ❌ quality_analyze
• ❌ quality_gate_execute
• ❌ quality_validate_metrics
• ❌ quality_risk_assess
• ❌ quality_decision_make
• ❌ quality_policy_check

Prediction & Risk Analysis (6 tools)

• ❌ predict_defects
• ❌ flaky_test_detect
• ❌ predict_defects_ai
• ❌ regression_risk_analyze
• ❌ visual_test_regression
• ❌ deployment_readiness_check

Performance & Analysis (5 tools)

• ❌ coverage_analyze_sublinear
• ❌ coverage_gaps_detect
• ❌ performance_benchmark_run
• ❌ performance_monitor_realtime
• ❌ security_scan_comprehensive

Advanced QE Tools (7 tools)

• ❌ requirements_validate
• ❌ requirements_generate_bdd
• ❌ production_incident_replay
• ❌ production_rum_analyze
• ❌ api_breaking_changes
• ❌ mutation_test_execute

Streaming Tools (2 tools) - NOT IMPLEMENTED

• ❌ test_execute_stream - Handler missing
• ❌ coverage_analyze_stream - Handler missing

Total Missing Tests: 35 tools (33 with handlers, 2 without handlers)

---

Impact Assessment

Current State

• Production Impact: 🟢 LOW - All 54 tools have handler implementations (except 2 streaming)
• Test Confidence: 🟡 MEDIUM - 35% of tools verified by unit tests
• CI/CD Impact: 🟡 MEDIUM - Tests pass for covered tools, validation reports gaps
• Maintenance Risk: 🟠 HIGH - Untested tools may break without detection

Risk Matrix

Tool Category	Tools	Tested	Coverage	Risk
Memory & Coordination	17	17	100%	🟢 LOW
Fleet Management	2	0	0%	🔴 HIGH
Test Lifecycle	9	1	11%	🔴 HIGH
Quality Tools	6	0	0%	🔴 HIGH
Prediction & Risk	6	0	0%	🔴 HIGH
Performance & Analysis	5	0	0%	🔴 HIGH
Advanced QE	7	0	0%	🔴 HIGH
Streaming	2	0	0%	🔴 CRITICAL

---

Root Cause Analysis

Why Tests Are Missing

1. Rapid Development: MCP tools added quickly for feature completeness
2. Validation Script Bugs:
   • Only checked flat paths (missed subdirectories)
   • Didn't search within test files
   • Masked the testing gap
3. Test Organization: Tests organized by category, not individual files
4. Streaming Tools: Intended for future v1.1.0 release, added to spec early

Why Gap Wasn't Detected Earlier

1. CI had continue-on-error: true - Tests could fail without blocking
2. Validation reported "2/54" - Looked like minor issue, not 65% gap
3. No coverage requirements - No minimum threshold enforced
4. Manual testing focus - Integration tests prioritized over unit tests

---

Remediation Plan

Phase 1: v1.3.7 (Quick Wins) - 2-3 days

Target: High-risk, high-usage tools

• Fleet management (2 tools): fleet_init, agent_spawn
• Core test tools (3 tools): test_execute, task_orchestrate, optimize_tests
• Quality gate (1 tool): quality_gate_execute
• Remove streaming tools from spec (2 tools) or implement handlers

Estimated Work: 6-8 comprehensive test suites (~400-500 lines each)

Phase 2: v1.3.8 (Medium Priority) - 1 week

Target: Quality and prediction tools

• Quality tools (5 remaining)
• Prediction tools (6 tools)
• Coverage analysis (2 tools)

Estimated Work: 13 test suites (~4,000-5,000 lines)

Phase 3: v1.4.0 (Comprehensive) - 2 weeks

Target: Complete coverage + integration tests

• Test lifecycle tools (6 remaining)
• Performance tools (3 remaining)
• Advanced QE tools (7 tools)
• Integration test suites for all categories
• End-to-end workflow tests

Estimated Work: 16 test suites + integration tests (~8,000-10,000 lines)

---

Immediate Actions (v1.3.6)

✅ Completed

1. Fixed validation script to search recursively
2. Fixed test imports (removed .js extensions)
3. Added Database mock for test environment
4. Documented testing gap (this file)

🔄 In Progress

5. Create GitHub issue for tracking
6. Update CI/CD comments with accurate status

📋 Next Steps

7. Prioritize 6-8 high-risk tools for v1.3.7
8. Create test templates for each tool category
9. Set up coverage thresholds (target: 80% by v1.4.0)
10. Add pre-commit hook for new MCP tools (require tests)

---

Testing Guidelines

Required for New MCP Tools

All new MCP tools MUST include:

1. Unit Test File: tests/mcp/handlers/{category}/{tool-name}.test.ts
2. Test Coverage:
   • Handler initialization
   • Input validation (valid, invalid, edge cases)
   • Error handling
   • Success responses
3. Integration Tests: Optional but recommended for complex workflows
4. Documentation: Tool usage examples in test descriptions

Test Template

/**
 * Tests for {Tool Name} MCP Tool
 *
 * @group mcp
 * @group {category}
 */

import { AgenticQEMCPServer } from '@mcp/server';
import { TOOL_NAMES } from '@mcp/tools';

describe('{Tool Name}', () => {
  let server: AgenticQEMCPServer;

  beforeEach(async () => {
    server = new AgenticQEMCPServer();
  });

  afterEach(async () => {
    await server.stop();
    jest.clearAllMocks();
  });

  describe('Valid inputs', () => {
    it('should handle {basic scenario}', async () => {
      // Test implementation
    });
  });

  describe('Invalid inputs', () => {
    it('should reject {invalid scenario}', async () => {
      // Test implementation
    });
  });

  describe('Error handling', () => {
    it('should handle {error scenario}', async () => {
      // Test implementation
    });
  });
});

---

Metrics & Tracking

Current Metrics (v1.3.6)

• Total MCP Tools: 54
• Tools Tested: 19 (35%)
• Tools Missing Tests: 35 (65%)
• Tools Missing Handlers: 2 (4%)
• Test Files: 4 category files
• Test Lines: ~2,000 lines

Target Metrics (v1.4.0)

• Tools Tested: 50+ (90%+)
• Code Coverage: 80%+
• Test Files: 15-20 files
• Test Lines: ~12,000-15,000 lines
• Integration Tests: 5-10 comprehensive suites

Tracking

• GitHub Issue: #26
• Milestone: v1.3.7 / v1.4.0
• Priority: HIGH
• Assignee: TBD

---

References

• Validation Script: scripts/validate-mcp-tools.js
• MCP Server: src/mcp/server.ts (63 handlers registered)
• Tool Definitions: src/mcp/tools.ts (54 tools defined)
• Test Directory: tests/mcp/
• CI Workflow: .github/workflows/mcp-tools-test.yml

---

Conclusion

The MCP testing gap is real and significant (65% of tools untested), but was masked by validation bugs. With fixes applied in v1.3.6, we now have:

1. ✅ Accurate reporting - Validation finds 19/54 correctly
2. ✅ Fixed test infrastructure - Tests run properly
3. ✅ Clear visibility - Gap documented and tracked
4. 📋 Remediation plan - Phased approach for v1.3.7+

Recommendation: Accept v1.3.6 with documented gap, prioritize high-risk tools for v1.3.7 quick follow-up.

---

Last Updated: 2025-10-30 Document Owner: AQE Development Team Status: 🟡 Active - Remediation in progress