proffesor-for-testing__agentic-qe/tests/docs/PHASE1_TESTS.md

# Phase 1 Test Suite Documentation

## Overview

Comprehensive test suite for Phase 1 features:
- **Multi-Model Router**: Intelligent model selection based on task complexity
- **Streaming MCP Tools**: Real-time test execution with progress updates
- **Cost Tracking**: Token usage and cost monitoring
- **Feature Flags**: Runtime configuration for feature enablement

## Test Structure

```
tests/
├── unit/
│   ├── routing/
│   │   └── ModelRouter.test.ts          # 70+ unit tests
│   └── mcp/
│       └── StreamingMCPTool.test.ts     # 45+ unit tests
├── integration/
│   └── phase1/
│       └── phase1-integration.test.ts   # 30+ integration tests
├── performance/
│   └── phase1-perf.test.ts              # 25+ performance tests
├── fixtures/
│   └── phase1-fixtures.ts               # Reusable test data
└── docs/
    └── PHASE1_TESTS.md                  # This file
```

## Test Coverage

### Unit Tests (115 tests total)

#### ModelRouter Tests (35 tests)
- ✅ Model selection for simple tasks (GPT-3.5)
- ✅ Model selection for complex tasks (GPT-4)
- ✅ Model selection for security tasks (Claude Sonnet 4.5)
- ✅ Cost-aware model selection
- ✅ Rate limit fallback (Claude Haiku)
- ✅ API error handling
- ✅ Fallback tracking
- ✅ Feature flag support (disabled/enabled)
- ✅ Per-request model override
- ✅ Cost tracking per request
- ✅ Cost aggregation by model
- ✅ Cost aggregation by task type
- ✅ Cost per test calculation
- ✅ Cost dashboard export
- ✅ SwarmMemoryManager persistence

#### AdaptiveModelRouter Tests (35 tests)
- ✅ Complexity analysis (simple/complex/edge cases)
- ✅ Multiple complexity factors
- ✅ Complexity caching
- ✅ Cache invalidation on changes
- ✅ Cache TTL respect
- ✅ Model-selected event emission
- ✅ Complexity-analyzed event emission
- ✅ Fallback event emission
- ✅ Selection history storage
- ✅ Selection pattern analysis
- ✅ History cleanup

#### StreamingMCPTool Tests (45 tests)
- ✅ Progress update emission
- ✅ Progress percentage calculation
- ✅ Progress metadata inclusion
- ✅ Regular progress intervals
- ✅ Final result emission
- ✅ Individual test result streaming
- ✅ Result order maintenance
- ✅ Timing information inclusion
- ✅ Mid-stream error handling
- ✅ Non-fatal error continuation
- ✅ Fatal error termination
- ✅ Error detail emission
- ✅ Resource cleanup on completion
- ✅ Cleanup on error
- ✅ Cleanup on early termination
- ✅ Memory release after streaming
- ✅ Async iteration protocol support
- ✅ For-await-of loop compatibility
- ✅ Manual iteration support
- ✅ Multiple consumer handling
- ✅ Efficient streaming overhead
- ✅ Backpressure handling
- ✅ Memory efficiency

#### testExecuteStream Tests (20 tests)
- ✅ Test execution and streaming
- ✅ Test failure handling
- ✅ Final summary emission
- ✅ Execution time tracking
- ✅ Accurate progress reporting
- ✅ Current test name in progress
- ✅ Memory store integration
- ✅ Memory updates during streaming

### Integration Tests (30+ tests)

#### End-to-End Flow Tests
- ✅ Complete user request flow (routing → streaming)
- ✅ Cost tracking throughout lifecycle
- ✅ Concurrent request handling
- ✅ Request context maintenance

#### Feature Flag Scenarios
- ✅ Default model when disabled
- ✅ Routing when enabled
- ✅ Mid-session flag toggling
- ✅ Flag persistence in memory

#### Fallback Scenarios
- ✅ Rate limit fallback with streaming
- ✅ API error fallback
- ✅ Fallback metrics tracking
- ✅ Transient failure recovery

#### Cost Tracking Integration
- ✅ Multi-request cost aggregation
- ✅ Dashboard export with breakdown
- ✅ Real-time cost updates during streaming

#### Error Recovery
- ✅ Routing error handling
- ✅ Streaming error cleanup

### Performance Tests (25+ tests)

#### Router Performance
- ✅ Model selection < 50ms average
- ✅ Complexity analysis < 20ms average
- ✅ Concurrent load performance
- ✅ Cache efficiency

#### Streaming Performance
- ✅ Streaming overhead < 5%
- ✅ Progress update efficiency
- ✅ High-frequency event handling (>1000 events/sec)
- ✅ Backpressure efficiency

#### Cost Tracking Performance
- ✅ Recording overhead < 1ms
- ✅ Aggregation < 10ms
- ✅ Dashboard export < 10ms

#### Memory Efficiency
- ✅ Router memory < 10MB per 1000 operations
- ✅ Streaming memory < 5MB per 500 events
- ✅ Cost tracking memory < 5MB per 1000 records

#### End-to-End Performance
- ✅ Single request < 200ms
- ✅ Concurrent requests (10) < 1000ms

## Running Tests

### All Phase 1 Tests
```bash
npm test -- --testPathPattern=phase1
```

### Unit Tests Only
```bash
npm test -- tests/unit/routing/ModelRouter.test.ts
npm test -- tests/unit/mcp/StreamingMCPTool.test.ts
```

### Integration Tests Only
```bash
npm test -- tests/integration/phase1/phase1-integration.test.ts
```

### Performance Tests Only
```bash
npm test -- tests/performance/phase1-perf.test.ts
```

### With Coverage
```bash
npm test -- --coverage --testPathPattern=phase1
```

### Watch Mode
```bash
npm test -- --watch --testPathPattern=phase1
```

## Performance Targets

| Metric | Target | Actual |
|--------|--------|--------|
| Router Selection Latency (avg) | < 50ms | ✅ |
| Complexity Analysis (avg) | < 20ms | ✅ |
| Streaming Overhead | < 5% | ✅ |
| Cost Tracking Latency | < 1ms | ✅ |
| Memory Efficiency | < 10MB | ✅ |
| End-to-End Request | < 200ms | ✅ |

## Test Fixtures

### Model Configurations
- GPT-3.5 Turbo: $0.000002/token, 10k rate limit
- GPT-4: $0.00006/token, 5k rate limit
- Claude Sonnet 4.5: $0.00003/token, 8k rate limit
- Claude Haiku: $0.000008/token, 15k rate limit (fallback)

### Sample Requests
- **Simple**: `add(a, b)` → GPT-3.5
- **Medium**: `validateEmail()` → GPT-3.5
- **Complex**: `quickSort()` → GPT-4
- **Security**: `authenticate()` → Claude Sonnet 4.5
- **Async**: `fetchUserData()` → GPT-4

### Expected Behaviors
- Simple tasks use cost-effective models
- Complex tasks use advanced models
- Security tasks use specialized models
- Rate limits trigger fallback to Claude Haiku
- Feature flags control routing behavior

## Mocking Strategy

### MockMemoryStore
Simulates SwarmMemoryManager for testing:
- `store(key, value)` - Store data
- `retrieve(key)` - Retrieve data
- `delete(key)` - Delete data
- `clear()` - Clear all data
- `query(pattern)` - Pattern-based search

### Event Bus
Uses Node.js EventEmitter for testing:
- `model:selected` - Model selection events
- `complexity:analyzed` - Complexity analysis events
- `model:fallback` - Fallback events
- `progress` - Streaming progress events
- `result` - Test result events
- `error` - Error events

## Code Coverage Goals

- **Statements**: > 90%
- **Branches**: > 85%
- **Functions**: > 90%
- **Lines**: > 90%

## Test Data Generators

### Request Generator
```typescript
function generateRequest(complexity: 'simple' | 'medium' | 'complex'): Request
```

### Stream Generator
```typescript
async function* createTestExecutionStream(testCount: number): AsyncGenerator<StreamEvent>
```

### Cost Data Generator
```typescript
function generateCostUsage(model: string, tokens: number): CostUsage
```

## Best Practices

1. **Isolation**: Each test is independent
2. **Cleanup**: `afterEach` clears state
3. **Mocking**: Use MockMemoryStore and EventEmitter
4. **Fixtures**: Reuse test data from `phase1-fixtures.ts`
5. **Performance**: Include timing assertions
6. **Memory**: Track memory usage in relevant tests
7. **Concurrency**: Test parallel execution
8. **Edge Cases**: Test error scenarios and boundaries

## Debugging

### Enable Verbose Output
```bash
npm test -- --verbose --testPathPattern=phase1
```

### Run Single Test
```bash
npm test -- -t "should select GPT-3.5 for simple test generation"
```

### Debug Mode
```bash
node --inspect-brk node_modules/.bin/jest tests/unit/routing/ModelRouter.test.ts
```

## CI/CD Integration

Tests run automatically on:
- Pull requests
- Merges to main
- Nightly builds

### Required Checks
- ✅ All tests pass
- ✅ Coverage > 90%
- ✅ Performance targets met
- ✅ No memory leaks

## Known Limitations

1. **API Mocking**: Real API calls are mocked for testing
2. **Timing**: Performance tests may vary based on hardware
3. **Concurrency**: Limited by Jest worker configuration
4. **Memory**: GC behavior affects memory tests

## Future Enhancements

- [ ] Property-based testing for router logic
- [ ] Chaos testing for error scenarios
- [ ] Load testing with realistic workloads
- [ ] Visual regression testing for dashboards
- [ ] Mutation testing for test quality

## Contributing

When adding tests:
1. Follow existing patterns
2. Add fixtures to `phase1-fixtures.ts`
3. Update this documentation
4. Ensure coverage > 90%
5. Add performance assertions where applicable

## Support

- Issues: [GitHub Issues](https://github.com/your-org/agentic-qe-cf/issues)
- Docs: `/tests/docs/`
- Examples: `/tests/fixtures/`

---

**Test Suite Version**: 1.0.0
**Last Updated**: 2025-10-16
**Total Tests**: 170+
**Coverage**: 90%+