proffesor-for-testing__agentic-qe/docs/plans/llm-independence-implementation-plan.md

AQE Fleet LLM Independence - Parallelizable Implementation Plan

Date: December 23, 2025 Version: 1.1 Status: Phase 1-4 COMPLETE ✅ | LLM Independence Achieved Source RFC: GitHub Issue #142 Strategic Plan: GOAP Plan Last Updated: December 2025

---

Executive Summary

This plan provides a detailed, parallelizable implementation roadmap for achieving LLM independence in the AQE Fleet. The plan is optimized for concurrent execution by multiple agents, with clear file ownership boundaries to prevent merge conflicts.

Phase 1 Status (December 2024) ✅ COMPLETE

Component	Status	Notes
OllamaProvider	✅ Complete	All 13 ILLMProvider methods
IAgentLLM Interface	✅ Complete	Provider-independent API
AgentLLMAdapter	✅ Complete	Bridges providers to agents
LLMProviderFactory	✅ Complete	Auto-detection, selection
Agent Migration	✅ Complete	See categorization below

Phase 2 Status (December 2025) ✅ COMPLETE

Component	Status	Notes
ComplexityClassifier	✅ Complete	ML-based task complexity analysis
ModelCapabilityRegistry	✅ Complete	20+ models with benchmarks
HybridRouterComplexityIntegration	✅ Complete	Complexity-aware routing
HybridRouterModelSelection	✅ Complete	Intelligent model selection
CostOptimizationStrategies	✅ Complete	Prompt compression, batching

Phase 3 Status (December 2025) ✅ COMPLETE

Component	Status	Notes
ProviderHealthMonitor	✅ Complete	Health checks, error rates, availability
HybridRouterHealthIntegration	✅ Complete	Health-aware routing with fallback
CLI Health Dashboard	✅ Complete	aqe providers status command
Circuit Breaker	✅ Complete	Failure detection, recovery

Phase 4 Status (December 2025) ✅ COMPLETE

Component	Status	Notes
GroqProvider	✅ Complete	Free tier 14.4k req/day
GitHubModelsProvider	✅ Complete	Codespaces detection, GITHUB_TOKEN
QuotaManager	✅ Complete	Per-provider tracking, alerts
OpenRouter Free Models	✅ Complete	30+ free models supported

Agent Categorization (Actual Implementation)

Category	Count	Description
LLM-Active	4	Use getAgentLLM() for AI features
N8n (Inherited)	15	LLM via N8nBaseAgent inheritance
Algorithmic	17	Rule/metric-based, no LLM needed

Success Criteria (Updated)

• ✅ Zero direct Claude API imports in agent files
• ✅ 4 agents actively use IAgentLLM for AI-enhanced features
• ✅ 15 n8n agents have LLM capability via inheritance
• ✅ 17 algorithmic agents work correctly without LLM (by design)
• ✅ Graceful multi-provider fallback chains
• ✅ Cost reduction of 60%+ through intelligent routing (Phase 2)
• ✅ Three deployment tiers: Local, Hosted, Free Cloud (Phase 2-4)

Estimated Timeline

• Phase 1 (Foundation): 4-6 weeks
• Phase 2 (Intelligent Routing): 3-4 weeks
• Phase 3 (Resilience): 2-3 weeks
• Phase 4 (Free Tier Optimization): 1-2 weeks
• Total: 10-15 weeks

Resource Requirements

• 3-5 coder agents for parallel implementation
• 2-3 tester agents for test suite creation
• 1-2 reviewer agents for code review
• 1 qe-test-generator for comprehensive testing

---

Table of Contents

1. Work Stream Breakdown
2. Phase 1: Foundation (Weeks 1-6)
3. Phase 2: Intelligent Routing (Weeks 7-10)
4. Phase 3: Resilience (Weeks 11-13)
5. Phase 4: Free Tier Optimization (Weeks 14-15)
6. Dependency Graph
7. File Ownership Matrix
8. Testing Strategy
9. Rollout Plan

---

Work Stream Breakdown

Stream 1: Provider Infrastructure

Owner: Coder Agent A Files: src/providers/* Goal: Implement OllamaProvider and enhance HybridRouter

Stream 2: Agent Abstraction

Owner: Coder Agent B Files: src/agents/* (20 files), src/subagents/* (11 files) Goal: Refactor all agents to use LLMProviderFactory

Stream 3: Configuration System

Owner: Coder Agent C Files: src/config/*, .aqe/providers.yaml Goal: Build multi-provider configuration management

Stream 4: Testing Infrastructure

Owner: Tester Agent A + qe-test-generator Files: tests/providers/*, tests/integration/* Goal: Comprehensive test coverage for all providers

Stream 5: Documentation & Examples

Owner: Coder Agent D Files: docs/*, examples/* Goal: User-facing documentation and example configurations

---

Phase 1: Foundation (Weeks 1-6)

🎯 Goal

Create provider-agnostic infrastructure with OllamaProvider and agent abstraction layer.

Milestone 1.1: OllamaProvider Implementation (Weeks 1-2)

Task 1.1.1: Core Provider Implementation

Agent: coder (Stream 1) Priority: CRITICAL Effort: High

Files to Create:

• src/providers/OllamaProvider.ts

Implementation Requirements:

// Must implement all 13 ILLMProvider methods:
export class OllamaProvider implements ILLMProvider {
  // 1. initialize(): Promise<void>
  // 2. complete(options: LLMCompletionOptions): Promise<LLMCompletionResponse>
  // 3. streamComplete(options): AsyncIterableIterator<LLMStreamEvent>
  // 4. embed(options: LLMEmbeddingOptions): Promise<LLMEmbeddingResponse>
  // 5. countTokens(options): Promise<number>
  // 6. healthCheck(): Promise<LLMHealthStatus>
  // 7. getMetadata(): LLMProviderMetadata
  // 8. shutdown(): Promise<void>
  // 9. trackCost(usage): number (returns 0 for local)
  // 10. Additional: Model discovery via Ollama API
  // 11. Additional: Model hot-swapping
  // 12. Additional: Streaming support
  // 13. Additional: Health check with model ping
}

Models to Support:

• qwen3-coder:30b (primary recommendation)
• rnj-1:8b (edge deployment)
• devstral-small-2:24b (laptop deployment)
• llama3.3:70b (high quality)
• deepseek-coder-v2 (multi-language)
• starcoder2:15b (budget-friendly)

API Endpoints:

• http://localhost:11434/api/generate (completion)
• http://localhost:11434/api/embeddings (embeddings)
• http://localhost:11434/api/tags (model discovery)
• http://localhost:11434/api/show (model info)

Success Criteria:

• ✅ All 13 ILLMProvider methods implemented
• ✅ Works with Ollama running locally
• ✅ Model discovery and listing
• ✅ Streaming support functional
• ✅ Health check returns model status
• ✅ Cost tracking returns $0

Dependencies: None (can start immediately)

---

Task 1.1.2: OllamaProvider Testing

Agent: tester (Stream 4) Priority: CRITICAL Effort: Medium

Files to Create:

• tests/providers/OllamaProvider.test.ts
• tests/integration/OllamaProvider.integration.test.ts

Test Coverage Requirements:

• Unit tests for all 13 methods (95% coverage target)
• Integration tests with real Ollama instance
• Model discovery tests
• Streaming tests
• Error handling tests (Ollama not running, model not found)
• Health check tests

Success Criteria:

• ✅ 95%+ code coverage
• ✅ All tests pass with Ollama running
• ✅ Graceful handling when Ollama unavailable

Dependencies: Task 1.1.1

---

Task 1.1.3: LLMProviderFactory Integration

Agent: coder (Stream 1) Priority: HIGH Effort: Low

Files to Modify:

• src/providers/LLMProviderFactory.ts

Changes Required:

// Add Ollama to provider selection matrix
interface EnvironmentSignals {
  hasOllamaRunning: boolean,  // NEW: Check localhost:11434
  // ... existing fields
}

// Update provider selection logic
async initialize(): Promise<void> {
  // 1. Check for Ollama on localhost:11434
  // 2. Add to available providers
  // 3. Prefer Ollama for local-first mode
}

// Add factory method
createOllamaProvider(config?: OllamaProviderConfig): ILLMProvider

// Update selection criteria
interface ProviderSelectionCriteria {
  preferLocal?: boolean,  // UPDATED: Now includes Ollama
  // ... existing fields
}

Success Criteria:

• ✅ Factory auto-detects Ollama availability
• ✅ createOllamaProvider() method works
• ✅ Smart selection includes Ollama option
• ✅ Tests updated and passing

Dependencies: Task 1.1.1

---

Milestone 1.2: Agent Abstraction Layer (Weeks 2-4)

Task 1.2.1: Define IAgentLLM Interface

Agent: coder (Stream 1) Priority: CRITICAL Effort: Low

Files to Create:

• src/agents/interfaces/IAgentLLM.ts

Interface Design:

/**
 * Agent-specific LLM interface that abstracts away provider details
 * Agents should ONLY import this interface, never specific providers
 */
export interface IAgentLLM {
  // Core completion (simplified for agents)
  complete(prompt: string, options?: AgentCompletionOptions): Promise<string>

  // Streaming for interactive sessions
  streamComplete(prompt: string, options?: AgentCompletionOptions): AsyncIterableIterator<string>

  // Embeddings for semantic tasks
  embed(text: string): Promise<number[]>

  // Agent-specific utilities
  getAvailableModels(): Promise<string[]>
  getCurrentModel(): string
  switchModel(model: string): Promise<void>

  // Health and cost tracking
  isHealthy(): Promise<boolean>
  getUsageStats(): AgentUsageStats
}

interface AgentCompletionOptions {
  temperature?: number
  maxTokens?: number
  systemPrompt?: string
  cacheKey?: string  // For RuVector caching
  complexity?: 'simple' | 'moderate' | 'complex' | 'very_complex'
}

interface AgentUsageStats {
  requestCount: number
  tokensUsed: number
  costIncurred: number
  averageLatency: number
}

Success Criteria:

• ✅ Interface designed and documented
• ✅ TypeScript types fully defined
• ✅ Agent-friendly simplified API

Dependencies: None (can run in parallel with 1.1.1)

---

Task 1.2.2: Implement IAgentLLM Adapter

Agent: coder (Stream 1) Priority: CRITICAL Effort: Medium

Files to Create:

• src/agents/adapters/AgentLLMAdapter.ts

Implementation:

/**
 * Adapter that wraps ILLMProvider and presents IAgentLLM interface
 * Handles complexity routing, caching, and agent-specific concerns
 */
export class AgentLLMAdapter implements IAgentLLM {
  constructor(
    private provider: ILLMProvider,
    private hybridRouter?: HybridRouter,
    private config?: AgentLLMConfig
  ) {}

  async complete(prompt: string, options?: AgentCompletionOptions): Promise<string> {
    // 1. Determine complexity if not provided
    // 2. Route to appropriate provider via HybridRouter
    // 3. Check RuVector cache if enabled
    // 4. Call provider.complete() with proper formatting
    // 5. Store in cache for future use
    // 6. Track usage stats
    // 7. Return string result
  }

  // ... implement all IAgentLLM methods
}

Success Criteria:

• ✅ All IAgentLLM methods implemented
• ✅ Integrates with HybridRouter
• ✅ RuVector caching support
• ✅ Usage tracking functional

Dependencies: Task 1.2.1

---

Task 1.2.3: Agent Migration Script

Agent: coder (Stream 2) Priority: HIGH Effort: Medium (actual scope refined) Status: ✅ COMPLETE (December 2024)

Implementation Note (December 2024): Analysis revealed that not all agents require LLM integration. Many agents are designed to be purely algorithmic (rule-based, metric-based, or pattern-based). The migration focused on agents that benefit from AI enhancement.

Agent Categorization:

Category	Count	Description
LLM-Active Agents	4	Actively use getAgentLLM() for AI features
N8n Agents (Inherited)	15	Have LLM capability via N8nBaseAgent
Algorithmic Agents	17	Designed without LLM dependency

Agents Actively Using IAgentLLM:

• TestGeneratorAgent.ts - AI test code generation
• CoverageAnalyzerAgent.ts - AI-enhanced gap analysis
• CodeIntelligenceAgent.ts - AI search summaries
• N8nBaseAgent.ts - AI workflow analysis (inherited by 15 n8n agents)

Algorithmic Agents (No LLM Required by Design):

• SecurityScannerAgent - OWASP rules, static patterns
• RegressionRiskAnalyzerAgent - Git diff analysis, risk scoring
• QualityGateAgent - Decision tree, threshold logic
• PerformanceTesterAgent - Metric-based measurements
• FlakyTestHunterAgent - Statistical flakiness patterns
• TestExecutorAgent - Test execution, no generation
• FleetCommanderAgent - Orchestration logic
• AccessibilityAllyAgent - WCAG rule validation
• ApiContractValidatorAgent - Schema validation
• CodeComplexityAnalyzerAgent - Cyclomatic complexity
• And 7 others...

Original Files to Modify: (31 total)

• src/agents/*.ts (20 files)
• src/subagents/*.ts (11 files)

Migration Pattern:

// BEFORE (❌ Direct import):
import { ClaudeProvider } from '../providers/ClaudeProvider';

export class QETestGeneratorAgent {
  private llm: ClaudeProvider;

  async initialize() {
    this.llm = new ClaudeProvider({
      apiKey: process.env.ANTHROPIC_API_KEY,
      defaultModel: 'claude-sonnet-4'
    });
    await this.llm.initialize();
  }
}

// AFTER (✅ Factory pattern):
import { IAgentLLM } from './interfaces/IAgentLLM';
import { getGlobalLLMFactory } from '../providers/LLMProviderFactory';
import { AgentLLMAdapter } from './adapters/AgentLLMAdapter';

export class QETestGeneratorAgent {
  private llm: IAgentLLM;

  async initialize() {
    const factory = getGlobalLLMFactory();
    await factory.initialize();

    const provider = factory.selectBestProvider({
      preferLocal: false,      // Quality over cost for test generation
      preferLowCost: true,     // But optimize when possible
      requiredCapabilities: ['streaming']
    }) || factory.getProvider('auto');

    this.llm = new AgentLLMAdapter(provider, factory.createHybridRouter());
  }

  async generateTests(code: string): Promise<string> {
    return this.llm.complete(`Generate tests for: ${code}`, {
      complexity: 'complex',
      systemPrompt: 'You are a QE test generation expert...',
      cacheKey: `test-gen-${hashCode(code)}`
    });
  }
}

Batch Migration Strategy:

1. Group 1 (Week 2): Core agents (5 files)

   • QETestGeneratorAgent.ts
   • QECoverageAnalyzerAgent.ts
   • QECodeReviewerAgent.ts
   • QEBugAnalyzerAgent.ts
   • QEDocumentationAgent.ts

2. Group 2 (Week 3): Specialized agents (7 files)

   • QEPerformanceTesterAgent.ts
   • QESecurityScannerAgent.ts
   • QEFlakyTestDetectorAgent.ts
   • QEAccessibilityAgent.ts
   • QEAPITesterAgent.ts
   • QEE2ETesterAgent.ts
   • QEContractTesterAgent.ts

3. Group 3 (Week 3): Remaining agents (8 files)

   • All other main agents

4. Group 4 (Week 4): Subagents (11 files)

   • All subagent files

Success Criteria (Updated December 2024):

• ✅ 0 direct ClaudeProvider imports in agent files
• ✅ BaseAgent provides getAgentLLM() to all child agents
• ✅ 4 agents actively use IAgentLLM for AI-enhanced features
• ✅ 15 n8n agents inherit LLM capability via N8nBaseAgent
• ✅ 17 algorithmic agents work correctly without LLM (by design)
• ✅ Graceful degradation when LLM unavailable
• ✅ No functionality regression

Original Success Criteria (Pre-Implementation):

• All 31 agents use IAgentLLM interface (revised: not all agents need LLM)
• Tests updated for each agent (tested via journey tests)

Dependencies: Task 1.2.2

---

Task 1.2.4: Agent Migration Testing

Agent: tester (Stream 4) + qe-test-generator Priority: CRITICAL Effort: High

Files to Create/Modify:

• tests/agents/*.test.ts (update all 31 agent tests)
• tests/integration/agent-migration.test.ts (new)

Test Requirements:

1. Per-Agent Tests:

   • Mock IAgentLLM interface
   • Test agent works with different providers
   • Verify no direct provider coupling
   • Test error handling

2. Integration Tests:

   • Test each agent with ClaudeProvider
   • Test each agent with OpenRouterProvider
   • Test each agent with OllamaProvider
   • Test automatic provider switching
   • Test fallback behavior

3. Quality Baseline:

   • Compare output quality vs. Claude-only baseline
   • Target: ≥90% quality retention
   • Document any quality differences

Success Criteria:

• ✅ All agent tests pass with mock IAgentLLM
• ✅ All agents work with 3+ provider types
• ✅ Quality baseline meets ≥90% threshold
• ✅ No regressions in agent functionality

Dependencies: Task 1.2.3

---

Milestone 1.3: Configuration System (Weeks 3-4)

Task 1.3.1: Provider Configuration Schema

Agent: coder (Stream 3) Priority: HIGH Effort: Medium

Files to Create:

• src/config/ProviderConfig.ts (types)
• src/config/ProviderConfigValidator.ts (validation)
• .aqe/providers.yaml.template (user template)

Configuration Schema:

export interface ProviderConfig {
  // Provider identification
  type: 'ollama' | 'openrouter' | 'groq' | 'claude' | 'ruvllm' | 'google' | 'together';
  enabled: boolean;
  priority: number; // Lower = higher priority (1 = first)

  // Authentication
  apiKey?: string;
  baseUrl?: string;

  // Model selection
  defaultModel: string;
  modelOverrides?: Record<TaskType, string>;

  // Rate limiting
  limits?: {
    requestsPerMinute?: number;
    requestsPerDay?: number;
    tokensPerMinute?: number;
    tokensPerDay?: number;
  };

  // Cost tracking
  costPer1MTokens?: {
    input: number;
    output: number;
  };

  // Health monitoring
  healthCheck?: {
    enabled: boolean;
    intervalMs: number;
    timeoutMs: number;
  };

  // Fallback chain
  fallbackProvider?: string;
}

export type TaskType =
  | 'test-generation'
  | 'coverage-analysis'
  | 'code-review'
  | 'bug-detection'
  | 'documentation'
  | 'refactoring'
  | 'performance-testing'
  | 'security-scanning';

export interface DeploymentMode {
  mode: 'local_first' | 'hosted' | 'free_only' | 'hybrid';
  providers: ProviderConfig[];
}

Example Configuration:

# .aqe/providers.yaml
mode: hybrid

providers:
  # Local-first (Ollama)
  - type: ollama
    enabled: true
    priority: 1
    baseUrl: http://localhost:11434
    defaultModel: qwen3-coder:30b
    fallbackProvider: groq
    healthCheck:
      enabled: true
      intervalMs: 60000
      timeoutMs: 5000

  # Free tier (Groq)
  - type: groq
    enabled: true
    priority: 2
    apiKey: ${GROQ_API_KEY}
    defaultModel: llama-3.3-70b-versatile
    limits:
      requestsPerDay: 14400
      tokensPerMinute: 6000
    fallbackProvider: openrouter

  # Multi-model gateway (OpenRouter)
  - type: openrouter
    enabled: true
    priority: 3
    apiKey: ${OPENROUTER_API_KEY}
    defaultModel: mistralai/devstral-2512:free
    modelOverrides:
      test-generation: moonshotai/kimi-dev-72b
      code-review: mistralai/devstral-2-123b
      documentation: meta-llama/llama-3.3-70b-instruct
    fallbackProvider: claude

  # Premium fallback (Claude)
  - type: claude
    enabled: true
    priority: 4
    apiKey: ${ANTHROPIC_API_KEY}
    defaultModel: claude-sonnet-4
    costPer1MTokens:
      input: 3.0
      output: 15.0

Success Criteria:

• ✅ TypeScript types fully defined
• ✅ YAML schema validated
• ✅ Template file created
• ✅ Documentation written

Dependencies: None (can run in parallel)

---

Task 1.3.2: Configuration Loader

Agent: coder (Stream 3) Priority: HIGH Effort: Medium

Files to Create:

• src/config/ProviderConfigLoader.ts

Implementation:

export class ProviderConfigLoader {
  private static instance: ProviderConfigLoader;
  private config: DeploymentMode | null = null;

  static getInstance(): ProviderConfigLoader {
    if (!this.instance) {
      this.instance = new ProviderConfigLoader();
    }
    return this.instance;
  }

  async loadConfig(path?: string): Promise<DeploymentMode> {
    // 1. Look for config in order:
    //    - Provided path
    //    - .aqe/providers.yaml
    //    - .aqe/providers.yml
    //    - Environment variables
    // 2. Parse YAML
    // 3. Validate against schema
    // 4. Substitute environment variables
    // 5. Return validated config
  }

  async reloadConfig(): Promise<void> {
    // Hot-reload configuration without restart
  }

  getProviderConfig(providerType: string): ProviderConfig | null {
    // Get config for specific provider
  }

  getProvidersByPriority(): ProviderConfig[] {
    // Get all enabled providers sorted by priority
  }
}

Success Criteria:

• ✅ Loads YAML configuration
• ✅ Validates against schema
• ✅ Environment variable substitution
• ✅ Error handling for missing/invalid config
• ✅ Hot-reload support

Dependencies: Task 1.3.1

---

Task 1.3.3: CLI Configuration Commands

Agent: coder (Stream 3) Priority: MEDIUM Effort: Low

Files to Modify:

• src/cli/commands/init.ts
• src/cli/commands/providers.ts (new)

New Commands:

# Initialize with provider selection
aqe init --provider ollama --model qwen3-coder:30b
aqe init --provider groq --free-tier
aqe init --providers groq,openrouter,ollama --auto-fallback

# Provider management
aqe providers list              # Show all configured providers
aqe providers status            # Show health and availability
aqe providers test <provider>   # Test specific provider
aqe providers quota             # Show quota usage
aqe providers set-default <provider>

Success Criteria:

• ✅ aqe init supports provider selection
• ✅ aqe providers commands work
• ✅ Clear error messages
• ✅ Documentation updated

Dependencies: Task 1.3.2

---

Phase 1 Summary

Deliverables:

• ✅ OllamaProvider fully implemented and tested
• ✅ All 31 agents refactored to use IAgentLLM
• ✅ Provider configuration system complete
• ✅ CLI commands for provider management
• ✅ 0 direct Claude imports remaining
• ✅ Quality baseline ≥90% retained

Files Modified: ~50 files Files Created: ~15 files Tests Added: ~30 test files

Parallel Work Streams: 4 concurrent streams Estimated Timeline: 4-6 weeks

---

Phase 2: Intelligent Routing (Weeks 7-10)

🎯 Goal

Implement smart model selection, cost-aware routing, and task complexity analysis.

Milestone 2.1: Task Complexity Analyzer (Week 7)

Task 2.1.1: ML-Based Complexity Classification

Agent: coder (Stream 1) Priority: HIGH Effort: Medium

Files to Create:

• src/routing/ComplexityClassifier.ts
• src/routing/training/ComplexityTrainingData.ts

Current Implementation (Heuristic-based):

// From HybridRouter.ts (lines 218-233)
enum TaskComplexity {
  SIMPLE,        // Pattern matching, simple Q&A
  MODERATE,      // Standard reasoning
  COMPLEX,       // Deep reasoning, code gen
  VERY_COMPLEX   // Advanced analysis
}

// Current scoring (basic heuristics):
- Content length >5000 chars = +2 points
- Max tokens >4000 = +2 points
- Message count >5 = +1 point
- Code patterns present = +1 point
- Complex keywords = +1 point

New ML-Based Implementation:

export class ComplexityClassifier {
  private model: LogisticRegressionModel | DecisionTreeModel;
  private trainingData: ComplexityTrainingData;

  constructor() {
    // Use lightweight ML model (logistic regression or decision tree)
    // Train on historical routing decisions
  }

  async classifyTask(request: LLMCompletionOptions): Promise<TaskComplexity> {
    // Extract features:
    const features = {
      contentLength: request.messages.reduce((sum, m) => sum + m.content.length, 0),
      tokenCount: await this.estimateTokens(request),
      messageCount: request.messages.length,
      hasCodeBlocks: this.detectCodePatterns(request),
      keywordComplexity: this.analyzeKeywords(request),
      promptEntropy: this.calculateEntropy(request),
      contextWindowUsage: this.estimateContextUsage(request)
    };

    // Classify using trained model
    return this.model.predict(features);
  }

  async train(historicalData: RoutingHistoryEntry[]): Promise<void> {
    // Train on last 10,000 routing decisions
    // Use actual outcomes (success/failure, latency, cost) as labels
  }
}

Training Data Collection:

export interface RoutingHistoryEntry {
  requestFeatures: TaskFeatures;
  selectedProvider: string;
  selectedModel: string;
  actualComplexity: TaskComplexity;  // Inferred from performance
  success: boolean;
  latency: number;
  cost: number;
  quality: number;  // User feedback or automated scoring
}

Success Criteria:

• ✅ ML classifier replaces heuristic scoring
• ✅ Trained on ≥1,000 historical decisions
• ✅ Accuracy ≥85% vs manual classification
• ✅ Inference time <10ms

Dependencies: None (uses existing routing history)

---

Task 2.1.2: Complexity-Based Model Routing

Agent: coder (Stream 1) Priority: HIGH Effort: Low

Files to Modify:

• src/providers/HybridRouter.ts (integrate ComplexityClassifier)

Changes:

// Update analyzeTaskComplexity() method
private async analyzeTaskComplexity(options: LLMCompletionOptions): Promise<TaskComplexity> {
  // NEW: Use ML classifier instead of heuristics
  return this.complexityClassifier.classifyTask(options);
}

// Update routeRequest() to use complexity for model selection
private async routeRequest(options: LLMCompletionOptions): Promise<ILLMProvider> {
  const complexity = await this.analyzeTaskComplexity(options);

  // Route based on complexity:
  // SIMPLE → Local (Ollama) or Free tier (Groq)
  // MODERATE → Free tier (Groq) or Low-cost cloud (OpenRouter)
  // COMPLEX → Mid-tier cloud (OpenRouter premium models)
  // VERY_COMPLEX → Premium cloud (Claude Sonnet/Opus)
}

Success Criteria:

• ✅ HybridRouter uses ML classifier
• ✅ Routing decisions improve over time
• ✅ Cost optimization vs. quality tradeoff

Dependencies: Task 2.1.1

---

Milestone 2.2: Model Capability Registry (Week 8)

Task 2.2.1: Model Metadata Store

Agent: coder (Stream 1) Priority: HIGH Effort: Medium

Files to Create:

• src/routing/ModelCapabilityRegistry.ts
• src/routing/data/model-capabilities.json

Implementation:

export interface ModelCapabilities {
  modelId: string;
  provider: string;

  // Core specs
  parameters: string;  // e.g., "30B", "72B", "123B"
  contextWindow: number;
  pricing?: {
    inputPer1M: number;
    outputPer1M: number;
  };

  // Capabilities
  supportedTasks: TaskType[];
  strengths: string[];  // e.g., "code generation", "reasoning"
  weaknesses: string[];

  // Performance benchmarks
  benchmarks?: {
    humanEval?: number;
    sweBench?: number;
    aiderPolyglot?: number;
  };

  // Deployment
  availableOn: ('ollama' | 'openrouter' | 'groq' | 'together' | 'claude')[];
  requiresGPU: boolean;
  vramRequired?: number;  // GB

  // Quality ratings (user feedback + automated scoring)
  qualityRatings?: {
    testGeneration?: number;  // 0-1
    codeReview?: number;
    bugDetection?: number;
    documentation?: number;
  };
}

export class ModelCapabilityRegistry {
  private models: Map<string, ModelCapabilities>;

  async loadModels(): Promise<void> {
    // Load from model-capabilities.json
  }

  getBestModelForTask(task: TaskType, complexity: TaskComplexity, constraints?: ModelConstraints): string {
    // Select best model matching:
    // - Task type
    // - Complexity level
    // - Cost constraints
    // - Availability constraints
  }

  async updateQualityRating(modelId: string, task: TaskType, rating: number): Promise<void> {
    // Update quality rating based on user feedback
  }
}

interface ModelConstraints {
  maxCostPer1M?: number;
  requiresLocal?: boolean;
  preferFree?: boolean;
  minContextWindow?: number;
}

Model Database (Initial Data):

{
  "models": [
    {
      "modelId": "qwen3-coder-30b",
      "provider": "ollama",
      "parameters": "30B (3.3B active)",
      "contextWindow": 262000,
      "pricing": { "inputPer1M": 0, "outputPer1M": 0 },
      "supportedTasks": ["test-generation", "code-review", "refactoring"],
      "strengths": ["agentic workflows", "cost efficiency", "local deployment"],
      "benchmarks": { "aiderPolyglot": 0.55 },
      "availableOn": ["ollama"],
      "requiresGPU": true,
      "vramRequired": 15
    },
    {
      "modelId": "moonshotai/kimi-dev-72b",
      "provider": "openrouter",
      "parameters": "72B",
      "contextWindow": 128000,
      "supportedTasks": ["test-generation", "bug-detection"],
      "strengths": ["SWE-bench SOTA", "autonomous bug fixing"],
      "benchmarks": { "sweBench": 0.604 },
      "availableOn": ["openrouter"],
      "requiresGPU": false
    },
    {
      "modelId": "mistralai/devstral-2-123b",
      "provider": "openrouter",
      "parameters": "123B",
      "contextWindow": 256000,
      "pricing": { "inputPer1M": 0.4, "outputPer1M": 2.0 },
      "supportedTasks": ["code-review", "refactoring"],
      "strengths": ["SWE-bench highest", "file system operations"],
      "benchmarks": { "sweBench": 0.722 },
      "availableOn": ["openrouter"],
      "requiresGPU": false
    }
    // ... add top 20 models from research
  ]
}

Success Criteria:

• ✅ Model registry with 20+ models
• ✅ Capability matching algorithm implemented
• ✅ Quality ratings updateable
• ✅ Fast lookup (<5ms)

Dependencies: None (can run in parallel)

---

Task 2.2.2: Intelligent Model Selection

Agent: coder (Stream 1) Priority: HIGH Effort: Medium

Files to Modify:

• src/providers/HybridRouter.ts
• src/providers/LLMProviderFactory.ts

Implementation:

// In HybridRouter
private async selectBestModel(
  taskType: TaskType,
  complexity: TaskComplexity,
  options: LLMCompletionOptions
): Promise<{ provider: ILLMProvider; model: string }> {

  // 1. Get model constraints from config
  const constraints = this.getModelConstraints(options);

  // 2. Query registry for best model
  const modelId = this.modelRegistry.getBestModelForTask(
    taskType,
    complexity,
    constraints
  );

  // 3. Get provider for that model
  const provider = this.getProviderForModel(modelId);

  // 4. Return provider + model combo
  return { provider, model: modelId };
}

Success Criteria:

• ✅ Routing considers model capabilities
• ✅ Cost vs. quality optimization
• ✅ Task-specific model selection

Dependencies: Task 2.2.1

---

Milestone 2.3: Cost-Aware Router Enhancement (Week 9-10)

Task 2.3.1: Advanced Cost Tracking

Agent: coder (Stream 1) Priority: MEDIUM Effort: Low

Files to Modify:

• src/providers/HybridRouter.ts (enhance cost tracking)

Features to Add:

interface CostSavingsReport {
  // Existing fields:
  totalRequests: number;
  localRequests: number;
  cloudRequests: number;
  totalCost: number;
  estimatedCloudCost: number;
  savings: number;
  savingsPercentage: number;
  cacheHits: number;
  cacheSavings: number;

  // NEW fields:
  costByProvider: Map<string, number>;
  costByTaskType: Map<TaskType, number>;
  costByModel: Map<string, number>;
  averageCostPerRequest: number;
  topCostlyTasks: Array<{ task: TaskType; cost: number }>;
  monthlyCostProjection: number;
  budgetUtilization?: number;  // If budget set
}

// NEW: Budget management
interface BudgetConfig {
  monthlyBudget?: number;
  dailyBudget?: number;
  perTaskBudget?: Map<TaskType, number>;
  alertThreshold: number;  // Percentage (e.g., 0.8 = 80%)
}

async setBudget(config: BudgetConfig): Promise<void>
async getBudgetStatus(): Promise<BudgetStatus>
async enforcebudget(options: LLMCompletionOptions): Promise<boolean>

Success Criteria:

• ✅ Per-provider cost tracking
• ✅ Per-task-type cost tracking
• ✅ Budget limits and alerts
• ✅ Monthly cost projections

Dependencies: None

---

Task 2.3.2: Cost Optimization Strategies

Agent: coder (Stream 1) Priority: MEDIUM Effort: Medium

Files to Modify:

• src/providers/HybridRouter.ts

Optimization Strategies:

// 1. Prompt compression (6-10% savings)
private compressPrompt(prompt: string): string {
  // Remove redundant whitespace
  // Abbreviate common patterns
  // Use token-efficient phrasing
}

// 2. Batch similar requests
private async batchSimilarRequests(requests: LLMCompletionOptions[]): Promise<LLMCompletionResponse[]> {
  // Group by similarity
  // Send as single batch request (50% savings on many providers)
}

// 3. Smart caching with expiration
private getCacheStrategy(taskType: TaskType): CacheStrategy {
  switch (taskType) {
    case 'test-generation':
      return { ttl: 3600, aggressive: true };  // Test patterns stable
    case 'code-review':
      return { ttl: 1800, aggressive: false }; // Code changes frequently
    case 'documentation':
      return { ttl: 7200, aggressive: true };  // Docs stable
  }
}

// 4. Model right-sizing
private shouldDowngradeModel(complexity: TaskComplexity, budget: BudgetConfig): boolean {
  // If over budget, use smaller models for simple/moderate tasks
}

Success Criteria:

• ✅ Prompt compression implemented
• ✅ Batch request support
• ✅ Task-specific caching strategies
• ✅ Model right-sizing when over budget

Dependencies: Task 2.3.1

---

Phase 2 Summary

Deliverables:

• ✅ ML-based complexity classification
• ✅ Model capability registry with 20+ models
• ✅ Intelligent model selection
• ✅ Advanced cost tracking and optimization
• ✅ Budget management system

Files Modified: ~10 files Files Created: ~5 files Tests Added: ~10 test files

Parallel Work Streams: 2 concurrent streams Estimated Timeline: 3-4 weeks

---

Phase 3: Resilience (Weeks 11-13)

🎯 Goal

Implement automatic failover, health monitoring, and graceful degradation.

Milestone 3.1: Provider Health Monitor (Week 11)

Task 3.1.1: Health Check System

Agent: coder (Stream 1) Priority: HIGH Effort: Medium

Files to Create:

• src/monitoring/ProviderHealthMonitor.ts

Implementation:

export interface ProviderHealth {
  providerId: string;
  status: 'healthy' | 'degraded' | 'unhealthy';
  lastCheck: Date;
  responseTime: number;  // ms
  errorRate: number;     // 0-1
  consecutiveFailures: number;
  availability: number;  // 0-1 (last 24h)
}

export class ProviderHealthMonitor {
  private healthStatus: Map<string, ProviderHealth>;
  private checkInterval: NodeJS.Timeout;

  async startMonitoring(intervalMs: number = 60000): Promise<void> {
    // Every 60 seconds:
    this.checkInterval = setInterval(async () => {
      for (const provider of this.providers) {
        await this.checkProviderHealth(provider);
      }
    }, intervalMs);
  }

  private async checkProviderHealth(provider: ILLMProvider): Promise<ProviderHealth> {
    const startTime = Date.now();

    try {
      const health = await provider.healthCheck();
      const responseTime = Date.now() - startTime;

      return {
        providerId: provider.getMetadata().name,
        status: health.status === 'healthy' ? 'healthy' : 'degraded',
        lastCheck: new Date(),
        responseTime,
        errorRate: this.calculateErrorRate(provider),
        consecutiveFailures: 0,
        availability: this.calculateAvailability(provider)
      };
    } catch (error) {
      return {
        providerId: provider.getMetadata().name,
        status: 'unhealthy',
        lastCheck: new Date(),
        responseTime: Date.now() - startTime,
        errorRate: 1.0,
        consecutiveFailures: this.incrementFailureCount(provider),
        availability: this.calculateAvailability(provider)
      };
    }
  }

  async getHealthReport(): Promise<ProviderHealth[]> {
    return Array.from(this.healthStatus.values());
  }

  async getHealthyProviders(): Promise<string[]> {
    return Array.from(this.healthStatus.values())
      .filter(h => h.status === 'healthy')
      .map(h => h.providerId);
  }
}

Success Criteria:

• ✅ Periodic health checks (60s default)
• ✅ Response time tracking
• ✅ Error rate calculation
• ✅ Availability metrics (24h window)

Dependencies: None

---

Task 3.1.2: Health Dashboard

Agent: coder (Stream 5) Priority: MEDIUM Effort: Low

Files to Create:

• src/cli/commands/providers.ts (add status command)
• src/monitoring/HealthDashboard.ts (optional web dashboard)

CLI Command:

$ aqe providers status

Provider Health Status (Last 24h)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Provider       Status    Uptime   Avg Latency  Error Rate
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ollama         ✅ Healthy 99.8%   850ms        0.2%
groq           ✅ Healthy 99.5%   1200ms       0.5%
openrouter     🟡 Degraded 95.2%  2500ms       4.8%
claude         ✅ Healthy 100%    1800ms       0.0%
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Circuit Breakers:
  openrouter: HALF_OPEN (testing recovery)

Recommendations:
  ⚠️  OpenRouter experiencing elevated latency
  ℹ️  Consider routing critical tasks to ollama or groq

Success Criteria:

• ✅ CLI status command works
• ✅ Clear health visualization
• ✅ Recommendations based on health

Dependencies: Task 3.1.1

---

Milestone 3.2: Multi-Provider Fallback Chains (Week 12)

Task 3.2.1: Fallback Chain Configuration

Agent: coder (Stream 3) Priority: HIGH Effort: Medium

Files to Modify:

• src/config/ProviderConfig.ts (add fallback chain support)
• .aqe/providers.yaml.template (add examples)

Enhanced Configuration:

# .aqe/providers.yaml
fallback_chains:
  # Development: Local → Free → Paid
  development:
    - ollama
    - groq
    - openrouter
    - claude

  # Production: Quality → Speed → Cost
  production:
    - claude
    - openrouter
    - groq
    - ollama

  # Free-only: Never use paid services
  free_only:
    - ollama
    - groq
    - openrouter:free  # Only free models

# Per-task fallback overrides
task_fallback_overrides:
  test-generation:
    - ollama:kimi-dev-72b  # Best for test gen
    - groq:llama-3.3-70b
    - openrouter:mistralai/devstral-2512:free

  code-review:
    - openrouter:mistralai/devstral-2-123b
    - claude:claude-sonnet-4
    - ollama:qwen3-coder:30b

Success Criteria:

• ✅ Named fallback chains configurable
• ✅ Per-task chain overrides
• ✅ Free-only constraints respected

Dependencies: None

---

Task 3.2.2: Fallback Logic Implementation

Agent: coder (Stream 1) Priority: HIGH Effort: High

Files to Modify:

• src/providers/HybridRouter.ts (major enhancement)

Implementation:

export class HybridRouter implements ILLMProvider {
  private fallbackChains: Map<string, string[]>;
  private healthMonitor: ProviderHealthMonitor;

  async complete(options: LLMCompletionOptions): Promise<LLMCompletionResponse> {
    // 1. Determine fallback chain
    const chain = this.getFallbackChain(options);

    // 2. Try each provider in chain
    for (let i = 0; i < chain.length; i++) {
      const providerId = chain[i];

      // Skip unhealthy providers
      if (!this.isProviderHealthy(providerId)) {
        continue;
      }

      // Skip providers with open circuit breakers
      if (this.circuitBreakerOpen(providerId)) {
        continue;
      }

      try {
        const provider = this.getProvider(providerId);
        const response = await this.executeWithTimeout(provider, options);

        // Success! Update metrics and return
        this.recordSuccess(providerId);
        return response;

      } catch (error) {
        this.recordFailure(providerId, error);

        // If last provider, re-throw
        if (i === chain.length - 1) {
          throw new Error(`All providers in fallback chain failed: ${error.message}`);
        }

        // Otherwise, continue to next provider
        console.warn(`Provider ${providerId} failed, trying next in chain...`);
      }
    }

    throw new Error('No healthy providers available');
  }

  private getFallbackChain(options: LLMCompletionOptions): string[] {
    // 1. Check for task-specific override
    const taskType = this.detectTaskType(options);
    if (this.taskFallbackOverrides.has(taskType)) {
      return this.taskFallbackOverrides.get(taskType)!;
    }

    // 2. Use mode-specific chain
    const mode = this.config.mode;  // 'development', 'production', 'free_only'
    return this.fallbackChains.get(mode) || this.getDefaultChain();
  }

  private async executeWithTimeout(
    provider: ILLMProvider,
    options: LLMCompletionOptions,
    timeoutMs: number = 30000
  ): Promise<LLMCompletionResponse> {
    return Promise.race([
      provider.complete(options),
      this.timeout(timeoutMs)
    ]);
  }
}

Success Criteria:

• ✅ Automatic provider failover
• ✅ Respects health status
• ✅ Respects circuit breakers
• ✅ Per-task chain support
• ✅ Timeout handling

Dependencies: Task 3.1.1, Task 3.2.1

---

Milestone 3.3: Circuit Breaker Dashboard (Week 13)

Task 3.3.1: Circuit Breaker Visualization

Agent: coder (Stream 5) Priority: LOW Effort: Low

Files to Modify:

• src/cli/commands/providers.ts (enhance status command)

Enhanced Status Command:

$ aqe providers status --detailed

Circuit Breaker Status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Provider       State         Failures  Next Retry
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ollama         CLOSED        0/5       -
groq           CLOSED        0/5       -
openrouter     HALF_OPEN     3/5       in 30s
claude         CLOSED        0/5       -
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Recent Failures:
  openrouter: Rate limit exceeded (429) at 14:23:15
  openrouter: Timeout after 30s at 14:22:45
  openrouter: Connection refused at 14:22:30

Active Fallback Routes:
  test-generation: ollama → groq → claude (openrouter bypassed)
  code-review: claude → ollama (openrouter bypassed)

Success Criteria:

• ✅ Circuit breaker state visualization
• ✅ Failure history
• ✅ Active fallback routes
• ✅ Clear recommendations

Dependencies: Task 3.2.2

---

Phase 3 Summary

Deliverables:

• ✅ Provider health monitoring system
• ✅ Multi-provider fallback chains (3+ providers)
• ✅ Circuit breaker visualization
• ✅ Automatic failover within 5 seconds
• ✅ Health dashboard (CLI + optional web)

Files Modified: ~8 files Files Created: ~4 files Tests Added: ~8 test files

Parallel Work Streams: 2 concurrent streams Estimated Timeline: 2-3 weeks

---

Phase 4: Free Tier Optimization (Weeks 14-15)

🎯 Goal

Support users without infrastructure or budget through free cloud services.

Milestone 4.1: Free Provider Integration (Week 14)

Task 4.1.1: Groq Free Tier Integration

Agent: coder (Stream 1) Priority: HIGH Effort: Low

Files to Create:

• src/providers/GroqProvider.ts (if not exists, extend OpenRouterProvider)

Configuration:

# .aqe/providers.yaml
providers:
  - type: groq
    enabled: true
    priority: 2
    apiKey: ${GROQ_API_KEY}
    defaultModel: llama-3.3-70b-versatile
    limits:
      requestsPerDay: 14400  # Highest free tier
      tokensPerMinute: 6000
    costPer1MTokens:
      input: 0
      output: 0
    features:
      - high_speed  # 300+ tokens/second
      - prompt_caching  # 50% discount on cached tokens

Features:

• ✅ 14,400 requests/day (free tier)
• ✅ 300+ tokens/second inference speed
• ✅ Prompt caching support (50% discount)
• ✅ Batch processing (50% cost reduction)

Success Criteria:

• ✅ Groq provider works with free API key
• ✅ Quota tracking implemented
• ✅ Prompt caching functional
• ✅ Speed optimizations utilized

Dependencies: None

---

Task 4.1.2: OpenRouter Free Models

Agent: coder (Stream 1) Priority: HIGH Effort: Low

Files to Modify:

• src/providers/OpenRouterProvider.ts (enhance free model support)

Free Model Configuration:

// Update OpenRouterProvider.ts
const FREE_MODELS = [
  'mistralai/devstral-2512:free',     // 123B, 256K context
  'mistralai/devstral-small-2505',    // $0.06/$0.12 (cheapest paid)
  'deepseek/deepseek-v3:free',        // Community free
  'meta-llama/llama-3.3-70b:free',    // Via certain providers
  // ... 30+ free models
];

// Fallback chain for free-only mode
const FREE_FALLBACK = [
  'mistralai/devstral-2512:free',     // Primary
  'deepseek/deepseek-v3:free',        // Secondary
  'meta-llama/llama-3.3-70b:free'     // Tertiary
];

Success Criteria:

• ✅ OpenRouter free models configured
• ✅ 50 requests/day limit tracked
• ✅ Automatic fallback to paid if quota exceeded
• ✅ Clear user messaging on quota status

Dependencies: None

---

Milestone 4.2: GitHub Models Integration (Week 14)

Task 4.2.1: GitHub Models Provider

Agent: coder (Stream 1) Priority: MEDIUM Effort: Medium

Files to Create:

• src/providers/GitHubModelsProvider.ts

Implementation:

export class GitHubModelsProvider implements ILLMProvider {
  constructor(private githubToken?: string) {
    // Use GITHUB_TOKEN env var if not provided
    this.githubToken = githubToken || process.env.GITHUB_TOKEN;
  }

  async initialize(): Promise<void> {
    // Verify GitHub token works
    // List available models
  }

  async complete(options: LLMCompletionOptions): Promise<LLMCompletionResponse> {
    // Call GitHub Models API
    // Format: https://models.github.com/v1/chat/completions
  }

  getMetadata(): LLMProviderMetadata {
    return {
      name: 'github-models',
      capabilities: {
        streaming: true,
        embeddings: true,
        multimodal: true,
        functionCalling: true
      },
      supportedModels: [
        'gpt-4o',
        'gpt-4o-mini',
        'o3-mini',
        'phi-4',
        'llama-3.3-70b',
        'mistral-large',
        // ... dozens more
      ],
      pricing: {
        inputCostPer1M: 0,   // FREE
        outputCostPer1M: 0
      }
    };
  }
}

Detection:

// In LLMProviderFactory.ts
private detectEnvironment(): EnvironmentSignals {
  return {
    // ... existing fields
    isGitHubCodespaces: process.env.CODESPACES === 'true',
    hasGitHubToken: !!process.env.GITHUB_TOKEN
  };
}

Success Criteria:

• ✅ GitHub Models provider implemented
• ✅ Auto-detects Codespaces environment
• ✅ Uses GITHUB_TOKEN automatically
• ✅ Access to 30+ models

Dependencies: None

---

Milestone 4.3: Rate Limit Management (Week 15)

Task 4.3.1: Quota Tracking System

Agent: coder (Stream 1) Priority: MEDIUM Effort: Medium

Files to Create:

• src/monitoring/QuotaManager.ts

Implementation:

export interface ProviderQuota {
  providerId: string;
  limits: {
    requestsPerMinute?: number;
    requestsPerDay?: number;
    tokensPerMinute?: number;
    tokensPerDay?: number;
  };
  usage: {
    requestsThisMinute: number;
    requestsToday: number;
    tokensThisMinute: number;
    tokensToday: number;
  };
  resetTimes: {
    minuteReset: Date;
    dayReset: Date;
  };
}

export class QuotaManager {
  private quotas: Map<string, ProviderQuota>;

  async checkQuota(providerId: string, estimatedTokens: number): Promise<boolean> {
    const quota = this.quotas.get(providerId);
    if (!quota) return true;  // No limits

    // Check if within limits
    if (quota.limits.requestsPerDay && quota.usage.requestsToday >= quota.limits.requestsPerDay) {
      return false;
    }

    if (quota.limits.tokensPerDay && quota.usage.tokensToday + estimatedTokens > quota.limits.tokensPerDay) {
      return false;
    }

    return true;
  }

  async recordUsage(providerId: string, tokens: number): Promise<void> {
    // Increment usage counters
    // Reset if past reset time
  }

  async getQuotaStatus(providerId: string): Promise<QuotaStatus> {
    return {
      requestsRemaining: this.calculateRemaining('requests', providerId),
      tokensRemaining: this.calculateRemaining('tokens', providerId),
      resetIn: this.calculateTimeUntilReset(providerId),
      percentUsed: this.calculatePercentUsed(providerId)
    };
  }

  async alertOnThreshold(providerId: string, threshold: number = 0.8): Promise<void> {
    const status = await this.getQuotaStatus(providerId);
    if (status.percentUsed >= threshold) {
      console.warn(`⚠️  ${providerId} at ${Math.round(status.percentUsed * 100)}% quota`);
    }
  }
}

Success Criteria:

• ✅ Quota tracking per provider
• ✅ Automatic quota reset (daily/minute)
• ✅ Alerts at 80% threshold
• ✅ Graceful handling when quota exceeded

Dependencies: None

---

Task 4.3.2: Free Tier CLI Commands

Agent: coder (Stream 3) Priority: MEDIUM Effort: Low

Files to Modify:

• src/cli/commands/providers.ts

New Commands:

# Show quota usage across all free providers
$ aqe providers quota

Free Provider Quota Usage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Provider       Daily Limit  Used    Remaining  Resets In
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
groq           14,400       3,245   11,155     21h 15m
openrouter     50           12      38         21h 15m
github-models  unlimited    456     unlimited  -
google-gemini  1,000        892     108        21h 15m ⚠️
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Warnings:
  ⚠️  google-gemini at 89% quota - consider using groq or github-models

Recommendations:
  ✅ groq has plenty of quota remaining (77%)
  ✅ github-models unlimited - no quota concerns

Success Criteria:

• ✅ aqe providers quota command works
• ✅ Clear visualization of quota status
• ✅ Warnings and recommendations
• ✅ Time until reset displayed

Dependencies: Task 4.3.1

---

Phase 4 Summary

Deliverables:

• ✅ Groq free tier integrated (14,400 req/day)
• ✅ OpenRouter free models configured (30+ models)
• ✅ GitHub Models provider (unlimited in Codespaces)
• ✅ Quota tracking and management
• ✅ User-friendly quota CLI commands
• ✅ Full free-tier workflow functional

Files Modified: ~5 files Files Created: ~3 files Tests Added: ~5 test files

Parallel Work Streams: 2 concurrent streams Estimated Timeline: 1-2 weeks

---

Dependency Graph

graph TD
    Start[Start] --> P1M1[Phase 1 M1: OllamaProvider]
    Start --> P1M2[Phase 1 M2: Agent Abstraction]
    Start --> P1M3[Phase 1 M3: Config System]

    P1M1 --> P1M1T1[Task 1.1.1: Core Provider]
    P1M1T1 --> P1M1T2[Task 1.1.2: Testing]
    P1M1T2 --> P1M1T3[Task 1.1.3: Factory Integration]

    P1M2 --> P1M2T1[Task 1.2.1: IAgentLLM Interface]
    P1M2T1 --> P1M2T2[Task 1.2.2: Adapter]
    P1M2T2 --> P1M2T3[Task 1.2.3: Agent Migration]
    P1M2T3 --> P1M2T4[Task 1.2.4: Testing]

    P1M3 --> P1M3T1[Task 1.3.1: Schema]
    P1M3T1 --> P1M3T2[Task 1.3.2: Loader]
    P1M3T2 --> P1M3T3[Task 1.3.3: CLI]

    P1M1T3 --> Phase1Complete[Phase 1 Complete]
    P1M2T4 --> Phase1Complete
    P1M3T3 --> Phase1Complete

    Phase1Complete --> P2M1[Phase 2 M1: Complexity Analyzer]
    Phase1Complete --> P2M2[Phase 2 M2: Model Registry]

    P2M1 --> P2M1T1[Task 2.1.1: ML Classifier]
    P2M1T1 --> P2M1T2[Task 2.1.2: Routing]

    P2M2 --> P2M2T1[Task 2.2.1: Registry]
    P2M2T1 --> P2M2T2[Task 2.2.2: Selection]

    P2M1T2 --> P2M3[Phase 2 M3: Cost Router]
    P2M2T2 --> P2M3

    P2M3 --> P2M3T1[Task 2.3.1: Tracking]
    P2M3T1 --> P2M3T2[Task 2.3.2: Optimization]

    P2M3T2 --> Phase2Complete[Phase 2 Complete]

    Phase2Complete --> P3M1[Phase 3 M1: Health Monitor]
    Phase2Complete --> P3M2[Phase 3 M2: Fallback Chains]

    P3M1 --> P3M1T1[Task 3.1.1: Health System]
    P3M1T1 --> P3M1T2[Task 3.1.2: Dashboard]

    P3M2 --> P3M2T1[Task 3.2.1: Config]
    P3M2T1 --> P3M2T2[Task 3.2.2: Logic]

    P3M1T1 --> P3M3[Phase 3 M3: Circuit Breaker Dashboard]
    P3M2T2 --> P3M3

    P3M3 --> Phase3Complete[Phase 3 Complete]

    Phase3Complete --> P4M1[Phase 4 M1: Free Providers]
    Phase3Complete --> P4M2[Phase 4 M2: GitHub Models]

    P4M1 --> P4M1T1[Task 4.1.1: Groq]
    P4M1 --> P4M1T2[Task 4.1.2: OpenRouter]

    P4M2 --> P4M2T1[Task 4.2.1: GitHub Provider]

    P4M1T1 --> P4M3[Phase 4 M3: Quota Management]
    P4M1T2 --> P4M3
    P4M2T1 --> P4M3

    P4M3 --> P4M3T1[Task 4.3.1: Quota System]
    P4M3T1 --> P4M3T2[Task 4.3.2: CLI]

    P4M3T2 --> Complete[LLM Independence Complete]

---

File Ownership Matrix

Prevent Merge Conflicts by Clear File Ownership

File/Directory	Owner Agent	Phase	Locked During
src/providers/OllamaProvider.ts	Coder A	1	Weeks 1-2
src/providers/GroqProvider.ts	Coder A	4	Week 14
src/providers/GitHubModelsProvider.ts	Coder A	4	Week 14
src/providers/HybridRouter.ts	Coder A	2-3	Weeks 7-13
src/providers/LLMProviderFactory.ts	Coder A	1	Weeks 2-3
src/agents/*.ts (Group 1)	Coder B	1	Week 2
src/agents/*.ts (Group 2)	Coder B	1	Week 3
src/agents/*.ts (Group 3)	Coder C	1	Week 3
src/subagents/*.ts	Coder D	1	Week 4
src/config/ProviderConfig.ts	Coder C	1	Weeks 3-4
src/config/ProviderConfigLoader.ts	Coder C	1	Weeks 3-4
src/routing/ComplexityClassifier.ts	Coder A	2	Week 7
src/routing/ModelCapabilityRegistry.ts	Coder A	2	Week 8
src/monitoring/ProviderHealthMonitor.ts	Coder A	3	Week 11
src/monitoring/QuotaManager.ts	Coder A	4	Week 15
src/cli/commands/providers.ts	Coder C	1,3,4	Weeks 4,13,15
tests/providers/*.test.ts	Tester A	All	Ongoing
tests/agents/*.test.ts	Tester B	1	Weeks 2-4
tests/integration/*.test.ts	qe-test-generator	All	Ongoing
docs/guides/*.md	Coder D	All	Ongoing
examples/*.ts	Coder D	All	Ongoing

---

Testing Strategy

Test Coverage Requirements

Unit Tests (95% coverage target)

• All provider implementations
• All routing logic
• All configuration loading
• All quota management

Integration Tests (90% coverage target)

• Provider interoperability
• Fallback chain execution
• Health monitoring
• Circuit breaker state transitions

End-to-End Tests (key scenarios)

• Agent using Ollama provider
• Agent using free tier providers
• Automatic failover on provider failure
• Cost tracking accuracy
• Quality baseline comparison

Test Execution Strategy

Per-Phase Testing:

• Phase 1: After each milestone, run full agent test suite
• Phase 2: Benchmark routing decisions against baseline
• Phase 3: Chaos testing (simulate provider failures)
• Phase 4: Free tier quota exhaustion scenarios

Quality Gates:

• All tests must pass before merging to main
• Code coverage must not decrease
• Quality baseline must meet ≥90% threshold
• Performance must not regress >10%

---

Rollout Plan

Gradual Production Rollout

Week 16: Internal Testing (10% traffic)

• Deploy to internal testing environment
• Route 10% of QE Fleet requests through new system
• Monitor for issues
• Collect performance metrics

Week 17: Beta Testing (20% traffic)

• Increase to 20% traffic
• Enable for beta users
• Gather feedback
• Fix any issues

Week 18: Expanded Rollout (50% traffic)

• Increase to 50% traffic
• Monitor cost savings
• Verify quality retention
• Optimize based on usage patterns

Week 19: Full Rollout (80% traffic)

• Increase to 80% traffic
• Keep Claude as 20% control group
• Final optimization

Week 20: LLM Independence Achieved (100% traffic)

• 100% of requests through new system
• Claude reserved for emergency fallback only
• Celebrate independence! 🎉

Rollback Plan

If quality drops below 85%:

1. Immediately revert to 50% traffic
2. Analyze quality issues
3. Fix root cause
4. Resume rollout

If costs exceed budget:

1. Enable aggressive caching
2. Shift more traffic to free tier
3. Implement model right-sizing
4. Adjust budget thresholds

If providers consistently fail:

1. Activate all fallback chains
2. Investigate provider health
3. Update circuit breaker settings
4. Consider provider alternatives

---

Success Metrics

Final Success Criteria

Metric	Target	Measurement
Independence	80%+ requests via non-Claude providers	Request logs
Quality	≥95% vs Claude baseline	Automated quality scoring
Cost Reduction	60%+ decrease	Monthly cost reports
Latency P50	<2s (simple), <5s (complex)	Prometheus metrics
Latency P95	<5s (simple), <10s (complex)	Prometheus metrics
Reliability	99.5% uptime	Health monitoring
Agent Migration	0/31 direct Claude imports	Static analysis
Free Tier Support	Users can run full workflow	Manual testing

Monitoring Dashboard

Post-implementation, monitor:

• Cost per request by provider
• Quality scores by agent type
• Fallback chain activation frequency
• Provider health trends
• Quota utilization rates

---

Budget & Resources

Estimated Costs

Phase	Description	Estimated Cost
Phase 1	Local development (hardware amortized)	$1,000
Phase 2	Cloud API testing	$2,000
Phase 3	Reliability testing	$1,000
Phase 4	Free tier testing	$0
Total	Development & Testing	$4,000

Production Operating Costs (Monthly)

Scenario	Current (Claude-only)	After Implementation	Savings
Light Usage (10M tokens)	$300	$50	83%
Medium Usage (50M tokens)	$1,500	$400	73%
Heavy Usage (100M tokens)	$3,000	$900	70%

ROI Analysis

Break-Even:

• Development cost: $4,000
• Monthly savings (medium usage): $1,100
• Break-even: ~3.6 months

12-Month ROI:

• Total savings: $13,200
• Net benefit: $9,200
• ROI: 230%

---

Risk Mitigation

Technical Risks

Risk	Probability	Impact	Mitigation
Ollama API changes	Low	Medium	Version pin, integration tests
Provider rate limiting	Medium	High	Multi-provider fallback
Model quality variance	High	Medium	Complexity-aware routing, quality monitoring
Breaking changes in agents	Medium	High	Comprehensive test suite, gradual rollout

Operational Risks

Risk	Probability	Impact	Mitigation
Free tier quota exhaustion	High	Medium	Quota tracking, user warnings, automatic fallback
Provider outages	Medium	High	Automatic failover chains, health monitoring
Configuration complexity	Medium	Medium	Sensible defaults, validation, examples

Migration Risks

Risk	Probability	Impact	Mitigation
Agent regression	Medium	High	Feature flags, gradual rollout, quality baseline
Performance degradation	Low	Medium	Benchmarks before/after, performance monitoring
User confusion	Medium	Low	Clear documentation, examples, migration guide

---

Next Steps

Immediate Actions (This Week)

1. Close GitHub Issue #142 with reference to this plan
2. Create new tracking issue with implementation milestones
3. Assign agents to work streams
4. Set up parallel execution branches
5. Kick off Phase 1 Milestone 1

Week 1 Execution Plan

Day 1:

• Coder A: Start Task 1.1.1 (OllamaProvider core)
• Coder B: Start Task 1.2.1 (IAgentLLM interface)
• Coder C: Start Task 1.3.1 (Config schema)
• Tester A: Prepare test infrastructure

Day 2-3:

• Coder A: Continue OllamaProvider implementation
• Coder B: Start Task 1.2.2 (Adapter)
• Coder C: Continue config schema
• Tester A: Write OllamaProvider test plan

Day 4-5:

• Coder A: Complete Task 1.1.1, start Task 1.1.3
• Coder B: Complete adapter, start agent migration Group 1
• Coder C: Complete Task 1.3.1, start Task 1.3.2
• Tester A: Start Task 1.1.2 (OllamaProvider tests)

End of Week:

• ✅ OllamaProvider 80% complete
• ✅ IAgentLLM interface defined
• ✅ Config schema designed
• ✅ Test infrastructure ready

---

Appendix A: Agent Assignment Table

Agent ID	Type	Work Stream	Primary Files	Availability
coder-a	coder	Stream 1 (Providers)	src/providers/*	Full-time
coder-b	coder	Stream 2 (Agents 1-2)	src/agents/* (Group 1-2)	Full-time
coder-c	coder	Stream 3 (Config)	src/config/*, CLI	Full-time
coder-d	coder	Stream 2 (Agents 3-4)	src/agents/* (Group 3), src/subagents/*	Part-time
coder-e	coder	Stream 5 (Docs)	docs/*, examples/*	Part-time
tester-a	tester	Stream 4 (Provider Tests)	tests/providers/*	Full-time
tester-b	tester	Stream 4 (Agent Tests)	tests/agents/*	Part-time
qe-test-gen	qe-test-generator	Stream 4 (Integration)	tests/integration/*	On-demand
reviewer-a	reviewer	All Streams	Code review	On-demand

---

Appendix B: Configuration Examples

Example 1: Local-First Configuration

# .aqe/providers.yaml
mode: local_first

providers:
  - type: ollama
    enabled: true
    priority: 1
    baseUrl: http://localhost:11434
    defaultModel: qwen3-coder:30b
    fallbackProvider: groq

  - type: groq
    enabled: true
    priority: 2
    apiKey: ${GROQ_API_KEY}
    defaultModel: llama-3.3-70b-versatile
    fallbackProvider: claude

  - type: claude
    enabled: true
    priority: 3
    apiKey: ${ANTHROPIC_API_KEY}
    defaultModel: claude-sonnet-4

fallback_chains:
  development:
    - ollama
    - groq
    - claude

Example 2: Free-Tier-Only Configuration

# .aqe/providers.yaml
mode: free_only

providers:
  - type: groq
    enabled: true
    priority: 1
    apiKey: ${GROQ_API_KEY}
    defaultModel: llama-3.3-70b-versatile
    limits:
      requestsPerDay: 14400

  - type: openrouter
    enabled: true
    priority: 2
    apiKey: ${OPENROUTER_API_KEY}
    defaultModel: mistralai/devstral-2512:free
    limits:
      requestsPerDay: 50

  - type: github-models
    enabled: true
    priority: 3
    # Uses GITHUB_TOKEN automatically

fallback_chains:
  free_only:
    - groq
    - openrouter
    - github-models

Example 3: Production Configuration

# .aqe/providers.yaml
mode: hybrid

providers:
  - type: claude
    enabled: true
    priority: 1
    apiKey: ${ANTHROPIC_API_KEY}
    defaultModel: claude-sonnet-4
    modelOverrides:
      test-generation: claude-opus-4
      code-review: claude-sonnet-4

  - type: openrouter
    enabled: true
    priority: 2
    apiKey: ${OPENROUTER_API_KEY}
    defaultModel: mistralai/devstral-2-123b
    modelOverrides:
      test-generation: moonshotai/kimi-dev-72b

  - type: groq
    enabled: true
    priority: 3
    apiKey: ${GROQ_API_KEY}
    defaultModel: llama-3.3-70b-versatile

fallback_chains:
  production:
    - claude
    - openrouter
    - groq

---

Document Metadata

Created: December 23, 2025 Authors: Claude Code GOAP Planner, Research Agents Version: 1.0 Status: Ready for Execution Estimated Completion: Week 20 (May 2026)

Related Documents:

• GitHub Issue #142
• GOAP Strategic Plan
• Multi-Model Router Analysis
• Open-Source LLM Models Research
• LLM Providers Comparison
• Free Cloud LLM Services

---

End of Implementation Plan