proffesor-for-testing__agentic-qe/docs/implementation/stream3-config-system.md

Stream 3: Configuration System Implementation

Overview

Implemented a comprehensive multi-provider configuration management system that enables LLM provider independence through YAML-based configuration files.

Implementation Date

2024-12-23

Components Delivered

1. Core Configuration Types (src/config/ProviderConfig.ts)

Key Features:

• Support for 8 LLM providers: Ollama, Groq, OpenRouter, Claude, ruvLLM, Google, Together, GitHub
• 4 deployment modes: local_first, hosted, free_only, hybrid
• 9 task types for intelligent model routing
• Rate limiting configuration
• Cost tracking and budgeting
• Fallback chain configuration
• Provider capabilities detection

Types Exported:

• ProviderConfig - Individual provider configuration
• MultiProviderConfig - Multi-provider system configuration
• TaskType - Task-specific routing types
• DeploymentMode - Deployment strategy enumeration
• RateLimitConfig - Rate limiting configuration
• CostConfig - Cost tracking configuration
• DEFAULT_PROVIDER_CONFIGS - Default configurations for all 8 providers
• DEFAULT_MODE_CONFIGS - Default configurations for all 4 modes

2. Configuration Loader (src/config/ConfigLoader.ts)

Key Features:

• YAML file loading from multiple locations
• Environment variable loading and interpolation
• Configuration validation with detailed error reporting
• Deep merging with defaults
• Configuration saving

Supported Locations:

1. .aqe/providers.yaml
2. .aqe/providers.yml
3. aqe.config.yaml
4. aqe.config.yml
5. Custom path via configPath option

Environment Variable Support:

• ${VAR_NAME} syntax interpolation
• $VAR_NAME syntax interpolation
• Automatic provider detection based on API keys
• Standard variable mapping:
  • ANTHROPIC_API_KEY → claude.apiKey
  • OPENROUTER_API_KEY → openrouter.apiKey
  • GROQ_API_KEY → groq.apiKey
  • GOOGLE_API_KEY → google.apiKey
  • TOGETHER_API_KEY → together.apiKey
  • GITHUB_TOKEN → github.apiKey
  • OLLAMA_HOST → ollama.baseUrl
  • RUVLLM_HOST → ruvllm.baseUrl
  • LLM_PROVIDER → defaultProvider
  • LLM_MODE → mode

Validation Features:

• Mode validation
• Provider type validation
• Required field validation
• Rate limit validation
• Cost configuration validation
• Fallback chain validation
• Warnings for potential issues

3. Module Exports (src/config/index.ts)

Central export point for all configuration functionality.

4. Example Configurations (examples/configs/)

Four production-ready configuration templates:

local-first.yaml

• Use Case: Development, cost-sensitive, privacy-focused
• Primary: Ollama (local, free)
• Fallback: OpenRouter (cost-effective cloud)
• Cost: $0-5/day
• Model Routing:
  • Test generation → Qwen 2.5 Coder 7B
  • Code review → DeepSeek Coder 6.7B
  • Documentation → Llama 3.2 3B

hosted-only.yaml

• Use Case: Production, CI/CD, enterprise
• Primary: Groq (fast, generous free tier)
• Secondary: OpenRouter (multi-model routing)
• Tertiary: Claude API (premium quality)
• Cost: $10-50/day
• Model Routing:
  • Test generation → Llama 3.3 70B
  • Code review → Claude 3.5 Sonnet
  • Security → Claude 3.5 Sonnet

free-tier.yaml

• Use Case: Open-source, learning, personal projects
• Providers: Groq → Google Gemini → GitHub Models → Ollama
• Cost: $0/day (100% free)
• Model Routing:
  • Test generation → Llama 3.3 70B (Groq)
  • Code review → Gemini 2.0 Flash
  • Documentation → Gemini 1.5 Flash

hybrid.yaml

• Use Case: Adaptive workloads, cost optimization
• Strategy: Intelligent routing based on task complexity
• Providers: Ollama (simple) → Groq (complex) → OpenRouter (critical)
• Cost: $5-20/day (adaptive)
• Model Routing:
  • Documentation → Llama 3.2 3B (local)
  • Test generation → Llama 3.3 70B (Groq)
  • Security → Claude 3.5 Sonnet (OpenRouter)

5. Documentation

Configuration Guide (docs/guides/configuration-guide.md)

Comprehensive 400+ line guide covering:

• Quick start instructions
• Deployment mode deep-dives
• Configuration syntax reference
• Task-specific routing examples
• Environment variable setup
• Cost management strategies
• Health monitoring
• Fallback chains
• Troubleshooting
• Best practices

Examples README (examples/configs/README.md)

Detailed README with:

• Configuration file descriptions
• Deployment mode comparison table
• Quick start instructions
• Configuration syntax examples
• Troubleshooting tips
• Next steps

6. Test Suite (tests/config/ConfigLoader.test.ts)

Comprehensive test coverage for:

• YAML file loading
• Environment variable loading
• Variable interpolation (${VAR} and $VAR)
• Configuration validation
• Default merging
• Configuration saving
• Integration scenarios

Test Scenarios (10+ tests):

• Valid YAML loading
• Non-existent file handling
• Default location detection
• Environment variable detection
• Auto-provider initialization
• Variable interpolation (both syntaxes)
• Mode-specific defaults
• Provider-specific defaults
• Validation (valid/invalid configs)
• Error reporting

Architecture Decisions

1. YAML as Primary Format

Rationale: Human-readable, widely supported, excellent for configuration

2. Environment Variable Interpolation

Rationale: Enables secrets management, CI/CD integration, 12-factor app compliance

3. Multi-Source Configuration Loading

Rationale: Flexibility for different deployment scenarios

4. Deep Merge Strategy

Rationale: Allows partial configuration with sensible defaults

5. Validation with Warnings

Rationale: Catch errors early while allowing flexibility

Integration Points

Current Integration Required

The configuration system is ready to integrate with:

1. LLMProviderFactory (src/providers/LLMProviderFactory.ts)

   • Update factory to accept MultiProviderConfig
   • Add configuration-based initialization
   • Respect priority ordering
   • Implement rate limiting
   • Implement cost budgets

2. CLI (future work)

   • aqe config show - Display current config
   • aqe config validate - Validate config file
   • aqe config init - Initialize from template
   • aqe providers test <provider> - Test specific provider

3. Agent System (future work)

   • Use task-specific model routing
   • Respect fallback chains
   • Track costs per agent

Usage Examples

Loading Configuration

import { loadConfig } from './src/config';

// Auto-load from default locations
const config = await loadConfig();

// Load from specific file
const config = await loadConfig({
  configPath: './my-config.yaml',
  loadFromEnv: true,
  mergeDefaults: true,
  validate: true,
});

Validation

import { validateConfig } from './src/config';

const result = validateConfig(config);

if (!result.valid) {
  console.error('Validation errors:', result.errors);
} else if (result.warnings.length > 0) {
  console.warn('Warnings:', result.warnings);
}

Environment Variables

# Set API keys
export GROQ_API_KEY="gsk_..."
export OPENROUTER_API_KEY="sk-or-v1-..."

# Override mode
export LLM_MODE="local_first"

# Initialize
aqe init

Cost Management

Budget Configuration

costBudget:
  daily: 10.0          # $10/day limit
  monthly: 250.0       # $250/month limit
  warnThreshold: 80    # Warn at 80%
  enforceLimit: true   # Block when exceeded

Cost Tracking

All providers include cost configuration:

providers:
  - type: groq
    costPer1MTokens:
      input: 0.59
      output: 0.79

Premium providers with caching:

providers:
  - type: claude
    costPer1MTokens:
      input: 3.0
      output: 15.0
      cacheReadMultiplier: 0.1   # 90% discount
      cacheWriteMultiplier: 1.25  # 25% premium

Task-Specific Routing

Route different task types to specialized models:

providers:
  - type: openrouter
    defaultModel: anthropic/claude-3.5-haiku

    modelOverrides:
      test-generation: qwen/qwen-2.5-coder-32b-instruct
      code-review: anthropic/claude-3.5-sonnet
      security-scanning: anthropic/claude-3.5-sonnet
      documentation: google/gemini-flash-1.5-8b

Next Steps

Stream 4: Factory Integration (Coder Agent D)

Update LLMProviderFactory to:

1. Accept MultiProviderConfig in constructor
2. Initialize providers based on configuration
3. Respect priority ordering
4. Implement rate limiting
5. Implement cost tracking and budgets
6. Support hot-swapping configurations

Stream 5: CLI Commands (Future)

Implement CLI commands:

• aqe config show
• aqe config validate
• aqe config init --template local-first
• aqe providers list
• aqe providers test groq
• aqe costs summary

Files Created

Source Code

1. /workspaces/agentic-qe-cf/src/config/ProviderConfig.ts - Core types (530 lines)
2. /workspaces/agentic-qe-cf/src/config/ConfigLoader.ts - Loader implementation (450 lines)
3. /workspaces/agentic-qe-cf/src/config/index.ts - Module exports (30 lines)

Example Configurations

4. /workspaces/agentic-qe-cf/examples/configs/local-first.yaml - Local-first template
5. /workspaces/agentic-qe-cf/examples/configs/hosted-only.yaml - Hosted-only template
6. /workspaces/agentic-qe-cf/examples/configs/free-tier.yaml - Free-tier template
7. /workspaces/agentic-qe-cf/examples/configs/hybrid.yaml - Hybrid template
8. /workspaces/agentic-qe-cf/examples/configs/README.md - Examples documentation

Documentation

9. /workspaces/agentic-qe-cf/docs/guides/configuration-guide.md - Comprehensive guide
10. /workspaces/agentic-qe-cf/docs/implementation/stream3-config-system.md - This file

Tests

11. /workspaces/agentic-qe-cf/tests/config/ConfigLoader.test.ts - Test suite

Success Criteria

• ✅ TypeScript types for all config options
• ✅ YAML config loader with validation
• ✅ Environment variable interpolation works
• ✅ Example configs for all 4 deployment modes (local_first, hosted, free_only, hybrid)
• ✅ Config schema documented
• ✅ Comprehensive test suite
• ✅ Production-ready examples
• ✅ Integration documentation

Dependencies

• yaml@2.8.2 - Already installed ✅
• @jest/globals - Already installed ✅
• No additional dependencies required

Summary

Successfully implemented a production-ready multi-provider configuration system with:

• 11 files created
• 8 providers supported
• 4 deployment modes
• 9 task types for routing
• Environment variable interpolation
• Comprehensive validation
• Extensive documentation
• Full test coverage

The system is ready for integration with the LLMProviderFactory (Stream 4) and future CLI commands (Stream 5).