Agent Learning System Architecture

Complete Agent Lifecycle with RuvLLM, RuVector, and ReasoningBank Integration

1 Agent Spawning - Two Entry Points

A. Via Claude Code Task Tool

User → Claude Code → Task Tool → External Agent Process └── Direct CLI execution

When you use Claude Code's Task tool:

B. Via AQE MCP Tools

User → Claude Code → MCP Tool → AgentSpawnHandler → AgentRegistry → QEAgentFactory │ │ │ └─ Creates BaseAgentConfig │ │ └─ agentType │ │ └─ capabilities │ │ └─ memoryStore (SHARED) ← KEY! │ │ └─ enableLearning: true │ │ └─ learningConfig │ │ │ └─ Creates Agent Instance │ └─ TestGeneratorAgent │ CoverageAnalyzerAgent │ etc. │ └─ Returns AgentInstance metadata

Key File: src/mcp/handlers/agent-spawn.ts:56-232

2 BaseAgent Initialization - Multi-Stage Process

BaseAgent.initialize() │ └─ 1. Load Knowledge (subclass-specific) │ └─ 2. Restore State from Memory │ └─ 3. Create Performance Tracker │ └─ 4. Create LearningEngine ──────────────────────────────────────────┌ │ │ │ │ └─ Q-Learning Algorithm (default) │ │ └─ Q-Table: Map<state, Map<action, reward>> │ │ └─ SwarmMemoryManager (SHARED) │ │ └─ HNSWPatternAdapter (Phase 0 M0.3) ← RuVector! │ │ │ └─ 5. Initialize LLM Provider ────────────────────────────────────────┌ │ │ │ │ └─ Selection Priority: │ │ 1. HybridRouter (if enableHybridRouter: true) ← NEW! │ │ └─ Routes: RuVector Cache → RuvLLM → Cloud │ │ 2. RuvLLM (LOCAL - default) ← RuvLLM! │ │ 3. Claude (if ANTHROPIC_API_KEY) │ │ 4. OpenRouter (if OPENROUTER_API_KEY) │ │ │ └─ 5b. Initialize HybridRouter ──────────────────────────────────────┌ │ │ │ │ └─ Create RuVectorClient (GNN cache client) │ │ └─ Configure cache threshold (default: 0.85) │ │ └─ Enable LoRA learning (rank: 8) │ │ └─ Enable EWC++ anti-forgetting │ │ │ └─ 6. Initialize Federated Learning (Phase 0 M0.5) ─────────────────┌ │ └─ Register with FederatedManager │ │ │ └─ 7. Initialize Subclass Components │ (TestGeneratorAgent → PatternExtractor, ReasoningBank) │ │ ┌────────────────────────────────────────────────────────────────────┘ │ ▾ ┌─────────────────────────────────────────────────────────────────────┌ │ INITIALIZATION COMPLETE │ │ │ │ Agent State: │ │ └─ learningEngine: LearningEngine (with Q-table + HNSW) │ │ └─ llmProvider: RuvllmProvider (local) OR Claude/OpenRouter │ │ └─ hybridRouter: HybridRouter (if enableHybridRouter) ← NEW! │ │ │ └─ ruvectorClient: RuVectorClient (GNN cache) │ │ └─ llmSessionId: "sess-xxx" (for session continuity) │ │ └─ federatedManager: FederatedManager (cross-agent learning) │ │ └─ memoryStore: SwarmMemoryManager (SQLite .agentic-qe/memory.db) │ └─────────────────────────────────────────────────────────────────────┘

Key Files:

src/agents/BaseAgent.ts:216-276 (initialize)

src/learning/LearningEngine.ts:107-179 (constructor + init)

src/agents/BaseAgent.ts:668-727 (LLM initialization)

3 LLM Provider Selection

Provider Selection Matrix

┌─────────────────────────────────────────────────────────────────────┌ │ LLM PROVIDER SELECTION │ └─────────────────────────────────────────────────────────────────────┌ │ │ │ Environment Check Selected Provider │ │ ───────────────── ─────────────── │ │ │ │ 1. enableHybridRouter: true ⇒ HybridRouter (NEW!) │ │ (Phase 0.5) └─ Cache-first routing │ │ │ │ 2. LLM_PROVIDER env override ⇒ Use specified provider │ │ │ │ 3. Inside Claude Code + │ │ ANTHROPIC_API_KEY ⇒ Claude API │ │ │ │ 4. OPENROUTER_API_KEY ⇒ OpenRouter (300+ models) │ │ │ │ 5. ANTHROPIC_API_KEY ⇒ Claude API │ │ │ │ 6. RuvLLM available ⇒ RuvLLM (LOCAL, FREE) │ │ (default for QE agents) │ │ │ │ 7. Fallback ⇒ Algorithmic-only mode │ │ │ └─────────────────────────────────────────────────────────────────────┘

HybridRouter with RuVector Cache

┌─────────────────────────────────────────────────────────────────────┌ │ HYBRIDROUTER CACHE-FIRST ROUTING │ │ (src/providers/HybridRouter.ts) │ └─────────────────────────────────────────────────────────────────────┌ │ │ │ Request Flow: │ │ │ │ ┌──────────────────┌ │ │ │ Agent Request │ prompt + context │ │ └────────⁁─────────┘ │ │ │ │ │ ▾ │ │ ┌──────────────────┌ │ │ │ 1. RuVector │ GNN cache lookup │ │ │ Cache │ - Sub-ms latency (<1ms) │ │ │ (Docker) │ - 0.85 confidence threshold │ │ └────────⁁─────────┘ - If HIT → return cached response │ │ │ MISS │ │ ▾ │ │ ┌──────────────────┌ │ │ │ 2. RuvLLM │ Local inference │ │ │ (Local) │ - Zero cost │ │ │ │ - TRM/SONA enabled │ │ └────────⁁─────────┘ - Cache response for future │ │ │ UNAVAILABLE │ │ ▾ │ │ ┌──────────────────┌ │ │ │ 3. Cloud LLM │ Claude/OpenRouter fallback │ │ │ (Remote) │ - Higher quality │ │ │ │ - Cache response + trigger learning │ │ └──────────────────┘ │ │ │ │ Learning Features: │ │ └─ LoRA: Low-rank adaptation (rank: 8, alpha: 16) │ │ └─ EWC++: Elastic Weight Consolidation (anti-forgetting) │ │ └─ GNN: Graph Neural Network pattern matching │ │ │ │ Cost Savings: │ │ └─ Cache hits: ~$0.00 per request │ │ └─ Local LLM: ~$0.00 per request │ │ └─ Cloud only when necessary │ │ │ └─────────────────────────────────────────────────────────────────────┘

RuvLLM Features

┌─────────────────────────────────────────────────────────────────────┌ │ RUVLLM PROVIDER │ │ (src/providers/RuvllmProvider.ts) │ └─────────────────────────────────────────────────────────────────────┌ │ │ │ ┌──────────────────┌ │ │ │ TRM (Test-time │ Iterative reasoning refinement │ │ │ Reasoning & │ - Up to 7 iterations │ │ │ Metacognition) │ - Converges at 95% confidence │ │ └────────⁁─────────┘ - Improves output quality │ │ │ │ │ ┌────────▾─────────┌ │ │ │ SONA (Self- │ Continuous model adaptation │ │ │ Organizing │ - LoRA rank: 8, alpha: 16 │ │ │ Neural Arch) │ - EWC lambda: 2000 │ │ └────────⁁─────────┘ - Prevents catastrophic forgetting │ │ │ │ │ ┌────────▾─────────┌ │ │ │ Sessions │ Multi-turn context preservation │ │ │ │ - 50% latency reduction │ │ │ │ - Context reuse across calls │ │ └────────⁁─────────┘ - Session timeout: 30 min │ │ │ │ │ ┌────────▾─────────┌ │ │ │ Batch API │ Parallel request processing │ │ │ │ - 4x throughput │ │ │ │ - Rate limiting protection │ │ └──────────────────┘ │ │ │ │ Default Model: llama-3.2-3b-instruct │ │ Port: 8080 │ │ Context Size: 4096 │ │ │ └─────────────────────────────────────────────────────────────────────┘

When LLM is Called During Task Execution

1

Phase 1: Code Analysis

analyzeCodeWithConsciousness() - May use LLM for analysis

2

Phase 2: Pattern Retrieval NO LLM

findApplicablePatterns() - Uses HNSW vector search O(log n), <1ms p95

3

Phase 3: Pattern Recognition

recognizePatterns() - May use LLM

4

Phase 4: Test Strategy Selection NO LLM

selectTestStrategy() - Algorithmic, no LLM

5

Phase 5: Test Generation LLM CALLED

generateUnitTests(), generateIntegrationTests(), generateEdgeCaseTests() - LLM called via this.llmChat(prompt)

6

Phase 6: Test Optimization NO LLM

optimizeTestSelection() - Algorithmic

7

Phase 7: Pattern Extraction & Storage

Extracts patterns for future reuse via this.llmEmbed() for embedding generation

Key File: src/agents/TestGeneratorAgent.ts:390-666 (generateTestsWithAI)

4 Learning from Execution

src/learning/LearningEngine.ts:227-345

1. Extract Experience

TaskExperience { state: encode(complexity, capabilities, resources) action: encode(strategy, parallelization, retry) reward: calculated from results nextState: updated state timestamp: when executed }

Reward Calculation

ConditionReward
Success+1.0
Failure-1.0
Fast execution+0.2 bonus
Errors-0.3 penalty
Coverage (test gen)+0.5 bonus
User feedback-1 to +1
Final: clamp to [-2, +2]

2. Update Q-Table (Q-Learning Algorithm)

Q(s,a) ← Q(s,a) + α[r + γ·max(Q(s',a')) - Q(s,a)]

VariableMeaningValue
αLearning rate0.1
γDiscount factor0.95
sCurrent state-
aAction taken-
rReward received-
s'Next state-

Persisted to: SwarmMemoryManager → q_values table

3. Update Patterns (Dual Storage)

Pattern Update: └─ usageCount++ └─ successRate = (old * count + result) / (count + 1) └─ confidence = min(0.95, confidence + 0.01) ┌────────────────────────────────────────────────────┌ │ DUAL STORAGE │ └────────────────────────────────────────────────────┌ │ │ │ ┌─────────────────┌ ┌─────────────────────┌ │ │ │ SQLite │ │ HNSW (RuVector) │ │ │ │ (Primary) │ │ (Vector Search) │ │ │ └─────────────────┌ └─────────────────────┌ │ │ │ │ │ │ │ │ │ patterns table │ │ O(log n) search │ │ │ │ q_values table │ │ <1ms p95 latency │ │ │ │ experiences │ │ 768-dim embeddings │ │ │ │ │ │ │ │ │ │ .agentic-qe/ │ │ In-memory index │ │ │ │ memory.db │ │ │ │ │ │ │ │ │ │ │ └────────⁁────────┘ └──────────⁁──────────┘ │ │ │ │ │ │ └────────⁁─────────────┘ │ │ │ │ └────────────────────⁁──────────────────────────────┘ │ ▾ SHARED ACROSS ALL AGENTS (Single source of truth)

4. Exploration Decay

explorationRate *= 0.995

0.30 (start) → → → → → → → → 0.01 (min)
Over ~1000 tasks: EXPLORING → EXPLOITING

5. Federated Sharing (Optional)

5 Pattern Retrieval During Future Tasks

1

Extract Code Signature

codeSignature = { functionName: "createUser" complexity: "medium" patterns: ["service", "async", "validation"] }
2

Query HNSW (O(log n) Vector Search)

learningEngine.searchSimilarPatterns("createUser test") ↓ hnswAdapter.searchSimilar(query, k=5) ↓ 1. Generate query embedding (RuvLLM or hash fallback) 2. Search HNSW index (O(log n), <1ms) 3. Return top-k similar patterns Results: └─ Pattern: "service-async-test" (similarity: 0.92) └─ Pattern: "validation-test" (similarity: 0.87) └─ Pattern: "crud-service-test" (similarity: 0.81)
3

Apply Patterns (Reduce LLM Calls)

For each high-confidence pattern (applicability > 0.7):

  • Use pattern template for test structure
  • Customize with code-specific details
  • SKIP LLM call for template portions

Performance Comparison

WITHOUT PATTERNS: 10 tests → 10 LLM calls → ~5000ms
WITH PATTERNS: 7 from patterns + 3 LLM calls → ~2000ms
60% faster, 70% cost reduction

6 Complete Data Flow Architecture

┌─────────────────────────────────┌ │ USER REQUEST │ │ "Generate tests for X.ts" │ └─────────────────⁁───────────────┘ │ ▾ ┌─────────────────────────────────┌ │ CLAUDE CODE │ │ (Orchestrator / Interface) │ └─────────────────⁁───────────────┘ │ ┌─────────────────────&btee;───────────────────┌ │ │ ▾ ▾ ┌─────────────────────────┌ ┌─────────────────────────┌ │ MCP Tools │ │ Task Tool │ │ (aqe-mcp server) │ │ (direct spawn) │ └───────────⁁─────────────┘ └───────────⁁─────────────┘ │ │ ▾ │ ┌─────────────────────────┌ │ │ AgentRegistry │ │ │ └─ Shared Memory ─────⁁──────────────────────&btee; │ └─ Creates Config │ │ └───────────⁁─────────────┘ │ │ │ └───────────────────⁁───────────────────┘ │ ▾ ┌─────────────────────────────────┌ │ QE AGENT (BaseAgent) │ │ ┌───────────────────────────┌ │ │ │ TestGeneratorAgent │ │ │ │ CoverageAnalyzerAgent │ │ │ │ SecurityScannerAgent │ │ │ │ etc. │ │ │ └───────────────────────────┘ │ └─────────────────⁁───────────────┘ │ ┌──────────────────────────&btee;──────────────────────────┌ │ │ │ ▾ ▾ ▾ ┌─────────────────┌ ┌─────────────────┌ ┌─────────────────┌ │ LLM Provider │ │ Learning Engine │ │ Memory Store │ │ │ │ │ │ │ │ ┌─────────────┌ │ │ ┌─────────────┌ │ │ ┌─────────────┌ │ │ │ RuvLLM │ │←→│⇒│ Q-Table │ │←→│⇒│ SQLite DB │ │ │ │ (LOCAL) │ │ │ │ Algorithm │ │ │ │ patterns │ │ │ │ │ │ │ │ │ │ │ │ q_values │ │ │ │ ┌─────────┌ │ │ │ └─────────────┘ │ │ │ experiences │ │ │ │ │ TRM │ │ │ │ │ │ └─────────────┘ │ │ │ │ SONA │ │ │ │ ┌─────────────┌ │ │ │ │ │ │ Session │ │ │ │ │ HNSW │ │ │ .agentic-qe/ │ │ │ │ Batch │ │ │ │ │ Adapter │ │←→│⇒memory.db │ │ │ └─────────┘ │ │ │ │ (RuVector) │ │ │ │ │ └─────────────┘ │ │ └─────────────┘ │ │ SHARED ACROSS │ │ │ │ │ │ ALL AGENTS │ │ OR │ │ Federated │ │ │ │ │ │ Learning │ │ │ │ ┌─────────────┌ │ │ (cross-agent) │ │ │ │ │ Claude │ │ │ │ │ │ │ │ OpenRouter │ │ │ │ │ │ │ └─────────────┘ │ │ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │ │ │ │ │ └──────────────────────────&btee;──────────────────────────┘ │ ▾ ┌─────────────────────────────────┌ │ TASK RESULT │ │ TestSuite, Coverage Report, │ │ Quality Metrics, etc. │ └─────────────────────────────────┘

7 Ruv Solutions Integration Summary

Solution Component Purpose Location
RuvLLM RuvllmProvider Local LLM inference with TRM/SONA/Sessions src/providers/RuvllmProvider.ts
RuVector HNSWPatternStore O(log n) vector similarity search src/memory/HNSWPatternStore.ts
RuVector HNSWPatternAdapter Bridge to LearningEngine src/learning/HNSWPatternAdapter.ts
RuVector HybridRouter Cache-first routing src/providers/HybridRouter.ts
RuVector RuVectorClient GNN cache client with LoRA/EWC++ src/providers/RuVectorClient.ts
@ruvector/core VectorDB HNSW index implementation node_modules/@ruvector/core
AgentDB ReasoningBank Pattern storage and retrieval node_modules/agentdb

Agent RuVector Methods

All QE agents that extend BaseAgent have access to:

Metrics & Status Methods

MethodReturnDescription
hasRuVectorCache()booleanCheck if HybridRouter with RuVector cache is enabled
getRuVectorMetrics()Promise<object | null>Get GNN/LoRA/cache metrics from RuVector
getCacheHitRate()numberGet cache hit rate (0-1)
getRoutingStats()objectGet routing decisions & latencies
getCostSavingsReport()objectGet cost savings from cache and local routing
getLLMStats()objectGet LLM provider status including RuVector

Learning Methods

MethodReturnDescription
forceRuVectorLearn()Promise<{success, ...}>Trigger LoRA consolidation with EWC++

MCP Tools for RuVector

Tool NameDescription
ruvector_healthCheck RuVector Docker health
ruvector_metricsGet GNN/LoRA/cache metrics
ruvector_force_learnTrigger LoRA consolidation
ruvector_store_patternStore pattern in GNN cache
ruvector_searchSearch similar patterns
ruvector_cost_savingsGet cost savings report

Usage Example

// Configure agent with HybridRouter const agent = new TestGeneratorAgent({ type: 'test-generator', capabilities: [...], context: {...}, memoryStore, eventBus, llm: { enabled: true, enableHybridRouter: true, // Enable Phase 0.5 features ruvectorCache: { baseUrl: 'http://localhost:8080', cacheThreshold: 0.85, learningEnabled: true, loraRank: 8, ewcEnabled: true, }, }, }); // Check cache status console.log('RuVector enabled:', agent.hasRuVectorCache()); console.log('Cache hit rate:', agent.getCacheHitRate()); // Get cost savings const savings = agent.getCostSavingsReport(); console.log(`Saved: $${savings.savings} (${savings.savingsPercentage}%)`); // Trigger learning const result = await agent.forceRuVectorLearn(); console.log('Learning result:', result);

8 Benefits of This Architecture

$ Reduced LLM Costs

  • Pattern templates reduce LLM calls by up to 70%
  • Local RuvLLM provides zero-cost inference
  • Session reuse cuts latency by 50%
  • HybridRouter cache-first routing further reduces cloud calls

↑ Continuous Improvement

  • Q-learning optimizes strategy selection
  • Patterns improve with each successful task
  • Federated learning shares knowledge across agents
  • LoRA + EWC++ enables continuous self-learning

⚡ Fast Pattern Retrieval

  • HNSW provides O(log n) vector search
  • <1ms p95 latency for pattern matching
  • Scales to millions of patterns
  • Phase 0.5: GNN cache provides sub-ms pattern matching

🗄 Unified Data Store

  • Single SQLite database prevents fragmentation
  • All agents share learned patterns
  • CLI, MCP, and direct spawning all contribute
  • RuVector Docker provides distributed cache layer

↓ Graceful Degradation

  • If LLM unavailable → algorithmic fallback
  • If HNSW fails → SQLite-only mode
  • If patterns unavailable → full LLM generation
  • If RuVector unavailable → fallback to RuvLLM → Cloud