mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Profa
4b0ec635f5
Add anti-sycophancy scoring, test quality gates, blind review, EMA calibration, edge-case injection, complexity-driven composition, and auto-escalation. All features enabled by default (opt-out). 178 tests, 14 new files, 6 config flags. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2902 lines
106 KiB
Markdown
2902 lines
106 KiB
Markdown
# Agentic QE v3 - Architecture Decision Records
|
|
|
|
**Project:** Agentic QE v3 Reimagining
|
|
**Date Range:** 2026-01-07 onwards
|
|
**Status:** Phase 16 — Loki-Mode Adversarial Quality Gates (ADR-074); Multi-Language Test Generation (ADR-075 through ADR-079)
|
|
**Decision Authority:** Architecture Team
|
|
**Last Verified:** 2026-03-04 (ADR-001-064: 58 Implemented, ADR-065-073: 9 Proposed, ADR-074: Implemented)
|
|
|
|
---
|
|
|
|
## ADR Index
|
|
|
|
| ADR | Title | Status | Date | Implementation |
|
|
|-----|-------|--------|------|----------------|
|
|
| ADR-001 | Adopt DDD for QE Bounded Contexts | **Implemented** | 2026-01-07 | ✅ 12/12 domains |
|
|
| ADR-002 | Event-Driven Domain Communication | **Implemented** | 2026-01-07 | ✅ EventBus + Router + 7 protocols |
|
|
| ADR-003 | Sublinear Algorithms for Coverage Analysis | **Implemented** | 2026-01-09 | ✅ REAL hnswlib-node + lcov-parse (verified) |
|
|
| ADR-004 | Plugin Architecture for QE Extensions | **Implemented** | 2026-01-07 | ✅ 12 plugins |
|
|
| ADR-005 | AI-First Test Generation | **Implemented** | 2026-01-09 | ✅ Full implementation (stubs replaced) |
|
|
| ADR-006 | Unified Learning System | **Implemented** | 2026-01-09 | ✅ Learning domain complete |
|
|
| ADR-007 | Quality Gate Decision Engine | **Implemented** | 2026-01-09 | ✅ ML-based gate engine |
|
|
| ADR-008 | Multi-Agent Hierarchical Coordination | **Implemented** | 2026-01-07 | ✅ Queen + protocols |
|
|
| ADR-009 | AgentDB as Primary Memory Backend | **Implemented** | 2026-01-07 | ✅ Backend with vector search |
|
|
| ADR-010 | MCP-First Tool Design | **Implemented** | 2026-01-09 | ✅ 14 domain tools + CLI wrappers |
|
|
| ADR-011 | LLM Provider System for QE | **Implemented** | 2026-01-09 | ✅ Claude/OpenAI/Ollama + circuit breaker |
|
|
| ADR-012 | MCP Security Features for QE | **Implemented** | 2026-01-09 | ✅ OAuth2.1 + rate limiter + CVE prevention |
|
|
| ADR-013 | Core Security Module for QE | **Implemented** | 2026-01-07 | ✅ OSV client + compliance patterns |
|
|
| ADR-014 | Background Workers for QE Monitoring | **Implemented** | 2026-01-09 | ✅ 10 workers + daemon |
|
|
| ADR-015 | Unified Plugin System for QE Extensions | **Implemented** | 2026-01-07 | ✅ Plugin loader + 12 domain plugins |
|
|
| ADR-016 | Collaborative Test Task Claims | **Implemented** | 2026-01-09 | ✅ ClaimService + WorkStealing + Handoff |
|
|
| ADR-017 | RuVector Integration for QE Intelligence | **Implemented** | 2026-01-09 | ✅ Q-Learning + AST + fallbacks |
|
|
| ADR-018 | Expanded 12-Domain Architecture | **Implemented** | 2026-01-07 | ✅ All 12 domains |
|
|
| ADR-019 | Phase 1-3 Foundation Implementation | **Implemented** | 2026-01-07 | ✅ 1954 tests passing |
|
|
| ADR-020 | Stub Implementation Replacement | **Implemented** | 2026-01-11 | ✅ Domain services complete, 9 orchestration stubs (acceptable) |
|
|
| ADR-021 | QE ReasoningBank for Pattern Learning | **Implemented** | 2026-01-09 | ✅ REAL impl: transformers + SQLite (114k/s) + 52 tests |
|
|
| ADR-022 | Adaptive QE Agent Routing | **Implemented** | 2026-01-09 | ✅ ML router: 80 agents + 62ms P95 + 83 tasks/sec |
|
|
| ADR-023 | Quality Feedback Loop System | **Implemented** | 2026-01-09 | ✅ 101 tests: TestOutcomeTracker + CoverageLearner + PatternPromotion |
|
|
| ADR-024 | Self-Optimization Engine | **Implemented** | 2026-01-09 | ✅ 103 tests: AutoTuner + MetricCollectors + 4 QE Workers + Applicators |
|
|
| ADR-025 | Enhanced Init with Self-Configuration | **Implemented** | 2026-01-10 | ✅ 73 tests: ProjectAnalyzer + SelfConfigurator + InitWizard |
|
|
| ADR-026 | AISP Parsing and Validation | **Rejected** | 2026-01-10 | ❌ Deleted - no integration points, duplicated existing EventBus |
|
|
| ADR-027 | AISP Agent Specifications for QE Fleet | **Rejected** | 2026-01-10 | ❌ Deleted - TypeScript interfaces sufficient |
|
|
| ADR-028 | AISP↔Prose Translator (Rosetta Stone) | **Rejected** | 2026-01-10 | ❌ Deleted - no consumers of translation |
|
|
| ADR-029 | AISP Agent Communication Protocol | **Rejected** | 2026-01-10 | ❌ Deleted - duplicated existing coordination layer |
|
|
| ADR-030 | Coherence-Gated Quality Gates | **Implemented** | 2026-01-11 | ✅ 68 tests, 2,345 LOC: λ-coherence + 4-tier compute allocation |
|
|
| ADR-031 | Strange Loop Self-Awareness | **Implemented** | 2026-01-11 | ✅ 61 tests, 4,700 LOC: SwarmObserver + SelfModel + HealingController |
|
|
| ADR-032 | Time Crystal Scheduling | **Implemented** | 2026-01-11 | ✅ 165 tests, 1,870 LOC: Kuramoto CPG + 4-phase test cycle |
|
|
| ADR-033 | Early Exit Testing | **Implemented** | 2026-01-11 | ✅ 126 tests, 2,800 LOC: λ-stability + speculative execution |
|
|
| ADR-034 | Neural Topology Optimizer | **Implemented** | 2026-01-11 | ✅ 143 tests, 1,820 LOC: Q-learning + value network + replay buffer |
|
|
| ADR-035 | Causal Discovery | **Implemented** | 2026-01-11 | ✅ 103 tests, 1,430 LOC: STDP + Floyd-Warshall + Tarjan SCC |
|
|
| ADR-036 | Language-Aware Result Persistence | **Implemented** | 2026-01-11 | ✅ 74 tests, 780 LOC: ResultSaver + 11 languages + SARIF/LCOV |
|
|
| ADR-037 | V3 QE Agent Naming Standardization | **Implemented** | 2026-01-14 | ✅ v3-qe-* prefix for 47 agents in completions |
|
|
| ADR-038 | V3 QE Memory System Unification | **Implemented** | 2026-01-14 | ✅ AgentDB + HNSW + hybrid backend |
|
|
| ADR-039 | V3 QE MCP Optimization | **Implemented** | 2026-01-14 | ✅ Simplified MCP command (aqe-v3-mcp) |
|
|
| ADR-040 | V3 QE Agentic-Flow Integration | **Implemented** | 2026-01-14 | ✅ WorkflowOrchestrator + YAML pipelines |
|
|
| ADR-041 | V3 QE CLI Enhancement | **Implemented** | 2026-01-14 | ✅ 196 tests: wizards + progress + streaming + completions + workflows |
|
|
| ADR-042 | V3 QE Token Tracking Integration | **Implemented** | 2026-01-14 | ✅ TokenMetricsCollector + TokenOptimizerService + CLI commands + persistence |
|
|
| [ADR-043](./ADR-043-vendor-independent-llm.md) | Vendor-Independent LLM Support | **Implemented** | 2026-01-15 | ✅ HybridRouter + 7 providers + model-mapping + MSW integration tests |
|
|
| [ADR-044](./ADR-044-domain-rl-integration-status.md) | Domain RL Integration Status | **Implemented** | 2026-01-14 | ✅ 6/6 domains with RL integration |
|
|
| [ADR-045](./ADR-045-version-agnostic-naming.md) | Version-Agnostic Naming | **Implemented** | 2026-01-14 | ✅ qe-* naming convention, backward compat aliases |
|
|
| [ADR-046](./ADR-046-v2-feature-integration.md) | V2 Feature Integration (Q-Values, GOAP, Dreams) | **Implemented** | 2026-01-16 | ✅ Q-Values persistence + GOAP planner (52 actions) + Dream cycles (ConceptGraph + InsightGenerator) |
|
|
| [ADR-047](./ADR-047-mincut-self-organizing-qe.md) | MinCut Self-Organizing QE Integration | **Implemented** | 2026-01-17 | ✅ 478 tests, 14 modules: MinCut health + Strange Loop (P1) + Causal Discovery (P2) + Morphogenetic (P3) + Time Crystal (P4) + Neural GOAP (P5) + Dream Integration (P6) + 3 MCP tools |
|
|
| [ADR-048](./ADR-048-v2-v3-agent-migration.md) | V2-to-V3 Agent Migration Upgrade Strategy | **Implemented** | 2026-01-17 | ✅ 5-tier migration: v2_compat fields (6 agents) + agent-compat.ts + 59 tests + v3_new markers (7 agents) |
|
|
| [ADR-049](./ADR-049-V3-MAIN-PUBLISH.md) | V3 Main Package Publication | **Accepted** | 2026-01-17 | ✅ Root package publishes v3 CLI + MCP bundles, version 3.0.0 release strategy |
|
|
| [ADR-050](./ADR-050-ruvector-neural-backbone.md) | RuVector as Primary Neural Backbone | **Implemented** | 2026-01-20 | ✅ ML-first architecture, Q-Learning/SONA persistence, hypergraph code intelligence |
|
|
| [ADR-051](./ADR-051-agentic-flow-integration.md) | Agentic-Flow Deep Integration | **Implemented** | 2026-01-21 | ✅ 100% success rate: Agent Booster, ReasoningBank (HNSW), Model Router, ONNX Embeddings |
|
|
| [ADR-052](./ADR-052-coherence-gated-qe.md) | Coherence-Gated Quality Engineering | **Implemented** | 2026-01-24 | ✅ 382+ tests: CoherenceService + 6 engines + ThresholdTuner + WASM Fallback + CI Badge + Test Gen Gate |
|
|
| [ADR-053](./ADR-053-ag-ui-protocol.md) | AG-UI Protocol Adoption | **Proposed** | 2026-01-30 | SSE transport, 19 event types, 100ms p95 latency target |
|
|
| [ADR-054](./ADR-054-a2a-protocol.md) | A2A Protocol Integration | **Proposed** | 2026-01-30 | Agent Cards for 68 agents, JSON-RPC 2.0, discovery endpoint |
|
|
| [ADR-055](./ADR-055-a2ui-declarative-ui.md) | A2UI Declarative UI Strategy | **Proposed** | 2026-01-30 | 15+ components, QE catalog, WCAG 2.2, AG-UI state sync |
|
|
| [ADR-056](./ADR-056-skill-validation-system.md) | Deterministic Skill Validation System | **Implemented** | 2026-02-02 | ✅ 46 Tier 3 skills, 52 validators, CLI commands (`aqe skill`, `aqe eval`) |
|
|
| [ADR-057](./ADR-057-infra-self-healing.md) | Infrastructure Self-Healing Extension | **Implemented** | 2026-02-02 | ✅ Extends Strange Loop to detect and recover infrastructure failures |
|
|
| [ADR-058](./ADR-058-guidance-governance-integration.md) | @claude-flow/guidance Governance Integration | **Implemented** | 2026-02-04 | ✅ 16 governance modules + GovernanceAwareDomainMixin wired to 12 domain coordinators + Queen/ReasoningBank/MCP integration + 657 tests |
|
|
| [ADR-059](./ADR-059-ghost-intent-coverage.md) | Ghost Intent Coverage Analysis | **Proposed** | 2026-02-06 | Phantom test surface detection via HNSW ghost vectors (inspired by AISP ψ_g) |
|
|
| [ADR-060](./ADR-060-semantic-anti-drift.md) | Semantic Anti-Drift Protocol | **Proposed** | 2026-02-06 | HNSW semantic fingerprinting on domain events for drift detection (inspired by AISP anti-drift) |
|
|
| [ADR-061](./ADR-061-asymmetric-learning-rates.md) | Asymmetric Learning Rates for ReasoningBank | **Proposed** | 2026-02-06 | 10:1 Hebbian failure penalty + pattern quarantine (inspired by AISP Hebbian learning) |
|
|
| [ADR-062](./ADR-062-strongdm-software-factory.md) | StrongDM Software Factory Integration | **Accepted** | 2026-02-08 | 6 actions across 3 tiers: loop detection, token dashboard, holdout testing, gate ratcheting, progressive context, meta-learning |
|
|
| [ADR-063](./ADR-063-enterprise-integration-testing.md) | Enterprise Integration Testing Gap Closure | **Implemented** | 2026-02-04 | ✅ 7 new agents + 4 new skills + enterprise-integration bounded context (13th domain) + QCSD flag extensions |
|
|
| [ADR-064](./ADR-064-agentic-teams-integration.md) | Agent Teams Integration for AQE Fleet | **Implemented** | 2026-02-09 | ✅ All 4 phases: mailbox messaging, fleet tiers, task DAG, circuit breakers, tracing, competing hypotheses, federation, dynamic scaling. 397+ tests. |
|
|
| [ADR-065](./ADR-065-rvf-integration-hybrid-architecture.md) | RVF Integration — Hybrid Architecture | **Proposed** | 2026-02-10 | Umbrella ADR: SQLite+RVF coexistence, vector workload migration path |
|
|
| [ADR-066](./ADR-066-rvf-pattern-store-progressive-hnsw.md) | RVF-Backed PatternStore with Progressive HNSW | **Proposed** | 2026-02-10 | RVF INDEX_SEG for vector workloads, 3-layer progressive HNSW |
|
|
| [ADR-067](./ADR-067-agent-memory-branching-rvf-cow.md) | Agent Memory Branching via RVF COW | **Proposed** | 2026-02-10 | RVCOW copy-on-write branching for agent memory isolation |
|
|
| [ADR-068](./ADR-068-mincut-gated-model-routing.md) | MinCut-Gated Model Routing | **Proposed** | 2026-02-22 | Replace static 3-tier thresholds with mincut lambda-based structural complexity routing |
|
|
| [ADR-069](./ADR-069-rvcow-dream-cycle-branching.md) | RVCOW Dream Cycle Branching | **Proposed** | 2026-02-22 | Safe dream experimentation via RVCOW branches with validation gates |
|
|
| [ADR-070](./ADR-070-witness-chain-audit-compliance.md) | Witness Chain Audit Compliance | **Proposed** | 2026-02-22 | Cryptographic SHAKE-256 witness chain for tamper-evident QE decision audit |
|
|
| [ADR-071](./ADR-071-hnsw-implementation-unification.md) | HNSW Implementation Unification | **Proposed** | 2026-02-22 | Consolidate 3 HNSW implementations behind single HnswIndexProvider + RVF INDEX_SEG |
|
|
| [ADR-072](./ADR-072-rvf-primary-persistence-migration.md) | RVF Primary Persistence Migration | **Proposed** | 2026-02-22 | 4-stage migration from SQLite hybrid to RVF primary with go/no-go gates |
|
|
| [ADR-073](./ADR-073-portable-intelligence-containers.md) | Portable Intelligence Containers | **Proposed** | 2026-02-22 | Export/import/version QE intelligence as distributable .rvf files |
|
|
| [ADR-074](./ADR-074-loki-mode-adversarial-quality-gates.md) | Loki-Mode Adversarial Quality Gates | **Implemented** | 2026-03-04 | ✅ 7 features: sycophancy scorer, test quality gates, blind review, EMA calibration, edge-case injection, complexity composition, auto-escalation. 178 tests, 14 new files, 6 config flags (enabled by default). |
|
|
| [ADR-075](./ADR-075-unified-test-framework-type-system.md) | Unified TestFramework Type System | **Proposed** | 2026-03-04 | Single source-of-truth `TestFramework` + `SupportedLanguage` types at `src/shared/types/test-frameworks.ts`, replacing two divergent definitions |
|
|
| [ADR-076](./ADR-076-tree-sitter-wasm-multi-language-parser.md) | tree-sitter WASM Multi-Language Parser Integration | **Proposed** | 2026-03-04 | `web-tree-sitter` (WASM) as universal parser for Java, C#, Go, Rust, Swift, Kotlin, Dart, Python behind `ILanguageParser` abstraction |
|
|
| [ADR-077](./ADR-077-compilation-validation-loop.md) | Compilation Validation Loop for Generated Tests | **Proposed** | 2026-03-04 | Optional compile-validate-repair loop (up to 3 retries) for type-strict languages; opt-in, graceful degradation |
|
|
| [ADR-078](./ADR-078-backward-compatible-multi-language-api.md) | Backward-Compatible Multi-Language API Extension | **Proposed** | 2026-03-04 | Optional `language`, `projectRoot`, `compileValidation` fields on `IGenerateTestsRequest`; auto-detection from file extensions |
|
|
| [ADR-079](./ADR-079-language-specific-test-file-path-resolution.md) | Language-Specific Test File Path Resolution | **Proposed** | 2026-03-04 | `TestFileResolver` with per-language `TestFileConvention` for correct test output paths (Maven mirror, `*_test.go`, Rust inline, etc.) |
|
|
|
|
---
|
|
|
|
## ADR-001: Adopt DDD for QE Bounded Contexts
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Decision Makers:** Architecture Team
|
|
**Context Owner:** Lead Architect
|
|
|
|
### Context
|
|
|
|
Current Agentic QE v2.x implements quality engineering as monolithic services in `src/mcp/tools/` and `src/core/agents/`. This creates:
|
|
- Tight coupling between test generation and execution
|
|
- Difficulty in scaling individual QE capabilities
|
|
- Complex testing due to interdependencies
|
|
- Limited ability to evolve domains independently
|
|
|
|
**Current State:**
|
|
- All MCP tools in single directory (40+ tools)
|
|
- Agents mixed with different responsibilities
|
|
- Memory implementations scattered
|
|
- No clear domain boundaries
|
|
|
|
**Analysis:**
|
|
```
|
|
Current Structure:
|
|
├── src/mcp/tools/ # 40+ MCP tool implementations
|
|
├── src/core/agents/ # Mixed agent responsibilities
|
|
├── src/core/memory/ # Multiple memory implementations
|
|
└── High coupling, difficult to test
|
|
|
|
v3 Target Structure:
|
|
├── src/domains/
|
|
│ ├── test-generation/ # Bounded Context 1
|
|
│ ├── test-execution/ # Bounded Context 2
|
|
│ ├── coverage-analysis/ # Bounded Context 3
|
|
│ ├── quality-assessment/ # Bounded Context 4
|
|
│ ├── defect-intelligence/ # Bounded Context 5
|
|
│ └── learning-optimization/ # Bounded Context 6
|
|
└── Clear boundaries, independent evolution
|
|
```
|
|
|
|
### Decision
|
|
|
|
**We will restructure Agentic QE v3 using Domain-Driven Design with 12 bounded contexts (expanded per ADR-018) focused on quality engineering capabilities.**
|
|
|
|
> **Note:** Original proposal was 6 domains. Per ADR-018, expanded to 12 domains to ensure feature parity with v2.
|
|
|
|
Bounded Contexts:
|
|
1. **Test Generation** - AI-powered test creation, pattern learning
|
|
2. **Test Execution** - Parallel execution, retry, flaky detection
|
|
3. **Coverage Analysis** - Sublinear gap detection, risk scoring
|
|
4. **Quality Assessment** - Quality gates, deployment decisions
|
|
5. **Defect Intelligence** - Prediction, root cause, pattern learning
|
|
6. **Requirements Validation** - BDD, testability scoring (per ADR-018)
|
|
7. **Code Intelligence** - Knowledge Graph, semantic search (per ADR-018)
|
|
8. **Security Compliance** - SAST/DAST, compliance (per ADR-018)
|
|
9. **Contract Testing** - API contracts, GraphQL (per ADR-018)
|
|
10. **Visual Accessibility** - Visual regression, a11y (per ADR-018)
|
|
11. **Chaos Resilience** - Chaos engineering, load testing (per ADR-018)
|
|
12. **Learning Optimization** - Cross-domain learning, transfer
|
|
|
|
### Rationale
|
|
|
|
**Pros:**
|
|
- Independent evolution of QE capabilities
|
|
- Clearer testing boundaries
|
|
- Team specialization possible
|
|
- Better scalability per domain
|
|
- Aligned with QE workflow stages
|
|
|
|
**Cons:**
|
|
- Migration effort from v2
|
|
- Cross-domain coordination complexity
|
|
- Learning curve for DDD concepts
|
|
|
|
**Alternatives Considered:**
|
|
|
|
1. **Status Quo (Keep Flat Structure)**
|
|
- Rejected: Scaling issues, tight coupling
|
|
|
|
2. **Layer-Based Architecture**
|
|
- Rejected: Doesn't align with QE workflow stages
|
|
|
|
3. **Feature-Based Modules**
|
|
- Rejected: Less clear boundaries than DDD
|
|
|
|
### Implementation Plan
|
|
|
|
**Phase 1: Foundation (Week 1-2)**
|
|
- Define domain interfaces
|
|
- Create shared kernel
|
|
- Set up event bus
|
|
|
|
**Phase 2: Core Domains (Week 3-6)**
|
|
- Extract test-generation domain
|
|
- Extract test-execution domain
|
|
- Extract coverage-analysis domain
|
|
|
|
**Phase 3: Supporting Domains (Week 7-10)**
|
|
- Extract quality-assessment domain
|
|
- Extract defect-intelligence domain
|
|
- Extract learning-optimization domain
|
|
|
|
### Success Metrics
|
|
|
|
- [x] 12 clearly defined bounded contexts (expanded per ADR-018)
|
|
- [x] No circular dependencies between domains (verified 2026-01-09)
|
|
- [x] Each domain testable in isolation (1171 tests passing)
|
|
- [x] Domain events for cross-domain communication
|
|
- [ ] <300 lines per domain service (FAILED: some services exceed 1500 lines)
|
|
|
|
> **Implementation Note (2026-01-09):** All 12 domains implemented with coordinators, services, and plugins. However, some service files exceed the 300-line target (e.g., `security-auditor.ts` at 1715 lines, `test-generator.ts` at 1881 lines). Consider splitting in Phase 4.
|
|
|
|
---
|
|
|
|
## ADR-002: Event-Driven Domain Communication
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
With DDD bounded contexts, we need a communication mechanism that maintains loose coupling while enabling reactive workflows (e.g., automatically generate tests when coverage gaps detected).
|
|
|
|
### Decision
|
|
|
|
**Use domain events for all cross-domain communication, enabling reactive QE workflows.**
|
|
|
|
Key Events:
|
|
- `TestCaseGeneratedEvent` - Triggers coverage analysis
|
|
- `CoverageGapDetectedEvent` - Triggers test generation
|
|
- `TestRunCompletedEvent` - Triggers quality gate evaluation
|
|
- `QualityGateEvaluatedEvent` - Triggers deployment decisions
|
|
- `DefectPredictedEvent` - Triggers targeted testing
|
|
|
|
### Implementation
|
|
|
|
```typescript
|
|
// Event Bus Interface
|
|
interface IDomainEventBus {
|
|
publish(event: DomainEvent): Promise<void>;
|
|
subscribe<T extends DomainEvent>(
|
|
eventType: new (...args: any[]) => T,
|
|
handler: (event: T) => Promise<void>
|
|
): void;
|
|
}
|
|
|
|
// Event Handler Example
|
|
@EventHandler(CoverageGapDetectedEvent)
|
|
class AutoTestGenerationHandler {
|
|
async handle(event: CoverageGapDetectedEvent): Promise<void> {
|
|
if (event.riskScore > 0.7) {
|
|
await this.testGenerator.generateForGap(event);
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] All cross-domain communication via events (CrossDomainEventRouter implemented)
|
|
- [x] Event handlers are idempotent (event correlation IDs)
|
|
- [ ] Event replay for debugging (not yet implemented)
|
|
- [x] <100ms event propagation latency (in-memory bus)
|
|
|
|
> **Implementation Note (2026-01-09):** CrossDomainEventRouter with 7 coordination protocols (morning-sync, quality-gate, regression-prevention, coverage-driven, tdd-cycle, security-audit, learning-consolidation). Event replay deferred to Phase 4.
|
|
|
|
---
|
|
|
|
## ADR-003: Sublinear Algorithms for Coverage Analysis
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Implemented:** 2026-01-09
|
|
|
|
### Context
|
|
|
|
Coverage analysis in large codebases (100k+ files) using linear O(n) algorithms is too slow for real-time feedback. We need O(log n) algorithms for interactive analysis.
|
|
|
|
### Decision
|
|
|
|
**Implement O(log n) coverage analysis using HNSW vector indexing via AgentDB.**
|
|
|
|
### Implementation
|
|
|
|
**Files Created:**
|
|
- `src/domains/coverage-analysis/services/hnsw-index.ts` - HNSW wrapper for O(log n) search
|
|
- `src/domains/coverage-analysis/services/coverage-embedder.ts` - Coverage to embedding converter
|
|
- `src/domains/coverage-analysis/services/sublinear-analyzer.ts` - Main sublinear analyzer
|
|
|
|
**Tests (49 passing):**
|
|
- `tests/unit/domains/coverage-analysis/hnsw-index.test.ts`
|
|
- `tests/unit/domains/coverage-analysis/coverage-embedder.test.ts`
|
|
- `tests/unit/domains/coverage-analysis/sublinear-analyzer.test.ts`
|
|
|
|
```typescript
|
|
// Usage Example
|
|
import { createSublinearAnalyzer, createHNSWIndex } from '@agentic-qe/v3';
|
|
|
|
const analyzer = createSublinearAnalyzer(memoryBackend);
|
|
await analyzer.initialize();
|
|
|
|
// Index coverage data - O(n log n)
|
|
await analyzer.indexCoverageData(coverageData);
|
|
|
|
// Find gaps - O(log n)
|
|
const gaps = await analyzer.findGapsSublinear({
|
|
maxLineCoverage: 60,
|
|
minRiskScore: 0.5
|
|
});
|
|
|
|
// Find similar patterns - O(log n)
|
|
const similar = await analyzer.findSimilarPatterns(gap, 5);
|
|
|
|
// Detect risk zones - O(log n)
|
|
const zones = await analyzer.detectRiskZones(0.7);
|
|
```
|
|
|
|
### Performance Targets
|
|
|
|
| Codebase Size | Traditional O(n) | v3 O(log n) | Improvement |
|
|
|---------------|-----------------|-------------|-------------|
|
|
| 1,000 files | 1,000 ops | 10 ops | 100x |
|
|
| 10,000 files | 10,000 ops | 13 ops | 770x |
|
|
| 100,000 files | 100,000 ops | 17 ops | 5,900x |
|
|
|
|
### Success Metrics
|
|
|
|
- [x] <100ms gap detection on 500 files (verified in tests)
|
|
- [x] HNSW index wrapper with configurable dimensions
|
|
- [x] Sublinear scaling verified in benchmark tests
|
|
- [x] CoverageEmbedder with 128-dimension vectors
|
|
- [ ] Real-time coverage updates (deferred to Phase 4)
|
|
- [x] Integration with actual HNSW native library (hnswlib-node)
|
|
|
|
> **Implementation Note (2026-01-09 - UPDATED):**
|
|
>
|
|
> **REAL IMPLEMENTATIONS ADDED:**
|
|
> - `hnsw-index.ts` - Uses real `hnswlib-node` native library with graceful fallback to brute-force when unavailable
|
|
> - `coverage-parser.ts` - Uses real `lcov-parse` to parse actual LCOV/JSON coverage files
|
|
> - `performance-benchmarks.ts` - Actual O(log n) verification benchmarks
|
|
>
|
|
> **Native HNSW usage:**
|
|
> ```typescript
|
|
> import hnswlib from 'hnswlib-node';
|
|
> const index = new HierarchicalNSW('cosine', dimensions);
|
|
> index.initIndex(maxElements, M, efConstruction);
|
|
> const result = index.searchKnn(query, k); // O(log n)
|
|
> ```
|
|
>
|
|
> **To verify O(log n) claims, run benchmarks:**
|
|
> ```bash
|
|
> npx tsx src/benchmarks/run-benchmarks.ts
|
|
> ```
|
|
|
|
---
|
|
|
|
## ADR-004: Plugin Architecture for QE Extensions
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Agentic QE has specialized testing capabilities (n8n workflows, visual regression, performance) that not all users need. Core should be lean, with extensions as plugins.
|
|
|
|
### Decision
|
|
|
|
**Implement microkernel architecture with plugin system for optional QE capabilities.**
|
|
|
|
**Core (Always Loaded):**
|
|
- Test generation (basic)
|
|
- Test execution
|
|
- Coverage analysis
|
|
- Quality gates
|
|
|
|
**Plugins (Optional):**
|
|
- n8n Workflow Testing Plugin
|
|
- Visual Regression Plugin
|
|
- Performance Testing Plugin
|
|
- Security Scanning Plugin
|
|
- Accessibility Testing Plugin
|
|
- API Contract Testing Plugin
|
|
|
|
### Plugin Interface
|
|
|
|
```typescript
|
|
interface QEPlugin {
|
|
name: string;
|
|
version: string;
|
|
dependencies: string[];
|
|
|
|
initialize(kernel: QEKernel): Promise<void>;
|
|
shutdown(): Promise<void>;
|
|
|
|
registerAgents?(): AgentDefinition[];
|
|
registerMCPTools?(): MCPTool[];
|
|
registerDomainServices?(): ServiceDefinition[];
|
|
}
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] Core <50MB (vs 100MB+ with all features)
|
|
- [ ] Plugin loading <200ms
|
|
- [ ] 6+ official plugins
|
|
- [ ] Plugin development guide
|
|
|
|
---
|
|
|
|
## ADR-005: AI-First Test Generation
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Traditional test generation relies on templates and heuristics. AI models (Claude, GPT) can generate higher-quality, context-aware tests.
|
|
|
|
### Decision
|
|
|
|
**Make AI-powered test generation the primary method, with template-based as fallback.**
|
|
|
|
### Implementation
|
|
|
|
```typescript
|
|
class AITestGenerationService {
|
|
async generate(request: GenerateTestsRequest): Promise<TestCase[]> {
|
|
// 1. Analyze source code
|
|
const analysis = await this.analyzeSource(request.sourceFile);
|
|
|
|
// 2. Retrieve learned patterns
|
|
const patterns = await this.patternRepo.findByContext(request);
|
|
|
|
// 3. Identify coverage gaps
|
|
const gaps = await this.coverageAnalyzer.findGaps(request.sourceFile);
|
|
|
|
// 4. Generate via AI
|
|
const tests = await this.aiClient.generate(
|
|
this.buildPrompt(analysis, patterns, gaps)
|
|
);
|
|
|
|
// 5. Store successful patterns
|
|
await this.storePatterns(tests);
|
|
|
|
return tests;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] >80% of generated tests are valid
|
|
- [ ] AI tests improve coverage by 20%+
|
|
- [ ] <30 seconds per test suite generation
|
|
- [ ] Pattern learning improves quality over time
|
|
|
|
---
|
|
|
|
## ADR-006: Unified Learning System
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
QE agents learn independently without sharing knowledge. Cross-agent and cross-project learning could significantly improve quality.
|
|
|
|
### Decision
|
|
|
|
**Implement unified learning system with pattern storage, cross-agent sharing, and transfer learning.**
|
|
|
|
### Components
|
|
|
|
1. **Pattern Storage** - Successful patterns indexed in AgentDB
|
|
2. **Cross-Agent Sharing** - Patterns shared via learning coordinator
|
|
3. **Transfer Learning** - Patterns transferred across projects
|
|
4. **Continuous Improvement** - Feedback loop for pattern refinement
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] 1000+ learned patterns per project
|
|
- [ ] 15% improvement per sprint cycle
|
|
- [ ] Cross-project transfer working
|
|
- [ ] Pattern quality scoring
|
|
|
|
---
|
|
|
|
## ADR-007: Quality Gate Decision Engine
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Quality gates are currently simple threshold checks. Need intelligent decision engine that considers trends, risk, and context.
|
|
|
|
### Decision
|
|
|
|
**Implement intelligent quality gate engine with ML-based risk assessment.**
|
|
|
|
### Gate Evaluation
|
|
|
|
```typescript
|
|
class QualityGateEngine {
|
|
async evaluate(context: GateContext): Promise<GateDecision> {
|
|
const [
|
|
coverage,
|
|
testResults,
|
|
riskScore,
|
|
trends,
|
|
defectPrediction
|
|
] = await Promise.all([
|
|
this.getCoverage(context),
|
|
this.getTestResults(context),
|
|
this.calculateRisk(context),
|
|
this.analyzeTrends(context),
|
|
this.predictDefects(context)
|
|
]);
|
|
|
|
// ML-based decision
|
|
const decision = await this.decisionModel.predict({
|
|
coverage,
|
|
testResults,
|
|
riskScore,
|
|
trends,
|
|
defectPrediction
|
|
});
|
|
|
|
return decision;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] >95% accuracy in gate decisions
|
|
- [ ] <5% false positives (blocking good releases)
|
|
- [ ] <1% false negatives (allowing bad releases)
|
|
- [ ] Trend analysis integrated
|
|
|
|
---
|
|
|
|
## ADR-008: Multi-Agent Hierarchical Coordination
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Current agents operate independently. Need hierarchical coordination for complex QE workflows.
|
|
|
|
### Decision
|
|
|
|
**Implement hierarchical multi-agent coordination with Queen Coordinator pattern.**
|
|
|
|
### Agent Hierarchy
|
|
|
|
```
|
|
Queen Coordinator
|
|
|
|
|
+--------+--------+--------+--------+
|
|
| | | | |
|
|
Test Quality Coverage Learning Execution
|
|
Gen Gates Analysis Coord Coord
|
|
| | | | |
|
|
2-5 6-8 9-11 12-14 15-17
|
|
agents agents agents agents agents
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] >85% agent utilization
|
|
- [ ] <5 minute end-to-end QE workflow
|
|
- [ ] Coordination overhead <10%
|
|
- [ ] Automatic work distribution
|
|
|
|
---
|
|
|
|
## ADR-009: AgentDB as Primary Memory Backend
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Multiple memory implementations exist. Need single, optimized backend with vector search and pattern learning.
|
|
|
|
### Decision
|
|
|
|
**Use AgentDB with HNSW indexing as primary memory backend for all QE domains.**
|
|
|
|
### Benefits
|
|
|
|
- 150x-12,500x faster search via HNSW
|
|
- Vector embeddings for semantic search
|
|
- Built-in RL algorithms
|
|
- Cross-agent memory sharing
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] <10ms vector search
|
|
- [ ] 1M+ patterns indexed
|
|
- [ ] Seamless cross-domain access
|
|
- [ ] Memory usage <500MB
|
|
|
|
---
|
|
|
|
## ADR-010: MCP-First Tool Design
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Accepted:** 2026-01-09
|
|
|
|
### Context
|
|
|
|
QE functionality should be accessible via multiple interfaces (CLI, API, programmatic). MCP provides standard interface.
|
|
|
|
### Decision
|
|
|
|
**All QE functionality exposed as MCP tools first, with CLI as wrapper.**
|
|
|
|
### Implementation
|
|
|
|
**14 MCP Tools Implemented:**
|
|
- `qe/tests/generate` - AI-powered test generation
|
|
- `qe/tests/execute` - Parallel test execution
|
|
- `qe/coverage/analyze` - Sublinear coverage analysis
|
|
- `qe/coverage/gaps` - Coverage gap detection
|
|
- `qe/quality/evaluate` - Quality gate evaluation
|
|
- `qe/defects/predict` - Defect prediction
|
|
- `qe/requirements/validate` - Requirements validation
|
|
- `qe/code/analyze` - Code intelligence analysis
|
|
- `qe/security/scan` - Security scanning (SAST/DAST)
|
|
- `qe/contracts/validate` - Contract validation
|
|
- `qe/visual/compare` - Visual regression testing
|
|
- `qe/a11y/audit` - Accessibility auditing
|
|
- `qe/chaos/inject` - Chaos engineering
|
|
- `qe/learning/optimize` - Learning optimization
|
|
|
|
**Key Files:**
|
|
- `src/mcp/tools/base.ts` - Base MCP tool class with validation, streaming, abort
|
|
- `src/mcp/tools/registry.ts` - Tool registry with 14 tools across 12 domains
|
|
- `src/mcp/tools/domain/*.ts` - Domain-specific tool implementations
|
|
- `src/cli/commands/qe-tools.ts` - CLI wrappers (thin, delegates to MCP)
|
|
- `tests/unit/mcp/tools/*.test.ts` - 130 unit tests
|
|
|
|
```typescript
|
|
// MCP Tool (primary) - Example from actual implementation
|
|
export class TestGenerateTool extends MCPToolBase<GenerateParams, GenerateResult> {
|
|
readonly config: MCPToolConfig = {
|
|
name: 'qe/tests/generate',
|
|
description: 'Generate tests using AI-powered analysis',
|
|
domain: 'test-generation',
|
|
schema: { /* JSON Schema */ },
|
|
streaming: true,
|
|
timeout: 60000,
|
|
};
|
|
|
|
async execute(params: GenerateParams, context: MCPToolContext): Promise<ToolResult<GenerateResult>> {
|
|
// Implementation with streaming, abort support, validation
|
|
}
|
|
}
|
|
|
|
// CLI Command (wrapper) - Thin wrapper pattern
|
|
program.command('generate')
|
|
.option('--file <path>', 'Source file path')
|
|
.action(async (options) => {
|
|
const tool = getQETool('qe/tests/generate');
|
|
const result = await tool.invoke(options);
|
|
console.log(JSON.stringify(result, null, 2));
|
|
});
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] 100% MCP coverage (14 tools covering all 12 domains)
|
|
- [x] CLI adds <10% code (thin wrappers only)
|
|
- [x] Consistent API across interfaces (JSON Schema validation)
|
|
- [x] Auto-generated documentation (from tool schemas)
|
|
- [x] 130 unit tests passing
|
|
|
|
---
|
|
|
|
## Decision Framework
|
|
|
|
For future ADRs, use this template:
|
|
|
|
### ADR-XXX: [Title]
|
|
|
|
**Status:** Proposed | Accepted | Deprecated | Superseded
|
|
**Date:** YYYY-MM-DD
|
|
|
|
#### Context
|
|
What is the issue we're trying to solve?
|
|
|
|
#### Decision
|
|
What are we going to do?
|
|
|
|
#### Rationale
|
|
Why this decision? What alternatives did we consider?
|
|
|
|
#### Consequences
|
|
What are the trade-offs?
|
|
|
|
#### Implementation
|
|
How will we do this?
|
|
|
|
#### Success Metrics
|
|
How do we know it worked?
|
|
|
|
---
|
|
|
|
## ADR-011: LLM Provider System for QE
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Test generation requires multiple LLM providers for different scenarios:
|
|
- Claude for complex reasoning and test design
|
|
- GPT-4 for alternative test perspectives
|
|
- Local models (Ollama) for fast iteration and cost control
|
|
- Cost tracking for test generation budget management
|
|
|
|
### Decision
|
|
|
|
**Implement unified LLM provider system with cost tracking, failover, and circuit breaker.**
|
|
|
|
Key Components:
|
|
- Provider Manager with round-robin/cost-based load balancing
|
|
- Circuit breaker for cascading failure prevention
|
|
- Cost tracking per test generation request
|
|
- LRU caching for repeated test generation patterns
|
|
|
|
Provider Priority:
|
|
1. Claude (primary) - Complex test generation
|
|
2. GPT-4 (secondary) - Alternative test perspectives
|
|
3. Ollama (local) - Fast iteration, unlimited runs
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] 3+ LLM providers supported
|
|
- [ ] Cost tracking with budget alerts
|
|
- [ ] Circuit breaker prevents cascading failures
|
|
- [ ] <50ms overhead for provider selection
|
|
|
|
---
|
|
|
|
## ADR-012: MCP Security Features for QE
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Implemented:** 2026-01-09
|
|
|
|
### Context
|
|
|
|
QE MCP tools handle sensitive operations: test execution, code analysis, credential management for integration tests. Need security hardening.
|
|
|
|
### Decision
|
|
|
|
**Implement security features following claude-flow v3 patterns.**
|
|
|
|
Security Features:
|
|
- **JSON Schema Validation** - Validate all MCP tool inputs (schema-validator.ts)
|
|
- **Rate Limiting** - Token bucket (100 req/s, 200 burst) + sliding window (rate-limiter.ts)
|
|
- **OAuth 2.1 + PKCE** - Enterprise authentication (oauth21-provider.ts)
|
|
- **Sampling** - Server-initiated LLM for AI-driven test decisions (sampling-server.ts)
|
|
|
|
CVE Prevention (cve-prevention.ts):
|
|
- Path traversal protection (validatePath, normalizePath, joinPaths)
|
|
- ReDoS prevention with regex escaping (isRegexSafe, createSafeRegex)
|
|
- Timing-safe authentication comparison (timingSafeCompare, timingSafeHashCompare)
|
|
- Input sanitization (sanitizeInput, escapeHtml, stripHtmlTags)
|
|
- Command injection prevention (validateCommand, escapeShellArg)
|
|
|
|
### Implementation
|
|
|
|
Files created in `v3/src/mcp/security/`:
|
|
- `index.ts` - Exports and security middleware factory
|
|
- `schema-validator.ts` - JSON Schema validation with type checking, formats, combinators
|
|
- `rate-limiter.ts` - Token bucket and sliding window rate limiters
|
|
- `oauth21-provider.ts` - OAuth 2.1 + PKCE authorization flows
|
|
- `sampling-server.ts` - Server-initiated LLM sampling with QE decision prompts
|
|
- `cve-prevention.ts` - Security utilities for CVE prevention
|
|
|
|
Test coverage: 207 tests in `v3/tests/unit/mcp/security/`
|
|
|
|
### Success Metrics
|
|
|
|
- [x] All MCP tools have input schema validation
|
|
- [x] Rate limiting prevents abuse (token bucket + sliding window)
|
|
- [x] OAuth 2.1 for enterprise deployments (authorization_code, refresh_token, client_credentials)
|
|
- [x] Zero critical CVEs (path traversal, ReDoS, timing attacks, injection attacks prevented)
|
|
|
|
> **Implementation Note (2026-01-09 - UPDATED):**
|
|
>
|
|
> **REAL SECURITY SCANNER ADDED:**
|
|
> - `semgrep-integration.ts` - Real Semgrep SAST integration that shells out to semgrep when available
|
|
> - Graceful fallback to pattern-based scanning when semgrep not installed
|
|
> - Supports OWASP Top 10, CWE Top 25, and custom rule sets
|
|
>
|
|
> ```bash
|
|
> # Install semgrep for full functionality
|
|
> pip install semgrep
|
|
> ```
|
|
|
|
---
|
|
|
|
## ADR-013: Core Security Module for QE
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
QE executes tests with potentially dangerous operations: file access, command execution, credential handling for integration tests.
|
|
|
|
### Decision
|
|
|
|
**Create `@agentic-qe/security` package with defense-in-depth.**
|
|
|
|
Components:
|
|
1. **Safe Executor** - No shell interpretation, command allowlist
|
|
2. **Path Validator** - Traversal prevention, jail directories
|
|
3. **Credential Manager** - Secure storage for integration test credentials
|
|
4. **Input Sanitizer** - XSS, injection prevention
|
|
|
|
```typescript
|
|
const security = createSecurityModule({
|
|
projectRoot: process.cwd(),
|
|
allowedCommands: ['npm', 'jest', 'vitest', 'playwright'],
|
|
allowedPaths: ['src/', 'tests/', 'coverage/']
|
|
});
|
|
|
|
// Safe test execution
|
|
await security.safeExecutor.execute('jest', ['--coverage']);
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] No command injection vulnerabilities
|
|
- [ ] Path traversal blocked
|
|
- [ ] Credentials encrypted at rest
|
|
- [ ] 400+ security tests passing
|
|
|
|
---
|
|
|
|
## ADR-014: Background Workers for QE Monitoring
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
QE needs continuous monitoring: test health, coverage trends, flaky test detection, security scanning. Claude-flow v3 has 12 background workers.
|
|
|
|
### Decision
|
|
|
|
**Implement QE-specific background worker system with daemon support.**
|
|
|
|
QE Workers (10 total):
|
|
|
|
| Worker | Interval | Description |
|
|
|--------|----------|-------------|
|
|
| test-health | 5 min | Monitor test suite health |
|
|
| coverage-tracker | 10 min | Track coverage trends |
|
|
| flaky-detector | 15 min | Detect flaky test patterns |
|
|
| security-scan | 30 min | Security vulnerability scanning |
|
|
| quality-gate | 5 min | Continuous gate evaluation |
|
|
| learning-consolidation | 30 min | Pattern consolidation |
|
|
| defect-predictor | 15 min | ML defect prediction |
|
|
| regression-monitor | 10 min | Watch for regressions |
|
|
| performance-baseline | 1 hour | Performance trend tracking |
|
|
| compliance-checker | 30 min | ADR/DDD compliance |
|
|
|
|
Session Integration:
|
|
```bash
|
|
# Auto-start on session begin
|
|
aqe daemon start --workers test-health,coverage-tracker,flaky-detector
|
|
|
|
# Check worker status
|
|
aqe daemon status
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] 10 QE workers implemented
|
|
- [ ] Daemon survives session restarts
|
|
- [ ] Historical metrics stored (30 days)
|
|
- [ ] <500ms worker execution time
|
|
|
|
---
|
|
|
|
## ADR-015: Unified Plugin System for QE Extensions
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
QE has optional capabilities (n8n testing, visual regression, performance testing) that not all users need. Following ADR-004, we need a formal plugin SDK.
|
|
|
|
### Decision
|
|
|
|
**Implement plugin SDK with builder pattern following claude-flow v3.**
|
|
|
|
Plugin Types:
|
|
1. **n8n Workflow Testing Plugin**
|
|
2. **Visual Regression Plugin**
|
|
3. **Performance Testing Plugin**
|
|
4. **API Contract Testing Plugin**
|
|
5. **Accessibility Testing Plugin**
|
|
6. **Security Scanning Plugin**
|
|
|
|
Builder Pattern:
|
|
```typescript
|
|
const visualPlugin = new QEPluginBuilder('visual-regression', '1.0.0')
|
|
.withDescription('Visual regression testing with AI')
|
|
.withMCPTools([screenshotTool, compareTool, baselinesTool])
|
|
.withAgents(['visual-tester', 'accessibility-checker'])
|
|
.withWorkers([baselineWorker, regressionWorker])
|
|
.build();
|
|
|
|
await registry.register(visualPlugin);
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] Plugin SDK published
|
|
- [ ] 6+ official QE plugins
|
|
- [ ] <50ms plugin load time
|
|
- [ ] Plugin development guide
|
|
|
|
---
|
|
|
|
## ADR-016: Collaborative Test Task Claims
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Multiple QE agents and humans work on test coverage. Need claim system to prevent duplicate work and enable handoffs.
|
|
|
|
### Decision
|
|
|
|
**Implement collaborative claim system for test tasks following claude-flow v3 patterns.**
|
|
|
|
Claim Types:
|
|
- Coverage gaps (uncovered code → agent claims to write tests)
|
|
- Flaky tests (detected → agent claims to fix)
|
|
- Defect investigations (predicted defect → agent claims to analyze)
|
|
- Test reviews (generated tests → human claims to review)
|
|
|
|
Work Stealing:
|
|
```typescript
|
|
// Agent B steals stale test gap claim
|
|
const stealable = await claimService.getStealable('test-generation');
|
|
if (stealable.length > 0) {
|
|
await claimService.steal(stealable[0].gapId, agentB);
|
|
}
|
|
```
|
|
|
|
Handoff Patterns:
|
|
1. Test Generator → Test Reviewer (human reviews AI tests)
|
|
2. Coverage Analyzer → Test Generator (gaps trigger generation)
|
|
3. Flaky Detector → Flaky Hunter (detection triggers investigation)
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] Claim service for test tasks
|
|
- [ ] Work stealing for idle agents
|
|
- [ ] Human ↔ Agent handoffs working
|
|
- [ ] >90% QE agent utilization
|
|
|
|
---
|
|
|
|
## ADR-017: RuVector Integration for QE Intelligence
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Implemented:** 2026-01-09
|
|
|
|
### Context
|
|
|
|
RuVector provides ML-based code intelligence that enhances QE:
|
|
- Q-Learning agent routing (80%+ accuracy)
|
|
- AST complexity for test prioritization
|
|
- Diff risk classification for targeted testing
|
|
- Graph analysis for module boundary testing
|
|
|
|
### Decision
|
|
|
|
**Integrate RuVector as OPTIONAL dependency with graceful fallback.**
|
|
|
|
Integration Points:
|
|
|
|
| RuVector Feature | QE Application |
|
|
|------------------|----------------|
|
|
| Q-Learning Router | Route test tasks to optimal agents |
|
|
| AST Complexity | Prioritize tests by code complexity |
|
|
| Diff Risk Classification | Target tests at high-risk changes |
|
|
| Coverage Routing | Test coverage-aware agent selection |
|
|
| Graph Boundaries | Focus integration tests at module boundaries |
|
|
|
|
CLI Commands:
|
|
```bash
|
|
# Route test task with Q-Learning
|
|
aqe route --task "Test user authentication" --q-learning
|
|
|
|
# Analyze code for test prioritization
|
|
aqe analyze complexity --path src/auth/
|
|
|
|
# Classify PR risk for targeted testing
|
|
aqe analyze diff --risk
|
|
```
|
|
|
|
Fallback: When RuVector not installed, fall back to rule-based routing.
|
|
|
|
### Implementation (2026-01-09)
|
|
|
|
**Files Created:**
|
|
|
|
```
|
|
v3/src/integrations/ruvector/
|
|
├── index.ts # Barrel export + client factory
|
|
├── interfaces.ts # Core interfaces and config types
|
|
├── fallback.ts # Rule-based fallback implementations
|
|
├── q-learning-router.ts # Q-Learning agent routing
|
|
├── ast-complexity.ts # AST complexity analyzer
|
|
├── diff-risk-classifier.ts # Diff risk classification
|
|
├── coverage-router.ts # Coverage-aware routing
|
|
└── graph-boundaries.ts # Module boundary analysis
|
|
```
|
|
|
|
**Test Files:**
|
|
|
|
```
|
|
v3/tests/unit/integrations/ruvector/
|
|
├── q-learning-router.test.ts
|
|
├── ast-complexity.test.ts
|
|
├── diff-risk-classifier.test.ts
|
|
├── coverage-router.test.ts
|
|
└── graph-boundaries.test.ts
|
|
```
|
|
|
|
**Key Architecture Patterns:**
|
|
|
|
1. **Factory Functions** - Each component has `createXXX(config)` factory
|
|
2. **Graceful Fallback** - Every feature works without RuVector via rule-based fallback
|
|
3. **Result Tracking** - All results include `usedFallback: boolean` indicator
|
|
4. **Caching Support** - TTL-based caching when `cacheEnabled: true`
|
|
5. **Error Types** - `RuVectorError`, `RuVectorUnavailableError`, `RuVectorTimeoutError`
|
|
|
|
**Usage Example:**
|
|
|
|
```typescript
|
|
import {
|
|
createRuVectorClient,
|
|
createQLearningRouter,
|
|
createDiffRiskClassifier,
|
|
DEFAULT_RUVECTOR_CONFIG
|
|
} from '@agentic-qe/v3/integrations/ruvector';
|
|
|
|
// Create client with fallback support
|
|
const client = await createRuVectorClient({
|
|
...DEFAULT_RUVECTOR_CONFIG,
|
|
enabled: true,
|
|
endpoint: 'http://ruvector:8080'
|
|
});
|
|
|
|
// Route task to optimal agent
|
|
const router = createQLearningRouter(client.config);
|
|
const result = await router.routeTask({
|
|
id: 'task-1',
|
|
name: 'Test auth service',
|
|
type: 'unit',
|
|
complexity: 0.7,
|
|
priority: 'p1'
|
|
});
|
|
|
|
// result.usedFallback indicates if RuVector was available
|
|
console.log(`Routed to ${result.agentType} (fallback: ${result.usedFallback})`);
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] RuVector optional dependency configured
|
|
- [x] Q-Learning routing implementation with epsilon-greedy action selection
|
|
- [x] AST complexity analysis with cyclomatic, cognitive, coupling, cohesion metrics
|
|
- [x] Graceful fallback when unavailable (all 5 components have fallback)
|
|
- [x] 5 test files with comprehensive coverage
|
|
- [x] Factory pattern for all components
|
|
- [x] Caching support with TTL
|
|
- [x] Model export/import for Q-Learning persistence
|
|
|
|
---
|
|
|
|
## ADR-018: Expanded 12-Domain Architecture
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
|
|
### Context
|
|
|
|
Initial v3 architecture (ADR-001) proposed 6 bounded contexts with 21 agents. Analysis of v2 capabilities revealed critical gaps:
|
|
|
|
**Missing Capabilities from v2:**
|
|
- Code Intelligence (Knowledge Graph) - `qe-code-intelligence`
|
|
- Requirements Validation - `qe-requirements-validator`
|
|
- Regression Risk Analysis - `qe-regression-risk-analyzer`
|
|
- API Contract Testing - `qe-api-contract-validator`
|
|
- Visual Testing - `qe-visual-tester`
|
|
- Accessibility Testing - `qe-a11y-ally`
|
|
- Chaos Engineering - `qe-chaos-engineer`
|
|
- Production Intelligence - `qe-production-intelligence`
|
|
|
|
**Comparison with claude-flow v3:**
|
|
- claude-flow: 60+ agent types, 26 core commands, 140+ subcommands
|
|
- Initial AQE v3: 21 agents, 6 domains - insufficient
|
|
|
|
### Decision
|
|
|
|
**Expand from 6 domains/21 agents to 12 domains/47 agents to ensure complete v2 migration and feature parity with enterprise needs.**
|
|
|
|
**12 Bounded Contexts:**
|
|
|
|
| # | Domain | Agents | Key Capabilities |
|
|
|---|--------|--------|------------------|
|
|
| 1 | test-generation | 5 | AI test creation, TDD, data generation |
|
|
| 2 | test-execution | 4 | Parallel execution, flaky detection |
|
|
| 3 | coverage-analysis | 4 | O(log n) gaps, mutation testing |
|
|
| 4 | quality-assessment | 4 | Quality gates, deployment readiness |
|
|
| 5 | defect-intelligence | 4 | Prediction, regression risk |
|
|
| 6 | **requirements-validation** | 4 | BDD, testability scoring (NEW) |
|
|
| 7 | **code-intelligence** | 4 | Knowledge Graph, semantic search (NEW) |
|
|
| 8 | **security-compliance** | 4 | SAST/DAST, compliance (EXPANDED) |
|
|
| 9 | **contract-testing** | 4 | API contracts, GraphQL (NEW) |
|
|
| 10 | **visual-accessibility** | 4 | Visual regression, a11y (NEW) |
|
|
| 11 | **chaos-resilience** | 4 | Chaos engineering, load testing (NEW) |
|
|
| 12 | learning-optimization | 5 | Pattern learning, knowledge transfer |
|
|
|
|
**Agent Breakdown:**
|
|
- 1 Queen Coordinator
|
|
- 46 Domain Agents (across 12 domains)
|
|
- 7 Subagents (task-specific workers)
|
|
- 2 Specialized (cross-domain: QX Partner, Fleet Commander)
|
|
- **Total: 56 agents**
|
|
|
|
### Migration Map (v2 → v3)
|
|
|
|
```
|
|
v2 Agent → v3 Agent Domain
|
|
─────────────────────────────────────────────────────────────────
|
|
qe-code-intelligence → v3-qe-code-intelligence code-intelligence
|
|
qe-requirements-validator → v3-qe-requirements-validator requirements-validation
|
|
qe-regression-risk-analyzer→ v3-qe-regression-analyzer defect-intelligence
|
|
qe-api-contract-validator → v3-qe-contract-validator contract-testing
|
|
qe-visual-tester → v3-qe-visual-tester visual-accessibility
|
|
qe-a11y-ally → v3-qe-a11y-specialist visual-accessibility
|
|
qe-chaos-engineer → v3-qe-chaos-engineer chaos-resilience
|
|
qe-security-scanner → v3-qe-security-scanner security-compliance
|
|
qe-performance-tester → v3-qe-performance-profiler chaos-resilience
|
|
qe-production-intelligence→ v3-qe-production-intel learning-optimization
|
|
qe-code-complexity → v3-qe-code-complexity quality-assessment
|
|
qe-test-data-architect → v3-qe-test-data-architect test-generation
|
|
qx-partner → v3-qe-qx-partner specialized
|
|
qe-fleet-commander → v3-qe-fleet-commander specialized
|
|
```
|
|
|
|
### New Coordination Protocols
|
|
|
|
```yaml
|
|
protocols:
|
|
code-intelligence-index:
|
|
description: "Knowledge graph indexing"
|
|
participants: [v3-qe-code-intelligence, v3-qe-semantic-analyzer]
|
|
triggers: [code-change, hourly, "aqe kg index"]
|
|
|
|
security-audit:
|
|
description: "Security and compliance audit"
|
|
participants: [v3-qe-security-scanner, v3-qe-compliance-validator]
|
|
triggers: [daily-2am, dependency-update, "aqe security audit"]
|
|
|
|
requirements-validation:
|
|
description: "Pre-development requirements analysis"
|
|
participants: [v3-qe-requirements-validator, v3-qe-bdd-scenario-writer]
|
|
triggers: [user-story-created, sprint-planning]
|
|
```
|
|
|
|
### Rationale
|
|
|
|
**Pros:**
|
|
- Complete v2 feature migration
|
|
- Addresses user feedback about missing agents
|
|
- Knowledge Graph crucial for intelligent test targeting
|
|
- Requirements validation enables shift-left testing
|
|
- Visual/a11y testing for modern web applications
|
|
- Chaos engineering for distributed systems
|
|
|
|
**Cons:**
|
|
- More complex coordination (47 vs 21 agents)
|
|
- Longer migration timeline
|
|
- Higher resource requirements
|
|
|
|
**Mitigation:**
|
|
- Hierarchical coordination (Queen + 6 group leaders)
|
|
- Max 15 concurrent agents (per claude-flow v3)
|
|
- Lazy loading of domains (only activate when needed)
|
|
|
|
### Configuration
|
|
|
|
```yaml
|
|
# .agentic-qe/config.yaml
|
|
v3:
|
|
domains: 12
|
|
maxConcurrentAgents: 15
|
|
lazyLoading: true
|
|
memoryBackend: hybrid
|
|
hnswEnabled: true
|
|
backgroundWorkers: 12
|
|
hooks: 17
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [ ] All 22 v2 QE agents migrated to v3
|
|
- [ ] 12 domains implemented
|
|
- [ ] Knowledge Graph operational with O(log n) search
|
|
- [ ] Requirements validation integrated with CI/CD
|
|
- [ ] Visual/a11y testing automated
|
|
- [ ] <15 concurrent agents during peak load
|
|
|
|
---
|
|
|
|
---
|
|
|
|
## ADR-019: Phase 1 Foundation Implementation
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Decision Makers:** Architecture Team
|
|
|
|
### Context
|
|
|
|
Phase 1 of AQE v3 implementation requires establishing the foundational architecture before domain-specific work can begin. This ADR tracks the implementation progress of Phase 1 Foundation (Weeks 1-4 per Master Plan).
|
|
|
|
### Decision
|
|
|
|
**Implement Phase 1 Foundation with the following components:**
|
|
|
|
### Implementation Progress
|
|
|
|
#### ✅ Completed Tasks
|
|
|
|
| Task | Status | Date | Details |
|
|
|------|--------|------|---------|
|
|
| Directory structure for 12 domains | ✅ Complete | 2026-01-07 | All domain directories created |
|
|
| Shared kernel types | ✅ Complete | 2026-01-07 | `src/shared/types/index.ts` |
|
|
| Value objects | ✅ Complete | 2026-01-07 | FilePath, Coverage, RiskScore, TimeRange, Version |
|
|
| Domain entities | ✅ Complete | 2026-01-07 | Agent, TestCase, TestSuite with AggregateRoot |
|
|
| Domain events | ✅ Complete | 2026-01-07 | 15 core domain events defined |
|
|
| Event bus | ✅ Complete | 2026-01-07 | InMemoryEventBus with pub/sub |
|
|
| Agent coordinator | ✅ Complete | 2026-01-07 | Max 15 concurrent agents enforced |
|
|
| Plugin loader | ✅ Complete | 2026-01-07 | Lazy loading with dependency resolution |
|
|
| Memory backend | ✅ Complete | 2026-01-07 | In-memory store with vector search |
|
|
| QE Kernel | ✅ Complete | 2026-01-07 | QEKernelImpl microkernel |
|
|
| Domain interfaces (12/12) | ✅ Complete | 2026-01-07 | All 12 domains defined |
|
|
| Foundation tests | ✅ Complete | 2026-01-07 | Event bus, coordinator, value objects |
|
|
| Build configuration | ✅ Complete | 2026-01-07 | package.json, tsconfig.json, vitest.config.ts |
|
|
|
|
#### Domain Interfaces Implemented
|
|
|
|
| # | Domain | File | Status |
|
|
|---|--------|------|--------|
|
|
| 1 | test-generation | `src/domains/test-generation/interfaces.ts` | ✅ |
|
|
| 2 | test-execution | `src/domains/test-execution/interfaces.ts` | ✅ |
|
|
| 3 | coverage-analysis | `src/domains/coverage-analysis/interfaces.ts` | ✅ |
|
|
| 4 | quality-assessment | `src/domains/quality-assessment/interfaces.ts` | ✅ |
|
|
| 5 | defect-intelligence | `src/domains/defect-intelligence/interfaces.ts` | ✅ |
|
|
| 6 | code-intelligence | `src/domains/code-intelligence/interfaces.ts` | ✅ |
|
|
| 7 | requirements-validation | `src/domains/requirements-validation/interfaces.ts` | ✅ |
|
|
| 8 | security-compliance | `src/domains/security-compliance/interfaces.ts` | ✅ |
|
|
| 9 | contract-testing | `src/domains/contract-testing/interfaces.ts` | ✅ |
|
|
| 10 | visual-accessibility | `src/domains/visual-accessibility/interfaces.ts` | ✅ |
|
|
| 11 | chaos-resilience | `src/domains/chaos-resilience/interfaces.ts` | ✅ |
|
|
| 12 | learning-optimization | `src/domains/learning-optimization/interfaces.ts` | ✅ |
|
|
|
|
### Key Architecture Decisions Made
|
|
|
|
1. **Max 15 Concurrent Agents**: Per claude-flow v3 spec, enforced in `agent-coordinator.ts`
|
|
2. **HNSW Indexing**: Prepared for O(log n) vector search in memory backend
|
|
3. **Event-Driven Communication**: All domains communicate via domain events
|
|
4. **Microkernel Pattern**: Core QEKernel with lazy-loaded domain plugins
|
|
5. **Value Objects**: Immutable value objects for domain concepts
|
|
6. **Result Type**: Functional error handling with `Result<T, E>`
|
|
|
|
### Files Created
|
|
|
|
```
|
|
v3/
|
|
├── package.json (v3.0.0-alpha.1)
|
|
├── tsconfig.json
|
|
├── vitest.config.ts
|
|
├── src/
|
|
│ ├── shared/
|
|
│ │ ├── types/index.ts
|
|
│ │ ├── value-objects/index.ts
|
|
│ │ ├── entities/
|
|
│ │ │ ├── base.ts
|
|
│ │ │ ├── agent.ts
|
|
│ │ │ ├── test-case.ts
|
|
│ │ │ └── test-suite.ts
|
|
│ │ └── events/index.ts
|
|
│ ├── kernel/
|
|
│ │ ├── interfaces.ts
|
|
│ │ ├── event-bus.ts
|
|
│ │ ├── agent-coordinator.ts
|
|
│ │ ├── plugin-loader.ts
|
|
│ │ ├── memory-backend.ts
|
|
│ │ └── kernel.ts
|
|
│ └── domains/ (12 domains with interfaces.ts)
|
|
└── tests/
|
|
└── unit/
|
|
├── kernel/
|
|
│ ├── event-bus.test.ts
|
|
│ └── agent-coordinator.test.ts
|
|
└── shared/
|
|
└── value-objects.test.ts
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] 12 domain directories created
|
|
- [x] Shared kernel implemented
|
|
- [x] All domain interfaces defined
|
|
- [x] QE Kernel operational
|
|
- [x] Event bus functional
|
|
- [x] Agent concurrency limit enforced (max 15)
|
|
- [x] Foundation tests written
|
|
- [x] All tests passing ✅
|
|
- [x] TypeScript compilation successful ✅
|
|
|
|
### Phase 2: Domain Services - **COMPLETE**
|
|
|
|
- 48 service files implemented across 12 domains
|
|
- Note: 18 stub implementations remain (to be replaced per ADR-020)
|
|
|
|
### Phase 3: Event Integration - **COMPLETE**
|
|
|
|
- CrossDomainEventRouter implemented
|
|
- 7 coordination protocols (morning-sync, quality-gate, regression-prevention, coverage-driven, tdd-cycle, security-audit, learning-consolidation)
|
|
- WorkflowOrchestrator with built-in workflows
|
|
- Queen Coordinator with hierarchical coordination
|
|
|
|
### Current Stats (2026-01-09 - Verified by Code Review Agent)
|
|
|
|
| Metric | Count |
|
|
|--------|-------|
|
|
| Source Files | 182 |
|
|
| Test Files | 46 |
|
|
| Tests Passing | 1171 |
|
|
| Domains | 12 |
|
|
| Stubs Remaining | 10 (visual-accessibility + security-audit protocol) |
|
|
| Shared Modules | 9 (embeddings, entities, events, git, http, io, llm, parsers, security) |
|
|
| Coordination Protocols | 7 |
|
|
| MCP Handlers | 6 groups (core, agent, task, memory, domain, index) |
|
|
|
|
### Next Steps (Phase 4)
|
|
|
|
1. Replace remaining 10 stub implementations (visual testing needs Playwright)
|
|
2. Implement full MCP tool wrappers for all domain services
|
|
3. Add background worker daemon implementation
|
|
4. Complete RuVector integration for intelligent routing
|
|
5. Implement claim-based work distribution for multi-agent coordination
|
|
|
|
### Rationale
|
|
|
|
Phase 1 establishes the architectural foundation that all subsequent phases depend on. By completing the shared kernel, event bus, and domain interfaces first, we ensure:
|
|
|
|
- Consistent patterns across all 12 domains
|
|
- Type safety through comprehensive interfaces
|
|
- Testable foundation before adding complexity
|
|
- Clear boundaries for parallel development
|
|
|
|
---
|
|
|
|
## ADR-020: Stub Implementation Replacement
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-07
|
|
**Updated:** 2026-01-11 (Final Verification)
|
|
|
|
### Context
|
|
|
|
Phase 2 domain services contained stub implementations. A brutal honesty review revealed more stubs than initially claimed. Final verification on 2026-01-11 confirmed all domain service stubs have been replaced.
|
|
|
|
### Decision
|
|
|
|
Replace stub implementations with REAL functionality - not facades or wrappers that call simulated code.
|
|
|
|
### Final Status (2026-01-11 - Verified)
|
|
|
|
**Stub Analysis Results:**
|
|
- `// Stub:` comments: **0** ✅
|
|
- `// Simulated` comments: **9** (all in orchestration/MCP layers - acceptable)
|
|
- Domain service stubs: **0** ✅
|
|
|
|
**REAL Implementations Added:**
|
|
|
|
| Component | File | Status | Notes |
|
|
|-----------|------|--------|-------|
|
|
| HNSW Index | `coverage-analysis/services/hnsw-index.ts` | ✅ REAL | Uses `hnswlib-node` native library |
|
|
| Coverage Parser | `coverage-analysis/services/coverage-parser.ts` | ✅ REAL | Uses `lcov-parse` for actual LCOV files |
|
|
| Security Scanner | `security-compliance/services/semgrep-integration.ts` | ✅ REAL | Shells to semgrep CLI |
|
|
| A11y Audit | `visual-accessibility/services/axe-core-audit.ts` | ✅ REAL | Uses axe-core + Playwright |
|
|
| Benchmarks | `benchmarks/performance-benchmarks.ts` | ✅ REAL | Actual performance measurement |
|
|
| Defect Predictor | `defect-intelligence/services/defect-predictor.ts` | ✅ REAL | ML-based prediction (27k LOC) |
|
|
| Root Cause Analyzer | `defect-intelligence/services/root-cause-analyzer.ts` | ✅ REAL | Pattern-based analysis (18k LOC) |
|
|
| Pattern Learner | `defect-intelligence/services/pattern-learner.ts` | ✅ REAL | Learning system (23k LOC) |
|
|
|
|
**Shared Modules Created:**
|
|
- [x] `src/shared/parsers/` - TypeScript parser utilities
|
|
- [x] `src/shared/io/` - File I/O abstractions
|
|
- [x] `src/shared/http/` - HTTP client utilities
|
|
- [x] `src/shared/embeddings/` - Nomic embedder integration
|
|
- [x] `src/shared/security/` - Security utilities (OSV client, compliance patterns)
|
|
- [x] `src/shared/git/` - Git integration utilities
|
|
- [x] `src/shared/metrics/` - Metrics collection
|
|
- [x] `src/shared/llm/` - Circuit breaker, cost tracker, cache
|
|
|
|
**Remaining Simulated Code (Acceptable):**
|
|
|
|
| Location | Count | Reason |
|
|
|----------|-------|--------|
|
|
| `coordination/task-executor.ts` | 6 | Orchestration layer facades (delegates to domain services) |
|
|
| `mcp/handlers/agent-handlers.ts` | 2 | Resource metrics for dev/demo |
|
|
| `mcp/security/sampling-server.ts` | 1 | Test response for validation |
|
|
|
|
> **Note:** These are acceptable because they are in orchestration/MCP layers, not domain services. The actual domain logic is fully implemented.
|
|
|
|
### Success Criteria
|
|
|
|
- [x] 0 "// Stub:" in production code ✅
|
|
- [x] Domain service stubs replaced with real implementations ✅
|
|
- [x] Real HNSW with native library (graceful fallback)
|
|
- [x] Real LCOV parsing with lcov-parse
|
|
- [x] Real security scanning with semgrep
|
|
- [x] Real a11y testing with axe-core + Playwright
|
|
- [x] Real benchmarks to verify O(log n) claims
|
|
|
|
### Build & Test Status (Verified 2026-01-11)
|
|
|
|
- [x] TypeScript build passes (`npm run build`)
|
|
- [x] All **2,382 tests** passing (`npm test -- --run`)
|
|
- [x] **91 test files** across unit and integration tests
|
|
- [x] 166 source files compiled successfully
|
|
- [x] No circular dependencies detected
|
|
|
|
### Issues Fixed During Verification
|
|
|
|
1. **cli-adapter.ts (TS6133):** Unused `agentMapper` variable - Added getter to expose
|
|
2. **config-migrator.ts (TS2352):** Type conversion error - Fixed with proper casting
|
|
3. **knowledge-graph.ts (TS2339):** `KGEdge.id` property missing - Changed to composite key
|
|
4. **mcp/tools/index.ts:** Missing module imports - Replaced with base types
|
|
|
|
---
|
|
|
|
## ADR-021: QE ReasoningBank for Pattern Learning
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-09
|
|
**Implemented:** 2026-01-09
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** [AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md](../../docs/plans/AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md)
|
|
|
|
### Context
|
|
|
|
AQE v3 has basic pattern storage via SQLite and in-memory HNSW, but lacks QE-specific pattern learning. Claude-flow v3's ReasoningBank provides:
|
|
- HNSW vector indexing (150x-12,500x faster search)
|
|
- Pattern storage with quality scoring
|
|
- Short-term to long-term promotion
|
|
- Agent routing via similarity
|
|
|
|
AQE needs a QE-specific ReasoningBank that understands quality engineering domains and can guide test generation, coverage improvement, and defect prediction.
|
|
|
|
**Gap Analysis:**
|
|
|
|
| Capability | Claude-Flow V3 | AQE Current | Gap |
|
|
|------------|----------------|-------------|-----|
|
|
| Pattern Learning | TypeScript + AgentDB | SQLite + in-memory HNSW | Medium |
|
|
| Domain Templates | 5 generic templates | None | High |
|
|
| Quality Scoring | Outcome-based promotion | Partial | Medium |
|
|
| ONNX Embeddings | @claude-flow/embeddings | Hash fallback | Medium |
|
|
|
|
### Decision
|
|
|
|
**Implement QEReasoningBank extending claude-flow's ReasoningBank with QE-specific domains, patterns, and guidance templates.**
|
|
|
|
### QE Domain Pattern Types
|
|
|
|
```typescript
|
|
export const QE_DOMAINS = {
|
|
'test-generation': /test|spec|describe|it\(|expect|assert/i,
|
|
'coverage-analysis': /coverage|branch|line|uncovered|gap/i,
|
|
'mutation-testing': /mutant|mutation|kill|survive/i,
|
|
'api-testing': /endpoint|request|response|api|contract/i,
|
|
'security-testing': /vuln|cve|owasp|xss|sqli|injection/i,
|
|
'visual-testing': /screenshot|visual|snapshot|regression/i,
|
|
'accessibility': /a11y|aria|wcag|screen.?reader/i,
|
|
'performance': /load|stress|benchmark|latency|throughput/i,
|
|
};
|
|
|
|
export type QEPatternType =
|
|
| 'test-template'
|
|
| 'assertion-pattern'
|
|
| 'mock-pattern'
|
|
| 'coverage-strategy'
|
|
| 'mutation-strategy'
|
|
| 'api-contract'
|
|
| 'visual-baseline'
|
|
| 'a11y-check'
|
|
| 'perf-benchmark';
|
|
```
|
|
|
|
### Implementation Plan
|
|
|
|
**Package:** `v3/@aqe-platform/learning/`
|
|
|
|
**Files:**
|
|
- `src/qe-reasoning-bank.ts` - QE-specific ReasoningBank
|
|
- `src/qe-patterns.ts` - Pattern types and validation
|
|
- `src/qe-guidance.ts` - Domain-specific guidance templates
|
|
- `src/pattern-store.ts` - Persistent pattern storage
|
|
|
|
**Phase 1 (Week 1-2):**
|
|
1. Create `@aqe-platform/learning` package
|
|
2. Implement `QEReasoningBank` extending ReasoningBank
|
|
3. Define QE pattern types and domains
|
|
4. Set up SQLite persistence with HNSW indexing
|
|
5. Create QE hooks for pattern capture
|
|
|
|
### Implementation (2026-01-09)
|
|
|
|
**Files Created:**
|
|
|
|
```
|
|
v3/src/learning/
|
|
├── index.ts # Module exports
|
|
├── qe-patterns.ts # 8 QE domains, 12 pattern types
|
|
├── qe-guidance.ts # Guidance templates for all domains
|
|
├── pattern-store.ts # HNSW-indexed pattern storage
|
|
├── qe-reasoning-bank.ts # Main QEReasoningBank class
|
|
├── qe-hooks.ts # Event handlers for pattern capture
|
|
├── real-qe-reasoning-bank.ts # REAL implementation extending HybridReasoningBank
|
|
├── real-embeddings.ts # REAL transformer embeddings (all-MiniLM-L6-v2)
|
|
└── sqlite-persistence.ts # REAL SQLite persistence (better-sqlite3)
|
|
|
|
v3/tests/unit/learning/
|
|
├── qe-reasoning-bank.test.ts # 46 comprehensive unit tests
|
|
└── real-qe-reasoning-bank.benchmark.test.ts # REAL benchmarks with measured performance
|
|
```
|
|
|
|
**Key Features Implemented:**
|
|
|
|
1. **8 QE Domains**: test-generation, coverage-analysis, mutation-testing, api-testing, security-testing, visual-testing, accessibility, performance
|
|
2. **12 Pattern Types**: test-template, assertion-pattern, mock-pattern, coverage-strategy, mutation-strategy, api-contract, visual-baseline, a11y-check, perf-benchmark, flaky-fix, refactor-safe, error-handling
|
|
3. **HNSW Vector Indexing**: Real `hnswlib-node` integration
|
|
4. **Quality Scoring**: Combines confidence (30%), usage (20%), success rate (50%)
|
|
5. **Pattern Promotion**: Short-term → Long-term after 3+ uses, >70% success, >60% confidence
|
|
6. **Agent Routing**: 11 QE agents mapped with capabilities
|
|
7. **Guidance Templates**: Best practices, anti-patterns, framework/language-specific guidance
|
|
|
|
### REAL Implementation (2026-01-09)
|
|
|
|
**Honest Assessment After Code Review:**
|
|
|
|
The initial implementation made claims that weren't fully backed by code. A brutal-honesty review identified gaps:
|
|
- "Extends claude-flow ReasoningBank" - Was NOT actually extending anything
|
|
- "ONNX embeddings" - Was using hash-based fallback
|
|
- "SQLite persistence" - Was using MemoryBackend
|
|
|
|
**REAL Implementation Added:**
|
|
|
|
1. **`real-qe-reasoning-bank.ts`** - Standalone implementation with REAL components
|
|
2. **`real-embeddings.ts`** - REAL transformer embeddings using `@xenova/transformers` (all-MiniLM-L6-v2, 384 dimensions)
|
|
3. **`sqlite-persistence.ts`** - REAL SQLite using `better-sqlite3` with WAL mode, prepared statements
|
|
|
|
**Note on HybridReasoningBank Decision:** We initially attempted to extend `agentic-flow`'s `HybridReasoningBank`, but abandoned this due to:
|
|
- **Bug #1:** `SharedMemoryPool` uses `new Database()` without importing `better-sqlite3`
|
|
- **Bug #2:** Singleton behavior causes repeated re-initialization on every method call (OOM in CI)
|
|
- **Solution:** Standalone implementation gives full control and proper resource management
|
|
|
|
**Verified Benchmarks (REAL numbers - 56 tests passing):**
|
|
|
|
| Component | Metric | Result |
|
|
|-----------|--------|--------|
|
|
| Transformer Embeddings | Model Load | ~70-100ms |
|
|
| Transformer Embeddings | Semantic Similarity (related texts) | 65.4% |
|
|
| Transformer Embeddings | Semantic Similarity (unrelated texts) | 23.3% |
|
|
| SQLite Writes | Throughput | 127,212/sec |
|
|
| SQLite Reads | Throughput | 41,657/sec |
|
|
|
|
**Usage Example:**
|
|
|
|
```typescript
|
|
import {
|
|
createRealQEReasoningBank, // Standalone REAL implementation
|
|
createQEReasoningBank, // or in-memory implementation
|
|
QEDomain,
|
|
generateGuidanceContext
|
|
} from '@agentic-qe/v3';
|
|
|
|
// REAL implementation with SQLite + HNSW + Transformers
|
|
const realBank = createRealQEReasoningBank({
|
|
sqlite: { dbPath: '.agentic-qe/qe-patterns.db' },
|
|
});
|
|
await realBank.initialize();
|
|
|
|
// Standalone implementation (no agentic-flow dependency)
|
|
const standaloneBank = createQEReasoningBank();
|
|
await standaloneBank.initialize();
|
|
|
|
// Both support same API
|
|
const routing = await realBank.routeTask({
|
|
task: 'Generate unit tests for UserService',
|
|
context: { language: 'typescript', framework: 'vitest' },
|
|
});
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] QEReasoningBank with 8 QE domains
|
|
- [x] Pattern storage with quality scoring
|
|
- [x] Short-term to long-term promotion (3+ successful uses)
|
|
- [x] REAL transformer embeddings (all-MiniLM-L6-v2, 384d)
|
|
- [x] REAL SQLite persistence (114k+ writes/sec, 33k+ reads/sec)
|
|
- [x] QE guidance templates for each domain
|
|
- [ ] Full HybridReasoningBank integration (upstream database dependency)
|
|
- [x] 46 unit tests + 6 benchmark tests passing
|
|
|
|
---
|
|
|
|
## ADR-022: Adaptive QE Agent Routing
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-09
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** [AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md](../../docs/plans/AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md)
|
|
|
|
### Context
|
|
|
|
Current AQE agent routing uses basic regex matching and rule-based selection. This leads to:
|
|
- Suboptimal agent selection (wrong agent for task)
|
|
- No learning from past performance
|
|
- No consideration of historical success rates
|
|
- No capability matching
|
|
|
|
Claude-flow v3 provides ML-based routing via ReasoningBank patterns. AQE has 80 agents that need intelligent routing.
|
|
|
|
### Decision
|
|
|
|
**Implement ML-based task routing that combines vector similarity, historical performance, and capability matching.**
|
|
|
|
### Routing Algorithm
|
|
|
|
```typescript
|
|
class QETaskRouter {
|
|
async route(task: QETask): Promise<QERoutingDecision> {
|
|
// 1. Get embedding for task description
|
|
const taskEmbedding = await this.reasoningBank.embed(task.description);
|
|
|
|
// 2. Search for similar past successful tasks
|
|
const similarTasks = await this.reasoningBank.searchPatterns(taskEmbedding, 10);
|
|
|
|
// 3. Get historical performance by agent
|
|
const agentPerformance = this.calculateAgentPerformance(similarTasks);
|
|
|
|
// 4. Match task requirements to agent capabilities
|
|
const capabilityScores = this.matchCapabilities(task, this.agentRegistry);
|
|
|
|
// 5. Combine scores with learned weights
|
|
const finalScores = this.combineScores({
|
|
similarity: 0.3, // How similar to past successful tasks
|
|
performance: 0.4, // Historical agent performance
|
|
capabilities: 0.3, // Capability match
|
|
}, similarTasks, agentPerformance, capabilityScores);
|
|
|
|
return {
|
|
recommended: finalScores[0].agent,
|
|
confidence: finalScores[0].score,
|
|
alternatives: finalScores.slice(1, 4),
|
|
reasoning: this.generateReasoning(task, finalScores[0]),
|
|
};
|
|
}
|
|
}
|
|
```
|
|
|
|
### Agent Registry
|
|
|
|
All 78 QE agents mapped with capabilities:
|
|
|
|
```typescript
|
|
export const QE_AGENT_REGISTRY: Record<string, QEAgentProfile> = {
|
|
'qe-test-generator': {
|
|
capabilities: ['test-generation', 'tdd', 'bdd'],
|
|
frameworks: ['jest', 'vitest', 'pytest', 'junit'],
|
|
languages: ['typescript', 'javascript', 'python', 'java'],
|
|
complexity: { min: 'simple', max: 'complex' },
|
|
performanceScore: 0.85, // Updated via feedback
|
|
},
|
|
'qe-coverage-analyzer': {
|
|
capabilities: ['coverage-analysis', 'gap-detection'],
|
|
algorithms: ['sublinear', 'O(log n)'],
|
|
scalability: '10M+ lines',
|
|
performanceScore: 0.92,
|
|
},
|
|
// ... 76 more agents
|
|
};
|
|
```
|
|
|
|
### Implementation Plan
|
|
|
|
**Package:** `v3/@aqe-platform/routing/`
|
|
|
|
**Phase 2 (Week 2-3):**
|
|
1. Create QE agent registry (78 agents with capabilities)
|
|
2. Implement QETaskRouter with similarity search
|
|
3. Add routing feedback collection
|
|
4. Build routing accuracy monitoring
|
|
5. Integrate with existing Queen Coordinator
|
|
|
|
### Success Metrics
|
|
|
|
- [x] 80 agents registered with capabilities (6 categories: v3QE, n8n, general, v3Specialized, swarm, consensus)
|
|
- [ ] >85% routing accuracy (needs production data)
|
|
- [x] <100ms routing decision latency (P95: 62ms achieved)
|
|
- [x] Routing feedback loop operational (RoutingFeedbackCollector)
|
|
- [x] Agent performance scores updated from outcomes
|
|
|
|
### Implementation (2026-01-09)
|
|
|
|
**Package:** `v3/src/routing/`
|
|
|
|
**Files Created:**
|
|
- `types.ts` - Comprehensive type definitions (QEAgentProfile, QETask, QERoutingDecision, etc.)
|
|
- `qe-agent-registry.ts` - 80 QE agents with capabilities, domains, frameworks, languages
|
|
- `qe-task-router.ts` - ML-based router using transformer embeddings (all-MiniLM-L6-v2)
|
|
- `routing-feedback.ts` - Feedback collection for continuous learning
|
|
- `index.ts` - Module exports
|
|
|
|
**Agent Categories:**
|
|
- V3 QE Agents: 38 (test-generation, coverage, security, visual, performance, etc.)
|
|
- n8n Workflow Agents: 15 (workflow testing, triggers, security, chaos)
|
|
- General Agents: 7 (tester, reviewer, security, performance)
|
|
- V3 Specialized: 11 (ReasoningBank, ADR, DDD, SPARC, SONA, etc.)
|
|
- Swarm Agents: 8 (queen, mesh, hierarchical coordination)
|
|
- Consensus Agents: 4 (Byzantine, Raft, CRDT)
|
|
|
|
**Routing Algorithm:**
|
|
1. Detect domain and capabilities from task description
|
|
2. Pre-filter candidates by hard requirements (language, framework, complexity)
|
|
3. Compute vector similarity using transformer embeddings
|
|
4. Score historical performance (from feedback)
|
|
5. Match capabilities to task requirements
|
|
6. Combine scores with configurable weights (default: 30% similarity, 40% performance, 30% capabilities)
|
|
|
|
**Benchmarks (Real):**
|
|
- P95 Latency: 62ms (target: <100ms) ✓
|
|
- Throughput: 83 tasks/sec (target: >5) ✓
|
|
- 74 unit tests passing
|
|
|
|
---
|
|
|
|
## ADR-023: Quality Feedback Loop System
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-09
|
|
**Implemented:** 2026-01-09
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** [AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md](../../docs/plans/AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md)
|
|
|
|
### Context
|
|
|
|
AQE generates tests, analyzes coverage, and predicts defects but doesn't learn from outcomes:
|
|
- Generated tests pass/fail without feeding back
|
|
- Coverage improvements not tracked
|
|
- Defect predictions not validated
|
|
- No pattern quality updates
|
|
|
|
Need closed-loop feedback to improve patterns over time.
|
|
|
|
### Decision
|
|
|
|
**Implement comprehensive feedback loop that tracks outcomes and updates pattern quality.**
|
|
|
|
### Feedback Components
|
|
|
|
#### 1. Test Outcome Tracker
|
|
|
|
```typescript
|
|
export interface TestOutcome {
|
|
testId: string;
|
|
generatedBy: string; // Agent that generated
|
|
patternId?: string; // Pattern used
|
|
framework: string;
|
|
passed: boolean;
|
|
coverage: {
|
|
lines: number;
|
|
branches: number;
|
|
functions: number;
|
|
};
|
|
mutationScore?: number;
|
|
executionTime: number;
|
|
flaky: boolean;
|
|
maintainability: number; // 0-1 score
|
|
}
|
|
|
|
class TestOutcomeTracker {
|
|
async track(outcome: TestOutcome): Promise<void> {
|
|
// 1. Store outcome
|
|
await this.db.insert('test_outcomes', outcome);
|
|
|
|
// 2. Update pattern quality
|
|
if (outcome.patternId) {
|
|
const qualityDelta = this.calculateQualityDelta(outcome);
|
|
await this.reasoningBank.recordOutcome(
|
|
outcome.patternId,
|
|
outcome.passed && !outcome.flaky
|
|
);
|
|
}
|
|
|
|
// 3. Check for pattern promotion
|
|
await this.checkPatternPromotion(outcome.patternId);
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 2. Coverage Improvement Learner
|
|
|
|
```typescript
|
|
class CoverageLearner {
|
|
async learnFromCoverageSession(session: CoverageSession): Promise<void> {
|
|
const improvement = session.afterCoverage - session.beforeCoverage;
|
|
|
|
if (improvement > 5) { // Significant improvement
|
|
const strategy = this.extractStrategy(session);
|
|
await this.reasoningBank.storePattern(
|
|
strategy.description,
|
|
'coverage-analysis',
|
|
{ improvement, technique: session.technique }
|
|
);
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 3. Routing Feedback Collector
|
|
|
|
```typescript
|
|
class RoutingFeedbackCollector {
|
|
async recordRoutingOutcome(
|
|
taskId: string,
|
|
routingDecision: QERoutingDecision,
|
|
outcome: TaskOutcome
|
|
): Promise<void> {
|
|
await this.db.insert('routing_outcomes', {
|
|
taskId,
|
|
recommendedAgent: routingDecision.recommended,
|
|
actualAgent: outcome.usedAgent,
|
|
followedRecommendation: routingDecision.recommended === outcome.usedAgent,
|
|
success: outcome.success,
|
|
quality: outcome.qualityScore,
|
|
duration: outcome.durationMs,
|
|
});
|
|
|
|
// Update agent performance metrics
|
|
await this.updateAgentMetrics(outcome.usedAgent, outcome);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Implementation Plan
|
|
|
|
**Package:** `v3/src/feedback/`
|
|
|
|
**Phase 3 (Week 3-4):**
|
|
1. ✅ Implement TestOutcomeTracker
|
|
2. ✅ Build CoverageLearner
|
|
3. ⏳ Add RoutingFeedbackCollector (future)
|
|
4. ✅ Create quality score calculator
|
|
5. ✅ Implement pattern promotion logic
|
|
|
|
### Implementation Notes
|
|
|
|
**Implemented Components:**
|
|
- `types.ts` - Comprehensive types for feedback system
|
|
- `test-outcome-tracker.ts` - Tracks test outcomes with pattern correlation
|
|
- `coverage-learner.ts` - Learns from coverage sessions, extracts strategies
|
|
- `quality-score-calculator.ts` - Multi-dimensional quality scoring
|
|
- `pattern-promotion.ts` - Tier-based promotion/demotion
|
|
- `feedback-loop.ts` - Main integrator connecting all components
|
|
- `index.ts` - Module exports
|
|
|
|
**Integration with ADR-021:**
|
|
- Added `recordPatternOutcome()`, `promotePattern()`, `demotePattern()` to ReasoningBank
|
|
- Added `updatePattern()` to SQLite persistence layer
|
|
|
|
**Benchmarks (Real):**
|
|
- 82 unit tests passing
|
|
- Quality dimensions: effectiveness, coverage, mutation, stability, maintainability, performance
|
|
- Pattern tiers: short-term → working → long-term → permanent
|
|
|
|
### Success Metrics
|
|
|
|
- [x] Test outcomes tracked with pattern correlation
|
|
- [x] Coverage improvement patterns captured
|
|
- [x] Pattern quality scores updated from outcomes
|
|
- [x] Multi-dimensional quality scoring (6 dimensions)
|
|
- [x] Tier-based pattern promotion/demotion
|
|
- [x] 87 unit tests + 14 integration tests = 101 tests passing
|
|
- [ ] >20% quality improvement after 1 sprint (needs runtime measurement)
|
|
- [ ] <5ms feedback recording latency (P95) (needs runtime measurement)
|
|
|
|
---
|
|
|
|
## ADR-024: Self-Optimization Engine
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-09
|
|
**Implemented:** 2026-01-09
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** [AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md](../../docs/plans/AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md)
|
|
|
|
### Context
|
|
|
|
AQE system parameters are manually configured:
|
|
- HNSW efSearch (search quality vs speed)
|
|
- Routing confidence threshold
|
|
- Pattern promotion threshold
|
|
- Test complexity limits
|
|
|
|
These should be auto-tuned based on measured performance.
|
|
|
|
### Decision
|
|
|
|
**Implement self-optimization engine that automatically tunes system parameters based on performance metrics.**
|
|
|
|
### Auto-Tunable Parameters
|
|
|
|
```typescript
|
|
const tunableParams: TunableParameter[] = [
|
|
{
|
|
name: 'hnsw.efSearch',
|
|
current: 100,
|
|
min: 50,
|
|
max: 500,
|
|
metric: 'search_latency_ms',
|
|
target: 1, // <1ms
|
|
},
|
|
{
|
|
name: 'routing.confidence_threshold',
|
|
current: 0.7,
|
|
min: 0.5,
|
|
max: 0.95,
|
|
metric: 'routing_accuracy',
|
|
target: 0.9, // 90% accuracy
|
|
},
|
|
{
|
|
name: 'pattern.promotion_threshold',
|
|
current: 3,
|
|
min: 2,
|
|
max: 10,
|
|
metric: 'pattern_quality_long_term',
|
|
target: 0.8,
|
|
},
|
|
{
|
|
name: 'test_gen.complexity_limit',
|
|
current: 'complex',
|
|
options: ['simple', 'medium', 'complex'],
|
|
metric: 'test_maintainability',
|
|
target: 0.7,
|
|
},
|
|
];
|
|
```
|
|
|
|
### QE-Specific Background Workers
|
|
|
|
```typescript
|
|
export const QE_OPTIMIZATION_WORKERS = {
|
|
'pattern-consolidator': {
|
|
interval: 30 * 60 * 1000, // 30 min
|
|
handler: async () => {
|
|
return await qeReasoningBank.consolidate();
|
|
},
|
|
},
|
|
|
|
'coverage-gap-scanner': {
|
|
interval: 60 * 60 * 1000, // 1 hour
|
|
handler: async () => {
|
|
const gaps = await coverageAnalyzer.findGaps();
|
|
const prioritized = await riskAssessor.prioritize(gaps);
|
|
await taskQueue.addMany(prioritized.map(g => ({
|
|
type: 'coverage-improvement',
|
|
target: g.file,
|
|
priority: g.riskScore,
|
|
})));
|
|
return { gapsFound: gaps.length };
|
|
},
|
|
},
|
|
|
|
'flaky-test-detector': {
|
|
interval: 2 * 60 * 60 * 1000, // 2 hours
|
|
handler: async () => {
|
|
const flaky = await testAnalyzer.detectFlakyTests();
|
|
for (const test of flaky) {
|
|
await taskQueue.add({
|
|
type: 'flaky-test-fix',
|
|
testId: test.id,
|
|
flakinessScore: test.score,
|
|
});
|
|
}
|
|
return { flakyTests: flaky.length };
|
|
},
|
|
},
|
|
|
|
'routing-accuracy-monitor': {
|
|
interval: 15 * 60 * 1000, // 15 min
|
|
handler: async () => {
|
|
const stats = await routingFeedback.getStats('1h');
|
|
if (stats.accuracy < 0.8) {
|
|
await qeRouter.retrain();
|
|
}
|
|
return stats;
|
|
},
|
|
},
|
|
};
|
|
```
|
|
|
|
### Implementation (2026-01-09)
|
|
|
|
**Package:** `v3/src/optimization/`
|
|
|
|
**Files Created:**
|
|
```
|
|
v3/src/optimization/
|
|
├── index.ts # Module exports
|
|
├── types.ts # TunableParameter, MetricStats, ParameterApplicator interfaces
|
|
├── metric-collectors.ts # 4 collectors: SearchLatency, RoutingAccuracy, PatternQuality, TestMaintainability
|
|
├── tuning-algorithm.ts # CoordinateDescentTuner (gradient-free optimization)
|
|
├── auto-tuner.ts # AQEAutoTuner orchestrator + ParameterApplicatorRegistry
|
|
└── qe-workers.ts # 4 QE optimization workers
|
|
|
|
v3/tests/unit/optimization/
|
|
├── auto-tuner.test.ts # 37 tests (including applicator registry tests)
|
|
├── metric-collectors.test.ts # 29 tests (including division-by-zero edge cases)
|
|
├── tuning-algorithm.test.ts # 17 tests
|
|
└── qe-workers.test.ts # 20 tests
|
|
```
|
|
|
|
**Key Features Implemented:**
|
|
|
|
1. **4 Default Tunable Parameters:**
|
|
- `hnsw.efSearch` (50-500, target: <1ms latency)
|
|
- `routing.confidenceThreshold` (0.5-0.95, target: 90% accuracy)
|
|
- `pattern.promotionThreshold` (2-10, target: 0.8 quality)
|
|
- `testGen.complexityLimit` (simple/medium/complex, target: 0.7 maintainability)
|
|
|
|
2. **Metric Collectors:**
|
|
- `SearchLatencyCollector` - Tracks HNSW search performance
|
|
- `RoutingAccuracyCollector` - Monitors agent routing success
|
|
- `PatternQualityCollector` - Measures pattern quality scores
|
|
- `TestMaintainabilityCollector` - Tracks test maintainability
|
|
|
|
3. **CoordinateDescentTuner:**
|
|
- Gradient-free optimization (no gradients needed)
|
|
- Exploration vs exploitation balance
|
|
- Handles both numeric and categorical parameters
|
|
|
|
4. **ParameterApplicatorRegistry (Critical Fix):**
|
|
- Bridges tuning suggestions to real system changes
|
|
- `registerApplicator()` for each tunable system component
|
|
- Validation before applying changes
|
|
- Graceful fallback to simulation mode
|
|
|
|
5. **4 QE Optimization Workers:**
|
|
- `pattern-consolidator` (30min) - Merges duplicate patterns
|
|
- `coverage-gap-scanner` (1hr) - Prioritizes test gaps by risk
|
|
- `flaky-test-detector` (2hr) - Identifies unstable tests
|
|
- `routing-accuracy-monitor` (15min, critical) - Monitors router performance
|
|
|
|
**Critical Bug Fixes (via brutal-honesty-review):**
|
|
- Fixed `runEvaluation()` to actually apply configurations via applicators
|
|
- Fixed `recordMetric()` dead code for routing accuracy (now type-safe methods)
|
|
- Fixed division by zero in base class trend calculation
|
|
- Added `ParameterApplicator` interface for real system integration
|
|
|
|
**Usage Example:**
|
|
```typescript
|
|
import { createAutoTuner, DEFAULT_TUNABLE_PARAMETERS } from '@agentic-qe/v3';
|
|
|
|
const tuner = createAutoTuner(DEFAULT_TUNABLE_PARAMETERS, {
|
|
evaluationPeriodMs: 5000,
|
|
tuningIntervalMs: 7 * 24 * 60 * 60 * 1000, // Weekly
|
|
});
|
|
|
|
// Register real system integration
|
|
tuner.registerApplicator({
|
|
parameterName: 'hnsw.efSearch',
|
|
getCurrentValue: async () => hnswIndex.getEfSearch(),
|
|
setValue: async (v) => hnswIndex.setEfSearch(v as number),
|
|
});
|
|
|
|
// Type-safe metric recording
|
|
tuner.recordSearchLatency(5.2);
|
|
tuner.recordRoutingOutcome(true, true);
|
|
tuner.recordPatternQuality(0.85);
|
|
tuner.recordTestMaintainability(0.75);
|
|
|
|
// Start auto-tuning
|
|
tuner.start();
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] 4 auto-tunable parameters (numeric + categorical)
|
|
- [x] Weekly auto-tuning cycles (configurable interval)
|
|
- [x] 4 QE optimization workers implemented
|
|
- [x] Parameter history tracked (evaluationHistory)
|
|
- [x] ParameterApplicatorRegistry for real system integration
|
|
- [x] Type-safe metric recording methods
|
|
- [x] Division-by-zero protection in trend calculation
|
|
- [x] 103 unit tests passing
|
|
- [ ] <500ms worker execution time (needs runtime measurement)
|
|
|
|
---
|
|
|
|
## ADR-025: Enhanced Init with Self-Configuration
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-09
|
|
**Implemented:** 2026-01-10
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** [AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md](../../docs/plans/AQE-SELF-LEARNING-IMPROVEMENT-PLAN.md)
|
|
|
|
### Context
|
|
|
|
Current `aqe init` is manual:
|
|
- User must configure frameworks
|
|
- No analysis of existing codebase
|
|
- No pre-trained patterns loaded
|
|
- No integration with learning system
|
|
|
|
Need intelligent initialization that analyzes the project and configures itself.
|
|
|
|
### Decision
|
|
|
|
**Implement enhanced init with project analysis, self-configuration, and learning hooks integration.**
|
|
|
|
### Implementation (2026-01-10)
|
|
|
|
**Package:** `v3/src/init/`
|
|
|
|
**Files Created:**
|
|
```
|
|
v3/src/init/
|
|
├── index.ts # Module exports
|
|
├── types.ts # All types + defaults (AQEInitConfig, ProjectAnalysis, etc.)
|
|
├── project-analyzer.ts # Framework/language/test detection
|
|
├── self-configurator.ts # Rule-based configuration recommendations
|
|
└── init-wizard.ts # Wizard steps + orchestrator
|
|
|
|
v3/tests/unit/init/
|
|
├── project-analyzer.test.ts # 17 tests
|
|
├── self-configurator.test.ts # 33 tests
|
|
└── init-wizard.test.ts # 23 tests
|
|
```
|
|
|
|
**Key Features Implemented:**
|
|
|
|
1. **ProjectAnalyzer** - Detects project structure automatically:
|
|
- Framework detection: jest, vitest, mocha, jasmine, pytest, playwright, cypress
|
|
- Language detection: TypeScript, JavaScript, Python, Java, Go, Rust
|
|
- Test detection: counts and categorizes existing tests
|
|
- Complexity analysis: file count, LOC, recommendations
|
|
- Coverage parsing: reads existing coverage reports
|
|
- CI detection: GitHub Actions, GitLab CI, CircleCI, Jenkins
|
|
|
|
2. **SelfConfigurator** - Rule-based configuration with 14 rules:
|
|
- `typescript-vitest`: Transformer embeddings + higher confidence
|
|
- `large-codebase`: Larger HNSW index (M=32, efConstruction=400)
|
|
- `small-project`: Smaller settings, fewer workers
|
|
- `high-complexity`: All domains enabled
|
|
- `low-coverage`: Coverage gap scanner enabled
|
|
- `monorepo`: 15 agents, code intelligence domain
|
|
- `has-e2e`: Visual/accessibility domains
|
|
- `has-ci`: CI integration hooks
|
|
- `github-actions`: Claude Code hooks enabled
|
|
- `python-project`: Hybrid routing, security focus
|
|
- `java-project`: Extended timeout (2 min)
|
|
- `no-tests`: Focus on test generation
|
|
- `many-tests`: Flaky detection enabled
|
|
- `security-focus`: Security compliance domain
|
|
|
|
3. **InitOrchestrator** - Multi-step initialization with progress tracking:
|
|
- Step 1: Project Analysis
|
|
- Step 2: Configuration Generation
|
|
- Step 3: Learning System Setup
|
|
- Step 4: Hooks Configuration
|
|
- Step 5: Background Workers Start
|
|
- Step 6: Save Configuration
|
|
|
|
4. **Wizard Steps** - Interactive configuration:
|
|
- Welcome info
|
|
- Project type (auto-detect, single, monorepo, library)
|
|
- Learning mode (full, basic, disabled)
|
|
- Pre-trained patterns (yes/no)
|
|
- Claude Code hooks (yes/no)
|
|
- Background workers (yes/no)
|
|
|
|
### Enhanced Init Wizard
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ AQE v3 Initialization │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ ✓ Project Analysis 50ms │
|
|
│ ✓ Configuration Generation 10ms │
|
|
│ ✓ Learning System Setup 5ms │
|
|
│ ✓ Hooks Configuration 2ms │
|
|
│ ✓ Background Workers 3ms │
|
|
│ ✓ Save Configuration 1ms │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Project: test-project │
|
|
│ Type: single │
|
|
│ Patterns Loaded: 500 │
|
|
│ Workers Started: 4 │
|
|
│ Hooks Configured: Yes │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ ✓ AQE v3 initialized as self-learning platform │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Configuration Schema
|
|
|
|
```typescript
|
|
export interface AQEInitConfig {
|
|
version: string;
|
|
project: {
|
|
name: string;
|
|
root: string;
|
|
type: 'monorepo' | 'single' | 'library';
|
|
};
|
|
learning: {
|
|
enabled: boolean;
|
|
embeddingModel: 'transformer' | 'hash';
|
|
hnswConfig: HNSWConfig;
|
|
qualityThreshold: number;
|
|
promotionThreshold: number;
|
|
pretrainedPatterns: boolean;
|
|
};
|
|
routing: {
|
|
mode: 'ml' | 'rules' | 'hybrid';
|
|
confidenceThreshold: number;
|
|
fallbackEnabled: boolean;
|
|
};
|
|
workers: {
|
|
enabled: string[];
|
|
intervals: Record<string, number>;
|
|
maxConcurrent: number;
|
|
daemonAutoStart: boolean;
|
|
};
|
|
hooks: {
|
|
claudeCode: boolean;
|
|
preCommit: boolean;
|
|
ciIntegration: boolean;
|
|
};
|
|
autoTuning: {
|
|
enabled: boolean;
|
|
parameters: string[];
|
|
evaluationPeriodMs: number;
|
|
};
|
|
domains: {
|
|
enabled: string[];
|
|
disabled: string[];
|
|
};
|
|
agents: {
|
|
maxConcurrent: number;
|
|
defaultTimeout: number;
|
|
};
|
|
}
|
|
```
|
|
|
|
### Usage Examples
|
|
|
|
```typescript
|
|
import {
|
|
createProjectAnalyzer,
|
|
createSelfConfigurator,
|
|
createInitOrchestrator,
|
|
quickInit,
|
|
formatInitResult,
|
|
} from '@agentic-qe/v3';
|
|
|
|
// Quick auto-configuration
|
|
const result = await quickInit('/path/to/project');
|
|
console.log(formatInitResult(result));
|
|
|
|
// Full control with custom options
|
|
const orchestrator = createInitOrchestrator({
|
|
projectRoot: '/path/to/project',
|
|
autoMode: false,
|
|
wizardAnswers: {
|
|
'project-type': 'monorepo',
|
|
'learning-mode': 'full',
|
|
'load-patterns': true,
|
|
'hooks': true,
|
|
'workers': true,
|
|
},
|
|
});
|
|
const result = await orchestrator.initialize();
|
|
|
|
// Analyze project only
|
|
const analyzer = createProjectAnalyzer('/path/to/project');
|
|
const analysis = await analyzer.analyze();
|
|
|
|
// Get configuration recommendations
|
|
const configurator = createSelfConfigurator();
|
|
const config = configurator.recommend(analysis);
|
|
const appliedRules = configurator.getApplicableRules(analysis);
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] Project analysis detects frameworks (7 frameworks)
|
|
- [x] Project analysis detects languages (8 languages)
|
|
- [x] Project analysis detects tests (count, type, framework)
|
|
- [x] Self-configuration with 14 rules
|
|
- [x] Wizard steps with progress tracking
|
|
- [x] Auto-configuration mode (`quickInit()`)
|
|
- [x] Custom wizard answers support
|
|
- [x] Pre-trained pattern library loading
|
|
- [x] Claude Code hooks integration
|
|
- [x] CI provider detection
|
|
- [x] Package manager detection
|
|
- [x] 73 unit tests passing (17 + 33 + 23)
|
|
- [ ] <30 seconds init time (needs runtime measurement)
|
|
- [ ] Migration from v2 automated (deferred)
|
|
|
|
### ADR-025 Addendum: V2 Component Migration Analysis (2026-01-11)
|
|
|
|
**Status:** ✅ IMPLEMENTED (2026-01-11)
|
|
|
|
**Purpose:** Document which V2 init components are needed in V3 vs obsolete.
|
|
|
|
#### V2 Components NOT Needed in V3
|
|
|
|
1. **Slash Commands (`.claude/commands/`)** - OBSOLETE
|
|
- V2: 9 slash commands (`/aqe-execute`, `/aqe-generate`, etc.)
|
|
- V3: Replaced by MCP tools (`mcp__agentic-qe-v3__*`)
|
|
- **Decision:** Do NOT create `.claude/commands/` directory
|
|
|
|
2. **V2 Agent Definitions (`.claude/agents/` flat)** - OBSOLETE
|
|
- V2: 24 flat agent markdown files
|
|
- V3: Replaced by V3 QE agents with DDD domain structure
|
|
- **Decision:** Do NOT copy V2 agents
|
|
|
|
3. **Claude-Flow Core Agents** - NOT COPIED
|
|
- Agents like `adr-architect`, `memory-specialist`, `security-architect`, etc.
|
|
- These are **claude-flow** agents, not AQE agents (use `mcp__claude-flow__*` tools)
|
|
- **Decision:** Do NOT copy - available via claude-flow separately
|
|
|
|
4. **Helper Scripts (`.claude/helpers/`)** - MOSTLY OBSOLETE
|
|
- V2: 38 helper scripts
|
|
- V3: Background workers + MCP tools replace most functionality
|
|
- **Decision:** Keep only `statusline.js` and git hooks (optional)
|
|
|
|
5. **Reference Docs (`.agentic-qe/docs/`)** - OBSOLETE
|
|
- V2: Copied documentation files
|
|
- V3: Skills contain embedded documentation
|
|
- **Decision:** Do NOT create `.agentic-qe/docs/` directory
|
|
|
|
6. **Flat Agent Definitions (`.agentic-qe/agents/`)** - OBSOLETE
|
|
- V2: Agent configuration in separate directory
|
|
- V3: DDD domains + MCP tools replace
|
|
- **Decision:** Do NOT create `.agentic-qe/agents/` directory
|
|
|
|
#### V3 Components REQUIRED (Implemented)
|
|
|
|
1. **V3 QE Agents (`.claude/agents/v3/`)** - ✅ IMPLEMENTED
|
|
- 40 V3 QE domain agents (v3-qe-*) mapped to 12 DDD contexts
|
|
- 7 V3 QE subagents for TDD and code review
|
|
- **Total:** 47 QE agents installed via `AgentsInstaller`
|
|
|
|
2. **Pattern Database (`.agentic-qe/data/patterns.db`)** - PLANNED
|
|
- Required for: Learning system, pattern storage, HNSW indexing
|
|
- **Action:** Add database initialization to init
|
|
|
|
3. **Project Configuration (`CLAUDE.md`)** - PLANNED
|
|
- Required for: Project-specific V3 configuration, quick reference
|
|
- **Action:** Add V3-specific CLAUDE.md generation
|
|
|
|
4. **MCP Configuration (`.claude/mcp.json`)** - ✅ IMPLEMENTED
|
|
- Required for: MCP server definition, tool endpoint configuration
|
|
- Configures `agentic-qe-v3` MCP server with `npx @agentic-qe/v3 mcp`
|
|
|
|
#### V3 Init Comparison
|
|
|
|
| Metric | V2 | V3 | Reason |
|
|
|--------|----|----|--------|
|
|
| Files created | ~100+ | ~60 | MCP tools replace commands |
|
|
| Directories | 8 | 6 | DDD consolidates structure |
|
|
| Helper scripts | 38 | 2-3 | Workers replace helpers |
|
|
| V2 Agent defs | 24 | 0 | Replaced by V3 QE agents |
|
|
| V3 QE Agents | 0 | 47 | v3-qe-* agents for Task tool |
|
|
| Commands | 9 | 0 | MCP tools replace |
|
|
|
|
#### V3 Init File Structure (Final)
|
|
|
|
```
|
|
Project Root
|
|
├── CLAUDE.md # V3 project configuration [PLANNED]
|
|
├── .agentic-qe/
|
|
│ ├── config.yaml # ✅ Implemented
|
|
│ ├── data/
|
|
│ │ ├── patterns.db # [PLANNED] Pattern database
|
|
│ │ ├── learning-config.json # ✅ Implemented
|
|
│ │ └── hnsw/ # ✅ Implemented
|
|
│ └── workers/
|
|
│ ├── registry.json # ✅ Implemented
|
|
│ ├── *.json # ✅ Implemented
|
|
│ └── start-daemon.sh # ✅ Implemented
|
|
└── .claude/
|
|
├── settings.json # ✅ Implemented
|
|
├── mcp.json # ✅ Implemented (MCP server config)
|
|
├── skills/ # ✅ Implemented (SkillsInstaller)
|
|
│ ├── README.md
|
|
│ └── [skill-dirs]/
|
|
└── agents/v3/ # ✅ Implemented (AgentsInstaller)
|
|
├── README.md # Agent index
|
|
├── v3-qe-*.md # 40 QE domain agents
|
|
└── subagents/ # 7 QE subagents
|
|
└── v3-qe-*.md
|
|
```
|
|
|
|
---
|
|
|
|
## ADR-036: Language-Aware Result Persistence
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-01-10
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** MCP v3 Task Execution Pipeline
|
|
|
|
### Context
|
|
|
|
The MCP v3 task executor now returns real results from domain services, but these results are only returned in the API response and not persisted for later analysis. Users need to:
|
|
|
|
1. Review results after task completion
|
|
2. Compare results across multiple runs
|
|
3. Track quality trends over time
|
|
4. Generate reports for stakeholders
|
|
5. Use generated tests directly in their codebase
|
|
|
|
Current limitations:
|
|
- Results lost after API response
|
|
- No historical tracking
|
|
- Cannot diff between runs
|
|
- Generated tests not saved as usable files
|
|
|
|
### Decision
|
|
|
|
**Implement language/framework-aware result persistence that saves outputs in appropriate formats based on task type and target stack.**
|
|
|
|
### Output Format Matrix
|
|
|
|
| Task Type | Primary Format | Secondary Format | Extension Pattern |
|
|
|-----------|---------------|------------------|-------------------|
|
|
| Test Generation | Source Code | JSON manifest | Language-specific |
|
|
| Coverage Analysis | LCOV | JSON + HTML | `.lcov`, `.json`, `.html` |
|
|
| Security Scan | SARIF | JSON + MD report | `.sarif`, `.json`, `.md` |
|
|
| Quality Assessment | JSON | MD report | `.json`, `.md` |
|
|
| Code Indexing | JSON graph | GraphML | `.json`, `.graphml` |
|
|
| Defect Prediction | JSON | MD report | `.json`, `.md` |
|
|
| Contract Testing | JSON | OpenAPI diff | `.json`, `.yaml` |
|
|
| Accessibility | JSON | HTML report | `.json`, `.html` |
|
|
| Chaos/Load Test | JSON | HTML dashboard | `.json`, `.html` |
|
|
|
|
### Test File Extensions by Language/Framework
|
|
|
|
```typescript
|
|
const TEST_FILE_PATTERNS: Record<string, Record<string, string>> = {
|
|
typescript: { jest: '.test.ts', vitest: '.test.ts', mocha: '.spec.ts' },
|
|
javascript: { jest: '.test.js', vitest: '.test.js', mocha: '.spec.js' },
|
|
python: { pytest: 'test_*.py', unittest: '*_test.py' },
|
|
java: { junit: '*Test.java', testng: '*Test.java' },
|
|
go: { testing: '*_test.go' },
|
|
rust: { cargo: '*_test.rs' },
|
|
ruby: { rspec: '*_spec.rb', minitest: '*_test.rb' },
|
|
php: { phpunit: '*Test.php', pest: '*.test.php' },
|
|
csharp: { xunit: '*Tests.cs', nunit: '*Tests.cs' },
|
|
kotlin: { junit: '*Test.kt', kotest: '*Spec.kt' },
|
|
swift: { xctest: '*Tests.swift' },
|
|
};
|
|
```
|
|
|
|
### Directory Structure
|
|
|
|
```
|
|
.agentic-qe/
|
|
├── results/
|
|
│ ├── security/
|
|
│ │ ├── 2026-01-10T15-30-00_scan.sarif
|
|
│ │ ├── 2026-01-10T15-30-00_scan.json
|
|
│ │ └── 2026-01-10T15-30-00_report.md
|
|
│ ├── coverage/
|
|
│ │ ├── 2026-01-10T15-30-00_coverage.lcov
|
|
│ │ └── 2026-01-10T15-30-00_gaps.md
|
|
│ ├── tests/
|
|
│ │ ├── generated/
|
|
│ │ │ ├── user-service.test.ts
|
|
│ │ │ └── test_auth_module.py
|
|
│ │ └── manifest.json
|
|
│ └── index.json # Index of all results
|
|
```
|
|
|
|
### Implementation (2026-01-10) ✅ COMPLETE
|
|
|
|
**Files Created:**
|
|
- `v3/src/coordination/result-saver.ts` - Core ResultSaver class (780 LOC)
|
|
- `v3/src/coordination/task-executor.ts` - TaskExecutor with ResultSaver integration
|
|
|
|
**Implemented Features:**
|
|
|
|
| Component | Status | Details |
|
|
|-----------|--------|---------|
|
|
| ResultSaver class | ✅ | 780 lines, full implementation |
|
|
| Task-type savers | ✅ | 10 savers (test-gen, coverage, security, quality, code-index, defect, contract, a11y, chaos, generic) |
|
|
| LCOV generator | ✅ | `generateLcov()` for coverage reports |
|
|
| SARIF generator | ✅ | `generateSarif()` for security findings |
|
|
| MD report generators | ✅ | Test, coverage, security, quality reports |
|
|
| Language patterns | ✅ | 11 languages (TS, JS, Python, Java, Go, Rust, Ruby, PHP, C#, Kotlin, Swift) |
|
|
| Framework patterns | ✅ | jest, vitest, mocha, pytest, junit, testng, rspec, phpunit, xunit, kotest, xctest |
|
|
| Result index | ✅ | `updateIndex()` with trend tracking |
|
|
| TaskExecutor integration | ✅ | Auto-saves results after task execution |
|
|
|
|
**Key Methods:**
|
|
```typescript
|
|
class ResultSaver {
|
|
save(taskId, taskType, result, options): Promise<SavedResult>
|
|
// 10 private task-type handlers
|
|
generateLcov(data): string
|
|
generateSarif(data): string
|
|
generateTestReport(data): string
|
|
updateIndex(savedResult): Promise<void>
|
|
}
|
|
```
|
|
|
|
### Success Metrics
|
|
|
|
- [x] Results persist in standard formats (SARIF, LCOV, JSON)
|
|
- [x] Generated tests saved as ready-to-use source files
|
|
- [x] Language/framework conventions respected (11 languages, 11+ frameworks)
|
|
- [x] Result index with trend tracking
|
|
- [ ] Result comparison/diff (future enhancement)
|
|
- [ ] 30-day retention with compression (future enhancement)
|
|
|
|
---
|
|
|
|
## Implementation Roadmap Summary
|
|
|
|
### Sprint 1 (Days 1-5): Foundation - ADR-021
|
|
- Create `v3/@aqe-platform/learning` package
|
|
- Implement QEReasoningBank
|
|
- Define QE pattern types
|
|
|
|
### Sprint 2 (Days 6-10): Routing - ADR-022
|
|
- Create `v3/@aqe-platform/routing` package
|
|
- Build QE agent registry (78 agents)
|
|
- Implement ML-based task router
|
|
|
|
### Sprint 3 (Days 11-15): Feedback - ADR-023
|
|
- Create `v3/@aqe-platform/feedback` package
|
|
- Implement TestOutcomeTracker
|
|
- Build CoverageLearner
|
|
|
|
### Sprint 4 (Days 16-20): Optimization - ADR-024
|
|
- Create `v3/@aqe-platform/optimization` package
|
|
- Implement AQEAutoTuner
|
|
- Create QE optimization workers
|
|
|
|
### Sprint 5 (Days 21-25): Init - ADR-025
|
|
- Enhance `aqe init --wizard`
|
|
- Implement self-configuration
|
|
- Create pre-trained pattern library
|
|
|
|
### Sprint 6 (Days 26-30): Integration
|
|
- Integration tests
|
|
- Performance benchmarks
|
|
- Documentation
|
|
|
|
---
|
|
|
|
---
|
|
|
|
## ADR-037: V3 QE Agent Naming Standardization
|
|
|
|
**Status:** Proposed
|
|
**Date:** 2026-01-11
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** V3 Skills Improvement Analysis
|
|
|
|
### Context
|
|
|
|
V3 QE skills currently reference agents using V2-style short names (e.g., `'test-architect'`, `'coverage-specialist'`) while the actual V3 QE agent definitions use the full `v3-qe-*` prefix (e.g., `'v3-qe-test-architect'`, `'v3-qe-coverage-specialist'`).
|
|
|
|
This inconsistency causes:
|
|
1. Confusion about which agent version is being used
|
|
2. Potential routing errors in the Task tool
|
|
3. Documentation mismatches
|
|
4. Difficulty in agent discovery
|
|
|
|
### Decision
|
|
|
|
Standardize all V3 QE skills to use the full `v3-qe-*` agent naming convention.
|
|
|
|
**Naming Convention:** `v3-qe-{domain}-{specialty}`
|
|
|
|
**Files Updated:**
|
|
- `.claude/skills/v3-qe-fleet-coordination/SKILL.md`
|
|
- `.claude/skills/v3-qe-mcp/SKILL.md`
|
|
- `.claude/skills/v3-qe-integration/SKILL.md`
|
|
|
|
**Full ADR:** [ADR-037-v3-qe-agent-naming.md](./ADR-037-v3-qe-agent-naming.md)
|
|
|
|
---
|
|
|
|
## ADR-038: V3 QE Memory System Unification
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-11 (Proposed) / 2026-02-01 (Implemented)
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** V3 Skills Improvement Analysis
|
|
**Implemented In:** v3.4.0 - All 12 DDD domains now enabled by default with unified memory.db
|
|
|
|
### Context
|
|
|
|
The existing `v3-qe-memory-system` skill provides basic AgentDB integration, but lacks the comprehensive unification approach from `v3-memory-unification` in claude-flow.
|
|
|
|
### Decision
|
|
|
|
Create enhanced `v3-qe-memory-unification` skill with:
|
|
- AgentDB with HNSW indexing (150x-12,500x faster search)
|
|
- Migration from legacy QE memory systems (SQLite, markdown, in-memory)
|
|
- Cross-domain memory sharing for 12 DDD domains
|
|
- SONA integration for pattern learning
|
|
- Domain-specific HNSW configurations
|
|
|
|
**New Skill:** `.claude/skills/v3-qe-memory-unification/SKILL.md`
|
|
|
|
**Full ADR:** [ADR-038-v3-qe-memory-unification.md](./ADR-038-v3-qe-memory-unification.md)
|
|
|
|
---
|
|
|
|
## ADR-039: V3 QE MCP Optimization
|
|
|
|
**Status:** Proposed
|
|
**Date:** 2026-01-11
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** V3 Skills Improvement Analysis
|
|
|
|
### Context
|
|
|
|
The existing `v3-qe-mcp` skill provides basic MCP tool definitions but lacks the performance optimizations from `v3-mcp-optimization` in claude-flow.
|
|
|
|
### Decision
|
|
|
|
Create enhanced `v3-qe-mcp-optimization` skill with:
|
|
- Connection pooling (50 max, 5 min pre-warmed)
|
|
- O(1) tool lookup via hash-indexed registry
|
|
- Load balancing for fleet operations
|
|
- <100ms p95 response time targets
|
|
- Performance monitoring dashboard
|
|
|
|
**New Skill:** `.claude/skills/v3-qe-mcp-optimization/SKILL.md`
|
|
|
|
**Full ADR:** [ADR-039-v3-qe-mcp-optimization.md](./ADR-039-v3-qe-mcp-optimization.md)
|
|
|
|
---
|
|
|
|
## ADR-040: V3 QE Agentic-Flow Integration
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-11 (Proposed) / 2026-02-01 (Implemented)
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** V3 Skills Improvement Analysis
|
|
**Implemented In:** v3.4.0 - AG-UI, A2A, and A2UI protocol implementations
|
|
|
|
### Context
|
|
|
|
The existing `v3-qe-integration` skill provides basic cross-domain integration but lacks deep agentic-flow integration patterns from `v3-integration-deep` in claude-flow.
|
|
|
|
### Decision
|
|
|
|
Create enhanced `v3-qe-agentic-flow-integration` skill with:
|
|
- SONA learning mode integration (<0.05ms adaptation)
|
|
- Flash Attention (2.49x-7.47x speedup) for QE workloads
|
|
- 9 RL algorithms for intelligent QE decisions
|
|
- Code deduplication with agentic-flow
|
|
|
|
**New Skill:** `.claude/skills/v3-qe-agentic-flow-integration/SKILL.md`
|
|
|
|
**Full ADR:** [ADR-040-v3-qe-agentic-flow-integration.md](./ADR-040-v3-qe-agentic-flow-integration.md)
|
|
|
|
---
|
|
|
|
## ADR-041: V3 QE CLI Enhancement
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-11 (Proposed) / 2026-01-14 (Implemented)
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** V3 Skills Improvement Analysis
|
|
|
|
### Context
|
|
|
|
The existing `v3-qe-cli` skill provides basic CLI commands for QE operations but lacks modern CLI features from `v3-cli-modernization` in claude-flow.
|
|
|
|
### Decision
|
|
|
|
Enhance `v3-qe-cli` with:
|
|
- Interactive test generation wizard
|
|
- Progress bars for fleet operations
|
|
- Workflow automation for QE pipelines
|
|
- Intelligent command completion
|
|
- Streaming output for test execution
|
|
|
|
### Implementation Status (2026-01-14)
|
|
|
|
#### Fully Implemented (100%)
|
|
|
|
| Component | Description | Lines | Tests |
|
|
|-----------|-------------|-------|-------|
|
|
| **Interactive Wizards** | All 4 wizards (test, coverage, security, fleet) | 618-757 each | 61+ |
|
|
| **Config System** | Validation, caching, persistence to ~/.aqe-v3/cli-config.json | Complete | 75 |
|
|
| **Streaming Output** | TestResultStreamer, CoverageStreamer, AgentActivityStreamer, UnifiedStreamer | Complete | Yes |
|
|
| **Persistent Scheduler** | File-based with locking, backup, corruption recovery | 528 lines | 26 |
|
|
|
|
#### Integrated (Working)
|
|
|
|
- Test Generation Wizard (`--wizard` flag on `tests generate`)
|
|
- Coverage Analysis Wizard (`--wizard` flag on `coverage analyze`)
|
|
- Security Scan Wizard (`--wizard` flag on `security scan`)
|
|
- Streaming output (`--stream` flag on test/coverage commands)
|
|
- Config integration in progress.ts and streaming.ts
|
|
|
|
#### All Integrations Complete ✅
|
|
|
|
| Component | Status | CLI Command |
|
|
|-----------|--------|-------------|
|
|
| Fleet Init Wizard | ✅ Integrated | `fleet init --wizard` (lines 2764-2884) |
|
|
| Completions Command | ✅ Integrated | `completions bash\|zsh\|fish\|powershell` (lines 2655-2753) |
|
|
| Workflow Commands | ✅ Integrated | `workflow run\|validate\|schedule\|list` (lines 952-1501) |
|
|
| FleetProgressManager | ✅ Available | `progress.ts` with multi-bar support |
|
|
|
|
#### Security Fixes Applied (2026-01-14)
|
|
|
|
- Path traversal protection in test-wizard.ts
|
|
- File size limits in YAML/JSON parsing
|
|
- Prototype pollution protection in config deepMerge
|
|
|
|
#### Test Coverage
|
|
|
|
| Suite | Tests |
|
|
|-------|-------|
|
|
| Scheduler | 26 |
|
|
| Config | 75 |
|
|
| Wizards | 61+ |
|
|
| **Total ADR-041** | 196+ |
|
|
| **Overall v3** | 4032 |
|
|
|
|
#### Files Added for ADR-041
|
|
|
|
```
|
|
src/cli/wizards/
|
|
├── test-wizard.ts (618 lines)
|
|
├── coverage-wizard.ts (631 lines)
|
|
├── fleet-wizard.ts (709 lines)
|
|
└── security-wizard.ts (757 lines)
|
|
|
|
src/cli/utils/
|
|
├── progress.ts (multi-bar progress)
|
|
├── streaming.ts (streaming output)
|
|
└── workflow-parser.ts (990 lines)
|
|
|
|
src/cli/scheduler/
|
|
└── persistent-scheduler.ts (528 lines)
|
|
|
|
src/cli/completions/
|
|
└── index.ts (shell completions)
|
|
|
|
src/cli/config/
|
|
└── cli-config.ts (config system)
|
|
```
|
|
|
|
**Enhanced Skill:** `.claude/skills/v3-qe-cli/SKILL.md` (updated)
|
|
|
|
**Full ADR:** [ADR-041-v3-qe-cli-enhancement.md](./ADR-041-v3-qe-cli-enhancement.md)
|
|
|
|
---
|
|
|
|
## Implementation Roadmap Summary
|
|
|
|
### Sprint 1 (Days 1-5): Foundation - ADR-021
|
|
- Create `v3/@aqe-platform/learning` package
|
|
- Implement QEReasoningBank
|
|
- Define QE pattern types
|
|
|
|
### Sprint 2 (Days 6-10): Routing - ADR-022
|
|
- Create `v3/@aqe-platform/routing` package
|
|
- Build QE agent registry (78 agents)
|
|
- Implement ML-based task router
|
|
|
|
### Sprint 3 (Days 11-15): Feedback - ADR-023
|
|
- Create `v3/@aqe-platform/feedback` package
|
|
- Implement TestOutcomeTracker
|
|
- Build CoverageLearner
|
|
|
|
### Sprint 4 (Days 16-20): Optimization - ADR-024
|
|
- Create `v3/@aqe-platform/optimization` package
|
|
- Implement AQEAutoTuner
|
|
- Create QE optimization workers
|
|
|
|
### Sprint 5 (Days 21-25): Init - ADR-025
|
|
- Enhance `aqe init --wizard`
|
|
- Implement self-configuration
|
|
- Create pre-trained pattern library
|
|
|
|
### Sprint 6 (Days 26-30): Integration
|
|
- Integration tests
|
|
- Performance benchmarks
|
|
- Documentation
|
|
|
|
### Sprint 7 (Current): V3 Skills Improvement - ADR-037 to ADR-041
|
|
- V3 agent naming standardization
|
|
- Memory unification with HNSW
|
|
- MCP optimization with pooling
|
|
- Agentic-flow deep integration
|
|
- CLI enhancement
|
|
|
|
---
|
|
|
|
## ADR-042: V3 QE Token Tracking Integration
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-01-14
|
|
**Decision Makers:** Architecture Team
|
|
**Source:** Analysis of [agentic-flow](https://github.com/ruvnet/agentic-flow) token tracking capabilities
|
|
|
|
### Context
|
|
|
|
V3 QE agents lack token consumption tracking and optimization. Analysis of agentic-flow revealed proven mechanisms for -25% token reduction via pattern reuse, caching, and early-exit optimizations.
|
|
|
|
### Decision
|
|
|
|
Implement comprehensive token tracking and consumption reduction:
|
|
- Token tracking per task/agent/domain
|
|
- Pattern reuse for -25% token reduction
|
|
- MCP tool `token_usage` for analysis
|
|
- Early-exit optimizer for high-confidence patterns
|
|
|
|
### Implementation (2026-01-14)
|
|
|
|
- **TokenMetricsCollector**: Singleton for tracking token usage across agents/domains
|
|
- **TokenOptimizerService**: Wires EarlyExitTokenOptimizer into execution flow
|
|
- **token-bootstrap.ts**: Initializes optimization on MCP/CLI startup
|
|
- **CLI commands**: `aqe token-usage --by-agent/--by-domain/--recommendations`
|
|
- **Persistence**: JSON file persistence with auto-save
|
|
- **Integration tests**: 29 tests covering real LLM provider flows
|
|
- **LLM providers**: Claude, OpenAI, Ollama all report token usage
|
|
|
|
**Full ADR:** [ADR-042-v3-qe-token-tracking-integration.md](./ADR-042-v3-qe-token-tracking-integration.md)
|
|
|
|
---
|
|
|
|
**Document Maintained By:** Architecture Team
|
|
**Last Updated:** 2026-01-14 (ADR-042 Token Tracking Implementation)
|
|
**Next Review:** After V3 Skills Implementation Complete
|
|
|
|
### Current Implementation Stats (2026-01-14)
|
|
|
|
| Metric | Count |
|
|
|--------|-------|
|
|
| Source Files | 320+ |
|
|
| Test Files | 146+ |
|
|
| Tests Passing | 4,203 |
|
|
| ADRs Complete | 42/43 (ADR-001-042 Implemented, ADR-043 proposed) |
|
|
| RuVector MinCut ADRs | 6 (ADR-030 to ADR-035) |
|
|
| Result Persistence ADR | 1 (ADR-036) |
|
|
| V3 Skills Improvement ADRs | 6 implemented (ADR-037 to ADR-042), 1 proposed (ADR-043) |
|
|
| V3 QE Skills | 24 (21 existing + 3 new enhanced) |
|
|
| Init Module Tests | 73 |
|
|
| Feedback Module Tests | 101 |
|
|
| Optimization Module Tests | 103 |
|
|
| CLI Enhancement Tests (ADR-041) | 196 |
|