mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
3bc09cda6d
A post-Phase-4 devil's-advocate audit identified 10 issues. All addressed
with code fixes + tests. Some were real bugs (silent billing regression,
2 unwired MCP tools, 2 HybridRouter routing bugs); others were robustness
improvements (split singletons, silent errors, decorative type fields).
Findings → fixes (severity tagged in audit report):
1. CRITICAL — silent billing regression on upgrade
Users with ANTHROPIC_API_KEY in env would silently start paying for
Claude API calls on every Fleet operation. The CHANGELOG's "opt out
via code changes" wasn't sufficient — needed an env-only escape.
Fix: AQE_LLM_ROUTER_DISABLED=1 env kill-switch honored in both
QEKernelImpl._initializeLLMRouter and src/mcp/tools/base.ts
getSharedLLMRouter. Truthy values disable; 'false'/'0'/'no'/'off'/''
pass. Unified parse rule between kernel and MCP base.
2. HIGH — 2 unwired MCP standalone tools (audit said 3; QualityCriteria
isn't LLM-aware so excluded)
src/mcp/tools/test-generation/generate.ts:82 was calling
createTestGeneratorService(memory) without router. Same for
visual-accessibility/index.ts:177. Both now use the
createXxxServiceWithDependencies factories with { memory, llmRouter }.
3. CRITICAL — E2E only tested manual routing mode, not production
Production default is rule-based. Adding two new E2E tests for
rule-based mode SURFACED two real bugs in HybridRouter that I fixed:
3a. executeWithFallback hardcoded ['claude','openai','ollama'] as the
only fallback providers; gemini/openrouter/azure-openai/bedrock
users had no fallback even though ProviderManager can construct
them. Plus the user's defaultProvider wasn't auto-included in the
chain. Both fixed at src/shared/llm/router/hybrid-router.ts:612.
3b. A non-retryable error from one provider (e.g. Claude "invalid
x-api-key") aborted the entire fallback loop via `throw error`,
defeating the whole point of a fallback chain. Fixed: skip the
inter-attempt delay for non-retryable but continue to the next
fallback entry.
4. HIGH — MCPToolContext.llmRouter field was decorative
Added the field but no code populated it. Every getLLMRouter(context)
call fell through to getSharedLLMRouter().
Fix: added llmRouter and memory to BaseMCPTool.invoke() options;
qe-tool-bridge.ts and registry.ts now explicitly forward the shared
singleton via { llmRouter } at invoke time.
5. HIGH — 14 of 15 wired services had only STRUCTURAL verification, not
behavioral (only TestExecutor had a chat() round-trip test)
Fix: new tests/integration/llm-router-behavioral.test.ts asserts
isLLMAnalysisAvailable()===true (the exact gate the audit named) for
all 14 services + 2 full behavioral round-trips. 16 tests total.
6. HIGH — MCP and kernel built SEPARATE HybridRouter instances
Different cost trackers, caches, circuit breakers per side.
routing_economics/routing_metrics would only see half the traffic.
Fix: kernel.ts._initializeLLMRouter calls setSharedLLMRouter(router)
on init so both paths share one instance. kernel.dispose() calls
resetSharedLLMRouter() so the next kernel doesn't inherit the
previous instance.
7. HIGH — Silent error swallowing in _initializeLLMRouter
console.warn'd then continued without LLM, with no signal for
monitoring. A misconfigured provider would silently degrade with no
alert.
Fix: now publishes kernel.llm-router.init-failed (caught error) and
kernel.llm-router.init-no-provider (enabled=true but nothing
available) events on the event bus. Test confirms event delivery.
8. MEDIUM — Service count mismatch in docs (ADR/CHANGELOG said "14
services / 12 domains"; test enumerates 15 services / 11
LLM-enhanced domains)
Fix: ADR-043 addendum + CHANGELOG corrected to 15/11. Added
explicit service list and noted the 2 non-LLM domains
(enterprise-integration, coordination).
9. MEDIUM — aqe llm config --set apiKey=... silently dropped the key
stripApiKeys() returned a successfully-saved file without the key,
user thought it was saved.
Fix: CLI updateConfig() now throws a clear error pointing to env
vars BEFORE calling save. stripApiKeys also now logs a console.warn
when it strips anything, so programmatic callers see the strip.
10. LOW — Env vs disk precedence was counterintuitive and undocumented
Force-enable-on-env overrides explicit `enabled: false` on disk.
Fix: doc comment in config-store.applyEnvProviderDetection +
ADR-043 "Precedence & env-var conventions" section explains.
Verification
- npx tsc --noEmit: clean
- 191 unit + integration tests pass (16 new behavioral tests for #5,
17 wiring tests, gemini provider, config-store, llm-router-service,
kernel-llm-router with 17 tests including all kill-switch cases,
CLI persistence with 28 tests)
- 5 real-provider E2E tests pass (Gemini 2.5 Flash, OpenAI gpt-4o-mini,
OpenRouter, plus 2 rule-based-mode tests that previously failed and
now pass with the HybridRouter fallback fixes)
Refs ADR-043 audit findings (2026-05-21)
Builds on 94033e3 (Phase 1), be1e60d (Phase 2), c206c81 (Phase 3+4)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Agentic QE v3 Implementation Documentation
This directory contains all implementation documentation, planning, and research for Agentic QE v3.
Directory Structure
implementation/
├── adrs/ # Architecture Decision Records
├── architecture/ # Architecture diagrams & analysis
├── planning/ # Master plans & implementation strategies
├── security/ # Security considerations
└── agents/ # Agent specifications
Document Index
ADRs (Architecture Decision Records)
| Document | Description |
|---|---|
| v3-adrs.md | All 10 ADRs (001-010) master document |
Planning
| Document | Description |
|---|---|
| AQE-V3-MASTER-PLAN.md | Complete v3 master plan with DDD architecture |
Quick Links
- Skills: /.claude/skills/v3-qe-ddd-architecture/
- Agents: /.claude/agents/v3/
- Source (future): /v3/src/ - v3 source code (to be implemented)
Key Concepts
Domain-Driven Design for QE
Agentic QE v3 uses DDD with 6 bounded contexts:
- Test Generation - AI-powered test creation
- Test Execution - Parallel execution and retry
- Coverage Analysis - O(log n) gap detection
- Quality Assessment - Quality gates and metrics
- Defect Intelligence - Prediction and analysis
- Learning Optimization - Pattern learning and transfer
Agent Hierarchy
21 specialized v3 agents organized hierarchically:
- Queen Coordinator (orchestration)
- Test Generation Group (4 agents)
- Quality Gates Group (3 agents)
- Intelligence Group (3 agents)
- Execution Group (3 agents)
- Coverage Group (3 agents)
- Learning Group (4 agents)
Key ADRs
| ADR | Decision |
|---|---|
| ADR-001 | Adopt DDD for QE bounded contexts |
| ADR-002 | Event-driven domain communication |
| ADR-003 | Sublinear O(log n) coverage analysis |
| ADR-004 | Plugin architecture for extensions |
| ADR-005 | AI-first test generation |
Getting Started
# Use v3 DDD skill
Task("Analyze QE architecture",
"Design DDD bounded contexts for quality engineering",
"system-architect")
# Use v3 Queen Coordinator
Task("Orchestrate QE workflow",
"Coordinate full quality engineering workflow",
"v3-qe-queen-coordinator")
# Use v3 Test Architect
Task("Generate tests",
"Create AI-powered test suite for UserService",
"v3-qe-test-architect")
Last Updated: 2026-01-07 Version: 1.0.0