mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
working-june
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dragan Spiridonov
|
3bc09cda6d |
fix(llm-router): address 10 findings from qe-devils-advocate audit
A post-Phase-4 devil's-advocate audit identified 10 issues. All addressed
with code fixes + tests. Some were real bugs (silent billing regression,
2 unwired MCP tools, 2 HybridRouter routing bugs); others were robustness
improvements (split singletons, silent errors, decorative type fields).
Findings → fixes (severity tagged in audit report):
1. CRITICAL — silent billing regression on upgrade
Users with ANTHROPIC_API_KEY in env would silently start paying for
Claude API calls on every Fleet operation. The CHANGELOG's "opt out
via code changes" wasn't sufficient — needed an env-only escape.
Fix: AQE_LLM_ROUTER_DISABLED=1 env kill-switch honored in both
QEKernelImpl._initializeLLMRouter and src/mcp/tools/base.ts
getSharedLLMRouter. Truthy values disable; 'false'/'0'/'no'/'off'/''
pass. Unified parse rule between kernel and MCP base.
2. HIGH — 2 unwired MCP standalone tools (audit said 3; QualityCriteria
isn't LLM-aware so excluded)
src/mcp/tools/test-generation/generate.ts:82 was calling
createTestGeneratorService(memory) without router. Same for
visual-accessibility/index.ts:177. Both now use the
createXxxServiceWithDependencies factories with { memory, llmRouter }.
3. CRITICAL — E2E only tested manual routing mode, not production
Production default is rule-based. Adding two new E2E tests for
rule-based mode SURFACED two real bugs in HybridRouter that I fixed:
3a. executeWithFallback hardcoded ['claude','openai','ollama'] as the
only fallback providers; gemini/openrouter/azure-openai/bedrock
users had no fallback even though ProviderManager can construct
them. Plus the user's defaultProvider wasn't auto-included in the
chain. Both fixed at src/shared/llm/router/hybrid-router.ts:612.
3b. A non-retryable error from one provider (e.g. Claude "invalid
x-api-key") aborted the entire fallback loop via `throw error`,
defeating the whole point of a fallback chain. Fixed: skip the
inter-attempt delay for non-retryable but continue to the next
fallback entry.
4. HIGH — MCPToolContext.llmRouter field was decorative
Added the field but no code populated it. Every getLLMRouter(context)
call fell through to getSharedLLMRouter().
Fix: added llmRouter and memory to BaseMCPTool.invoke() options;
qe-tool-bridge.ts and registry.ts now explicitly forward the shared
singleton via { llmRouter } at invoke time.
5. HIGH — 14 of 15 wired services had only STRUCTURAL verification, not
behavioral (only TestExecutor had a chat() round-trip test)
Fix: new tests/integration/llm-router-behavioral.test.ts asserts
isLLMAnalysisAvailable()===true (the exact gate the audit named) for
all 14 services + 2 full behavioral round-trips. 16 tests total.
6. HIGH — MCP and kernel built SEPARATE HybridRouter instances
Different cost trackers, caches, circuit breakers per side.
routing_economics/routing_metrics would only see half the traffic.
Fix: kernel.ts._initializeLLMRouter calls setSharedLLMRouter(router)
on init so both paths share one instance. kernel.dispose() calls
resetSharedLLMRouter() so the next kernel doesn't inherit the
previous instance.
7. HIGH — Silent error swallowing in _initializeLLMRouter
console.warn'd then continued without LLM, with no signal for
monitoring. A misconfigured provider would silently degrade with no
alert.
Fix: now publishes kernel.llm-router.init-failed (caught error) and
kernel.llm-router.init-no-provider (enabled=true but nothing
available) events on the event bus. Test confirms event delivery.
8. MEDIUM — Service count mismatch in docs (ADR/CHANGELOG said "14
services / 12 domains"; test enumerates 15 services / 11
LLM-enhanced domains)
Fix: ADR-043 addendum + CHANGELOG corrected to 15/11. Added
explicit service list and noted the 2 non-LLM domains
(enterprise-integration, coordination).
9. MEDIUM — aqe llm config --set apiKey=... silently dropped the key
stripApiKeys() returned a successfully-saved file without the key,
user thought it was saved.
Fix: CLI updateConfig() now throws a clear error pointing to env
vars BEFORE calling save. stripApiKeys also now logs a console.warn
when it strips anything, so programmatic callers see the strip.
10. LOW — Env vs disk precedence was counterintuitive and undocumented
Force-enable-on-env overrides explicit `enabled: false` on disk.
Fix: doc comment in config-store.applyEnvProviderDetection +
ADR-043 "Precedence & env-var conventions" section explains.
Verification
- npx tsc --noEmit: clean
- 191 unit + integration tests pass (16 new behavioral tests for #5,
17 wiring tests, gemini provider, config-store, llm-router-service,
kernel-llm-router with 17 tests including all kill-switch cases,
CLI persistence with 28 tests)
- 5 real-provider E2E tests pass (Gemini 2.5 Flash, OpenAI gpt-4o-mini,
OpenRouter, plus 2 rule-based-mode tests that previously failed and
now pass with the HybridRouter fallback fixes)
Refs ADR-043 audit findings (2026-05-21)
Builds on
|
||
|
Dragan Spiridonov
|
c206c81751 |
feat(llm-router): Phases 3 + 4 — MCP path, integration tests, real-provider E2E, ADRs
Phase 3 — MCP tool path
- 8 standalone MCP tools (coverage-analyze, coverage-gaps, contract-validate,
chaos-inject, code-intel-analyze, defect-predict, learning-optimize) now
resolve the kernel HybridRouter via getLLMRouter(context) and pass it to
their service via the dependency-bag form.
- src/mcp/tools/base.ts: added getSharedLLMRouter() singleton, mirroring the
existing getSharedMemoryBackend() pattern. resetSharedLLMRouter() for
tests. Added MCPToolContext.llmRouter field so future tools (and the MCP
runtime when wiring per-request kernel context) can inject explicitly
rather than via the shared singleton.
Phase 4a — Mock-provider integration tests
- tests/integration/llm-router-wiring.test.ts (new, 17 tests).
- Per-domain wiring assertions (15): for each LLM-enhanced
plugin → coordinator → service path, asserts service.llmRouter ===
kernel.llmRouter after kernel boot with an injected mock provider.
This is the precise structural test for the gap the author's friend
flagged — a regression in any domain's wiring fails immediately.
- Behavioral end-to-end (1): TestExecutorService.analyzeFailuresWithLLM
actually invokes the mock provider; assert generateCalls counter
incremented and the response surfaced.
- Sanity (1): kernel.llmRouter is defined when a router was built.
Phase 4b — Real-provider E2E smoke test
- tests/e2e/llm-router-real-providers.test.ts (new, 3 tests).
- Gated by AQE_LLM_E2E=1; each provider auto-skips when its API key
isn't in env.
- Uses mode: 'manual' to bypass DEFAULT_QE_ROUTING_RULES (which would
otherwise route generic tasks to Claude even when defaultProvider
is set to something else — worth knowing).
- Verified live on 2026-05-21 against:
Gemini (gemini-2.5-flash) → "Hi!"
OpenAI (gpt-4o-mini) → "Hi!"
OpenRouter (openai/gpt-4o-mini) → "Hi!"
Claude intentionally skipped — the ANTHROPIC_API_KEY in the dev env
is a 24-char stub (real Anthropic keys are ~108 chars), would 401.
Phase 4c — Documentation
- docs/implementation/adrs/ADR-043-vendor-independent-llm.md: added
"Wiring Verified" status entry (2026-05-21) + full addendum documenting
the problem found, the four-phase resolution, the verification tests,
and the behavioral consequence for users.
- docs/implementation/adrs/ADR-051-agentic-flow-integration.md: added
"LLM-Enhancement Wiring Verified" status entry pointing to ADR-043's
addendum.
- CHANGELOG.md: added [Unreleased] section under Changed — behavioral,
explicitly noting the behavior shift (fleets with provider keys in env
will now start routing LLM calls through that provider), the opt-out
path (KernelConfig.llmRouter: { enabled: false }), and the persistence
fix for `aqe llm config --set`.
Test totals across all four phases on this branch
- 39 unit tests across 5 files (Phase 1)
- 17 integration tests in 1 file (Phase 4a)
- 3 real-provider E2E tests in 1 file (Phase 4b, AQE_LLM_E2E=1)
- 34 pre-existing kernel tests still pass (no regression)
- 22 pre-existing test-generation coordinator tests still pass (no regression)
- Clean tsc --noEmit across the full project
Refs ADR-043, ADR-051
Closes the wiring gap identified by user audit on 2026-05-21
Builds on
|
||
|
Dragan Spiridonov
|
4e513b431d |
fix: resolve 3 P0 blockers and CI failures from QE v3.8.13 audit
- Delete dead MCP server.ts (911 lines, never used in production — entry.ts imports MCPProtocolServer). Add legacy aliases in index.ts for backwards compat. - Fix ESLint: rename .eslintrc.js to .eslintrc.cjs (CommonJS in ESM project) - Fix hardcoded version '3.0.0' in protocol-server.ts and http-server.ts — now reads dynamically from package.json - Fix vitest process hang: add worker-level afterAll force-exit (setup-worker.ts) and global teardown safety net. Native modules (better-sqlite3, hnswlib-node) held open handles preventing fork worker exit. - Add test gate to npm-publish.yml — 194 unit test files must pass before publish - Disable benchmark.yml release trigger (benchmarks/suite.ts doesn't exist) - Remove continue-on-error from test steps in optimized-ci.yml (was masking failures) - Add getToolDefinitions() to MCPProtocolServer for test introspection - Update e2e and integration tests to use production server API Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Profa
|
2253d05618 |
feat: flatten project structure — archive v2, promote v3 to root
Remove the nested v3/ subdirectory by moving all active source code, tests, configs, and assets directly to the project root. Archive v2/ (dead code, 847MB) to v2-archive.tar.gz. Key changes: - Move v3/src/ → src/, v3/tests/ → tests/, v3/packages/ → packages/ - Move v3/tsconfig.json, v3/vitest.config.ts to root - Merge v3/package.json into root package.json (ESM, scripts, deps, exports) - Merge v3/scripts/ into scripts/, v3/implementation/ into docs/implementation/ - Update all CI workflows to remove "cd v3 &&" and v3/ path prefixes - Fix all init installer path resolution (3-level-up → 2-level-up) - Fix hooks to use "node ./dist/cli/bundle.js" instead of bare "aqe" - Update settings.json ADR/DDD directories to docs/implementation/ - Fix helper scripts (statusline, adr-compliance, ddd-tracker, guidance-hooks) - Update 58 agent definitions: aqe/v3/ → aqe/ memory namespaces - Update 15 guidance shards: v3/src/ → src/ path references - Fix sync interfaces: remove v3/ and v2/ path references - Fix test path calculations (process.cwd() + '..' no longer needed) - Fix flaky perf test threshold (500ms → 1000ms for CI environments) - Fix complexity-analyzer test to match ADR-051 keyword-based eligibility Build passes, 17,901 tests pass, CLI and MCP bundles verified. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Profa
|
fd2f37eaa0 |
fix(v3): update README claims and init hooks to use correct package name
Changes: - Fix README.md agent counts (48 QE agents = 41 main + 7 subagents) - Fix README.md skill counts (15 QE-specific skills) - Fix v3/README.md agent counts (47 → 48) - Fix init-wizard.ts hooks from 'npx @agentic-qe/v3' to 'npx agentic-qe' - Complete v2 → v2/ folder reorganization (54 items moved) - Update root package.json to point to v3 as main version Verified claims: - 48 QE agents (41 main + 7 TDD subagents) - 15 QE-specific skills - 12 DDD bounded contexts - ~5,600+ tests - HNSW, ReasoningBank, Queen Coordinator features exist Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Profa
|
7363bb1278 |
chore(release): bump version to v2.8.0
## @ruvector/edge Integration - Phases 0-3 Complete ### Phase 0: Browser Runtime - WASM shims and browser compatibility - BrowserAgent, BrowserHNSWAdapter, IndexedDBStorage - Chrome DevTools panel ### Phase 1: VS Code Extension MVP - Full extension with real-time analysis - Inline test suggestions, coverage visualization - Offline-first storage with sync ### Phase 2: P2P Foundation - Ed25519 crypto, WebRTC connections - SignalingServer, AgentSpawnAPI, EdgeServer - Pattern sharing via data channels - CRDT conflict resolution ### Phase 3: Web Dashboard - React dashboard with dark theme - P2P connection management UI - QE Agent Launcher from browser - React hooks (useP2P, usePatternSync, etc.) ### Documentation - Edge Server Guide - P2P Pattern Sharing Guide - Updated CLI agent-commands ### Also includes - Dead code cleanup (~22k lines) - 143 new P2P integration tests - Version updates across all files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Profa
|
b91627f282 |
fix(quality): implement GOAP Issue #149 code quality remediation
Phase 1: Test Determinism - Create SeededRandom utility with Mulberry32 PRNG (src/utils/SeededRandom.ts) - Migrate 258+ Math.random() calls to SeededRandom across 65+ test files - Add timer test utilities (tests/helpers/timerTestUtils.ts) - Add jest.useFakeTimers() pattern to 13 test files Phase 2: Type Safety (63.5% reduction) - Reduce `any` types from ~1,800 to 656 - Add 50+ new interfaces across agents, core, mcp, cli modules - Fix all TypeScript errors (0 errors, build passes) - Replace Record<string, any> with Record<string, unknown> - Add proper type guards for hook methods (PostTaskData, TaskErrorData) Key improvements: - agents/: 105 → 0 any types (100% reduction) - mcp/: 226 → 102 any types (55% reduction) - cli/: 277 → 168 any types (39% reduction) - core/: 199 → 175 any types (12% reduction) Closes #149 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Profa
|
517bd5a5a0 |
feat: Complete RC 1.2.0 - AgentDB Integration & Production Hardening
🎉 Major Release: AgentDB Integration Complete ## Critical Fixes ### AgentDB API Compatibility (Blocker Resolved) - Fixed field name mismatch: `data` → `embedding` in vector operations - Fixed field name mismatch: `similarity` → `score` in search results - Fixed method name: `getStats()` → `stats()` (synchronous) - Removed unnecessary Float32Array conversion - **Impact**: 6/6 AgentDB tests passing (100% pass rate) - **Release Score**: 78/100 → 90/100 (+12 points, +15.4%) - **Resolution Time**: 2 hours systematic investigation ### Test Results - AgentDB Integration: 6/6 passing ✅ - Core Agent Tests: 53/53 passing ✅ - Build Quality: Clean TypeScript compilation ✅ - Regression Testing: Zero new failures ✅ - Performance: <1ms operations validated ✅ ## Files Modified ### Source Code - `src/core/memory/RealAgentDBAdapter.ts` - Fixed 4 API issues - `src/mcp/MCPToolRegistry.ts` - New MCP tool registry - `tests/e2e/cli.test.ts` - Fixed test syntax errors ### Scripts - `scripts/test-real-agentdb.js` - AgentDB integration test - `scripts/test-agentdb-e2e.ts` - E2E validation script ## Cleanup & Optimization ### Documentation (Archived 24 reports, saved 4.1MB) - Archived old version reports (v1.0.x, v1.1.x) - Archived development reports (TEST-*, VALIDATION-*, REGRESSION-*) - Created new RC 1.2.0 comprehensive reports - Reports: RC-1.2.0-FINAL-STATUS.md, RC-1.2.0-FINAL-VERIFICATION.md, RC-1.2.0-CLEANUP-RECOMMENDATIONS.md, RC-1.2.0-RELEASE-READY.md ### Dependencies (Removed 89 packages, saved ~7.3MB) - Removed: @cucumber/cucumber (5.4MB, unused) - Removed: axios (420KB, unused) - Removed: c8 (dev, using jest --coverage) - Updated: @types/node, @anthropic-ai/sdk, chokidar (latest patches) - **Impact**: Faster installs, smaller package size ### Repository Cleanup - Removed .DS_Store files (macOS system files) - Removed .bak backup files - Updated .gitignore with temp file patterns ## CHANGELOG Updates - Added comprehensive v1.2.0 section dated 2025-10-22 - Documented AgentDB fixes and verification results - Included performance metrics and test results ## What's Working ### AgentDB Integration ✅ Vector storage (single + batch, <1ms latency) ✅ Similarity search (cosine, euclidean, dot product, <1ms) ✅ Database statistics and monitoring ✅ QUIC synchronization (<1ms latency, 36/36 tests) ✅ Automatic mock adapter fallback ✅ Real AgentDB v1.0.12 validated ### Core Functionality ✅ All v1.1.0 features intact (104/104 tests) ✅ Q-Learning algorithms ✅ Pattern bank operations ✅ Performance tracking ✅ Zero regressions detected ## Performance Metrics - Single Insert: <1ms - Batch Insert (10): <10ms - Vector Search (k=5): <1ms - QUIC Sync: <1ms (validated) - Memory Usage: 0.09MB/12 vectors - Database Size: 88KB for 12 vectors (384 dims) ## Release Readiness - Quality Score: **90/100** ✅ (Target: 90) - Tests Passing: **59/59** (100%) - Build Status: **Clean** ✅ - Regressions: **Zero** ✅ - Package Size: **~9MB** (reduced from 12MB) - Ready for Production: **YES** 🚀 ## Documentation - docs/reports/RC-1.2.0-FINAL-STATUS.md - docs/reports/RC-1.2.0-FINAL-VERIFICATION.md - docs/reports/RC-1.2.0-CLEANUP-RECOMMENDATIONS.md - docs/reports/RC-1.2.0-RELEASE-READY.md - docs/reports/archive/ (organized old reports) --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Profa
|
5a2883a4a0 |
chore: Initial release v1.0.0
Agentic Quality Engineering Fleet - AI-driven quality management platform Features: - 16 specialized QE agents for comprehensive testing - AI-powered test generation with 95% coverage targets - O(log n) sublinear algorithms for massive scale - Parallel test execution with intelligent orchestration - Real-time coverage analysis and gap detection - Smart quality gates with ML-driven decisions - Security scanning (SAST, DAST, dependency analysis) - Performance testing with load orchestration - Flaky test detection with automatic stabilization - API contract validation with breaking change detection - Test data generation (10,000+ records/sec) - Production intelligence (convert incidents to tests) - Chaos engineering for resilience testing - MCP integration with Claude Code - Multi-framework support (Jest, Mocha, Cypress, Playwright) Architecture: - TypeScript implementation - MCP (Model Context Protocol) integration - Claude Code agent orchestration - Event-driven agent coordination - Distributed fleet management - Cross-session memory persistence Documentation: - Complete API documentation (TypeDoc) - User guides (Getting Started, MCP Integration, Test Generation, etc.) - Architecture deep-dive - CLI reference - Contributing guidelines - MIT License Tech Stack: - TypeScript 5.3+ - Node.js 18+ - @anthropic-ai/sdk ^0.64.0 - @modelcontextprotocol/sdk ^1.18.2 - Jest for testing - Winston for logging - SQLite for persistence Project Structure: - src/ - Source code (agents, core, MCP, CLI) - tests/ - Comprehensive test suite - docs/ - Complete documentation - .claude/ - Agent definitions and commands - config/ - Configuration files 🤖 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com> |