Evolve the QE framework from PACT (Proactive, Autonomous, Collaborative,
Targeted) to PACTS by adding a fifth principle: Structured — governance,
observability, and explainability of agent behavior. We measure confidence,
not trust (trust is a human feeling; confidence can be mathematically
explained). Especially important in regulated industries. Inspired by DORA's
research on AI-assisted delivery and platform-engineering quality measurement.
Anti-pattern: bolting agentic AI onto chaos.
Scope:
- Source skills (.claude/skills): holistic-testing-pact + agentic-quality-
engineering SKILL.md, agentic-quality-engineering schema (added optional
`structured` dimension, kept slug keys), README, skills-manifest.json
- Regenerated derived artifacts: assets/skills (via prepare-assets equivalent
copy) and the two affected .opencode yaml (via generate-opencode-skills)
- Hand-maintained trees updated consistently: .kiro skills, docs (catalog,
glossary, intro), plugin/marketplace manifests, qe-learning-coordinator
Stability:
- Kept skill id/dir `holistic-testing-pact` (and `.kiro` `qe-holistic-testing-pact`)
unchanged — referenced across other repos. Only titles/descriptions/content
changed to PACTS.
Attribution: PACT originated with Reuven Cohen (Agentics Foundation); adapted
to QE by Dragan Spiridonov. PACTS adds Structured.
Co-Authored-By: claude-flow & Agentic QE fleet
MCP server reliability + database-free completeness (community issues from
@nagoodman): in-process `agentic-qe mcp` (#528), memory_delete stale-cache fix
(#535), db-free propagation to all editor MCP configs (#533), and the opt-in
--no-claude exclusive install (#532). Plus the js-yaml dev bump (#540) and the
devil's-advocate follow-up fixes (routing/merge/startup + CI guard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the 2026-06-17 evidence: ADR-038 (DRACO ceiling) is now Accepted
upstream → the A3 gate leans G-ABORT; MetaHarness v0.1.15 ships mcp-scan --json
+ scorecard, shrinking A1 to "consume, not build"; and two new standalone
candidate actions lifted from ADR-150 (A9 graceful-degradation, A10 KRR router).
Cross-links the ADR-150 spawnSync bug class to AQE #528.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to d95f0d7b / 2f4d119c — the earlier passes updated .claude/skills
and assets/skills but missed several other eval surfaces that still pinned
retired IDs (claude-3.5-sonnet, claude-3-haiku):
- `.github/workflows/test-qe-browser.yml` — the `aqe eval run --model` flag
(inert in command-eval mode, but a real dead-ID leftover) -> claude-sonnet-4-6.
- `plugins/agentic-qe-fleet/skills/` — the tracked plugin-marketplace skill copy
(8 eval suites + a sample-output fixture); chaos skills get the opus-4-8 ceiling.
- `docs/schemas/skill-eval.schema.json` — models `enum` was missing claude-opus-4-8
and still allowed retired/deprecated IDs; trimmed to current+active, default
flipped to claude-sonnet-4-6. Plus description examples in the sibling schemas
and the modelUsed values in docs/examples/*.json fixtures.
- `docs/templates/skill-frontmatter.example.yaml` validation_models list.
- `src/cli/commands/eval.ts` — the `aqe eval` `--model` help example and the
`run-all --models` default (`claude-3.5-sonnet` -> `claude-sonnet-4-6`).
Verified: typecheck clean; all edited JSON/YAML parse; zero retired IDs remain in
any eval suite, schema, template, workflow, or eval-CLI default. Intentionally
left untouched: the consensus-provider subsystem (pricing tables / type-unions /
defaults — separate routing layer, bucket-2 data) and historical release notes /
QE audit reports (which document the old IDs by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cross-model consistency matrix (models_to_test) in 54 skill evals + both
eval templates pinned retired/malformed model IDs — `claude-3.5-sonnet`
(malformed: dashes, not dots; resolves to Sonnet 3.5, retired Oct 2025) and
`claude-3-haiku` (retired Apr 19 2026). Both 404, so those evals could not run
a meaningful cross-model check.
Refresh to current pinned IDs, preserving the original "primary + fast floor"
design:
- claude-sonnet-4-6 # primary (high accuracy)
- claude-haiku-4-5 # fast floor (minimum quality)
High-stakes skills additionally get the capability ceiling claude-opus-4-8
(security-testing, security-visual-testing, n8n-security-testing,
pentest-validation, chaos-engineering-resilience, qe-chaos-resilience,
qe-defect-intelligence) and the security eval template. Opus is scoped to these
7 + the security template rather than all 54 to avoid a fleet-wide eval-cost
blowup.
Also fixes the two `--model claude-3.5-sonnet/claude-3-haiku` examples in
run-skill-eval.ts and a nested cross-model test-case input in
qe-learning-optimization.yaml.
Verified: all eval YAML parses; zero retired/deprecated Claude IDs remain across
skills/evals + templates + runner. (src/domains/constants.ts MODEL_TIERS was
already current — tier 1 claude-haiku-4-5 — so it is intentionally unchanged;
the legacy claude-3-haiku-20240307 entries in cost-tracker/bedrock are retained
historical pricing/availability data.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Statuses updated to reflect reality (were stale 'Proposed' despite
implementation): 107/108/110 Implemented; 105 Implemented w/ INFERRED
domain-finding follow-up; 106 Implemented (live matrix, native tiers
pass every cell); 109 Implemented infrastructure w/ corpus-scaling
follow-up. Gitignore transient run outputs, force-keeping canonical
evidence files.
Verified: full branch tsc clean; both new workflows valid YAML; 58
locally-runnable tests green; no .env or key material in any branch
commit (audited).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome
(hidden acceptance test), style-neutral verbatim judge instruction (no
treatment-revealing framing in either direction), two-pass cross-family
judging with INFERRED-labeled fallback, clustered-stats reporting rule.
Provably rubric-before-data: this commit precedes any results-* artifact.
2 scenarios with seeded defects + personas + hidden tests (off-by-one
pagination; swallowed async rejection). Hidden tests verified both
directions: FAIL on seeded bug with precise reasons, PASS on the fix.
lib/stats.ts encodes the three Pattern Space harness fixes as code:
isGoodRow enforced inside aggregate() (no path to a number that skips
it), seeded per-item counterbalance (their null cell was 72% position-
skewed), exact scenario-clustered sign test (validated against the
known 27/11 → p≈0.0136 re-analysis).
Verified: 14/14 stats tests; hidden-test ground truth both directions;
tsc clean. Remaining: live conversation driver + judge integration
(API-gated, same decision as ADR-106 live mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds EvidenceClass (EXECUTED|STATIC|INFERRED|CONJECTURE) and
EvidenceArtifact to the Finding schema, required at the type level.
requirements.ts factory defaults to STATIC (all 26 steps are
deterministic content analysis); pipeline crash-findings are EXECUTED
with the error output as artifact. Field is evidenceClass, not
evidence — StepResult.evidence: string[] already exists (ADR amended).
Verified: tsc --noEmit clean; validation-pipeline-handler tests 7/7
(MCP handler passes findings through, new field flows to MCP output).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lineage registry with rubric-before-data, changelog, kept-data,
clustered-statistics, and style-neutral-judge rules. Seeded with
arena Phase 1 (incl. its retracted measured-duration variant as the
model entry) and the planned ADR-109 row. Pre-registry numbers are
uncited-by-default pending backfill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six proposed ADRs from the universal-pattern-space QE assessment:
evidence-class labels on findings, behavioral safety evals per model
tier, shipped-agent invariant verification, benchmark lineage with
pre-registered rubrics (incl. antipattern list), interaction benchmark
for qualitative agents, kept-nulls negative pattern records.
Cross-checked against existing ADR corpus: Related fields reference
ADR-058/061/070/074/083/086/098/099/102/104; mid-task safety injection,
restraint probes, and null-vs-quarantine sequencing folded in from the
cross-check findings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aqe arena run: seeded tournaments where test strategies compete on REAL
fitness — built-in first-order operator mutants (string/comment/arrow-safe
enumeration) executed per strategy via node --test, real line coverage
from --experimental-test-coverage, and a deterministic suite-cost term.
Tournament emits a Wolfram-style pairwise competitive array; --evolve
hill-climbs the winner by seeded group flips with real re-evaluation.
Runs persist fail-soft to memory.db kv_store (namespace 'arena');
aqe arena list reads them back. Self-contained: no core changes, no new
dependencies; fixtures/arena-demo ships 4 test groups of varying kill
power over a branch-dense pricing module.
Reproducibility contract: same seed + fixture => byte-identical envelope
(verified E2E by diffing two runs), with informational.runtimesMs the
only excluded field. Honest finding during verification: the original
measured-runtime rank penalty was jitter-dominated at fixture scale and
flipped rankings between identical-seed runs — replaced with the
deterministic suite-size ratio (documented in ADR-104).
Phase 2 (documented): LLM-generated strategies frozen as artifacts,
winners distilled into qe-test-architect priors; MCP arena_run
registration (CLI and MCP share runArena() — parity by construction).
10 tests green (incl. real-engine reproducibility); tsc + eslint clean.
Part of #520 (improvement 7/7 — initiative complete).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
qcsd-development-swarm now runs as a deterministic workflow script
(.claude/workflows/qcsd-development-review.js) instead of prompt-driven
Task-tool fan-out:
- pipeline: one finder per quality dimension (TDD adherence, complexity,
coverage gaps; qe agents as agentType) -> 3 blind adversarial refuters
per finding -> deterministic synthesis, no agent editorializes the report
- Loki-mode constraints (ADR-074) encoded in refuter prompts: blind review
(bare claim + evidence only), anti-sycophancy (attack the claim;
uncertainty defaults to refuted), 3 distinct lenses
- majority kill (>= ceil(N/2) refuted votes); killed findings retained
with refutations for audit; output is finding-verdict@1 envelopes
(ADR-103)
- SKILL.md v1.1.0: execution.primary workflow, Task-tool fallback
documented (reports marked verification: none)
Validation run (wf_254f1271-022): 3 dimensions over src/, 48 agents,
15 raw findings -> 14 confirmed, 1 killed by unanimous refutation of a
factually-wrong claim — the adversarial gate demonstrably filtering a
false positive. Several confirmed findings target code added earlier in
this initiative (honest signal, queued as follow-ups).
Part of #520 (improvement 5/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Versioned, additive-only verdict envelopes for agent/MCP handoffs:
RiskDecision (quality gates), FindingVerdict (review findings with
adversarial refutations), CoverageGap (risk-weighted gaps).
- src/contracts/verdicts.ts: dependency-free validators (source of
truth) + mirrored draft-07 JSON Schemas; no zod/ajv runtime deps
(light-install posture preserved)
- quality_assess MCP boundary now attaches a validated riskDecision
envelope derived from the gate outcome (approve/block, escalate when
indeterminate) — additive, existing payload unchanged; direct and
wrapped handlers share the same mapToResult so MCP parity is inherent
- schemas/*.schema.json published via scripts/generate-verdict-schemas.mjs
for external ajv validation and workflow agent({schema}) enforcement
(consumed by improvement 5)
17 contract tests (goldens validate; mutations rejected with field-level
errors; builder always emits valid envelopes). E2E: envelope emitted by
the compiled boundary builder validates with stock ajv against the
published schema; mutated sample rejected. tsc + eslint clean.
Part of #520 (improvement 6/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Anthropic announced nested subagent support (depth=5) on 2026-06-09; the
runtime currently strips Task at parent->child spawn (upstream denylist,
empirically confirmed by our committed probe: level=1 status=NO_AGENT_TOOL,
matching ruflo ADR-147). This lands the readiness infrastructure that is
independently useful today and activates with zero changes when the gate
lifts:
- aqe hooks post-task gains --parent-agent-id / --depth (validated:
identifier charset, integer 0 <= d <= 32; depth=0 survives as 0);
provenance persisted to qe_trajectories.metadata_json (stage 1 — no
schema migration) and forwarded into ReasoningBank feedback so learning
can segment patterns per hierarchy level
- TaskExecution gains optional parentAgentId/depth (additive types)
- scripts/probe-nested-spawn-depth.mjs: single-shot empirical probe,
committed transcript under docs/probes/ — re-run after every
`claude update`; TASK_TOOL_LIVE is the signal for follow-up phases
- fresh-project fix found by the E2E: persistTaskOutcome assumed
captured_experiences/experience_applications exist, but they are
middleware-created — Stream B silently no-oped on fresh installs;
canonical DDL now ensured in the hook path
Deliberately deferred (documented in ADR-101 P1): tools: frontmatter
retrofit on shipped qe-*.md agents — Claude Code frontmatter is a
restrictive allowlist and would sever inherited MCP tool access;
needs maintainer sign-off + wildcard verification first.
8 unit tests green; E2E on a fresh temp project verified persistence,
rejections, and depth=0 boundary. tsc + eslint clean.
Part of #520 (improvement 4/7, Tier 2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Breaks runaway retry loops: when the same Bash command fails consecutively
(normalized signature, cross-process via a small JSON sidecar under
.agentic-qe/), pre-command warns at 3 failures and emits a BLOCK
recommendation with a recovery hint at 5. Advisory by default;
AQE_STRICT_TOOL_LOOP=1 upgrades the block to a PreToolUse deny. A
successful run resets the breaker; an open breaker grants a half-open
probe after 2 minutes; stale entries pruned after 30 minutes.
Mirrors the ADR-064 domain-circuit-breaker semantics at command level.
Sidecar instead of memory.db: pre-command deliberately runs without a DB
open, and each hook call is a fresh process so in-process state is
impossible. Fail-open everywhere (corrupt state file => allow).
11 unit tests green; E2E verified with real aqe hooks pre/post-command:
warn after 3, BLOCK after 5, strict-mode deny envelope, reset on success.
Part of #520 (improvement 3/7, Tier 1 complete).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The enableCache flag existed since ADR-011 but was never read: every
advisor/LLM call paid full input price. Now:
- ClaudeProvider.generate() sends `system` as a content-block array with
cache_control {type: ephemeral} when enableCache (default on); plain
string when disabled. Anthropic-native provider only (ADR-092 — other
providers untouched).
- cache_creation_input_tokens / cache_read_input_tokens surfaced into
TokenUsage.cacheCreationTokens/cacheReadTokens; totalTokens now counts
the full prompt (input_tokens alone is the uncached remainder).
- CostTracker bills cache reads at 0.1x and writes at 1.25x input rate
instead of ignoring them.
5 new provider tests assert the serialized request shape (cache_control
present when enabled, absent when disabled, no system field when no
prompt) and the usage mapping. 64/64 LLM tests green, tsc clean.
Part of #520 (improvement 2/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fable 5-era models emit large extended-thinking blocks inside task results
and step actions; embedding them verbatim contaminates the 384-dim pattern
vectors in memory.db and degrades HNSW retrieval.
- new scrubReasoningBlocks()/scrubReasoningDeep() (src/shared/reasoning-scrub.ts):
boundary-gated — only well-formed <think|thinking|reasoning|
REASONING_SCRATCHPAD>...</> pairs are removed; prose mentioning a tag
survives; clean input returned byte-identical
- applied at ingestion: TrajectoryTracker startTrajectory task,
recordStep action/result.data/result.error
- applied at distillation: ExperienceReplay.storeExperience task/strategy/
keyActions (insert + embedding text)
- applied at flow-forward: both init-time embedding backfills (legacy rows
cleaned as they get embedded — no memory.db rewrite) and retrieveSimilar
query embeddings (symmetric with stored vectors)
- 15 unit tests; E2E verified against a temp DB: trajectory + experience
rows persisted with all scratchpad blocks stripped
Part of #520 (improvement 1/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aqe memory/agent/task/team/pipeline CLI commands always failed with
"Fleet not initialized": autoInitialize() built kernel+queen in its own
CLIContext while the shared MCP handlers guard on the core-handlers
fleet-state singleton. New adoptExternalFleet() bridges the CLI's
components into that state (idempotent, never clobbers an existing
fleet), restoring MCP-CLI parity. Verified E2E: store/get/delete green.
- devcontainer: node_modules named-volume overlay + npm ci in postCreate
so container installs Linux natives without touching the host's macOS
binaries (takes effect on next rebuild; devcontainer users only)
- docs/setup: drop stale ruflo@3.5.18 pins (CLAUDE.md quick setup,
install-tools.sh) in favor of ruflo@3 + global `ruflo mcp start`
- docs: add Fable 5 / ruflo-parity improvement plan (tracked in #520)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Switch the cross-project knowledge hub from the cloud endpoint to a
locally-runnable Nagual instance, and add an optional local LLM judge
for scoring pattern quality.
- nagual-client.ts: REST client for a local `nagual serve` instance
(NAGUAL_URL, default http://localhost:3333); search/recordOutcome/
promotePattern are all fire-and-forget safe.
- local-judge-client.ts: Ollama-compatible judge (NAGUAL_JUDGE_URL/MODEL)
blending 0.7*outcome + 0.3*judge_score, with an ambiguous-zone review flag.
- task-hooks.ts: pre-task surfaces Nagual cross-project context for
complex tasks in relevant QE domains (best-effort, gated on env + length).
- settings.json: add NAGUAL_JUDGE_URL/MODEL; fix invalid mcp allow-rule
globs (mcp__ruflo__* / mcp__claude-flow__*) and add Bash(rm -rf:*) deny.
- docs: de-reference the cloud hub in favor of a local instance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code-intelligence TS/JS via bundled tree-sitter WASM (no typescript dep, #511) +
fix for MCP domain tools hanging on fresh install. Also fixes the release skill's
repo path (agentic-qe-new -> agentic-qe).
Repo-health cleanup from an external code audit:
- eslint: ignore src/_archived (already excluded by tsconfig, so linting it only
produced 6 parserOptions.project errors).
- Align the architecture claim: src/index.ts and docs said "12"/"6" bounded
contexts; there are 13 domain dirs (matching package.json). Updated src/index.ts
(comment + ARCHITECTURE const) and the docs list to the actual 13.
- Apply stock `eslint --fix` (prefer-const, prefer-as-const, no-useless-escape,
no-regex-spaces) across 24 files.
Lint problems: 1072 -> 1031. Typecheck and build pass; no behavioral changes.
Per request: aqe init now installs the cross-platform Node hook shim into each
user project and routes hooks through it (instead of `npx agentic-qe hooks`).
- package.json: add `.claude/hooks` to `files` so aqe-hook.cjs ships in the npm
tarball (verified via `npm pack --dry-run`).
- 07-hooks.ts: new installHookShim() copies aqe-hook.cjs into the project's
.claude/hooks/ (source: node_modules/agentic-qe, then the package's own copy
resolved relative to the compiled module — works for npx too).
- 07-hooks.ts + init-wizard-hooks.ts: generated hooks now run
`node "${CLAUDE_PROJECT_DIR:-.}/.claude/hooks/aqe-hook.cjs" <cmd> ...`
(README/help examples left as `npx agentic-qe hooks` for manual use).
- aqe-hook.cjs: added an npx fallback (`npx -y --prefer-offline agentic-qe`)
so npx-only installs with no project-local bundle still work; disable with
AQE_HOOK_NPX=0 (tests use this for determinism). Header/scope updated (now a
shipped, init-copied file, not repo-internal).
Migration: isAqeHookEntry already matches the new `aqe-hook.cjs` command (via
the `\baqe\b` pattern) AND the old `npx agentic-qe hooks` form, so re-running
`aqe init` replaces a user's existing hooks with the shim form — no duplicates.
Verified end-to-end (real `aqe init --auto` in a temp project):
- aqe-hook.cjs copied into .claude/hooks/ ("Installed aqe-hook.cjs (copied)")
- all 11 generated hooks use `node .../aqe-hook.cjs <cmd>` (0 npx commands)
- post-task retains the #508 fix (--agent/--success true/--description)
- isAqeHookEntry matches new+old forms, not unrelated user hooks
- shim 7/7, settings-merge 17/17, task-hooks 5/5 tests pass; typecheck clean
ADR-098 documents the durable behavioural decisions from the ruflo adoption
pass (#510): domain-score de-dilution (#2), EWC++ importance-weighted forgetting
(#6), contradiction suppression (#7), the two pre-existing bug fixes
(consecutive_failures, MCP -32601), and the explicit NEGLECTED decisions —
ruflo's Beta-priors (#2) and RaBitQ speedup (#8) did not transfer, and
multi-bundle packaging is deferred. AQE WH(Y) format, Accepted.
Marks the consecutive_failures (ae7c8b71) and MCP error-code (9b189115)
side-findings FIXED in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit (qe-devils-advocate) verdict: 10/10 items have real committed code + real
passing tests at claimed counts; production changes (#2/#6/#7) genuinely wired
into live paths; no real-store mutation; #8 honestly UNWIRED. Three fair
discrepancies, now corrected:
1. #2 had one TAUTOLOGICAL test (asserted (1/3)*0.4 > (1/11)*0.4 — arithmetic on
literals, not a code call). Rewritten to score a matching vs non-matching
agent through calculateAgentScores and assert the delta equals the CAPPED
domain term (~0.1333), which fails if the cap is removed. 4/4 still pass.
2. #2 status row softened to VERIFIED (PARTIAL): +5.6pp shipped, but the
structural Q-maturity / fixed-perf-term ceiling on confidence remains.
3. #10 status row corrected to PARTIAL: only the smoke contract was delivered;
the original multi-bundle 'subset packaging' scope is deferred.
No production code changed by this commit. Side-findings re-confirmed real
(dead consecutive_failures branch, MCP [object Object] error, RVF/SQLite drift).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports ruflo's perf-M4 (sign-random-projection + Hamming popcount). Adds
src/shared/utils/rabitq.ts: signSignature (384-bit packed into 12 Uint32
words, 48 bytes), hammingDistance (SWAR popcount), hammingSimilarity, and
topKBySignature (Hamming prefilter gated at N>=100 + exact-cosine rerank of a
recall-aware pool, so output equals exact topK when the true top-k fall in the
pool). 16 tests + scripts/benchmark-rabitq.ts (npm run benchmark:rabitq).
HONEST FINDING (verified, does NOT match ruflo's optimistic 2.7x/10.9x):
N=1000: speedup 1.31x, recall@10 100%, index 1.46MB -> 47KB (32x smaller)
N=5000: speedup 1.18x, recall@10 99.9%, 7.32MB -> 234KB (32x smaller)
The 32x INDEX-MEMORY reduction is real and unconditional. The SPEEDUP is only
~1.2-1.5x in JS, not ruflo's figure: a 1-bit sign signature in 384-dim is
coarse, so preserving recall@10 >= 0.95 needs a large exact-rerank pool
(~40-55% of N), which eats most of the prefilter win. ruflo's big win came
from replacing a LINEAR scan; AQE's main search is already native HNSW
(sublinear), so that win doesn't transfer. Tests pin this tradeoff (recall==1.0
with a sufficient pool; small pool sacrifices recall) so it can't be
mis-sold later.
NOT INTEGRATED — no zero-regression consumer exists:
- unified-memory-hnsw search() IS the ranking (no separate exact rerank to
prefilter), so Hamming-prefiltering its beam would drop true neighbors;
- coherence-gate-energy O(N^2) SUM and k-means clustering are numerically
sensitive (Hamming would change results).
Committed as a tested, benchmarked, ready-to-use utility (real memory lever for
patterns.rvf and a future prefilter->exact-rerank consumer), not wired into any
production path. No production code or .agentic-qe stores touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigation (verified with real `aqe hooks pre-task` routes) shows AQE's
40% routing confidence is NOT a Beta-bandit prior, so ruflo ADR-142's
per-bucket fix does not apply. Confidence is the top agent's static combined
score from calculateAgentScores (agent-routing.ts) blended with an immature
Q-table (qWeight=0). Real root cause: domain-score dilution —
domainScore = (domainMatch / detectedDomains.length) * 0.4 — with the domain
detector returning 7-12 domains, washing out relevance. Measured easy=37.3%,
hard=20.5%. Proposed scoped fix to calculateAgentScores is a published
hot-path change → deferred pending user sign-off (no production code touched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports ruflo's benchmark-self-learning.mjs discipline (the 'reports success
but persists nothing' detector) to AQE. Proves the learning stores actually
persist what they claim, that stat aggregators agree with the underlying
index, that nothing writes accidentally, and that data survives restart.
Sections (each isolated in a temp dir, each gating):
A store persistence — N creates => totalPatterns delta == N
B outcome learning — N outcomes => learningOutcomes delta == N, successRate matches fed ratio
C negative control — read-only ops => 0 delta (no accidental writes)
D restart survival — dispose + reopen on same temp path => counts persist
E consistency — RvfPatternStore.getStats == adapter.totalVectors == bank.totalPatterns
F audit-real — copies real stores read-only, checks banner numbers round-trip
Data protection (CLAUDE.md): synthetic mode uses fresh temp dirs; audit-real
copies stores and opens only copies. Verified real .rvf stores byte-identical
(mtime+size+sha256) before/after a run.
Audit-real finding on the live store: banner numbers corroborated
(72.5% success, 39.7% confidence, 2131 routings) BUT RVF vector count (338)
!= SQLite qe_patterns rows (345), a ~7-pattern index/table drift the banner
hides — exactly the divergence this harness exists to surface (reported as a
diagnostic note, non-gating). Synthetic sections A-E pass; exit 1 on failure.
npm run benchmark:self-learning
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audited AQE routing against two router bugs ruflo fixed 2026-05-29:
- #2229 stale route-cache: NOT PRESENT — Q-values read live from SQLite
(qe-reasoning-bank buildQValueLookup) and in-memory qTable; no cached
routing decision exists to go stale, so no invalidation needed.
- #2239 state-encoder truncation: NOT PRESENT — state keys are plain
string joins (agent-routing buildRoutingStateKey: taskType|priority|domain),
no bit-folded quantized vector, no mask. Distinct keyword tasks map to
distinct states.
Adds a guard test locking in non-collapsing state encoding so a future
refactor toward a bit-folding encoder fails here. 4 tests passing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan to port highest-value mechanisms from ruvnet/ruflo's last month
(307 commits, ADR-120 to ADR-143) into AQE: self-learning verification
harness, per-bucket bandit priors, router bug audit, MCP smoke layer,
EWC++ forgetting protection, and 9 more items across 3 tiers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the ADR-043 LLM router into the production swarm path: kernel
constructs HybridRouter singleton during initialize(), threads it
through all 11 LLM-enhanced domains and MCP standalone tools. Adds
persistent aqe llm config --set, Gemini GOOGLE_API_KEY alias, 2 real
fallback-chain bug fixes (provider-discriminatory hardcoded list +
non-retryable short-circuit), and a qe-devils-advocate audit hardening
sweep closing 10 findings.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A post-Phase-4 devil's-advocate audit identified 10 issues. All addressed
with code fixes + tests. Some were real bugs (silent billing regression,
2 unwired MCP tools, 2 HybridRouter routing bugs); others were robustness
improvements (split singletons, silent errors, decorative type fields).
Findings → fixes (severity tagged in audit report):
1. CRITICAL — silent billing regression on upgrade
Users with ANTHROPIC_API_KEY in env would silently start paying for
Claude API calls on every Fleet operation. The CHANGELOG's "opt out
via code changes" wasn't sufficient — needed an env-only escape.
Fix: AQE_LLM_ROUTER_DISABLED=1 env kill-switch honored in both
QEKernelImpl._initializeLLMRouter and src/mcp/tools/base.ts
getSharedLLMRouter. Truthy values disable; 'false'/'0'/'no'/'off'/''
pass. Unified parse rule between kernel and MCP base.
2. HIGH — 2 unwired MCP standalone tools (audit said 3; QualityCriteria
isn't LLM-aware so excluded)
src/mcp/tools/test-generation/generate.ts:82 was calling
createTestGeneratorService(memory) without router. Same for
visual-accessibility/index.ts:177. Both now use the
createXxxServiceWithDependencies factories with { memory, llmRouter }.
3. CRITICAL — E2E only tested manual routing mode, not production
Production default is rule-based. Adding two new E2E tests for
rule-based mode SURFACED two real bugs in HybridRouter that I fixed:
3a. executeWithFallback hardcoded ['claude','openai','ollama'] as the
only fallback providers; gemini/openrouter/azure-openai/bedrock
users had no fallback even though ProviderManager can construct
them. Plus the user's defaultProvider wasn't auto-included in the
chain. Both fixed at src/shared/llm/router/hybrid-router.ts:612.
3b. A non-retryable error from one provider (e.g. Claude "invalid
x-api-key") aborted the entire fallback loop via `throw error`,
defeating the whole point of a fallback chain. Fixed: skip the
inter-attempt delay for non-retryable but continue to the next
fallback entry.
4. HIGH — MCPToolContext.llmRouter field was decorative
Added the field but no code populated it. Every getLLMRouter(context)
call fell through to getSharedLLMRouter().
Fix: added llmRouter and memory to BaseMCPTool.invoke() options;
qe-tool-bridge.ts and registry.ts now explicitly forward the shared
singleton via { llmRouter } at invoke time.
5. HIGH — 14 of 15 wired services had only STRUCTURAL verification, not
behavioral (only TestExecutor had a chat() round-trip test)
Fix: new tests/integration/llm-router-behavioral.test.ts asserts
isLLMAnalysisAvailable()===true (the exact gate the audit named) for
all 14 services + 2 full behavioral round-trips. 16 tests total.
6. HIGH — MCP and kernel built SEPARATE HybridRouter instances
Different cost trackers, caches, circuit breakers per side.
routing_economics/routing_metrics would only see half the traffic.
Fix: kernel.ts._initializeLLMRouter calls setSharedLLMRouter(router)
on init so both paths share one instance. kernel.dispose() calls
resetSharedLLMRouter() so the next kernel doesn't inherit the
previous instance.
7. HIGH — Silent error swallowing in _initializeLLMRouter
console.warn'd then continued without LLM, with no signal for
monitoring. A misconfigured provider would silently degrade with no
alert.
Fix: now publishes kernel.llm-router.init-failed (caught error) and
kernel.llm-router.init-no-provider (enabled=true but nothing
available) events on the event bus. Test confirms event delivery.
8. MEDIUM — Service count mismatch in docs (ADR/CHANGELOG said "14
services / 12 domains"; test enumerates 15 services / 11
LLM-enhanced domains)
Fix: ADR-043 addendum + CHANGELOG corrected to 15/11. Added
explicit service list and noted the 2 non-LLM domains
(enterprise-integration, coordination).
9. MEDIUM — aqe llm config --set apiKey=... silently dropped the key
stripApiKeys() returned a successfully-saved file without the key,
user thought it was saved.
Fix: CLI updateConfig() now throws a clear error pointing to env
vars BEFORE calling save. stripApiKeys also now logs a console.warn
when it strips anything, so programmatic callers see the strip.
10. LOW — Env vs disk precedence was counterintuitive and undocumented
Force-enable-on-env overrides explicit `enabled: false` on disk.
Fix: doc comment in config-store.applyEnvProviderDetection +
ADR-043 "Precedence & env-var conventions" section explains.
Verification
- npx tsc --noEmit: clean
- 191 unit + integration tests pass (16 new behavioral tests for #5,
17 wiring tests, gemini provider, config-store, llm-router-service,
kernel-llm-router with 17 tests including all kill-switch cases,
CLI persistence with 28 tests)
- 5 real-provider E2E tests pass (Gemini 2.5 Flash, OpenAI gpt-4o-mini,
OpenRouter, plus 2 rule-based-mode tests that previously failed and
now pass with the HybridRouter fallback fixes)
Refs ADR-043 audit findings (2026-05-21)
Builds on 94033e3 (Phase 1), be1e60d (Phase 2), c206c81 (Phase 3+4)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 3 — MCP tool path
- 8 standalone MCP tools (coverage-analyze, coverage-gaps, contract-validate,
chaos-inject, code-intel-analyze, defect-predict, learning-optimize) now
resolve the kernel HybridRouter via getLLMRouter(context) and pass it to
their service via the dependency-bag form.
- src/mcp/tools/base.ts: added getSharedLLMRouter() singleton, mirroring the
existing getSharedMemoryBackend() pattern. resetSharedLLMRouter() for
tests. Added MCPToolContext.llmRouter field so future tools (and the MCP
runtime when wiring per-request kernel context) can inject explicitly
rather than via the shared singleton.
Phase 4a — Mock-provider integration tests
- tests/integration/llm-router-wiring.test.ts (new, 17 tests).
- Per-domain wiring assertions (15): for each LLM-enhanced
plugin → coordinator → service path, asserts service.llmRouter ===
kernel.llmRouter after kernel boot with an injected mock provider.
This is the precise structural test for the gap the author's friend
flagged — a regression in any domain's wiring fails immediately.
- Behavioral end-to-end (1): TestExecutorService.analyzeFailuresWithLLM
actually invokes the mock provider; assert generateCalls counter
incremented and the response surfaced.
- Sanity (1): kernel.llmRouter is defined when a router was built.
Phase 4b — Real-provider E2E smoke test
- tests/e2e/llm-router-real-providers.test.ts (new, 3 tests).
- Gated by AQE_LLM_E2E=1; each provider auto-skips when its API key
isn't in env.
- Uses mode: 'manual' to bypass DEFAULT_QE_ROUTING_RULES (which would
otherwise route generic tasks to Claude even when defaultProvider
is set to something else — worth knowing).
- Verified live on 2026-05-21 against:
Gemini (gemini-2.5-flash) → "Hi!"
OpenAI (gpt-4o-mini) → "Hi!"
OpenRouter (openai/gpt-4o-mini) → "Hi!"
Claude intentionally skipped — the ANTHROPIC_API_KEY in the dev env
is a 24-char stub (real Anthropic keys are ~108 chars), would 401.
Phase 4c — Documentation
- docs/implementation/adrs/ADR-043-vendor-independent-llm.md: added
"Wiring Verified" status entry (2026-05-21) + full addendum documenting
the problem found, the four-phase resolution, the verification tests,
and the behavioral consequence for users.
- docs/implementation/adrs/ADR-051-agentic-flow-integration.md: added
"LLM-Enhancement Wiring Verified" status entry pointing to ADR-043's
addendum.
- CHANGELOG.md: added [Unreleased] section under Changed — behavioral,
explicitly noting the behavior shift (fleets with provider keys in env
will now start routing LLM calls through that provider), the opt-out
path (KernelConfig.llmRouter: { enabled: false }), and the persistence
fix for `aqe llm config --set`.
Test totals across all four phases on this branch
- 39 unit tests across 5 files (Phase 1)
- 17 integration tests in 1 file (Phase 4a)
- 3 real-provider E2E tests in 1 file (Phase 4b, AQE_LLM_E2E=1)
- 34 pre-existing kernel tests still pass (no regression)
- 22 pre-existing test-generation coordinator tests still pass (no regression)
- Clean tsc --noEmit across the full project
Refs ADR-043, ADR-051
Closes the wiring gap identified by user audit on 2026-05-21
Builds on 94033e3 (Phase 1) and be1e60d (Phase 2)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dragan Spiridonov