Let a downstream tool give a host it supervises its own AQE provider
identity, instead of forking the provider enum or projecting the host
onto an unrelated vendor.
Two surfaces:
- `externalProviders` in .agentic-qe/llm-config.json — declared as data,
so one config file is picked up by both the CLI and MCP bundles.
- registerProvider() from agentic-qe/shared/llm for in-process embedders.
Also:
- Open LLMProviderType to registered identities while keeping the shipped
set closed, so per-provider maps still fail the build when a built-in is
added. ALL_PROVIDER_TYPES is unchanged.
- Fix billing misattribution: any subscription-billed provider was reported
as running on the user's Claude Code subscription.
- Export ./shared/llm, an import path the module had documented but which
was never actually exported.
chore(release): bump version to v3.13.12
Also brings .claude-plugin/plugin.json (stale since v3.13.3) and
.claude/skills/skills-manifest.json fleetVersion (stale since v3.13.10)
back in step with package.json.
Closes the two security questions raised by the adversarial review that the
prosecutors never returned a ruling on, so I settled them with evidence.
1. Coverage measurement executes the analyzed repository.
Collecting real coverage means running the project's own tests: the JS/TS path
runs `npx vitest|jest|nyc` (pre-existing), and the Rust path added in this
release runs `cargo llvm-cov`, which compiles and runs test binaries, executes
`build.rs`, and honors any `runner`/linker directive in that repo's
`.cargo/config.toml`. For a trusted project that is exactly what the caller
wants; it is surprising for an operation that reads like a static query, and an
agent can be pointed at an untrusted checkout.
Execution stays the default — requiring opt-in would make measured coverage the
exceptional case and estimation the norm, inverting ADR-126's whole point — but
is now opt-out via `AQE_COVERAGE_NO_EXEC=1`, which disables ALL build-tool
execution. Existing coverage reports are still parsed; everything else degrades
to clearly-labelled static estimation with guidance explaining why.
Verified end-to-end through the real MCP server with a fake `cargo` shim on
PATH: switch off -> cargo invoked; switch on -> not invoked. The unit test
includes the same control, so it cannot pass vacuously.
2. Credential stripping in agentOverrides was already complete, but under-reported.
`sanitizeAgentOverrides` builds its result as an ALLOW-LIST, so any field that
is not provider/model/temperature/maxTokens/priority never survives — apiKey,
token, password and friends were all dropped already. Only the *warning* was
apiKey-specific, so a user who pasted a token got no feedback. The warning now
covers all credential-shaped keys, and a test asserts no warning ever echoes a
secret value.
ADR-126 gains a section recording the execution decision and what was rejected
(opt-in-by-default, sandboxing), plus the requirement that future language
delegations honor the same switch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four independent user-reported issues, each verified end-to-end through the real
MCP protocol server and adversarially reviewed via qe-court.
#565 — HIGH CVE reachable by consumers
`@huggingface/transformers` was an *optional* peerDependency, but npm >=7
auto-installs optional peers — `optional` only suppresses the error when the peer
cannot be resolved. Every `npm install agentic-qe` therefore pulled
onnxruntime-node -> adm-zip@0.5.x (GHSA-xcpc-8h2w-3j85), and no resolvable
version of that chain is unaffected. Dropped the peer (it stays a devDependency
and an opt-in for local in-process embeddings; both load sites already degrade
with an actionable message). Consumer audit: 4 HIGH -> 0.
#567 — test_generate_enhanced never invoked the LLM
The DI factory behind the task-executor path was registered as `(memory) => ...`,
so `llmRouter` never reached TestGeneratorService and the ADR-051 branch was
unreachable regardless of provider configuration. Threaded the router through the
registry -> resolveTestGeneratorService -> getTestGenerator (now async), honored
the documented `aiEnhancement` opt-out, and surfaced `llmEnhanced` /
`generationMode` / `note` so a silent fallback is diagnosable. `llmEnhanced` now
reflects what the LLM actually produced — a broken provider (401/timeout/empty)
no longer reports AI-enhanced output it did not produce. The CLI was unaffected
(it goes through the domain plugin, which was already wired).
#568 — per-agent provider routing had no on-disk surface
Added `agentOverrides` to `.agentic-qe/llm-config.json` with keyed-object merge,
validation that drops bad entries individually and strips leaked API keys, and
resolution ahead of the category defaults. Materialized the overrides into
routing rules the router actually evaluates, and threaded `agentType` through all
nine router-calling domain services — without it a parsed override built a rule
that never fired. Also added the missing `codex` entry to ALL_PROVIDER_TYPES,
which had made that provider unselectable through the router.
#569 — coverage_analyze_sublinear reported fabricated data for Rust
Coverage now carries provenance (`estimated` / `measured` / `coverageMethod`),
uncollected branch and function metrics report `null` instead of 0 or 100, gaps
derived from an estimate carry low confidence and an `estimated` flag, and Rust
crates are measured with `cargo llvm-cov` where available. Test code (`tests/`,
`benches/`, brace-matched `#[cfg(test)]` blocks) is excluded from production
coverage, and line numbers can no longer exceed the file length. Security scans
now state their analysis depth so "zero findings" is not read as "verified clean".
Adversarial review (qe-court, cross-vendor) found and this commit fixes: an
anchoring bug that dropped `latest_parser.rs` as a test file; brace matching that
counted braces inside string literals and comments; `#[cfg(test)]` on a
non-module item swallowing the following production function; crate detection
that skipped instrumentation for subdirectory targets, over-claimed nonexistent
targets, and hijacked polyglot subtrees; and a Cobertura fallback that rounded a
measured 47% to 0%.
ADR-125 (on-disk agent routing overrides) and ADR-126 (measurement provenance)
document the decisions; ADR-115 gains an amendment on optional peers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QE-Court convenes independent, cross-vendor AI reviewers against a delivery
(diff / PR / test-suite), makes them file charges from their own probe sets,
then forces any SHIP verdict to SURVIVE an escalating deeper review before it
stands (loop-until-dry overturn). Verdict is 3-valued — SHIP / REMAND / BLOCK —
with a human as final judge. It is the reference implementation of ADR-124; the
hosted sibling is the planned Cognitum /v1/qe/verdict endpoint.
The skill is an orchestration layer that COMPOSES existing assets — it does not
reimplement critics:
- Prosecutors: qe-devils-advocate, brutal-honesty-review, sherlock-review,
qe-security-scanner, qe-mutation-tester, and `codex exec review` (cross-vendor
GPT lens — the strongest form of writer≠juror).
- Kill round: src/verification/adversarial-verify. Jury: two-gate judge (ADR-119).
- Evidence: provenance tiers (ADR-121), signed receipt (ADR-118).
Configurable per-role model routing (config.json `routing`) across Claude /
Cognitum tiers / OpenRouter / Codex / local, all through the ADR-123 provider
layer (budget caps + receipts). Invariants: >=2 distinct vendors, jury vendor
!= writer, DoE-gated scoring (no un-validated "91/100").
Self-learning (ADR-124 M0.B/C): signed court records feed the flywheel; reproduced
charges become oracle-tier qe_patterns; an overturned SHIP is persisted as a
discriminator training pair for the frozen anchor + two-gate judge.
Ships via `aqe init` (added to V3_DOMAIN_SKILLS; auto-installed, not excluded)
and the assets/ npm mirror. Includes a runnable acceptance eval + a seeded-mutant
fixture (a boundary off-by-one a shallow reviewer rates SHIP; the overturn round
must flip it, and MUST regress to SHIP at overturnDepth=0). trust_tier 1 until the
eval runs green (>=5 oracle cases) → then tier 3.
Skill count: 75 -> 76 user-facing (totalQESkills 81 -> 82). Parity clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the structural gap from issue #557: AQE could only bill the paid
Anthropic API even for Pro/Max subscribers, and its budget caps were never
enforced.
- claude-code provider: run QE via `claude -p` on the user's subscription;
child env strips API-billing keys so it can't revert to API billing
- Enforced cross-process spend caps (--max-budget-usd / AQE_MAX_BUDGET_USD),
persisted to memory.db (llm_spend); enforced on the ProviderManager AND
HybridRouter paths; over-budget aborts with COST_LIMIT_EXCEEDED
- cognitum provider: OpenAI/Anthropic-compatible gateway with per-request
cost receipts, embeddings, and a server-side hard cap
- Billing-mode taxonomy + startup notice + `aqe health` LLM Billing section
- config-store: honor explicit `enabled: false` over env key presence;
AQE_LLM_PROVIDER selector
- Fix: `aqe eval run` no longer scores fabricated mock responses — real
provider by default, hard-error when none configured, `--mock` opt-in
- Consensus verification path prints a billing warning (deferred full port)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes and supersedes the 2026-07-09 directional DoE (which was 25/27 cells, had
no F-test, and — the reproducibility gap found this session — never persisted its raw
data). Re-run full under the CRIT-2 design (N=5 stochastic replicates of fixed item
A1-inRange, real within-cell error term), qwen local $0 + anthropic/claude-haiku-4.5
+ anthropic/claude-opus-4.8 via OpenRouter. $1.538 total, well under the $11 budget.
Results (real main-effects ANOVA, 135 observations, first real data the aggregator
has ever consumed — previously stub-only):
- scaffold: none 1.000 >> plan-and-solve 0.311 >> reflexion 0.067
F(2,108)=218.9, p=1.0e-38, 71.9% of variance — DOMINANT and strongly NEGATIVE.
Both scaffolds DROP. retort's "more harness lowers reliability", now overwhelming.
- model: qwen3-coder:30b 0.644 > haiku-4.5 0.400 > opus-4.8 0.333
F(2,108)=25.1, p=1.1e-9 — local qwen best & free; opus 5x+ cost = negative value.
(Complete 27/27 reverses the prior unequal-n haiku>opus>qwen ordering, significantly.)
- retrieval: off 0.480 > on 0.433 F(1,108)=1.5, p=0.22 NOT significant, slightly
negative — the ADR-118 flywheel feature-on-trial formally fails as `beads`.
EXPLAINED by the 2026-07-10 coupling finding: A1-inRange is a no-headroom item
the models know cold, so a retrieved exemplar has nothing to add.
- prompt: TDD 0.533 > ATDD 0.467 > neutral 0.378 F(2,108)=5.7, p=4.5e-3.
Changes:
- scripts/doe-run.ts: cloud path rewired from raw Anthropic fetch to OpenRouter
(vendor/model ids, opus-4.8 not opus-4-7, exact per-call cost from usage.cost).
- docs/implementation/doe-run-2026-07-10.jsonl: raw per-cell data persisted IN-REPO
(fixes the gap that lost the 2026-07-09 run's data).
- DOE-HARNESS-SCREEN-DESIGN.md §Results (DEFINITIVE) + ADR-122 status/history updated;
prior directional run retained, marked SUPERSEDED.
Verified: tsc 0 errors; $0 --stub and --smoke pipelines green before the metered run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Runs the CRIT-1 coupling experiment live on qwen3-coder:30b (Ollama, $0) and
characterizes EXACTLY when the ADR-118 flywheel's retrieval helps generated tests.
Result (N=5/condition, mutation-oracle graded, end-to-end "runs∧kills" response):
- POSITIVE CONTROL (POS-reserved — an arbitrary system-specific rule the model
cannot know a priori, a proxy for project-specific corpus knowledge):
NONE 0.00 / IRRELEVANT 0.00 / RELEVANT 1.00 → gain +1.00 COUPLING VALIDATED.
Without the exemplar qwen can't even write a passing test; with it, a perfect one.
- TEXTBOOK (B4-isLeapYear — the century/400 rule):
NONE 1.00 / RELEVANT 1.00 → NO HEADROOM (the model knows the trap cold).
Conclusion: the coupling mechanism is REAL (the harness demonstrably detects it, so
its nulls are live measurements) but HEADROOM-GATED — retrieval improves tests iff
the corpus supplies knowledge the base model lacks. Generic QE edge cases are
already known to code models (30b AND 7b), which is WHY the ADR-122 DoE rated
retrieval `beads`. This explains the prior honest nulls rather than contradicting them.
Getting here required fixing three measure-the-measurement traps that had produced a
FALSE "no coupling" verdict on the first run:
- baseline floored by 500-token truncation (→ 1400) and empty-input over-reach;
- spec leaked the trap (anchor inputUnderTest spells out the edge) → no headroom →
switched to UNDER-SPECIFIED specs graded against the frozen trap-encoding reference;
- metric conditioned on baseline, hiding retrieval's strongest effect (rescuing a
test that otherwise can't be written) → switched to end-to-end usable-quality.
Also:
- OpenRouter wired as the cloud path (vendor/model ids; anthropic/claude-haiku-4.5
et al. confirmed reachable), replacing the rate-limited Anthropic-direct fetch.
- coupled-anchor.test.ts: adds an honest AQE_LIVE_COUPLING-gated end-to-end test that
fails loudly (never fabricates a pass) rather than asserting the mechanism the
scripted unit test only assumes.
- ADR-118 status → "INFRASTRUCTURE WITH A CHARACTERIZED SIGNAL": the flywheel's
retrieval-tuning pays off iff qe_patterns holds knowledge the base model lacks.
Verified: tsc 0 errors; coupled-anchor 1 passed + 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the CRITICAL/HIGH/MED findings from the brutal-honesty review of the
learning-integrity cluster. The theme: the machinery was real but several claims
outran the evidence. This makes the claims honest and the evidence falsifiable.
CRIT-1 / HIGH-3 (flywheel gated vapor / toy anchor can't move):
- verification/anchors/qe-anchor-v2.json — 5 oracle-verified DISCRIMINATING items
(13 mutants), each with a trap edge a happy-path test misses, so the anchor CAN
actually move. v1's items were too easy to discriminate a retrieval effect.
- scripts/coupling-experiment.ts — falsifiable NONE/IRRELEVANT/RELEVANT controlled
test (N reps, temp>0, $0 mutation-oracle grading). Live run remains blocked on
model availability; the harness refuses to fabricate a verdict.
CRIT-2 (DoE "stats" were a range proxy):
- scripts/doe-run.ts — N real replicates of a FIXED item at temp=0.6 (was 5
different items x1 at temp=0), per-replicate seed, plus a deterministic $0 --stub.
- scripts/doe-aggregate.mjs — real main-effects factorial ANOVA: within-cell error
term, F-test via dependency-free regularized incomplete beta; beads verdict now
gates on p<0.05, not an eyeballed mean delta.
- tests/unit/scripts/doe-aggregate.test.ts — F-CDF + incomplete-beta vs closed
forms/R, planted-effect recovery (p<0.001), null factor (p>0.5), unbalanced cells.
HIGH-4 / MED-6 (over-claimed gate guarantees):
- ADR-120 gains a "Threat model and limits (honest)" section: gateReExecutes
catches a lying log, not a buggy gate (same-process seal+rule).
- ADR-118 status reframed to "infrastructure awaiting a validated signal."
- tests/unit/validation/rule-consistency.test.ts — property test scoping what the
gate actually guarantees (determinism, stable fingerprints, flipped-verdict).
- DOE-HARNESS-SCREEN-DESIGN.md — unequal-n note.
HIGH-5 / MED-7 (self-fulfilling test / brittle scripts):
- tests/unit/learning/mmr-scaling.test.ts — FALSIFIABLE regression for the real
O(n^4) scoreHeldOut hang the MMR_POOL_SIZE cap fixed (96s->420ms at n=300).
- coupled-anchor.test.ts de-faked: relabeled as a MECHANISM/wiring test, documents
that the scripted generator hardcodes the outcome and the real model was an
honest null — no longer readable as evidence that coupling helps.
- tests/unit/validation/sealed-hash-roundtrip.test.ts — regression for the
canonicalJson undefined-key seal bug.
Verification: tsc 0 errors; 33/33 new remediation tests + 290/290 full
validation+flywheel cluster green. No live model run (Ollama offline + Anthropic
account at usage limit) — coupling live-validation surfaced as the one honest
open item, not faked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the receipt-gated self-improvement discipline from the 2026-07 cross-repo
analysis of ruflo (flywheel), MetaHarness (gateReExecutes), and retort (two-gate
spec conformance + DoE). All DI'd and unit-tested (178 tests); run end-to-end on
real qe_patterns; every limitation honestly scoped.
- ADR-117 frozen oracle anchor: hash-pinned, constant-denominator eval set that
no promotion may regress; loader throws on drift (verification/anchors/qe-anchor-v1.json).
- ADR-118 receipt-gated QE-policy flywheel: generational loop over qe_patterns,
conjunctive frozen accept(), Ed25519-signed receipts + compounding lineage,
durable persistence (flywheel_receipts), corpus + semantic-retriever coupling.
- ADR-119 two-gate quality verdicts: mechanical gate + frontier judge, three-valued
(pass/fail/inconclusive) with preflight; new `aqe quality-gate` CLI + `qe/quality/gate` MCP tool.
- ADR-120 gateReExecutes: re-execute a versioned frozen rule on sealed inputs;
catches the A8-EXT forged-promote class; retrofit onto pattern-promote + dream-apply.
- ADR-121 provenance tiers: only oracle:test-exec evidence changes fleet behavior;
additive, conservative-backfill migration.
- ADR-122 DoE screen: model x prompt x retrieval x scaffold, run on real data
(25/27 cells, $1.25) — scaffolding (esp. reflexion) and retrieval degrade
reliability (the retort `beads` signature); simplest+cheapest stack wins.
- safe-db: AQE_DISABLE_WAL / AQE_JOURNAL_MODE env-gate (macOS virtiofs bind-mount
WAL corruption mitigation; default behavior unchanged).
Honest nulls documented: the flywheel lift is on a self-supervised proxy with the
anchor held flat; making the coupled anchor move needs harder items + a relevant
corpus (an ADR-117 re-freeze), not a retriever change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full remediation of docs/analysis/SYSTEM-INTEGRITY-AUDIT-2026-07-04.md's
P0-P2 findings, tracked in docs/plans/SYSTEM-INTEGRITY-REMEDIATION-GOAP-PLAN-2026-07-04.md.
Highlights:
- Dream engine: real pattern creation on insight apply, error/outcome
concept nodes from real ADR-110 data, and a follow-up fix ensuring the
concept refresh actually runs on the real automatic path (not just via
test wrappers); persistTaskOutcome now genuinely promotes pending
insights instead of blindly incrementing an applied counter
- Witness chain: Ed25519 signing wired via persistent key directory,
archival/append-after-archival bug fixed, CI verification gate added
- Mincut: real event-bus wiring (was a no-op stub) and scaffold-vertex
detection fixes
- GOAP: real domain-API dispatch executor replacing the always-mock one
- SONA: EWC++ cold-start deadlock fixed via independent request-count
persistence
- ADR-110 pattern-nulls wired into the real production capture path
- Daemon/snapshot visibility, TTL unit bugs, and 8 ADRs re-statused with
additive verification notes
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The proof-gate work cited only upstream RuVector 'ADR-194' — which is not an
AQE in-repo ADR. Adds ADR-116 (the AQE-side integration decision: port the
hash-chain to TS, opt-in/fail-soft/single-store wiring, and why NOT to depend
on the @ruvector npm packages — backed by the verified ruvllm mock-mode/demo
findings) and cross-references it from the four source/test files that cite
ADR-194, so the in-repo decision is discoverable from the code.
Comment/docs only — no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidate C4 diagram generation onto a single engine and expose it to
users via CLI and MCP, with a deterministic confidence gate and real
Knowledge-Graph-derived relationships.
- Consolidation (C1/C6): C4ModelService is the single render/analyze/store
engine; the bridge delegates and FAILS LOUD on a render error instead of
silently degrading. Duplicate inline generators removed.
- C2: real component relationships from the Knowledge Graph (AST import/call
edges) replace the naming heuristic; project-scoped via a new KG basePath
so repos outside cwd don't trip the path-traversal guard.
- C3: deterministic confidence gate (high/medium/low + reasons) on every
diagram — surfaces the detector's known limits instead of hiding them.
- C4/C5: `aqe code c4` CLI + `qe/code/c4` MCP tool (generate/search),
verified at MCP-CLI parity and through the protocol-server bridge.
- Search: generate persists embeddings (opt-in `enableC4Embeddings`, on for
MCP) so `qe/code/c4 search` returns hits in the standard flow.
- Fixed pre-existing lint in touched files (require->import, unused catches).
Docs: ADR-112 + docs/guides/c4-architecture-diagrams.md. tsc + lint clean;
C4 surface fully green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Statuses updated to reflect reality (were stale 'Proposed' despite
implementation): 107/108/110 Implemented; 105 Implemented w/ INFERRED
domain-finding follow-up; 106 Implemented (live matrix, native tiers
pass every cell); 109 Implemented infrastructure w/ corpus-scaling
follow-up. Gitignore transient run outputs, force-keeping canonical
evidence files.
Verified: full branch tsc clean; both new workflows valid YAML; 58
locally-runnable tests green; no .env or key material in any branch
commit (audited).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds EvidenceClass (EXECUTED|STATIC|INFERRED|CONJECTURE) and
EvidenceArtifact to the Finding schema, required at the type level.
requirements.ts factory defaults to STATIC (all 26 steps are
deterministic content analysis); pipeline crash-findings are EXECUTED
with the error output as artifact. Field is evidenceClass, not
evidence — StepResult.evidence: string[] already exists (ADR amended).
Verified: tsc --noEmit clean; validation-pipeline-handler tests 7/7
(MCP handler passes findings through, new field flows to MCP output).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six proposed ADRs from the universal-pattern-space QE assessment:
evidence-class labels on findings, behavioral safety evals per model
tier, shipped-agent invariant verification, benchmark lineage with
pre-registered rubrics (incl. antipattern list), interaction benchmark
for qualitative agents, kept-nulls negative pattern records.
Cross-checked against existing ADR corpus: Related fields reference
ADR-058/061/070/074/083/086/098/099/102/104; mid-task safety injection,
restraint probes, and null-vs-quarantine sequencing folded in from the
cross-check findings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aqe arena run: seeded tournaments where test strategies compete on REAL
fitness — built-in first-order operator mutants (string/comment/arrow-safe
enumeration) executed per strategy via node --test, real line coverage
from --experimental-test-coverage, and a deterministic suite-cost term.
Tournament emits a Wolfram-style pairwise competitive array; --evolve
hill-climbs the winner by seeded group flips with real re-evaluation.
Runs persist fail-soft to memory.db kv_store (namespace 'arena');
aqe arena list reads them back. Self-contained: no core changes, no new
dependencies; fixtures/arena-demo ships 4 test groups of varying kill
power over a branch-dense pricing module.
Reproducibility contract: same seed + fixture => byte-identical envelope
(verified E2E by diffing two runs), with informational.runtimesMs the
only excluded field. Honest finding during verification: the original
measured-runtime rank penalty was jitter-dominated at fixture scale and
flipped rankings between identical-seed runs — replaced with the
deterministic suite-size ratio (documented in ADR-104).
Phase 2 (documented): LLM-generated strategies frozen as artifacts,
winners distilled into qe-test-architect priors; MCP arena_run
registration (CLI and MCP share runArena() — parity by construction).
10 tests green (incl. real-engine reproducibility); tsc + eslint clean.
Part of #520 (improvement 7/7 — initiative complete).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
qcsd-development-swarm now runs as a deterministic workflow script
(.claude/workflows/qcsd-development-review.js) instead of prompt-driven
Task-tool fan-out:
- pipeline: one finder per quality dimension (TDD adherence, complexity,
coverage gaps; qe agents as agentType) -> 3 blind adversarial refuters
per finding -> deterministic synthesis, no agent editorializes the report
- Loki-mode constraints (ADR-074) encoded in refuter prompts: blind review
(bare claim + evidence only), anti-sycophancy (attack the claim;
uncertainty defaults to refuted), 3 distinct lenses
- majority kill (>= ceil(N/2) refuted votes); killed findings retained
with refutations for audit; output is finding-verdict@1 envelopes
(ADR-103)
- SKILL.md v1.1.0: execution.primary workflow, Task-tool fallback
documented (reports marked verification: none)
Validation run (wf_254f1271-022): 3 dimensions over src/, 48 agents,
15 raw findings -> 14 confirmed, 1 killed by unanimous refutation of a
factually-wrong claim — the adversarial gate demonstrably filtering a
false positive. Several confirmed findings target code added earlier in
this initiative (honest signal, queued as follow-ups).
Part of #520 (improvement 5/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Versioned, additive-only verdict envelopes for agent/MCP handoffs:
RiskDecision (quality gates), FindingVerdict (review findings with
adversarial refutations), CoverageGap (risk-weighted gaps).
- src/contracts/verdicts.ts: dependency-free validators (source of
truth) + mirrored draft-07 JSON Schemas; no zod/ajv runtime deps
(light-install posture preserved)
- quality_assess MCP boundary now attaches a validated riskDecision
envelope derived from the gate outcome (approve/block, escalate when
indeterminate) — additive, existing payload unchanged; direct and
wrapped handlers share the same mapToResult so MCP parity is inherent
- schemas/*.schema.json published via scripts/generate-verdict-schemas.mjs
for external ajv validation and workflow agent({schema}) enforcement
(consumed by improvement 5)
17 contract tests (goldens validate; mutations rejected with field-level
errors; builder always emits valid envelopes). E2E: envelope emitted by
the compiled boundary builder validates with stock ajv against the
published schema; mutated sample rejected. tsc + eslint clean.
Part of #520 (improvement 6/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Anthropic announced nested subagent support (depth=5) on 2026-06-09; the
runtime currently strips Task at parent->child spawn (upstream denylist,
empirically confirmed by our committed probe: level=1 status=NO_AGENT_TOOL,
matching ruflo ADR-147). This lands the readiness infrastructure that is
independently useful today and activates with zero changes when the gate
lifts:
- aqe hooks post-task gains --parent-agent-id / --depth (validated:
identifier charset, integer 0 <= d <= 32; depth=0 survives as 0);
provenance persisted to qe_trajectories.metadata_json (stage 1 — no
schema migration) and forwarded into ReasoningBank feedback so learning
can segment patterns per hierarchy level
- TaskExecution gains optional parentAgentId/depth (additive types)
- scripts/probe-nested-spawn-depth.mjs: single-shot empirical probe,
committed transcript under docs/probes/ — re-run after every
`claude update`; TASK_TOOL_LIVE is the signal for follow-up phases
- fresh-project fix found by the E2E: persistTaskOutcome assumed
captured_experiences/experience_applications exist, but they are
middleware-created — Stream B silently no-oped on fresh installs;
canonical DDL now ensured in the hook path
Deliberately deferred (documented in ADR-101 P1): tools: frontmatter
retrofit on shipped qe-*.md agents — Claude Code frontmatter is a
restrictive allowlist and would sever inherited MCP tool access;
needs maintainer sign-off + wildcard verification first.
8 unit tests green; E2E on a fresh temp project verified persistence,
rejections, and depth=0 boundary. tsc + eslint clean.
Part of #520 (improvement 4/7, Tier 2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Breaks runaway retry loops: when the same Bash command fails consecutively
(normalized signature, cross-process via a small JSON sidecar under
.agentic-qe/), pre-command warns at 3 failures and emits a BLOCK
recommendation with a recovery hint at 5. Advisory by default;
AQE_STRICT_TOOL_LOOP=1 upgrades the block to a PreToolUse deny. A
successful run resets the breaker; an open breaker grants a half-open
probe after 2 minutes; stale entries pruned after 30 minutes.
Mirrors the ADR-064 domain-circuit-breaker semantics at command level.
Sidecar instead of memory.db: pre-command deliberately runs without a DB
open, and each hook call is a fresh process so in-process state is
impossible. Fail-open everywhere (corrupt state file => allow).
11 unit tests green; E2E verified with real aqe hooks pre/post-command:
warn after 3, BLOCK after 5, strict-mode deny envelope, reset on success.
Part of #520 (improvement 3/7, Tier 1 complete).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The enableCache flag existed since ADR-011 but was never read: every
advisor/LLM call paid full input price. Now:
- ClaudeProvider.generate() sends `system` as a content-block array with
cache_control {type: ephemeral} when enableCache (default on); plain
string when disabled. Anthropic-native provider only (ADR-092 — other
providers untouched).
- cache_creation_input_tokens / cache_read_input_tokens surfaced into
TokenUsage.cacheCreationTokens/cacheReadTokens; totalTokens now counts
the full prompt (input_tokens alone is the uncached remainder).
- CostTracker bills cache reads at 0.1x and writes at 1.25x input rate
instead of ignoring them.
5 new provider tests assert the serialized request shape (cache_control
present when enabled, absent when disabled, no system field when no
prompt) and the usage mapping. 64/64 LLM tests green, tsc clean.
Part of #520 (improvement 2/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fable 5-era models emit large extended-thinking blocks inside task results
and step actions; embedding them verbatim contaminates the 384-dim pattern
vectors in memory.db and degrades HNSW retrieval.
- new scrubReasoningBlocks()/scrubReasoningDeep() (src/shared/reasoning-scrub.ts):
boundary-gated — only well-formed <think|thinking|reasoning|
REASONING_SCRATCHPAD>...</> pairs are removed; prose mentioning a tag
survives; clean input returned byte-identical
- applied at ingestion: TrajectoryTracker startTrajectory task,
recordStep action/result.data/result.error
- applied at distillation: ExperienceReplay.storeExperience task/strategy/
keyActions (insert + embedding text)
- applied at flow-forward: both init-time embedding backfills (legacy rows
cleaned as they get embedded — no memory.db rewrite) and retrieveSimilar
query embeddings (symmetric with stored vectors)
- 15 unit tests; E2E verified against a temp DB: trajectory + experience
rows persisted with all scratchpad blocks stripped
Part of #520 (improvement 1/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aqe memory/agent/task/team/pipeline CLI commands always failed with
"Fleet not initialized": autoInitialize() built kernel+queen in its own
CLIContext while the shared MCP handlers guard on the core-handlers
fleet-state singleton. New adoptExternalFleet() bridges the CLI's
components into that state (idempotent, never clobbers an existing
fleet), restoring MCP-CLI parity. Verified E2E: store/get/delete green.
- devcontainer: node_modules named-volume overlay + npm ci in postCreate
so container installs Linux natives without touching the host's macOS
binaries (takes effect on next rebuild; devcontainer users only)
- docs/setup: drop stale ruflo@3.5.18 pins (CLAUDE.md quick setup,
install-tools.sh) in favor of ruflo@3 + global `ruflo mcp start`
- docs: add Fable 5 / ruflo-parity improvement plan (tracked in #520)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Switch the cross-project knowledge hub from the cloud endpoint to a
locally-runnable Nagual instance, and add an optional local LLM judge
for scoring pattern quality.
- nagual-client.ts: REST client for a local `nagual serve` instance
(NAGUAL_URL, default http://localhost:3333); search/recordOutcome/
promotePattern are all fire-and-forget safe.
- local-judge-client.ts: Ollama-compatible judge (NAGUAL_JUDGE_URL/MODEL)
blending 0.7*outcome + 0.3*judge_score, with an ambiguous-zone review flag.
- task-hooks.ts: pre-task surfaces Nagual cross-project context for
complex tasks in relevant QE domains (best-effort, gated on env + length).
- settings.json: add NAGUAL_JUDGE_URL/MODEL; fix invalid mcp allow-rule
globs (mcp__ruflo__* / mcp__claude-flow__*) and add Bash(rm -rf:*) deny.
- docs: de-reference the cloud hub in favor of a local instance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo-health cleanup from an external code audit:
- eslint: ignore src/_archived (already excluded by tsconfig, so linting it only
produced 6 parserOptions.project errors).
- Align the architecture claim: src/index.ts and docs said "12"/"6" bounded
contexts; there are 13 domain dirs (matching package.json). Updated src/index.ts
(comment + ARCHITECTURE const) and the docs list to the actual 13.
- Apply stock `eslint --fix` (prefer-const, prefer-as-const, no-useless-escape,
no-regex-spaces) across 24 files.
Lint problems: 1072 -> 1031. Typecheck and build pass; no behavioral changes.
ADR-098 documents the durable behavioural decisions from the ruflo adoption
pass (#510): domain-score de-dilution (#2), EWC++ importance-weighted forgetting
(#6), contradiction suppression (#7), the two pre-existing bug fixes
(consecutive_failures, MCP -32601), and the explicit NEGLECTED decisions —
ruflo's Beta-priors (#2) and RaBitQ speedup (#8) did not transfer, and
multi-bundle packaging is deferred. AQE WH(Y) format, Accepted.
Marks the consecutive_failures (ae7c8b71) and MCP error-code (9b189115)
side-findings FIXED in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A post-Phase-4 devil's-advocate audit identified 10 issues. All addressed
with code fixes + tests. Some were real bugs (silent billing regression,
2 unwired MCP tools, 2 HybridRouter routing bugs); others were robustness
improvements (split singletons, silent errors, decorative type fields).
Findings → fixes (severity tagged in audit report):
1. CRITICAL — silent billing regression on upgrade
Users with ANTHROPIC_API_KEY in env would silently start paying for
Claude API calls on every Fleet operation. The CHANGELOG's "opt out
via code changes" wasn't sufficient — needed an env-only escape.
Fix: AQE_LLM_ROUTER_DISABLED=1 env kill-switch honored in both
QEKernelImpl._initializeLLMRouter and src/mcp/tools/base.ts
getSharedLLMRouter. Truthy values disable; 'false'/'0'/'no'/'off'/''
pass. Unified parse rule between kernel and MCP base.
2. HIGH — 2 unwired MCP standalone tools (audit said 3; QualityCriteria
isn't LLM-aware so excluded)
src/mcp/tools/test-generation/generate.ts:82 was calling
createTestGeneratorService(memory) without router. Same for
visual-accessibility/index.ts:177. Both now use the
createXxxServiceWithDependencies factories with { memory, llmRouter }.
3. CRITICAL — E2E only tested manual routing mode, not production
Production default is rule-based. Adding two new E2E tests for
rule-based mode SURFACED two real bugs in HybridRouter that I fixed:
3a. executeWithFallback hardcoded ['claude','openai','ollama'] as the
only fallback providers; gemini/openrouter/azure-openai/bedrock
users had no fallback even though ProviderManager can construct
them. Plus the user's defaultProvider wasn't auto-included in the
chain. Both fixed at src/shared/llm/router/hybrid-router.ts:612.
3b. A non-retryable error from one provider (e.g. Claude "invalid
x-api-key") aborted the entire fallback loop via `throw error`,
defeating the whole point of a fallback chain. Fixed: skip the
inter-attempt delay for non-retryable but continue to the next
fallback entry.
4. HIGH — MCPToolContext.llmRouter field was decorative
Added the field but no code populated it. Every getLLMRouter(context)
call fell through to getSharedLLMRouter().
Fix: added llmRouter and memory to BaseMCPTool.invoke() options;
qe-tool-bridge.ts and registry.ts now explicitly forward the shared
singleton via { llmRouter } at invoke time.
5. HIGH — 14 of 15 wired services had only STRUCTURAL verification, not
behavioral (only TestExecutor had a chat() round-trip test)
Fix: new tests/integration/llm-router-behavioral.test.ts asserts
isLLMAnalysisAvailable()===true (the exact gate the audit named) for
all 14 services + 2 full behavioral round-trips. 16 tests total.
6. HIGH — MCP and kernel built SEPARATE HybridRouter instances
Different cost trackers, caches, circuit breakers per side.
routing_economics/routing_metrics would only see half the traffic.
Fix: kernel.ts._initializeLLMRouter calls setSharedLLMRouter(router)
on init so both paths share one instance. kernel.dispose() calls
resetSharedLLMRouter() so the next kernel doesn't inherit the
previous instance.
7. HIGH — Silent error swallowing in _initializeLLMRouter
console.warn'd then continued without LLM, with no signal for
monitoring. A misconfigured provider would silently degrade with no
alert.
Fix: now publishes kernel.llm-router.init-failed (caught error) and
kernel.llm-router.init-no-provider (enabled=true but nothing
available) events on the event bus. Test confirms event delivery.
8. MEDIUM — Service count mismatch in docs (ADR/CHANGELOG said "14
services / 12 domains"; test enumerates 15 services / 11
LLM-enhanced domains)
Fix: ADR-043 addendum + CHANGELOG corrected to 15/11. Added
explicit service list and noted the 2 non-LLM domains
(enterprise-integration, coordination).
9. MEDIUM — aqe llm config --set apiKey=... silently dropped the key
stripApiKeys() returned a successfully-saved file without the key,
user thought it was saved.
Fix: CLI updateConfig() now throws a clear error pointing to env
vars BEFORE calling save. stripApiKeys also now logs a console.warn
when it strips anything, so programmatic callers see the strip.
10. LOW — Env vs disk precedence was counterintuitive and undocumented
Force-enable-on-env overrides explicit `enabled: false` on disk.
Fix: doc comment in config-store.applyEnvProviderDetection +
ADR-043 "Precedence & env-var conventions" section explains.
Verification
- npx tsc --noEmit: clean
- 191 unit + integration tests pass (16 new behavioral tests for #5,
17 wiring tests, gemini provider, config-store, llm-router-service,
kernel-llm-router with 17 tests including all kill-switch cases,
CLI persistence with 28 tests)
- 5 real-provider E2E tests pass (Gemini 2.5 Flash, OpenAI gpt-4o-mini,
OpenRouter, plus 2 rule-based-mode tests that previously failed and
now pass with the HybridRouter fallback fixes)
Refs ADR-043 audit findings (2026-05-21)
Builds on 94033e3 (Phase 1), be1e60d (Phase 2), c206c81 (Phase 3+4)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 3 — MCP tool path
- 8 standalone MCP tools (coverage-analyze, coverage-gaps, contract-validate,
chaos-inject, code-intel-analyze, defect-predict, learning-optimize) now
resolve the kernel HybridRouter via getLLMRouter(context) and pass it to
their service via the dependency-bag form.
- src/mcp/tools/base.ts: added getSharedLLMRouter() singleton, mirroring the
existing getSharedMemoryBackend() pattern. resetSharedLLMRouter() for
tests. Added MCPToolContext.llmRouter field so future tools (and the MCP
runtime when wiring per-request kernel context) can inject explicitly
rather than via the shared singleton.
Phase 4a — Mock-provider integration tests
- tests/integration/llm-router-wiring.test.ts (new, 17 tests).
- Per-domain wiring assertions (15): for each LLM-enhanced
plugin → coordinator → service path, asserts service.llmRouter ===
kernel.llmRouter after kernel boot with an injected mock provider.
This is the precise structural test for the gap the author's friend
flagged — a regression in any domain's wiring fails immediately.
- Behavioral end-to-end (1): TestExecutorService.analyzeFailuresWithLLM
actually invokes the mock provider; assert generateCalls counter
incremented and the response surfaced.
- Sanity (1): kernel.llmRouter is defined when a router was built.
Phase 4b — Real-provider E2E smoke test
- tests/e2e/llm-router-real-providers.test.ts (new, 3 tests).
- Gated by AQE_LLM_E2E=1; each provider auto-skips when its API key
isn't in env.
- Uses mode: 'manual' to bypass DEFAULT_QE_ROUTING_RULES (which would
otherwise route generic tasks to Claude even when defaultProvider
is set to something else — worth knowing).
- Verified live on 2026-05-21 against:
Gemini (gemini-2.5-flash) → "Hi!"
OpenAI (gpt-4o-mini) → "Hi!"
OpenRouter (openai/gpt-4o-mini) → "Hi!"
Claude intentionally skipped — the ANTHROPIC_API_KEY in the dev env
is a 24-char stub (real Anthropic keys are ~108 chars), would 401.
Phase 4c — Documentation
- docs/implementation/adrs/ADR-043-vendor-independent-llm.md: added
"Wiring Verified" status entry (2026-05-21) + full addendum documenting
the problem found, the four-phase resolution, the verification tests,
and the behavioral consequence for users.
- docs/implementation/adrs/ADR-051-agentic-flow-integration.md: added
"LLM-Enhancement Wiring Verified" status entry pointing to ADR-043's
addendum.
- CHANGELOG.md: added [Unreleased] section under Changed — behavioral,
explicitly noting the behavior shift (fleets with provider keys in env
will now start routing LLM calls through that provider), the opt-out
path (KernelConfig.llmRouter: { enabled: false }), and the persistence
fix for `aqe llm config --set`.
Test totals across all four phases on this branch
- 39 unit tests across 5 files (Phase 1)
- 17 integration tests in 1 file (Phase 4a)
- 3 real-provider E2E tests in 1 file (Phase 4b, AQE_LLM_E2E=1)
- 34 pre-existing kernel tests still pass (no regression)
- 22 pre-existing test-generation coordinator tests still pass (no regression)
- Clean tsc --noEmit across the full project
Refs ADR-043, ADR-051
Closes the wiring gap identified by user audit on 2026-05-21
Builds on 94033e3 (Phase 1) and be1e60d (Phase 2)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes 11 findings from a post-merge audit of the external embedder endpoint
implementation. The "no silent hash fallback" correctness promise was being
contradicted by two callers; the connect timeout was actually an idle timeout;
the identity probe could be bypassed in several paths; and identity drift
across runs was undetectable. All hardened.
Critical/High fixes:
- sqlite-persistence backfill no longer falls back to hash embeddings when the
endpoint is configured; re-raises to protect the HNSW index from non-comparable
vectors (was poisoning silently on every endpoint blip).
- qe-reasoning-bank.embed() propagates the throw under isUsingEndpoint() instead
of silently substituting a hash vector.
- EmbedderEndpointClient.embed() now gates on probe() — lazy probe on first use
with in-flight dedup so 20+ concurrent embeds share one probe.
- Circuit breaker recovery (window expires) clears cached identity so the next
embed re-probes — endpoint restarts often coincide with model swaps.
- True TCP-connect timeout enforced at the socket layer (was req.setTimeout,
which is an idle-after-write timeout — SYN blackholes hung ~75s instead of 5s).
- Endpoint identity persisted to memory.db kv_store (_system namespace) — cross-
run fingerprint drift now surfaces as a loud warning on next boot.
Medium fixes:
- encoding_format: 'float' pinned in request; base64 responses fail loud.
- TLS knobs (ca, cert, key, rejectUnauthorized, servername) passthrough for
self-hosted https endpoints without NODE_TLS_REJECT_UNAUTHORIZED=0.
- URL userinfo stripped + warned at parse; redacted from all error messages
and logs.
- Embedding cache key namespaced by mode (endpoint:<fingerprint>:<text> vs
inproc:<text>) — cross-mode collisions structurally impossible.
Tests:
- 33 unit tests pass (was 23): adds probe-gate, breaker re-probe, connect
timeout, DNS failure, concurrent embed dedup, base64 fail-loud, URL
redaction, encoding_format wire format.
- 6 live integration tests against real llama-server (all-MiniLM-L6-v2.Q8_0.gguf):
probe, single embed, batch, lazy probe gate, dim mismatch, 50-input batch.
Replaces the prior "9ms in-process fake" with honest numbers (cold 15ms,
warm 2ms against localhost llama.cpp).
- New tests/integration/embedder-endpoint-llamacpp.test.ts is gated on
AQE_LLAMA_EMBED_URL so it skips cleanly in CI without the binary.
ADR-097 updated with the per-caller audit table (12 writers verified
non-poisoning) and "drop-in for 6 providers" claim softened to "verified
against llama-server; OpenAI shape compat expected but unverified" per
provider.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dragan Spiridonov