mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
release/v3.13.3
82 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dragan Spiridonov
|
e5a7719aa7 |
feat(#569): add AQE_COVERAGE_NO_EXEC opt-out; broaden credential stripping (#568)
Closes the two security questions raised by the adversarial review that the prosecutors never returned a ruling on, so I settled them with evidence. 1. Coverage measurement executes the analyzed repository. Collecting real coverage means running the project's own tests: the JS/TS path runs `npx vitest|jest|nyc` (pre-existing), and the Rust path added in this release runs `cargo llvm-cov`, which compiles and runs test binaries, executes `build.rs`, and honors any `runner`/linker directive in that repo's `.cargo/config.toml`. For a trusted project that is exactly what the caller wants; it is surprising for an operation that reads like a static query, and an agent can be pointed at an untrusted checkout. Execution stays the default — requiring opt-in would make measured coverage the exceptional case and estimation the norm, inverting ADR-126's whole point — but is now opt-out via `AQE_COVERAGE_NO_EXEC=1`, which disables ALL build-tool execution. Existing coverage reports are still parsed; everything else degrades to clearly-labelled static estimation with guidance explaining why. Verified end-to-end through the real MCP server with a fake `cargo` shim on PATH: switch off -> cargo invoked; switch on -> not invoked. The unit test includes the same control, so it cannot pass vacuously. 2. Credential stripping in agentOverrides was already complete, but under-reported. `sanitizeAgentOverrides` builds its result as an ALLOW-LIST, so any field that is not provider/model/temperature/maxTokens/priority never survives — apiKey, token, password and friends were all dropped already. Only the *warning* was apiKey-specific, so a user who pasted a token got no feedback. The warning now covers all credential-shaped keys, and a test asserts no warning ever echoes a secret value. ADR-126 gains a section recording the execution decision and what was rejected (opt-in-by-default, sandboxing), plus the requirement that future language delegations honor the same switch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
9b6de39792 |
fix: resolve issues #565, #567, #568, #569 (deps, LLM wiring, agent routing, coverage honesty)
Four independent user-reported issues, each verified end-to-end through the real MCP protocol server and adversarially reviewed via qe-court. #565 — HIGH CVE reachable by consumers `@huggingface/transformers` was an *optional* peerDependency, but npm >=7 auto-installs optional peers — `optional` only suppresses the error when the peer cannot be resolved. Every `npm install agentic-qe` therefore pulled onnxruntime-node -> adm-zip@0.5.x (GHSA-xcpc-8h2w-3j85), and no resolvable version of that chain is unaffected. Dropped the peer (it stays a devDependency and an opt-in for local in-process embeddings; both load sites already degrade with an actionable message). Consumer audit: 4 HIGH -> 0. #567 — test_generate_enhanced never invoked the LLM The DI factory behind the task-executor path was registered as `(memory) => ...`, so `llmRouter` never reached TestGeneratorService and the ADR-051 branch was unreachable regardless of provider configuration. Threaded the router through the registry -> resolveTestGeneratorService -> getTestGenerator (now async), honored the documented `aiEnhancement` opt-out, and surfaced `llmEnhanced` / `generationMode` / `note` so a silent fallback is diagnosable. `llmEnhanced` now reflects what the LLM actually produced — a broken provider (401/timeout/empty) no longer reports AI-enhanced output it did not produce. The CLI was unaffected (it goes through the domain plugin, which was already wired). #568 — per-agent provider routing had no on-disk surface Added `agentOverrides` to `.agentic-qe/llm-config.json` with keyed-object merge, validation that drops bad entries individually and strips leaked API keys, and resolution ahead of the category defaults. Materialized the overrides into routing rules the router actually evaluates, and threaded `agentType` through all nine router-calling domain services — without it a parsed override built a rule that never fired. Also added the missing `codex` entry to ALL_PROVIDER_TYPES, which had made that provider unselectable through the router. #569 — coverage_analyze_sublinear reported fabricated data for Rust Coverage now carries provenance (`estimated` / `measured` / `coverageMethod`), uncollected branch and function metrics report `null` instead of 0 or 100, gaps derived from an estimate carry low confidence and an `estimated` flag, and Rust crates are measured with `cargo llvm-cov` where available. Test code (`tests/`, `benches/`, brace-matched `#[cfg(test)]` blocks) is excluded from production coverage, and line numbers can no longer exceed the file length. Security scans now state their analysis depth so "zero findings" is not read as "verified clean". Adversarial review (qe-court, cross-vendor) found and this commit fixes: an anchoring bug that dropped `latest_parser.rs` as a test file; brace matching that counted braces inside string literals and comments; `#[cfg(test)]` on a non-module item swallowing the following production function; crate detection that skipped instrumentation for subdirectory targets, over-claimed nonexistent targets, and hijacked polyglot subtrees; and a Cobertura fallback that rounded a measured 47% to 0%. ADR-125 (on-disk agent routing overrides) and ADR-126 (measurement provenance) document the decisions; ADR-115 gains an amendment on optional peers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
5f323ec77c |
feat(skills): add qe-court adversarial review skill (ADR-124)
QE-Court convenes independent, cross-vendor AI reviewers against a delivery
(diff / PR / test-suite), makes them file charges from their own probe sets,
then forces any SHIP verdict to SURVIVE an escalating deeper review before it
stands (loop-until-dry overturn). Verdict is 3-valued — SHIP / REMAND / BLOCK —
with a human as final judge. It is the reference implementation of ADR-124; the
hosted sibling is the planned Cognitum /v1/qe/verdict endpoint.
The skill is an orchestration layer that COMPOSES existing assets — it does not
reimplement critics:
- Prosecutors: qe-devils-advocate, brutal-honesty-review, sherlock-review,
qe-security-scanner, qe-mutation-tester, and `codex exec review` (cross-vendor
GPT lens — the strongest form of writer≠juror).
- Kill round: src/verification/adversarial-verify. Jury: two-gate judge (ADR-119).
- Evidence: provenance tiers (ADR-121), signed receipt (ADR-118).
Configurable per-role model routing (config.json `routing`) across Claude /
Cognitum tiers / OpenRouter / Codex / local, all through the ADR-123 provider
layer (budget caps + receipts). Invariants: >=2 distinct vendors, jury vendor
!= writer, DoE-gated scoring (no un-validated "91/100").
Self-learning (ADR-124 M0.B/C): signed court records feed the flywheel; reproduced
charges become oracle-tier qe_patterns; an overturned SHIP is persisted as a
discriminator training pair for the frozen anchor + two-gate judge.
Ships via `aqe init` (added to V3_DOMAIN_SKILLS; auto-installed, not excluded)
and the assets/ npm mirror. Includes a runnable acceptance eval + a seeded-mutant
fixture (a boundary off-by-one a shallow reviewer rates SHIP; the overturn round
must flip it, and MUST regress to SHIP at overturnDepth=0). trust_tier 1 until the
eval runs green (>=5 oracle cases) → then tier 3.
Skill count: 75 -> 76 user-facing (totalQESkills 81 -> 82). Parity clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
b9a4b4c6e8 |
feat(llm): billing-aware execution — subscription provider, enforced budgets, real eval (ADR-123)
Closes the structural gap from issue #557: AQE could only bill the paid Anthropic API even for Pro/Max subscribers, and its budget caps were never enforced. - claude-code provider: run QE via `claude -p` on the user's subscription; child env strips API-billing keys so it can't revert to API billing - Enforced cross-process spend caps (--max-budget-usd / AQE_MAX_BUDGET_USD), persisted to memory.db (llm_spend); enforced on the ProviderManager AND HybridRouter paths; over-budget aborts with COST_LIMIT_EXCEEDED - cognitum provider: OpenAI/Anthropic-compatible gateway with per-request cost receipts, embeddings, and a server-side hard cap - Billing-mode taxonomy + startup notice + `aqe health` LLM Billing section - config-store: honor explicit `enabled: false` over env key presence; AQE_LLM_PROVIDER selector - Fix: `aqe eval run` no longer scores fabricated mock responses — real provider by default, hard-error when none configured, `--mock` opt-in - Consensus verification path prints a billing warning (deferred full port) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
7537fefb4e |
feat(doe): definitive 27/27 DoE re-run via OpenRouter — real F-tests + persisted raw data
Completes and supersedes the 2026-07-09 directional DoE (which was 25/27 cells, had
no F-test, and — the reproducibility gap found this session — never persisted its raw
data). Re-run full under the CRIT-2 design (N=5 stochastic replicates of fixed item
A1-inRange, real within-cell error term), qwen local $0 + anthropic/claude-haiku-4.5
+ anthropic/claude-opus-4.8 via OpenRouter. $1.538 total, well under the $11 budget.
Results (real main-effects ANOVA, 135 observations, first real data the aggregator
has ever consumed — previously stub-only):
- scaffold: none 1.000 >> plan-and-solve 0.311 >> reflexion 0.067
F(2,108)=218.9, p=1.0e-38, 71.9% of variance — DOMINANT and strongly NEGATIVE.
Both scaffolds DROP. retort's "more harness lowers reliability", now overwhelming.
- model: qwen3-coder:30b 0.644 > haiku-4.5 0.400 > opus-4.8 0.333
F(2,108)=25.1, p=1.1e-9 — local qwen best & free; opus 5x+ cost = negative value.
(Complete 27/27 reverses the prior unequal-n haiku>opus>qwen ordering, significantly.)
- retrieval: off 0.480 > on 0.433 F(1,108)=1.5, p=0.22 NOT significant, slightly
negative — the ADR-118 flywheel feature-on-trial formally fails as `beads`.
EXPLAINED by the 2026-07-10 coupling finding: A1-inRange is a no-headroom item
the models know cold, so a retrieved exemplar has nothing to add.
- prompt: TDD 0.533 > ATDD 0.467 > neutral 0.378 F(2,108)=5.7, p=4.5e-3.
Changes:
- scripts/doe-run.ts: cloud path rewired from raw Anthropic fetch to OpenRouter
(vendor/model ids, opus-4.8 not opus-4-7, exact per-call cost from usage.cost).
- docs/implementation/doe-run-2026-07-10.jsonl: raw per-cell data persisted IN-REPO
(fixes the gap that lost the 2026-07-09 run's data).
- DOE-HARNESS-SCREEN-DESIGN.md §Results (DEFINITIVE) + ADR-122 status/history updated;
prior directional run retained, marked SUPERSEDED.
Verified: tsc 0 errors; $0 --stub and --smoke pipelines green before the metered run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
68c8914f92 |
feat(learning-integrity): live coupling validation closes CRIT-1 — mechanism real but headroom-gated
Runs the CRIT-1 coupling experiment live on qwen3-coder:30b (Ollama, $0) and
characterizes EXACTLY when the ADR-118 flywheel's retrieval helps generated tests.
Result (N=5/condition, mutation-oracle graded, end-to-end "runs∧kills" response):
- POSITIVE CONTROL (POS-reserved — an arbitrary system-specific rule the model
cannot know a priori, a proxy for project-specific corpus knowledge):
NONE 0.00 / IRRELEVANT 0.00 / RELEVANT 1.00 → gain +1.00 COUPLING VALIDATED.
Without the exemplar qwen can't even write a passing test; with it, a perfect one.
- TEXTBOOK (B4-isLeapYear — the century/400 rule):
NONE 1.00 / RELEVANT 1.00 → NO HEADROOM (the model knows the trap cold).
Conclusion: the coupling mechanism is REAL (the harness demonstrably detects it, so
its nulls are live measurements) but HEADROOM-GATED — retrieval improves tests iff
the corpus supplies knowledge the base model lacks. Generic QE edge cases are
already known to code models (30b AND 7b), which is WHY the ADR-122 DoE rated
retrieval `beads`. This explains the prior honest nulls rather than contradicting them.
Getting here required fixing three measure-the-measurement traps that had produced a
FALSE "no coupling" verdict on the first run:
- baseline floored by 500-token truncation (→ 1400) and empty-input over-reach;
- spec leaked the trap (anchor inputUnderTest spells out the edge) → no headroom →
switched to UNDER-SPECIFIED specs graded against the frozen trap-encoding reference;
- metric conditioned on baseline, hiding retrieval's strongest effect (rescuing a
test that otherwise can't be written) → switched to end-to-end usable-quality.
Also:
- OpenRouter wired as the cloud path (vendor/model ids; anthropic/claude-haiku-4.5
et al. confirmed reachable), replacing the rate-limited Anthropic-direct fetch.
- coupled-anchor.test.ts: adds an honest AQE_LIVE_COUPLING-gated end-to-end test that
fails loudly (never fabricates a pass) rather than asserting the mechanism the
scripted unit test only assumes.
- ADR-118 status → "INFRASTRUCTURE WITH A CHARACTERIZED SIGNAL": the flywheel's
retrieval-tuning pays off iff qe_patterns holds knowledge the base model lacks.
Verified: tsc 0 errors; coupled-anchor 1 passed + 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
df183d1ea4 |
fix(learning-integrity): remediate brutal-honesty-review findings on ADR-117..122
Addresses the CRITICAL/HIGH/MED findings from the brutal-honesty review of the learning-integrity cluster. The theme: the machinery was real but several claims outran the evidence. This makes the claims honest and the evidence falsifiable. CRIT-1 / HIGH-3 (flywheel gated vapor / toy anchor can't move): - verification/anchors/qe-anchor-v2.json — 5 oracle-verified DISCRIMINATING items (13 mutants), each with a trap edge a happy-path test misses, so the anchor CAN actually move. v1's items were too easy to discriminate a retrieval effect. - scripts/coupling-experiment.ts — falsifiable NONE/IRRELEVANT/RELEVANT controlled test (N reps, temp>0, $0 mutation-oracle grading). Live run remains blocked on model availability; the harness refuses to fabricate a verdict. CRIT-2 (DoE "stats" were a range proxy): - scripts/doe-run.ts — N real replicates of a FIXED item at temp=0.6 (was 5 different items x1 at temp=0), per-replicate seed, plus a deterministic $0 --stub. - scripts/doe-aggregate.mjs — real main-effects factorial ANOVA: within-cell error term, F-test via dependency-free regularized incomplete beta; beads verdict now gates on p<0.05, not an eyeballed mean delta. - tests/unit/scripts/doe-aggregate.test.ts — F-CDF + incomplete-beta vs closed forms/R, planted-effect recovery (p<0.001), null factor (p>0.5), unbalanced cells. HIGH-4 / MED-6 (over-claimed gate guarantees): - ADR-120 gains a "Threat model and limits (honest)" section: gateReExecutes catches a lying log, not a buggy gate (same-process seal+rule). - ADR-118 status reframed to "infrastructure awaiting a validated signal." - tests/unit/validation/rule-consistency.test.ts — property test scoping what the gate actually guarantees (determinism, stable fingerprints, flipped-verdict). - DOE-HARNESS-SCREEN-DESIGN.md — unequal-n note. HIGH-5 / MED-7 (self-fulfilling test / brittle scripts): - tests/unit/learning/mmr-scaling.test.ts — FALSIFIABLE regression for the real O(n^4) scoreHeldOut hang the MMR_POOL_SIZE cap fixed (96s->420ms at n=300). - coupled-anchor.test.ts de-faked: relabeled as a MECHANISM/wiring test, documents that the scripted generator hardcodes the outcome and the real model was an honest null — no longer readable as evidence that coupling helps. - tests/unit/validation/sealed-hash-roundtrip.test.ts — regression for the canonicalJson undefined-key seal bug. Verification: tsc 0 errors; 33/33 new remediation tests + 290/290 full validation+flywheel cluster green. No live model run (Ollama offline + Anthropic account at usage limit) — coupling live-validation surfaced as the one honest open item, not faked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
0a8254f7a9 |
feat(learning): ADR-117..122 learning-integrity cluster (frozen anchor, two-gate judge, gate-reexecute, provenance tiers, receipt-gated flywheel, DoE)
Adopts the receipt-gated self-improvement discipline from the 2026-07 cross-repo analysis of ruflo (flywheel), MetaHarness (gateReExecutes), and retort (two-gate spec conformance + DoE). All DI'd and unit-tested (178 tests); run end-to-end on real qe_patterns; every limitation honestly scoped. - ADR-117 frozen oracle anchor: hash-pinned, constant-denominator eval set that no promotion may regress; loader throws on drift (verification/anchors/qe-anchor-v1.json). - ADR-118 receipt-gated QE-policy flywheel: generational loop over qe_patterns, conjunctive frozen accept(), Ed25519-signed receipts + compounding lineage, durable persistence (flywheel_receipts), corpus + semantic-retriever coupling. - ADR-119 two-gate quality verdicts: mechanical gate + frontier judge, three-valued (pass/fail/inconclusive) with preflight; new `aqe quality-gate` CLI + `qe/quality/gate` MCP tool. - ADR-120 gateReExecutes: re-execute a versioned frozen rule on sealed inputs; catches the A8-EXT forged-promote class; retrofit onto pattern-promote + dream-apply. - ADR-121 provenance tiers: only oracle:test-exec evidence changes fleet behavior; additive, conservative-backfill migration. - ADR-122 DoE screen: model x prompt x retrieval x scaffold, run on real data (25/27 cells, $1.25) — scaffolding (esp. reflexion) and retrieval degrade reliability (the retort `beads` signature); simplest+cheapest stack wins. - safe-db: AQE_DISABLE_WAL / AQE_JOURNAL_MODE env-gate (macOS virtiofs bind-mount WAL corruption mitigation; default behavior unchanged). Honest nulls documented: the flywheel lift is on a self-supervised proxy with the anchor held flat; making the coupled anchor move needs harder items + a relevant corpus (an ADR-117 re-freeze), not a retriever change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
53ee377aec |
fix(learning): close system-integrity audit gaps in dream engine, witness chain, mincut, GOAP executor, and SONA
Full remediation of docs/analysis/SYSTEM-INTEGRITY-AUDIT-2026-07-04.md's P0-P2 findings, tracked in docs/plans/SYSTEM-INTEGRITY-REMEDIATION-GOAP-PLAN-2026-07-04.md. Highlights: - Dream engine: real pattern creation on insight apply, error/outcome concept nodes from real ADR-110 data, and a follow-up fix ensuring the concept refresh actually runs on the real automatic path (not just via test wrappers); persistTaskOutcome now genuinely promotes pending insights instead of blindly incrementing an applied counter - Witness chain: Ed25519 signing wired via persistent key directory, archival/append-after-archival bug fixed, CI verification gate added - Mincut: real event-bus wiring (was a no-op stub) and scaffold-vertex detection fixes - GOAP: real domain-API dispatch executor replacing the always-mock one - SONA: EWC++ cold-start deadlock fixed via independent request-count persistence - ADR-110 pattern-nulls wired into the real production capture path - Daemon/snapshot visibility, TTL unit bugs, and 8 ADRs re-statused with additive verification notes Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
9ca2c128ef | Merge remote-tracking branch 'origin/main' into feat/local-model-improvements | ||
|
Dragan Spiridonov
|
f2742c8651 |
docs(adr): add ADR-116 for proof-gate + cross-ref from code
The proof-gate work cited only upstream RuVector 'ADR-194' — which is not an AQE in-repo ADR. Adds ADR-116 (the AQE-side integration decision: port the hash-chain to TS, opt-in/fail-soft/single-store wiring, and why NOT to depend on the @ruvector npm packages — backed by the verified ruvllm mock-mode/demo findings) and cross-references it from the four source/test files that cite ADR-194, so the in-repo decision is discoverable from the code. Comment/docs only — no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
6dcb919aee |
feat(deps): dependency-security system + fix grpc-js HIGH (ADR-115)
Closes the gap that let @grpc/grpc-js@1.14.3 (CVE-2026-48068/48069, HIGH) sit undetected: every existing dependency check only fires on dep-file changes, so a CVE disclosed against a frozen pin was invisible. Adds: - scripts/dependency-audit.mjs in-repo + consumer-POV audit, severity gate - scripts/dependency-update.mjs security-first in-range updater with a guard that refuses to drop optional @ruvector platform deps (the Dependabot lockfile bug) - dependency-audit.yml daily consumer-POV scan -> tracking issue - dependency-update.yml weekly in-range security PR (gh CLI) - docs/guides/dependency-security-policy.md + ADR-115 - tests/unit/scripts/dependency-security.test.ts (11 hermetic tests) - npm scripts: deps:audit[:consumer], deps:update[:all] Also bumps @grpc/grpc-js 1.14.3 -> 1.14.4 in-range (HIGH 1 -> 0); optional platform deps preserved (90 -> 90). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
af4140efb9 |
feat(quality): oracle evals, durable-first tests, conservation guard + provider fixes
Apply Phoenix Architecture principles to AQE — regenerate internals, conserve the interface. ADR-113 (evals are oracles, durable-first tests, mutation-score gate): - src/validation/oracle-eval.ts grades a generated test by running it against a reference impl + operator mutants (reuses qe-arena, ADR-104); rejects assertion-less output - oracle mode wired into scripts/run-skill-eval.ts (skips under the simulating runner; runs live) - qe-test-generation rewritten durable-first; its eval flipped to oracle - src/feedback/regenerability-gate.ts: mutation-score + regenerability gate (warn-default, opt-in block) - live-validated: Claude sonnet-4-6 clears the oracle 5/5; weak local-model output correctly fails ADR-114 (conservation layer, Phoenix essay 14): - src/validation/conservation-guard.ts + scripts/conservation-guard.ts guard CLI commands, output schemas, and dashboard API against breaking removals (deprecation registry exempts) - policy doc + baselines in verification/conservation/; wired into invariant-check.yml non-blocking LLM provider fixes: - OpenRouterProvider.type was 'openai' but registered under 'openrouter' -> ProviderManager failover never found it; fixed to 'openrouter' end-to-end - Gemini default model gemini-1.5-pro was retired -> gemini-2.5-flash; models/pricing refreshed from the live Gemini ListModels API - scripts/provider-health.ts (npm run providers:health) Cheaper-model eval lanes: - scripts/oracle-model-bench.ts + openrouter-models.ts; openai/gpt-oss-120b clears the oracle ~50-70x cheaper than Claude Sonnet; guide in docs/guides/cheaper-model-eval-lanes.md Verification: npm run build ok; 115 new+touched tests pass; 724 LLM tests pass; lint + parity clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
9fd18ed00a |
feat(code-intelligence): first-class C4 architecture diagrams (ADR-112)
Consolidate C4 diagram generation onto a single engine and expose it to users via CLI and MCP, with a deterministic confidence gate and real Knowledge-Graph-derived relationships. - Consolidation (C1/C6): C4ModelService is the single render/analyze/store engine; the bridge delegates and FAILS LOUD on a render error instead of silently degrading. Duplicate inline generators removed. - C2: real component relationships from the Knowledge Graph (AST import/call edges) replace the naming heuristic; project-scoped via a new KG basePath so repos outside cwd don't trip the path-traversal guard. - C3: deterministic confidence gate (high/medium/low + reasons) on every diagram — surfaces the detector's known limits instead of hiding them. - C4/C5: `aqe code c4` CLI + `qe/code/c4` MCP tool (generate/search), verified at MCP-CLI parity and through the protocol-server bridge. - Search: generate persists embeddings (opt-in `enableC4Embeddings`, on for MCP) so `qe/code/c4 search` returns hits in the standard flow. - Fixed pre-existing lint in touched files (require->import, unused catches). Docs: ADR-112 + docs/guides/c4-architecture-diagrams.md. tsc + lint clean; C4 surface fully green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
41bfa05bee |
docs(metaharness): ADR-111 + D3 fixture-diverse gate + plan-05 reconciliation + A11 judge harness
- ADR-111 (Accepted, scoped to the escalation lane): cheap-replaces-frontier = G-ABORT; the asymmetric escalation lane is the committed product. - D3 gate harness now corpus-driven (docs/metaharness/prototype/d3-corpus/: pricing, strings, stats, validate, timefmt). Fixture-diverse n=30 confirmation: §12 best-of-k and §11 coder-binds hold 5/5 modules; escalation lane ≈ frontier within composite SE. - d3-judge.mjs: A11 writer≠evaluator LLM-judge discriminator precision-validation (writer=local qwen3:30b, judge=OpenRouter deepseek-v4-flash, no Anthropic), per ADR-183. - docs 06: D3 progression + honesty caveats (adversarial review); 00: exec-summary brought current; 05: status reconciliation (A1 done, A6 retired, A3 resolved) + oracle-arc actions A11-A13 from Ruv's ADR-178/182/183/184. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
772af839a4 |
docs(adr): mark ADR-105..110 Implemented; gitignore transient eval outputs (issue #522)
Statuses updated to reflect reality (were stale 'Proposed' despite implementation): 107/108/110 Implemented; 105 Implemented w/ INFERRED domain-finding follow-up; 106 Implemented (live matrix, native tiers pass every cell); 109 Implemented infrastructure w/ corpus-scaling follow-up. Gitignore transient run outputs, force-keeping canonical evidence files. Verified: full branch tsc clean; both new workflows valid YAML; 58 locally-runnable tests green; no .env or key material in any branch commit (audited). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
69dd3a6a23 |
feat(validation): evidenceClass on Finding per ADR-105 (issue #522)
Adds EvidenceClass (EXECUTED|STATIC|INFERRED|CONJECTURE) and EvidenceArtifact to the Finding schema, required at the type level. requirements.ts factory defaults to STATIC (all 26 steps are deterministic content analysis); pipeline crash-findings are EXECUTED with the error output as artifact. Field is evidenceClass, not evidence — StepResult.evidence: string[] already exists (ADR amended). Verified: tsc --noEmit clean; validation-pipeline-handler tests 7/7 (MCP handler passes findings through, new field flows to MCP output). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
bc8a37696a |
docs(adr): ADR-105..110 — Pattern Space cross-pollination (issue #522)
Six proposed ADRs from the universal-pattern-space QE assessment: evidence-class labels on findings, behavioral safety evals per model tier, shipped-agent invariant verification, benchmark lineage with pre-registered rubrics (incl. antipattern list), interaction benchmark for qualitative agents, kept-nulls negative pattern records. Cross-checked against existing ADR corpus: Related fields reference ADR-058/061/070/074/083/086/098/099/102/104; mid-task safety injection, restraint probes, and null-vs-quarantine sequencing folded in from the cross-check findings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
07326b9802 |
feat(arena): ADR-104 qe-arena Phase 1 — competitive test-strategy tournaments
aqe arena run: seeded tournaments where test strategies compete on REAL fitness — built-in first-order operator mutants (string/comment/arrow-safe enumeration) executed per strategy via node --test, real line coverage from --experimental-test-coverage, and a deterministic suite-cost term. Tournament emits a Wolfram-style pairwise competitive array; --evolve hill-climbs the winner by seeded group flips with real re-evaluation. Runs persist fail-soft to memory.db kv_store (namespace 'arena'); aqe arena list reads them back. Self-contained: no core changes, no new dependencies; fixtures/arena-demo ships 4 test groups of varying kill power over a branch-dense pricing module. Reproducibility contract: same seed + fixture => byte-identical envelope (verified E2E by diffing two runs), with informational.runtimesMs the only excluded field. Honest finding during verification: the original measured-runtime rank penalty was jitter-dominated at fixture scale and flipped rankings between identical-seed runs — replaced with the deterministic suite-size ratio (documented in ADR-104). Phase 2 (documented): LLM-generated strategies frozen as artifacts, winners distilled into qe-test-architect priors; MCP arena_run registration (CLI and MCP share runArena() — parity by construction). 10 tests green (incl. real-engine reproducibility); tsc + eslint clean. Part of #520 (improvement 7/7 — initiative complete). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
e04f05742f |
feat(qcsd): ADR-102 deterministic workflow orchestration with adversarial verification
qcsd-development-swarm now runs as a deterministic workflow script (.claude/workflows/qcsd-development-review.js) instead of prompt-driven Task-tool fan-out: - pipeline: one finder per quality dimension (TDD adherence, complexity, coverage gaps; qe agents as agentType) -> 3 blind adversarial refuters per finding -> deterministic synthesis, no agent editorializes the report - Loki-mode constraints (ADR-074) encoded in refuter prompts: blind review (bare claim + evidence only), anti-sycophancy (attack the claim; uncertainty defaults to refuted), 3 distinct lenses - majority kill (>= ceil(N/2) refuted votes); killed findings retained with refutations for audit; output is finding-verdict@1 envelopes (ADR-103) - SKILL.md v1.1.0: execution.primary workflow, Task-tool fallback documented (reports marked verification: none) Validation run (wf_254f1271-022): 3 dimensions over src/, 48 agents, 15 raw findings -> 14 confirmed, 1 killed by unanimous refutation of a factually-wrong claim — the adversarial gate demonstrably filtering a false positive. Several confirmed findings target code added earlier in this initiative (honest signal, queued as follow-ups). Part of #520 (improvement 5/7, Tier 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
dc6fce5e70 |
feat(contracts): ADR-103 structured verdict handoffs — RiskDecision et al
Versioned, additive-only verdict envelopes for agent/MCP handoffs:
RiskDecision (quality gates), FindingVerdict (review findings with
adversarial refutations), CoverageGap (risk-weighted gaps).
- src/contracts/verdicts.ts: dependency-free validators (source of
truth) + mirrored draft-07 JSON Schemas; no zod/ajv runtime deps
(light-install posture preserved)
- quality_assess MCP boundary now attaches a validated riskDecision
envelope derived from the gate outcome (approve/block, escalate when
indeterminate) — additive, existing payload unchanged; direct and
wrapped handlers share the same mapToResult so MCP parity is inherent
- schemas/*.schema.json published via scripts/generate-verdict-schemas.mjs
for external ajv validation and workflow agent({schema}) enforcement
(consumed by improvement 5)
17 contract tests (goldens validate; mutations rejected with field-level
errors; builder always emits valid envelopes). E2E: envelope emitted by
the compiled boundary builder validates with stock ajv against the
published schema; mutated sample rejected. tsc + eslint clean.
Part of #520 (improvement 6/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
f2a350ba7f |
feat(agents): ADR-101 nested-subagent readiness — provenance, probe, validation
Anthropic announced nested subagent support (depth=5) on 2026-06-09; the runtime currently strips Task at parent->child spawn (upstream denylist, empirically confirmed by our committed probe: level=1 status=NO_AGENT_TOOL, matching ruflo ADR-147). This lands the readiness infrastructure that is independently useful today and activates with zero changes when the gate lifts: - aqe hooks post-task gains --parent-agent-id / --depth (validated: identifier charset, integer 0 <= d <= 32; depth=0 survives as 0); provenance persisted to qe_trajectories.metadata_json (stage 1 — no schema migration) and forwarded into ReasoningBank feedback so learning can segment patterns per hierarchy level - TaskExecution gains optional parentAgentId/depth (additive types) - scripts/probe-nested-spawn-depth.mjs: single-shot empirical probe, committed transcript under docs/probes/ — re-run after every `claude update`; TASK_TOOL_LIVE is the signal for follow-up phases - fresh-project fix found by the E2E: persistTaskOutcome assumed captured_experiences/experience_applications exist, but they are middleware-created — Stream B silently no-oped on fresh installs; canonical DDL now ensured in the hook path Deliberately deferred (documented in ADR-101 P1): tools: frontmatter retrofit on shipped qe-*.md agents — Claude Code frontmatter is a restrictive allowlist and would sever inherited MCP tool access; needs maintainer sign-off + wildcard verification first. 8 unit tests green; E2E on a fresh temp project verified persistence, rejections, and depth=0 boundary. tsc + eslint clean. Part of #520 (improvement 4/7, Tier 2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
2d5126a053 |
feat(hooks): ADR-100 tool-loop circuit breaker — warn at 3, block at 5
Breaks runaway retry loops: when the same Bash command fails consecutively (normalized signature, cross-process via a small JSON sidecar under .agentic-qe/), pre-command warns at 3 failures and emits a BLOCK recommendation with a recovery hint at 5. Advisory by default; AQE_STRICT_TOOL_LOOP=1 upgrades the block to a PreToolUse deny. A successful run resets the breaker; an open breaker grants a half-open probe after 2 minutes; stale entries pruned after 30 minutes. Mirrors the ADR-064 domain-circuit-breaker semantics at command level. Sidecar instead of memory.db: pre-command deliberately runs without a DB open, and each hook call is a fresh process so in-process state is impossible. Fail-open everywhere (corrupt state file => allow). 11 unit tests green; E2E verified with real aqe hooks pre/post-command: warn after 3, BLOCK after 5, strict-mode deny envelope, reset on success. Part of #520 (improvement 3/7, Tier 1 complete). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
9c54a70656 |
feat(llm): ADR-088 amendment — wire enableCache to ephemeral cache breakpoints
The enableCache flag existed since ADR-011 but was never read: every
advisor/LLM call paid full input price. Now:
- ClaudeProvider.generate() sends `system` as a content-block array with
cache_control {type: ephemeral} when enableCache (default on); plain
string when disabled. Anthropic-native provider only (ADR-092 — other
providers untouched).
- cache_creation_input_tokens / cache_read_input_tokens surfaced into
TokenUsage.cacheCreationTokens/cacheReadTokens; totalTokens now counts
the full prompt (input_tokens alone is the uncached remainder).
- CostTracker bills cache reads at 0.1x and writes at 1.25x input rate
instead of ignoring them.
5 new provider tests assert the serialized request shape (cache_control
present when enabled, absent when disabled, no system field when no
prompt) and the usage mapping. 64/64 LLM tests green, tsc clean.
Part of #520 (improvement 2/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
8ff39f2c6e |
feat(learning): ADR-099 reasoning-tag scrub before distillation/embedding
Fable 5-era models emit large extended-thinking blocks inside task results and step actions; embedding them verbatim contaminates the 384-dim pattern vectors in memory.db and degrades HNSW retrieval. - new scrubReasoningBlocks()/scrubReasoningDeep() (src/shared/reasoning-scrub.ts): boundary-gated — only well-formed <think|thinking|reasoning| REASONING_SCRATCHPAD>...</> pairs are removed; prose mentioning a tag survives; clean input returned byte-identical - applied at ingestion: TrajectoryTracker startTrajectory task, recordStep action/result.data/result.error - applied at distillation: ExperienceReplay.storeExperience task/strategy/ keyActions (insert + embedding text) - applied at flow-forward: both init-time embedding backfills (legacy rows cleaned as they get embedded — no memory.db rewrite) and retrieveSimilar query embeddings (symmetric with stored vectors) - 15 unit tests; E2E verified against a temp DB: trajectory + experience rows persisted with all scratchpad blocks stripped Part of #520 (improvement 1/7, Tier 1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
a76fbf98e1 |
fix(cli): bridge CLI auto-init into MCP fleet state + dev-env fixes
- aqe memory/agent/task/team/pipeline CLI commands always failed with "Fleet not initialized": autoInitialize() built kernel+queen in its own CLIContext while the shared MCP handlers guard on the core-handlers fleet-state singleton. New adoptExternalFleet() bridges the CLI's components into that state (idempotent, never clobbers an existing fleet), restoring MCP-CLI parity. Verified E2E: store/get/delete green. - devcontainer: node_modules named-volume overlay + npm ci in postCreate so container installs Linux natives without touching the host's macOS binaries (takes effect on next rebuild; devcontainer users only) - docs/setup: drop stale ruflo@3.5.18 pins (CLAUDE.md quick setup, install-tools.sh) in favor of ruflo@3 + global `ruflo mcp start` - docs: add Fable 5 / ruflo-parity improvement plan (tracked in #520) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
dc5dc14ebd |
feat(learning): use local Nagual instance + add local LLM judge
Switch the cross-project knowledge hub from the cloud endpoint to a locally-runnable Nagual instance, and add an optional local LLM judge for scoring pattern quality. - nagual-client.ts: REST client for a local `nagual serve` instance (NAGUAL_URL, default http://localhost:3333); search/recordOutcome/ promotePattern are all fire-and-forget safe. - local-judge-client.ts: Ollama-compatible judge (NAGUAL_JUDGE_URL/MODEL) blending 0.7*outcome + 0.3*judge_score, with an ambiguous-zone review flag. - task-hooks.ts: pre-task surfaces Nagual cross-project context for complex tasks in relevant QE domains (best-effort, gated on env + length). - settings.json: add NAGUAL_JUDGE_URL/MODEL; fix invalid mcp allow-rule globs (mcp__ruflo__* / mcp__claude-flow__*) and add Bash(rm -rf:*) deny. - docs: de-reference the cloud hub in favor of a local instance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
bec5a57f1c |
chore(lint): ignore src/_archived, align bounded-context count, apply safe autofixes
Repo-health cleanup from an external code audit: - eslint: ignore src/_archived (already excluded by tsconfig, so linting it only produced 6 parserOptions.project errors). - Align the architecture claim: src/index.ts and docs said "12"/"6" bounded contexts; there are 13 domain dirs (matching package.json). Updated src/index.ts (comment + ARCHITECTURE const) and the docs list to the actual 13. - Apply stock `eslint --fix` (prefer-const, prefer-as-const, no-useless-escape, no-regex-spaces) across 24 files. Lint problems: 1072 -> 1031. Typecheck and build pass; no behavioral changes. |
||
|
Dragan Spiridonov
|
84fb9ca65d |
docs(adr): ADR-098 learning-store & routing hardening + mark 2 side-findings fixed
ADR-098 documents the durable behavioural decisions from the ruflo adoption pass (#510): domain-score de-dilution (#2), EWC++ importance-weighted forgetting (#6), contradiction suppression (#7), the two pre-existing bug fixes (consecutive_failures, MCP -32601), and the explicit NEGLECTED decisions — ruflo's Beta-priors (#2) and RaBitQ speedup (#8) did not transfer, and multi-bundle packaging is deferred. AQE WH(Y) format, Accepted. Marks the consecutive_failures ( |
||
|
Dragan Spiridonov
|
3bc09cda6d |
fix(llm-router): address 10 findings from qe-devils-advocate audit
A post-Phase-4 devil's-advocate audit identified 10 issues. All addressed
with code fixes + tests. Some were real bugs (silent billing regression,
2 unwired MCP tools, 2 HybridRouter routing bugs); others were robustness
improvements (split singletons, silent errors, decorative type fields).
Findings → fixes (severity tagged in audit report):
1. CRITICAL — silent billing regression on upgrade
Users with ANTHROPIC_API_KEY in env would silently start paying for
Claude API calls on every Fleet operation. The CHANGELOG's "opt out
via code changes" wasn't sufficient — needed an env-only escape.
Fix: AQE_LLM_ROUTER_DISABLED=1 env kill-switch honored in both
QEKernelImpl._initializeLLMRouter and src/mcp/tools/base.ts
getSharedLLMRouter. Truthy values disable; 'false'/'0'/'no'/'off'/''
pass. Unified parse rule between kernel and MCP base.
2. HIGH — 2 unwired MCP standalone tools (audit said 3; QualityCriteria
isn't LLM-aware so excluded)
src/mcp/tools/test-generation/generate.ts:82 was calling
createTestGeneratorService(memory) without router. Same for
visual-accessibility/index.ts:177. Both now use the
createXxxServiceWithDependencies factories with { memory, llmRouter }.
3. CRITICAL — E2E only tested manual routing mode, not production
Production default is rule-based. Adding two new E2E tests for
rule-based mode SURFACED two real bugs in HybridRouter that I fixed:
3a. executeWithFallback hardcoded ['claude','openai','ollama'] as the
only fallback providers; gemini/openrouter/azure-openai/bedrock
users had no fallback even though ProviderManager can construct
them. Plus the user's defaultProvider wasn't auto-included in the
chain. Both fixed at src/shared/llm/router/hybrid-router.ts:612.
3b. A non-retryable error from one provider (e.g. Claude "invalid
x-api-key") aborted the entire fallback loop via `throw error`,
defeating the whole point of a fallback chain. Fixed: skip the
inter-attempt delay for non-retryable but continue to the next
fallback entry.
4. HIGH — MCPToolContext.llmRouter field was decorative
Added the field but no code populated it. Every getLLMRouter(context)
call fell through to getSharedLLMRouter().
Fix: added llmRouter and memory to BaseMCPTool.invoke() options;
qe-tool-bridge.ts and registry.ts now explicitly forward the shared
singleton via { llmRouter } at invoke time.
5. HIGH — 14 of 15 wired services had only STRUCTURAL verification, not
behavioral (only TestExecutor had a chat() round-trip test)
Fix: new tests/integration/llm-router-behavioral.test.ts asserts
isLLMAnalysisAvailable()===true (the exact gate the audit named) for
all 14 services + 2 full behavioral round-trips. 16 tests total.
6. HIGH — MCP and kernel built SEPARATE HybridRouter instances
Different cost trackers, caches, circuit breakers per side.
routing_economics/routing_metrics would only see half the traffic.
Fix: kernel.ts._initializeLLMRouter calls setSharedLLMRouter(router)
on init so both paths share one instance. kernel.dispose() calls
resetSharedLLMRouter() so the next kernel doesn't inherit the
previous instance.
7. HIGH — Silent error swallowing in _initializeLLMRouter
console.warn'd then continued without LLM, with no signal for
monitoring. A misconfigured provider would silently degrade with no
alert.
Fix: now publishes kernel.llm-router.init-failed (caught error) and
kernel.llm-router.init-no-provider (enabled=true but nothing
available) events on the event bus. Test confirms event delivery.
8. MEDIUM — Service count mismatch in docs (ADR/CHANGELOG said "14
services / 12 domains"; test enumerates 15 services / 11
LLM-enhanced domains)
Fix: ADR-043 addendum + CHANGELOG corrected to 15/11. Added
explicit service list and noted the 2 non-LLM domains
(enterprise-integration, coordination).
9. MEDIUM — aqe llm config --set apiKey=... silently dropped the key
stripApiKeys() returned a successfully-saved file without the key,
user thought it was saved.
Fix: CLI updateConfig() now throws a clear error pointing to env
vars BEFORE calling save. stripApiKeys also now logs a console.warn
when it strips anything, so programmatic callers see the strip.
10. LOW — Env vs disk precedence was counterintuitive and undocumented
Force-enable-on-env overrides explicit `enabled: false` on disk.
Fix: doc comment in config-store.applyEnvProviderDetection +
ADR-043 "Precedence & env-var conventions" section explains.
Verification
- npx tsc --noEmit: clean
- 191 unit + integration tests pass (16 new behavioral tests for #5,
17 wiring tests, gemini provider, config-store, llm-router-service,
kernel-llm-router with 17 tests including all kill-switch cases,
CLI persistence with 28 tests)
- 5 real-provider E2E tests pass (Gemini 2.5 Flash, OpenAI gpt-4o-mini,
OpenRouter, plus 2 rule-based-mode tests that previously failed and
now pass with the HybridRouter fallback fixes)
Refs ADR-043 audit findings (2026-05-21)
Builds on
|
||
|
Dragan Spiridonov
|
c206c81751 |
feat(llm-router): Phases 3 + 4 — MCP path, integration tests, real-provider E2E, ADRs
Phase 3 — MCP tool path
- 8 standalone MCP tools (coverage-analyze, coverage-gaps, contract-validate,
chaos-inject, code-intel-analyze, defect-predict, learning-optimize) now
resolve the kernel HybridRouter via getLLMRouter(context) and pass it to
their service via the dependency-bag form.
- src/mcp/tools/base.ts: added getSharedLLMRouter() singleton, mirroring the
existing getSharedMemoryBackend() pattern. resetSharedLLMRouter() for
tests. Added MCPToolContext.llmRouter field so future tools (and the MCP
runtime when wiring per-request kernel context) can inject explicitly
rather than via the shared singleton.
Phase 4a — Mock-provider integration tests
- tests/integration/llm-router-wiring.test.ts (new, 17 tests).
- Per-domain wiring assertions (15): for each LLM-enhanced
plugin → coordinator → service path, asserts service.llmRouter ===
kernel.llmRouter after kernel boot with an injected mock provider.
This is the precise structural test for the gap the author's friend
flagged — a regression in any domain's wiring fails immediately.
- Behavioral end-to-end (1): TestExecutorService.analyzeFailuresWithLLM
actually invokes the mock provider; assert generateCalls counter
incremented and the response surfaced.
- Sanity (1): kernel.llmRouter is defined when a router was built.
Phase 4b — Real-provider E2E smoke test
- tests/e2e/llm-router-real-providers.test.ts (new, 3 tests).
- Gated by AQE_LLM_E2E=1; each provider auto-skips when its API key
isn't in env.
- Uses mode: 'manual' to bypass DEFAULT_QE_ROUTING_RULES (which would
otherwise route generic tasks to Claude even when defaultProvider
is set to something else — worth knowing).
- Verified live on 2026-05-21 against:
Gemini (gemini-2.5-flash) → "Hi!"
OpenAI (gpt-4o-mini) → "Hi!"
OpenRouter (openai/gpt-4o-mini) → "Hi!"
Claude intentionally skipped — the ANTHROPIC_API_KEY in the dev env
is a 24-char stub (real Anthropic keys are ~108 chars), would 401.
Phase 4c — Documentation
- docs/implementation/adrs/ADR-043-vendor-independent-llm.md: added
"Wiring Verified" status entry (2026-05-21) + full addendum documenting
the problem found, the four-phase resolution, the verification tests,
and the behavioral consequence for users.
- docs/implementation/adrs/ADR-051-agentic-flow-integration.md: added
"LLM-Enhancement Wiring Verified" status entry pointing to ADR-043's
addendum.
- CHANGELOG.md: added [Unreleased] section under Changed — behavioral,
explicitly noting the behavior shift (fleets with provider keys in env
will now start routing LLM calls through that provider), the opt-out
path (KernelConfig.llmRouter: { enabled: false }), and the persistence
fix for `aqe llm config --set`.
Test totals across all four phases on this branch
- 39 unit tests across 5 files (Phase 1)
- 17 integration tests in 1 file (Phase 4a)
- 3 real-provider E2E tests in 1 file (Phase 4b, AQE_LLM_E2E=1)
- 34 pre-existing kernel tests still pass (no regression)
- 22 pre-existing test-generation coordinator tests still pass (no regression)
- Clean tsc --noEmit across the full project
Refs ADR-043, ADR-051
Closes the wiring gap identified by user audit on 2026-05-21
Builds on
|
||
|
Dragan Spiridonov
|
a9f7082f15 |
fix(embeddings): harden ADR-097 endpoint client against devil's-advocate audit
Closes 11 findings from a post-merge audit of the external embedder endpoint implementation. The "no silent hash fallback" correctness promise was being contradicted by two callers; the connect timeout was actually an idle timeout; the identity probe could be bypassed in several paths; and identity drift across runs was undetectable. All hardened. Critical/High fixes: - sqlite-persistence backfill no longer falls back to hash embeddings when the endpoint is configured; re-raises to protect the HNSW index from non-comparable vectors (was poisoning silently on every endpoint blip). - qe-reasoning-bank.embed() propagates the throw under isUsingEndpoint() instead of silently substituting a hash vector. - EmbedderEndpointClient.embed() now gates on probe() — lazy probe on first use with in-flight dedup so 20+ concurrent embeds share one probe. - Circuit breaker recovery (window expires) clears cached identity so the next embed re-probes — endpoint restarts often coincide with model swaps. - True TCP-connect timeout enforced at the socket layer (was req.setTimeout, which is an idle-after-write timeout — SYN blackholes hung ~75s instead of 5s). - Endpoint identity persisted to memory.db kv_store (_system namespace) — cross- run fingerprint drift now surfaces as a loud warning on next boot. Medium fixes: - encoding_format: 'float' pinned in request; base64 responses fail loud. - TLS knobs (ca, cert, key, rejectUnauthorized, servername) passthrough for self-hosted https endpoints without NODE_TLS_REJECT_UNAUTHORIZED=0. - URL userinfo stripped + warned at parse; redacted from all error messages and logs. - Embedding cache key namespaced by mode (endpoint:<fingerprint>:<text> vs inproc:<text>) — cross-mode collisions structurally impossible. Tests: - 33 unit tests pass (was 23): adds probe-gate, breaker re-probe, connect timeout, DNS failure, concurrent embed dedup, base64 fail-loud, URL redaction, encoding_format wire format. - 6 live integration tests against real llama-server (all-MiniLM-L6-v2.Q8_0.gguf): probe, single embed, batch, lazy probe gate, dim mismatch, 50-input batch. Replaces the prior "9ms in-process fake" with honest numbers (cold 15ms, warm 2ms against localhost llama.cpp). - New tests/integration/embedder-endpoint-llamacpp.test.ts is gated on AQE_LLAMA_EMBED_URL so it skips cleanly in CI without the binary. ADR-097 updated with the per-caller audit table (12 writers verified non-poisoning) and "drop-in for 6 providers" claim softened to "verified against llama-server; OpenAI shape compat expected but unverified" per provider. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
b17309b3f8 |
feat(embeddings): optional external embedder endpoint (ADR-097, #503)
Add opt-in OpenAI-compatible /v1/embeddings client to eliminate duplicate in-process model loads in ruflo/ruvector co-deployments and the per-hook cold-load cost in fresh agentic-qe CLI processes. When EmbeddingConfig.endpoint (or AQE_EMBEDDER_ENDPOINT) is set, the @huggingface/transformers import is bypassed entirely and feature-extraction routes through the remote service. Endpoint unset = today's behavior. Wire format is OpenAI's /v1/embeddings so any existing embedder server (TEI, vLLM, llama.cpp, Ollama, LocalAI, LM Studio, OpenAI) works as a drop-in, not just bespoke ruvector servers. Transport is HTTP or HTTP-over-Unix-socket. Errors hard-fail with a circuit breaker (3 failures / 60s window) — no silent hash-embedding fallback, which would poison the HNSW index by mixing non-comparable vectors. A startup probe asserts dim === 384 and computes a stable identity fingerprint that catches model/quantization drift at the boundary. - src/learning/embedder-endpoint-client.ts: OpenAI client with keep-alive agent, 5s/30s timeouts, Bearer auth from env, circuit breaker, probe - src/learning/real-embeddings.ts: endpoint branch placed strictly before the dynamic transformers import so the cold-load fix holds at runtime; new isUsingEndpoint() / getEndpointIdentity() observability helpers - 23 new unit tests (real http.createServer fixtures, no http mocking) - End-to-end smoke: 9ms for import + init + probe + embed, transformers never appears in the loaded-module list Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
d698c58721 |
fix(routing): collapse Q-state to 3-dim (drop complexityBucket) (#499)
ADR-096 fix #3 (Option A): the previous 4-dim state_key taskType|priority|domain|complexityBucket fragmented semantically identical tasks across cells. complexityBucket was derived from raw description length (round(min(len/200,1)*10)) — so a 55-char and 130-char phrasing of the same task landed in different Q-rows, and QWEIGHT_RAMP_VISITS=20 was rarely reachable per cell even after the fix #1/#2 loop closure. State key is now 3-dim: taskType|priority|domain Same change applied at both ends of the loop: - buildRoutingStateKey (agent-routing.ts) — parameter and key shape - updateHookRouterQValue (hooks-dream-learning.ts) — state_key SQL - qe-reasoning-bank.ts reader-side call - task-hooks.ts post-task writer call - routing-hooks.ts post-route writer call (new in fix #1) `deriveComplexityBucket` stays exported (marked @deprecated for state_key use) — the pre-task bridge payload still carries `complexityBucket` for kv_store schema compatibility and any downstream telemetry. `TaskBridgePayload` interface unchanged. Migration: existing rl_q_values rows written under the previous 4-dim key are orphaned. Field reports show real installs have at most ~2 such rows (the loop was broken; no real Q-data accumulated), so we accept the orphans rather than ship a migration. If a future install carries substantial 4-dim data, a one-shot script that strips the trailing |N segment is straightforward. Test contract extended: - agent-routing-exploration.test.ts pins the 3-segment key shape - route-q-loop-closure.test.ts asserts complexityBucket does NOT leak into updateHookRouterQValue payloads (writer-side check) and that bridge.complexityBucket is ignored even when present. ADR-096 updated: complexity-bucket promoted from Open Questions to Decision (Option A). Open Questions now lists "re-introducing a coarse complexity tier if telemetry shows the 3-dim key conflating decisions that benefit from differentiation" as a wait-and-measure follow-up rather than a planned change. Refs: #499 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
d41bbc22b0 |
fix(routing): close Q-loop on routing surface (#499)
The route hook (UserPromptSubmit) reads `rl_q_values` to produce qWeight, but the only writer was task-hooks `post-task` gated on `outcome.bridge`. On real projects, `routing_outcomes` reached 139 rows while `rl_q_values` stayed at 2 — the route surface that displays qWeight had nothing to read, so `qWeight: 0` was structurally pinned. Two complementary producer additions, ADR-096: Fix #1 — post-route trains rl_q_values When the Stop hook resolves a routing_outcomes sentinel, derive a state_key from the sentinel's `task_json.description` using the same exported helpers the route hook used (`deriveTaskType`, `deriveComplexityBucket`, `detectQEDomains`) and call `updateHookRouterQValue`. Writer and reader address byte-identical rows. Best-effort + try/catch: Stop hooks must never crash the host. Fix #2 — post-task fires even when bridge is absent Drop the `if (outcome.bridge)` gate. When bridge is missing, derive state from `options.description` instead. Adds a new `--description` option to `aqe hooks post-task` so the PostToolUse hook command can pass `tool_input.description` for Task tools. Skip the update only when neither bridge nor description is available (would pollute an "unknown" sentinel state). Tests: tests/unit/cli/commands/route-q-loop-closure.test.ts pins the producer/consumer state_key alignment contract — if a future refactor desynchronizes the three call sites (routing-hooks.ts post-route, task-hooks.ts post-task, qe-reasoning-bank.ts:530), assertions fail the build. Closes the Open Question flagged in ADR-095. The deriveComplexityBucket fragmentation Jordi noted is acknowledged in ADR-096's Open Questions and tracked for a follow-up — orthogonal to closing the loop. Refs: #499 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
70154c1b72 |
feat(routing): three-signal routing — static + Q-value blend + mincut-gated ε-greedy (ADR-095, #488 Phase 4)
Closes the self-learning loop: the Q-value Bellman updates that #487 populates now have a routing-side reader; structurally-aware exploration (via the mincut infrastructure from ADR-047 / ADR-068) gathers Q-table signal beyond the deterministic greedy winner; the catalog diversifies without sacrificing safety on bottleneck decisions. Three signals blended at the agent-routing decision point: 1. Static score (existing): domain match + capability match + performance prior + language boost + pattern similarity. Computed by `calculateAgentScores` in src/learning/agent-routing.ts. 2. Q-value blend (NEW — reads from rl_q_values): effectiveScore = staticScore * (1 - qWeight) + sigmoid(qValue) * qWeight qWeight ramps from 0 (no Q-data) to MAX_Q_WEIGHT=0.4 over QWEIGHT_RAMP_VISITS=20 visits per (state_key, action_key). Cold-start behavior is identical to pre-ADR-095 static-only routing; the learned signal contributes meaningfully only after the Q-table has matured for that state. 3. Mincut-gated ε-greedy exploration (NEW): baseEpsilon = AQE_ROUTER_EXPLORATION_RATE env or 0.05 default safetyMultiplier = getSharedMinCutMonitor().isCritical() ? 0.2 : 1.0 epsilon = baseEpsilon * safetyMultiplier With probability ε, swap the greedy winner with a uniformly-random pick from the top-3 alternatives. Uses crypto.randomInt (NOT Math.random) to match the codebase convention (randomUUID is the established crypto entry-point in the same modules). Maintainer decisions captured in ADR-095 v2: 1. Default ON (Option A): mincut-modulated exploration ships enabled. AQE_ROUTER_EXPLORATION_RATE=0 is the documented escape hatch. 2. Telemetry surface: extended `aqe learning loop-health` with a new "Routing diversification" section showing 7-day exploit/explore counts, avg quality_score per bucket, avg criticality, avg q_weight. 3. crypto.randomInt over Math.random — codebase convention preserved. 4. Q-value ramp constants are best-guess (MAX_Q_WEIGHT=0.4, QWEIGHT_RAMP_VISITS=20); tune post-deploy from telemetry. Source-level findings worth noting: - `rl_q_values` was populated by `updateHookRouterQValue` since #487 but NEVER read during routing. Verified via grep across src/learning/, src/routing/, src/cli/commands/hooks-handlers/. This commit adds the reader; closes that loop. - Mincut graph is populated by Queen coordinator swarm activity, NOT per-task. So the "criticality" signal is graph-level safety (the `isCritical()` boolean from MinCutHealthMonitor) rather than per-task structural weight. Honest framing reflected in the ADR. What landed: src/kernel/unified-memory-schemas.ts - routing_outcomes gains exploration (INTEGER NOT NULL DEFAULT 0), criticality (REAL), q_weight (REAL) columns + an exploration index for the loop-health bucket queries src/routing/routing-outcomes-migration.ts (NEW) - ensureRoutingOutcomesAdr095Columns(db) — idempotent ALTER TABLE for pre-existing databases. Process-local flag avoids repeat work. src/routing/routing-feedback.ts - hooks the migration helper into the existing schema-migration block src/learning/agent-routing.ts - new helpers: blendStaticAndQValue, resolveExplorationRate, applyExplorationPolicy (uses crypto.randomInt), sigmoid normalization - new helpers: deriveTaskType, buildRoutingStateKey, deriveComplexityBucket — moved from task-hooks.ts so the routing path and the post-task Q-update path share the same state_key derivation - calculateAgentScores accepts optional qValueLookup; backward-compat when undefined (identical pre-ADR-095 behavior) - ScoredAgent gains optional telemetry fields: staticScore, qWeight, qValue, qVisits, exploration src/learning/qe-reasoning-bank-types.ts - QERoutingResult gains optional exploration / criticality / qWeight src/learning/qe-reasoning-bank.ts - routeTask builds the Q-value lookup from rl_q_values keyed by stateKey = deriveTaskType(task) | normal | domain | complexityBucket - applies exploration policy after sort; reports avgQWeight + the mincut safetyMultiplier in the QERoutingResult - new private buildQValueLookup helper with table-exists guard so the method is a graceful no-op when the schema isn't applied yet (tests, fresh-init scenarios) src/cli/commands/hooks-handlers/task-hooks.ts - private deriveTaskType removed (DRY: now imported from agent-routing) - pre-task sentinel INSERT writes the three new columns - migration helper invoked before write src/cli/commands/hooks-handlers/routing-hooks.ts - JSON output passes exploration / criticality / qWeight through - INSERT writes the three new columns; migration helper invoked src/cli/commands/learning.ts - `aqe learning loop-health` queries routing_outcomes for 7-day rolling exploration stats and avg quality_score per bucket. Skipped silently when the columns don't exist (pre-ADR-095 databases). Tests: - tests/unit/learning/agent-routing-exploration.test.ts: 35 tests cover sigmoid blend math, qWeight ramp, env override priority, mincut safety multiplier, exploration rate statistical tolerance, only-top-3 alternative constraint, deriveTaskType regex coverage, stateKey format, complexityBucket scaling, end-to-end Q-blend - tests/unit/routing/routing-outcomes-migration.test.ts: 5 tests cover fresh-DB migration, idempotent re-run, missing-table tolerance, exploration index creation, column type/default checks 634/634 tests pass across CLI commands, workers, bridge, learning, init/phases, architecture, kernel, and routing suites; build clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
6236a2cdc7 |
feat(kernel): kernel-side dream cycles + hooks-as-producers boundary (ADR-094, #488 Phase 2)
Moves dream cycles out of hook subprocesses into the long-lived kernel
process via the DreamScheduler that already existed but was never wired.
Single cutover, no feature flag — per maintainer decision on ADR-094.
Why this matters
================
Before this commit, every ~20 Edit/Write tool uses triggered a dream cycle
inside a short-lived `npx aqe hooks post-edit` subprocess. The cycle held
a SQLite write transaction for up to 10 seconds while the kernel-owning
MCP server, other hook subprocesses, and user MCP tool calls queued
behind it. stderr from the cycle was consumed by Claude Code's hook
reader and lost from operator visibility. Cold DreamEngine init paid
every ~20 edits.
Now the kernel's `QEKernelImpl.initialize()` boots a `DreamScheduler`
that runs cycles on its own cadence inside the long-lived process. Hook
subprocesses keep only `incrementDreamExperience` (a counter bump that
returns in ~50ms). Operators see `aqe learning loop-health` report the
`dreamScheduler` component as live within one tick.
What landed
===========
src/kernel/kernel.ts
- new private `_dreamScheduler?: DreamScheduler`
- new config flag `enableDreamScheduler` (default `true`, mirrors
`enableExperienceBridge`)
- initialize() constructs DreamEngine + DreamScheduler after the
bridge starts, wraps in try/catch (best-effort), failure logs a
warning and continues (kernel boot never blocks on this)
- dispose() stops the scheduler BEFORE bus/memory cleanup so a dream
in-progress doesn't write to a disposed backend
src/learning/dream/dream-scheduler.ts
- executeDream() now records loop-health on every tick (success or
failure) via the helper from #488 Phase 1 B.2. Operators see
`learning:loop-health.components.dreamScheduler.successesSinceBoot`
grow on each cycle.
src/cli/commands/hooks-handlers/editing-hooks.ts
src/cli/commands/hooks-handlers/task-hooks.ts
- dropped `checkAndTriggerDream` import + call
- kept `incrementDreamExperience` (the counter is the producer-side
signal the kernel scheduler reads)
- JSON output now emits `dreamTriggered: false, dreamReason:
'deferred-to-kernel'` so operator scripts grepping the field see
*where* the cycle actually runs
src/cli/commands/hooks-handlers/hooks-shared.ts
- removed the `checkAndTriggerDream` re-export. The function itself
is still defined in hooks-dream-learning.ts and exported there for
direct test access and the documented rollback path.
tests/unit/architecture/hooks-boundary.test.ts (NEW)
- greps `src/cli/commands/hooks-handlers/` for forbidden patterns:
imports from `src/learning/dream/` and references to
`checkAndTriggerDream`. Skips comment lines so explanatory mentions
don't false-positive. `hooks-dream-learning.ts` is in `allowedFiles`
since it DEFINES the function.
- This is the enforcement mechanism for the boundary contract. A
contributor accidentally re-adding heavy hook-side work gets a
clear CI failure pointing them at ADR-094.
tests/unit/kernel/dream-scheduler-wiring.test.ts (NEW)
- asserts `enableDreamScheduler` default, opt-out path, init+dispose
cleanliness with and without the scheduler, idempotent dispose
tests/unit/cli/commands/post-edit-dream-trigger.test.ts (REWRITTEN)
- was asserting checkAndTriggerDream was called from post-edit (the
#480 fix). That contract is now wrong — the kernel is the only
place that triggers dreams.
- now asserts: counter is bumped (the kernel reads it), no inline
cycle trigger, JSON emits `deferred-to-kernel`, hook stays robust
when the memory backend throws.
docs/implementation/adrs/ADR-094-kernel-side-dream-cycles.md
- status: Implemented
- migration section replaced with the single-cutover rationale and
a documented 4-line rollback path
583/583 tests pass across CLI, workers, bridge, learning, init/phases,
architecture, and kernel suites. Build clean.
Boundary contract (now enforced in CI)
======================================
Hook subprocesses MUST NOT do work that exceeds ~100ms. Acceptable
hook-side work: producer SQLite writes, counter increments, in-memory
routing lookups. Unacceptable: DreamEngine init, multi-second SQLite
transactions, anything that depends on long-lived state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
5d187de2a1 |
fix(install,rvf,hypergraph): unblock Windows install, fix RVF FsyncFailed, synthesize covers edges
Bundles fixes for issue #439 (Windows install) and Jordi RUFLO patches P020 (RVF init) + P220 (hypergraph blindness). Windows install (#439): - Move hnswlib-node from dependencies to optionalDependencies so installs no longer fail on Windows boxes without VS C++ Build Tools; HnswAdapter already routes around a missing native binary at runtime. - Replace HNSWIndex.ts top-level static `import hnswlib from 'hnswlib-node'` with a lazy require so package load survives the optional dep being absent. - README + preinstall.cjs: document toolchain requirement and disclose that the JS fallback degrades to O(N) brute-force when @ruvector/gnn is also unavailable (Windows default — no win32 prebuilds ship). - ADR-090 amendment reconciling the move with ADR-081's optional-dep + JS-fallback intent. RVF init (Jordi P020): - Root cause: shared-rvf-adapter and pattern-store factory always called RvfDatabase.create(), which throws `0x0303 FsyncFailed` when the .rvf file already exists from a prior init phase. Reproduced and confirmed against both 0.1.7 and 0.1.8 native binaries — not a version regression. - Fix: race-tolerant open-or-create ladder (try open → fall back to create → retry open on create-race) with dim() verification after open. Mismatched-dim files are closed and the caller degrades rather than corrupting silently. Hypergraph (Jordi P220): - Drop the original flushEdgesToHypergraph (wrote `call`/`extends` edges that no production query reads — devil's-advocate review caught it as dead infrastructure). - Add HypergraphEngine.synthesizeTestCoverage that re-tags test files type='test' and writes `covers` edges from each test file to functions in the source files it imports. This is the shape findUntestedFunctions / findImpactedTests actually filter on. - E2E verified: untested correctly excludes covered functions; impacted-tests returns the right test file. Verification: - `npm run build` clean - 276/276 tests pass across HNSW, RVF, KG, coordinator, hypergraph suites (5 new synthesizeTestCoverage tests; replaced 4 obsolete flush tests) - E2E on fresh project: aqe init clean, hooks stats shows nativeAvailable: true, code index logs `Test coverage synthesized`, hg untested + hg impacted return useful results Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
70e900e0f9 |
feat(adr-091): promote to Implemented + trust_tier 3
Promotion gate documented in ADR-091 (first green CI run of
test-qe-browser.yml against real Vibium) cleared at commit
|
||
|
Dragan Spiridonov
|
cda9fb3c05 |
fixup(adr-091): address devil's-advocate review
Single commit covering the 5 concrete findings from the qe-devils-advocate review. Does not change the underlying CommandEvalRunner design or the 5 primitives — only the claims around them and the fixture-server hack the yaml had silently inherited. 1. Revert promotion (G4): trust_tier 3 -> 2 in SKILL.md; ADR-091 status Implemented -> In Progress (2026-04-21); v3-adrs.md index row updated to reflect the same; counts 80->79 Implemented, 6->7 In Progress. The promotion gate is now documented on ADR-091: first green CI run of test-qe-browser.yml (unit + smoke + eval against real Vibium) before trust_tier 3 / Implemented are restored. 2. Fix yaml fixtures (G3): tc006/tc007 were referencing http://localhost:8088/qe-browser/SKILL.md.html which required an unstarted fixture server, so `aqe eval run --skill qe-browser` would have failed on a fresh machine. Rewrote both to use https://httpbin.org/html (matching scripts/smoke-test.sh tc006/tc007). Dropped tc009 (poisoned-page check-injection) from the yaml — it needs the local serve-skills.js fixture which is out of scope for CommandEvalRunner. Documented as an explicit GAP comment in the yaml, noted that check-injection.js's severity path is unit-tested at tests/unit/scripts/qe-browser-check-injection.test.ts. yaml test_cases now 11 -> 10. 3. Correct pin comment wording (G1): the previous comment claimed ^26.3.18 was a "major.minor-line pin" — that's factually wrong under npm semver (^26.3.18 accepts 26.99.x). Rewritten to accurately describe the intent: major-line pin that blocks 27.0+ while allowing auto-uptake of 26.x security patches and additive features, with scripts/smoke-test.sh as the belt-and-suspenders API contract gate. The spec itself stays ^26.3.18 — the correct choice for an end-user tool under semver assumptions. 4. Add eval CI job (G2): third job in test-qe-browser.yml runs `aqe eval run --skill qe-browser --model claude-3.5-sonnet` against a real Vibium install after `npm run build`. This is the end-to-end proof that CommandEvalRunner dispatch works that the previous unit tests (all mocked CommandRunner) could not provide. Uploads the eval JSON as a CI artifact for post-run inspection. actionlint clean. 5. Scrub stray files (G5): git rm the coherence-comparison artifacts from docs/reports/ that were swept in by a prior `git add -A`, and added a focused pattern to .gitignore (not a blanket docs/reports/ ignore) so future sessions can't repeat the mistake while intentional reports can still be committed. Tests: 51 relevant tests still pass (3 files: command-eval-runner, command-eval-runner-qe-browser, browser-engine-installer). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
23416e1577 |
feat(adr-091): promote to Implemented — trust_tier 3, platform hint, yaml/smoke unify
Closes GAP-05, GAP-08, and final verification gate. - Linux ARM64 diagnostic (GAP-05): browser-engine-installer now probes os.platform + os.arch on Linux aarch64 and checks a prioritized list of system chromium locations (/usr/bin/chromium, chromium-browser, google-chrome, google-chrome-stable, /snap/bin/chromium). When a browser is found, the install result carries a PlatformHint with an exact VIBIUM_BROWSER_PATH export the user can copy. When none exists, the hint surfaces the `apt-get install chromium` remediation. Wired into phase 09 so it prints after vibium install succeeds. 5 new tests. - GAP-08: added explicit `vibium --headless viewport 1280 720` setup steps to yaml tc006/tc007 so the eval runs are as deterministic in CI as smoke-test.sh. Updated smoke-test.sh header to note the yaml is the canonical spec and the bash script is its mirror (covers the F1 vibium-missing contract case the yaml can't express cleanly). - Trust tier promoted 2 -> 3 now that (a) the eval is runnable via CommandEvalRunner, (b) CI runs both unit tests + smoke gate, and (c) all 11 dependent browser-using skills either reference qe-browser or document their intentional Playwright fallback. - ADR-091 status: Proposed -> Implemented (2026-04-21). - v3-adrs.md index reflects 80 Implemented / 6 In Progress / 3 Proposed. Verification: 643/643 tests pass across tests/unit/validation/, tests/unit/init/, tests/unit/scripts/qe-browser-*. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
864741a757 |
docs(adr-093): mark Accepted — Phases 1, 2, 4 implemented; Phase 3 not required
Status moves from Proposed to Accepted. Phase 3 thinking-config shim
dropped from scope after codebase sweep confirmed no raw-Messages-API
callers pass {type: enabled, budget_tokens: N}. Remaining work tracked:
Cyber Verification application submission (external) and 30-day
telemetry recalibration (~2026-05-17).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
b055560e81 |
feat(adr-093): set Opus 4.7 as default ADR-092 advisor model
Commit 5 of 6 for ADR-093 (Commit 4 skipped — no thinking-config callers to shim; documented in the implementation plan). * src/routing/advisor/multi-model-executor.ts: DEFAULT_ADVISOR_MODEL upgraded from 'anthropic/claude-opus-4' to 'anthropic/claude-opus-4.7'. When TinyDancerRouter sets triggerMultiModel=true (confidence < 0.80, or < 0.85 for security tasks), MultiModelExecutor.consult() now dispatches to Opus 4.7 by default via HybridRouter.chat() + OpenRouter provider. * tests/routing/advisor/multi-model-executor.test.ts, tiny-dancer-router-advisor.test.ts: update expected provider model IDs. SWE-bench Verified 87.6% (vs 80.8% Opus 4.6), SWE-bench Pro 64.3% (vs 53.4%), CursorBench 70% (vs 58%). Vendor-independence preserved — consultations still flow through HybridRouter's 7 providers per ADR-043. Verification: * 0 tsc errors * 65/65 advisor unit tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
3ee1fbf518 |
feat(adr-093): add Opus 4.7, Sonnet 4.6, Haiku 4.5 to model registry
Commit 1 of 6 for ADR-093 Opus 4.7 migration. No runtime behavior change.
* src/shared/llm/model-mapping.ts: add MODEL_MAPPINGS entries for
claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5 with
anthropic / openrouter / bedrock IDs.
* src/shared/llm/model-registry.ts:
- extend ModelCapabilities with optional supportsAdaptiveThinking,
supportsEffortXHigh, tokenizerVersion fields (non-breaking)
- add MODEL_REGISTRY entries for 4.7 (1M context, xhigh, opus-4-7
tokenizer, $5/$25/MTok), 4.6 (standard 200k, legacy tokenizer,
$3/$15), 4.5 haiku
- mark claude-sonnet-4 deprecationDate 2026-06-15 (retired
alongside Sonnet 4 per Anthropic announcement)
- mark claude-haiku-3-5 deprecationDate 2026-02-19 (already retired)
- export DEFAULT_SONNET_MODEL = 'claude-sonnet-4-6',
DEFAULT_OPUS_MODEL = 'claude-opus-4-7',
DEFAULT_HAIKU_MODEL = 'claude-haiku-4-5',
RETIRING_MODELS registry with ISO retirement dates
* tests/unit/shared/llm/model-registry.test.ts: 14 new tests covering
new registry entries, central constants, and deprecation metadata.
All 154 tests across registry/mapping/cost-tracker pass.
* docs/implementation/adrs/ADR-093-opus-4-7-migration.md: new ADR
capturing migration decisions (Sonnet 4.6 as Tier 3 default, Opus 4.7
as escalation target, xhigh fleet-wide, thinking-config shim,
Cyber Verification for security agents, Routines/Monitor deferred
to ADR-094).
* docs/implementation/adrs/v3-adrs.md: ADR-093 index row added.
* docs/implementation/plans/ADR-093-IMPLEMENTATION-PLAN.md: 6-commit
execution plan with per-commit verification + rollback strategy.
* docs/security/cyber-verification-application.md: Cyber Verification
Program application draft (submitted 2026-04-17).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
24e30e0584 |
feat(advisor): implement ADR-092 provider-agnostic advisor strategy
Multi-model routing layer with circuit breaking, PII redaction, and domain-specific prompts for the Queen coordinator. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
9c52823e0e |
fix(qe-browser): F1 — implement documented vibium-unavailable contract
Phase 5 UAT discovered the helper scripts surfaced missing-vibium as `actual: "eval error: vibium binary not found..."` inside a `failed` envelope. The Fallback Policy in SKILL.md said scripts MUST return status:"skipped" with reason "browser-engine-unavailable" — that contract was aspirational. F1 makes it real. Contract changes: - lib/vibium.js: ENOENT now throws typed VibiumUnavailableError (with stable code:"BROWSER_ENGINE_UNAVAILABLE" for cross-module duck typing) - New helper unavailableEnvelope() returns canonical skipped envelope with top-level vibiumUnavailable:true and output.reason field - New runOrSkip() wraps each helper's main() and converts the typed error into the documented envelope on the way out - New rethrowIfUnavailable() helper used inside per-script catch blocks so missing-vibium bubbles past local "convert exception to fail()" handlers — assert.runConsoleCheck/runNetworkCheck/runBrowserSideCheck, batch's per-step catch, check-injection main, intent-score main, visual-diff main all updated - emit() now returns exit code 2 for status:"skipped" (was 1) - Exit code contract documented: 0=success, 1=failed, 2=skipped Downstream skills can now branch on result.vibiumUnavailable === true or exit code === 2 instead of grepping the actual string for "vibium binary not found". Tests: 103 (was 86). New file qe-browser-unavailable-e2e.test.ts spawns each of the 5 helpers under PATH=/tmp/fake-bin (node only, no vibium) and asserts exit 2 + skipped envelope shape. vibium-lib test grew from 10 to 22 cases including VibiumUnavailableError, runOrSkip, exit codes. Smoke test: 10/10 PASS (was 9/9). New tc011 builds a fake bin dir inside the smoke test work dir and verifies the contract end-to-end against the actual installed scripts. SKILL.md updated: - Output Contract section now documents all 3 statuses with shapes and exit codes - Fallback Policy replaced generic prose with concrete bash + Node snippets that branch on the structured field ADR-091 Phase 6 documents the change and verification. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
e468038c4d |
docs(adr-091): Phase 5 — user-perspective verification of qe-browser
Ran fresh `aqe init --auto` against an empty test project at /tmp/qe-browser-uat and exercised every helper script as a real user. 12 checks documented in ADR-091, all passing: 1. qe-browser skill installed cleanly 2. navigate + assert end-to-end against httpbin.org 3. M5 --threshold=0.42 form parsed live (verified non-default value) 4. M6 batch pre-validation aborts BEFORE side effects 5. M7 intent-score returns scored candidates from real form 6. M4+M8 check-injection --exclude-selector strips body (3595→35 chars) 7. M2 fixture server binds 127.0.0.1 (verified via banner) 8. M3 path traversal blocked (HTTP 404 on /etc/passwd.html) 9. Vibium-missing fallback returns clear actionable error 10. Installed smoke-test.sh: 9/9 PASS from inside test project 11. Idempotent re-init: Browser engine line still printed 12. JSON envelope contract validated on every emit H6 pre-flight short-circuit confirmed in production: "Browser engine: vibium 26.3.18 (already installed)" The "this can take 1-3 minutes" banner is suppressed when vibium is already on PATH, matching the documented Phase 2 behavior. New finding logged as F1 (NOT a Phase 4 regression): Helper scripts surface missing-vibium as `actual: "eval error: ..."` inside a `failed` envelope. The SKILL.md Fallback Policy says downstream skills should report status:"skipped" reason: "browser-engine-unavailable". A cleaner contract would be a top-level vibiumUnavailable:true flag. Not a blocker. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
3eb9ccc461 |
fix(qe-browser): Phase 4 — all 9 medium devil's-advocate findings
M1 visual-diff threshold: documented "max difference fraction" convention matching Playwright maxDiffPixelRatio (was inconsistent in docs/code). M2 fixture server bind: default to 127.0.0.1 instead of 0.0.0.0; QE_BROWSER_FIXTURE_HOST=0.0.0.0 opt-in for explicit external access. Codespaces auto-forwards 0.0.0.0 ports — this was an unintentional open server every time the eval ran. M3 path-traversal guard: replaced startsWith() with path.relative() + '..'/absolute check. The old guard false-passed sibling-prefix dirs (/foo vs /foo2) and was fragile on Windows mixed separators. M4 ANSI escape injection: added sanitizeSnippet() that strips C0 controls (0x00-0x1F except \t\n) and DEL (0x7F) from every check-injection finding snippet. Prevents `cat findings.json` from triggering terminal control sequences embedded in malicious page content. M5 parseArgs --key=value: split on first '=' so --threshold=0.05 works alongside --threshold 0.05. Only the first '=' is split so URL/base64 values survive intact. M6 batch.js pre-validation: added validateAllSteps() that walks every step's required fields BEFORE the first vibium call. Step 17 typo now aborts before steps 1-16 execute their side effects. M7 intent-score bare-x regex: anchored \bx\b in close_dialog scorer so "fix", "exit", "extra", "sixteen" no longer match. Unicode × and ✕ unchanged. M8 check-injection --exclude-selector: new flag drops subtrees from the scan. Page text is read from a CLONE of <body> with matching elements removed — live page unchanged. Lets docs sites scan their own pages without self-flagging documentation about prompt injection. M9 09-assets vibium catch: wrapped raw error with actionable recovery guidance (`npm install -g vibium`, then re-run `aqe init`). Hidden bug found and fixed during M2/M3 testing: fixtures/serve-skills.js existed but had never been run because the repo root is ESM and the file uses CommonJS require(). Added scoped fixtures/package.json with "type": "commonjs" matching the existing scripts/package.json pattern. Tests: 86 unit tests across 7 files (was 50). All pass. Smoke test: 9/9 still passing against real Vibium v26.3.18. Mirrored .claude/skills/qe-browser → assets/skills/qe-browser. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Dragan Spiridonov
|
4e52d573f7 |
fix(qe-browser): Phase 3 — real Vibium smoke test verified 9/9 passing
User pushed back on Phase 3 attempt 1 ("blocked on linux-arm64"). After
re-verifying upstream — Vibium 26.3.18 IS the latest on npm, Chrome for
Testing across Stable/Beta/Dev/Canary still ships only linux64 (x86_64),
mac-arm64, mac-x64, win32, win64 — installed Debian's native ARM64
chromium + chromium-driver via apt and symlinked them into Vibium's
cache. Then ran the smoke test for real and found four real bugs in
the qe-browser helper scripts that would have shipped broken.
Bugs found and fixed:
1. Eval contract was wrong. lib/vibium.js's vibiumEvalStdin sent
`console.log(...)` expecting Vibium to capture it. Vibium doesn't —
`eval` returns the LAST EXPRESSION's value as
`{"ok":true,"result":"<stringified>"}`. Fix: dropped console.log
wrappers from assert.js/intent-score.js/check-injection.js; added
unwrapEvalResult() in lib/vibium.js that parses the envelope and
JSON-decodes the result string back into our object.
2. lib/vibium.js did not inject --headless. Vibium defaults to visible
browser, which fails in headless containers with "Missing X server
or $DISPLAY". Fix: lib/vibium.js prepends --headless unless
QE_BROWSER_HEADED=1 is set in the env.
3. `vibium screenshot -o /abs/path/foo.png` ignores the directory and
saves to `~/Pictures/Vibium/foo.png`. Verified live. Fix:
visual-diff.js passes just the basename, reads from Vibium's actual
output dir, copies to the requested path, unlinks the source.
4. `vibium screenshot --selector` flag does NOT exist in v26.3.x.
Verified via `vibium screenshot --help`. Fix: visual-diff.js throws
a clear error with remediation guidance if a caller passes
--selector. (B3's "fall back to eval-clip" comment is now removed.)
Plus a real environment-flakiness fix: smoke-test.sh sets
`vibium viewport 1280 720` before tc006/tc007 visual-diff captures
because httpbin.org/html renders at non-deterministic dimensions
(765x672 vs 780x654 observed) when no viewport is set, causing
spurious pixel-diff failures.
Verbatim smoke test result on this codespace:
Smoke testing against vibium v26.3.18
PASS tc001 url_contains on httpbin form
PASS tc002 selector_visible h1 on httpbin /html
PASS tc003 failing assertion exits 1
PASS tc004 batch 3-step happy path
PASS tc005 batch stops on first failure
PASS tc006 visual-diff baseline created
PASS tc007 visual-diff second run matches
PASS tc008 check-injection clean page
PASS tc010 intent-score submit_form on httpbin form
PASS: 9 FAIL: 0 SKIPPED: 0
Unit tests still pass: 50/50 (one test had to be updated — it asserted
`buildScript` includes 'console.log', which is now correctly NOT in the
script after the eval-contract fix).
Phase 3 verification COMPLETE per ADR-091. The qe-browser helpers work
end-to-end against real Vibium + real Chromium + pinned httpbin.org
fixtures. The Linux ARM64 chromium-symlink workaround is documented in
ADR-091; user-facing docs in references/migration-from-playwright.md
still need updating (marked as Phase 4 follow-up).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|
Dragan Spiridonov
|
07f20e13cf |
docs(adr-091): Phase 3 attempt 1 — blocked on Linux ARM64 chromedriver
Ran the smoke test in this codespace (linux-arm64). Vibium installed
cleanly (v26.3.18) and the H1 detectVibium fix verified against real
binary output. tc003 and tc005 passed (failure-path tests that don't
need a browser). The other 7 tests failed because Google does not
publish chrome-for-testing for linux-arm64; Vibium downloaded the
linux64 (x86_64) variant which dies under Rosetta with an elf-loader
error.
Workarounds attempted and rejected:
- apt install chromium → native ARM64 binary works but Vibium
hardcodes its own cached chromedriver, no --browser-path flag
- searched the binary for VIBIUM_CHROME_PATH / --browser-path /
BROWSER env var support → none in v26.3.18
- System chromium responds at /json/version with CDP, not BiDi;
would need a BiDi-speaking chromedriver to use --connect
Conclusion: Phase 3 verification cannot complete on linux-arm64. This
is a Vibium upstream platform gap, NOT a qe-browser bug. The qe-browser
script-level integration with the real vibium binary IS partially
verified by the 2 passing tests.
Added "Vibium does not support Linux ARM64 today" to the Negative
Consequences section. PR reopen still gated on running smoke-test.sh
on linux-x64 or mac-arm64.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|