skills/proffesor-for-testing__agentic-qe
1
0
Fork 0
mirror of https://github.com/proffesor-for-testing/agentic-qe.git synced 2026-09-19 08:45:47 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,802 Commits 166 Branches 294 Tags
release/v3.13.3
Commit Graph

26 Commits

Author SHA1 Message Date
Dragan Spiridonov 3bd03ba2cc fix(memory): stop AQE_DISABLE_WAL from silently killing all learning capture 
Learning capture froze for six days (2026-07-23 .. 2026-07-29) while every hook
kept reporting success: true. Root cause chain:

  AQE_DISABLE_WAL=1
    -> safe-db ran `journal_mode = DELETE` on EVERY writable open
      -> that switch needs an EXCLUSIVE lock (busy_timeout does not help)
        -> a stale `npm exec agentic-qe mcp` held memory.db + -wal open
          -> SQLITE_BUSY -> openDatabase threw -> UnifiedMemoryManager init
             aborted -> ALL persistence died, silently

It could not self-heal: the switch can never succeed while any reader exists.

safe-db.ts now:
- reads the current journal_mode first and attempts the switch ONLY if it
  differs, so the steady state takes no lock at all and the one-time migration
  leaves the hot path;
- catches ONLY SQLITE_BUSY / SQLITE_LOCKED and rethrows everything else
  unchanged, so I/O errors and corruption are no longer misreported as contention;
- FAILS CLOSED on contention rather than writing anyway. AQE_DISABLE_WAL=1 is the
  operator declaring WAL unsafe on this mount, so continuing in WAL would trade a
  loud outage for silent corruption;
- verifies the switch actually took effect before allowing writes;
- names the real database file and the command that finds the actual holder.

The hook shim's health-log guidance named "concurrent AQE processes (MCP server /
daemon)". Operators checked exactly those and found nothing, because the culprit
was an npx-cache process matching neither "aqe" nor .mcp.json — that wrong hint is
most of why this ran six days. It now prints a /proc scan that finds ANY holder,
and only claims contention when the failure really was a lock error.

Also raises the statusline refresh from 5s to 60s; it reopened the 158MB DB on
every tick, which was constant added pressure on the same lock.

11 tests covering fail-closed, the non-lock rethrow, the skip-when-already-correct
path, durable write across reopen, and the diagnostic contents. Verified live:
journal_mode=delete, integrity_check=ok, captured_experiences growing again after
freezing at 21157.
 2026-07-29 14:46:48 +00:00
Dragan Spiridonov e5a7719aa7 feat(#569): add AQE_COVERAGE_NO_EXEC opt-out; broaden credential stripping (#568) 
Closes the two security questions raised by the adversarial review that the
prosecutors never returned a ruling on, so I settled them with evidence.

1. Coverage measurement executes the analyzed repository.
   Collecting real coverage means running the project's own tests: the JS/TS path
   runs `npx vitest|jest|nyc` (pre-existing), and the Rust path added in this
   release runs `cargo llvm-cov`, which compiles and runs test binaries, executes
   `build.rs`, and honors any `runner`/linker directive in that repo's
   `.cargo/config.toml`. For a trusted project that is exactly what the caller
   wants; it is surprising for an operation that reads like a static query, and an
   agent can be pointed at an untrusted checkout.

   Execution stays the default — requiring opt-in would make measured coverage the
   exceptional case and estimation the norm, inverting ADR-126's whole point — but
   is now opt-out via `AQE_COVERAGE_NO_EXEC=1`, which disables ALL build-tool
   execution. Existing coverage reports are still parsed; everything else degrades
   to clearly-labelled static estimation with guidance explaining why.

   Verified end-to-end through the real MCP server with a fake `cargo` shim on
   PATH: switch off -> cargo invoked; switch on -> not invoked. The unit test
   includes the same control, so it cannot pass vacuously.

2. Credential stripping in agentOverrides was already complete, but under-reported.
   `sanitizeAgentOverrides` builds its result as an ALLOW-LIST, so any field that
   is not provider/model/temperature/maxTokens/priority never survives — apiKey,
   token, password and friends were all dropped already. Only the *warning* was
   apiKey-specific, so a user who pasted a token got no feedback. The warning now
   covers all credential-shaped keys, and a test asserts no warning ever echoes a
   secret value.

ADR-126 gains a section recording the execution decision and what was rejected
(opt-in-by-default, sandboxing), plus the requirement that future language
delegations honor the same switch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-23 10:35:11 +00:00
Dragan Spiridonov 9b6de39792 fix: resolve issues #565, #567, #568, #569 (deps, LLM wiring, agent routing, coverage honesty) 
Four independent user-reported issues, each verified end-to-end through the real
MCP protocol server and adversarially reviewed via qe-court.

#565 — HIGH CVE reachable by consumers
`@huggingface/transformers` was an *optional* peerDependency, but npm >=7
auto-installs optional peers — `optional` only suppresses the error when the peer
cannot be resolved. Every `npm install agentic-qe` therefore pulled
onnxruntime-node -> adm-zip@0.5.x (GHSA-xcpc-8h2w-3j85), and no resolvable
version of that chain is unaffected. Dropped the peer (it stays a devDependency
and an opt-in for local in-process embeddings; both load sites already degrade
with an actionable message). Consumer audit: 4 HIGH -> 0.

#567 — test_generate_enhanced never invoked the LLM
The DI factory behind the task-executor path was registered as `(memory) => ...`,
so `llmRouter` never reached TestGeneratorService and the ADR-051 branch was
unreachable regardless of provider configuration. Threaded the router through the
registry -> resolveTestGeneratorService -> getTestGenerator (now async), honored
the documented `aiEnhancement` opt-out, and surfaced `llmEnhanced` /
`generationMode` / `note` so a silent fallback is diagnosable. `llmEnhanced` now
reflects what the LLM actually produced — a broken provider (401/timeout/empty)
no longer reports AI-enhanced output it did not produce. The CLI was unaffected
(it goes through the domain plugin, which was already wired).

#568 — per-agent provider routing had no on-disk surface
Added `agentOverrides` to `.agentic-qe/llm-config.json` with keyed-object merge,
validation that drops bad entries individually and strips leaked API keys, and
resolution ahead of the category defaults. Materialized the overrides into
routing rules the router actually evaluates, and threaded `agentType` through all
nine router-calling domain services — without it a parsed override built a rule
that never fired. Also added the missing `codex` entry to ALL_PROVIDER_TYPES,
which had made that provider unselectable through the router.

#569 — coverage_analyze_sublinear reported fabricated data for Rust
Coverage now carries provenance (`estimated` / `measured` / `coverageMethod`),
uncollected branch and function metrics report `null` instead of 0 or 100, gaps
derived from an estimate carry low confidence and an `estimated` flag, and Rust
crates are measured with `cargo llvm-cov` where available. Test code (`tests/`,
`benches/`, brace-matched `#[cfg(test)]` blocks) is excluded from production
coverage, and line numbers can no longer exceed the file length. Security scans
now state their analysis depth so "zero findings" is not read as "verified clean".

Adversarial review (qe-court, cross-vendor) found and this commit fixes: an
anchoring bug that dropped `latest_parser.rs` as a test file; brace matching that
counted braces inside string literals and comments; `#[cfg(test)]` on a
non-module item swallowing the following production function; crate detection
that skipped instrumentation for subdirectory targets, over-claimed nonexistent
targets, and hijacked polyglot subtrees; and a Cobertura fallback that rounded a
measured 47% to 0%.

ADR-125 (on-disk agent routing overrides) and ADR-126 (measurement provenance)
document the decisions; ADR-115 gains an amendment on optional peers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-23 09:59:49 +00:00
Dragan Spiridonov 4952d4eebe fix: remediate qe-court findings on the release diff (3 fatal + 4 significant) 
A qe-court adversarial review (cross-vendor Codex GPT + AQE devils-advocate +
security prosecutors, filing blind) BLOCKED the release by catching edge cases the
green unit suite missed. All confirmed charges fixed:

FATAL:
- F1 codex.ts / claude-code.ts: no 'error' listener on child.stdin — an early
  child exit (not-logged-in path) raised an uncaught EPIPE that crashed the host.
  Added a stdin error guard + try/catch (both providers — same template bug).
- F2 db-recovery.ts: non-atomic copyFileSync + no lock let concurrent hook
  processes tear the restored memory.db. Now: copy to a temp on the same fs →
  atomic rename, under a best-effort exclusive lock (skip if a peer is restoring).
- F3 db-recovery.ts: isDbHealthy opened readonly + integrity_check, which
  FALSE-NEGATIVES a HEALTHY WAL db when -shm is zeroed over virtiofs — the exact
  target failure mode — and would have parked the good DB + reverted to an older
  backup (silent data loss). Now the live-corruption signal is the SQLite HEADER:
  a -wal/-shm fault never corrupts the main file, so a header-valid DB is treated
  healthy and NEVER parked. Backups still get a full integrity_check before use.

SIGNIFICANT:
- spend-ledger.ts: a swallowed ALTER migration made every INSERT hit a missing
  column, throw, get caught, and silently drop ALL spend → budget-cap bypass.
  record() now tracks whether the column exists and INSERTs accordingly (spend is
  always recorded; attribution degrades gracefully).
- witness-verify.ts: verifyAqeReceipt was fail-OPEN under an empty allowlist (the
  default state) — a self-signed forgery returned valid:true. Now fail-closed
  ('no-trust-root') unless an explicit allowSelfSigned opt-in; harness path asserts
  alg==='ed25519'; unparseable allowlist JSON now logs loudly.
- gate-reexecute.ts accept/v1+sig: enforce a minimum paired-sample size (n>=8 —
  below it a "95% CI" is just the raw delta) and assert mean(samples)==sealed
  scalar (defends against a forged scalar/vector pairing).

Verified: qe-court acceptance eval 100%; 1498 shared/validation/llm/flywheel tests
pass (incl. new F2/F3 regressions, fail-closed, alg, min-n, consistency cases);
tsc clean; build green. Deferred (non-blocking rough edges, noted in the court
record): codex maxTokens/temperature passthrough, healthCheck-while-logged-out,
semaphore microtask race, stableStringify byte-parity (needs a real qe-harness sample).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-18 15:33:38 +00:00
Dragan Spiridonov 3fe85eace7 feat(kernel): auto-restore memory.db from verified backup on corrupt open (M3.2) 
Adopts the transferable half of ruflo v3.25.2 (backup auto-restore), adapted to
better-sqlite3. Previously a malformed memory.db surfaced errors and told the
operator to "restore manually" (unified-memory.ts:365); now the corrupt case
self-heals like the already-handled missing-file case.

- src/shared/db-recovery.ts: isDbHealthy() (integrity_check) + attemptAutoRestore()
  — on a malformed DB, park the corrupt original as <db>.corrupt-<ts> (NEVER delete),
  park stale -wal/-shm, and copy in the NEWEST backup that itself passes
  integrity_check from .agentic-qe/backups/verified/. Non-destructive + no-op when
  healthy or when no verified backup exists (left for manual inspection).
- safe-db.ts: opt-in `autoRestore` option (writable opens only), runs before open.
- unified-memory.ts: enables autoRestore on the memory.db open (missing-file case
  was already covered; this adds the exists-but-malformed case).

Data-protection: corrupt file is renamed not deleted; only integrity-verified
backups are used; matches scripts/aqe-db-backup.sh's verified-backup layout.

Verified: db-recovery 7/7 (restore newest, skip corrupt backup, park stale WAL,
non-destructive when no backup); 1171 shared tests pass; tsc clean; build green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-18 14:19:53 +00:00
Dragan Spiridonov 64d4936cc5 feat(llm): per-agent cost attribution in the spend ledger (ADR-124 M2.3) 
Extends the ADR-123 cross-process spend ledger to attribute each LLM charge to
the QE agent type that incurred it — the seed for a Cognitum per-agent billing
dashboard across the 60+ agent fleet.

- spend-ledger.ts: SpendEntry.agent; additive `agent_type` column via a guarded
  ALTER TABLE ADD COLUMN (non-destructive — existing rows get NULL, never drops
  or rewrites memory.db); new spentByAgentSince() grouping (unattributed charges
  bucket under '(unattributed)'); implemented on both Sqlite + InMemory ledgers.
- provider-manager.ts: recordSpend/recordResponseSpend take an optional agent.
- hybrid-router.ts: passes params.agentType at the record site (already in scope).

Verified: spend-ledger tests 9/9 (incl. an old-schema DB migrating without losing
its legacy row); full LLM suite 780/780; tsc clean; build green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-18 13:37:50 +00:00
Dragan Spiridonov 31fb100a8f feat(llm): add Codex CLI subscription provider (ADR-124 M3.5) 
Adds a `codex` provider that runs QE inference on the user's ChatGPT
subscription by shelling out to `codex exec`, mirroring the claude-code
subscription provider (ADR-123). Gives AQE a cross-vendor (GPT-family)
engine for provider fallback and — the motivating use — an independent
prosecutor/juror in qe-court (ADR-124), where a non-Claude, non-Cognitum
brain is the strongest form of the writer≠juror invariant.

- src/shared/llm/providers/codex.ts: subscription provider. Captures the
  final message via `codex exec -o <file>` (read-only sandbox), estimates
  token usage, cost $0 (subscription). LOAD-BEARING: strips OPENAI_API_KEY /
  CODEX_API_KEY from the child env so it cannot silently revert to API billing.
- interfaces.ts: 'codex' added to LLMProviderType; CodexConfig; providers map.
- provider-manager: case 'codex'. billing-modes: codex => 'subscription' +
  ChatGPT-aware startup notice. config-store: no env key (subscription) +
  runtime-constructible. Totality-required Record maps updated (cost-tracker,
  llm-router default model, message-formatter strategy).
- tests: env-strip parity + metadata (7 tests).

Verified: 209 provider/router unit tests pass; tsc clean; full build green;
a faithful inline reproduction of the provider's spawn hit the real `codex`
binary → "OK", exit 0, ~5.5s (gpt-5.5 via ChatGPT subscription).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-18 12:45:18 +00:00
Dragan Spiridonov b9a4b4c6e8 feat(llm): billing-aware execution — subscription provider, enforced budgets, real eval (ADR-123) 
Closes the structural gap from issue #557: AQE could only bill the paid
Anthropic API even for Pro/Max subscribers, and its budget caps were never
enforced.

- claude-code provider: run QE via `claude -p` on the user's subscription;
  child env strips API-billing keys so it can't revert to API billing
- Enforced cross-process spend caps (--max-budget-usd / AQE_MAX_BUDGET_USD),
  persisted to memory.db (llm_spend); enforced on the ProviderManager AND
  HybridRouter paths; over-budget aborts with COST_LIMIT_EXCEEDED
- cognitum provider: OpenAI/Anthropic-compatible gateway with per-request
  cost receipts, embeddings, and a server-side hard cap
- Billing-mode taxonomy + startup notice + `aqe health` LLM Billing section
- config-store: honor explicit `enabled: false` over env key presence;
  AQE_LLM_PROVIDER selector
- Fix: `aqe eval run` no longer scores fabricated mock responses — real
  provider by default, hard-error when none configured, `--mock` opt-in
- Consensus verification path prints a billing warning (deferred full port)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-13 11:49:28 +00:00
Dragan Spiridonov 0a8254f7a9 feat(learning): ADR-117..122 learning-integrity cluster (frozen anchor, two-gate judge, gate-reexecute, provenance tiers, receipt-gated flywheel, DoE) 
Adopts the receipt-gated self-improvement discipline from the 2026-07 cross-repo
analysis of ruflo (flywheel), MetaHarness (gateReExecutes), and retort (two-gate
spec conformance + DoE). All DI'd and unit-tested (178 tests); run end-to-end on
real qe_patterns; every limitation honestly scoped.

- ADR-117 frozen oracle anchor: hash-pinned, constant-denominator eval set that
  no promotion may regress; loader throws on drift (verification/anchors/qe-anchor-v1.json).
- ADR-118 receipt-gated QE-policy flywheel: generational loop over qe_patterns,
  conjunctive frozen accept(), Ed25519-signed receipts + compounding lineage,
  durable persistence (flywheel_receipts), corpus + semantic-retriever coupling.
- ADR-119 two-gate quality verdicts: mechanical gate + frontier judge, three-valued
  (pass/fail/inconclusive) with preflight; new `aqe quality-gate` CLI + `qe/quality/gate` MCP tool.
- ADR-120 gateReExecutes: re-execute a versioned frozen rule on sealed inputs;
  catches the A8-EXT forged-promote class; retrofit onto pattern-promote + dream-apply.
- ADR-121 provenance tiers: only oracle:test-exec evidence changes fleet behavior;
  additive, conservative-backfill migration.
- ADR-122 DoE screen: model x prompt x retrieval x scaffold, run on real data
  (25/27 cells, $1.25) — scaffolding (esp. reflexion) and retrieval degrade
  reliability (the retort `beads` signature); simplest+cheapest stack wins.
- safe-db: AQE_DISABLE_WAL / AQE_JOURNAL_MODE env-gate (macOS virtiofs bind-mount
  WAL corruption mitigation; default behavior unchanged).

Honest nulls documented: the flywheel lift is on a self-supervised proxy with the
anchor held flat; making the coupled anchor move needs harder items + a relevant
corpus (an ADR-117 re-freeze), not a retriever change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-09 10:26:54 +00:00
Dragan Spiridonov af4140efb9 feat(quality): oracle evals, durable-first tests, conservation guard + provider fixes 
Apply Phoenix Architecture principles to AQE — regenerate internals, conserve the interface.

ADR-113 (evals are oracles, durable-first tests, mutation-score gate):
- src/validation/oracle-eval.ts grades a generated test by running it against a
  reference impl + operator mutants (reuses qe-arena, ADR-104); rejects assertion-less output
- oracle mode wired into scripts/run-skill-eval.ts (skips under the simulating runner; runs live)
- qe-test-generation rewritten durable-first; its eval flipped to oracle
- src/feedback/regenerability-gate.ts: mutation-score + regenerability gate (warn-default, opt-in block)
- live-validated: Claude sonnet-4-6 clears the oracle 5/5; weak local-model output correctly fails

ADR-114 (conservation layer, Phoenix essay 14):
- src/validation/conservation-guard.ts + scripts/conservation-guard.ts guard CLI commands,
  output schemas, and dashboard API against breaking removals (deprecation registry exempts)
- policy doc + baselines in verification/conservation/; wired into invariant-check.yml non-blocking

LLM provider fixes:
- OpenRouterProvider.type was 'openai' but registered under 'openrouter' -> ProviderManager
  failover never found it; fixed to 'openrouter' end-to-end
- Gemini default model gemini-1.5-pro was retired -> gemini-2.5-flash; models/pricing refreshed
  from the live Gemini ListModels API
- scripts/provider-health.ts (npm run providers:health)

Cheaper-model eval lanes:
- scripts/oracle-model-bench.ts + openrouter-models.ts; openai/gpt-oss-120b clears the oracle
  ~50-70x cheaper than Claude Sonnet; guide in docs/guides/cheaper-model-eval-lanes.md

Verification: npm run build ok; 115 new+touched tests pass; 724 LLM tests pass; lint + parity clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-27 15:37:46 +00:00
Dragan Spiridonov 9fd18ed00a feat(code-intelligence): first-class C4 architecture diagrams (ADR-112) 
Consolidate C4 diagram generation onto a single engine and expose it to
users via CLI and MCP, with a deterministic confidence gate and real
Knowledge-Graph-derived relationships.

- Consolidation (C1/C6): C4ModelService is the single render/analyze/store
  engine; the bridge delegates and FAILS LOUD on a render error instead of
  silently degrading. Duplicate inline generators removed.
- C2: real component relationships from the Knowledge Graph (AST import/call
  edges) replace the naming heuristic; project-scoped via a new KG basePath
  so repos outside cwd don't trip the path-traversal guard.
- C3: deterministic confidence gate (high/medium/low + reasons) on every
  diagram — surfaces the detector's known limits instead of hiding them.
- C4/C5: `aqe code c4` CLI + `qe/code/c4` MCP tool (generate/search),
  verified at MCP-CLI parity and through the protocol-server bridge.
- Search: generate persists embeddings (opt-in `enableC4Embeddings`, on for
  MCP) so `qe/code/c4 search` returns hits in the standard flow.
- Fixed pre-existing lint in touched files (require->import, unused catches).

Docs: ADR-112 + docs/guides/c4-architecture-diagrams.md. tsc + lint clean;
C4 surface fully green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-27 11:27:25 +00:00
Dragan Spiridonov a42c3073e0 feat(shared): graceful optional-module loading + CI absent-drill (plan 05 / A9) 
ADR-150 graceful-degradation core. AQE's native/optional deps (@ruvector/*,
hnswlib-node, rvlite) must degrade — not crash — when a platform binary or a
Dependabot-pruned lockfile is missing.

- src/shared/optional-module.ts: loadOptionalModule(name, req?) → {available}|
  {degraded}. Fixes the #528 anti-pattern in the existing ad-hoc
  try{require}catch{} (6+ ruvector wrappers): degrades ONLY on genuine absence
  (MODULE_NOT_FOUND) and RE-THROWS real load failures (ABI mismatch / corrupt
  .node) instead of masking them. 8 tests incl. the rethrow property.
- .github/workflows/degraded-mode.yml: the absent drill — npm ci --omit=optional
  --ignore-scripts, then run the degraded-path tests. Pairs with the regular
  install job (proves both paths, #528).

Follow-up (held, CI-verified): migrate the ad-hoc sites + the static import in
sona-wrapper.ts onto loadOptionalModule (changes a production native-load path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-25 07:59:04 +00:00
Dragan Spiridonov 9c54a70656 feat(llm): ADR-088 amendment — wire enableCache to ephemeral cache breakpoints 
The enableCache flag existed since ADR-011 but was never read: every
advisor/LLM call paid full input price. Now:

- ClaudeProvider.generate() sends `system` as a content-block array with
  cache_control {type: ephemeral} when enableCache (default on); plain
  string when disabled. Anthropic-native provider only (ADR-092 — other
  providers untouched).
- cache_creation_input_tokens / cache_read_input_tokens surfaced into
  TokenUsage.cacheCreationTokens/cacheReadTokens; totalTokens now counts
  the full prompt (input_tokens alone is the uncached remainder).
- CostTracker bills cache reads at 0.1x and writes at 1.25x input rate
  instead of ignoring them.

5 new provider tests assert the serialized request shape (cache_control
present when enabled, absent when disabled, no system field when no
prompt) and the usage mapping. 64/64 LLM tests green, tsc clean.

Part of #520 (improvement 2/7, Tier 1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-10 09:41:49 +00:00
Dragan Spiridonov 8ff39f2c6e feat(learning): ADR-099 reasoning-tag scrub before distillation/embedding 
Fable 5-era models emit large extended-thinking blocks inside task results
and step actions; embedding them verbatim contaminates the 384-dim pattern
vectors in memory.db and degrades HNSW retrieval.

- new scrubReasoningBlocks()/scrubReasoningDeep() (src/shared/reasoning-scrub.ts):
  boundary-gated — only well-formed <think|thinking|reasoning|
  REASONING_SCRATCHPAD>...</> pairs are removed; prose mentioning a tag
  survives; clean input returned byte-identical
- applied at ingestion: TrajectoryTracker startTrajectory task,
  recordStep action/result.data/result.error
- applied at distillation: ExperienceReplay.storeExperience task/strategy/
  keyActions (insert + embedding text)
- applied at flow-forward: both init-time embedding backfills (legacy rows
  cleaned as they get embedded — no memory.db rewrite) and retrieveSimilar
  query embeddings (symmetric with stored vectors)
- 15 unit tests; E2E verified against a temp DB: trajectory + experience
  rows persisted with all scratchpad blocks stripped

Part of #520 (improvement 1/7, Tier 1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-10 09:35:25 +00:00
Dragan Spiridonov c3173a88ad feat(code-intelligence): parse TS/JS via bundled tree-sitter WASM — no typescript dependency (#511) 
Code-intelligence (index/deps/impact/search) required the multi-megabyte
`typescript` compiler at runtime; without it every TS/JS file errored
("TypeScript is required for code analysis") and produced 0 nodes/edges — i.e.
the feature was broken out of the box for the entire JS/TS ecosystem.

Bundle tree-sitter WASM grammars for typescript/tsx/javascript and extract
entities/imports from them instead:

- assets/grammars/tree-sitter-{typescript,tsx,javascript}.wasm (shipped via
  package `files: assets/**`).
- treesitter-ts-extractor.ts: extracts functions (incl. async arrows), classes +
  methods (with visibility), interfaces, and imports (incl. require()).
- knowledge-graph.ts: TS/JS extraction now prefers tree-sitter (typescript-free),
  falling back to the TypeScript compiler API only if tree-sitter is unavailable
  AND `typescript` happens to be installed.
- tree-sitter-wasm-parser.ts: export loadWasmGrammar() reusing the proven WASM
  init + grammar-path resolution.

Built with tree-sitter-cli 0.26.9 (matching web-tree-sitter ~0.26) via the new
scripts/build-grammars.sh (docker + emscripten/emsdk; reproducible — byte
identical across runs). Grammars built with the pre-0.25 emscripten dylink format
do NOT load with web-tree-sitter 0.26 (getDylinkMetadata error), which is why the
build pins the toolchain.

Verified on a real fresh project with NO typescript installed: code index → 8
nodes / 7 edges, deps → 2 connecting edges, search → 3 results. 232
code-intelligence tests + new extractor unit tests pass.

Note: the pre-existing assets/grammars/*.wasm for python/java/c#/rust/swift were
built with the old toolchain and currently fail to load with web-tree-sitter
0.26 (silently falling back to regex). They can be rebuilt with
`scripts/build-grammars.sh all` — tracked as a follow-up.
 2026-06-03 12:51:10 +00:00
Dragan Spiridonov 0d80eb56ec feat(perf): RaBitQ 1-bit signature utility + benchmark (#510 item 8) 
Ports ruflo's perf-M4 (sign-random-projection + Hamming popcount). Adds
src/shared/utils/rabitq.ts: signSignature (384-bit packed into 12 Uint32
words, 48 bytes), hammingDistance (SWAR popcount), hammingSimilarity, and
topKBySignature (Hamming prefilter gated at N>=100 + exact-cosine rerank of a
recall-aware pool, so output equals exact topK when the true top-k fall in the
pool). 16 tests + scripts/benchmark-rabitq.ts (npm run benchmark:rabitq).

HONEST FINDING (verified, does NOT match ruflo's optimistic 2.7x/10.9x):
  N=1000: speedup 1.31x, recall@10 100%, index 1.46MB -> 47KB (32x smaller)
  N=5000: speedup 1.18x, recall@10 99.9%, 7.32MB -> 234KB (32x smaller)
The 32x INDEX-MEMORY reduction is real and unconditional. The SPEEDUP is only
~1.2-1.5x in JS, not ruflo's figure: a 1-bit sign signature in 384-dim is
coarse, so preserving recall@10 >= 0.95 needs a large exact-rerank pool
(~40-55% of N), which eats most of the prefilter win. ruflo's big win came
from replacing a LINEAR scan; AQE's main search is already native HNSW
(sublinear), so that win doesn't transfer. Tests pin this tradeoff (recall==1.0
with a sufficient pool; small pool sacrifices recall) so it can't be
mis-sold later.

NOT INTEGRATED — no zero-regression consumer exists:
  - unified-memory-hnsw search() IS the ranking (no separate exact rerank to
    prefilter), so Hamming-prefiltering its beam would drop true neighbors;
  - coherence-gate-energy O(N^2) SUM and k-means clustering are numerically
    sensitive (Hamming would change results).
Committed as a tested, benchmarked, ready-to-use utility (real memory lever for
patterns.rvf and a future prefilter->exact-rerank consumer), not wired into any
production path. No production code or .agentic-qe stores touched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-02 10:30:10 +00:00
Dragan Spiridonov 95cd6ea7e1 fix(llm-router): post-audit sweep — update test for new fallback semantics + README 
Two follow-up fixes after re-running the broader test suite (2064 tests):

1. tests/unit/shared/llm/router/hybrid-router.test.ts:619 — the
   pre-existing test "should not retry non-retryable errors" was
   asserting the OLD short-circuit-on-non-retryable behavior that
   Fix #3 in the audit corrected. Updated to assert the new (correct)
   semantics: non-retryable errors don't re-try the SAME provider but
   DO fall through to the NEXT provider in the fallback chain.
   Renamed accordingly. Both providers fail non-retryable so the
   request still rejects, preserving the original "rejects" assertion.

2. README.md "LLM Providers" section was under-documented relative to
   the wired implementation. Updated to:
   - List all 7 ExtendedProviderType providers with their env var names
     (including the GOOGLE_API_KEY alias added in Phase 1)
   - Document `aqe llm config --set` persistence path
   - Document the AQE_LLM_ROUTER_DISABLED kill-switch
   - Mention that apiKey can't be saved via CLI (env-only)
   - Drop Groq from the table since it's not actually in the supported
     provider list

Verification
- npx tsc --noEmit: clean
- 2063/2063 unit + integration tests pass (1 skipped, no failures)
- Includes 38/38 hybrid-router tests after the assertion update
 2026-05-21 12:16:43 +00:00
Dragan Spiridonov 94033e3500 feat(llm-router): Phase 1 — kernel singleton + persistence + Gemini key alias 
Wires the foundation for ADR-043's HybridRouter into the kernel boot
sequence so domain plugins can finally receive an llmRouter dependency.
Phase 1 lands the kernel/CLI/provider changes; Phase 2 will plumb the
router through all 13 domains.

Background: the LLM-enhancement code paths added in ADR-051
(isLLMAnalysisAvailable() across 14 services) were unreachable because
no construction site ever passed an llmRouter — services accepted the
optional dependency but no caller injected it. ProviderManager +
HybridRouter were only instantiated from the `aqe llm advise` CLI
subcommand.

Changes
- src/shared/llm/providers/gemini.ts: read GOOGLE_API_KEY as a third
  fallback after GOOGLE_AI_API_KEY and GEMINI_API_KEY, so users with
  the conventional Google key name don't need to alias it themselves.
- src/shared/llm/router/config-store.ts (new): load/save
  .agentic-qe/llm-config.json with env-detection layering
  (defaults < disk < env < explicit override). Strips apiKey fields
  defensively before persisting. Filters out providers the
  ProviderManager doesn't yet construct (onnx) to keep boot logs
  clean.
- src/shared/llm/llm-router-service.ts (new): builds a HybridRouter
  from the loaded config, picks primary/fallbacks from the available
  set honoring the user's defaultProvider when possible. Accepts an
  injected ProviderManager for tests.
- src/kernel/interfaces.ts: extends QEKernel with llmRouter? and
  KernelConfig with llmRouter.enabled ('auto'|true|false) +
  configOverride + providerManager test-injection.
- src/kernel/kernel.ts: kernel-singleton llmRouter built during
  initialize(), exposed via kernel.llmRouter, forwarded to all 13
  DOMAIN_FACTORIES as a 4th arg. Phase 2 wires each factory through
  to its coordinator/services; for now the wrappers accept-and-discard.
- src/cli/commands/llm-router.ts: replaces the
  `// would be persisted in real implementation` stub with real
  read/write through the config-store. `aqe llm config --set` now
  survives a process restart. Adds AQE_CONFIG_ROOT env var as a test
  seam so tests don't mutate the real .agentic-qe/.

Tests (110 total, all passing)
- gemini.test.ts: +3 new tests for each env-var alias, existing
  tests updated to clear all three names
- config-store.test.ts (new, 29 tests): load/save/merge precedence,
  env detection, secret stripping, atomicity
- llm-router-service.test.ts (new, 10 tests): pickEnabledProviders,
  pickPrimaryAndFallbacks, end-to-end with injected mock provider
- kernel-llm-router.test.ts (new, 4 tests): kernel boot with router
  enabled=false / auto-no-providers / injected mock, chat() round-trip
- llm-router.test.ts (CLI): +2 persistence-to-disk tests, env-isolated
- tests/mocks/index.ts: shared createMockLLMProvider() helper

Verification
- npx tsc --noEmit: clean
- 110 tests pass; no regressions in existing kernel.test.ts (34/34)
- Stderr noise from PersistentSONAEngine init is pre-existing and
  unrelated to this change.

What this commit does NOT do
- Does NOT change behavior for any user yet — DOMAIN_FACTORIES
  wrappers still discard the llmRouter param. Phase 2 lights up the
  domain services.

Refs ADR-043 (vendor-independent LLM), ADR-051 (LLM-enhanced services)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-21 09:38:40 +00:00
Dragan Spiridonov b62eb83ad4 fix(learning): rehydrate Date fields at all kv-read seams (#493 follow-up) 
#493 was not the last instance of the kv-Date hazard — it was the most
reachable. Audit of the rest of the learning pipeline found 11 more
unprotected kv-read sites where TypeScript declares `Date` but JSON
round-trip through `memory.get` yields an ISO string at runtime.

Two failure modes, both already documented at
learning-coordinator.ts:898-904 (#491 Bug 3):

  1. THROW — `record.someDate.getTime()` raises TypeError. Surfaces
     immediately on the first invocation once any record exists.
  2. SILENT — `timeRange.contains(record.someDate)` or
     `new Date() > record.expiresAt` coerces string vs Date to NaN; all
     comparisons return false; records are silently filtered out or
     expiry is never recognised. No error, just empty/stale analytics.

The silent variant is the worse one — #491 Bug 3 was the same class
and shipped for a release cycle undetected.

Strategic fix: a single `rehydrateDates<T>(record, dateFields[])` helper
at `src/shared/utils/kv-date-rehydrate.ts` applied at every kv-read seam
that touches Date-bearing types. Shallow-copies the record and coerces
named string fields back to Date; pass-through for already-Date, null,
undefined.

Sites patched (11 total):

  src/domains/learning-optimization/services/
    production-intel.ts       — 4 sites (HIGH #2-#5 throw, MEDIUM #8 silent)
                                 getMetricsHistory, getRecentIncidents (×2),
                                 resolveIncident, getRecentMilestones
    metrics-optimizer.ts      — 1 site (HIGH #1 throw)
                                 getMetricsHistory.sort
    learning-coordinator.ts   — 1 site (HIGH #6 throw)
                                 getReplayBuffer.sort
    transfer-specialist.ts    — 4 sites (HIGH #7 throw, MEDIUM #9 silent)
                                 queryKnowledge (×2), getKnowledgeById,
                                 bulkTransfer
  src/domains/learning-optimization/
    coordinator-helpers.ts    — 1 site (SILENT)
                                 getExperiencesForDomain
    coordinator.ts            — 1 site (SILENT, duplicate of above)
                                 getExperiencesForDomain (private)

Sites verified SAFE (kv-read but only access non-Date fields):
  - coordinator.ts:1045/1134/1175 — uses .domain/.confidence/.version
  - metrics-optimizer.ts:676      — uses .metrics map
  - learning-coordinator.ts:379/438/483/547/830/842 — uses
    .confidence/.successRate/.domain/.usageCount
  - learning-coordinator.ts:901 — already has #491 Bug 3 inline
    rehydration with instanceof guard
  - experience-capture.ts:520/563 — TaskExperience uses number
    timestamps, not Date

Sites NOT patched (deferred per scope):
  - coordination/protocols/learning-consolidation.ts — defined but
    never imported anywhere. Dead code; would resurrect the bug if
    reactivated. Documented in the audit but not fixed.

Regression tests (29 new, 6 files):
  tests/unit/shared/utils/kv-date-rehydrate.test.ts             (10)
  tests/unit/domains/learning-optimization/
    production-intel-date-coercion.test.ts                       (5)
    metrics-optimizer-date-coercion.test.ts                      (2)
    learning-coordinator-date-coercion.test.ts                   (2)
    transfer-specialist-date-coercion.test.ts                    (5)
    coordinator-helpers-date-coercion.test.ts                    (2)

Each service test seeds its mock memory backend with kv-shape records
(ISO strings instead of Date instances — matching the post-JSON.parse
shape that better-sqlite3-backed memory yields in production) and
exercises the kv-read → consumer path that would fail pre-fix. Both
throw and silent failure modes are pinned: throws are asserted not to
fire, silent failures are caught by behavioural assertions (e.g. the
expired-knowledge test asserts `expired.score < live.score * 0.5 ±`,
which would fail pre-fix because the 0.5 penalty never triggered).

Verification:
  - 29/29 new regression tests green
  - 135/135 existing tests for patched services green (production-intel,
    metrics-optimizer, transfer-specialist, learning-coordinator,
    learning-consolidation, learning-consolidation-mining,
    learning-consolidation-liveness)
  - `tsc --noEmit` clean
  - `npm run build` produces dist with rehydrateDates at all 17
    expected call sites across the 6 patched files

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 08:38:53 +00:00
Dragan Spiridonov d918e78813 feat(adr-093): xhigh effort level fleet-wide with priority chain 
Commit 3 of 6 for ADR-093 Opus 4.7 migration. Adds effort-level plumbing
with xhigh as the fleet-wide default and max override on security agents.

* src/shared/llm/effort-resolver.ts (new): resolveEffortLevel() implements
  the ADR-093 priority chain:
    runtime override > per-agent frontmatter > QE_EFFORT_LEVEL env
      > config/fleet-defaults.yaml > DEFAULT_EFFORT_LEVEL (xhigh)
  Plus downgradeEffort() for models that don't support xhigh.
  Minimal regex-based YAML parse avoids a new dependency.
* config/fleet-defaults.yaml (new): effort_level: xhigh.
* src/shared/llm/interfaces.ts: GenerateOptions gains optional
  `effort?: 'low' | 'medium' | 'high' | 'xhigh' | 'max'`.
* src/shared/llm/providers/claude.ts: resolve effort per request and
  include in /v1/messages body; silently downgrade to 'high' when the
  target model lacks supportsEffortXHigh capability.
* .claude/agents/v3/qe-security-auditor.md,
  qe-pentest-validator.md,
  qe-security-scanner.md: add `effort: max` frontmatter per user
  direction. Also corrects stale `advisor.model: claude-opus-4-6`
  (non-existent ID) to `claude-opus-4-7` on the two agents that had it.

Note: fleet has 3 security/pentest qe-* agents (qe-security-auditor,
qe-security-scanner, qe-pentest-validator). The Cyber Verification
application lists 4 (including qe-security-reviewer which does not
exist in .claude/agents/v3/). If the submitted application needs an
amendment, re-file with the correct 3-agent list.

* tests/unit/shared/llm/effort-resolver.test.ts (new): 13 tests
  covering the priority chain, invalid-input fallback, YAML read,
  and downgrade helper.

Verification:
* 0 tsc errors (npx tsc --noEmit)
* 677/677 LLM unit tests pass (+13 new)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 12:44:10 +00:00
Dragan Spiridonov d55fcfbbda refactor(adr-093): sweep retiring model IDs to Opus 4.7/Sonnet 4.6/Haiku 4.5 
Commit 2 of 6 for ADR-093 Opus 4.7 migration. Real behavior change:
the AQE fleet now targets Sonnet 4.6, Haiku 4.5, Opus 4.7.

43 files changed, 253 insertions, 246 deletions.

Mechanical sed sweep across 29 src/ files and 14 tests/ files:
* 'claude-sonnet-4-20250514' → 'claude-sonnet-4-6'
* 'claude-3-5-haiku-20241022' → 'claude-haiku-4-5-20251001'
* 'claude-opus-4-5-20251101' → 'claude-opus-4-7'
* 'claude-opus-4-20250514' → 'claude-opus-4-7'

Data-definition files (model-mapping.ts, model-registry.ts,
cost-tracker.ts, model-mapping.test.ts, model-registry.test.ts)
deliberately excluded from sweep — they keep retiring IDs as
canonical data + regression coverage for backward-compat paths.

* src/shared/llm/cost-tracker.ts: added claude-opus-4-7 ($5/$25),
  claude-sonnet-4-6 ($3/$15), claude-haiku-4-5-20251001 ($1/$5)
  pricing entries alongside retained retiring entries.
* src/shared/llm/providers/bedrock.ts: rebuilt BEDROCK_MODEL_MAPPING
  to preserve both new ARNs (claude-opus-4-7-v1:0 etc.) and retiring
  ARNs; fixes duplicate-key compile error caused by sed collapsing
  'claude-opus-4-20250514' and 'claude-opus-4-5-20251101' to same key.
* src/shared/llm/interfaces.ts: ClaudeConfig.model type union
  updated to hint at new IDs; '| string' fallback keeps backward
  compat for lingering callers.
* tests updated: 3 provider/bedrock/cost-tracker assertions
  corrected to match new pricing + new ARN mappings.

Verification:
* 0 tsc errors (npx tsc --noEmit)
* 664/664 LLM unit tests pass
* 2302/2302 coordination + CLI tests pass
* 2884/2885 domain tests pass (1 pre-existing skip)
* grep -rcE 'retired-id-pattern' src/ returns 0 outside data files

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 12:32:24 +00:00
Dragan Spiridonov 3ee1fbf518 feat(adr-093): add Opus 4.7, Sonnet 4.6, Haiku 4.5 to model registry 
Commit 1 of 6 for ADR-093 Opus 4.7 migration. No runtime behavior change.

* src/shared/llm/model-mapping.ts: add MODEL_MAPPINGS entries for
  claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5 with
  anthropic / openrouter / bedrock IDs.
* src/shared/llm/model-registry.ts:
  - extend ModelCapabilities with optional supportsAdaptiveThinking,
    supportsEffortXHigh, tokenizerVersion fields (non-breaking)
  - add MODEL_REGISTRY entries for 4.7 (1M context, xhigh, opus-4-7
    tokenizer, $5/$25/MTok), 4.6 (standard 200k, legacy tokenizer,
    $3/$15), 4.5 haiku
  - mark claude-sonnet-4 deprecationDate 2026-06-15 (retired
    alongside Sonnet 4 per Anthropic announcement)
  - mark claude-haiku-3-5 deprecationDate 2026-02-19 (already retired)
  - export DEFAULT_SONNET_MODEL = 'claude-sonnet-4-6',
    DEFAULT_OPUS_MODEL = 'claude-opus-4-7',
    DEFAULT_HAIKU_MODEL = 'claude-haiku-4-5',
    RETIRING_MODELS registry with ISO retirement dates
* tests/unit/shared/llm/model-registry.test.ts: 14 new tests covering
  new registry entries, central constants, and deprecation metadata.
  All 154 tests across registry/mapping/cost-tracker pass.
* docs/implementation/adrs/ADR-093-opus-4-7-migration.md: new ADR
  capturing migration decisions (Sonnet 4.6 as Tier 3 default, Opus 4.7
  as escalation target, xhigh fleet-wide, thinking-config shim,
  Cyber Verification for security agents, Routines/Monitor deferred
  to ADR-094).
* docs/implementation/adrs/v3-adrs.md: ADR-093 index row added.
* docs/implementation/plans/ADR-093-IMPLEMENTATION-PLAN.md: 6-commit
  execution plan with per-commit verification + rollback strategy.
* docs/security/cyber-verification-application.md: Cyber Verification
  Program application draft (submitted 2026-04-17).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 12:16:24 +00:00
Dragan Spiridonov 0cc11ea05b fix: unify hypergraph persistence to memory.db, add CLI/MCP query tools 
Eliminates the separate hypergraph.db file — all hypergraph data now lives
in the unified .agentic-qe/memory.db alongside other QE tables. Fixes stale
v3/ path references in governance shards. Adds user-facing hypergraph query
surface via CLI (`aqe hypergraph stats/untested/impacted/gaps`) and MCP
(`hypergraph_query` tool).

Key changes:
- coordinator + coordinator-hypergraph default path: hypergraph.db → memory.db
- governance shards: v3/src/domains/ → src/domains/ (12 files + constitution)
- init phase 06: now builds hypergraph tables during `aqe init --auto`
- coordinator.index(): rebuilds hypergraph on every code_index MCP call
- protocol: removed redundant buildHypergraphFromIndex (coordinator handles it)
- shared code-index-extractor: async batched I/O, arrow/method/interface patterns
- CLI handler: ensureInitialized guard, connection leak fix, path resolution
- MCP handler: hypergraph_query tool with stats/untested/impacted/gaps queries
- tool-scoping: hypergraph_query added to 5 agent roles
- completions: bash/zsh/fish/powershell support for hypergraph subcommands
- coordinator: publishes HypergraphDegraded event on init failure
- 12 new tests for code-index-extractor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-23 13:48:09 +00:00
Dragan Spiridonov 00e4d2cfa6 fix(tests): add afterEach cleanup to 104 test files missing it (#374 P2) 
125 test files had beforeEach/beforeAll but no afterEach/afterAll,
leaking state between tests and causing order-dependent failures.

Added cleanup to 104 files (skipped 4 archived, 16 import-only):
- 2 critical DB handle leaks (witness-chain): added db.close()
- 7 high-risk service files: added vi.restoreAllMocks()
- 7 beforeAll files: added matching afterAll
- 88 medium/low risk: added afterEach with appropriate cleanup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-21 12:51:47 +00:00
Profa ec2c243172 fix(embeddings): complete 384-dim standardization across tests and providers 
- Update default embed model from nomic-embed-text to all-MiniLM-L6-v2
  in Ollama LLM provider
- Fix 15 test files with hardcoded 768-dim references
- Update mock model names in Ollama health checks
- Update comments referencing old model name

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 12:33:47 +00:00
Profa 2253d05618 feat: flatten project structure — archive v2, promote v3 to root 
Remove the nested v3/ subdirectory by moving all active source code,
tests, configs, and assets directly to the project root. Archive v2/
(dead code, 847MB) to v2-archive.tar.gz.

Key changes:
- Move v3/src/ → src/, v3/tests/ → tests/, v3/packages/ → packages/
- Move v3/tsconfig.json, v3/vitest.config.ts to root
- Merge v3/package.json into root package.json (ESM, scripts, deps, exports)
- Merge v3/scripts/ into scripts/, v3/implementation/ into docs/implementation/
- Update all CI workflows to remove "cd v3 &&" and v3/ path prefixes
- Fix all init installer path resolution (3-level-up → 2-level-up)
- Fix hooks to use "node ./dist/cli/bundle.js" instead of bare "aqe"
- Update settings.json ADR/DDD directories to docs/implementation/
- Fix helper scripts (statusline, adr-compliance, ddd-tracker, guidance-hooks)
- Update 58 agent definitions: aqe/v3/ → aqe/ memory namespaces
- Update 15 guidance shards: v3/src/ → src/ path references
- Fix sync interfaces: remove v3/ and v2/ path references
- Fix test path calculations (process.cwd() + '..' no longer needed)
- Fix flaky perf test threshold (500ms → 1000ms for CI environments)
- Fix complexity-analyzer test to match ADR-051 keyword-based eligibility

Build passes, 17,901 tests pass, CLI and MCP bundles verified.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 11:35:41 +00:00