Version bumped in all four places that carry it: package.json,
.claude-plugin/plugin.json, and the fleetVersion in both the .claude/ and
assets/ skills manifests (the assets mirror drifts easily and is shipped).
Adds the v3.13.1 changelog entry and release notes covering the four fixed
issues (#565, #567, #568, #569) plus the Codex QE skills and lifecycle hooks
from a parallel session (763ea385), and draft issue-closing comments to post
once the release is published.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-vendor (Codex/GPT) review of the finished work. Three of its five findings
reproduced; one did not, and is recorded below so the review stays calibrated.
CONFIRMED — a report containing only test files reported them as production
coverage. `productionOnly` filtering fell back to the UNFILTERED file list when
no production files survived, so an LCOV whose only entry was
`tests/verifier_matrix.rs` came back as a measured 79% coverage target. That is
issue #569's third contradiction, verbatim, on a path the original fix did not
touch. An empty production set now yields no production coverage and says why.
CONFIRMED — the aggregate was carried over from the unfiltered set. Filtering
test files out of `files` while keeping `existing.summary` meant the headline
figure still blended test coverage in. In practice `CoverageAnalyzerService.analyze`
recomputes the summary from `files`, so the emitted number was already correct —
but the handler was relying on a downstream accident. The summary is now
recomputed where the filtering happens.
CONFIRMED — saved LCOV presented estimates as instrumented output. LCOV has
nowhere to record provenance, so `generateLcov` turned a `null` into
`DA:1,0 / LF:100 / LH:0` — a measured 0% over 100 lines that nothing measured —
and a 78% estimate into an apparently instrumented 78%. LCOV is now written only
for `measured: true`; the JSON artifact continues to carry the full result with
its provenance. This was the reviewer's stated highest-value fix and its reason
for a do-not-ship verdict.
NOT CONFIRMED — the reviewer reported the headline figure as 55% for a
10%-production/100%-test fixture. Reproduced and measured: it returns 10. The
finding was derived statically and missed the downstream recomputation. Fixed the
underlying fragility anyway, but the defect as described did not exist.
Also reverts three `.claude/helpers/` files (`.helpers-version`,
`helpers.manifest.json`, `statusline.cjs`) that a hook swept into 9b6de397. They
are unrelated to all four issues, and they shipped internally inconsistent
metadata — `.helpers-version` said 3.32.7 while the manifest said 3.32.4 — plus a
whole-file CRLF rewrite of the statusline. Restored to the 001c042a baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nullable coverage metrics were a visible contract change with nothing a user
would ever read to discover it. Adds an Unreleased section covering all four
issues, the two new opt-ins (`agentOverrides`, `AQE_COVERAGE_NO_EXEC`), the
session-cache staleness fix, and — under Changed — the BREAKING note that
`lineCoverage`/`statementCoverage`/`branchCoverage`/`functionCoverage` are now
`number | null`, with `0` meaning a measured zero and `null` meaning unknown.
Written outcome-first: what the tool used to tell you that wasn't true, and what
it does now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found by the qe-devils-advocate session review, AFTER #569 was already committed.
The handler was producing honest provenance; `coverageAnalyzeConfig.mapToResult`
was converting it back into fabricated certainty on three paths.
1. Absent provenance was treated as "collected".
`data.branchDataCollected !== false` — and `undefined !== false` is true. Any
producer that supplies no provenance (notably the coverage-analysis domain
plugin, whose handler returns a nested-`summary` CoverageReport) therefore
surfaced issue #569's EXACT reported symptom: 100% branch coverage alongside
0% function coverage, labelled `measured: true`, with `confidence: 0.7` on its
gaps. Now requires an explicit `true`; absence means unknown, so an
unprovenanced result reports `null`.
2. `measured` was derived as `!estimated`.
These are not complements — there is a third state where nothing was collected
at all. An empty project reported `lineCoverage: 0, measured: true`: "we
measured 0%" when the truth was "we measured nothing". Verified through the
real MCP server before and after. `measured` now requires an explicit claim.
3. `null` was coerced to `0`, and risk was graded off it.
`?? 0` turned "no data" into a plottable 0%, and `null < 70` is `false` in JS,
so an unguarded comparison graded "no data" as 'low' risk — the most dangerous
possible default. Nulls now survive; risk grading requires a real number.
Also fixes the stale-result mystery that dogged verification all session.
`src/optimization/session-cache.ts` fingerprints on domain+action+input only, and
persists to kv_store with a 1-hour TTL, so entries survive package upgrades: a
project analyzed before a fix keeps replaying the old answer, making the fix look
broken. I twice mis-diagnosed this (first as a cache, then wrongly retracted that;
it WAS the cache). The fingerprint now includes the build identity, so an upgrade
misses cleanly and old entries age out. `AQE_SESSION_CACHE_SALT` forces a cold
cache without deleting anything.
This matters beyond #569: every fix in this release would have appeared not to
work for any user with a warm cache.
`CoverageAnalyzeResult.lineCoverage`/`statementCoverage` are now `number | null`,
joining `branchCoverage`/`functionCoverage`. `0` means a measured zero; `null`
means unknown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the two security questions raised by the adversarial review that the
prosecutors never returned a ruling on, so I settled them with evidence.
1. Coverage measurement executes the analyzed repository.
Collecting real coverage means running the project's own tests: the JS/TS path
runs `npx vitest|jest|nyc` (pre-existing), and the Rust path added in this
release runs `cargo llvm-cov`, which compiles and runs test binaries, executes
`build.rs`, and honors any `runner`/linker directive in that repo's
`.cargo/config.toml`. For a trusted project that is exactly what the caller
wants; it is surprising for an operation that reads like a static query, and an
agent can be pointed at an untrusted checkout.
Execution stays the default — requiring opt-in would make measured coverage the
exceptional case and estimation the norm, inverting ADR-126's whole point — but
is now opt-out via `AQE_COVERAGE_NO_EXEC=1`, which disables ALL build-tool
execution. Existing coverage reports are still parsed; everything else degrades
to clearly-labelled static estimation with guidance explaining why.
Verified end-to-end through the real MCP server with a fake `cargo` shim on
PATH: switch off -> cargo invoked; switch on -> not invoked. The unit test
includes the same control, so it cannot pass vacuously.
2. Credential stripping in agentOverrides was already complete, but under-reported.
`sanitizeAgentOverrides` builds its result as an ALLOW-LIST, so any field that
is not provider/model/temperature/maxTokens/priority never survives — apiKey,
token, password and friends were all dropped already. Only the *warning* was
apiKey-specific, so a user who pasted a token got no feedback. The warning now
covers all credential-shaped keys, and a test asserts no warning ever echoes a
secret value.
ADR-126 gains a section recording the execution decision and what was rejected
(opt-in-by-default, sandboxing), plus the requirement that future language
delegations honor the same switch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up court charge, confirmed by reproduction.
`isTestPath()` checked every segment of the ABSOLUTE path against the test-directory
set. A project that merely *lives* under a directory called `tests`, `spec`, or
`examples` — a CI workspace at `/build/spec/proj`, `~/examples/myapp`, a monorepo
`packages/test-utils/` subtree — therefore had every one of its source files
classified as test code. `buildEstimatedCoverage()` then returned `null` and
coverage analysis reported nothing at all for the whole project.
Only directories *inside* the analyzed tree are meaningful, so classification now
takes the analysis root and evaluates the relative path. Threaded through all four
call sites (source walk, cargo LCOV filter, and both instrumented-report filters).
Verified: a crate at `<tmp>/tests/myapp` now analyzes its `src/lib.rs` instead of
returning null, while a `tests/` directory *inside* the project is still excluded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four independent user-reported issues, each verified end-to-end through the real
MCP protocol server and adversarially reviewed via qe-court.
#565 — HIGH CVE reachable by consumers
`@huggingface/transformers` was an *optional* peerDependency, but npm >=7
auto-installs optional peers — `optional` only suppresses the error when the peer
cannot be resolved. Every `npm install agentic-qe` therefore pulled
onnxruntime-node -> adm-zip@0.5.x (GHSA-xcpc-8h2w-3j85), and no resolvable
version of that chain is unaffected. Dropped the peer (it stays a devDependency
and an opt-in for local in-process embeddings; both load sites already degrade
with an actionable message). Consumer audit: 4 HIGH -> 0.
#567 — test_generate_enhanced never invoked the LLM
The DI factory behind the task-executor path was registered as `(memory) => ...`,
so `llmRouter` never reached TestGeneratorService and the ADR-051 branch was
unreachable regardless of provider configuration. Threaded the router through the
registry -> resolveTestGeneratorService -> getTestGenerator (now async), honored
the documented `aiEnhancement` opt-out, and surfaced `llmEnhanced` /
`generationMode` / `note` so a silent fallback is diagnosable. `llmEnhanced` now
reflects what the LLM actually produced — a broken provider (401/timeout/empty)
no longer reports AI-enhanced output it did not produce. The CLI was unaffected
(it goes through the domain plugin, which was already wired).
#568 — per-agent provider routing had no on-disk surface
Added `agentOverrides` to `.agentic-qe/llm-config.json` with keyed-object merge,
validation that drops bad entries individually and strips leaked API keys, and
resolution ahead of the category defaults. Materialized the overrides into
routing rules the router actually evaluates, and threaded `agentType` through all
nine router-calling domain services — without it a parsed override built a rule
that never fired. Also added the missing `codex` entry to ALL_PROVIDER_TYPES,
which had made that provider unselectable through the router.
#569 — coverage_analyze_sublinear reported fabricated data for Rust
Coverage now carries provenance (`estimated` / `measured` / `coverageMethod`),
uncollected branch and function metrics report `null` instead of 0 or 100, gaps
derived from an estimate carry low confidence and an `estimated` flag, and Rust
crates are measured with `cargo llvm-cov` where available. Test code (`tests/`,
`benches/`, brace-matched `#[cfg(test)]` blocks) is excluded from production
coverage, and line numbers can no longer exceed the file length. Security scans
now state their analysis depth so "zero findings" is not read as "verified clean".
Adversarial review (qe-court, cross-vendor) found and this commit fixes: an
anchoring bug that dropped `latest_parser.rs` as a test file; brace matching that
counted braces inside string literals and comments; `#[cfg(test)]` on a
non-module item swallowing the following production function; crate detection
that skipped instrumentation for subdirectory targets, over-claimed nonexistent
targets, and hijacked polyglot subtrees; and a Cobertura fallback that rounded a
measured 47% to 0%.
ADR-125 (on-disk agent routing overrides) and ADR-126 (measurement provenance)
document the decisions; ADR-115 gains an amendment on optional peers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A self-contained harness that measures, per quality-engineering task, the
cheapest tier that clears the quality bar against api.cognitum.one — so tier
routing (e.g. the qe-court model table) is set by evidence, not guessed.
Structure (scripts/cognitum-qe-bench/):
- 7 QE tasks spanning the oracle-provenance ladder: test-generation &
mutation-adequacy (execution), security-triage / pr-severity / flaky-diagnosis
/ adversarial-review (labeled), coverage-gap (structural).
- test-generation is graded by MUTATION SCORE over a diverse mutant set with
per-test validity, truncation-repair, and emission-vs-quality separation —
operators grounded in .claude/skills/mutation-testing/references.
- Three policies per task: P1 always-cheap, P2 always-frontier, P3 cheap+escalate
(high tier only as an advisor on low cheap-confidence).
- Cost is ground truth from each x_cognitum.price_usd receipt; each row carries
resolvedModel + requestId (reconcilable 1:1 against usage_ledger).
- Keys come from local config (env / .env), never code; two-key split (worker
low/mid, judge high) is enforced by scope. mint-bench-keys.mjs is a separate
one-time owner tool (safe two-part write: metered subscription + scope-split
key). README documents setup for team members.
- Safety: dry-run default, --max-cost pre-flight abort, mid-run cost stop,
AQE_MAX_BUDGET_USD backstop. Emits ruflo-compatible cognitum-qe.v1 records.
First measured finding: on these QE tasks the cheap tier (glm-5.2) matches or
beats the frontier tier at 3-4x less cost; on test-generation the frontier tier
failed to emit runnable code (emit 0%) while cheap scored a full mutation score.
Co-Authored-By: Ruflo & AQE
The user-facing intro's "Selected QE skills available" table omitted qe-court and
still read 75. Adds /qe-court to the Review category and updates the count to 76.
AQE-QE-scoped only (excludes ruflo/platform/dev skills); the on-disk "115 skill
definitions" line is already correct (qe-court made it 114→115).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A qe-court adversarial review (cross-vendor Codex GPT + AQE devils-advocate +
security prosecutors, filing blind) BLOCKED the release by catching edge cases the
green unit suite missed. All confirmed charges fixed:
FATAL:
- F1 codex.ts / claude-code.ts: no 'error' listener on child.stdin — an early
child exit (not-logged-in path) raised an uncaught EPIPE that crashed the host.
Added a stdin error guard + try/catch (both providers — same template bug).
- F2 db-recovery.ts: non-atomic copyFileSync + no lock let concurrent hook
processes tear the restored memory.db. Now: copy to a temp on the same fs →
atomic rename, under a best-effort exclusive lock (skip if a peer is restoring).
- F3 db-recovery.ts: isDbHealthy opened readonly + integrity_check, which
FALSE-NEGATIVES a HEALTHY WAL db when -shm is zeroed over virtiofs — the exact
target failure mode — and would have parked the good DB + reverted to an older
backup (silent data loss). Now the live-corruption signal is the SQLite HEADER:
a -wal/-shm fault never corrupts the main file, so a header-valid DB is treated
healthy and NEVER parked. Backups still get a full integrity_check before use.
SIGNIFICANT:
- spend-ledger.ts: a swallowed ALTER migration made every INSERT hit a missing
column, throw, get caught, and silently drop ALL spend → budget-cap bypass.
record() now tracks whether the column exists and INSERTs accordingly (spend is
always recorded; attribution degrades gracefully).
- witness-verify.ts: verifyAqeReceipt was fail-OPEN under an empty allowlist (the
default state) — a self-signed forgery returned valid:true. Now fail-closed
('no-trust-root') unless an explicit allowSelfSigned opt-in; harness path asserts
alg==='ed25519'; unparseable allowlist JSON now logs loudly.
- gate-reexecute.ts accept/v1+sig: enforce a minimum paired-sample size (n>=8 —
below it a "95% CI" is just the raw delta) and assert mean(samples)==sealed
scalar (defends against a forged scalar/vector pairing).
Verified: qe-court acceptance eval 100%; 1498 shared/validation/llm/flywheel tests
pass (incl. new F2/F3 regressions, fail-closed, alg, min-n, consistency cases);
tsc clean; build green. Deferred (non-blocking rough edges, noted in the court
record): codex maxTokens/temperature passthrough, healthCheck-while-logged-out,
semaphore microtask race, stableStringify byte-parity (needs a real qe-harness sample).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes a real gap: AQE's frozen accept/v1 promotes when the candidate's held-out
MEAN merely beats baseline by ANY amount (gate-reexecute.ts:72) — i.e. it can
promote a within-noise gain. ruflo's flywheel already gates on significance
(accept/v1+sig, paired bootstrap CI > 0); this is the symmetric ADOPTION of that
technique (reimplemented dependency-free — AQE does not import ruflo).
- src/validation/paired-bootstrap.ts: deterministic (fixed-seed LCG) one-sided
95% lower bound on the mean of PAIRED per-task deltas. Determinism is required —
the promotion is replay-verified (ADR-120), so the bound must reproduce exactly.
- gate-reexecute.ts: new frozen rule `accept/v1+sig` = accept/v1 AND
bootstrapDeltaCILow(paired held-out deltas) > 0. accept/v1 is UNTOUCHED (added a
new registry key the ADR-120 way). Fail-closed: rejects if paired samples are
absent/mismatched. SealedInputs gains optional per-task sample vectors (omitted
from the hash when undefined → accept/v1 receipts byte-identical).
- corpus-scorer.ts: scoreHeldOut now returns the per-query reciprocal-rank vector
(the paired sample) alongside its mean → PolicyScores.heldOutSamples.
- generation.ts: seals the samples ONLY when ruleVersion === 'accept/v1+sig'
(opt-in); default accept/v1 behavior + receipt bytes unchanged.
Why paired-bootstrap not ANVOA: for 2 groups ANOVA F = t² and unpaired; the
paired CI exploits the task-pairing and is strictly stronger. (AQE's ADR-122
ANOVA stays the right tool for offline multi-factor screening — complementary.)
Verified: new accept/v1+sig tests incl. the marquee gap case (a within-noise gain
accept/v1 PROMOTES but accept/v1+sig REJECTS); 321 flywheel+validation tests pass;
tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Source-verified in /workspaces/ruflo: the flywheel already refuses to promote a
candidate whose paired candidate-vs-baseline delta isn't significant —
`significant = bootstrapDeltaCILow(paired holdout deltas) > 0`, rule accept/v1+sig,
one-sided 95% paired bootstrap, n>=20 (evolve-proof.ts:174-183). That is STRONGER
than the unpaired ANOVA we were going to contribute (2-group ANOVA F = t², and it
drops the pairing). Our premise ("maybe ruflo promotes noise") was wrong.
Outcome: NO contribution PR (would be redundant and weaker). Corrects the strategy
report's open-Q #3 (ruflo's flywheel IS rigorously DoE-validated) and retires the
M3.1 action. The draft is kept, marked superseded, documenting the one genuinely-
additive idea (an ANOVA candidate-SELECTION veto) and why it's a bigger, replay-
verifier-touching change — deferred, not pursued.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the transferable half of ruflo v3.25.2 (backup auto-restore), adapted to
better-sqlite3. Previously a malformed memory.db surfaced errors and told the
operator to "restore manually" (unified-memory.ts:365); now the corrupt case
self-heals like the already-handled missing-file case.
- src/shared/db-recovery.ts: isDbHealthy() (integrity_check) + attemptAutoRestore()
— on a malformed DB, park the corrupt original as <db>.corrupt-<ts> (NEVER delete),
park stale -wal/-shm, and copy in the NEWEST backup that itself passes
integrity_check from .agentic-qe/backups/verified/. Non-destructive + no-op when
healthy or when no verified backup exists (left for manual inspection).
- safe-db.ts: opt-in `autoRestore` option (writable opens only), runs before open.
- unified-memory.ts: enables autoRestore on the memory.db open (missing-file case
was already covered; this adds the exists-but-malformed case).
Data-protection: corrupt file is renamed not deleted; only integrity-verified
backups are used; matches scripts/aqe-db-backup.sh's verified-backup layout.
Verified: db-recovery 7/7 (restore newest, skip corrupt backup, park stale WAL,
non-destructive when no backup); 1171 shared tests pass; tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M3.1: a ready-to-post, collegial issue draft offering AQE's DoE/ANOVA screen
(ADR-122) to validate ruflo's ADR-176 flywheel promotions — the "2 compounding
promotions / flat human anchor" shape matches AQE's own "beads" null. HELD from
posting to ruvnet/ruflo pending the user (a ruflo collaborator).
M3.2: assessment — ruflo v3.25.2's torn-full-image-flush fix does not apply to
AQE (better-sqlite3 WAL = atomic commits, not sql.js full-image flushes); AQE's
actual corruption class (virtiofs WAL/shm) is already mitigated in safe-db.ts.
Transferable lesson filed: auto-restore-on-malformed-open (AQE restores manually
today) — a scoped follow-up, not built here (memory.db recovery is data-sensitive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md references ruflo/claude-flow MCP tools heavily, which can read as a
runtime dependency. It isn't: the shipped agentic-qe package imports only
@ruvector/* primitives + two optional @claude-flow peers, never the ruflo npm
package. Adds a CONTRIBUTING note so contributors don't wire product code to a
ruflo CLI/MCP call or add it as a runtime dep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(A) npm "@ruvector Lattice WASM" = vaporware (E404; claude-flow's own source admits
the default package doesn't exist; inert opt-in slot) — the ruvllm mock-mode pattern.
(B) github.com/ohdearquant/lattice = a REAL pure-Rust inference engine (runs on
arm64 here, pushed 2026-07-18). But the shipped binary is chat-only — serve exposes
/v1/chat/completions,/v1/models,/health, NO /v1/embeddings (source
crates/inference/src/serve/mod.rs:248). Embeddings are a separate lattice-embed
crate, source-build-only, CLI not HTTP. Models incl. BGE-small 384 (default) …
Qwen3-Embedding 1024.
Verdict: GO-with-caveats, NOT a drop-in and not the cheapest path — the near-term
win (BGE-small @384, MTEB-stronger than MiniLM, same dim) is reachable via AQE's
existing transformers.js path (Xenova/bge-small-en-v1.5), no Rust/shim. Keep lattice
for a bigger move (Qwen3-1024, SIMD perf, Cognitum-side embedding serving).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the single AQE-side verifier that reconciles AQE flywheel receipts with
Cognitum qe-harness witness bundles, so QE evidence signed by the shared platform
identity (fingerprint f1ac28607da49ec1) is verifiable on either side.
Investigation (read-only, cognitum-one/qe-harness) established the bridge is
HALF-wired already: both sides use Ed25519, the same QE_WITNESS_SIGNING_KEY (JSON
PEM), and the IDENTICAL fingerprint scheme sha256(pem.trim())[:16] — both sign as
f1ac28607da49ec1. What was NOT aligned: the signed artifacts (AQE = fixed-order
JSON body, hex; qe-harness = stableStringify sha256: digest, base64, nested
witness{}) are not natively cross-verifiable.
- src/learning/qe-flywheel/witness-verify.ts: allowlist-driven verifier for BOTH
formats + stableStringify/canonicalDigest (qe-harness parity), tamper-detecting,
fail-closed allowlist. node:crypto only; reuses receipt.ts for the AQE path.
- 15 tests: fingerprint parity (keyFingerprint === signer.keyId), both formats
round-trip, tamper → bad-signature/digest-mismatch, allowlist enforcement.
Two external dependencies remain (need the qe-harness owner — documented in the
GOAP plan, NOT done unilaterally): provision the f1ac28607da49ec1 PUBLIC key via
QE_WITNESS_PUBLIC_KEYS_JSON; confirm byte-parity vs a real qe-harness sample;
decide canonicalization convergence (AQE's hand-listed body silently drops new
fields — recommend adopting stableStringify).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the ADR-123 cross-process spend ledger to attribute each LLM charge to
the QE agent type that incurred it — the seed for a Cognitum per-agent billing
dashboard across the 60+ agent fleet.
- spend-ledger.ts: SpendEntry.agent; additive `agent_type` column via a guarded
ALTER TABLE ADD COLUMN (non-destructive — existing rows get NULL, never drops
or rewrites memory.db); new spentByAgentSince() grouping (unattributed charges
bucket under '(unattributed)'); implemented on both Sqlite + InMemory ledgers.
- provider-manager.ts: recordSpend/recordResponseSpend take an optional agent.
- hybrid-router.ts: passes params.agentType at the record site (already in scope).
Verified: spend-ledger tests 9/9 (incl. an old-schema DB migrating without losing
its legacy row); full LLM suite 780/780; tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hosted qe-court: POST a delivery (diff/patch/files), get a signed,
overturn-tested verdict. Grounded in the M1 live probe — a NEW route on the
existing api.cognitum.one inference service (X-API-Key, x_cognitum receipts),
NOT the ADR-308 funnel (which 404s on that host); /v1/qe/* confirmed free.
Verdict semantics reference the OSS referee (src/skills/qe-court/referee.ts) so
client and server agree: 3-valued verdict, the overturn mechanic (overturnDepth),
writer≠juror + vendor-diversity (422 PANEL_INVARIANT_VIOLATION), DoE-gated score,
Ed25519-signed record (ADR-118), provenance-tiered charges (ADR-121). Sync + async
job models; budget/cap semantics mirror /v1/chat/completions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M1 resolved: api.cognitum.one's inference API and the ADR-308 funnel/proxy API
are NOT the same backend. /v1/proxy/*, /v1/credits, /v1/events, /v1/funnel-policy
all 404 ROUTE_NOT_FOUND on the inference host; the funnel layer is a separate
service (confirms ADR-308's addendum). /v1/qe/* namespace is free (404) — build
ADR-124 Phase 2 as a new route on the existing inference Express app. Hard cap
confirmed: hardCapUsd $20, headroom $19.34. Tiers route to a model marketplace
(cognitum-low -> z-ai/glm-5.2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turns qe-court's falsifiable claims into enforced code and earns trust_tier 3 on
real, executed evidence (no self-report):
- src/skills/qe-court/referee.ts: the court's rules as pure functions —
resolveVerdict() with the overturn mechanic (a SHIP must SURVIVE escalation;
a mutant surfaced only at depth>=1 does NOT count at overturnDepth=0, so it
regresses to SHIP — the false-SHIP the court exists to catch), validatePanel()
(writer!=juror, >=2 distinct vendors), shouldEmitScore() (DoE-gated, no
un-validated number).
- tests/unit/skills/qe-court/referee.test.ts: 7 oracle cases — each passes on
correct logic and FAILS on a seeded regression.
- evals/qe-court.yaml: converted to a command-eval suite (6 cases) that runs the
oracles through the `aqe eval` CLI.
- SKILL.md: trust_tier 1 -> 3, validation status passing (6/6, 2026-07-18).
- skills-manifest (both trees): Tier-3 count 49 -> 50, additional 33 -> 32.
Verified: `aqe eval run --skill qe-court` -> PASSED 6/6 100%; referee suite 7/7;
skill parity clean; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six Thinking Hats strategy report and the precondition-ordered GOAP execution
plan for the AQE <-> ruflo <-> Cognitum integration work. Captures: what's new
in ruflo (Cognitum funnel ADR-301..319), AQE's current integration state, the
gap analysis / top-10 opportunities, the QE-Court addendum (= ADR-124 /
/v1/qe/verdict from the user-facing end), and the tiered plan with per-item
approval gates (incl. Codex-as-prosecutor and Codex-as-provider M3.5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QE-Court convenes independent, cross-vendor AI reviewers against a delivery
(diff / PR / test-suite), makes them file charges from their own probe sets,
then forces any SHIP verdict to SURVIVE an escalating deeper review before it
stands (loop-until-dry overturn). Verdict is 3-valued — SHIP / REMAND / BLOCK —
with a human as final judge. It is the reference implementation of ADR-124; the
hosted sibling is the planned Cognitum /v1/qe/verdict endpoint.
The skill is an orchestration layer that COMPOSES existing assets — it does not
reimplement critics:
- Prosecutors: qe-devils-advocate, brutal-honesty-review, sherlock-review,
qe-security-scanner, qe-mutation-tester, and `codex exec review` (cross-vendor
GPT lens — the strongest form of writer≠juror).
- Kill round: src/verification/adversarial-verify. Jury: two-gate judge (ADR-119).
- Evidence: provenance tiers (ADR-121), signed receipt (ADR-118).
Configurable per-role model routing (config.json `routing`) across Claude /
Cognitum tiers / OpenRouter / Codex / local, all through the ADR-123 provider
layer (budget caps + receipts). Invariants: >=2 distinct vendors, jury vendor
!= writer, DoE-gated scoring (no un-validated "91/100").
Self-learning (ADR-124 M0.B/C): signed court records feed the flywheel; reproduced
charges become oracle-tier qe_patterns; an overturned SHIP is persisted as a
discriminator training pair for the frozen anchor + two-gate judge.
Ships via `aqe init` (added to V3_DOMAIN_SKILLS; auto-installed, not excluded)
and the assets/ npm mirror. Includes a runnable acceptance eval + a seeded-mutant
fixture (a boundary off-by-one a shallow reviewer rates SHIP; the overturn round
must flip it, and MUST regress to SHIP at overturnDepth=0). trust_tier 1 until the
eval runs green (>=5 oracle cases) → then tier 3.
Skill count: 75 -> 76 user-facing (totalQESkills 81 -> 82). Parity clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a `codex` provider that runs QE inference on the user's ChatGPT
subscription by shelling out to `codex exec`, mirroring the claude-code
subscription provider (ADR-123). Gives AQE a cross-vendor (GPT-family)
engine for provider fallback and — the motivating use — an independent
prosecutor/juror in qe-court (ADR-124), where a non-Claude, non-Cognitum
brain is the strongest form of the writer≠juror invariant.
- src/shared/llm/providers/codex.ts: subscription provider. Captures the
final message via `codex exec -o <file>` (read-only sandbox), estimates
token usage, cost $0 (subscription). LOAD-BEARING: strips OPENAI_API_KEY /
CODEX_API_KEY from the child env so it cannot silently revert to API billing.
- interfaces.ts: 'codex' added to LLMProviderType; CodexConfig; providers map.
- provider-manager: case 'codex'. billing-modes: codex => 'subscription' +
ChatGPT-aware startup notice. config-store: no env key (subscription) +
runtime-constructible. Totality-required Record maps updated (cost-tracker,
llm-router default model, message-formatter strategy).
- tests: env-strip parity + metadata (7 tests).
Verified: 209 provider/router unit tests pass; tsc clean; full build green;
a faithful inline reproduction of the provider's spawn hit the real `codex`
binary → "OK", exit 0, ~5.5s (gpt-5.5 via ChatGPT subscription).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consumer dependency audit flagged 4 HIGH findings, all tracing to one
real CVE — adm-zip@0.5.17 (GHSA-xcpc-8h2w-3j85, crafted ZIP → 4GB alloc) —
pulled in transitively:
agentic-qe → @huggingface/transformers@4.2.0 → onnxruntime-node@1.24.3 → adm-zip@0.5.17
@huggingface/transformers@4.2.0 is the latest release (no patched version)
and pins the vulnerable onnxruntime-node, so it cannot be overridden away for
consumers (root overrides don't reach downstream installs).
Since transformers is only needed for in-process/local embeddings — consumers
using an embedding endpoint (e.g. Cognitum /v1/embeddings) never load it — move
it out of the default consumer install:
- dependencies → optional peerDependency (peerDependenciesMeta.optional),
mirroring how @claude-flow/browser is handled. optionalDependencies is NOT
sufficient: npm installs those by default (that's why the @ruvector native
binaries live there); only an optional peer keeps the package — and its
onnxruntime-node → adm-zip chain — out of a plain `npm install agentic-qe`.
- kept in devDependencies so our own build/typecheck/tests still resolve it.
- EmbeddingGenerator.ts: the one static value-import of `pipeline` is now a
lazy dynamic import inside initialize(), with an actionable error when the
optional package is absent. The `Tensor` import is `import type` (erased).
Verified:
- consumer-POV audit (scripts/dependency-audit.mjs --consumer): PASS, 0 high/critical (was 4 high)
- npm run build: green
- embeddings tests (endpoint + local backend): 28/28 pass
- lockfile regenerated --package-lock-only; 14 @ruvector optional platform entries preserved
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two "no provider available" tests save and clear provider keys to be
hermetic, but the list went stale when ADR-123 (v3.12.2) added the cognitum and
claude-code providers. Any developer with COGNITUM_API_KEY exported got a
HybridRouter where the test expected undefined — failing locally while passing
in CI, where none of these are set.
Adds COGNITUM_API_KEY, CLAUDE_API_KEY, and ANTHROPIC_AUTH_TOKEN to both
clear-lists. Verified: 17/17 pass with COGNITUM_API_KEY still exported.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An export killed mid-write (hook timeout, session teardown, SIGKILL) left a
store created but never completed. It could neither be opened (0x0106
ManifestNotFound) nor created over (0x0303 FsyncFailed, path exists), so the
RVF backend stayed silently disabled for the life of the project.
Export is now atomic: build at `<path>.tmp.<pid>`, close, then rename over the
final path. An interrupted export leaves the previous good store untouched.
Stale tmps are swept, skipping any whose owning pid is still alive.
Recovery is wired into all three open paths (dual-writer/brain.rvf, shared
adapter/patterns.rvf, pattern-store ladder): when open AND create both fail the
file is provably unusable, so it is quarantined aside — not deleted — and the
derived cache is rebuilt.
Recovery is deliberately NOT keyed on the signature reported in the issue.
Measured against @ruvector/rvf-node 0.1.8: `FLVR` is the lock record's own
magic (the store's is `SFVR`), so a 104-byte FLVR lock is an ordinary stale
lock, not leaked store bytes; and a valid empty store is exactly 162 bytes,
the size reported as "truncated". Keying on either would quarantine healthy
stores out from under live processes. The lock record does carry its owner's
pid (u32 LE at offset 4), which now guards recovery and also fixes a
pre-existing hazard: stale-lock cleanup previously unlinked locks
unconditionally, breaking live peers.
Also converts three require() calls of source modules to static imports. They
resolve in the esbuild bundle but throw under the test runner, so tests were
silently exercising a different code path than production.
Tests: real SIGKILL mid-export reproduction (previously truncated the store to
162 bytes + stale lock, matching the report), fault-injected export failure,
and recovery through each open path. All fail without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flywheel signed evolve receipts with a self-generated key, so its lineage
could not be attributed to the platform identity that qe-harness evidence bundles
chain to. This wires the bridge:
- platform-signer.ts: platformSigner() derives the deterministic Ed25519 seed from
QE_WITNESS_SIGNING_KEY (JSON {publicKey,privateKey}, the Secret Manager shape
qe-harness Stage-B reads) and hands it to createSigner — reproducing the exact
platform keypair. Graceful: null when the env is absent/malformed (keeps the
self-generated-key behaviour). Env-reading stays out of the receipt.ts core.
- receipt.ts keyIdOf now trims the PEM, matching the platform witness fingerprint
scheme (sha256(pem.trim()) truncated to 16 hex — qe-harness witness.ts /
meta-llm qe-witness.ts). So a platform-signed receipt's keyId equals the
allowlist fingerprint.
Verified against the REAL QE_WITNESS_SIGNING_KEY (no key material logged): the
derived signer's keyId == f1ac28607da49ec1, it is in the real
QE_WITNESS_PUBLIC_KEYS_JSON allowlist, and a signed receipt verifies (tamper
rejected). Tests: +5; existing flywheel suite (34) green; tsc clean.
Refs cognitum-one/qe-harness#7.
Co-Authored-By: Ruflo & AQE
Closes the structural gap from issue #557: AQE could only bill the paid
Anthropic API even for Pro/Max subscribers, and its budget caps were never
enforced.
- claude-code provider: run QE via `claude -p` on the user's subscription;
child env strips API-billing keys so it can't revert to API billing
- Enforced cross-process spend caps (--max-budget-usd / AQE_MAX_BUDGET_USD),
persisted to memory.db (llm_spend); enforced on the ProviderManager AND
HybridRouter paths; over-budget aborts with COST_LIMIT_EXCEEDED
- cognitum provider: OpenAI/Anthropic-compatible gateway with per-request
cost receipts, embeddings, and a server-side hard cap
- Billing-mode taxonomy + startup notice + `aqe health` LLM Billing section
- config-store: honor explicit `enabled: false` over env key presence;
AQE_LLM_PROVIDER selector
- Fix: `aqe eval run` no longer scores fabricated mock responses — real
provider by default, hard-error when none configured, `--mock` opt-in
- Consensus verification path prints a billing warning (deferred full port)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two flaky tests depended on shared/real SQLite state, causing failures only
in the full parallel suite run:
- metrics-tracker.test.ts opened the REAL project .agentic-qe/memory.db via
createLearningMetricsTracker(process.cwd()). A live-DB read threw "database
is locked" under the bind-mounted/parallel runner (and risked corrupting the
project's learning data). Now uses an isolated temp projectRoot with an
empty-but-valid memory.db, exercising the real getDashboardData* code paths
(the tracker probes sqlite_master, so an empty DB returns zeroed metrics).
Adds an explicit "Database not found" test against a missing temp DB.
- sona-persistence.test.ts used a fixed /tmp path, so persisted kv_store state
(the request counter) leaked between tests and flaked the "starts at 0"
assertion. Each test now gets a unique mkdtemp dir, cleaned up in afterEach.
Full unit suite: 17639 passed, 0 failed (previously 5 failed / 2 files).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dragan Spiridonov