A self-contained harness that measures, per quality-engineering task, the
cheapest tier that clears the quality bar against api.cognitum.one — so tier
routing (e.g. the qe-court model table) is set by evidence, not guessed.
Structure (scripts/cognitum-qe-bench/):
- 7 QE tasks spanning the oracle-provenance ladder: test-generation &
mutation-adequacy (execution), security-triage / pr-severity / flaky-diagnosis
/ adversarial-review (labeled), coverage-gap (structural).
- test-generation is graded by MUTATION SCORE over a diverse mutant set with
per-test validity, truncation-repair, and emission-vs-quality separation —
operators grounded in .claude/skills/mutation-testing/references.
- Three policies per task: P1 always-cheap, P2 always-frontier, P3 cheap+escalate
(high tier only as an advisor on low cheap-confidence).
- Cost is ground truth from each x_cognitum.price_usd receipt; each row carries
resolvedModel + requestId (reconcilable 1:1 against usage_ledger).
- Keys come from local config (env / .env), never code; two-key split (worker
low/mid, judge high) is enforced by scope. mint-bench-keys.mjs is a separate
one-time owner tool (safe two-part write: metered subscription + scope-split
key). README documents setup for team members.
- Safety: dry-run default, --max-cost pre-flight abort, mid-run cost stop,
AQE_MAX_BUDGET_USD backstop. Emits ruflo-compatible cognitum-qe.v1 records.
First measured finding: on these QE tasks the cheap tier (glm-5.2) matches or
beats the frontier tier at 3-4x less cost; on test-generation the frontier tier
failed to emit runnable code (emit 0%) while cheap scored a full mutation score.
Co-Authored-By: Ruflo & AQE
The user-facing intro's "Selected QE skills available" table omitted qe-court and
still read 75. Adds /qe-court to the Review category and updates the count to 76.
AQE-QE-scoped only (excludes ruflo/platform/dev skills); the on-disk "115 skill
definitions" line is already correct (qe-court made it 114→115).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A qe-court adversarial review (cross-vendor Codex GPT + AQE devils-advocate +
security prosecutors, filing blind) BLOCKED the release by catching edge cases the
green unit suite missed. All confirmed charges fixed:
FATAL:
- F1 codex.ts / claude-code.ts: no 'error' listener on child.stdin — an early
child exit (not-logged-in path) raised an uncaught EPIPE that crashed the host.
Added a stdin error guard + try/catch (both providers — same template bug).
- F2 db-recovery.ts: non-atomic copyFileSync + no lock let concurrent hook
processes tear the restored memory.db. Now: copy to a temp on the same fs →
atomic rename, under a best-effort exclusive lock (skip if a peer is restoring).
- F3 db-recovery.ts: isDbHealthy opened readonly + integrity_check, which
FALSE-NEGATIVES a HEALTHY WAL db when -shm is zeroed over virtiofs — the exact
target failure mode — and would have parked the good DB + reverted to an older
backup (silent data loss). Now the live-corruption signal is the SQLite HEADER:
a -wal/-shm fault never corrupts the main file, so a header-valid DB is treated
healthy and NEVER parked. Backups still get a full integrity_check before use.
SIGNIFICANT:
- spend-ledger.ts: a swallowed ALTER migration made every INSERT hit a missing
column, throw, get caught, and silently drop ALL spend → budget-cap bypass.
record() now tracks whether the column exists and INSERTs accordingly (spend is
always recorded; attribution degrades gracefully).
- witness-verify.ts: verifyAqeReceipt was fail-OPEN under an empty allowlist (the
default state) — a self-signed forgery returned valid:true. Now fail-closed
('no-trust-root') unless an explicit allowSelfSigned opt-in; harness path asserts
alg==='ed25519'; unparseable allowlist JSON now logs loudly.
- gate-reexecute.ts accept/v1+sig: enforce a minimum paired-sample size (n>=8 —
below it a "95% CI" is just the raw delta) and assert mean(samples)==sealed
scalar (defends against a forged scalar/vector pairing).
Verified: qe-court acceptance eval 100%; 1498 shared/validation/llm/flywheel tests
pass (incl. new F2/F3 regressions, fail-closed, alg, min-n, consistency cases);
tsc clean; build green. Deferred (non-blocking rough edges, noted in the court
record): codex maxTokens/temperature passthrough, healthCheck-while-logged-out,
semaphore microtask race, stableStringify byte-parity (needs a real qe-harness sample).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes a real gap: AQE's frozen accept/v1 promotes when the candidate's held-out
MEAN merely beats baseline by ANY amount (gate-reexecute.ts:72) — i.e. it can
promote a within-noise gain. ruflo's flywheel already gates on significance
(accept/v1+sig, paired bootstrap CI > 0); this is the symmetric ADOPTION of that
technique (reimplemented dependency-free — AQE does not import ruflo).
- src/validation/paired-bootstrap.ts: deterministic (fixed-seed LCG) one-sided
95% lower bound on the mean of PAIRED per-task deltas. Determinism is required —
the promotion is replay-verified (ADR-120), so the bound must reproduce exactly.
- gate-reexecute.ts: new frozen rule `accept/v1+sig` = accept/v1 AND
bootstrapDeltaCILow(paired held-out deltas) > 0. accept/v1 is UNTOUCHED (added a
new registry key the ADR-120 way). Fail-closed: rejects if paired samples are
absent/mismatched. SealedInputs gains optional per-task sample vectors (omitted
from the hash when undefined → accept/v1 receipts byte-identical).
- corpus-scorer.ts: scoreHeldOut now returns the per-query reciprocal-rank vector
(the paired sample) alongside its mean → PolicyScores.heldOutSamples.
- generation.ts: seals the samples ONLY when ruleVersion === 'accept/v1+sig'
(opt-in); default accept/v1 behavior + receipt bytes unchanged.
Why paired-bootstrap not ANVOA: for 2 groups ANOVA F = t² and unpaired; the
paired CI exploits the task-pairing and is strictly stronger. (AQE's ADR-122
ANOVA stays the right tool for offline multi-factor screening — complementary.)
Verified: new accept/v1+sig tests incl. the marquee gap case (a within-noise gain
accept/v1 PROMOTES but accept/v1+sig REJECTS); 321 flywheel+validation tests pass;
tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Source-verified in /workspaces/ruflo: the flywheel already refuses to promote a
candidate whose paired candidate-vs-baseline delta isn't significant —
`significant = bootstrapDeltaCILow(paired holdout deltas) > 0`, rule accept/v1+sig,
one-sided 95% paired bootstrap, n>=20 (evolve-proof.ts:174-183). That is STRONGER
than the unpaired ANOVA we were going to contribute (2-group ANOVA F = t², and it
drops the pairing). Our premise ("maybe ruflo promotes noise") was wrong.
Outcome: NO contribution PR (would be redundant and weaker). Corrects the strategy
report's open-Q #3 (ruflo's flywheel IS rigorously DoE-validated) and retires the
M3.1 action. The draft is kept, marked superseded, documenting the one genuinely-
additive idea (an ANOVA candidate-SELECTION veto) and why it's a bigger, replay-
verifier-touching change — deferred, not pursued.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the transferable half of ruflo v3.25.2 (backup auto-restore), adapted to
better-sqlite3. Previously a malformed memory.db surfaced errors and told the
operator to "restore manually" (unified-memory.ts:365); now the corrupt case
self-heals like the already-handled missing-file case.
- src/shared/db-recovery.ts: isDbHealthy() (integrity_check) + attemptAutoRestore()
— on a malformed DB, park the corrupt original as <db>.corrupt-<ts> (NEVER delete),
park stale -wal/-shm, and copy in the NEWEST backup that itself passes
integrity_check from .agentic-qe/backups/verified/. Non-destructive + no-op when
healthy or when no verified backup exists (left for manual inspection).
- safe-db.ts: opt-in `autoRestore` option (writable opens only), runs before open.
- unified-memory.ts: enables autoRestore on the memory.db open (missing-file case
was already covered; this adds the exists-but-malformed case).
Data-protection: corrupt file is renamed not deleted; only integrity-verified
backups are used; matches scripts/aqe-db-backup.sh's verified-backup layout.
Verified: db-recovery 7/7 (restore newest, skip corrupt backup, park stale WAL,
non-destructive when no backup); 1171 shared tests pass; tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M3.1: a ready-to-post, collegial issue draft offering AQE's DoE/ANOVA screen
(ADR-122) to validate ruflo's ADR-176 flywheel promotions — the "2 compounding
promotions / flat human anchor" shape matches AQE's own "beads" null. HELD from
posting to ruvnet/ruflo pending the user (a ruflo collaborator).
M3.2: assessment — ruflo v3.25.2's torn-full-image-flush fix does not apply to
AQE (better-sqlite3 WAL = atomic commits, not sql.js full-image flushes); AQE's
actual corruption class (virtiofs WAL/shm) is already mitigated in safe-db.ts.
Transferable lesson filed: auto-restore-on-malformed-open (AQE restores manually
today) — a scoped follow-up, not built here (memory.db recovery is data-sensitive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md references ruflo/claude-flow MCP tools heavily, which can read as a
runtime dependency. It isn't: the shipped agentic-qe package imports only
@ruvector/* primitives + two optional @claude-flow peers, never the ruflo npm
package. Adds a CONTRIBUTING note so contributors don't wire product code to a
ruflo CLI/MCP call or add it as a runtime dep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(A) npm "@ruvector Lattice WASM" = vaporware (E404; claude-flow's own source admits
the default package doesn't exist; inert opt-in slot) — the ruvllm mock-mode pattern.
(B) github.com/ohdearquant/lattice = a REAL pure-Rust inference engine (runs on
arm64 here, pushed 2026-07-18). But the shipped binary is chat-only — serve exposes
/v1/chat/completions,/v1/models,/health, NO /v1/embeddings (source
crates/inference/src/serve/mod.rs:248). Embeddings are a separate lattice-embed
crate, source-build-only, CLI not HTTP. Models incl. BGE-small 384 (default) …
Qwen3-Embedding 1024.
Verdict: GO-with-caveats, NOT a drop-in and not the cheapest path — the near-term
win (BGE-small @384, MTEB-stronger than MiniLM, same dim) is reachable via AQE's
existing transformers.js path (Xenova/bge-small-en-v1.5), no Rust/shim. Keep lattice
for a bigger move (Qwen3-1024, SIMD perf, Cognitum-side embedding serving).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the single AQE-side verifier that reconciles AQE flywheel receipts with
Cognitum qe-harness witness bundles, so QE evidence signed by the shared platform
identity (fingerprint f1ac28607da49ec1) is verifiable on either side.
Investigation (read-only, cognitum-one/qe-harness) established the bridge is
HALF-wired already: both sides use Ed25519, the same QE_WITNESS_SIGNING_KEY (JSON
PEM), and the IDENTICAL fingerprint scheme sha256(pem.trim())[:16] — both sign as
f1ac28607da49ec1. What was NOT aligned: the signed artifacts (AQE = fixed-order
JSON body, hex; qe-harness = stableStringify sha256: digest, base64, nested
witness{}) are not natively cross-verifiable.
- src/learning/qe-flywheel/witness-verify.ts: allowlist-driven verifier for BOTH
formats + stableStringify/canonicalDigest (qe-harness parity), tamper-detecting,
fail-closed allowlist. node:crypto only; reuses receipt.ts for the AQE path.
- 15 tests: fingerprint parity (keyFingerprint === signer.keyId), both formats
round-trip, tamper → bad-signature/digest-mismatch, allowlist enforcement.
Two external dependencies remain (need the qe-harness owner — documented in the
GOAP plan, NOT done unilaterally): provision the f1ac28607da49ec1 PUBLIC key via
QE_WITNESS_PUBLIC_KEYS_JSON; confirm byte-parity vs a real qe-harness sample;
decide canonicalization convergence (AQE's hand-listed body silently drops new
fields — recommend adopting stableStringify).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the ADR-123 cross-process spend ledger to attribute each LLM charge to
the QE agent type that incurred it — the seed for a Cognitum per-agent billing
dashboard across the 60+ agent fleet.
- spend-ledger.ts: SpendEntry.agent; additive `agent_type` column via a guarded
ALTER TABLE ADD COLUMN (non-destructive — existing rows get NULL, never drops
or rewrites memory.db); new spentByAgentSince() grouping (unattributed charges
bucket under '(unattributed)'); implemented on both Sqlite + InMemory ledgers.
- provider-manager.ts: recordSpend/recordResponseSpend take an optional agent.
- hybrid-router.ts: passes params.agentType at the record site (already in scope).
Verified: spend-ledger tests 9/9 (incl. an old-schema DB migrating without losing
its legacy row); full LLM suite 780/780; tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hosted qe-court: POST a delivery (diff/patch/files), get a signed,
overturn-tested verdict. Grounded in the M1 live probe — a NEW route on the
existing api.cognitum.one inference service (X-API-Key, x_cognitum receipts),
NOT the ADR-308 funnel (which 404s on that host); /v1/qe/* confirmed free.
Verdict semantics reference the OSS referee (src/skills/qe-court/referee.ts) so
client and server agree: 3-valued verdict, the overturn mechanic (overturnDepth),
writer≠juror + vendor-diversity (422 PANEL_INVARIANT_VIOLATION), DoE-gated score,
Ed25519-signed record (ADR-118), provenance-tiered charges (ADR-121). Sync + async
job models; budget/cap semantics mirror /v1/chat/completions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M1 resolved: api.cognitum.one's inference API and the ADR-308 funnel/proxy API
are NOT the same backend. /v1/proxy/*, /v1/credits, /v1/events, /v1/funnel-policy
all 404 ROUTE_NOT_FOUND on the inference host; the funnel layer is a separate
service (confirms ADR-308's addendum). /v1/qe/* namespace is free (404) — build
ADR-124 Phase 2 as a new route on the existing inference Express app. Hard cap
confirmed: hardCapUsd $20, headroom $19.34. Tiers route to a model marketplace
(cognitum-low -> z-ai/glm-5.2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turns qe-court's falsifiable claims into enforced code and earns trust_tier 3 on
real, executed evidence (no self-report):
- src/skills/qe-court/referee.ts: the court's rules as pure functions —
resolveVerdict() with the overturn mechanic (a SHIP must SURVIVE escalation;
a mutant surfaced only at depth>=1 does NOT count at overturnDepth=0, so it
regresses to SHIP — the false-SHIP the court exists to catch), validatePanel()
(writer!=juror, >=2 distinct vendors), shouldEmitScore() (DoE-gated, no
un-validated number).
- tests/unit/skills/qe-court/referee.test.ts: 7 oracle cases — each passes on
correct logic and FAILS on a seeded regression.
- evals/qe-court.yaml: converted to a command-eval suite (6 cases) that runs the
oracles through the `aqe eval` CLI.
- SKILL.md: trust_tier 1 -> 3, validation status passing (6/6, 2026-07-18).
- skills-manifest (both trees): Tier-3 count 49 -> 50, additional 33 -> 32.
Verified: `aqe eval run --skill qe-court` -> PASSED 6/6 100%; referee suite 7/7;
skill parity clean; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six Thinking Hats strategy report and the precondition-ordered GOAP execution
plan for the AQE <-> ruflo <-> Cognitum integration work. Captures: what's new
in ruflo (Cognitum funnel ADR-301..319), AQE's current integration state, the
gap analysis / top-10 opportunities, the QE-Court addendum (= ADR-124 /
/v1/qe/verdict from the user-facing end), and the tiered plan with per-item
approval gates (incl. Codex-as-prosecutor and Codex-as-provider M3.5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QE-Court convenes independent, cross-vendor AI reviewers against a delivery
(diff / PR / test-suite), makes them file charges from their own probe sets,
then forces any SHIP verdict to SURVIVE an escalating deeper review before it
stands (loop-until-dry overturn). Verdict is 3-valued — SHIP / REMAND / BLOCK —
with a human as final judge. It is the reference implementation of ADR-124; the
hosted sibling is the planned Cognitum /v1/qe/verdict endpoint.
The skill is an orchestration layer that COMPOSES existing assets — it does not
reimplement critics:
- Prosecutors: qe-devils-advocate, brutal-honesty-review, sherlock-review,
qe-security-scanner, qe-mutation-tester, and `codex exec review` (cross-vendor
GPT lens — the strongest form of writer≠juror).
- Kill round: src/verification/adversarial-verify. Jury: two-gate judge (ADR-119).
- Evidence: provenance tiers (ADR-121), signed receipt (ADR-118).
Configurable per-role model routing (config.json `routing`) across Claude /
Cognitum tiers / OpenRouter / Codex / local, all through the ADR-123 provider
layer (budget caps + receipts). Invariants: >=2 distinct vendors, jury vendor
!= writer, DoE-gated scoring (no un-validated "91/100").
Self-learning (ADR-124 M0.B/C): signed court records feed the flywheel; reproduced
charges become oracle-tier qe_patterns; an overturned SHIP is persisted as a
discriminator training pair for the frozen anchor + two-gate judge.
Ships via `aqe init` (added to V3_DOMAIN_SKILLS; auto-installed, not excluded)
and the assets/ npm mirror. Includes a runnable acceptance eval + a seeded-mutant
fixture (a boundary off-by-one a shallow reviewer rates SHIP; the overturn round
must flip it, and MUST regress to SHIP at overturnDepth=0). trust_tier 1 until the
eval runs green (>=5 oracle cases) → then tier 3.
Skill count: 75 -> 76 user-facing (totalQESkills 81 -> 82). Parity clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a `codex` provider that runs QE inference on the user's ChatGPT
subscription by shelling out to `codex exec`, mirroring the claude-code
subscription provider (ADR-123). Gives AQE a cross-vendor (GPT-family)
engine for provider fallback and — the motivating use — an independent
prosecutor/juror in qe-court (ADR-124), where a non-Claude, non-Cognitum
brain is the strongest form of the writer≠juror invariant.
- src/shared/llm/providers/codex.ts: subscription provider. Captures the
final message via `codex exec -o <file>` (read-only sandbox), estimates
token usage, cost $0 (subscription). LOAD-BEARING: strips OPENAI_API_KEY /
CODEX_API_KEY from the child env so it cannot silently revert to API billing.
- interfaces.ts: 'codex' added to LLMProviderType; CodexConfig; providers map.
- provider-manager: case 'codex'. billing-modes: codex => 'subscription' +
ChatGPT-aware startup notice. config-store: no env key (subscription) +
runtime-constructible. Totality-required Record maps updated (cost-tracker,
llm-router default model, message-formatter strategy).
- tests: env-strip parity + metadata (7 tests).
Verified: 209 provider/router unit tests pass; tsc clean; full build green;
a faithful inline reproduction of the provider's spawn hit the real `codex`
binary → "OK", exit 0, ~5.5s (gpt-5.5 via ChatGPT subscription).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consumer dependency audit flagged 4 HIGH findings, all tracing to one
real CVE — adm-zip@0.5.17 (GHSA-xcpc-8h2w-3j85, crafted ZIP → 4GB alloc) —
pulled in transitively:
agentic-qe → @huggingface/transformers@4.2.0 → onnxruntime-node@1.24.3 → adm-zip@0.5.17
@huggingface/transformers@4.2.0 is the latest release (no patched version)
and pins the vulnerable onnxruntime-node, so it cannot be overridden away for
consumers (root overrides don't reach downstream installs).
Since transformers is only needed for in-process/local embeddings — consumers
using an embedding endpoint (e.g. Cognitum /v1/embeddings) never load it — move
it out of the default consumer install:
- dependencies → optional peerDependency (peerDependenciesMeta.optional),
mirroring how @claude-flow/browser is handled. optionalDependencies is NOT
sufficient: npm installs those by default (that's why the @ruvector native
binaries live there); only an optional peer keeps the package — and its
onnxruntime-node → adm-zip chain — out of a plain `npm install agentic-qe`.
- kept in devDependencies so our own build/typecheck/tests still resolve it.
- EmbeddingGenerator.ts: the one static value-import of `pipeline` is now a
lazy dynamic import inside initialize(), with an actionable error when the
optional package is absent. The `Tensor` import is `import type` (erased).
Verified:
- consumer-POV audit (scripts/dependency-audit.mjs --consumer): PASS, 0 high/critical (was 4 high)
- npm run build: green
- embeddings tests (endpoint + local backend): 28/28 pass
- lockfile regenerated --package-lock-only; 14 @ruvector optional platform entries preserved
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two "no provider available" tests save and clear provider keys to be
hermetic, but the list went stale when ADR-123 (v3.12.2) added the cognitum and
claude-code providers. Any developer with COGNITUM_API_KEY exported got a
HybridRouter where the test expected undefined — failing locally while passing
in CI, where none of these are set.
Adds COGNITUM_API_KEY, CLAUDE_API_KEY, and ANTHROPIC_AUTH_TOKEN to both
clear-lists. Verified: 17/17 pass with COGNITUM_API_KEY still exported.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An export killed mid-write (hook timeout, session teardown, SIGKILL) left a
store created but never completed. It could neither be opened (0x0106
ManifestNotFound) nor created over (0x0303 FsyncFailed, path exists), so the
RVF backend stayed silently disabled for the life of the project.
Export is now atomic: build at `<path>.tmp.<pid>`, close, then rename over the
final path. An interrupted export leaves the previous good store untouched.
Stale tmps are swept, skipping any whose owning pid is still alive.
Recovery is wired into all three open paths (dual-writer/brain.rvf, shared
adapter/patterns.rvf, pattern-store ladder): when open AND create both fail the
file is provably unusable, so it is quarantined aside — not deleted — and the
derived cache is rebuilt.
Recovery is deliberately NOT keyed on the signature reported in the issue.
Measured against @ruvector/rvf-node 0.1.8: `FLVR` is the lock record's own
magic (the store's is `SFVR`), so a 104-byte FLVR lock is an ordinary stale
lock, not leaked store bytes; and a valid empty store is exactly 162 bytes,
the size reported as "truncated". Keying on either would quarantine healthy
stores out from under live processes. The lock record does carry its owner's
pid (u32 LE at offset 4), which now guards recovery and also fixes a
pre-existing hazard: stale-lock cleanup previously unlinked locks
unconditionally, breaking live peers.
Also converts three require() calls of source modules to static imports. They
resolve in the esbuild bundle but throw under the test runner, so tests were
silently exercising a different code path than production.
Tests: real SIGKILL mid-export reproduction (previously truncated the store to
162 bytes + stale lock, matching the report), fault-injected export failure,
and recovery through each open path. All fail without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flywheel signed evolve receipts with a self-generated key, so its lineage
could not be attributed to the platform identity that qe-harness evidence bundles
chain to. This wires the bridge:
- platform-signer.ts: platformSigner() derives the deterministic Ed25519 seed from
QE_WITNESS_SIGNING_KEY (JSON {publicKey,privateKey}, the Secret Manager shape
qe-harness Stage-B reads) and hands it to createSigner — reproducing the exact
platform keypair. Graceful: null when the env is absent/malformed (keeps the
self-generated-key behaviour). Env-reading stays out of the receipt.ts core.
- receipt.ts keyIdOf now trims the PEM, matching the platform witness fingerprint
scheme (sha256(pem.trim()) truncated to 16 hex — qe-harness witness.ts /
meta-llm qe-witness.ts). So a platform-signed receipt's keyId equals the
allowlist fingerprint.
Verified against the REAL QE_WITNESS_SIGNING_KEY (no key material logged): the
derived signer's keyId == f1ac28607da49ec1, it is in the real
QE_WITNESS_PUBLIC_KEYS_JSON allowlist, and a signed receipt verifies (tamper
rejected). Tests: +5; existing flywheel suite (34) green; tsc clean.
Refs cognitum-one/qe-harness#7.
Co-Authored-By: Ruflo & AQE
Closes the structural gap from issue #557: AQE could only bill the paid
Anthropic API even for Pro/Max subscribers, and its budget caps were never
enforced.
- claude-code provider: run QE via `claude -p` on the user's subscription;
child env strips API-billing keys so it can't revert to API billing
- Enforced cross-process spend caps (--max-budget-usd / AQE_MAX_BUDGET_USD),
persisted to memory.db (llm_spend); enforced on the ProviderManager AND
HybridRouter paths; over-budget aborts with COST_LIMIT_EXCEEDED
- cognitum provider: OpenAI/Anthropic-compatible gateway with per-request
cost receipts, embeddings, and a server-side hard cap
- Billing-mode taxonomy + startup notice + `aqe health` LLM Billing section
- config-store: honor explicit `enabled: false` over env key presence;
AQE_LLM_PROVIDER selector
- Fix: `aqe eval run` no longer scores fabricated mock responses — real
provider by default, hard-error when none configured, `--mock` opt-in
- Consensus verification path prints a billing warning (deferred full port)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two flaky tests depended on shared/real SQLite state, causing failures only
in the full parallel suite run:
- metrics-tracker.test.ts opened the REAL project .agentic-qe/memory.db via
createLearningMetricsTracker(process.cwd()). A live-DB read threw "database
is locked" under the bind-mounted/parallel runner (and risked corrupting the
project's learning data). Now uses an isolated temp projectRoot with an
empty-but-valid memory.db, exercising the real getDashboardData* code paths
(the tracker probes sqlite_master, so an empty DB returns zeroed metrics).
Adds an explicit "Database not found" test against a missing temp DB.
- sona-persistence.test.ts used a fixed /tmp path, so persisted kv_store state
(the request counter) leaked between tests and flaked the "starts at 0"
assertion. Each test now gets a unique mkdtemp dir, cleaned up in afterEach.
Full unit suite: 17639 passed, 0 failed (previously 5 failed / 2 files).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Non-destructive `aqe init` for `.claude/settings.json`: merge instead of
overwrite, preserve user-owned config (custom statusLine, explicit
includeCoAuthoredBy, AQE_* env overrides, other tools' hooks such as ruflo /
Claude Flow), deep-merge v3 sections, and back up the pristine original before
writing (mirroring CLAUDE.md.backup).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to 961f3feb. The structural rewrite removed the regex sink (the 9
incomplete-hostname-regexp alerts cleared) but introduced one new alert:
`allowed.replace('*', MARKER)` replaces only the FIRST '*' (CodeQL
js/incomplete-sanitization, "Incomplete string escaping or encoding",
routes.ts:709).
Use the global form `allowed.replace(/\*/g, MARKER)` so every '*' is substituted.
A multi-wildcard pattern is still rejected safely: its suffix then contains the
placeholder label, which no real host can end with. Adds a regression test for a
multi-star pattern.
All 10 validateRedirectUri cases pass; tsc 0 errors; build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to 2d63c88c. Bounding the wildcard to `[^./]+` fixed the open-redirect
bypass, but CodeQL's js/incomplete-hostname-regexp still flagged every hostname
allow-list literal that flows into `new RegExp` (it does not model the custom
escaping), so the alert count grew to 9 as the new regression tests added more
`*.example.com` literals.
Remove the RegExp sink entirely: validateRedirectUri now compares parsed URL
components. For a wildcard entry, only the single leftmost subdomain label may be
wildcarded — it must be a non-empty, dot-free label — and protocol/port must match
with an exact-or-prefix path match. This is strictly more secure (a host can never
be spoofed via extra dots or path segments) and eliminates the CodeQL finding at
its root (no URL string is ever compiled into a regex).
All 9 validateRedirectUri cases pass unchanged (6 original + 3 bypass regressions);
tsc 0 errors; build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
validateRedirectUri expanded an allow-list wildcard with `.replace(/\*/g, '.*')`,
so `https://*.example.com/cb` compiled to `^https://.*\.example\.com/cb$`. Because
`.*` also matches `.` and `/`, an attacker-controlled host could satisfy the
pattern by carrying the allowed suffix in the URL PATH — e.g.
`https://evil.com/x.example.com/cb` validated as allowed → open redirect. This is
the CodeQL js/incomplete-hostname-regexp finding (6 alerts, #214–219) surfaced by
the new a2a auth route tests.
Fix: expand the wildcard to `[^./]+` so it binds to a single subdomain label and
cannot span dots or slashes. Exact, same-origin, path-prefix, and single-label
wildcard matches are preserved; only the previously-exploitable path-injection
spoof and multi-level-wildcard span are now rejected (standard wildcard-cert
semantics). Adds regression tests for the path-injection, multi-label, and
embedded-slash bypasses.
Scope: grepped all `.replace(/\*/g, '.*')` sites — the others are glob→regex
matchers for memory keys / event names / worker patterns (not hostname
validation) where `.*` is intended; only the auth redirect validator is affected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes and supersedes the 2026-07-09 directional DoE (which was 25/27 cells, had
no F-test, and — the reproducibility gap found this session — never persisted its raw
data). Re-run full under the CRIT-2 design (N=5 stochastic replicates of fixed item
A1-inRange, real within-cell error term), qwen local $0 + anthropic/claude-haiku-4.5
+ anthropic/claude-opus-4.8 via OpenRouter. $1.538 total, well under the $11 budget.
Results (real main-effects ANOVA, 135 observations, first real data the aggregator
has ever consumed — previously stub-only):
- scaffold: none 1.000 >> plan-and-solve 0.311 >> reflexion 0.067
F(2,108)=218.9, p=1.0e-38, 71.9% of variance — DOMINANT and strongly NEGATIVE.
Both scaffolds DROP. retort's "more harness lowers reliability", now overwhelming.
- model: qwen3-coder:30b 0.644 > haiku-4.5 0.400 > opus-4.8 0.333
F(2,108)=25.1, p=1.1e-9 — local qwen best & free; opus 5x+ cost = negative value.
(Complete 27/27 reverses the prior unequal-n haiku>opus>qwen ordering, significantly.)
- retrieval: off 0.480 > on 0.433 F(1,108)=1.5, p=0.22 NOT significant, slightly
negative — the ADR-118 flywheel feature-on-trial formally fails as `beads`.
EXPLAINED by the 2026-07-10 coupling finding: A1-inRange is a no-headroom item
the models know cold, so a retrieved exemplar has nothing to add.
- prompt: TDD 0.533 > ATDD 0.467 > neutral 0.378 F(2,108)=5.7, p=4.5e-3.
Changes:
- scripts/doe-run.ts: cloud path rewired from raw Anthropic fetch to OpenRouter
(vendor/model ids, opus-4.8 not opus-4-7, exact per-call cost from usage.cost).
- docs/implementation/doe-run-2026-07-10.jsonl: raw per-cell data persisted IN-REPO
(fixes the gap that lost the 2026-07-09 run's data).
- DOE-HARNESS-SCREEN-DESIGN.md §Results (DEFINITIVE) + ADR-122 status/history updated;
prior directional run retained, marked SUPERSEDED.
Verified: tsc 0 errors; $0 --stub and --smoke pipelines green before the metered run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Runs the CRIT-1 coupling experiment live on qwen3-coder:30b (Ollama, $0) and
characterizes EXACTLY when the ADR-118 flywheel's retrieval helps generated tests.
Result (N=5/condition, mutation-oracle graded, end-to-end "runs∧kills" response):
- POSITIVE CONTROL (POS-reserved — an arbitrary system-specific rule the model
cannot know a priori, a proxy for project-specific corpus knowledge):
NONE 0.00 / IRRELEVANT 0.00 / RELEVANT 1.00 → gain +1.00 COUPLING VALIDATED.
Without the exemplar qwen can't even write a passing test; with it, a perfect one.
- TEXTBOOK (B4-isLeapYear — the century/400 rule):
NONE 1.00 / RELEVANT 1.00 → NO HEADROOM (the model knows the trap cold).
Conclusion: the coupling mechanism is REAL (the harness demonstrably detects it, so
its nulls are live measurements) but HEADROOM-GATED — retrieval improves tests iff
the corpus supplies knowledge the base model lacks. Generic QE edge cases are
already known to code models (30b AND 7b), which is WHY the ADR-122 DoE rated
retrieval `beads`. This explains the prior honest nulls rather than contradicting them.
Getting here required fixing three measure-the-measurement traps that had produced a
FALSE "no coupling" verdict on the first run:
- baseline floored by 500-token truncation (→ 1400) and empty-input over-reach;
- spec leaked the trap (anchor inputUnderTest spells out the edge) → no headroom →
switched to UNDER-SPECIFIED specs graded against the frozen trap-encoding reference;
- metric conditioned on baseline, hiding retrieval's strongest effect (rescuing a
test that otherwise can't be written) → switched to end-to-end usable-quality.
Also:
- OpenRouter wired as the cloud path (vendor/model ids; anthropic/claude-haiku-4.5
et al. confirmed reachable), replacing the rate-limited Anthropic-direct fetch.
- coupled-anchor.test.ts: adds an honest AQE_LIVE_COUPLING-gated end-to-end test that
fails loudly (never fabricates a pass) rather than asserting the mechanism the
scripted unit test only assumes.
- ADR-118 status → "INFRASTRUCTURE WITH A CHARACTERIZED SIGNAL": the flywheel's
retrieval-tuning pays off iff qe_patterns holds knowledge the base model lacks.
Verified: tsc 0 errors; coupled-anchor 1 passed + 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the CRITICAL/HIGH/MED findings from the brutal-honesty review of the
learning-integrity cluster. The theme: the machinery was real but several claims
outran the evidence. This makes the claims honest and the evidence falsifiable.
CRIT-1 / HIGH-3 (flywheel gated vapor / toy anchor can't move):
- verification/anchors/qe-anchor-v2.json — 5 oracle-verified DISCRIMINATING items
(13 mutants), each with a trap edge a happy-path test misses, so the anchor CAN
actually move. v1's items were too easy to discriminate a retrieval effect.
- scripts/coupling-experiment.ts — falsifiable NONE/IRRELEVANT/RELEVANT controlled
test (N reps, temp>0, $0 mutation-oracle grading). Live run remains blocked on
model availability; the harness refuses to fabricate a verdict.
CRIT-2 (DoE "stats" were a range proxy):
- scripts/doe-run.ts — N real replicates of a FIXED item at temp=0.6 (was 5
different items x1 at temp=0), per-replicate seed, plus a deterministic $0 --stub.
- scripts/doe-aggregate.mjs — real main-effects factorial ANOVA: within-cell error
term, F-test via dependency-free regularized incomplete beta; beads verdict now
gates on p<0.05, not an eyeballed mean delta.
- tests/unit/scripts/doe-aggregate.test.ts — F-CDF + incomplete-beta vs closed
forms/R, planted-effect recovery (p<0.001), null factor (p>0.5), unbalanced cells.
HIGH-4 / MED-6 (over-claimed gate guarantees):
- ADR-120 gains a "Threat model and limits (honest)" section: gateReExecutes
catches a lying log, not a buggy gate (same-process seal+rule).
- ADR-118 status reframed to "infrastructure awaiting a validated signal."
- tests/unit/validation/rule-consistency.test.ts — property test scoping what the
gate actually guarantees (determinism, stable fingerprints, flipped-verdict).
- DOE-HARNESS-SCREEN-DESIGN.md — unequal-n note.
HIGH-5 / MED-7 (self-fulfilling test / brittle scripts):
- tests/unit/learning/mmr-scaling.test.ts — FALSIFIABLE regression for the real
O(n^4) scoreHeldOut hang the MMR_POOL_SIZE cap fixed (96s->420ms at n=300).
- coupled-anchor.test.ts de-faked: relabeled as a MECHANISM/wiring test, documents
that the scripted generator hardcodes the outcome and the real model was an
honest null — no longer readable as evidence that coupling helps.
- tests/unit/validation/sealed-hash-roundtrip.test.ts — regression for the
canonicalJson undefined-key seal bug.
Verification: tsc 0 errors; 33/33 new remediation tests + 290/290 full
validation+flywheel cluster green. No live model run (Ollama offline + Anthropic
account at usage limit) — coupling live-validation surfaced as the one honest
open item, not faked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ruflo daemon's audit worker (now disabled) dumped 9 inconsistently-named
LLM-generated SECURITY_AUDIT_*.json / security-audit-*.json reports to the repo
root — headline "hardcoded secrets" was a false positive (public SauceLabs demo
creds), the rest unverified/inconsistent LLM noise. Ignore the pattern so future
stray reports don't clutter git status. Also ignore the option-C WAL-monitor
local artifacts (.agentic-qe/wal-monitor.*, *.moncopy-*).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reflexion critique-and-revise step sometimes emits an opening ```javascript
fence with no closing ```; the old match-a-closed-block regex then left the fence
line in the generated test → a syntax error → false baseline failure. Strip fence
lines directly so unclosed fences no longer inflate cell failures. Verified
2026-07-09: does not change the ADR-122 scaffold verdict (reflexion's hit is a
genuine over-iteration effect), only removes cross-cell noise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the receipt-gated self-improvement discipline from the 2026-07 cross-repo
analysis of ruflo (flywheel), MetaHarness (gateReExecutes), and retort (two-gate
spec conformance + DoE). All DI'd and unit-tested (178 tests); run end-to-end on
real qe_patterns; every limitation honestly scoped.
- ADR-117 frozen oracle anchor: hash-pinned, constant-denominator eval set that
no promotion may regress; loader throws on drift (verification/anchors/qe-anchor-v1.json).
- ADR-118 receipt-gated QE-policy flywheel: generational loop over qe_patterns,
conjunctive frozen accept(), Ed25519-signed receipts + compounding lineage,
durable persistence (flywheel_receipts), corpus + semantic-retriever coupling.
- ADR-119 two-gate quality verdicts: mechanical gate + frontier judge, three-valued
(pass/fail/inconclusive) with preflight; new `aqe quality-gate` CLI + `qe/quality/gate` MCP tool.
- ADR-120 gateReExecutes: re-execute a versioned frozen rule on sealed inputs;
catches the A8-EXT forged-promote class; retrofit onto pattern-promote + dream-apply.
- ADR-121 provenance tiers: only oracle:test-exec evidence changes fleet behavior;
additive, conservative-backfill migration.
- ADR-122 DoE screen: model x prompt x retrieval x scaffold, run on real data
(25/27 cells, $1.25) — scaffolding (esp. reflexion) and retrieval degrade
reliability (the retort `beads` signature); simplest+cheapest stack wins.
- safe-db: AQE_DISABLE_WAL / AQE_JOURNAL_MODE env-gate (macOS virtiofs bind-mount
WAL corruption mitigation; default behavior unchanged).
Honest nulls documented: the flywheel lift is on a self-supervised proxy with the
anchor held flat; making the coupled anchor move needs harder items + a relevant
corpus (an ADR-117 re-freeze), not a retriever change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real-time hook learning-capture was failing on this macOS Docker bind mount
with `Failed to initialize UnifiedMemoryManager: database is locked`, and a
brain-checkpoint hook was leaking deadlocked `brain export` subprocesses
(4 found, ~1.5GB RSS). Root cause is SQLite concurrency on a WAL-hostile
virtiofs mount, NOT a native-binary mismatch (better-sqlite3 loads fine).
- brain-checkpoint.cjs: add AQE_DISABLE_BRAIN_CHECKPOINT env / marker-file kill
switch (opt-in; shipped installs unaffected). The export spawns a native RVF
writer that deadlocks in a futex and ignores the SIGTERM the timeout sends,
so orphaned exporters piled up. Disabling stops the leak at the source.
- aqe-hook.cjs: make the FATAL health-log fix-hint conditional. Only suggest
`npm rebuild better-sqlite3` for real native markers; lock-contention
failures now point at the actual cause instead of sending users down the
wrong path.
- feature-flags.ts: add the missing RUVECTOR_USE_RVF_PATTERN_STORE env override
(the only RuVector flag lacking one) so an RVF FsyncFailed can fall back to
the SQLite HNSW PatternStore. Activates after build.
- scripts/aqe-db-backup.sh: consistent VACUUM INTO snapshots (safe with
concurrent writers), integrity-verified before promotion, rotating + daily,
to the host mount so they survive container loss. Restore round-trip tested.
- devcontainer.json: auto-restart the backup loop (postStartCommand) and set
bind-mount env mitigations (remoteEnv: AQE_DISABLE_WAL,
RUVECTOR_USE_RVF_PATTERN_STORE=false, AQE_DISABLE_BRAIN_CHECKPOINT).
- learning-config.json: clear the stale rebuildMode flag (no src reader; on
since 2025-12).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dragan Spiridonov