A qe-court adversarial review (cross-vendor Codex GPT + AQE devils-advocate +
security prosecutors, filing blind) BLOCKED the release by catching edge cases the
green unit suite missed. All confirmed charges fixed:
FATAL:
- F1 codex.ts / claude-code.ts: no 'error' listener on child.stdin — an early
child exit (not-logged-in path) raised an uncaught EPIPE that crashed the host.
Added a stdin error guard + try/catch (both providers — same template bug).
- F2 db-recovery.ts: non-atomic copyFileSync + no lock let concurrent hook
processes tear the restored memory.db. Now: copy to a temp on the same fs →
atomic rename, under a best-effort exclusive lock (skip if a peer is restoring).
- F3 db-recovery.ts: isDbHealthy opened readonly + integrity_check, which
FALSE-NEGATIVES a HEALTHY WAL db when -shm is zeroed over virtiofs — the exact
target failure mode — and would have parked the good DB + reverted to an older
backup (silent data loss). Now the live-corruption signal is the SQLite HEADER:
a -wal/-shm fault never corrupts the main file, so a header-valid DB is treated
healthy and NEVER parked. Backups still get a full integrity_check before use.
SIGNIFICANT:
- spend-ledger.ts: a swallowed ALTER migration made every INSERT hit a missing
column, throw, get caught, and silently drop ALL spend → budget-cap bypass.
record() now tracks whether the column exists and INSERTs accordingly (spend is
always recorded; attribution degrades gracefully).
- witness-verify.ts: verifyAqeReceipt was fail-OPEN under an empty allowlist (the
default state) — a self-signed forgery returned valid:true. Now fail-closed
('no-trust-root') unless an explicit allowSelfSigned opt-in; harness path asserts
alg==='ed25519'; unparseable allowlist JSON now logs loudly.
- gate-reexecute.ts accept/v1+sig: enforce a minimum paired-sample size (n>=8 —
below it a "95% CI" is just the raw delta) and assert mean(samples)==sealed
scalar (defends against a forged scalar/vector pairing).
Verified: qe-court acceptance eval 100%; 1498 shared/validation/llm/flywheel tests
pass (incl. new F2/F3 regressions, fail-closed, alg, min-n, consistency cases);
tsc clean; build green. Deferred (non-blocking rough edges, noted in the court
record): codex maxTokens/temperature passthrough, healthCheck-while-logged-out,
semaphore microtask race, stableStringify byte-parity (needs a real qe-harness sample).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes a real gap: AQE's frozen accept/v1 promotes when the candidate's held-out
MEAN merely beats baseline by ANY amount (gate-reexecute.ts:72) — i.e. it can
promote a within-noise gain. ruflo's flywheel already gates on significance
(accept/v1+sig, paired bootstrap CI > 0); this is the symmetric ADOPTION of that
technique (reimplemented dependency-free — AQE does not import ruflo).
- src/validation/paired-bootstrap.ts: deterministic (fixed-seed LCG) one-sided
95% lower bound on the mean of PAIRED per-task deltas. Determinism is required —
the promotion is replay-verified (ADR-120), so the bound must reproduce exactly.
- gate-reexecute.ts: new frozen rule `accept/v1+sig` = accept/v1 AND
bootstrapDeltaCILow(paired held-out deltas) > 0. accept/v1 is UNTOUCHED (added a
new registry key the ADR-120 way). Fail-closed: rejects if paired samples are
absent/mismatched. SealedInputs gains optional per-task sample vectors (omitted
from the hash when undefined → accept/v1 receipts byte-identical).
- corpus-scorer.ts: scoreHeldOut now returns the per-query reciprocal-rank vector
(the paired sample) alongside its mean → PolicyScores.heldOutSamples.
- generation.ts: seals the samples ONLY when ruleVersion === 'accept/v1+sig'
(opt-in); default accept/v1 behavior + receipt bytes unchanged.
Why paired-bootstrap not ANVOA: for 2 groups ANOVA F = t² and unpaired; the
paired CI exploits the task-pairing and is strictly stronger. (AQE's ADR-122
ANOVA stays the right tool for offline multi-factor screening — complementary.)
Verified: new accept/v1+sig tests incl. the marquee gap case (a within-noise gain
accept/v1 PROMOTES but accept/v1+sig REJECTS); 321 flywheel+validation tests pass;
tsc clean; build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the structural gap from issue #557: AQE could only bill the paid
Anthropic API even for Pro/Max subscribers, and its budget caps were never
enforced.
- claude-code provider: run QE via `claude -p` on the user's subscription;
child env strips API-billing keys so it can't revert to API billing
- Enforced cross-process spend caps (--max-budget-usd / AQE_MAX_BUDGET_USD),
persisted to memory.db (llm_spend); enforced on the ProviderManager AND
HybridRouter paths; over-budget aborts with COST_LIMIT_EXCEEDED
- cognitum provider: OpenAI/Anthropic-compatible gateway with per-request
cost receipts, embeddings, and a server-side hard cap
- Billing-mode taxonomy + startup notice + `aqe health` LLM Billing section
- config-store: honor explicit `enabled: false` over env key presence;
AQE_LLM_PROVIDER selector
- Fix: `aqe eval run` no longer scores fabricated mock responses — real
provider by default, hard-error when none configured, `--mock` opt-in
- Consensus verification path prints a billing warning (deferred full port)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the CRITICAL/HIGH/MED findings from the brutal-honesty review of the
learning-integrity cluster. The theme: the machinery was real but several claims
outran the evidence. This makes the claims honest and the evidence falsifiable.
CRIT-1 / HIGH-3 (flywheel gated vapor / toy anchor can't move):
- verification/anchors/qe-anchor-v2.json — 5 oracle-verified DISCRIMINATING items
(13 mutants), each with a trap edge a happy-path test misses, so the anchor CAN
actually move. v1's items were too easy to discriminate a retrieval effect.
- scripts/coupling-experiment.ts — falsifiable NONE/IRRELEVANT/RELEVANT controlled
test (N reps, temp>0, $0 mutation-oracle grading). Live run remains blocked on
model availability; the harness refuses to fabricate a verdict.
CRIT-2 (DoE "stats" were a range proxy):
- scripts/doe-run.ts — N real replicates of a FIXED item at temp=0.6 (was 5
different items x1 at temp=0), per-replicate seed, plus a deterministic $0 --stub.
- scripts/doe-aggregate.mjs — real main-effects factorial ANOVA: within-cell error
term, F-test via dependency-free regularized incomplete beta; beads verdict now
gates on p<0.05, not an eyeballed mean delta.
- tests/unit/scripts/doe-aggregate.test.ts — F-CDF + incomplete-beta vs closed
forms/R, planted-effect recovery (p<0.001), null factor (p>0.5), unbalanced cells.
HIGH-4 / MED-6 (over-claimed gate guarantees):
- ADR-120 gains a "Threat model and limits (honest)" section: gateReExecutes
catches a lying log, not a buggy gate (same-process seal+rule).
- ADR-118 status reframed to "infrastructure awaiting a validated signal."
- tests/unit/validation/rule-consistency.test.ts — property test scoping what the
gate actually guarantees (determinism, stable fingerprints, flipped-verdict).
- DOE-HARNESS-SCREEN-DESIGN.md — unequal-n note.
HIGH-5 / MED-7 (self-fulfilling test / brittle scripts):
- tests/unit/learning/mmr-scaling.test.ts — FALSIFIABLE regression for the real
O(n^4) scoreHeldOut hang the MMR_POOL_SIZE cap fixed (96s->420ms at n=300).
- coupled-anchor.test.ts de-faked: relabeled as a MECHANISM/wiring test, documents
that the scripted generator hardcodes the outcome and the real model was an
honest null — no longer readable as evidence that coupling helps.
- tests/unit/validation/sealed-hash-roundtrip.test.ts — regression for the
canonicalJson undefined-key seal bug.
Verification: tsc 0 errors; 33/33 new remediation tests + 290/290 full
validation+flywheel cluster green. No live model run (Ollama offline + Anthropic
account at usage limit) — coupling live-validation surfaced as the one honest
open item, not faked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the receipt-gated self-improvement discipline from the 2026-07 cross-repo
analysis of ruflo (flywheel), MetaHarness (gateReExecutes), and retort (two-gate
spec conformance + DoE). All DI'd and unit-tested (178 tests); run end-to-end on
real qe_patterns; every limitation honestly scoped.
- ADR-117 frozen oracle anchor: hash-pinned, constant-denominator eval set that
no promotion may regress; loader throws on drift (verification/anchors/qe-anchor-v1.json).
- ADR-118 receipt-gated QE-policy flywheel: generational loop over qe_patterns,
conjunctive frozen accept(), Ed25519-signed receipts + compounding lineage,
durable persistence (flywheel_receipts), corpus + semantic-retriever coupling.
- ADR-119 two-gate quality verdicts: mechanical gate + frontier judge, three-valued
(pass/fail/inconclusive) with preflight; new `aqe quality-gate` CLI + `qe/quality/gate` MCP tool.
- ADR-120 gateReExecutes: re-execute a versioned frozen rule on sealed inputs;
catches the A8-EXT forged-promote class; retrofit onto pattern-promote + dream-apply.
- ADR-121 provenance tiers: only oracle:test-exec evidence changes fleet behavior;
additive, conservative-backfill migration.
- ADR-122 DoE screen: model x prompt x retrieval x scaffold, run on real data
(25/27 cells, $1.25) — scaffolding (esp. reflexion) and retrieval degrade
reliability (the retort `beads` signature); simplest+cheapest stack wins.
- safe-db: AQE_DISABLE_WAL / AQE_JOURNAL_MODE env-gate (macOS virtiofs bind-mount
WAL corruption mitigation; default behavior unchanged).
Honest nulls documented: the flywheel lift is on a self-supervised proxy with the
anchor held flat; making the coupled anchor move needs harder items + a relevant
corpus (an ADR-117 re-freeze), not a retriever change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the biggest remaining drift gap (ADR-114 follow-up).
- skill-parity now gates plugins/agentic-qe-fleet/skills as a strict body-mirror alongside
assets/skills; reconcileBody + `check-skill-parity --sync` resync drifted mirrors while
preserving each tree's frontmatter (used to fix the plugins qe-quality-assessment drift)
- .kiro/skills is an intentionally divergent 86-skill variant (qe- prefixed, some large
.kiro-only bodies) — gated by PRESENCE via the conservation guard's kiro-skills surface,
not body-equality (which would destroy the divergence)
- both guards (verify:skill-parity blocking, verify:conservation non-blocking) wired into
invariant-check.yml with plugins/.kiro paths; policy doc updated
- tests: reconcileBody/splitFrontmatter covered; 16 validation tests pass
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only EXECUTED/STATIC findings from failed blocking steps may block or
halt the pipeline. INFERRED findings queue in PipelineResult.
needsVerification (the ADR-102 adversarial-verification dispatch
surface) and downgrade overall to warn. CONJECTURE never gates. A
failed blocking step without gating evidence reads as warn, never as a
clean pass. Markdown report shows evidence class on blockers and a
'Needs Verification' section.
Verified: tsc clean; 8 new gating tests + 7 MCP handler tests, 15/15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Single commit covering the 5 concrete findings from the qe-devils-advocate
review. Does not change the underlying CommandEvalRunner design or the 5
primitives — only the claims around them and the fixture-server hack the
yaml had silently inherited.
1. Revert promotion (G4): trust_tier 3 -> 2 in SKILL.md; ADR-091 status
Implemented -> In Progress (2026-04-21); v3-adrs.md index row updated
to reflect the same; counts 80->79 Implemented, 6->7 In Progress.
The promotion gate is now documented on ADR-091: first green CI run
of test-qe-browser.yml (unit + smoke + eval against real Vibium)
before trust_tier 3 / Implemented are restored.
2. Fix yaml fixtures (G3): tc006/tc007 were referencing
http://localhost:8088/qe-browser/SKILL.md.html which required an
unstarted fixture server, so `aqe eval run --skill qe-browser` would
have failed on a fresh machine. Rewrote both to use https://httpbin.org/html
(matching scripts/smoke-test.sh tc006/tc007). Dropped tc009
(poisoned-page check-injection) from the yaml — it needs the local
serve-skills.js fixture which is out of scope for CommandEvalRunner.
Documented as an explicit GAP comment in the yaml, noted that
check-injection.js's severity path is unit-tested at
tests/unit/scripts/qe-browser-check-injection.test.ts. yaml test_cases
now 11 -> 10.
3. Correct pin comment wording (G1): the previous comment claimed
^26.3.18 was a "major.minor-line pin" — that's factually wrong under
npm semver (^26.3.18 accepts 26.99.x). Rewritten to accurately
describe the intent: major-line pin that blocks 27.0+ while
allowing auto-uptake of 26.x security patches and additive features,
with scripts/smoke-test.sh as the belt-and-suspenders API contract
gate. The spec itself stays ^26.3.18 — the correct choice for an
end-user tool under semver assumptions.
4. Add eval CI job (G2): third job in test-qe-browser.yml runs
`aqe eval run --skill qe-browser --model claude-3.5-sonnet` against
a real Vibium install after `npm run build`. This is the
end-to-end proof that CommandEvalRunner dispatch works that the
previous unit tests (all mocked CommandRunner) could not provide.
Uploads the eval JSON as a CI artifact for post-run inspection.
actionlint clean.
5. Scrub stray files (G5): git rm the coherence-comparison artifacts
from docs/reports/ that were swept in by a prior `git add -A`, and
added a focused pattern to .gitignore (not a blanket docs/reports/
ignore) so future sessions can't repeat the mistake while intentional
reports can still be committed.
Tests: 51 relevant tests still pass (3 files: command-eval-runner,
command-eval-runner-qe-browser, browser-engine-installer).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes GAP-05, GAP-08, and final verification gate.
- Linux ARM64 diagnostic (GAP-05): browser-engine-installer now probes
os.platform + os.arch on Linux aarch64 and checks a prioritized list
of system chromium locations (/usr/bin/chromium, chromium-browser,
google-chrome, google-chrome-stable, /snap/bin/chromium). When a
browser is found, the install result carries a PlatformHint with an
exact VIBIUM_BROWSER_PATH export the user can copy. When none exists,
the hint surfaces the `apt-get install chromium` remediation. Wired
into phase 09 so it prints after vibium install succeeds. 5 new tests.
- GAP-08: added explicit `vibium --headless viewport 1280 720` setup
steps to yaml tc006/tc007 so the eval runs are as deterministic in CI
as smoke-test.sh. Updated smoke-test.sh header to note the yaml is
the canonical spec and the bash script is its mirror (covers the F1
vibium-missing contract case the yaml can't express cleanly).
- Trust tier promoted 2 -> 3 now that (a) the eval is runnable via
CommandEvalRunner, (b) CI runs both unit tests + smoke gate, and
(c) all 11 dependent browser-using skills either reference qe-browser
or document their intentional Playwright fallback.
- ADR-091 status: Proposed -> Implemented (2026-04-21).
- v3-adrs.md index reflects 80 Implemented / 6 In Progress / 3 Proposed.
Verification: 643/643 tests pass across tests/unit/validation/,
tests/unit/init/, tests/unit/scripts/qe-browser-*.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes GAP-01, GAP-02, GAP-04 from the ADR-091 completion plan.
- Adds CommandEvalRunner (src/validation/command-eval-runner.ts) as a
sibling to ParallelEvalRunner. Detects shell-based eval suites by
shape (first test case has input.command), runs setup/main commands
sequentially, and evaluates exit_code, json_fields (dotted JSONPath),
severity_at_least (none < low < medium < high < critical), and
candidate_count_at_least assertions against script JSON envelopes.
- Wires aqe eval run to dispatch to CommandEvalRunner when the suite is
command-mode. LLM-prompt suites keep the pre-existing path unchanged.
- Activates .claude/skills/qe-browser/evals/qe-browser.yaml (status:
active, was design-spec) and removes the honesty note now that the
runner supports its structured assertions.
- Pins vibium@^26.3.18 via DEFAULT_VIBIUM_SPEC in browser-engine-
installer.ts so npm install -g can't silently upgrade past a version
we've verified against the 5 primitives + smoke tests.
- Drops qe-browser SKILL.md trust_tier from 3 to 2 pending eval
integration in CI; will re-promote once the full verification pass
runs green.
Tests: 31 new cases (pure helpers, runner integration with injected
runner, happy-path against the shipped qe-browser.yaml).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
125 test files had beforeEach/beforeAll but no afterEach/afterAll,
leaking state between tests and causing order-dependent failures.
Added cleanup to 104 files (skipped 4 archived, 16 import-only):
- 2 critical DB handle leaks (witness-chain): added db.close()
- 7 high-risk service files: added vi.restoreAllMocks()
- 7 beforeAll files: added matching afterAll
- 88 medium/low risk: added afterEach with appropriate cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add three subsystems for fleet orchestration intelligence:
- Pre-spawn MCP validation scans agent files for tool references and
validates availability at install and spawn time (advisory only)
- Agent dependency graph with YAML frontmatter parsing, topological
sort, and phased spawn plans for multi-agent orchestration
- Co-execution repository tracks agent pair success rates, feeding
behavioral signals into the signal merger for routing decisions
Wire all subsystems into the queen coordinator: dependency graph built
on initialize(), MCP validation in requestAgentSpawn(), co-execution
recording in handleTaskCompletionCallback(), and getSpawnPlan()/
getDependencyGraph() exposed on the public interface.
Add structured dependencies frontmatter to 5 agent definitions and
475 passing tests including 7 end-to-end integration tests proving
the wiring works.
Closes#342
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements three improvements from Claude Blog skill-creator analysis:
- Trigger Optimizer: analyzes skill descriptions/tags for false positive/negative activation risks
- Version Comparator: A/B testing between skill versions with Cohen's d effect size
- Skill Intent: classify skills as capability_uplift, encoded_preference, or hybrid
Includes 52 unit tests (23 + 29), schema sync across all 3 copies, and ADR-056 update.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dragan Spiridonov